[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[David Ribeiro Alves (Code Review)]

David Ribeiro Alves has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 22: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 22
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: No

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[David Ribeiro Alves (Code Review)]

David Ribeiro Alves has submitted this change and it was merged.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was
failing) and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other assorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

Restrictions:
  1. Kudu nodes should run recent Debian/Ubuntu Linux distro
 (that's due to the internal Jepsen's restrictions).

  2. The 'kudu-master', 'kudu-tserver' and 'kudu' binaries in
 the $KUDU_HOME/build/latest/bin should be built for the OS/distro
 running on the Kudu cluster nodes.

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Reviewed-on: http://gerrit.cloudera.org:8080/5492
Tested-by: Kudu Jenkins
Reviewed-by: David Ribeiro Alves 
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/resources/ntp.conf.common
A java/kudu-jepsen/resources/ntp.conf.server
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
15 files changed, 1,557 insertions(+), 0 deletions(-)

Approvals:
  David Ribeiro Alves: Looks good to me, approved
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 23
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[David Ribeiro Alves (Code Review)]

David Ribeiro Alves has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 22:

I'm merging this anyway. We can address any docker specific issues in follow up 
patches.

-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 22
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: No

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Alexey Serbin has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 22:

> (7 comments)
 > 
 > final q: does this still work with docker instances?

Last time I checked that before I had robust setup with Jenkins.  That I'll 
verify today.

-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 22
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: No

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Alexey Serbin has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 18:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/5492/18//COMMIT_MSG
Commit Message:

PS18, Line 10: failing)
> nit: wrap this
Done


PS18, Line 15: unsorted
> you mean asorted, no? (i.e. miscellaneous, not out of order)
yup, I meant 'assorted'.  Fixed.


PS18, Line 25: All Jepsen control operations on the DB nodes (i.e. Kudu master 
and
 : tserver nodes) are run via SSH.  The private SSH key should be 
set prior
 : to running the test:
 : 
 :   1. The public part of the SSH key should be added into the
 :  'authorized_keys' file for the root user on all cluster 
nodes.
 : 
 :   2. The private part of the SSH key should be provided to the 
test
 :  either by:
 :* adding the key into the SSH agent on the control node
 :* specifying the path to the key via 'sshKeyPath' property
 : 
 : Having the Kudu cluster provisioned and SSH keys deployed, to run
 : the tests against the cluster with master node m0 and tserver 
nodes
 : {t0..t4}, build the Kudu project from sources and then execute
 : the following in the $KUDU_HOME/java/kudu-jepsen directory:
 : 
 :   mvn clojure:run -DmasterNodes=m0 
-DtserverNodes="t0,t1,t2,t3,t4"
 : 
 : after bulding the top-level project with
 : 
 :   mvn clean compile test-compile -Pjepsen
> don't think this info needs to be here. Is it in the README?
Yep, that also present in the README.adoc, in a little different format.  Will 
remove.


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj:

PS18, Line 87: into
> s/into/to
Done


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj:

PS18, Line 99: but no server sees
 :   the *same* majority as any other.
> is there a guarantee that the majorities are different, or is it just likel
This is a good question!

That part I didn't check -- just took the jepsen's word for it.  These nemeses 
are modeled after the originals in 
$JEPSEN_GIT_ROOT/jepsen/src/jepsen/nemesis.clj, but they are acting within the 
subset of nodes where tservers are running (master nodes are not affected).  
Probably, there is a better way to achieve that, but since I'm not so good at 
Clojure yet, that's what I came up with :)


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj:

PS18, Line 34: 1.2.0
> isn't this 1.3.0-SNAPSHOT or whatever now?
Well, it's repo pkg version, so for the latest snapshot from the main trunk it 
should be just an empty string.  Will update, thank you for pointing at this!


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
File java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj:

PS18, Line 53: (def register-test-configs
 :   [
 :{:suffix "tserver-random-halves"
 : :nemesis '(kn/tserver-partition-random-halves)}
 :{:suffix "tserver-majorities-ring"
 : :nemesis '(kn/tserver-partition-majorities-ring)}
 :{:suffix "kill-restart-2-tservers"
 : :nemesis '(kn/kill-restart-tserver (comp (partial take 2) 
shuffle))}
 :{:suffix "kill-restart-3-tservers"
 : :nemesis '(kn/kill-restart-tserver (comp (partial take 3) 
shuffle))}
 :{:suffix "kill-restart-all-tservers"
 : :nemesis '(kn/kill-restart-tserver shuffle)}
 :{:suffix "all-random-halves"
 : :nemesis '(jn/partition-random-halves)}
 :{:suffix "all-majorities-ring"
 : :nemesis '(jn/partition-majorities-ring)}
 :{:suffix "hammer-2-tservers"
 : :nemesis '(kn/tserver-hammer-time (comp (partial take 2) 
shuffle))}
 :{:suffix "hammer-3-tservers"
 : :nemesis '(kn/tserver-hammer-time (comp (partial take 3) 
shuffle))}
 :{:suffix "hammer-all-tservers"
 : :nemesis '(kn/tserver-hammer-time shuffle)}
 :])
> clap clap! neato, thanks for adding all there scenarios
I'm happy that you find this useful :)  We can add more scenarios down the road.


-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5492

to look at the new patch set (#22).

Change subject: [kudu-jepsen] Kudu Jepsen tests
..

[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was
failing) and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other assorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

Restrictions:
  1. Kudu nodes should run recent Debian/Ubuntu Linux distro
 (that's due to the internal Jepsen's restrictions).

  2. The 'kudu-master', 'kudu-tserver' and 'kudu' binaries in
 the $KUDU_HOME/build/latest/bin should be built for the OS/distro
 running on the Kudu cluster nodes.

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/resources/ntp.conf.common
A java/kudu-jepsen/resources/ntp.conf.server
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
15 files changed, 1,557 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/5492/22
-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 22
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[David Ribeiro Alves (Code Review)]

David Ribeiro Alves has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 18:

(7 comments)

final q: does this still work with docker instances?

http://gerrit.cloudera.org:8080/#/c/5492/18//COMMIT_MSG
Commit Message:

PS18, Line 10: failing)
nit: wrap this


PS18, Line 15: unsorted
you mean asorted, no? (i.e. miscellaneous, not out of order)


PS18, Line 25: All Jepsen control operations on the DB nodes (i.e. Kudu master 
and
 : tserver nodes) are run via SSH.  The private SSH key should be 
set prior
 : to running the test:
 : 
 :   1. The public part of the SSH key should be added into the
 :  'authorized_keys' file for the root user on all cluster 
nodes.
 : 
 :   2. The private part of the SSH key should be provided to the 
test
 :  either by:
 :* adding the key into the SSH agent on the control node
 :* specifying the path to the key via 'sshKeyPath' property
 : 
 : Having the Kudu cluster provisioned and SSH keys deployed, to run
 : the tests against the cluster with master node m0 and tserver 
nodes
 : {t0..t4}, build the Kudu project from sources and then execute
 : the following in the $KUDU_HOME/java/kudu-jepsen directory:
 : 
 :   mvn clojure:run -DmasterNodes=m0 
-DtserverNodes="t0,t1,t2,t3,t4"
 : 
 : after bulding the top-level project with
 : 
 :   mvn clean compile test-compile -Pjepsen
don't think this info needs to be here. Is it in the README?


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj:

PS18, Line 87: into
s/into/to
same below


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj:

PS18, Line 99: but no server sees
 :   the *same* majority as any other.
is there a guarantee that the majorities are different, or is it just likely 
that they are?


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj:

PS18, Line 34: 1.2.0
isn't this 1.3.0-SNAPSHOT or whatever now?


http://gerrit.cloudera.org:8080/#/c/5492/18/java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
File java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj:

PS18, Line 53: (def register-test-configs
 :   [
 :{:suffix "tserver-random-halves"
 : :nemesis '(kn/tserver-partition-random-halves)}
 :{:suffix "tserver-majorities-ring"
 : :nemesis '(kn/tserver-partition-majorities-ring)}
 :{:suffix "kill-restart-2-tservers"
 : :nemesis '(kn/kill-restart-tserver (comp (partial take 2) 
shuffle))}
 :{:suffix "kill-restart-3-tservers"
 : :nemesis '(kn/kill-restart-tserver (comp (partial take 3) 
shuffle))}
 :{:suffix "kill-restart-all-tservers"
 : :nemesis '(kn/kill-restart-tserver shuffle)}
 :{:suffix "all-random-halves"
 : :nemesis '(jn/partition-random-halves)}
 :{:suffix "all-majorities-ring"
 : :nemesis '(jn/partition-majorities-ring)}
 :{:suffix "hammer-2-tservers"
 : :nemesis '(kn/tserver-hammer-time (comp (partial take 2) 
shuffle))}
 :{:suffix "hammer-3-tservers"
 : :nemesis '(kn/tserver-hammer-time (comp (partial take 3) 
shuffle))}
 :{:suffix "hammer-all-tservers"
 : :nemesis '(kn/tserver-hammer-time shuffle)}
 :])
clap clap! neato, thanks for adding all there scenarios


-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 18
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5492

to look at the new patch set (#21).

Change subject: [kudu-jepsen] Kudu Jepsen tests
..

[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was failing)
and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other unsorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

All Jepsen control operations on the DB nodes (i.e. Kudu master and
tserver nodes) are run via SSH.  The private SSH key should be set prior
to running the test:

  1. The public part of the SSH key should be added into the
 'authorized_keys' file for the root user on all cluster nodes.

  2. The private part of the SSH key should be provided to the test
 either by:
   * adding the key into the SSH agent on the control node
   * specifying the path to the key via 'sshKeyPath' property

Having the Kudu cluster provisioned and SSH keys deployed, to run
the tests against the cluster with master node m0 and tserver nodes
{t0..t4}, build the Kudu project from sources and then execute
the following in the $KUDU_HOME/java/kudu-jepsen directory:

  mvn clojure:run -DmasterNodes=m0 -DtserverNodes="t0,t1,t2,t3,t4"

after bulding the top-level project with

  mvn clean compile test-compile -Pjepsen

Restrictions:
  1. Kudu nodes should run recent Debian/Ubuntu Linux distro
 (that's due to the internal Jepsen's restrictions).

  2. The 'kudu-master', 'kudu-tserver' and 'kudu' binaries in
 the $KUDU_HOME/build/latest/bin should be built for the OS/distro
 running on the Kudu cluster nodes.

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/resources/ntp.conf.common
A java/kudu-jepsen/resources/ntp.conf.server
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
15 files changed, 1,555 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/5492/21
-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 21
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5492

to look at the new patch set (#19).

Change subject: [kudu-jepsen] Kudu Jepsen tests
..

[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was failing)
and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other unsorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

All Jepsen control operations on the DB nodes (i.e. Kudu master and
tserver nodes) are run via SSH.  The private SSH key should be set prior
to running the test:

  1. The public part of the SSH key should be added into the
 'authorized_keys' file for the root user on all cluster nodes.

  2. The private part of the SSH key should be provided to the test
 either by:
   * adding the key into the SSH agent on the control node
   * specifying the path to the key via 'sshKeyPath' property

Having the Kudu cluster provisioned and SSH keys deployed, to run
the tests against the cluster with master node m0 and tserver nodes
{t0..t4}, build the Kudu project from sources and then execute
the following in the $KUDU_HOME/java/kudu-jepsen directory:

  mvn clojure:run -DmasterNodes=m0 -DtserverNodes="t0,t1,t2,t3,t4"

after bulding the top-level project with

  mvn clean compile test-compile -Pjepsen

Restrictions:
  1. Kudu nodes should run recent Debian/Ubuntu Linux distro
 (that's due to the internal Jepsen's restrictions).

  2. The 'kudu-master', 'kudu-tserver' and 'kudu' binaries in
 the $KUDU_HOME/build/latest/bin should be built for the OS/distro
 running on the Kudu cluster nodes.

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/resources/ntp.conf.common
A java/kudu-jepsen/resources/ntp.conf.server
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
15 files changed, 1,544 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/5492/19
-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 19
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5492

to look at the new patch set (#18).

Change subject: [kudu-jepsen] Kudu Jepsen tests
..

[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was failing)
and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other unsorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

All Jepsen control operations on the DB nodes (i.e. Kudu master and
tserver nodes) are run via SSH.  The private SSH key should be set prior
to running the test:

  1. The public part of the SSH key should be added into the
 'authorized_keys' file for the root user on all cluster nodes.

  2. The private part of the SSH key should be provided to the test
 either by:
   * adding the key into the SSH agent on the control node
   * specifying the path to the key via 'sshKeyPath' property

Having the Kudu cluster provisioned and SSH keys deployed, to run
the tests against the cluster with master node m0 and tserver nodes
{t0..t4}, build the Kudu project from sources and then execute
the following in the $KUDU_HOME/java/kudu-jepsen directory:

  mvn clojure:run -DmasterNodes=m0 -DtserverNodes="t0,t1,t2,t3,t4"

after bulding the top-level project with

  mvn clean compile test-compile -Pjepsen

Restrictions:
  1. Kudu nodes should run recent Debian/Ubuntu Linux distro
 (that's due to the internal Jepsen's restrictions).

  2. The 'kudu-master', 'kudu-tserver' and 'kudu' binaries in
 the $KUDU_HOME/build/latest/bin should be built for the OS/distro
 running on the Kudu cluster nodes.

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/resources/ntp.conf.common
A java/kudu-jepsen/resources/ntp.conf.server
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
15 files changed, 1,507 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/5492/18
-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 18
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Alexey Serbin has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 16:

(19 comments)

http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/.gitignore
File java/kudu-jepsen/.gitignore:

Line 1: # Ignore files containing information on prior Leiningen runs/sessions.
> add apache header
Done


Line 5: /store
> why do you need '/store' instead of just 'store' or 'store/'?
It seems it's a typo -- should be store/ instead.


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/README.adoc
File java/kudu-jepsen/README.adoc:

PS16, Line 57: The Kudu master and tserver nodes should be created prior to 
running the test:
 : the tests does not create those itself
> can you clarify whether you just mean that the machines should be created, 
Done


PS16, Line 147: liear
> typo
Done


PS16, Line 152: activisation
> activation
Done


PS16, Line 156: correspondning
> corresponding
Done


PS16, Line 158: should under
> should be under
Done


PS16, Line 162: The Jepsen code is distributed under the Eclipse Public License 
either
  : version 1.0 or (at your option) any later version.
> If we're not shipping Jepsen itself, I dont think it's necessary to specify
Done


Line 165: The kudu-jepsen is licensed under the Apache License, Version 2.0
> this should be implicitly covered by the top-level LICENSE file we already 
Done


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/resources/kudu.flags
File java/kudu-jepsen/resources/kudu.flags:

Line 1: # This overrides all flags in a flag file
> nit: add license (or add to rat excludes if not possible)
Done


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/resources/ntp.conf.common
File java/kudu-jepsen/resources/ntp.conf.common:

Line 1: tinker panic 0
> same (license)
Done


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj:

PS16, Line 92: do (let [rr-iter (.nextRows scanner)]
 : (while (.hasNext rr-iter)
 :   (do 
> are these two 'do's necessary given the single-expression loop bodies?
Good point: I found it's not necessary here, but some tutorial have (do) in 
their examples: 
https://www.tutorialspoint.com/clojure/clojure_while_statement.htm

I'll remove that (do) -- at least it seems working without (do).


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj:

PS16, Line 34: respond
> response
Done


PS16, Line 35: respond
> response
Done


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj:

PS16, Line 71: client (atom false) nil nil)
> hrm, not entirely following this here. Does this actually run once per clie
As I understand, the value associated with the :client key in the result map 
returned by this function is used to instantiate all clients for the.  So, 
having that atom is the way to store a state between those invocations -- the 
very first client which sets the atom creates the table, all other clients just 
open already existing table when jepsen calls setup.


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj:

PS16, Line 31: reduce (fn [col e] (conj col e)) [] components)
> can you explain this to me? not following. isn't this equivalent to (reduce
Yes, you are right.  It seems I wrote this code in an iterative way and missed 
the point of simplifying it in the very end :)

The sequence should not be vec-ified, using list here is OK as well.

Basically, it should be just (str/join "/" components)


Line 351:   ;; Copy appropriate binaries to the node.
> can we add some kind of check before copying that the binaries we are going
Good idea, but I would like that to work also with binaries from other 
architecture put into the build/latest/bin: e.g., I use my OS X laptop to copy 
pre-build Debian/Ubuntu binaries into the DB test nodes.

I'll think how to achieve that.


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/utils/kudu_test_runner.clj
File java/kudu-jepsen/src/utils/kudu_test_runner.clj:

PS16, Line 82: {:masters (:master-nodes options)
 :  :tservers (:tserver-nodes options)
 :  :ssh-key-path (:ssh-key-path options)
 :  :iter-num (:iter-num options)}))
> seems odd to be mapping the 'options' map to new keys here, and then again 
ok, passing those transparently would make more sense, yes.


PS16, Line 88: def cmd-line-opts (get-cmd-line-opts))
 :   (def private-key-path (:ssh-key-path cmd-line-opts))

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Todd Lipcon (Code Review)]

Todd Lipcon has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 16:

(20 comments)

http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/.gitignore
File java/kudu-jepsen/.gitignore:

Line 1: # Ignore files containing information on prior Leiningen runs/sessions.
add apache header


Line 5: /store
why do you need '/store' instead of just 'store' or 'store/'?


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/README.adoc
File java/kudu-jepsen/README.adoc:

PS16, Line 57: The Kudu master and tserver nodes should be created prior to 
running the test:
 : the tests does not create those itself
can you clarify whether you just mean that the machines should be created, or 
whether there should actually be a tserver/master installed?


PS16, Line 147: liear
typo


PS16, Line 152: activisation
activation


PS16, Line 156: correspondning
corresponding


PS16, Line 158: should under
should be under


PS16, Line 162: The Jepsen code is distributed under the Eclipse Public License 
either
  : version 1.0 or (at your option) any later version.
If we're not shipping Jepsen itself, I dont think it's necessary to specify the 
license of the Maven dependency


Line 165: The kudu-jepsen is licensed under the Apache License, Version 2.0
this should be implicitly covered by the top-level LICENSE file we already have 
in the repo


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/resources/kudu.flags
File java/kudu-jepsen/resources/kudu.flags:

Line 1: # This overrides all flags in a flag file
nit: add license (or add to rat excludes if not possible)


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/resources/ntp.conf.common
File java/kudu-jepsen/resources/ntp.conf.common:

Line 1: tinker panic 0
same (license)


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj:

PS16, Line 92: do (let [rr-iter (.nextRows scanner)]
 : (while (.hasNext rr-iter)
 :   (do 
are these two 'do's necessary given the single-expression loop bodies?


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj:

PS16, Line 34: respond
response


PS16, Line 35: respond
response


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj:

PS16, Line 71: client (atom false) nil nil)
hrm, not entirely following this here. Does this actually run once per client? 
If so, do you want them to be sharing an atom? it seems like it ends up 
creating a separate atom per client, in which case the locking and CAS on it 
wouldn't do anything?


PS16, Line 73: ;; (count (:tservers test))
 :;; :num_replicas 1
remove this commented-out code


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj:

PS16, Line 31: reduce (fn [col e] (conj col e)) [] components)
can you explain this to me? not following. isn't this equivalent to (reduce 
conj [] components)? And isn't that equivalent to just (vec components)?

(and not sure why it has to be 'vec'ified anyway)?


Line 351:   ;; Copy appropriate binaries to the node.
can we add some kind of check before copying that the binaries we are going to 
copy are not a shared-lib build? eg run 'ldd' and check that it doesn't have 
dependencies in the build dir?


http://gerrit.cloudera.org:8080/#/c/5492/16/java/kudu-jepsen/src/utils/kudu_test_runner.clj
File java/kudu-jepsen/src/utils/kudu_test_runner.clj:

PS16, Line 82: {:masters (:master-nodes options)
 :  :tservers (:tserver-nodes options)
 :  :ssh-key-path (:ssh-key-path options)
 :  :iter-num (:iter-num options)}))
seems odd to be mapping the 'options' map to new keys here, and then again map 
to new keys on line 88-91 below


PS16, Line 88: def cmd-line-opts (get-cmd-line-opts))
 :   (def private-key-path (:ssh-key-path cmd-line-opts))
 :   (def iter-num (:iter-num cmd-line-opts))
 :   (def test-opts (dissoc cmd-line-opts :ssh-key-path :iter-num))
would 'let' be better here?


-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 16
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5492

to look at the new patch set (#17).

Change subject: [kudu-jepsen] Kudu Jepsen tests
..

[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was failing)
and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other unsorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

All Jepsen control operations on the DB nodes (i.e. Kudu master and
tserver nodes) are run via SSH.  The private SSH key should be set prior
to running the test:

  1. The public part of the SSH key should be added into the
 'authorized_keys' file for the root user on all cluster nodes.

  2. The private part of the SSH key should be provided to the test
 either by:
   * adding the key into the SSH agent on the control node
   * specifying the path to the key via 'sshKeyPath' property

Having the Kudu cluster provisioned and SSH keys deployed, to run
the tests against the cluster with master node m0 and tserver nodes
{t0..t4}, build the Kudu project from sources and then execute
the following in the $KUDU_HOME/java/kudu-jepsen directory:

  mvn clojure:run -DmasterNodes=m0 -DtserverNodes="t0,t1,t2,t3,t4"

after bulding the top-level project with

  mvn clean compile test-compile -Pjepsen

Restrictions:
  1. Kudu nodes should run recent Debian/Ubuntu Linux distro
 (that's due to the internal Jepsen's restrictions).

  2. The 'kudu-master', 'kudu-tserver' and 'kudu' binaries in
 the $KUDU_HOME/build/latest/bin should be built for the OS/distro
 running on the Kudu cluster nodes.

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/resources/ntp.conf.common
A java/kudu-jepsen/resources/ntp.conf.server
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
15 files changed, 1,479 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/5492/17
-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 17
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5492

to look at the new patch set (#15).

Change subject: [kudu-jepsen] Kudu Jepsen tests
..

[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was failing)
and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other unsorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

All Jepsen control operations on the DB nodes (i.e. Kudu master and
tserver nodes) are run via SSH.  The private SSH key should be set prior
to running the test:

  1. The public part of the SSH key should be added into the
 'authorized_keys' file for the root user on all cluster nodes.

  2. The private part of the SSH key should be provided to the test
 either by:
   * adding the key into the SSH agent on the control node
   * specifying the path to the key via 'sshKeyPath' property

Having the Kudu cluster provisioned and SSH keys deployed, to run
the tests against the cluster with master node m0 and tserver nodes
{t0..t4}, build the Kudu project from sources and then execute
the following in the $KUDU_HOME/java/kudu-jepsen directory:

  mvn clojure:run -DmasterNodes=m0 -DtserverNodes="t0,t1,t2,t3,t4"

after bulding the top-level project with

  mvn compile -Pjepsen

Restrictions:
  1. Kudu nodes should run recent Debian/Ubuntu Linux distro
 (that's due to the internal Jepsen's restrictions).

  2. The 'kudu-master', 'kudu-tserver' and 'kudu' binaries in
 the $KUDU_HOME/build/latest/bin should be built for the OS/distro
 running on the Kudu cluster nodes.

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/resources/ntp.conf.common
A java/kudu-jepsen/resources/ntp.conf.server
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
15 files changed, 1,386 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/5492/15
-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 15
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Alexey Serbin has posted comments on this change.

Change subject: [kudu-jepsen] Kudu Jepsen tests
..


Patch Set 12:

(13 comments)

http://gerrit.cloudera.org:8080/#/c/5492/12/java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj:

Line 1: (ns jepsen.kudu
> nit: need license headers on this and all other files in this commit
Done


Line 28:   (def flags ["--fs_wal_dir=/var/lib/kudu/master"
> eastwood lint says this is a "def-in-def" which is bad (since it makes 'fla
Done


PS12, Line 32:   (when (> (count (:masters test)) 1)
 : (conj flags (str "--master_addresses=" 
(concatenate-addresses (:masters test)
> confused by this - it seems like a no-op, since the result of the (when...)
Good catch -- yes, indeed.  That's a bug.  Fixed.


Line 46: (def ntp-common-opts ["statistics loopstats peerstats clockstats"
> could you 'slurp' these from a resource file instead?
Done


PS12, Line 64: masters
> I don't think NTP uses the term 'masters'. Probably better to say 'servers'
Done


Line 66:  [(str "server " (name (first masters))
> shouldn't this use all of the configured masters, not just the first? eg so
Done


PS12, Line 69: (defn db
 :   []
 :   "Apache Kudu."
> src/main/clojure/jepsen/kudu.clj:69:7: misplaced-docstrings: Possibly mispl
Done


PS12, Line 151: (into []
> https://github.com/bbatsov/clojure-style-guide says to use (vec ...) instea
Done


http://gerrit.cloudera.org:8080/#/c/5492/12/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
File java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj:

Line 37:   ([name type] (.build (.key (new ColumnSchema$ColumnSchemaBuilder 
name, type) false)))
> can this be defined as (column-schema name type false) to avoid redundancy?
Good idea!  Done.


Line 38:   ([name type key?] (.build (.key (new 
ColumnSchema$ColumnSchemaBuilder name, type) key?
> how about:
Done


PS12, Line 51: columns (.getColumns (.getSchema row-result)
> again maybe the java programmer in me but I think (-> row-result .getSchema
Done


PS12, Line 55: case (.ordinal (.getType column))
 :   ;; Clojure transforms enums in literals
 :   ;; so we have to use ordinals :(
> what about using cond instead?
Good idea -- I replaced that with condp and it works.


PS12, Line 71: ->t
> the style guide I'm looking at says to use -> for "conversion functions" bu
Done: renamed into 'drain-scanner-into-tuples'


-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 12
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] [kudu-jepsen] Kudu Jepsen tests

[Alexey Serbin (Code Review)]

Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5492

to look at the new patch set (#13).

Change subject: [kudu-jepsen] Kudu Jepsen tests
..

[kudu-jepsen] Kudu Jepsen tests

This patch contains David's code for the initial kudu-jepsen tests
as it was before KUDU-798 was resolved (i.e. as it was when it was failing)
and additional updates/fixes:
  * Extra nemeses for the read/write register linearizability test
  * Run multiple test scenarios in the scope of the register test
  * Starting up master server: wait for the catalog manager
  * Other unsorted fixes for more robust operation

The clojure code is integrated into the Kudu maven build and is compiled
along with the other projects in a separate 'jepsen' profile.

The patch also adds functionality to run the kudu-jepsen tests
from the clojure-maven-plugin.  The test uses the build machine
as the Jepsen control node, running the control logic and the Kudu
Java client there.

All Jepsen control operations on the DB nodes (i.e. Kudu master and
tserver nodes) are run via SSH.  The private SSH key should be set prior
to running the test:
  1. The public part of the SSH key should be added into the
 'authorized_keys' file for the root user on all cluster nodes.
  2. The private part of the SSH key should be provided to the test
 either by:
   * adding the key into the SSH agent on the control node
   * specifying the path to the key via 'sshKeyPath' property

Having the Kudu cluster provisioned and SSH keys deployed, to run
the tests against the cluster with master node m0 and tserver nodes
{t0..t4}, build the Kudu project from sources and then execute
the following in the $KUDU_HOME/java/kudu-jepsen directory:

  mvn clojure:run -DmasterNodes=m0 -DtserverNodes="t0,t1,t2,t3,t4"

after bulding the top-level project with

  mvn compile -Pjepsen

Restrictions:
  1. The Kudu cluster should consist of Linux machines of the same
 architecture and OS distro as the machine where the Kudu
 kudu-master, kudu-tserver, and the 'kudu' CLI utility are built.

  2. As for now, the kudu-jepsen test runs on recent Debian distros
 only (jepsen supports only Debian Linux).

Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
---
A java/kudu-jepsen/.gitignore
A java/kudu-jepsen/README.adoc
A java/kudu-jepsen/pom.xml
A java/kudu-jepsen/resources/kudu.flags
A java/kudu-jepsen/src/main/clojure/jepsen/kudu.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/nemesis.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/register.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj
A java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
A java/kudu-jepsen/src/test/clojure/jepsen/kudu_test.clj
A java/kudu-jepsen/src/utils/kudu_test_runner.clj
M java/pom.xml
13 files changed, 1,387 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/5492/13
-- 
To view, visit http://gerrit.cloudera.org:8080/5492
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I590c6e78840304b3131666c7037ff9a08dc77dea
Gerrit-PatchSet: 13
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: David Ribeiro Alves 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Alexey Serbin 
Gerrit-Reviewer: Dan Burkert 
Gerrit-Reviewer: David Ribeiro Alves 
Gerrit-Reviewer: Jean-Daniel Cryans 
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon