[
https://issues.apache.org/jira/browse/CALCITE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231144#comment-15231144
]
Josh Elser commented on CALCITE-1190:
-------------------------------------
Some progress on what I've been hacking together. A simple Java program which
can find a bunch of JUnit tests and invoke them given a Driver against a given
Avatica server (a URL). We can enumerate many versions to test via a YAML
config file:
{noformat}
tck_jar:
/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar
versions:
v1.6.0:
client_jar:
/Users/jelser/.m2/repository/org/apache/calcite/calcite-avatica/1.6.0/calcite-avatica-1.6.0.jar
server_url: http://localhost:8765
client_url_template: jdbc:avatica:remote:url=<url>;serialization=PROTOBUF
v1.7.1:
client_jar:
/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.7.1/avatica-1.7.1.jar
server_url: http://localhost:8766
client_url_template: jdbc:avatica:remote:url=<url>;serialization=PROTOBUF
v1.8.0-SNAPSHOT:
client_jar:
/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.8.0-SNAPSHOT/avatica-1.8.0-SNAPSHOT.jar
server_url: http://localhost:8767
client_url_template: jdbc:avatica:remote:url=<url>;serialization=PROTOBUF
{noformat}
Then, a small ruby script will take the versions specified in the above YAML
file, generate a cartesian product of all of the versions (removing identity
mappings so those are already implicitly tested), and invoke the java tests
against each client=>server pair.
I need to write some more tests (just made a simple insert test so far), but
I'm pretty happy with it so far). Some example output:
{noformat}
Running v1.6.0 against v1.7.1
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/calcite-avatica/1.6.0/calcite-avatica-1.6.0.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8766;serialization=PROTOBUF''
2016-04-07 17:41:15,073 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:15,208 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.6.0 against v1.7.1 PASSED!
Running v1.6.0 against v1.8.0-SNAPSHOT
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/calcite-avatica/1.6.0/calcite-avatica-1.6.0.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8766;serialization=PROTOBUF''
2016-04-07 17:41:16,020 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:16,163 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.6.0 against v1.8.0-SNAPSHOT PASSED!
Running v1.7.1 against v1.6.0
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.7.1/avatica-1.7.1.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8765;serialization=PROTOBUF''
2016-04-07 17:41:17,044 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:17,176 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.7.1 against v1.6.0 PASSED!
Running v1.7.1 against v1.8.0-SNAPSHOT
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.7.1/avatica-1.7.1.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8765;serialization=PROTOBUF''
2016-04-07 17:41:18,012 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:18,150 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.7.1 against v1.8.0-SNAPSHOT PASSED!
Running v1.8.0-SNAPSHOT against v1.6.0
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.8.0-SNAPSHOT/avatica-1.8.0-SNAPSHOT.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8765;serialization=PROTOBUF''
2016-04-07 17:41:18,980 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:19,113 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.8.0-SNAPSHOT against v1.6.0 PASSED!
Running v1.8.0-SNAPSHOT against v1.7.1
Java command: 'java -cp
'/Users/jelser/projects/calcite.git/avatica/tck/target/avatica-tck-1.8.0-SNAPSHOT-shaded.jar:/Users/jelser/.m2/repository/org/apache/calcite/avatica/avatica/1.8.0-SNAPSHOT/avatica-1.8.0-SNAPSHOT.jar'
org.apache.calcite.avatica.tck.TestRunner -u
'jdbc:avatica:remote:url=http://localhost:8765;serialization=PROTOBUF''
2016-04-07 17:41:19,984 [main] INFO tck.TestRunner - Running
org.apache.calcite.avatica.tck.tests.InsertTest
2016-04-07 17:41:20,115 [main] INFO tck.TestRunner - Tests run: 1, Failures:
0, Skipped: 0, Time elapsed: 0 - in
org.apache.calcite.avatica.tck.tests.InsertTest
Tests run: 1, Failures: 0
Test of v1.8.0-SNAPSHOT against v1.7.1 PASSED!
All tests finished (ran 6)
{noformat}
I also hacked together a simple Maven project to get an HSQLDB-backed Avatica
server running against older versions of Avatica. I've included the same code
in Avatica going forward.
> Cross-Version Compatibility Test Harness
> ----------------------------------------
>
> Key: CALCITE-1190
> URL: https://issues.apache.org/jira/browse/CALCITE-1190
> Project: Calcite
> Issue Type: Test
> Components: avatica
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: avatica-1.8.0
>
>
> One thing that the Protobuf serialization aimed to provide was a library
> which provides us the tools to make Avatica compatible across versions.
> However, Protobuf is just a tool and the developers can still misuse protobuf
> in such a way that breaks compatibility across versions. Not to mention,
> compatibility isn't even remotely certain without any tests.
> Because of Avatica's position as a library less than a product, we have to
> defer some logic to the concrete product being tested (e.g. Phoenix or
> Drill). I'm thinking something like the following:
> The user provides pairs of client and server "definitions" for a given
> version of compatibility. This would include some version of Avatica and some
> backing database. For example, Avatica-1.7.1 and Phoenix-4.7.0 or
> Avatica-1.8.0-SNAPSHOT and HSQLDB-2.3.1.
> The client half would be some template for the appropriate JDBC url to use
> (sans the URL to the Avatica server) and a JAR file containing the necessary
> classes to run the j.s.Driver. The server half would just be a URL to the
> Avatica server instance.
> The test harness itself can provide the logic to test the remote driver
> against the other servers, enumerating the possible combinations of
> client-server communication. Starting the server for each version to test is
> likely too difficult to automate well given the unknown of what the server
> will be, so that will be left as a prerequisite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
