[ https://issues.apache.org/jira/browse/THRIFT-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610256#comment-14610256 ]
Nobuaki Sukegawa commented on THRIFT-3213: ------------------------------------------ {quote} Ideally I would like to see make cross actually try the test anyway, {quote} This can be achieved by removing {{-s}} flag from the command in Makefile.am. Downside of this is the time it takes. Many of them result in timeout with client hang, taking more time than others. I believe it was far less than 50min limit of Travis so it should be definitely doable, though. By the way, I recommend running {{test/test.py}} directly rather than {{make cross}} if you're looking into this. Minimal documentation can be found in {{test/README.md}}. > make cross should indicate when it skips a known failing test > ------------------------------------------------------------- > > Key: THRIFT-3213 > URL: https://issues.apache.org/jira/browse/THRIFT-3213 > Project: Thrift > Issue Type: Improvement > Components: Test Suite > Affects Versions: 0.9.2 > Environment: Ubuntu 12.04 LTS with cpp, java, python, perl, php, and > go enabled. > Reporter: James E. King, III > Assignee: Roger Meier > Priority: Minor > > I do not know if this affects earlier releases. I am trying to get {{cpp}} > and {{go}} to run in a {{make cross}}. Currently I can make cpp on both > sides, and go on both sides, but when I try to mix them up then no tests run: > {noformat} > jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server cpp --client > cpp > Apache Thrift - Integration Test Suite > Tue Jun 30 16:26:24 2015 > ====================================================================== > server-client: protocol: transport: result: > cpp-cpp compact buffered-domain success(0) > cpp-cpp compact framed-ip success(0) > cpp-cpp compact buffered-ip success(0) > cpp-cpp compact http-ip-ssl success(0) > cpp-cpp compact framed-ip-ssl success(0) > cpp-cpp compact buffered-ip-ssl success(0) > cpp-cpp binary buffered-ip-ssl success(0) > cpp-cpp compact framed-domain success(0) > cpp-cpp binary http-ip-ssl success(0) > cpp-cpp binary buffered-ip success(0) > cpp-cpp binary framed-ip-ssl success(0) > cpp-cpp binary buffered-domain success(0) > cpp-cpp binary framed-domain success(0) > cpp-cpp binary framed-ip success(0) > cpp-cpp json http-ip-ssl success(0) > cpp-cpp json buffered-ip success(0) > cpp-cpp json buffered-ip-ssl success(0) > cpp-cpp json buffered-domain success(0) > cpp-cpp json framed-ip-ssl success(0) > cpp-cpp json framed-ip success(0) > cpp-cpp json framed-domain success(0) > ====================================================================== > No unexpected failures. > You can browse results at: > file:///home/jking/thrift/test/result.html > # If you use Chrome, run: > # cd /home/jking/thrift/test > # python -m SimpleHTTPServer 8001 > # then browse: > # http://localhost:8001/result.html > Full log for each test is here: > test/log/client_server_protocol_transport_client.log > test/log/client_server_protocol_transport_server.log > 0 failed of 21 tests in total. > Test execution took 12.6 seconds. > Tue Jun 30 16:26:37 2015 > jking@dvm-4:~/thrift$ ps ax | grep est > 20778 pts/4 S+ 0:00 grep --color=auto est > jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server cpp --client go > Apache Thrift - Integration Test Suite > Tue Jun 30 16:26:45 2015 > ====================================================================== > server-client: protocol: transport: result: > ====================================================================== > No unexpected failures. > You can browse results at: > file:///home/jking/thrift/test/result.html > # If you use Chrome, run: > # cd /home/jking/thrift/test > # python -m SimpleHTTPServer 8001 > # then browse: > # http://localhost:8001/result.html > Full log for each test is here: > test/log/client_server_protocol_transport_client.log > test/log/client_server_protocol_transport_server.log > 0 failed of 0 tests in total. > Test execution took 0.1 seconds. > Tue Jun 30 16:26:45 2015 > jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server go --client go > Apache Thrift - Integration Test Suite > Tue Jun 30 16:26:49 2015 > ====================================================================== > server-client: protocol: transport: result: > go-go compact framed-ip success(0) > go-go binary buffered-ip success(0) > go-go compact buffered-ip success(0) > go-go compact framed-ip-ssl success(0) > go-go compact buffered-ip-ssl success(0) > go-go binary buffered-ip-ssl success(0) > go-go json buffered-ip success(0) > go-go binary framed-ip success(0) > go-go json framed-ip success(0) > go-go json buffered-ip-ssl success(0) > go-go json framed-ip-ssl success(0) > go-go binary framed-ip-ssl success(0) > ====================================================================== > No unexpected failures. > You can browse results at: > file:///home/jking/thrift/test/result.html > # If you use Chrome, run: > # cd /home/jking/thrift/test > # python -m SimpleHTTPServer 8001 > # then browse: > # http://localhost:8001/result.html > Full log for each test is here: > test/log/client_server_protocol_transport_client.log > test/log/client_server_protocol_transport_server.log > 0 failed of 12 tests in total. > Test execution took 6.3 seconds. > Tue Jun 30 16:26:55 2015 > jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server go --client cpp > Apache Thrift - Integration Test Suite > Tue Jun 30 16:26:59 2015 > ====================================================================== > server-client: protocol: transport: result: > ====================================================================== > No unexpected failures. > You can browse results at: > file:///home/jking/thrift/test/result.html > # If you use Chrome, run: > # cd /home/jking/thrift/test > # python -m SimpleHTTPServer 8001 > # then browse: > # http://localhost:8001/result.html > Full log for each test is here: > test/log/client_server_protocol_transport_client.log > test/log/client_server_protocol_transport_server.log > 0 failed of 0 tests in total. > Test execution took 0.1 seconds. > Tue Jun 30 16:26:59 2015 > {noformat} > By all accounts, the tests.json file indicates these tests have commonality > and should be tested against each-other. Curiously when I use java as the > server and go as the client, it works: > {noformat} > jking@dvm-4:~/thrift$ /usr/bin/python test/test.py -s --server java --client > go > Apache Thrift - Integration Test Suite > Tue Jun 30 16:37:52 2015 > ====================================================================== > server-client: protocol: transport: result: > java-go compact framed-ip success(0) > java-go compact fastframed-framed-ip success(0) > java-go compact buffered-ip success(0) > java-go compact framed-ip-ssl success(0) > java-go compact buffered-ip-ssl success(0) > java-go compact fastframed-framed-ip-ssl success(0) > java-go binary framed-ip success(0) > java-go binary buffered-ip success(0) > java-go binary fastframed-framed-ip success(0) > java-go binary framed-ip-ssl success(0) > java-go binary fastframed-framed-ip-ssl success(0) > java-go binary buffered-ip-ssl success(0) > java-go json framed-ip success(0) > java-go json framed-ip-ssl success(0) > java-go json buffered-ip success(0) > java-go json fastframed-framed-ip success(0) > java-go json buffered-ip-ssl success(0) > java-go json fastframed-framed-ip-ssl success(0) > ====================================================================== > {noformat} > Assigning this to [~roger.meier] for initial triage. > It turns out that this is a function of the > {{test/known_failures_Linux.json}} file. If a test is omitted because it is > specifically skipped, there should be an entry on the output that indicates > it was skipped so that it doesn't look like something went wrong. Ideally I > would like to see {{make cross}} actually try the test anyway, and then if it > fails and the entry is in the known failures list, then mark it as skipped. > That way tests that were marked as failed in the past will be tried and might > even succeed, giving better test coverage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)