On 05/08/2017 07:10 AM, Galder Zamarreño wrote: > I think there's some value in Radim's suggestion. The email was not fully > clear to me initially but after reading a few times I understood what he was > referring to. @Radim, correct me if I'm wrong... > > Right now clients verify that they behave as expected, e.g. JS client uses > its asserts, Java client uses other asserts. What Radim is trying to say is > that there needs to be a way to verify they work adequately independent of > their implementations. > > So, the only way to do that is to verify it at the server level. Not sure > what exactly he means by the fake server, but more than a fake server, I'd be > more inclined to modify the server to that it can somehow act as TCK > verifier. This is to avoid having to reimplement transport logic, protocol > decoder...etc in a new fake server.
I think you got the idea. I am not trying to push any particular implementation of the "fake server" - you could just tweak existing one, but the purest and most deterministic approach would be having a script that could look like: expect connection A to serverX/any server expect receive from A <base64 encoded bytes/hex encoded bytes> send to A <... bytes> expect connection A closed Implementing a server that interprets such script isn't that complex; you don't have to deal with protocol decoder (what's transport logic on server?), because you just expect and send bytes. Radim > > Cheers, > -- > Galder Zamarreño > Infinispan, Red Hat > >> On 11 Apr 2017, at 15:57, Radim Vansa <rva...@redhat.com> wrote: >> >> Since these tests use real server(s), many of them test not only the >> client behaviour (generating correct commands according to the >> protocol), but server, too. While this is practical (we need to test >> server somehow, too), there's nothing all the tests across languages >> will have physically in common and all comparison is prone to human error. >> >> If we want to test various implementations of the client, maybe it would >> make sense to give the clients a fake server that will have just a >> scenario of expected commands to receive and pre-defined responses. We >> could use audit log to generate such scenario based on the actual Java >> tests. >> >> But then we'd have to test the actual behaviour on server, and we'd need >> a way to issue the commands. >> >> Just my 2c >> >> Radim >> >> On 04/11/2017 02:33 PM, Martin Gencur wrote: >>> Hello all, >>> we have been working on https://issues.jboss.org/browse/ISPN-7120. >>> >>> Anna has finished the first step from the JIRA - collecting information >>> about tests in the Java HotRod client test suite (including server >>> integration tests) and it is now prepared for wider review. >>> >>> She created a spreadsheet [1]. The spread sheet includes for each Java >>> test its name, the suggested target package in the TCK, whether to >>> include it in the TCK or not, and some other notes. The suggested >>> package also poses grouping for the tests (e.g. tck.query, tck.near, >>> tck.xsite, ...) >>> >>> Let me add that right now the goal is not to create a true TCK [2]. The >>> goal is to make sure that all implementations of the HotRod protocol >>> have sufficient test coverage and possibly the same server side of the >>> client-server test (including the server version and configuration). >>> >>> What are the next step? >>> >>> * Please review the list (at least a quick look) and see if some of the >>> tests which are NOT suggested for the TCK should be added or vice versa. >>> * I suppose the next step would then be to check other implementations >>> (C#, C++, NodeJS, ..) and identify tests which are missing there (there >>> will surely be some). >>> * Gradually implement the missing tests in the other implementations >>> Note: Here we should ensure that the server is configured in the same >>> way for all implementations. One way to achieve this (thanks Anna for >>> suggestion!) is to have a shell/batch scripts for CLI which would be >>> executed before the tests. This can probably be done for all impls. and >>> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes >>> useless because it uses Creaper (Java) and we need a language-neutral >>> solution for configuring the server. >>> >>> Some other notes: >>> * there are some duplicated tests in hotrod-client and server >>> integration test suites, in this case it probably makes sense to only >>> include in the TCK the server integration test >>> * tests from the hotrod-client module which are supposed to be part of >>> the TCK should be copied to the server integration test suite one day >>> (possibly later) >>> >>> Please let us know what you think. >>> >>> Thanks, >>> Martin >>> >>> >>> [1] >>> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0 >>> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit >>> [3] https://github.com/infinispan/infinispan/pull/5012 >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Radim Vansa <rva...@redhat.com> >> JBoss Performance Team >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Radim Vansa <rva...@redhat.com> JBoss Performance Team _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
