On Mon, May 8, 2017 at 12:10 PM, Galder Zamarreño <gal...@redhat.com> wrote:
> I think there's some value in Radim's suggestion. The email was not fully > clear to me initially but after reading a few times I understood what he > was referring to. @Radim, correct me if I'm wrong... > > Right now clients verify that they behave as expected, e.g. JS client uses > its asserts, Java client uses other asserts. What Radim is trying to say is > that there needs to be a way to verify they work adequately independent of > their implementations. > > So, the only way to do that is to verify it at the server level. > Not sure what exactly he means by the fake server, but more than a fake > server, I'd be more inclined to modify the server to that it can somehow > act as TCK verifier. We had a thread about Hot Rod testing last year, and another possible strategy is to use real unmodified servers have the TCK written once in a neutral language, compile/distribute it to each of the clients, where the tests would run as part of the build. More details on [1] [1] http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Hot-Rod-testing-tt4031152.html#a4031213 Gustavo > This is to avoid having to reimplement transport logic, protocol > decoder...etc in a new fake server. > > Cheers, > -- > Galder Zamarreño > Infinispan, Red Hat > > > On 11 Apr 2017, at 15:57, Radim Vansa <rva...@redhat.com> wrote: > > > > Since these tests use real server(s), many of them test not only the > > client behaviour (generating correct commands according to the > > protocol), but server, too. While this is practical (we need to test > > server somehow, too), there's nothing all the tests across languages > > will have physically in common and all comparison is prone to human > error. > > > > If we want to test various implementations of the client, maybe it would > > make sense to give the clients a fake server that will have just a > > scenario of expected commands to receive and pre-defined responses. We > > could use audit log to generate such scenario based on the actual Java > > tests. > > > > But then we'd have to test the actual behaviour on server, and we'd need > > a way to issue the commands. > > > > Just my 2c > > > > Radim > > > > On 04/11/2017 02:33 PM, Martin Gencur wrote: > >> Hello all, > >> we have been working on https://issues.jboss.org/browse/ISPN-7120. > >> > >> Anna has finished the first step from the JIRA - collecting information > >> about tests in the Java HotRod client test suite (including server > >> integration tests) and it is now prepared for wider review. > >> > >> She created a spreadsheet [1]. The spread sheet includes for each Java > >> test its name, the suggested target package in the TCK, whether to > >> include it in the TCK or not, and some other notes. The suggested > >> package also poses grouping for the tests (e.g. tck.query, tck.near, > >> tck.xsite, ...) > >> > >> Let me add that right now the goal is not to create a true TCK [2]. The > >> goal is to make sure that all implementations of the HotRod protocol > >> have sufficient test coverage and possibly the same server side of the > >> client-server test (including the server version and configuration). > >> > >> What are the next step? > >> > >> * Please review the list (at least a quick look) and see if some of the > >> tests which are NOT suggested for the TCK should be added or vice versa. > >> * I suppose the next step would then be to check other implementations > >> (C#, C++, NodeJS, ..) and identify tests which are missing there (there > >> will surely be some). > >> * Gradually implement the missing tests in the other implementations > >> Note: Here we should ensure that the server is configured in the same > >> way for all implementations. One way to achieve this (thanks Anna for > >> suggestion!) is to have a shell/batch scripts for CLI which would be > >> executed before the tests. This can probably be done for all impls. and > >> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes > >> useless because it uses Creaper (Java) and we need a language-neutral > >> solution for configuring the server. > >> > >> Some other notes: > >> * there are some duplicated tests in hotrod-client and server > >> integration test suites, in this case it probably makes sense to only > >> include in the TCK the server integration test > >> * tests from the hotrod-client module which are supposed to be part of > >> the TCK should be copied to the server integration test suite one day > >> (possibly later) > >> > >> Please let us know what you think. > >> > >> Thanks, > >> Martin > >> > >> > >> [1] > >> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_ > EA0giQNDZWzFNPWrF5G4/edit#gid=0 > >> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit > >> [3] https://github.com/infinispan/infinispan/pull/5012 > >> _______________________________________________ > >> infinispan-dev mailing list > >> infinispan-dev@lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > -- > > Radim Vansa <rva...@redhat.com> > > JBoss Performance Team > > > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev@lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >
_______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev