Hi Tommaso, Mocking up a third-party system is preferable to having no runnable tests at all, and it would likely prevent us from introducing regressions in the code. On the other hand, such an approach does not support forward development or configuration exploration. As you point out, these systems release new versions periodically, and no doubt people will eventually want to add features to the existing connectors, or support different configurations, for which a mock-up approach will not be sufficient. So I don't think mock-ups will be a complete replacement for end-to-end testing against an actual instance, although as I said, it would be better than nothing. This logic applies most completely to:
- FileNet - Documentum - LiveLink - Meridio - SharePoint Other connectors, such as the web connector and rss connector, are more difficult to test via mockup and need instead something to crawl against. We used a DNS fakeout trick at MetaCarta - basically, messing with the DNS setup to point all servers at the same box, but that's hard to do under Java because it caches DNS, and tests that mess with local DNS settings would probably upset some people who ran them unknowingly. ;-) Karl On Tue, May 17, 2011 at 9:10 AM, Tommaso Teofili <tommaso.teof...@gmail.com> wrote: > Hello Karl, > thanks so much again for the warm welcome and for these insights. > I agree with you that testability is a major concern we need to engage and > get sorted. > One thing I think right now is that having tests rely on a testing > infrastructure would possibly expose ManifoldCF to this issue again in case > of maintenance, versions evolution and so on therefore, although I still > have to investigate if that can be sorted technically for the supported > systems, my "design time" opinion is that we should try to mock those > systems. > Did you try to mock any of them yet? Does this sound good/bad to you? > Regards, > Tommaso > > > 2011/5/17 Karl Wright <daddy...@gmail.com> > >> For those who have just entered the ManifoldCF project, I'd like to >> first extend my congratulations once again! >> >> You are probably still trying to figure out exactly what's going on >> and where we are going. Unfortunately, this being Apache, I cannot >> actually answer your question, because you are part of the process >> now, and you will now be able to act on your own ideas and goals for >> the project. But in the interests of planning and consensus building, >> I'd like to share my thoughts as to what I think are the major >> technical issues the project faces. >> >> The first and foremost issue is one of testability. ManifoldCF is >> unique in that every connector requires a system to test against. In >> some cases this is a proprietary system, such as Documentum or >> LiveLink. In other cases, it's a large number of web servers. >> >> When MetaCarta developed the project in the first place, we had a >> stable of VMs against which our tests worked. They had proprietary >> software installed in some cases, or some Apache hackery to emulate a >> large number of individual web sites. When MetaCarta assets were sold >> to qBase, those assets included those VMs. We desperately need >> something like this again. I've tried to get access to the old >> MetaCarta VM's, but qBase has not yet granted this, and may never >> grant it due to ongoing technical reasons. >> >> Given this, it seems that we need some way of doing the same thing. >> It would be great to hear your ideas, especially if you have access to >> any of the proprietary systems we support and would be willing to set >> up and maintain testing infrastructure. >> >> Again, welcome to our new committers and new mentor! >> Karl >> >
