On 11 April 2016 at 07:56, Jonathan Holloway <jhollo...@redhat.com> wrote:
> > > ------------------------------ > > *From: *"M S Vishwanath Bhat" <msvb...@gmail.com> > *To: *"Niels de Vos" <nde...@redhat.com> > *Cc: *"Raghavendra Talur" <rta...@redhat.com>, "Jonathan Holloway" < > jhollo...@redhat.com>, "Gluster Devel" <gluster-devel@gluster.org>, > "Kaushal M" <kshlms...@gmail.com> > *Sent: *Wednesday, March 30, 2016 7:05:52 AM > *Subject: *Re: [Gluster-devel] Location of distaf tests > > > > On 24 March 2016 at 15:00, Niels de Vos <nde...@redhat.com> wrote: > >> On Thu, Mar 24, 2016 at 12:05:29PM +0530, Raghavendra Talur wrote: >> > On Wed, Mar 23, 2016 at 9:56 PM, M S Vishwanath Bhat <msvb...@gmail.com >> > >> > wrote: >> > >> > > >> > > >> > > On 18 March 2016 at 02:47, Jonathan Holloway <jhollo...@redhat.com> >> wrote: >> > > >> > >> >> > >> >> > >> ------------------------------ >> > >> >> > >> *From: *"M S Vishwanath Bhat" <msvb...@gmail.com> >> > >> *To: *"Niels de Vos" <nde...@redhat.com> >> > >> *Cc: *"Gluster Devel" <gluster-devel@gluster.org> >> > >> *Sent: *Thursday, March 17, 2016 8:18:23 AM >> > >> *Subject: *Re: [Gluster-devel] Location of distaf tests >> > >> >> > >> >> > >> >> > >> >> > >> On 17 March 2016 at 10:50, Niels de Vos <nde...@redhat.com> wrote: >> > >> >> > >>> On Wed, Mar 09, 2016 at 08:26:44PM +0530, M S Vishwanath Bhat wrote: >> > >>> > On 9 March 2016 at 19:39, Kaushal M <kshlms...@gmail.com> wrote: >> > >>> > >> > >>> > > On Wed, Mar 9, 2016 at 7:02 PM, M S Vishwanath Bhat < >> > >>> msvb...@gmail.com> >> > >>> > > wrote: >> > >>> > > > Hi, >> > >>> > > > >> > >>> > > > When we were discussing about the readiness of distaf for >> upstream >> > >>> test >> > >>> > > > automation, this question came up, That we should have a >> process or >> > >>> > > workflow >> > >>> > > > for proposing, reviewing and including the tests somewhere. >> > >>> > > > >> > >>> > > > Right now the tests are part of distaf repository >> > >>> > > > (github.com/gluster/distaf) itself. And contributing to >> distaf is >> > >>> by >> > >>> > > sending >> > >>> > > > a PR. But we want this to be included in gerrit so that >> review and >> > >>> > > > contributing process becomes much easier. But the question >> still >> > >>> > > remains... >> > >>> > > > where? Right now I can think of below options. >> > >>> > > > >> > >>> > > > * Use the same distaf repo in github for tests as well. >> > >>> > > > * Create a separate repo distaf_gluster_tests (or something >> > >>> similar) and >> > >>> > > > have all the tests there. >> > >>> > > > * Or have a tests/distaf/ directory inside glusterfs >> repository. >> > >>> And this >> > >>> > > > tests can be bundled in a rpm and distributed. This directory >> will >> > >>> have >> > >>> > > both >> > >>> > > > the test cases and related library functions. >> > >>> > > >> > >>> > > I prefer this approach. It makes it easier for developers to >> submit >> > >>> > > tests along with their changes, as is the case with our >> regression >> > >>> > > tests now. >> > >>> > > >> > >>> > > By library functions, I'm assuming you mean helper libraries >> related >> > >>> > > to gluster, which will be used in the tests which will be >> written. >> > >>> > > >> > >>> > >> > >>> > Yes, I mean helper functions which are related to gluster. The >> > >>> framework >> > >>> > itself will be made a python package. At least that's the plan. >> > >>> >> > >>> +1 for the tests in tests/distaf/ . I think I would like the libs >> to be >> > >>> part of the main distaf project. That would make it easier to run >> distaf >> > >>> for other projects that integrate with Gluster (like Samba, >> > >>> NFS-Ganesha and the like). The upstream projects can that use >> distaf for >> > >>> their integration with Gluster if they choose so. >> > >>> >> > >> >> > >> My concern with having libs in main distaf project is that, libs and >> test >> > >> cases are very much tied together. So when someone writes a >> testcase, they >> > >> write and/or modify the related gluster libs along with it. And the >> one >> > >> needs to go back and forth between the testcase and gluster libs >> while >> > >> writing and debugging the case. If they are in separate >> repo/package, it >> > >> makes it bit difficult (not impossible) for test case writer. >> > >> >> > >> If everyone is of the same opinion that gluster libs should be part >> of >> > >> main distaf repo, I can still do it when I package it next week. >> > >> >> > >> >> > >> I'm now thinking we should go ahead and keep it three distinct >> projects: >> > >> - Gluster: testcase--and supporting functions that are not a good >> fit for >> > >> the libs >> > >> - distaf-libs-gluster: core libarary and utility functions >> > >> - distaf: framework only >> > >> >> > > >> > > But that (having more than one repo) kind of defeats the purpose of >> > > developers having to concern themselves with only single repo >> > > (glusterfs.git). Since I couldn't discuss this in weekly gluster >> meeting in >> > > #gluster-meeting, let me propose my solution here. >> > > >> > >> > This proposal from Jonathan of 3 repos is what I like the best for long >> > term. >> >> Same here, although having distaf-libs-gluster in the man distaf >> framework might be good too. Many frameworks also provide plugins of >> some kind, I see the libs as something like a plugin. >> >> In the end, I do not really care very much. Just check with whoever is >> writing test-cases that need library extensions. Hopefully the library >> changes are rare (on future) and new test-cases do not need any >> framework/libs modifications. Once the libs are (more) stable, maybe >> move them to the main distaf repo. >> > > Okay, We had an internal meeting to finalise on this. Hope it is Okay. > > So as of now, we'll have both distaf test cases and libraries together. > This will ease the process of sending test cases for all people involved. > > I have sent a initial patch for that http://review.gluster.org/#/c/13853/. > Please take some time to review it. Since I will be on leave for next two > weeks until 15th April, someone please address any comments that is given > to the patch. > > > I'm getting a 500 Server Error when trying to access upstream gerrit under > my id at the moment, so I'll comment on the patch at next opportunity. > > Anyone know of any snags we might encounter with namespace_packages? > Since we're going ahead and housing distaf-libs in the glusterfs repo now > with the idea of moving to a separate distaflibs repo later, > we could leverage namespace_packages to have all libraries fall under a > single installed namespace while development stays completely separate. > The distaf_libs tree under the glusterfs repo could be something like the > following: > > tests/distaf/distaflibs-gluster/ <--- root dir > for python config files, docs, etc. > tests/distaf/distaflibs-gluster/setup.py > tests/distaf/distaflibs-gluster/distaflibs/__init__.py <--- > namespace_package placeholder > tests/distaf/distaflibs-gluster/distaflibs/gluster <--- contains > the actual package and module files > tests/distaf/distaflibs-gluster/distaflibs/gluster/__init__.py > tests/distaf/distaflibs-gluster/distaflibs/gluster/brick_ops.py > ... > tests/distaf/distaflibs_gluster/distaflibs/gluster/volume_ops.py > > This would install to /usr/lib/python2.7/site-packages/distaflibs/gluster > and import via "from distaflibs.gluster import volume_ops", etc. > > If I write a separate library, for say lvm, under my own github, etc., it > would be setup similar to... > <wherever>/distaflibs-lvm/ > <wherever>/distaflibs-lvm/setup.py > <wherever>/distaflibs-lvm/distaflibs/__init__.py > <wherever>/distaflibs-lvm/distaflibs/lvm > <wherever>/distaflibs-lvm/distaflibs/lvm/__init__.py > <wherever>/distaflibs-lvm/distaflibs/lvm/pv.py > > Like distaflibs-gluster, it would also be installed under > /usr/lib/python2.7/distaflibs > and import via "from distaflibs.lvm import pv", etc. > > The directory structure seems a little redundant, but allows for easy > separation of package configuration while installing into the same package > tree. > When we are ready to move out of glusterfs repo, it's just a matter of > moving the distaflibs-gluster directory somewhere else and all imports will > remain the same. > If down the road we decide to host other common/core distaflibs packages, > the directory structure makes it simple to copy them in with no changes. > > Sphinx uses something similar for sphinxcontrib and zope uses > namespace_packages pretty heavily. > For some reasons this wasn't delivered to my inbox and I found it in sent-mail. Anyway, I like this Idea of namespace packaging. I see that you have sent a patch for the same. I am reviewing that as well. So one thing this mandates, is to install both distaf package and distaflibs package before running any test cases. I think that is perfectly Okay. Thanks for the suggestion. Best Regards, Vishwanath > Cheers, > Jonathan > > > So after 3.8, when we have some tests already in repo, we can analyse the > robustness of the libs and move out the related libs to different repo. But > we can take that call later and for now, at least for ease of use, lets > have both together. > > Best Regards, > Vishwanath > > >> > > Library functions and test cases both go into a subdirectory inside >> > > glusterfs.git. This helps for devs and people writing test case for >> > > gluster. And we find a way to sync the libraries to distaf.git, so >> that >> > > each time a package is made, it is available for integrating projects >> to >> > > make use of. I know this is not very elegant way of dealing with the >> > > problem, but I couldn't think of anything which would fit all >> requirements. >> > > >> > >> > If you think distaf-libs-gluster can have more improvements and it makes >> > sense to have them with tests for now; we can have 2 repos temporarily >> > 1. Gluster git has tests + distaf-libs-gluster ( We can even avoid >> > packaging these two for now. Just let them be part of Gluster repo. ) >> > 2. distaf repo has distaf framework only >> >> I really would like to see the tests (and distaf-libs) packaged too. >> This makes it much easier for writing and running test cases. All the >> tests that we do in the CentOS CI should only install RPMs, that makes >> it very simple to run the same tests on different environments. >> >> At the moment we package the tests/ directory for non-official builds >> already. We do not want to provide the glusterfs-regression-tests RPM in >> repositories for users where they can install the package and easily >> break their existing deployment. But installing the package on a >> test-machine with the matching binary RPMs makes testing pretty smooth. >> >> Niels >> >> > >> > >> > > >> > > Jonathan, Niels, Kaushal, Raghavendra Talur - Your thoughts? >> > > >> > > Best Regards, >> > > Vishwanath >> > > >> > > >> > >> Here's some of my reasoning... >> > >> - We're already talking about not putting the libs in the Gluster >> repo as >> > >> an option, so from a convenience perspective it's not as relevant >> which >> > >> repo is being used (distaf or distaf-gluster-libs). >> > >> - And if we're already talking about the possibility of a refactor to >> > >> split the libs out down the road, it's not much different to talk >> refactor >> > >> to roll the libs into the distaf project should they become a >> problem to >> > >> maintain. >> > >> - With that said, if looking to have distaf adopted by a wider >> > >> non-Gluster audience, we'll want to split the libs out down the road >> > >> anyway. It would also lay the foundation for others >> > >> (distaf-libs-<your-project-here>). >> > >> - The Maintainer/Reviewer/Contributor demographic isn't necessarily >> the >> > >> same across all three and the Maintainer/Reviewer will most likely >> play a >> > >> larger role initially as we build out the libs and figure out what >> goes >> > >> where, what works, etc. >> > >> I'm just thinking it would be easier to manage that >> separately in >> > >> the beginning. >> > >> - If we are including the libs in the distaf project, how will the >> libs >> > >> be consumed? Roll them in the distaf package? Separate package? Git >> clone >> > >> and setup.py? >> > >> I like the concept of separate distaf and distaf-libs-gluster >> > >> packages that can be versioned and maintained separately. It keeps >> them >> > >> flexible and clean/clear of each other, and any framework changes >> that are >> > >> required by a library change can be managed through the packaging. >> > >> We can still do that if the libs are in the same repo, but >> in my >> > >> opinion it's not as clean and the value add in having them combined >> is >> > >> minimal. >> > >> - It shouldn't take that long to create the new project and may save >> us >> > >> some inevitable headache down the road. >> > >> >> > >> Either way, I agree with Niels that the sooner the better--so if a >> > >> separate repo for distaf-libs-gluster would create an unacceptable >> delay, >> > >> rolling libs into the distaf project works for me. >> > >> >> > >> Cheers, >> > >> Jonathan >> > >> >> > >> Best Regards, >> > >> Vishwanath >> > >> >> > >>> >> > >>> > > I'm also in favor of including them here as well. This will >> help keep >> > >>> > > DiSTAF free of an Gluster specific cruft and allow it to be >> > >>> (possibly) >> > >>> > > reusable by others. >> > >>> > > >> > >>> > >> > >>> > The recent changes makes it specific to gluster, but very easy to >> make >> > >>> it >> > >>> > generic. >> > >>> >> > >>> Starting Gluster specific is fine, later on when things are in use >> and >> > >>> working the way we need it should be trivial to make it more >> generic. >> > >>> Just keep a generic way for adding new functionalities for now, and >> file >> > >>> issues in GitHub for the parts that need to get refactored. >> > >>> >> > >>> The sooner we can start using distaf and gain (more) real world >> > >>> experience the better. >> > >>> >> > >>> Thanks, >> > >>> Niels >> > >>> >> > >>> >> > >>> > Best Regards, >> > >>> > Vishwanath >> > >>> > >> > >>> > >> > >>> > > >> > >>> > > > >> > >>> > > > Please let us know what your preferred option is. If you have >> any >> > >>> other >> > >>> > > > ideas, please let us know as well. >> > >>> > > > >> > >>> > > > Best Regards, >> > >>> > > > Vishwanath >> > >>> > > > >> > >>> > > > >> > >>> > > > _______________________________________________ >> > >>> > > > Gluster-devel mailing list >> > >>> > > > Gluster-devel@gluster.org >> > >>> > > > http://www.gluster.org/mailman/listinfo/gluster-devel >> > >>> > > >> > >>> >> > >>> > _______________________________________________ >> > >>> > Gluster-devel mailing list >> > >>> > Gluster-devel@gluster.org >> > >>> > http://www.gluster.org/mailman/listinfo/gluster-devel >> > >>> >> > >>> >> > >> >> > >> _______________________________________________ >> > >> Gluster-devel mailing list >> > >> Gluster-devel@gluster.org >> > >> http://www.gluster.org/mailman/listinfo/gluster-devel >> > >> >> > >> >> > >> >> > > >> > > >
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel