I want to comment on this for the meeting, but don't have much time so I'll keep it brief.
The point of modularity is that different features can be turned on/off as the application developer needs/wants. The point of asynchronous releases of modules are that each can develop on its own development timeline and/or be dropped in the future if they are not used by the community any longer (or even moved to new communities if they evolve beyond DSpace). The REST api can exist outside the "trunk" but you cannot create a dspace/modules/rest overly project for it in the trunk, theres catch-22 there, which is why we are discussing asynchronous release. The point of release a beta module is to test and get feedback on it to improve it. IMO, XMLUI, LNI and SWORD all entered into trunk in a Beta state and were "hardened' over many many months. It may be that authentication/authorization Basic auth capabilities need to be implemented before REST can go into the trunk. However, this both doesn't solve the problem of asych releasing nor getting us to the point where it is easier to add modules onto the build that are maintained/released outside the trunk. Recently (before I was sidetracked :-) ) I talked with Tim about utilizing Maven Archetypes and plugins to create a mechanism to add modules into your dspace build without having to edit the poms yourself. This would introduce a one-off that would be neccessary to begin not having to create the dspace/modules/xxxxx overlay directories in the trunk to turn on that feature in your build, instead the plugin would create and establish the overlay. I think this is the way out of the current catch-22 we are in. So, putting REST into Trunk is actually not a critical activity unless the community really wants to have it as a major addon like SWORD, LNI, XMLUI etc. as adding it in really deadlocks its development into the fixed release schedule. It should only be added if the developers outside the GSoC project are willing to maintain it, At this time I don't think this is 100% the case for LNI or SWORD, which are both maintained by a select few within the community. I'm more interested in how we can get the existing modules "out of core" rather than shoe horning more into it. I have a different opinion on the GSoC Testing framework because it is infrustructure just like dspace-api. So, I would like to see some discussion today on the overal build process and getting into async releases for modules as an important special topic (possibly next week when I can attend the commiters meetings again. Mark On Wed, Aug 11, 2010 at 7:03 AM, TAYLOR Robin <robin.tay...@ed.ac.uk> wrote: > Hi Tim, > > I have some concerns about this approach. I have been working recently with > a company called Atira who, on the basis of the documentation, chose to make > use of the LNI for a particular project. Having started down that road they > discovered that the documentation was more comprehensive than the actual > code. When I highlighted this on the mailing lists it was pointed out that > the LNI was an "experimental" piece of work and should not have been > considered ready for a 'live' situation. My private reaction to this was to > wonder what it was doing in the distributed code rather than in a sandbox > area. By this time a considerable commitment had been made by this company > to making use of a feature which turned out to be not ready. Let me stress > that I am not criticising the LNI in any way, I just think users have the > right to expect that code distributed in a release is considered ready for > use. Having said that, if the way we package DSpace changes in the future to > allow users to se! > lect modules from a list whose status is clearly designated then fine, but > we are not at that point. > > So, its not that I am against this approach per se, I'm just not sure how > we would do it right now. I would be happy to be convinced otherwise as it > would be a shame not to make the REST code available somehow. How is that > for sitting on the fence :) > > Cheers, Robin. > > > ________________________________________ > From: Tim Donohue [tdono...@duraspace.org] > Sent: 10 August 2010 22:40 > To: Sands Alden Fish > Cc: Bojan Suzic; azecko...@gmail.com; DSpace GSoC Students; dspace-devel > Subject: Re: [Dspace-devel] DSpace GSoC REST API project status? > > All, > > Here's yet another possible approach to how we could release the REST API. > > I asked the Fedora Commons folks how they went about releasing their > REST API. Here's the general steps they went through: > > 1. In Fedora 3.0 (July 2008), they released the REST API as > "experimental" with an explicit warning that they didn't recommend it > for production use. However, they packaged it with the release, and > made it optional to install (for anyone wanting to try it out & give > feedback). > > 2. In Fedora 3.1 (Oct 2008), they updated the REST API status to "beta" > > 3. In Fedora 3.2 (May 2009), they improved the stability and decide to > turn it on by default in all installs > > 4. In Fedora 3.3 (Dec 2009), they stabilized the code even more & it is > now considered fully production quality > > After talking with Chris Wilper (Fedora Tech Lead), he said he feels it > was beneficial to the uptake of the REST API to release it even in its > "experimental" phase. Even though it wasn't fully "production ready" > many people started developing against it and building clients (e.g. a > Ruby client, Python clients, etc). This meant that they received some > immediate feedback from developers, and also had several clients > available by the time it was released as a production quality REST API. > > They also discussed whether or not to package the "experimental" version > with Fedora 3.0. But, in the end, decided that developers were not > likely to download it separately (but, if it came in the normal install, > they felt that many would try it out, even if it was explicitly marked > "experimental"). > > Just thought I'd pass this summary along. I'm sure we can ask Chris > (and all the other Fedora folks) more questions about their experiences > with this sort of release process. > > - Tim > > > On 8/10/2010 9:58 AM, Tim Donohue wrote: > > Hi all, > > > > A few comments, which Sands' thoughts brought to mind. > > > > First, a quick FYI for everyone reading along. The REST API code is > > actually available in SVN already (this year, we required all GSoC > > projects to work out of our SVN). So, any developers can already pull it > > down and begin to play with the code (Obviously, Bojan is still working > > on some final testing, fixes). Here's where the GSoC REST API code is in > > SVN: > > > > http://scm.dspace.org/svn/repo/modules/rest/branches/dspace-rest-gsoc10/ > > > > The above code is actually a separate REST Maven project (with it's own > > webapp). So, you'd need to install it alongside an existing DSpace 1.6 > > test install (obviously). > > > > As for whether this makes it into DSpace 1.7. Obviously the decision > > still needs to be made in the coming weeks. The main reason I was hoping > > for its addition to Trunk is that's the current way to ensure the code > > is "fully supported" (by the Committers). However, this old "rule of > > thumb" (that the 'trunk' is the area of full support) is already > > starting to be broken (by 'dspace-services' and other modules whose code > > actually doesn't reside on SVN 'trunk', but are still 'fully supported' > > modules). > > > > It's very possible this REST API could be one of the first examples of a > > true 'asynchronously released module'. So, if it's not ready/stable > > enough, then it doesn't get released in 1.7.0 out-of-the-box. But, > > perhaps, the developers/committers release it separately when it's ready > > (even a month or two after 1.7.0) and then provide documentation for how > > institutions can install this "1.7 supported add-on" if they want it. > > Later on, maybe the REST API becomes "out-of-the-box" in 1.8, or maybe > > it always remains a separate "supported add-on" for DSpace. > > > > Just another option to think about... > > > > - Tim > > > > On 8/9/2010 7:23 PM, Sands Alden Fish wrote: > >> It's great that this code is being moved along with the target of a 1.7 > >> release. I think that it's one of the most needed features for DSpace to > >> grow. > >> > >> However, a comment / opinion on getting it into the repository. I wonder > >> if it's not too much of a rush to put it immediately into the trunk. > >> > >> Considering that there (as reads below) has been extremely minimal > >> testing of it's impact on stability as well as it's proper function, and > >> now with the completion of code as well as documentation under > >> relatively heavy deadline pressure given the time left, why not commit > >> it to a branch immediately and allow the broader developer community to > >> test/vet it, track the completion of its coding and perhaps even > >> contribute some fixes? > >> > >> This way, if October 22rd ;) comes around and there has not been enough > >> time to be certain of it's stability, we haven't introduced unknown > >> instability into the next release, it will still be available for people > >> to implement locally if they choose, and work can continue until it is > >> release-quality code. I have to say, I would love it if it was ready for > >> the 1.7 timeframe, but the description of it's current state below > >> doesn't leave me completely convinced. > >> > >> > >> IMHO, > >> > >> -- > >> sands fish > >> Software Engineer > >> MIT Libraries > >> Technology Research & Development > >> sa...@mit.edu <mailto:sa...@mit.edu> > >> E25-131 > >> > >> > >> > >> > >> On Aug 9, 2010, at 12:09 PM, Tim Donohue wrote: > >> > >>> Hi Bojan (and all), > >>> > >>> First, I'd agree with Kevin Clarke's point. At a minimum level, I think > >>> we really need support for Basic HTTP Auth before we could think about > >>> releasing the REST API as part of DSpace 1.7 (which I, personally, > would > >>> love to see happen). All our other "machine" interfaces (SWORD, > >>> LNI/WebDav) support Basic HTTP Auth -- so, this would bring the REST > API > >>> in alignment with both of them. > >>> > >>> More comments/suggestions in-line below... > >>> > >>> On 8/7/2010 9:40 AM, Bojan Suzic wrote: > >>>> Yes you are right, it is not mentioned there. Currently the > >>>> authentication is done simply by providing the user name and > >>>> password as > >>>> request parameters (e.g. GET /rest/items/1.json?user=xxx&pass=xxx). If > >>>> there is a a need within community for other mechanism I will gladly > >>>> extend the code to support it. > >>>> > >>>> Also regarding comments on the wiki, I have took them into account and > >>>> so it should be ok. > >>> > >>> Thanks for adding more details to the wiki docs. Again, support for > >>> Basic HTTP Auth would be what we'd want to add. > >>> > >>>> Yes, that it correct, the authorization is forwarded to the underlying > >>>> structures itself and the errors are catched and returned to the user. > >>>> In the most cases the application provides an message about failure to > >>>> process a request - in the case of unsuccessful Authorization or any > >>>> other case. Ofcourse, the code returns descriptions of the errors in > >>>> the > >>>> form provided by the underlying libraries. This means, if related > >>>> components do not return exact description, nor the rest api can. > >>> > >>> This sounds like a good approach to Authorization for the REST API. > >>> > >>> > >>>> Ok, I will comment this from my perspective. > >>>> First, the wiki is not currently following the status of the software > >>>> development. As I have been working on the particular aspects of the > >>>> code, updating the wiki after each change would produce unnecessary > >>>> context switching and overhead, thus causing inefficient use of the > >>>> time > >>>> resources available. Instead the wiki (in this phase) has been used > >>>> mostly for a draft proposal how all should look like e.g. the > >>>> organization of the endpoints, functions covered, the > >>>> handling/manipulation of resources and so. > >>>> > >>>> Now that some questions were raised, I will update the wiki with the > >>>> information requested and will also change it to better resemble > >>>> current > >>>> status of the project. E.g. some methods have been changed, especially > >>>> related to creation of new resources (update/getting hasn't been > >>>> changed > >>>> much). > >>> > >>> Thanks for the wiki updates. I definitely understand that the Wiki may > >>> not be 100% up-to-date at any one time. Just do you best to take some > >>> time every once and a while to get it back up-to-date. > >>> > >>> What is important is to ensure the Wiki is up-to-date (or mostly > >>> up-to-date) by the end of GSoC -- that way we have the absolute current > >>> status of the project. Firm pencils down is next week (Aug 16), so it > >>> would be good to ensure it is up-to-date to the latest status of the > >>> code by then. > >>> > >>>> For the issues/concerns, this time I have been focused more on the > >>>> following aspects: > >>>> - decoupling parts of the code and moving the direct handling routines > >>>> (like accessing the resources directly) from entity providers' level > to > >>>> entities; this should make easier support among different versions of > >>>> DSpace and make easier transition to new storage api > >>>> - taking into account the user need to manipulate with the resources, > >>>> e.g. create new communities, update/create items and so > >>>> > >>>> From this point, one issue may be the fact that implementation of all > >>>> dspace-api -> dspace-rest mappings and support for all those functions > >>>> could require too much effort and time, while many of those functions > >>>> would not be used by most of the users. Instead in this wiki I have > >>>> drafted/proposed the functions (endpoints/features) which would have > >>>> the > >>>> highest priority to implement and thus represent this first official > >>>> REST support for the DSpace. For this first (version) I think it is > ok, > >>>> and then we may see user/developer feedback and implement other > >>>> functions. Also from the last year's project I already got some > >>>> feedback > >>>> so it was good starting point for me. > >>> > >>> I agree that this is a good approach. An initial REST API does *not* > >>> need to do everything available to the 'dspace-api'. I think at a > >>> minimum, it should concentrate on 'GET' methods (to enable repository > >>> browsing via REST). If possible, it'd also be good to get some of the > >>> main Repository manipulation methods (POST & DELETE especially). > >>> However, if the REST API for DSpace 1.7 only had the 'GET' methods > ready > >>> (and tested), I still think this would be worth releasing as in itself > >>> it allows other systems to better "crawl" or harvest all the contents > >>> (including Communities/Collections) out of a DSpace. The manipulation > >>> methods could always be built in later on, if needed. > >>> > >>>> Other issue for me was the introduction of other new projects, e.g. > >>>> Backporting of DSpace 2 Storage Services and Semantic Content > >>>> Repository. Current version of the code uses DSpace1 approach and > >>>> libraries and once other implementations are ready we can think making > >>>> this code dependent/related to them. > >>> > >>> I think this is the right approach for the REST API. DSpace 1.7 won't > >>> be releasing any of the backporting functionality (as that work will > >>> require additional migration scripts and other work). So, the REST API > >>> should be using the DSpace1 approach (as that is what will be part of > >>> DSpace 1.7). > >>> > >>>> For the software stability etc, I must admit that I haven't had much > >>>> time to test functionality/correctness of the output results. While I > >>>> tested each feature manually during the implementation, I suppose > there > >>>> will be many small bugs to correct. One repository of say 100 or so > >>>> items could be good starting point for testing. It would be good if > >>>> someone could help me to get such repository, then I could write > >>>> automatic tests (or use other testing methodology) and this way many > of > >>>> those small bugs could be found prior to release. > >>>> And generally for the testing of the software I think also that > >>>> community should decide up to which extent we should test the code > >>>> before going into official release. As October 22rd is the freezing > >>>> point, can 5-6 weeks be enough for us all to check the code and > correct > >>>> the errors found? > >>> > >>> I think the real goal here would be the following: > >>> (1) Bojan, Aaron and anyone else do some immediate basic testing (to > >>> ensure the main code seems mostly stable & ready). This can just be > >>> done on a local repository with some test data. > >>> (2) We work to get approval to commit this code to Trunk soon (next few > >>> weeks -- if other Committers approve it & agree) > >>> (3) Once it is in Trunk, this gives more people a way to more easily > >>> test the REST code. It also means that it will receive more thorough > >>> testing as part of the DSpace 1.7 Test-a-Thon (and I can post the REST > >>> API code up on http://demo.dspace.org , which is a demo repository of > >>> ~200 demo items). > >>> > >>>> Other issue, timeline, I think not everything will be finished up to > >>>> the > >>>> official end of GSoC. However I will also continue to work on it after > >>>> GSoC end and I expect it would require a few weeks more before it > could > >>>> go to the testing. I would say maybe 2-3 weeks or before > mid-September. > >>>> This would give some 6 weeks for testing and fixing. What do you > think, > >>>> is it enough time for inclusion? > >>> > >>> All features *must* be committed by the deadline of Oct 22. So, as long > >>> as the main feature is approved & committed to Trunk before then, it > >>> would be fine. Since this is a new project, I'd recommend actually > >>> committing the code before then (hopefully in the next few weeks -- or > >>> by late Aug at the latest), so that others can help with the testing & > >>> development. In a large development environment like the DSpace > >>> development community, the *earlier* you commit code to Trunk, the more > >>> likely it will be bug-free before release (as it allows many other > >>> people to help with testing/commenting/bug fixing). > >>> > >>> So, I'd highly encourage you to begin to work to stabilize the code > that > >>> you have now. As GSoC is wrapping up, this is necessary anyways. > >>> Stabilize code, document it (by updating the wiki docs), and then we > can > >>> work to get it approved for committing to Trunk. Assuming it is > >>> approved, the developers can help find (and squash) any last bugs, and > >>> small amounts of additional development could even happen (e.g. adding > a > >>> few more 'dspace-api' functions, if some are missing). > >>> > >>> I hope this helps. Definitely ask more questions if anything doesn't > >>> make sense or needs more clarification. > >>> > >>> - Tim > >>> > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> > >>> This SF.net <http://SF.net> email is sponsored by > >>> > >>> Make an app they can't live without > >>> Enter the BlackBerry Developer Challenge > >>> http://p.sf.net/sfu/RIM-dev2dev > >>> _______________________________________________ > >>> Dspace-devel mailing list > >>> Dspace-devel@lists.sourceforge.net > >>> <mailto:Dspace-devel@lists.sourceforge.net> > >>> https://lists.sourceforge.net/lists/listinfo/dspace-devel > >> > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by > > Make an app they can't live without > Enter the BlackBerry Developer Challenge > http://p.sf.net/sfu/RIM-dev2dev > _______________________________________________ > Dspace-devel mailing list > Dspace-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-devel > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by > > Make an app they can't live without > Enter the BlackBerry Developer Challenge > http://p.sf.net/sfu/RIM-dev2dev > _______________________________________________ > Dspace-devel mailing list > Dspace-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-devel > -- Mark R. Diggory Head of U.S. Operations - @mire http://www.atmire.com - Institutional Repository Solutions http://www.togather.eu - Before getting together, get t...@ther
------------------------------------------------------------------------------ This SF.net email is sponsored by Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________ Dspace-devel mailing list Dspace-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-devel
