I agree -- it's a great feature and I'm really looking forward to its
completion, but I think it might be too early for trunk testing right now..
even the GET functionality isn't 100% there yet.
RE: Auth -- I've been implementing basic HTTP auth in Tomcat and including
authZ details in my query/headers, which seems to be working well for me,
but I agree it'd be nice to be consistent with the way LNI, SWORD etc. work.
I have some notes from my own testing, which may or may not be helpful
(trying a combination of things outlined on the Wiki and in the 'describe'
hint pages served up by the REST API webapp itself):
[begin testing notes]
/items/{id}
Individual item GETs seem to be working fairly well, although by default it
also returns IDs for all items also in the same parent container, which
might end up slowing things down in large collections...
/communities/{id}
This worked as I expected, though I only tested on a community with a single
collection (which contained ~2000 items). Full details were returned for the
collection, and IDs only for the items.
/collections/{id}
As at my last testing, doing a GET on a collection with ~2000 items resulted
in 65949 individual db queries, 7756 of which were
"Collection.getAllItems()" queries! -- my test client typically timed out or
crashed before I could receive results.. the results were fairly verbose too
at about ~200000 lines of JSON, but they looked fine once they made it
through...
I think there's some loop here during iteration / depth detection that is
causing a lot of unnecessary getAllItems / findAll calls.
/collections/{id}/items
This is nice and quick, but only returns IDs and 'idOnly=false' seems to
have no effect
Personally I'd really like to get to the bottom of the issues I'm seeing
with CollectionEntity/Provider... I think that's all that's holding me back
from making a nice demo 'Repository Explorer' client.
[end testing notes]
Thanks to Bojan for his hard work so far!
Cheers,
Kim
On 10 August 2010 12:23, Sands Alden Fish <sa...@mit.edu> wrote:
> It's great that this code is being moved along with the target of a 1.7
> release. I think that it's one of the most needed features for DSpace to
> grow.
>
> However, a comment / opinion on getting it into the repository. I wonder
> if it's not too much of a rush to put it immediately into the trunk.
>
> Considering that there (as reads below) has been extremely minimal testing
> of it's impact on stability as well as it's proper function, and now with
> the completion of code as well as documentation under relatively heavy
> deadline pressure given the time left, why not commit it to a branch
> immediately and allow the broader developer community to test/vet it, track
> the completion of its coding and perhaps even contribute some fixes?
>
> This way, if October 22rd ;) comes around and there has not been enough
> time to be certain of it's stability, we haven't introduced unknown
> instability into the next release, it will still be available for people to
> implement locally if they choose, and work can continue until it is
> release-quality code. I have to say, I would love it if it was ready for
> the 1.7 timeframe, but the description of it's current state below doesn't
> leave me completely convinced.
>
>
> IMHO,
>
> --
> sands fish
> Software Engineer
> MIT Libraries
> Technology Research & Development
> sa...@mit.edu
> E25-131
>
>
>
>
> On Aug 9, 2010, at 12:09 PM, Tim Donohue wrote:
>
> Hi Bojan (and all),
>
> First, I'd agree with Kevin Clarke's point. At a minimum level, I think
> we really need support for Basic HTTP Auth before we could think about
> releasing the REST API as part of DSpace 1.7 (which I, personally, would
> love to see happen). All our other "machine" interfaces (SWORD,
> LNI/WebDav) support Basic HTTP Auth -- so, this would bring the REST API
> in alignment with both of them.
>
> More comments/suggestions in-line below...
>
> On 8/7/2010 9:40 AM, Bojan Suzic wrote:
>
> Yes you are right, it is not mentioned there. Currently the
>
> authentication is done simply by providing the user name and password as
>
> request parameters (e.g. GET /rest/items/1.json?user=xxx&pass=xxx). If
>
> there is a a need within community for other mechanism I will gladly
>
> extend the code to support it.
>
>
> Also regarding comments on the wiki, I have took them into account and
>
> so it should be ok.
>
>
> Thanks for adding more details to the wiki docs. Again, support for
> Basic HTTP Auth would be what we'd want to add.
>
> Yes, that it correct, the authorization is forwarded to the underlying
>
> structures itself and the errors are catched and returned to the user.
>
> In the most cases the application provides an message about failure to
>
> process a request - in the case of unsuccessful Authorization or any
>
> other case. Ofcourse, the code returns descriptions of the errors in the
>
> form provided by the underlying libraries. This means, if related
>
> components do not return exact description, nor the rest api can.
>
>
> This sounds like a good approach to Authorization for the REST API.
>
>
> Ok, I will comment this from my perspective.
>
> First, the wiki is not currently following the status of the software
>
> development. As I have been working on the particular aspects of the
>
> code, updating the wiki after each change would produce unnecessary
>
> context switching and overhead, thus causing inefficient use of the time
>
> resources available. Instead the wiki (in this phase) has been used
>
> mostly for a draft proposal how all should look like e.g. the
>
> organization of the endpoints, functions covered, the
>
> handling/manipulation of resources and so.
>
>
> Now that some questions were raised, I will update the wiki with the
>
> information requested and will also change it to better resemble current
>
> status of the project. E.g. some methods have been changed, especially
>
> related to creation of new resources (update/getting hasn't been changed
>
> much).
>
>
> Thanks for the wiki updates. I definitely understand that the Wiki may
> not be 100% up-to-date at any one time. Just do you best to take some
> time every once and a while to get it back up-to-date.
>
> What is important is to ensure the Wiki is up-to-date (or mostly
> up-to-date) by the end of GSoC -- that way we have the absolute current
> status of the project. Firm pencils down is next week (Aug 16), so it
> would be good to ensure it is up-to-date to the latest status of the
> code by then.
>
> For the issues/concerns, this time I have been focused more on the
>
> following aspects:
>
> - decoupling parts of the code and moving the direct handling routines
>
> (like accessing the resources directly) from entity providers' level to
>
> entities; this should make easier support among different versions of
>
> DSpace and make easier transition to new storage api
>
> - taking into account the user need to manipulate with the resources,
>
> e.g. create new communities, update/create items and so
>
>
> From this point, one issue may be the fact that implementation of all
>
> dspace-api -> dspace-rest mappings and support for all those functions
>
> could require too much effort and time, while many of those functions
>
> would not be used by most of the users. Instead in this wiki I have
>
> drafted/proposed the functions (endpoints/features) which would have the
>
> highest priority to implement and thus represent this first official
>
> REST support for the DSpace. For this first (version) I think it is ok,
>
> and then we may see user/developer feedback and implement other
>
> functions. Also from the last year's project I already got some feedback
>
> so it was good starting point for me.
>
>
> I agree that this is a good approach. An initial REST API does *not*
> need to do everything available to the 'dspace-api'. I think at a
> minimum, it should concentrate on 'GET' methods (to enable repository
> browsing via REST). If possible, it'd also be good to get some of the
> main Repository manipulation methods (POST & DELETE especially).
> However, if the REST API for DSpace 1.7 only had the 'GET' methods ready
> (and tested), I still think this would be worth releasing as in itself
> it allows other systems to better "crawl" or harvest all the contents
> (including Communities/Collections) out of a DSpace. The manipulation
> methods could always be built in later on, if needed.
>
> Other issue for me was the introduction of other new projects, e.g.
>
> Backporting of DSpace 2 Storage Services and Semantic Content
>
> Repository. Current version of the code uses DSpace1 approach and
>
> libraries and once other implementations are ready we can think making
>
> this code dependent/related to them.
>
>
> I think this is the right approach for the REST API. DSpace 1.7 won't
> be releasing any of the backporting functionality (as that work will
> require additional migration scripts and other work). So, the REST API
> should be using the DSpace1 approach (as that is what will be part of
> DSpace 1.7).
>
> For the software stability etc, I must admit that I haven't had much
>
> time to test functionality/correctness of the output results. While I
>
> tested each feature manually during the implementation, I suppose there
>
> will be many small bugs to correct. One repository of say 100 or so
>
> items could be good starting point for testing. It would be good if
>
> someone could help me to get such repository, then I could write
>
> automatic tests (or use other testing methodology) and this way many of
>
> those small bugs could be found prior to release.
>
> And generally for the testing of the software I think also that
>
> community should decide up to which extent we should test the code
>
> before going into official release. As October 22rd is the freezing
>
> point, can 5-6 weeks be enough for us all to check the code and correct
>
> the errors found?
>
>
> I think the real goal here would be the following:
> (1) Bojan, Aaron and anyone else do some immediate basic testing (to
> ensure the main code seems mostly stable & ready). This can just be
> done on a local repository with some test data.
> (2) We work to get approval to commit this code to Trunk soon (next few
> weeks -- if other Committers approve it & agree)
> (3) Once it is in Trunk, this gives more people a way to more easily
> test the REST code. It also means that it will receive more thorough
> testing as part of the DSpace 1.7 Test-a-Thon (and I can post the REST
> API code up on http://demo.dspace.org , which is a demo repository of
> ~200 demo items).
>
> Other issue, timeline, I think not everything will be finished up to the
>
> official end of GSoC. However I will also continue to work on it after
>
> GSoC end and I expect it would require a few weeks more before it could
>
> go to the testing. I would say maybe 2-3 weeks or before mid-September.
>
> This would give some 6 weeks for testing and fixing. What do you think,
>
> is it enough time for inclusion?
>
>
> All features *must* be committed by the deadline of Oct 22. So, as long
> as the main feature is approved & committed to Trunk before then, it
> would be fine. Since this is a new project, I'd recommend actually
> committing the code before then (hopefully in the next few weeks -- or
> by late Aug at the latest), so that others can help with the testing &
> development. In a large development environment like the DSpace
> development community, the *earlier* you commit code to Trunk, the more
> likely it will be bug-free before release (as it allows many other
> people to help with testing/commenting/bug fixing).
>
> So, I'd highly encourage you to begin to work to stabilize the code that
> you have now. As GSoC is wrapping up, this is necessary anyways.
> Stabilize code, document it (by updating the wiki docs), and then we can
> work to get it approved for committing to Trunk. Assuming it is
> approved, the developers can help find (and squash) any last bugs, and
> small amounts of additional development could even happen (e.g. adding a
> few more 'dspace-api' functions, if some are missing).
>
> I hope this helps. Definitely ask more questions if anything doesn't
> make sense or needs more clarification.
>
> - Tim
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by
>
> Make an app they can't live without
> Enter the BlackBerry Developer Challenge
> http://p.sf.net/sfu/RIM-dev2dev
> _______________________________________________
> Dspace-devel mailing list
> Dspace-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-devel
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by
>
> Make an app they can't live without
> Enter the BlackBerry Developer Challenge
> http://p.sf.net/sfu/RIM-dev2dev
> _______________________________________________
> Dspace-devel mailing list
> Dspace-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-devel
>
>
------------------------------------------------------------------------------
This SF.net email is sponsored by
Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________
Dspace-devel mailing list
Dspace-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-devel
