Stefano, I assume that some sort of access control in Fedora (a policy) will be applied to the Fedora external datastream or object, or that the repository itself will not be publicly exposed? Otherwise, Fedora basically becomes an open back window to access content that's protected by a locked front door.
I think the primary reason why this issue hasn't come up before is because most repository owners have direct control over the objects within their repository, and can manage the authnz architecture to suit their own needs. If you have some degree of control over who can access the images on the backend webserver, then the simplest solution would be to configure the backend web server to allow requests from the Fedora host to pass through unimpeded, using IP access control. I took a look at the source code, and the method that makes the actual request is https://github.com/fcrepo/fcrepo/blob/master/fcrepo-server/src/main/java/org/fcrepo/server/storage/DefaultExternalContentManager.java line 280: private MIMETypedStream getFromWeb(ContentManagerParams params) Rich is correct: populating ContentManagerParams would get the credentials injected for you. I don't recall how those parameters get populated, though. Turning debug logging on will give you *a lot* of information about the handling of the request. -- Scott On 05/09/2013 04:02 PM, Stefano Cossu wrote: > @Scott: I can't afford to have this datastream managed by Fedora, > because it's several megabytes large and there are over a million of them. > > @Rich: the http://user@pass:/resource syntax works with cURL too, that's > why I gave it a shot even though I don't know what Fedora actually uses > to connect to remote servers. > Your source link is very interesting though. I'll give it a look. I > wonder how this hasn't been brought up before. Accessing resources > through authentication seems like quite a common task to me, and I hoped > I could do it without hacking the Fedora code. > > Thanks > sc > > > Stefano Cossu > Director of Application Services, Collections > > The Art Institute of Chicago > 116 S. Michigan Ave. > Chicago, IL 60603 > 312-499-4026 > > > On 5/9/13 3:45 PM, [email protected] wrote: >> Send Fedora-commons-users mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of Fedora-commons-users digest..." >> >> >> Today's Topics: >> >> 1. Re: [NEWBIE] Accessing external resource with HTTP >> authentication (Benjamin Armintor) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Thu, 9 May 2013 16:45:16 -0400 >> From: Benjamin Armintor <[email protected]> >> Subject: Re: [fcrepo-user] [NEWBIE] Accessing external resource with >> HTTP authentication >> To: "Support and info exchange list for Fedora users." >> <[email protected]> >> Message-ID: >> <cadqq8tpfxo8va7ub2ncb6xvbjzcbf1s7unu7hftcqhpntwb...@mail.gmail.com> >> Content-Type: text/plain; charset="iso-8859-1" >> >> Yes, in that sense it would be straightforward- though this question of >> where the credentials would be stored was one of things that derailed the >> feature in the first place! >> >> >> On Thu, May 9, 2013 at 4:40 PM, Rich d'Rich <[email protected]> wrote: >> >>> AFAIK the username:password@ syntax is a browser artefact that the Java >>> HTTP access libraries (apache commons httpclient) that Fedora uses doesn't >>> support. >>> >>> This also means that you can't do a server-server import where the source >>> Fedora server requires authentication, and it causes problems with >>> disseminators. >>> >>> However, looking at the code, most of the "wiring" is there: >>> >>> https://github.com/fcrepo/fcrepo/blob/master/fcrepo-server/src/main/java/org/fcrepo/server/access/DefaultAccess.java >>> >>> it just needs getDatastreamDissemination (around line 1145) to extract a >>> username and password from somewhere and put it into ContentManagerParams. >>> Ideally, there would be a configured table of known external servers and >>> credentials that could be kept secure so passwords aren't bandied about. >>> >>> I may be wrong though and there's already a way to do this :-) >>> >>> >>> On 10 May 2013 06:36, Scott Prater <[email protected]> wrote: >>> >>>> Stefano -- >>>> >>>> Are you ingesting the datastreams as managed datastreams, or as redirect >>>> or external datastreams? >>>> >>>> If the former, once Fedora ingests the FOXML, the object is referred to >>>> by its internal Fedora URI, and no source URLs or passwords are exposed >>>> in any object export. >>>> >>>> If the datastreams are managed, then you may want to take a compromise >>>> approach: fetch them to the local machine using curl or some such tool, >>>> then ingest the local file. Once it's ingested, you can delete the >>>> local file. >>>> >>>> Managed datastreams are usually preferred to external or redirect >>>> datastreams; there are use cases for external and redirect datastreams >>>> (which is why they exist), but the normal case is to store datastreams >>>> as managed. >>>> >>>> -- Scott >>>> >>>> On 05/09/2013 01:08 PM, Benjamin Armintor wrote: >>>>> Stefano- >>>>> I remember some conversation a couple of years ago about supporting >>>>> BASIC auth in services, but as far as I know they didn't go anywhere. >>>>> Maybe another committer remembers something? In any case, I don't see >>>>> why storing the credentials like that wouldn't work, if you can accept >>>>> the plain-text issues you cite. >>>>> >>>>> As far as certs, Im afraid you're on your own. I will warn you that >>>>> Java errs on the side of verification unless you instruct it not to, so >>>>> invalid certs will cause other problems. >>>>> >>>>> - Ben >>>>> >>>>> >>>>> On Thu, May 9, 2013 at 12:32 PM, Stefano Cossu <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Hi there, >>>>> I'm starting to tinker with Fedora and trying to write a CMA >>>> workflow. >>>>> I'm building a digital object that should grab an image datastream >>>> from >>>>> an HTTPS server which requires basic authentication. >>>>> I tried inserting the authentication data in the URL for the >>>> datastream, >>>>> but now I have 2 problems: >>>>> 1) Username and password are stored in plain text in the FOXML, >>>> visible >>>>> by everyone who looks up that record in Fedora, as well as all over >>>> the >>>>> logs. >>>>> 2) I still can't connect to the server this way. The server's >>>>> certificate is expired, I don't know if that plays a role. >>>>> >>>>> Fedora throws this error: >>>>> >>>>> ERROR 2013-05-09 11:04:28.618 [http-8080-1] (BaseRestResource) >>>>> Unexpected error fulfilling REST API request >>>>> org.fcrepo.server.errors.HttpServiceNotFoundException: >>>>> [DefaultExternalContentManager] returned an error. The underlying >>>> error >>>>> was a org.fcrepo.server.errors.GeneralException T >>>>> he message was "Error getting >>>>> https://username:password@imageserver/myHugePicture"; . >>>>> at >>>>> >>>> org.fcrepo.server.storage.DefaultExternalContentManager.getExternalContent(DefaultExternalContentManager.java:152) >>>>> ~[fcrepo-server-3.6.2.jar:na] >>>>> at >>>>> >>>> org.fcrepo.server.access.DefaultAccess.getDatastreamDissemination(DefaultAccess.java:1148) >>>>> ~[fcrepo-server-3.6.2.jar:na] >>>>> at >>>>> >>>> org.fcrepo.server.rest.DatastreamResource.getDatastream(DatastreamResource.java:247) >>>>> ~[fcrepo-server-3.6.2.jar:na] >>>>> [...] >>>>> >>>>> And the image server's Apache error log: >>>>> >>>>> Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] Connection to >>>>> child 0 established (server imageserver:443) >>>>> [Thu May 09 11:04:25 2013] [info] Seeding PRNG with 144 bytes of >>>> entropy >>>>> [Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] SSL library >>>> error >>>>> 1 in handshake (server imageserver:443) >>>>> [Thu May 09 11:04:25 2013] [info] SSL Library Error: 336151608 >>>>> error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal >>>> error >>>>> [Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] Connection >>>> closed >>>>> to child 0 with abortive shutdown (server imageserver:443) >>>>> [...] >>>>> >>>>> Of course, I can always use a redirect datastream and let the client >>>>> deal with authentication and SSL, but I'd like to hide the source >>>> URI if >>>>> possible. >>>>> >>>>> >>>>> Below is the FOXML representation of my object: >>>>> >>>>> <foxml:digitalObject VERSION="1.1" PID="test:dervPub_obj" >>>>> xsi:schemaLocation="info:fedora/fedora-system:def/foxml# >>>>> http://www.fedora.info/definitions/1/0/foxml1-1.xsd";> >>>>> <foxml:objectProperties> >>>>> <foxml:property >>>>> NAME="info:fedora/fedora-system:def/model#state" VALUE="Active"/> >>>>> <foxml:property >>>>> NAME="info:fedora/fedora-system:def/model#label" VALUE="Disseminator >>>>> object"/> >>>>> <foxml:property >>>>> NAME="info:fedora/fedora-system:def/model#ownerId" >>>> VALUE="fedoraAdmin"/> >>>>> <foxml:property >>>>> NAME="info:fedora/fedora-system:def/model#createdDate" >>>>> VALUE="2013-05-09T15:37:41.708Z"/> >>>>> <foxml:property >>>>> NAME="info:fedora/fedora-system:def/view#lastModifiedDate" >>>>> VALUE="2013-05-09T15:37:41.892Z"/> >>>>> </foxml:objectProperties> >>>>> <foxml:datastream ID="AUDIT" STATE="A" CONTROL_GROUP="X" >>>>> VERSIONABLE="false"> >>>>> <foxml:datastreamVersion ID="AUDIT.0" LABEL="Audit Trail >>>> for >>>>> this object" CREATED="2013-05-09T15:37:41.708Z" MIMETYPE="text/xml" >>>>> FORMAT_URI="info:fedora/fedora-system:format/xml.fedora.audit"> >>>>> <foxml:xmlContent> >>>>> <audit:auditTrail> >>>>> <audit:record ID="AUDREC1"> >>>>> <audit:process type="Fedora API-M"/> >>>>> <audit:action>addDatastream</audit:action> >>>>> <audit:componentID>SOURCE_IMG</audit:componentID> >>>>> <audit:responsibility>fedoraAdmin</audit:responsibility> >>>>> <audit:date>2013-05-09T15:37:41.892Z</audit:date> >>>>> <audit:justification/> >>>>> </audit:record> >>>>> </audit:auditTrail> >>>>> </foxml:xmlContent> >>>>> </foxml:datastreamVersion> >>>>> </foxml:datastream> >>>>> <foxml:datastream ID="DC" STATE="A" CONTROL_GROUP="X" >>>>> VERSIONABLE="true"> >>>>> <foxml:datastreamVersion ID="DC1.0" LABEL="Dublin Core >>>> Record >>>>> for this object" CREATED="2013-05-09T15:37:41.708Z" >>>> MIMETYPE="text/xml" >>>>> FORMAT_URI="http://www.openarchives.org/OAI/2.0/oai_dc/"; >>>> SIZE="388"> >>>>> <foxml:xmlContent> >>>>> <oai_dc:dc >>>>> xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ >>>>> http://www.openarchives.org/OAI/2.0/oai_dc.xsd";> >>>>> <dc:title>Disseminator object</dc:title> >>>>> <dc:identifier>test:dervPub_obj</dc:identifier> >>>>> </oai_dc:dc> >>>>> </foxml:xmlContent> >>>>> </foxml:datastreamVersion> >>>>> </foxml:datastream> >>>>> <foxml:datastream ID="RELS-EXT" STATE="A" CONTROL_GROUP="X" >>>>> VERSIONABLE="false"> >>>>> <foxml:datastreamVersion ID="RELS-EXT.0" >>>> LABEL="Relationships" >>>>> CREATED="2013-05-09T15:37:41.837Z" MIMETYPE="application/rdf+xml" >>>>> FORMAT_URI="info:fedora/fedora-system:FedoraRELSExt-1.0" SIZE="273"> >>>>> <foxml:xmlContent> >>>>> <rdf:RDF> >>>>> <rdf:Description >>>>> rdf:about="info:fedora/test:dervPub_obj"> >>>>> <hasModel >>>>> rdf:resource="info:fedora/test:dervPub_CModel"/> >>>>> </rdf:Description> >>>>> </rdf:RDF> >>>>> </foxml:xmlContent> >>>>> </foxml:datastreamVersion> >>>>> </foxml:datastream> >>>>> <foxml:datastream ID="SOURCE_IMG" STATE="A" CONTROL_GROUP="E" >>>>> VERSIONABLE="true"> >>>>> <foxml:datastreamVersion ID="SOURCE_IMG.0" LABEL="full >>>> sized >>>>> image" CREATED="2013-05-09T15:37:41.892Z" MIMETYPE="image/jpeg"> >>>>> <foxml:contentLocation TYPE="URL" >>>>> REF="https://username:password@imageserver/myHugePicture"/> >>>>> </foxml:datastreamVersion> >>>>> </foxml:datastream> >>>>> >>>>> I would really appreciate your help. >>>>> >>>>> >>>>> >>>>> >>>> ------------------------------------------------------------------------------ >>>>> Learn Graph Databases - Download FREE O'Reilly Book >>>>> "Graph Databases" is the definitive new guide to graph databases and >>>>> their applications. This 200-page book is written by three acclaimed >>>>> leaders in the field. The early access version is available now. >>>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>>>> _______________________________________________ >>>>> Fedora-commons-users mailing list >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>>> >>>>> >>>>> >>>>> >>>>> >>>> ------------------------------------------------------------------------------ >>>>> Learn Graph Databases - Download FREE O'Reilly Book >>>>> "Graph Databases" is the definitive new guide to graph databases and >>>>> their applications. This 200-page book is written by three acclaimed >>>>> leaders in the field. The early access version is available now. >>>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Fedora-commons-users mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>>> >>>> >>>> -- >>>> Scott Prater >>>> Shared Development Group >>>> General Library System >>>> University of Wisconsin - Madison >>>> [email protected] >>>> 5-5415 >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Learn Graph Databases - Download FREE O'Reilly Book >>>> "Graph Databases" is the definitive new guide to graph databases and >>>> their applications. This 200-page book is written by three acclaimed >>>> leaders in the field. The early access version is available now. >>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>>> _______________________________________________ >>>> Fedora-commons-users mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Learn Graph Databases - Download FREE O'Reilly Book >>> "Graph Databases" is the definitive new guide to graph databases and >>> their applications. This 200-page book is written by three acclaimed >>> leaders in the field. The early access version is available now. >>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >>> _______________________________________________ >>> Fedora-commons-users mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >>> >>> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> >> ------------------------------ >> >> ------------------------------------------------------------------------------ >> Learn Graph Databases - Download FREE O'Reilly Book >> "Graph Databases" is the definitive new guide to graph databases and >> their applications. This 200-page book is written by three acclaimed >> leaders in the field. The early access version is available now. >> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may >> >> ------------------------------ >> >> _______________________________________________ >> Fedora-commons-users mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users >> >> >> End of Fedora-commons-users Digest, Vol 75, Issue 7 >> *************************************************** > > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and > their applications. This 200-page book is written by three acclaimed > leaders in the field. The early access version is available now. > Download your free book today! http://p.sf.net/sfu/neotech_d2d_may > _______________________________________________ > Fedora-commons-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > -- Scott Prater Shared Development Group General Library System University of Wisconsin - Madison [email protected] 5-5415 ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
