>
> I can't afford to have this datastream managed by Fedora,
> because it's several megabytes large and there are over a million of them.
Currently it's under $100 per TB of disk. So, your immediate costs are not
that high. You may be spending more by engineering a more complicated
solution.
Justin
On Thu, May 9, 2013 at 4:02 PM, Stefano Cossu <[email protected]> wrote:
> @Scott: I can't afford to have this datastream managed by Fedora,
> because it's several megabytes large and there are over a million of them.
>
> @Rich: the http://user@pass:/resource syntax works with cURL too, that's
> why I gave it a shot even though I don't know what Fedora actually uses
> to connect to remote servers.
> Your source link is very interesting though. I'll give it a look. I
> wonder how this hasn't been brought up before. Accessing resources
> through authentication seems like quite a common task to me, and I hoped
> I could do it without hacking the Fedora code.
>
> Thanks
> sc
>
>
> Stefano Cossu
> Director of Application Services, Collections
>
> The Art Institute of Chicago
> 116 S. Michigan Ave.
> Chicago, IL 60603
> 312-499-4026
>
>
> On 5/9/13 3:45 PM, [email protected]:
> > Send Fedora-commons-users mailing list submissions to
> > [email protected]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> > or, via email, send a message with subject or body 'help' to
> > [email protected]
> >
> > You can reach the person managing the list at
> > [email protected]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Fedora-commons-users digest..."
> >
> >
> > Today's Topics:
> >
> > 1. Re: [NEWBIE] Accessing external resource with HTTP
> > authentication (Benjamin Armintor)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 9 May 2013 16:45:16 -0400
> > From: Benjamin Armintor <[email protected]>
> > Subject: Re: [fcrepo-user] [NEWBIE] Accessing external resource with
> > HTTP authentication
> > To: "Support and info exchange list for Fedora users."
> > <[email protected]>
> > Message-ID:
> > <
> cadqq8tpfxo8va7ub2ncb6xvbjzcbf1s7unu7hftcqhpntwb...@mail.gmail.com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Yes, in that sense it would be straightforward- though this question of
> > where the credentials would be stored was one of things that derailed the
> > feature in the first place!
> >
> >
> > On Thu, May 9, 2013 at 4:40 PM, Rich d'Rich <[email protected]>
> wrote:
> >
> >> AFAIK the username:password@ syntax is a browser artefact that the Java
> >> HTTP access libraries (apache commons httpclient) that Fedora uses
> doesn't
> >> support.
> >>
> >> This also means that you can't do a server-server import where the
> source
> >> Fedora server requires authentication, and it causes problems with
> >> disseminators.
> >>
> >> However, looking at the code, most of the "wiring" is there:
> >>
> >>
> https://github.com/fcrepo/fcrepo/blob/master/fcrepo-server/src/main/java/org/fcrepo/server/access/DefaultAccess.java
> >>
> >> it just needs getDatastreamDissemination (around line 1145) to extract a
> >> username and password from somewhere and put it into
> ContentManagerParams.
> >> Ideally, there would be a configured table of known external servers and
> >> credentials that could be kept secure so passwords aren't bandied about.
> >>
> >> I may be wrong though and there's already a way to do this :-)
> >>
> >>
> >> On 10 May 2013 06:36, Scott Prater <[email protected]> wrote:
> >>
> >>> Stefano --
> >>>
> >>> Are you ingesting the datastreams as managed datastreams, or as
> redirect
> >>> or external datastreams?
> >>>
> >>> If the former, once Fedora ingests the FOXML, the object is referred to
> >>> by its internal Fedora URI, and no source URLs or passwords are exposed
> >>> in any object export.
> >>>
> >>> If the datastreams are managed, then you may want to take a compromise
> >>> approach: fetch them to the local machine using curl or some such tool,
> >>> then ingest the local file. Once it's ingested, you can delete the
> >>> local file.
> >>>
> >>> Managed datastreams are usually preferred to external or redirect
> >>> datastreams; there are use cases for external and redirect datastreams
> >>> (which is why they exist), but the normal case is to store datastreams
> >>> as managed.
> >>>
> >>> -- Scott
> >>>
> >>> On 05/09/2013 01:08 PM, Benjamin Armintor wrote:
> >>>> Stefano-
> >>>> I remember some conversation a couple of years ago about
> supporting
> >>>> BASIC auth in services, but as far as I know they didn't go anywhere.
> >>>> Maybe another committer remembers something? In any case, I don't
> see
> >>>> why storing the credentials like that wouldn't work, if you can accept
> >>>> the plain-text issues you cite.
> >>>>
> >>>> As far as certs, Im afraid you're on your own. I will warn you that
> >>>> Java errs on the side of verification unless you instruct it not to,
> so
> >>>> invalid certs will cause other problems.
> >>>>
> >>>> - Ben
> >>>>
> >>>>
> >>>> On Thu, May 9, 2013 at 12:32 PM, Stefano Cossu <[email protected]
> >>>> <mailto:[email protected]>> wrote:
> >>>>
> >>>> Hi there,
> >>>> I'm starting to tinker with Fedora and trying to write a CMA
> >>> workflow.
> >>>> I'm building a digital object that should grab an image
> datastream
> >>> from
> >>>> an HTTPS server which requires basic authentication.
> >>>> I tried inserting the authentication data in the URL for the
> >>> datastream,
> >>>> but now I have 2 problems:
> >>>> 1) Username and password are stored in plain text in the FOXML,
> >>> visible
> >>>> by everyone who looks up that record in Fedora, as well as all
> over
> >>> the
> >>>> logs.
> >>>> 2) I still can't connect to the server this way. The server's
> >>>> certificate is expired, I don't know if that plays a role.
> >>>>
> >>>> Fedora throws this error:
> >>>>
> >>>> ERROR 2013-05-09 11:04:28.618 [http-8080-1] (BaseRestResource)
> >>>> Unexpected error fulfilling REST API request
> >>>> org.fcrepo.server.errors.HttpServiceNotFoundException:
> >>>> [DefaultExternalContentManager] returned an error. The
> underlying
> >>> error
> >>>> was a org.fcrepo.server.errors.GeneralException T
> >>>> he message was "Error getting
> >>>> https://username:password@imageserver/myHugePicture"; .
> >>>> at
> >>>>
> >>>
> org.fcrepo.server.storage.DefaultExternalContentManager.getExternalContent(DefaultExternalContentManager.java:152)
> >>>> ~[fcrepo-server-3.6.2.jar:na]
> >>>> at
> >>>>
> >>>
> org.fcrepo.server.access.DefaultAccess.getDatastreamDissemination(DefaultAccess.java:1148)
> >>>> ~[fcrepo-server-3.6.2.jar:na]
> >>>> at
> >>>>
> >>>
> org.fcrepo.server.rest.DatastreamResource.getDatastream(DatastreamResource.java:247)
> >>>> ~[fcrepo-server-3.6.2.jar:na]
> >>>> [...]
> >>>>
> >>>> And the image server's Apache error log:
> >>>>
> >>>> Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] Connection
> to
> >>>> child 0 established (server imageserver:443)
> >>>> [Thu May 09 11:04:25 2013] [info] Seeding PRNG with 144 bytes of
> >>> entropy
> >>>> [Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] SSL
> library
> >>> error
> >>>> 1 in handshake (server imageserver:443)
> >>>> [Thu May 09 11:04:25 2013] [info] SSL Library Error: 336151608
> >>>> error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal
> >>> error
> >>>> [Thu May 09 11:04:25 2013] [info] [client 10.80.25.47] Connection
> >>> closed
> >>>> to child 0 with abortive shutdown (server imageserver:443)
> >>>> [...]
> >>>>
> >>>> Of course, I can always use a redirect datastream and let the
> client
> >>>> deal with authentication and SSL, but I'd like to hide the source
> >>> URI if
> >>>> possible.
> >>>>
> >>>>
> >>>> Below is the FOXML representation of my object:
> >>>>
> >>>> <foxml:digitalObject VERSION="1.1" PID="test:dervPub_obj"
> >>>> xsi:schemaLocation="info:fedora/fedora-system:def/foxml#
> >>>> http://www.fedora.info/definitions/1/0/foxml1-1.xsd";>
> >>>> <foxml:objectProperties>
> >>>> <foxml:property
> >>>> NAME="info:fedora/fedora-system:def/model#state" VALUE="Active"/>
> >>>> <foxml:property
> >>>> NAME="info:fedora/fedora-system:def/model#label"
> VALUE="Disseminator
> >>>> object"/>
> >>>> <foxml:property
> >>>> NAME="info:fedora/fedora-system:def/model#ownerId"
> >>> VALUE="fedoraAdmin"/>
> >>>> <foxml:property
> >>>> NAME="info:fedora/fedora-system:def/model#createdDate"
> >>>> VALUE="2013-05-09T15:37:41.708Z"/>
> >>>> <foxml:property
> >>>> NAME="info:fedora/fedora-system:def/view#lastModifiedDate"
> >>>> VALUE="2013-05-09T15:37:41.892Z"/>
> >>>> </foxml:objectProperties>
> >>>> <foxml:datastream ID="AUDIT" STATE="A" CONTROL_GROUP="X"
> >>>> VERSIONABLE="false">
> >>>> <foxml:datastreamVersion ID="AUDIT.0" LABEL="Audit
> Trail
> >>> for
> >>>> this object" CREATED="2013-05-09T15:37:41.708Z"
> MIMETYPE="text/xml"
> >>>> FORMAT_URI="info:fedora/fedora-system:format/xml.fedora.audit">
> >>>> <foxml:xmlContent>
> >>>> <audit:auditTrail>
> >>>> <audit:record ID="AUDREC1">
> >>>> <audit:process type="Fedora API-M"/>
> >>>> <audit:action>addDatastream</audit:action>
> >>>> <audit:componentID>SOURCE_IMG</audit:componentID>
> >>>> <audit:responsibility>fedoraAdmin</audit:responsibility>
> >>>> <audit:date>2013-05-09T15:37:41.892Z</audit:date>
> >>>> <audit:justification/>
> >>>> </audit:record>
> >>>> </audit:auditTrail>
> >>>> </foxml:xmlContent>
> >>>> </foxml:datastreamVersion>
> >>>> </foxml:datastream>
> >>>> <foxml:datastream ID="DC" STATE="A" CONTROL_GROUP="X"
> >>>> VERSIONABLE="true">
> >>>> <foxml:datastreamVersion ID="DC1.0" LABEL="Dublin Core
> >>> Record
> >>>> for this object" CREATED="2013-05-09T15:37:41.708Z"
> >>> MIMETYPE="text/xml"
> >>>> FORMAT_URI="http://www.openarchives.org/OAI/2.0/oai_dc/";
> >>> SIZE="388">
> >>>> <foxml:xmlContent>
> >>>> <oai_dc:dc
> >>>> xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/
> >>>> http://www.openarchives.org/OAI/2.0/oai_dc.xsd";>
> >>>> <dc:title>Disseminator object</dc:title>
> >>>> <dc:identifier>test:dervPub_obj</dc:identifier>
> >>>> </oai_dc:dc>
> >>>> </foxml:xmlContent>
> >>>> </foxml:datastreamVersion>
> >>>> </foxml:datastream>
> >>>> <foxml:datastream ID="RELS-EXT" STATE="A" CONTROL_GROUP="X"
> >>>> VERSIONABLE="false">
> >>>> <foxml:datastreamVersion ID="RELS-EXT.0"
> >>> LABEL="Relationships"
> >>>> CREATED="2013-05-09T15:37:41.837Z" MIMETYPE="application/rdf+xml"
> >>>> FORMAT_URI="info:fedora/fedora-system:FedoraRELSExt-1.0"
> SIZE="273">
> >>>> <foxml:xmlContent>
> >>>> <rdf:RDF>
> >>>> <rdf:Description
> >>>> rdf:about="info:fedora/test:dervPub_obj">
> >>>> <hasModel
> >>>> rdf:resource="info:fedora/test:dervPub_CModel"/>
> >>>> </rdf:Description>
> >>>> </rdf:RDF>
> >>>> </foxml:xmlContent>
> >>>> </foxml:datastreamVersion>
> >>>> </foxml:datastream>
> >>>> <foxml:datastream ID="SOURCE_IMG" STATE="A"
> CONTROL_GROUP="E"
> >>>> VERSIONABLE="true">
> >>>> <foxml:datastreamVersion ID="SOURCE_IMG.0" LABEL="full
> >>> sized
> >>>> image" CREATED="2013-05-09T15:37:41.892Z" MIMETYPE="image/jpeg">
> >>>> <foxml:contentLocation TYPE="URL"
> >>>> REF="https://username:password@imageserver/myHugePicture"/>
> >>>> </foxml:datastreamVersion>
> >>>> </foxml:datastream>
> >>>>
> >>>> I would really appreciate your help.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> ------------------------------------------------------------------------------
> >>>> Learn Graph Databases - Download FREE O'Reilly Book
> >>>> "Graph Databases" is the definitive new guide to graph databases
> and
> >>>> their applications. This 200-page book is written by three
> acclaimed
> >>>> leaders in the field. The early access version is available now.
> >>>> Download your free book today!
> http://p.sf.net/sfu/neotech_d2d_may
> >>>> _______________________________________________
> >>>> Fedora-commons-users mailing list
> >>>> [email protected]
> >>>> <mailto:[email protected]>
> >>>>
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> ------------------------------------------------------------------------------
> >>>> Learn Graph Databases - Download FREE O'Reilly Book
> >>>> "Graph Databases" is the definitive new guide to graph databases and
> >>>> their applications. This 200-page book is written by three acclaimed
> >>>> leaders in the field. The early access version is available now.
> >>>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Fedora-commons-users mailing list
> >>>> [email protected]
> >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >>>>
> >>>
> >>> --
> >>> Scott Prater
> >>> Shared Development Group
> >>> General Library System
> >>> University of Wisconsin - Madison
> >>> [email protected]
> >>> 5-5415
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Learn Graph Databases - Download FREE O'Reilly Book
> >>> "Graph Databases" is the definitive new guide to graph databases and
> >>> their applications. This 200-page book is written by three acclaimed
> >>> leaders in the field. The early access version is available now.
> >>> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
> >>> _______________________________________________
> >>> Fedora-commons-users mailing list
> >>> [email protected]
> >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >>>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Learn Graph Databases - Download FREE O'Reilly Book
> >> "Graph Databases" is the definitive new guide to graph databases and
> >> their applications. This 200-page book is written by three acclaimed
> >> leaders in the field. The early access version is available now.
> >> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
> >> _______________________________________________
> >> Fedora-commons-users mailing list
> >> [email protected]
> >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >>
> >>
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> >
> > ------------------------------
> >
> >
> ------------------------------------------------------------------------------
> > Learn Graph Databases - Download FREE O'Reilly Book
> > "Graph Databases" is the definitive new guide to graph databases and
> > their applications. This 200-page book is written by three acclaimed
> > leaders in the field. The early access version is available now.
> > Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
> >
> > ------------------------------
> >
> > _______________________________________________
> > Fedora-commons-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
> >
> >
> > End of Fedora-commons-users Digest, Vol 75, Issue 7
> > ***************************************************
>
>
>
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and
> their applications. This 200-page book is written by three acclaimed
> leaders in the field. The early access version is available now.
> Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
> _______________________________________________
> Fedora-commons-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
>
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and
their applications. This 200-page book is written by three acclaimed
leaders in the field. The early access version is available now.
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
_______________________________________________
Fedora-commons-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
