Hey Tom,
On Jul 17, 2012, at 7:57 AM, Thomas Bennett wrote:
> Hi everyone,
>
> Thanks for the responses. I always appreciate the fact that post a
> question/problem on the forum and the next morning there are stack of replies.
>
> +1 to everyone!
Thanks dude!
>
> As it turns out the problem was entirely mine :-)
Well not entirely! :)
Read on below:
>
> However, I've discovered how the file manager works out the percentage file
> transferred and why it thinks the file Percentage Complete was 0%.
>
> The magic all happens with the following piece of code:
>
> private long getBytesTransferred(Reference r) {
> File destFile = null;
> try {
> destFile = new File(new URI(r.getDataStoreReference()));
> long bytesTransferred = destFile.length();
> return bytesTransferred;
> } catch (URISyntaxException e) {
> LOG.log(Level.WARNING,
> "URISyntaxException when checking size of destFile: ["
> + r.getDataStoreReference() + "]: Message: "
> + e.getMessage());
> return -1L;
> }
> }
>
> So the file transfer size is worked out on the fly - its not stored in the
> catalog.
>
> The origin of the weirdness relates to my Versioner.
>
> My versioner stores files to
> file:/var/kat/archive/data/comm/[StartDate]/1234567890.h5
>
> My original code (pre-OODT) stored the data in the same way. However, for
> OODT I store my dates in UTC, and thus there is a 2 hour difference between
> my preOODT and OODT implementations, so for my product 1287353769.h5 the
> references are:
>
> origReference "file:/var/kat/archive/data/comm/2010/10/18/1287353769.h5"
> (id=115)
> dataStoreReference
> "file:/var/kat/archive/data/comm/2010/10/17/1287353769.h5" (id=112)
>
> Since I used the InPlaceDataTransfer the data reference was updated but the
> file was not moved.
>
> So files captured between midnight and 2am will now have a new reference to
> the time shifted directory (e.g. moved from the 18'th to the 17'th)
>
> When the %complete is calculated it uses the dataStoreReference.
>
> Is it possible for the file manager to know that the InPlaceDataTransfer has
> been used and that the reference should not change? I guess this would not be
> possible.
Yep it's possible. You can use the InPlaceVersioner, and extend it, or extend
MetadataBasedVersioner and sub-class to do something similar to the
InPlaceVersioner.
Does that make sense?
Cheers,
Chris
>
> On 17 July 2012 05:29, Sheryl John <[email protected]> wrote:
> > On Jul 16, 2012 8:26 AM, "Thomas Bennett" <[email protected]> wrote:
> > Hi,
> >
> > I've recently created a Lucene file manager catalog. I cataloged about 2000
> > files "in place", but have encountered a problem.
> >
> > From the fmbrowser some of the files are listed as being:
> >
> > Product Transfer Status Percent Complete Received Time
> > References Metadata
> > 1287587805.h5 RECEIVED 100% UNKNOWN View
> > References View Metadata
> > 1287353769.h5 RECEIVED 0% UNKNOWN View
> > References View Metadata
> > 1287355996.h5 RECEIVED 0% UNKNOWN View
> > References View Metadata
> > 1287425397.h5 RECEIVED 100% UNKNOWN View
> > References View Metadata
> >
> > The problem lies with the two products that show % complete being 0%.
> >
> > When I use fmprod to download the product:
> >
> > wget http://..../fmprod/data?productID=26b9cb0b-c5dd-11e1-bb9f-036596780c6e
> >
> > I get:
> > HTTP request sent, awaiting response... 500 Internal Server Error
> > 2012-07-14 20:20:38 ERROR 500: Internal Server Error.
> >
> > So I'd like to "edit" the lucene catalog to update this piece of metadata
> > to reflect that the % complete is 100. Are there any tools available to do
> > this?
> >
> > Also at some point I must figure out why its happening :)
> >
> > Cheers,
> > Tom
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [email protected]
WWW: http://sunset.usc.edu/~mattmann/
Phone: +1 (818) 354-8810
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
