Hey YunHee,
This is expected behavoir now... pushpull only downloads the files... it talks to the filemgr to try to determine if it already has a file before it redownloads it... if you want automated file ingest, you must set up a crawler to crawl your pushpull staging directory... there is documentation for that as well... http://oodt.apache.org/components/maven/crawler/user/ ... the command line information is out of date there possibly, however just run: ./crawler_launcher --help ... the help menus are pretty involved... you can also learn how to customize your command-line options here: https://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt
-brian
On Aug 02, 2012, at 10:08 AM, Kang YunHee <yunh.k...@gmail.com> wrote:
On Aug 02, 2012, at 10:08 AM, Kang YunHee <yunh.k...@gmail.com> wrote:
Hi Brain,
I applied the patch(OODT-481.2012-08-01.txt) to the source of pushpull
framework, rebuilt it and swapped its jar out in my deployment according
to the following steps:
patch -p0 < OODT-481.2012-08-01.txt
mvn clean
mvn install
cd target
cp cas-pushpull-0.5-SNAPSHOT.jar ~/oodt-0.5/cas-pushpull/lib/
After I ran the pushpull script, I found out there is no "Catalog
exception" as you can see:
Aug 3, 2012 1:31:25 AM
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
addToDownloadQueue
WARNING: Skipping file {parent = 'null', path =
'/TES/TL2CO2N.005/2004.09.20/TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5',
isDir = 'false'} because it is already in staging area
PageSize: 8 PageLoc: 2
FileList size: 2
PageSize: 8 PageLoc: 952
FileList size: 952
Aug 3, 2012 1:35:38 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Aug 3, 2012 1:35:38 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90121 on RMI registry port 9012 is shutting down
But I am not sure that the file downloaded is ingested by my filemgr.
I think if it was ingested by my filemgr, it would be in the repository
directory.
However I did not find it in my repository directory:
Let me know how to check the status of cataloging mentioned above.
Thanks,
Yunhee
On 8/2/12 6:29 AM, "Brian Foster" <holeno...@me.com> wrote:
>
>hey YunHee,
>
>I've submitted the patch, so you can instead of having to patch the code
>you can just resync your pushpull code and rebuild it and swap the jar
>out in your deployment
>
>-brian
>
>On Aug 1, 2012, at 8:28 AM, YunHee Kang wrote:
>
>> Hi Chris and Brian,
>>
>> I am reading source codes for handling "Catalog exception" related the
>> runtime error described below.
>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> Failure writing request
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>RpcFileManagerClient.java:606)
>> at
>>org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.jav
>>a:284)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlread
>>yInDatabase(FileRetrievalSystem.java:254)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDow
>>nloadQueue(FileRetrievalSystem.java:463)
>> at
>>org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFil
>>e(RemoteCrawler.java:138)
>> at
>>org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles
>>(RetrievalSetup.java:109)
>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> I think that the exception was caused by the wrong value of the
>> parameter productName of the method hasProduct() in the following
>> codelet of XmlRpcFileManagerClient.java:
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(Xml
>>RpcFileManagerClient.java:606)
>> public boolean hasProduct(String productName) throws CatalogException {
>> Vector<Object> argList = new Vector<Object>();
>> argList.add(productName);
>>
>> boolean hasProduct = false;
>>
>> try {
>> hasProduct = ((Boolean) client.execute("filemgr.hasProduct",
>> argList)).booleanValue();
>> } catch (XmlRpcException e) {
>> throw new CatalogException(e.getMessage());
>> } catch (IOException e) {
>> throw new CatalogException(e.getMessage());
>> }
>> return hasProduct;
>> }
>>
>> I inspected that the element "mine-type" described below was added
>> in the file mimestypes.xml that is one of pushpull property files.
>> <mime-type type="product/tes">
>> <_comment>ProductType=MyTesProductType</_comment>
>> <glob
>> pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
>> isregex="true"/>
>> </mime-type>
>>
>> I would like to know what the element "mine-type" means.
>> I am wondering how I can check the value of productName in the
>> properties of Filemgr and Pushpull.
>>
>> I am sorry to bother you again.
>>
>> Thanks,
>> Yunhee
>
