Hi Brain,
I found two mime-types.xml files:
- pushpull/policy/mime-types.xml
- cas-filemgr/etc/mimet-ypes.xml
In the directory pushpull/policy, I put the element "mine-type" like
the following into mimetypes.xml:
<mime-type type="product/tes">
<_comment>ProductType=MyTesProductType</_comment>
<glob
pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
isregex="true"/>
</mime-type>
> fill in an actual filemgr ProductType in place of
> MyTesProdectType... and the pattern should be the regex describing the
> filenames of the files that belong to the give ProductType.
I also put the same "mime-type" into mime-types.xml in the directory
cas-filemgr/etc.
<mime-type type="product/tes">
<_comment>ProductType=MyTesProductType</_comment>
<glob
pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5"
isregex="true"/>
</mime-type>
Am I doing well ?
Thanks,
Yunhee
2012/7/30 Brian Foster <holeno...@me.com>:
> hey YunHee,
>
> I keep forgetting to add you to the email cc... are you on the dev list?...
> see email below, explains your commented out filemgr url making the pushpull
> work
>
> -brian
>
> Begin forwarded message:
>
> From: Brian Foster <holeno...@me.com>
> Date: July 29, 2012 10:12:48 AM PDT
> To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> Cc: Brian Foster <holeno...@mac.com>, "dev@oodt.apache.org"
> <dev@oodt.apache.org>, "Mattmann, Chris A (388J)"
> <chris.a.mattm...@jpl.nasa.gov>
> Subject: Re: Problem when running pushpull
> Reply-To: dev@oodt.apache.org
>
> Hey YunHee,
>
> You need to configure you mime-types.xml file in your policy directory to
> map your file names to filemgr ProductTypes... if you want to just download
> the files for now just comment out the filemgr
> URL property in your pushpull properties file... if you want to configure
> the mime-types.xml file, take a look at:
>
> https://svn.apache.org/repos/asf/oodt/trunk/pushpull/src/main/resources/policy/mimetypes.xml
>
> Here is basically how the file works:
>
> <mime-type type="product/tes">
> <_comment>ProductType=MyTesProductType</_comment>
> <glob pattern="any valid Pattern.java regex" is regex="true"/>
> </mime-type>
>
> the mime-type "type" must have a / and is just a name being used to title
> the mapping... fill in an actual filemgr ProductType in place of
> MyTesProdectType... and the pattern should be the regex describing the
> filenames of the files that belong to the give ProductType.
>
> then make sure that org.apache.oodt.cas.pushpull.config.type.dectection.file
> property points at your mime-types.xml file.
>
> -Brian
> On Jul 29, 2012, at 6:45 AM, YunHee Kang <yunh.k...@gmail.com> wrote:
>
> Hi Brian,
>
>
> I checked out the latest trunk of oodt 0.5 and installed it.
>
> After I set up the file manger and pushpull framework, I ran the
>
> pushpull script.
>
> As I described below, I think that the trunk is useful to fix the
>
> problem, which was caused by handling ftp protocol.
>
> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>
> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>
> username 'anonymous'
>
> Jul 29, 2012 10:09:30 PM
>
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
>
> initializeStagingArea
>
> INFO: Preparing staging area
> /home/yhkang/oodt-0.5/cas-pushpull/staging/TESL2CO2
>
> Jul 29, 2012 10:09:30 PM org.apache.commons.httpclient.HttpClient <clinit>
>
> FINE: Java version: 1.6.0_31
>
>
> But I got the difference problem that is related with filemgr.
>
> The following is part of error message in the file cas-pushpull0.log :
>
> Jul 29, 2012 10:09:30 PM
>
> org.apache.oodt.cas.filemgr.ingest.StdIngester hasProduct
>
> WARNING: Unable to check for existance of product: [null]: Message:
>
> Failure writing request
>
> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>
> Failure writing request
>
> at
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606)
>
> at
> org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284)
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254)
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463)
>
> at
> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138)
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Jul 29, 2012 10:09:30 PM
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>
> retrieveFiles
>
> SEVERE: Failed to finish downloading per property files
>
> /home/yhkang/oodt-0.5/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>
> : [RemoteCrawler] Failed to communicate with database : Failure
>
> writing request
>
> org.apache.oodt.cas.pushpull.exceptions.RetrievalMethodException:
>
> [RemoteCrawler] Failed to communicate with database : Failure writing
>
> request
>
> at
> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:150)
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Jul 29, 2012 10:09:30 PM
>
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
>
> INFO: Disconnecting protocol
>
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
>
> Jul 29, 2012 10:09:30 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>
> INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to
>
> sleep until Sun Jul 29 22:12:00 KST 2012
>
>
> Please let me know how to fix this problem.
>
>
> Thanks,
>
> Yunhee
>
>
>
>
> 2012/7/28 Brian Foster <holeno...@me.com>:
>
> Hey YunHee,
>
>
> Doesn't look like the patch took this time, but don't worry about that now,
> i have put the fixes into the trunk... if you just check out the latest
> trunk and mvn install it (make sure you check out more than just
> cas-pushpull, because if protocol-ftp doesn't get reinstalled your
> cas-pushpull mvn install will pick up the last mvn installed version of the
> protocol-ftp 0.5 in your local maven repository... another thing to note,
> you probably should utilize the environment variables in place in the
> pushpull config files because if you ever decide to move your install you
> are going to have to change all the paths in your configuration files
> instead of just being able to change environment variables (if you have any
> question about that let me know)... again if you run into any road blocks
> just hit up the list and sorry about the bumpy road of getting pushpull up
> and running... it should be good to go now though (it downloads the files on
> my machine at least... lol).
>
>
> -brian
>
>
> On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote:
>
>
> Hi Brain,
>
>
> Unfortunately your patch didn't fix the problem. I got the same
>
> error message before:
>
> SEVERE: Failed to finish downloading per property files
>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>
> : null
>
> java.lang.NullPointerException
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>
> at
> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>
>
> First of all, I replaced CommonsNetFtpProtocol.java with new one
>
> that you modified as follows:
>
> ftp.retrieveFile(fromFile.getName(), os);
>
> /*
>
> if (ftp.retrieveFile(fromFile.getName(), os))// {
>
> throw new ProtocolException("Failed to download file "
>
> + fromFile.getName());
>
> // }
>
>
> */
>
> Next I rebuild a new jar file cas-protocol-ftp-0.4.jar and then
>
> copied it to lib directory:
>
> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar
>
> ls -al cas-protocol-ftp-0.4.jar
>
>
> I am wondering that there is difference between the following codes(1 and
> 2) :
>
> 1) ftp.retrieveFile(fromFile.getName(), os);
>
> 2) if (ftp.retrieveFile(fromFile.getName(), os))// {
>
> throw new ProtocolException("Failed to download file "
>
> + fromFile.getName());
>
> // }
>
>
> Will you let me know if the method get() in the
>
> CommonsNetFtpProtocol.java was called when pushpull was run ?
>
>
> Thanks,
>
> Yunhee
>
>
>
> 2012/7/27 Brian Foster <holeno...@mac.com>:
>
> Hey YunHee,
>
>
> Okay the latest patch attached to the issue now (the on with todays date)
>
> works as far as pushpull is concerned, however FTPClient.retrieveFile being
>
> used in CommonsNetFtpProtocol is returning false for some reason... so i
>
> would recommend trying the following... first unpatch the first patch from
>
> your cas-pushpull and repatch with the new patch i've uploaded... then check
>
> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify
>
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this:
>
>
>
> if (ftp.retrieveFile(fromFile.getName(), os))// {
>
>
> throw new ProtocolException("Failed to download file "
>
>
> + fromFile.getName());
>
>
> // }
>
>
>
> to this:
>
>
>
> ftp.retrieveFile(fromFile.getName(), os);
>
>
>
> then mvn compile jar:jar and replace ftp protocol jar with that one in your
>
> pushpull lib directory and give pushpull another run and see what happens
>
> (validate the files downloaded to make sure they are complete)... hope this
>
> helps and let me know if you need any further help!
>
>
>
> -brian
>
>
>
> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.k...@gmail.com> wrote:
>
>
> Hi Brian,
>
>
> I attach a tar file including etc and policy directories.
>
>
> Thanks,
>
> Yunhee
>
>
> 2012/7/27 Brian Foster <holeno...@mac.com>:
>
> Hey YunHee,
>
>
> send me a tarball with your etc and policy directories i will spend some
>
> time today figuring out what is going on here
>
>
> -brian
>
>
>
> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.k...@gmail.com> wrote:
>
>
> Hi Brian,
>
>
> I just applied the patch you made according to the following steps:
>
> cd apache-oodt-0.4-src/pushpull
>
> patch -p0 < OODT-476.2012-07-25.txt
>
> mvn clean
>
> mvn install
>
> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/
>
>
> After I got a new cas-pushpull-0.4.jar,
>
> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar
>
> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar
>
> I ran the script pushpull.
>
>
>
> But the same error message was generated:
>
>
> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>
> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>
> username 'anonymous'
>
> Jul 26, 2012 5:12:56 PM
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>
> retrieveFiles
>
> SEVERE: Failed to finish downloading per property files
>
>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>
> : null
>
> java.lang.NullPointerException
>
> at
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>
> at
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>
> at
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>
> at
>
>
> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>
> at
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>
> at java.lang.Thread.run(Thread.java:662)
>
>
> Could you check a class name for handling ftp protocol ?
>
> I used "CommonsNetFtpProtocolFactory.class" in the jar named
>
> "cas-protocol-ftp-0.4.jar".
>
> The class file was described in the file ProtocolFactoryInfo.xml as
>
> follows:
>
> <protocol type="ftp">
>
> ....
>
> <!-- cas-protocol-ftp-0.4.jar -->
>
> <!--protocolFactory
>
> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>
> <protocolFactory
>
> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>
> </protocol>
>
>
> Thanks,
>
> Yunhee
>
>
>
>
>
> 2012/7/26 Brian Foster <holeno...@mac.com>:
>
> hey Yunhee,
>
>
> i just realized that your probably not on the dev list and my last email
>
> just went to the list and didn't have you cc'ed
>
>
> try the patch attached to this issue mentioned below...
>
>
> -brian
>
>
> Begin forwarded message:
>
>
> From: Brian Foster <holeno...@mac.com>
>
> Date: July 25, 2012 12:43:10 AM
>
> To: dev@oodt.apache.org,dev@oodt.apache.org
>
> Subject: Re: Problem when running pushpull
>
>
> hey Yunhee,
>
>
> Try patching your pushpull code with the fix i've attached to issue:
>
>
> https://issues.apache.org/jira/browse/OODT-476
>
>
> -brian
>
>
> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote:
>
>
> Hi Brian,
>
>
> I changed the file RetrievalSetup.java to get the stacktrace.
>
> So I got a new cas-pushpull-0.4.jar after rebuilt it.
>
>
> The following is a message including the stacktrace with information
>
> about excpetion propation.
>
>
> SEVERE: Failed to finish downloading per property files
>
>
>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>
> : null
>
> java.lang.NullPointerException
>
> at
>
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367)
>
> at
>
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311)
>
> at
>
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300)
>
> at
>
>
>
> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103)
>
> at
>
>
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>
> at java.lang.Thread.run(Thread.java:662)
>
>
> Please let me know what java programs I check frist.
>
>
> Thanks,
>
>
> Yunhee
>
> 2012/7/25 Brian Foster <holeno...@mac.com>:
>
> YunHee,
>
>
> There is a NullPointerException being thrown... on line 118 of
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change
>
> it
>
>
> rebuild and deploy and that should give you the stacktrace which will
>
> answer
>
> what is going on here.
>
>
> -brian
>
>
>
> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)"
>
>
> <chris.a.mattm...@jpl.nasa.gov> wrote:
>
>
> Hi YunHee,
>
>
> Thanks for your mail.
>
>
> Let me take a look at the configs again. Does anyone else using
>
> PushPull have any advice for YunHee here? That error message
>
> seems to indicate there is a config file error somewhere. I'm wondering
>
> if your etc/push_pull_framework.properties has all the right values
>
> in there. Did you go through Cameron's guide to getting Push Pull
>
> running on the wiki?
>
>
> Cheers,
>
> Chris
>
>
>
> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:
>
>
> Hi Chris,
>
>
> There are some problems in the configuration files.
>
> I found there is a dead link which does not connect two information
>
> in two files :RemoteSpecs.xml and ExternalSources.xml.
>
> For example an alias named "DewFtp" in RemoteSpecs.xml has no
>
> associated source in "ExternalSources.xml.
>
>
> After I tailored those two XML files, I just tested one of
>
> configurations, whose alias is PolarNcepNoaaFtp.
>
> I also faced the same problem as follows:
>
>
> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
>
> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>
> username 'anonymous'
>
> Jul 25, 2012 2:21:04 AM
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>
> retrieveFiles
>
> SEVERE: Failed to finish downloading per property files
>
>
>
>
>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>
> : null
>
> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>
> sleep
>
>
>
> Thanks,
>
> Yunhee
>
>
> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>
> Hi YunHee,
>
>
> Thanks!
>
>
> Cheers,
>
> Chris
>
>
>
> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>
>
> Hi Chris,
>
>
> Thanks, can you try some of the other XML file configs.
>
> Yes, I will try it and let you know its result.
>
>
> Maybe it's being blocked in your country,
>
> That site may be blocked.
>
>
>
> Thanks,
>
> Yunhee
>
>
> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>
> Hi YunHee,
>
>
>
> Thanks, can you try some of the other XML file configs. Maybe it's
>
> being
>
> blocked in your country, or something, but let's see if we can get
>
> some
>
> of the other XML RemoteSpec configs to work?
>
>
> Cheers,
>
> Chris
>
>
> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>
>
> Hi Chris,
>
>
>
> Try out the MODIS one there, does that work?
>
> No, it doesn't.
>
> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection
>
> problem.
>
>
> $ ftp e4ftl01u.ecs.nasa.gov
>
> ftp: connect: Connection timed out
>
> ftp>
>
>
> The message below also say the reason why the problem was occurred.
>
> WARNING: Error occurred while connecting to RemoteSite: alias =
>
> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username =
>
> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>
> connect to server : Connection timed out
>
> Jul 25, 2012 1:31:25 AM
>
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>
> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url =
>
> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir =
>
> 'null' maxConnections = '-1' in 5 seconds
>
> Waiting . . . . .
>
>
> Thanks,
>
> Yunhee
>
>
> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>
> Hi YunHee,
>
>
>
> OK, can you try some of the other files, e.g., the ones in:
>
>
> http://s.apache.org/ks0
>
>
> Try out the MODIS one there, does that work?
>
>
> Cheers,
>
> Chris
>
>
> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>
>
> Hi Chris,
>
>
> I added the header of XML as follows:
>
> <?xml version="1.0"?>
>
>
> Next the value of elements was changed into specific one.
>
> The following is the output of the XML:
>
> <?xml version="1.0"?>
>
> <root>
>
> <dirstruct starting_path="/TES/TL2CO2N.005">
>
> <nofiles/>
>
> <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>
> <dir name="2004\.09\.20">
>
> <nodirs/>
>
> <!--file
>
> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>
> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>
> </dir>
>
> </dirstruct>
>
> </root>
>
>
> Unfortunately I got the same result before:
>
> ...
>
> INFO: Testing protocol
>
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this
>
> ma
>
> y take a few minutes . . .
>
> Jul 25, 2012 12:30:03 AM
>
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>
> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>
> protocol 'org.apache.oodt.
>
> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>
> Jul 25, 2012 12:30:03 AM
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>
> retrie
>
> veFiles
>
> SEVERE: Failed to finish downloading per property files
>
> /home/yhkang/oodt/cas-pushpull/etc/
>
> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>
> Jul 25, 2012 12:30:04 AM
>
> org.apache.oodt.cas.pushpull.daemon.Daemon
>
> sleep
>
> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going
>
> to
>
> sleep until Wed Jul 25 0
>
> ...
>
>
>
> Thanks,
>
> Yunhee
>
>
>
> 2012/7/25 Mattmann, Chris A (388J)
>
> <chris.a.mattm...@jpl.nasa.gov>:
>
> Hi YunHee,
>
>
>
> Can you try simply putting a specific <dir name="..."> instead
>
> of
>
> \d{4}, etc., and
>
> then can you put a specific file name in the <file name="..">.
>
> If
>
> that works and
>
> downloads, then we know there is something up with the regex
>
> somehow.
>
> Also can you try adding the <?xml version="1.0"?> start to your
>
> file?
>
>
> Cheers,
>
> Chris
>
>
> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>
>
> Hi Chris,
>
>
>
> The following is the content of TESL2CO2.xml.
>
> $ cat
>
>
>
>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>
> <root>
>
> <dirstruct starting_path="/TES/TL2CO2N.005">
>
> <nofiles/>
>
> <dir name="\d{4}\.\d{2}\.\d{2}">
>
> <nodirs/>
>
> <file
>
> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>
> </dir>
>
> </dirstruct>
>
> </root>
>
>
> Let me know if there is a problem in the configuration file.
>
>
> Thanks,
>
> Yunhee
>
>
> 2012/7/24 Mattmann, Chris A (388J)
>
> <chris.a.mattm...@jpl.nasa.gov>:
>
> Hi YunHee,
>
>
>
> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>
>
> Hi Chris,
>
>
> I am stuck on the problem related with "pushpull" script
>
> occurred
>
> the following error message:
>
> ....
>
> INFO: Testing protocol
>
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . .
>
> this may
>
> take a few minutes . . .
>
> Jul 24, 2012 4:05:02 PM
>
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>
> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov
>
> with
>
> protocol
>
> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>
> username 'anonymous'
>
> Jul 24, 2012 4:05:02 PM
>
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>
> retrieveFiles
>
> SEVERE: Failed to finish downloading per property files
>
>
>
>
>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>
> : null
>
>
> Can you do me a favor and simply run this command:
>
>
> cat
>
>
>
>
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>
>
> What do you get?
>
>
> Jul 24, 2012 4:05:02 PM
>
> org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>
> INFO: Daemon with ID = 91131 on RMI registry port 9113 is
>
> going
>
> to
>
> sleep until Tue Jul 24 16:06:00 KST 2012
>
> Jul 24, 2012 4:06:00 PM
>
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
>
> setAsRunning
>
> INFO: Daemon with ID = 91131 was given permission to run
>
>
>
>
>
> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>
>
> First of all, I would like test the pushpull framework to
>
> move
>
> data
>
> written in NetCDF from a remote site by a network protocols
>
> (like ftp,
>
> http, etc).
>
>
> Yep, sounds good.
>
>
> I think that there is a test suite which can be used to check
>
> the
>
> pushpull framework.
>
> Can you give configuration files(a set of XML files) that I
>
> can
>
> test
>
> downloading files from a FTP site (public anonymous site) ?
>
>
> The examples in the resources directory that you found should
>
> work
>
> so we'll just need to figure out why they aren't working on
>
> your
>
> system.
>
>
> or
>
> Please let me know a reference implementation of pushpull
>
> framework
>
> that I can use to move NetCDF files.
>
>
> Thanks we'll figure this out.
>
>
> Cheers,
>
> Chris
>
>
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Chris Mattmann, Ph.D.
>
> Senior Computer Scientist
>
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>
> Office: 171-266B, Mailstop: 171-246
>
> Email: chris.a.mattm...@nasa.gov
>
> WWW: http://sunset.usc.edu/~mattmann/
>
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Adjunct Assistant Professor, Computer Science Department
>
> University of Southern California, Los Angeles, CA 90089 USA
>
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Chris Mattmann, Ph.D.
>
> Senior Computer Scientist
>
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>
> Office: 171-266B, Mailstop: 171-246
>
> Email: chris.a.mattm...@nasa.gov
>
> WWW: http://sunset.usc.edu/~mattmann/
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Adjunct Assistant Professor, Computer Science Department
>
> University of Southern California, Los Angeles, CA 90089 USA
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Chris Mattmann, Ph.D.
>
> Senior Computer Scientist
>
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>
> Office: 171-266B, Mailstop: 171-246
>
> Email: chris.a.mattm...@nasa.gov
>
> WWW: http://sunset.usc.edu/~mattmann/
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Adjunct Assistant Professor, Computer Science Department
>
> University of Southern California, Los Angeles, CA 90089 USA
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Chris Mattmann, Ph.D.
>
> Senior Computer Scientist
>
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>
> Office: 171-266B, Mailstop: 171-246
>
> Email: chris.a.mattm...@nasa.gov
>
> WWW: http://sunset.usc.edu/~mattmann/
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Adjunct Assistant Professor, Computer Science Department
>
> University of Southern California, Los Angeles, CA 90089 USA
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Chris Mattmann, Ph.D.
>
> Senior Computer Scientist
>
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>
> Office: 171-266B, Mailstop: 171-246
>
> Email: chris.a.mattm...@nasa.gov
>
> WWW: http://sunset.usc.edu/~mattmann/
>
> Phone: +1 (818) 354-8810
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Adjunct Assistant Professor, Computer Science Department
>
> University of Southern California, Los Angeles, CA 90089 USA
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Chris Mattmann, Ph.D.
>
> Senior Computer Scientist
>
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>
> Office: 171-266B, Mailstop: 171-246
>
> Email: chris.a.mattm...@nasa.gov
>
> WWW: http://sunset.usc.edu/~mattmann/
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Adjunct Assistant Professor, Computer Science Department
>
> University of Southern California, Los Angeles, CA 90089 USA
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
