Hi YunHee, Looking at: http://s.apache.org/coZ I think you need to define the environment variable FILEMGR_LOC, so that PushPull knows how to communicate with File Manager to determine whether or not you have already ingested a product that it is trying to download. Alternatively, you can remove the:
# ingester filemgr url org.apache.oodt.cas.filemgr.url=[FILEMGR_LOC] property from your push_pull_framework.properties file and you should be set. Cheers, Chris On Jul 29, 2012, at 6:45 AM, YunHee Kang wrote: > Hi Brian, > > I checked out the latest trunk of oodt 0.5 and installed it. > After I set up the file manger and pushpull framework, I ran the > pushpull script. > As I described below, I think that the trunk is useful to fix the > problem, which was caused by handling ftp protocol. > INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with > protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and > username 'anonymous' > Jul 29, 2012 10:09:30 PM > org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem > initializeStagingArea > INFO: Preparing staging area > /home/yhkang/oodt-0.5/cas-pushpull/staging/TESL2CO2 > Jul 29, 2012 10:09:30 PM org.apache.commons.httpclient.HttpClient <clinit> > FINE: Java version: 1.6.0_31 > > But I got the difference problem that is related with filemgr. > The following is part of error message in the file cas-pushpull0.log : > Jul 29, 2012 10:09:30 PM > org.apache.oodt.cas.filemgr.ingest.StdIngester hasProduct > WARNING: Unable to check for existance of product: [null]: Message: > Failure writing request > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: > Failure writing request > at > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606) > at > org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284) > at > org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254) > at > org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463) > at > org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138) > at > org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) > at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) > at java.lang.Thread.run(Thread.java:662) > Jul 29, 2012 10:09:30 PM > org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup > retrieveFiles > SEVERE: Failed to finish downloading per property files > /home/yhkang/oodt-0.5/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml > : [RemoteCrawler] Failed to communicate with database : Failure > writing request > org.apache.oodt.cas.pushpull.exceptions.RetrievalMethodException: > [RemoteCrawler] Failed to communicate with database : Failure writing > request > at > org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:150) > at > org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) > at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) > at java.lang.Thread.run(Thread.java:662) > Jul 29, 2012 10:09:30 PM > org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect > INFO: Disconnecting protocol > org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol > Jul 29, 2012 10:09:30 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep > INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to > sleep until Sun Jul 29 22:12:00 KST 2012 > > Please let me know how to fix this problem. > > Thanks, > Yunhee > > > > 2012/7/28 Brian Foster <holeno...@me.com>: >> Hey YunHee, >> >> Doesn't look like the patch took this time, but don't worry about that now, >> i have put the fixes into the trunk... if you just check out the latest >> trunk and mvn install it (make sure you check out more than just >> cas-pushpull, because if protocol-ftp doesn't get reinstalled your >> cas-pushpull mvn install will pick up the last mvn installed version of the >> protocol-ftp 0.5 in your local maven repository... another thing to note, >> you probably should utilize the environment variables in place in the >> pushpull config files because if you ever decide to move your install you >> are going to have to change all the paths in your configuration files >> instead of just being able to change environment variables (if you have any >> question about that let me know)... again if you run into any road blocks >> just hit up the list and sorry about the bumpy road of getting pushpull up >> and running... it should be good to go now though (it downloads the files on >> my machine at least... lol). >> >> -brian >> >> On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote: >> >>> Hi Brain, >>> >>> Unfortunately your patch didn't fix the problem. I got the same >>> error message before: >>> SEVERE: Failed to finish downloading per property files >>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>> : null >>> java.lang.NullPointerException >>> at >>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368) >>> at >>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312) >>> at >>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301) >>> at >>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104) >>> at >>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >>> at java.lang.Thread.run(Thread.java:662) >>> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>> >>> First of all, I replaced CommonsNetFtpProtocol.java with new one >>> that you modified as follows: >>> ftp.retrieveFile(fromFile.getName(), os); >>> /* >>> if (ftp.retrieveFile(fromFile.getName(), os))// { >>> throw new ProtocolException("Failed to download file " >>> + fromFile.getName()); >>> // } >>> >>> */ >>> Next I rebuild a new jar file cas-protocol-ftp-0.4.jar and then >>> copied it to lib directory: >>> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar >>> ls -al cas-protocol-ftp-0.4.jar >>> >>> I am wondering that there is difference between the following codes(1 and >>> 2) : >>> 1) ftp.retrieveFile(fromFile.getName(), os); >>> 2) if (ftp.retrieveFile(fromFile.getName(), os))// { >>> throw new ProtocolException("Failed to download file " >>> + fromFile.getName()); >>> // } >>> >>> Will you let me know if the method get() in the >>> CommonsNetFtpProtocol.java was called when pushpull was run ? >>> >>> Thanks, >>> Yunhee >>> >>> >>> 2012/7/27 Brian Foster <holeno...@mac.com>: >>>> Hey YunHee, >>>> >>>> Okay the latest patch attached to the issue now (the on with todays date) >>>> works as far as pushpull is concerned, however FTPClient.retrieveFile being >>>> used in CommonsNetFtpProtocol is returning false for some reason... so i >>>> would recommend trying the following... first unpatch the first patch from >>>> your cas-pushpull and repatch with the new patch i've uploaded... then >>>> check >>>> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify >>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this: >>>> >>>> >>>> if (ftp.retrieveFile(fromFile.getName(), os))// { >>>> >>>> throw new ProtocolException("Failed to download file " >>>> >>>> + fromFile.getName()); >>>> >>>> // } >>>> >>>> >>>> to this: >>>> >>>> >>>> ftp.retrieveFile(fromFile.getName(), os); >>>> >>>> >>>> then mvn compile jar:jar and replace ftp protocol jar with that one in your >>>> pushpull lib directory and give pushpull another run and see what happens >>>> (validate the files downloaded to make sure they are complete)... hope this >>>> helps and let me know if you need any further help! >>>> >>>> >>>> -brian >>>> >>>> >>>> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.k...@gmail.com> wrote: >>>> >>>> Hi Brian, >>>> >>>> I attach a tar file including etc and policy directories. >>>> >>>> Thanks, >>>> Yunhee >>>> >>>> 2012/7/27 Brian Foster <holeno...@mac.com>: >>>>> Hey YunHee, >>>>> >>>>> send me a tarball with your etc and policy directories i will spend some >>>>> time today figuring out what is going on here >>>>> >>>>> -brian >>>>> >>>>> >>>>> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.k...@gmail.com> wrote: >>>>> >>>>> Hi Brian, >>>>> >>>>> I just applied the patch you made according to the following steps: >>>>> cd apache-oodt-0.4-src/pushpull >>>>> patch -p0 < OODT-476.2012-07-25.txt >>>>> mvn clean >>>>> mvn install >>>>> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/ >>>>> >>>>> After I got a new cas-pushpull-0.4.jar, >>>>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar >>>>> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar >>>>> I ran the script pushpull. >>>>> >>>>> >>>>> But the same error message was generated: >>>>> >>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>> username 'anonymous' >>>>> Jul 26, 2012 5:12:56 PM >>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>> retrieveFiles >>>>> SEVERE: Failed to finish downloading per property files >>>>> >>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>> : null >>>>> java.lang.NullPointerException >>>>> at >>>>> >>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368) >>>>> at >>>>> >>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312) >>>>> at >>>>> >>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301) >>>>> at >>>>> >>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104) >>>>> at >>>>> >>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> Could you check a class name for handling ftp protocol ? >>>>> I used "CommonsNetFtpProtocolFactory.class" in the jar named >>>>> "cas-protocol-ftp-0.4.jar". >>>>> The class file was described in the file ProtocolFactoryInfo.xml as >>>>> follows: >>>>> <protocol type="ftp"> >>>>> .... >>>>> <!-- cas-protocol-ftp-0.4.jar --> >>>>> <!--protocolFactory >>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/--> >>>>> <protocolFactory >>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/> >>>>> </protocol> >>>>> >>>>> Thanks, >>>>> Yunhee >>>>> >>>>> >>>>> >>>>> >>>>> 2012/7/26 Brian Foster <holeno...@mac.com>: >>>>>> hey Yunhee, >>>>>> >>>>>> i just realized that your probably not on the dev list and my last email >>>>>> just went to the list and didn't have you cc'ed >>>>>> >>>>>> try the patch attached to this issue mentioned below... >>>>>> >>>>>> -brian >>>>>> >>>>>> Begin forwarded message: >>>>>> >>>>>> From: Brian Foster <holeno...@mac.com> >>>>>> Date: July 25, 2012 12:43:10 AM >>>>>> To: dev@oodt.apache.org,dev@oodt.apache.org >>>>>> Subject: Re: Problem when running pushpull >>>>>> >>>>>> hey Yunhee, >>>>>> >>>>>> Try patching your pushpull code with the fix i've attached to issue: >>>>>> >>>>>> https://issues.apache.org/jira/browse/OODT-476 >>>>>> >>>>>> -brian >>>>>> >>>>>> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote: >>>>>> >>>>>> Hi Brian, >>>>>> >>>>>> I changed the file RetrievalSetup.java to get the stacktrace. >>>>>> So I got a new cas-pushpull-0.4.jar after rebuilt it. >>>>>> >>>>>> The following is a message including the stacktrace with information >>>>>> about excpetion propation. >>>>>> >>>>>> SEVERE: Failed to finish downloading per property files >>>>>> >>>>>> >>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >>>>>> : null >>>>>> java.lang.NullPointerException >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >>>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> Please let me know what java programs I check frist. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Yunhee >>>>>> 2012/7/25 Brian Foster <holeno...@mac.com>: >>>>>>> YunHee, >>>>>>> >>>>>>> There is a NullPointerException being thrown... on line 118 of >>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change >>>>>>> it >>>>>>> >>>>>>> rebuild and deploy and that should give you the stacktrace which will >>>>>>> answer >>>>>>> what is going on here. >>>>>>> >>>>>>> -brian >>>>>>> >>>>>>> >>>>>>> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)" >>>>>> >>>>>>> <chris.a.mattm...@jpl.nasa.gov> wrote: >>>>>>> >>>>>>> Hi YunHee, >>>>>>> >>>>>>> Thanks for your mail. >>>>>>> >>>>>>> Let me take a look at the configs again. Does anyone else using >>>>>>> PushPull have any advice for YunHee here? That error message >>>>>>> seems to indicate there is a config file error somewhere. I'm wondering >>>>>>> if your etc/push_pull_framework.properties has all the right values >>>>>>> in there. Did you go through Cameron's guide to getting Push Pull >>>>>>> running on the wiki? >>>>>>> >>>>>>> Cheers, >>>>>>> Chris >>>>>> >>>>>>> >>>>>>> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote: >>>>>>> >>>>>>>> Hi Chris, >>>>>>>> >>>>>>>> There are some problems in the configuration files. >>>>>>>> I found there is a dead link which does not connect two information >>>>>>>> in two files :RemoteSpecs.xml and ExternalSources.xml. >>>>>>>> For example an alias named "DewFtp" in RemoteSpecs.xml has no >>>>>>>> associated source in "ExternalSources.xml. >>>>>>>> >>>>>>>> After I tailored those two XML files, I just tested one of >>>>>>>> configurations, whose alias is PolarNcepNoaaFtp. >>>>>>>> I also faced the same problem as follows: >>>>>>>> >>>>>>>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with >>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>>>>> username 'anonymous' >>>>>>>> Jul 25, 2012 2:21:04 AM >>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>> retrieveFiles >>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >>>>>>>> : null >>>>>>>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>> sleep >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Yunhee >>>>>>>> >>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>> Hi YunHee, >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Chris >>>>>> >>>>>>>>> >>>>>>>>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote: >>>>>>>>> >>>>>>>>>> Hi Chris, >>>>>>>>>> >>>>>>>>>>> Thanks, can you try some of the other XML file configs. >>>>>>>>>> Yes, I will try it and let you know its result. >>>>>>>>>> >>>>>>>>>>> Maybe it's being blocked in your country, >>>>>>>>>> That site may be blocked. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Yunhee >>>>>>>>>> >>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>> Hi YunHee, >>>>>>>>>>> >>>>>> >>>>>>>>>>> Thanks, can you try some of the other XML file configs. Maybe it's >>>>>>>>>>> being >>>>>>>>>>> blocked in your country, or something, but let's see if we can get >>>>>>>>>>> some >>>>>>>>>>> of the other XML RemoteSpec configs to work? >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Chris >>>>>>>>>>> >>>>>>>>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Chris, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Try out the MODIS one there, does that work? >>>>>>>>>>>> No, it doesn't. >>>>>>>>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection >>>>>>>>>>>> problem. >>>>>>>>>>>> >>>>>>>>>>>> $ ftp e4ftl01u.ecs.nasa.gov >>>>>>>>>>>> ftp: connect: Connection timed out >>>>>>>>>>>> ftp> >>>>>>>>>>>> >>>>>>>>>>>> The message below also say the reason why the problem was occurred. >>>>>>>>>>>> WARNING: Error occurred while connecting to RemoteSite: alias = >>>>>>>>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username = >>>>>>>>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to >>>>>>>>>>>> connect to server : Connection timed out >>>>>>>>>>>> Jul 25, 2012 1:31:25 AM >>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url = >>>>>>>>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir = >>>>>>>>>>>> 'null' maxConnections = '-1' in 5 seconds >>>>>>>>>>>> Waiting . . . . . >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Yunhee >>>>>>>>>>>> >>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>>> Hi YunHee, >>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>> OK, can you try some of the other files, e.g., the ones in: >>>>>>>>>>>>> >>>>>>>>>>>>> http://s.apache.org/ks0 >>>>>>>>>>>>> >>>>>>>>>>>>> Try out the MODIS one there, does that work? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Chris >>>>>>>>>>>>> >>>>>>>>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I added the header of XML as follows: >>>>>>>>>>>>>> <?xml version="1.0"?> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Next the value of elements was changed into specific one. >>>>>>>>>>>>>> The following is the output of the XML: >>>>>>>>>>>>>> <?xml version="1.0"?> >>>>>>>>>>>>>> <root> >>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>>>>>>> <nofiles/> >>>>>>>>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"--> >>>>>>>>>>>>>> <dir name="2004\.09\.20"> >>>>>>>>>>>>>> <nodirs/> >>>>>>>>>>>>>> <!--file >>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/--> >>>>>>>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/> >>>>>>>>>>>>>> </dir> >>>>>>>>>>>>>> </dirstruct> >>>>>>>>>>>>>> </root> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Unfortunately I got the same result before: >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> INFO: Testing protocol >>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this >>>>>>>>>>>>>> ma >>>>>>>>>>>>>> y take a few minutes . . . >>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>>>>>>>>> protocol 'org.apache.oodt. >>>>>>>>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' >>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>>>>>>> retrie >>>>>>>>>>>>>> veFiles >>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/ >>>>>>>>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null >>>>>>>>>>>>>> Jul 25, 2012 12:30:04 AM >>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>>>>>>>> sleep >>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going >>>>>>>>>>>>>> to >>>>>>>>>>>>>> sleep until Wed Jul 25 0 >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Yunhee >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) >>>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>>>>> Hi YunHee, >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> Can you try simply putting a specific <dir name="..."> instead >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>> \d{4}, etc., and >>>>>>>>>>>>>>> then can you put a specific file name in the <file name="..">. >>>>>>>>>>>>>>> If >>>>>>>>>>>>>>> that works and >>>>>>>>>>>>>>> downloads, then we know there is something up with the regex >>>>>>>>>>>>>>> somehow. >>>>>>>>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your >>>>>>>>>>>>>>> file? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The following is the content of TESL2CO2.xml. >>>>>>>>>>>>>>>> $ cat >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>>>>>> <root> >>>>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>>>>>>>>> <nofiles/> >>>>>>>>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}"> >>>>>>>>>>>>>>>> <nodirs/> >>>>>>>>>>>>>>>> <file >>>>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/> >>>>>>>>>>>>>>>> </dir> >>>>>>>>>>>>>>>> </dirstruct> >>>>>>>>>>>>>>>> </root> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Let me know if there is a problem in the configuration file. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Yunhee >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J) >>>>>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>>>>>>> Hi YunHee, >>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I am stuck on the problem related with "pushpull" script >>>>>>>>>>>>>>>>>> occurred >>>>>>>>>>>>>>>>>> the following error message: >>>>>>>>>>>>>>>>>> .... >>>>>>>>>>>>>>>>>> INFO: Testing protocol >>>>>>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . >>>>>>>>>>>>>>>>>> this may >>>>>>>>>>>>>>>>>> take a few minutes . . . >>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov >>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>>>>>>>>>>>>>>> username 'anonymous' >>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>>>>>>>>>>> retrieveFiles >>>>>>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>>>>>>>> : null >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can you do me a favor and simply run this command: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cat >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What do you get? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is >>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012 >>>>>>>>>>>>>>>>>> Jul 24, 2012 4:06:00 PM >>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager >>>>>>>>>>>>>>>>>> setAsRunning >>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> First of all, I would like test the pushpull framework to >>>>>>>>>>>>>>>>>> move >>>>>>>>>>>>>>>>>> data >>>>>>>>>>>>>>>>>> written in NetCDF from a remote site by a network protocols >>>>>>>>>>>>>>>>>> (like ftp, >>>>>>>>>>>>>>>>>> http, etc). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yep, sounds good. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I think that there is a test suite which can be used to check >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> pushpull framework. >>>>>>>>>>>>>>>>>> Can you give configuration files(a set of XML files) that I >>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The examples in the resources directory that you found should >>>>>>>>>>>>>>>>> work >>>>>>>>>>>>>>>>> so we'll just need to figure out why they aren't working on >>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>> system. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>> Please let me know a reference implementation of pushpull >>>>>>>>>>>>>>>>>> framework >>>>>>>>>>>>>>>>>> that I can use to move NetCDF files. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks we'll figure this out. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>> Senior Computer Scientist >>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>> Phone: +1 (818) 354-8810 >>>>>> >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>> Chris Mattmann, Ph.D. >>>>>>> Senior Computer Scientist >>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
