hey YunHee, I keep forgetting to add you to the email cc... are you on the dev list?... see email below, explains your commented out filemgr url making the pushpull work
-brian Begin forwarded message: > From: Brian Foster <holeno...@me.com> > Date: July 29, 2012 10:12:48 AM PDT > To: "dev@oodt.apache.org" <dev@oodt.apache.org> > Cc: Brian Foster <holeno...@mac.com>, "dev@oodt.apache.org" > <dev@oodt.apache.org>, "Mattmann, Chris A (388J)" > <chris.a.mattm...@jpl.nasa.gov> > Subject: Re: Problem when running pushpull > Reply-To: dev@oodt.apache.org > > Hey YunHee, > > You need to configure you mime-types.xml file in your policy directory to map > your file names to filemgr ProductTypes... if you want to just download the > files for now just comment out the filemgr > URL property in your pushpull properties file... if you want to configure the > mime-types.xml file, take a look at: > > https://svn.apache.org/repos/asf/oodt/trunk/pushpull/src/main/resources/policy/mimetypes.xml > > Here is basically how the file works: > > <mime-type type="product/tes"> > <_comment>ProductType=MyTesProductType</_comment> > <glob pattern="any valid Pattern.java regex" is regex="true"/> > </mime-type> > > the mime-type "type" must have a / and is just a name being used to title the > mapping... fill in an actual filemgr ProductType in place of > MyTesProdectType... and the pattern should be the regex describing the > filenames of the files that belong to the give ProductType. > > then make sure that org.apache.oodt.cas.pushpull.config.type.dectection.file > property points at your mime-types.xml file. > > -Brian > On Jul 29, 2012, at 6:45 AM, YunHee Kang <yunh.k...@gmail.com> wrote: > >> Hi Brian, >> >> I checked out the latest trunk of oodt 0.5 and installed it. >> After I set up the file manger and pushpull framework, I ran the >> pushpull script. >> As I described below, I think that the trunk is useful to fix the >> problem, which was caused by handling ftp protocol. >> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >> username 'anonymous' >> Jul 29, 2012 10:09:30 PM >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem >> initializeStagingArea >> INFO: Preparing staging area >> /home/yhkang/oodt-0.5/cas-pushpull/staging/TESL2CO2 >> Jul 29, 2012 10:09:30 PM org.apache.commons.httpclient.HttpClient <clinit> >> FINE: Java version: 1.6.0_31 >> >> But I got the difference problem that is related with filemgr. >> The following is part of error message in the file cas-pushpull0.log : >> Jul 29, 2012 10:09:30 PM >> org.apache.oodt.cas.filemgr.ingest.StdIngester hasProduct >> WARNING: Unable to check for existance of product: [null]: Message: >> Failure writing request >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >> Failure writing request >> at >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606) >> at >> org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284) >> at >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254) >> at >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463) >> at >> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138) >> at >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >> at java.lang.Thread.run(Thread.java:662) >> Jul 29, 2012 10:09:30 PM >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> retrieveFiles >> SEVERE: Failed to finish downloading per property files >> /home/yhkang/oodt-0.5/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> : [RemoteCrawler] Failed to communicate with database : Failure >> writing request >> org.apache.oodt.cas.pushpull.exceptions.RetrievalMethodException: >> [RemoteCrawler] Failed to communicate with database : Failure writing >> request >> at >> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:150) >> at >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >> at java.lang.Thread.run(Thread.java:662) >> Jul 29, 2012 10:09:30 PM >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect >> INFO: Disconnecting protocol >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol >> Jul 29, 2012 10:09:30 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >> INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to >> sleep until Sun Jul 29 22:12:00 KST 2012 >> >> Please let me know how to fix this problem. >> >> Thanks, >> Yunhee >> >> >> >> 2012/7/28 Brian Foster <holeno...@me.com>: >>> Hey YunHee, >>> >>> Doesn't look like the patch took this time, but don't worry about that now, >>> i have put the fixes into the trunk... if you just check out the latest >>> trunk and mvn install it (make sure you check out more than just >>> cas-pushpull, because if protocol-ftp doesn't get reinstalled your >>> cas-pushpull mvn install will pick up the last mvn installed version of the >>> protocol-ftp 0.5 in your local maven repository... another thing to note, >>> you probably should utilize the environment variables in place in the >>> pushpull config files because if you ever decide to move your install you >>> are going to have to change all the paths in your configuration files >>> instead of just being able to change environment variables (if you have any >>> question about that let me know)... again if you run into any road blocks >>> just hit up the list and sorry about the bumpy road of getting pushpull up >>> and running... it should be good to go now though (it downloads the files >>> on my machine at least... lol). >>> >>> -brian >>> >>> On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote: >>> >>>> Hi Brain, >>>> >>>> Unfortunately your patch didn't fix the problem. I got the same >>>> error message before: >>>> SEVERE: Failed to finish downloading per property files >>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>> : null >>>> java.lang.NullPointerException >>>> at >>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368) >>>> at >>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312) >>>> at >>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301) >>>> at >>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104) >>>> at >>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >>>> at java.lang.Thread.run(Thread.java:662) >>>> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>>> >>>> First of all, I replaced CommonsNetFtpProtocol.java with new one >>>> that you modified as follows: >>>> ftp.retrieveFile(fromFile.getName(), os); >>>> /* >>>> if (ftp.retrieveFile(fromFile.getName(), os))// { >>>> throw new ProtocolException("Failed to download file " >>>> + fromFile.getName()); >>>> // } >>>> >>>> */ >>>> Next I rebuild a new jar file cas-protocol-ftp-0.4.jar and then >>>> copied it to lib directory: >>>> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar >>>> ls -al cas-protocol-ftp-0.4.jar >>>> >>>> I am wondering that there is difference between the following codes(1 and >>>> 2) : >>>> 1) ftp.retrieveFile(fromFile.getName(), os); >>>> 2) if (ftp.retrieveFile(fromFile.getName(), os))// { >>>> throw new ProtocolException("Failed to download file " >>>> + fromFile.getName()); >>>> // } >>>> >>>> Will you let me know if the method get() in the >>>> CommonsNetFtpProtocol.java was called when pushpull was run ? >>>> >>>> Thanks, >>>> Yunhee >>>> >>>> >>>> 2012/7/27 Brian Foster <holeno...@mac.com>: >>>>> Hey YunHee, >>>>> >>>>> Okay the latest patch attached to the issue now (the on with todays date) >>>>> works as far as pushpull is concerned, however FTPClient.retrieveFile >>>>> being >>>>> used in CommonsNetFtpProtocol is returning false for some reason... so i >>>>> would recommend trying the following... first unpatch the first patch from >>>>> your cas-pushpull and repatch with the new patch i've uploaded... then >>>>> check >>>>> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify >>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this: >>>>> >>>>> >>>>> if (ftp.retrieveFile(fromFile.getName(), os))// { >>>>> >>>>> throw new ProtocolException("Failed to download file " >>>>> >>>>> + fromFile.getName()); >>>>> >>>>> // } >>>>> >>>>> >>>>> to this: >>>>> >>>>> >>>>> ftp.retrieveFile(fromFile.getName(), os); >>>>> >>>>> >>>>> then mvn compile jar:jar and replace ftp protocol jar with that one in >>>>> your >>>>> pushpull lib directory and give pushpull another run and see what happens >>>>> (validate the files downloaded to make sure they are complete)... hope >>>>> this >>>>> helps and let me know if you need any further help! >>>>> >>>>> >>>>> -brian >>>>> >>>>> >>>>> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.k...@gmail.com> wrote: >>>>> >>>>> Hi Brian, >>>>> >>>>> I attach a tar file including etc and policy directories. >>>>> >>>>> Thanks, >>>>> Yunhee >>>>> >>>>> 2012/7/27 Brian Foster <holeno...@mac.com>: >>>>>> Hey YunHee, >>>>>> >>>>>> send me a tarball with your etc and policy directories i will spend some >>>>>> time today figuring out what is going on here >>>>>> >>>>>> -brian >>>>>> >>>>>> >>>>>> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.k...@gmail.com> wrote: >>>>>> >>>>>> Hi Brian, >>>>>> >>>>>> I just applied the patch you made according to the following steps: >>>>>> cd apache-oodt-0.4-src/pushpull >>>>>> patch -p0 < OODT-476.2012-07-25.txt >>>>>> mvn clean >>>>>> mvn install >>>>>> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/ >>>>>> >>>>>> After I got a new cas-pushpull-0.4.jar, >>>>>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar >>>>>> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar >>>>>> I ran the script pushpull. >>>>>> >>>>>> >>>>>> But the same error message was generated: >>>>>> >>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>>> username 'anonymous' >>>>>> Jul 26, 2012 5:12:56 PM >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>> retrieveFiles >>>>>> SEVERE: Failed to finish downloading per property files >>>>>> >>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>> : null >>>>>> java.lang.NullPointerException >>>>>> at >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368) >>>>>> at >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312) >>>>>> at >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301) >>>>>> at >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104) >>>>>> at >>>>>> >>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >>>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> Could you check a class name for handling ftp protocol ? >>>>>> I used "CommonsNetFtpProtocolFactory.class" in the jar named >>>>>> "cas-protocol-ftp-0.4.jar". >>>>>> The class file was described in the file ProtocolFactoryInfo.xml as >>>>>> follows: >>>>>> <protocol type="ftp"> >>>>>> .... >>>>>> <!-- cas-protocol-ftp-0.4.jar --> >>>>>> <!--protocolFactory >>>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/--> >>>>>> <protocolFactory >>>>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/> >>>>>> </protocol> >>>>>> >>>>>> Thanks, >>>>>> Yunhee >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> 2012/7/26 Brian Foster <holeno...@mac.com>: >>>>>>> hey Yunhee, >>>>>>> >>>>>>> i just realized that your probably not on the dev list and my last email >>>>>>> just went to the list and didn't have you cc'ed >>>>>>> >>>>>>> try the patch attached to this issue mentioned below... >>>>>>> >>>>>>> -brian >>>>>>> >>>>>>> Begin forwarded message: >>>>>>> >>>>>>> From: Brian Foster <holeno...@mac.com> >>>>>>> Date: July 25, 2012 12:43:10 AM >>>>>>> To: dev@oodt.apache.org,dev@oodt.apache.org >>>>>>> Subject: Re: Problem when running pushpull >>>>>>> >>>>>>> hey Yunhee, >>>>>>> >>>>>>> Try patching your pushpull code with the fix i've attached to issue: >>>>>>> >>>>>>> https://issues.apache.org/jira/browse/OODT-476 >>>>>>> >>>>>>> -brian >>>>>>> >>>>>>> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote: >>>>>>> >>>>>>> Hi Brian, >>>>>>> >>>>>>> I changed the file RetrievalSetup.java to get the stacktrace. >>>>>>> So I got a new cas-pushpull-0.4.jar after rebuilt it. >>>>>>> >>>>>>> The following is a message including the stacktrace with information >>>>>>> about excpetion propation. >>>>>>> >>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>> >>>>>>> >>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >>>>>>> : null >>>>>>> java.lang.NullPointerException >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103) >>>>>>> at >>>>>>> >>>>>>> >>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >>>>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >>>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>>> >>>>>>> Please let me know what java programs I check frist. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Yunhee >>>>>>> 2012/7/25 Brian Foster <holeno...@mac.com>: >>>>>>>> YunHee, >>>>>>>> >>>>>>>> There is a NullPointerException being thrown... on line 118 of >>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change >>>>>>>> it >>>>>>>> >>>>>>>> rebuild and deploy and that should give you the stacktrace which will >>>>>>>> answer >>>>>>>> what is going on here. >>>>>>>> >>>>>>>> -brian >>>>>>>> >>>>>>>> >>>>>>>> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)" >>>>>>> >>>>>>>> <chris.a.mattm...@jpl.nasa.gov> wrote: >>>>>>>> >>>>>>>> Hi YunHee, >>>>>>>> >>>>>>>> Thanks for your mail. >>>>>>>> >>>>>>>> Let me take a look at the configs again. Does anyone else using >>>>>>>> PushPull have any advice for YunHee here? That error message >>>>>>>> seems to indicate there is a config file error somewhere. I'm wondering >>>>>>>> if your etc/push_pull_framework.properties has all the right values >>>>>>>> in there. Did you go through Cameron's guide to getting Push Pull >>>>>>>> running on the wiki? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Chris >>>>>>> >>>>>>>> >>>>>>>> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote: >>>>>>>> >>>>>>>>> Hi Chris, >>>>>>>>> >>>>>>>>> There are some problems in the configuration files. >>>>>>>>> I found there is a dead link which does not connect two information >>>>>>>>> in two files :RemoteSpecs.xml and ExternalSources.xml. >>>>>>>>> For example an alias named "DewFtp" in RemoteSpecs.xml has no >>>>>>>>> associated source in "ExternalSources.xml. >>>>>>>>> >>>>>>>>> After I tailored those two XML files, I just tested one of >>>>>>>>> configurations, whose alias is PolarNcepNoaaFtp. >>>>>>>>> I also faced the same problem as follows: >>>>>>>>> >>>>>>>>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with >>>>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>>>>>> username 'anonymous' >>>>>>>>> Jul 25, 2012 2:21:04 AM >>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>> retrieveFiles >>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >>>>>>>>> : null >>>>>>>>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>>> sleep >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Yunhee >>>>>>>>> >>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>> Hi YunHee, >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Chris >>>>>>> >>>>>>>>>> >>>>>>>>>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote: >>>>>>>>>> >>>>>>>>>>> Hi Chris, >>>>>>>>>>> >>>>>>>>>>>> Thanks, can you try some of the other XML file configs. >>>>>>>>>>> Yes, I will try it and let you know its result. >>>>>>>>>>> >>>>>>>>>>>> Maybe it's being blocked in your country, >>>>>>>>>>> That site may be blocked. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Yunhee >>>>>>>>>>> >>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>> Hi YunHee, >>>>>>>>>>>> >>>>>>> >>>>>>>>>>>> Thanks, can you try some of the other XML file configs. Maybe it's >>>>>>>>>>>> being >>>>>>>>>>>> blocked in your country, or something, but let's see if we can get >>>>>>>>>>>> some >>>>>>>>>>>> of the other XML RemoteSpec configs to work? >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Chris >>>>>>>>>>>> >>>>>>>>>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Try out the MODIS one there, does that work? >>>>>>>>>>>>> No, it doesn't. >>>>>>>>>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection >>>>>>>>>>>>> problem. >>>>>>>>>>>>> >>>>>>>>>>>>> $ ftp e4ftl01u.ecs.nasa.gov >>>>>>>>>>>>> ftp: connect: Connection timed out >>>>>>>>>>>>> ftp> >>>>>>>>>>>>> >>>>>>>>>>>>> The message below also say the reason why the problem was >>>>>>>>>>>>> occurred. >>>>>>>>>>>>> WARNING: Error occurred while connecting to RemoteSite: alias = >>>>>>>>>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username = >>>>>>>>>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to >>>>>>>>>>>>> connect to server : Connection timed out >>>>>>>>>>>>> Jul 25, 2012 1:31:25 AM >>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url >>>>>>>>>>>>> = >>>>>>>>>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir = >>>>>>>>>>>>> 'null' maxConnections = '-1' in 5 seconds >>>>>>>>>>>>> Waiting . . . . . >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Yunhee >>>>>>>>>>>>> >>>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) >>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>>>> Hi YunHee, >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> OK, can you try some of the other files, e.g., the ones in: >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://s.apache.org/ks0 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Try out the MODIS one there, does that work? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Chris >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I added the header of XML as follows: >>>>>>>>>>>>>>> <?xml version="1.0"?> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Next the value of elements was changed into specific one. >>>>>>>>>>>>>>> The following is the output of the XML: >>>>>>>>>>>>>>> <?xml version="1.0"?> >>>>>>>>>>>>>>> <root> >>>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>>>>>>>> <nofiles/> >>>>>>>>>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"--> >>>>>>>>>>>>>>> <dir name="2004\.09\.20"> >>>>>>>>>>>>>>> <nodirs/> >>>>>>>>>>>>>>> <!--file >>>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/--> >>>>>>>>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/> >>>>>>>>>>>>>>> </dir> >>>>>>>>>>>>>>> </dirstruct> >>>>>>>>>>>>>>> </root> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Unfortunately I got the same result before: >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> INFO: Testing protocol >>>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> ma >>>>>>>>>>>>>>> y take a few minutes . . . >>>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>>>>>>>>>> protocol 'org.apache.oodt. >>>>>>>>>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' >>>>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM >>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>>>>>>>> retrie >>>>>>>>>>>>>>> veFiles >>>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/ >>>>>>>>>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null >>>>>>>>>>>>>>> Jul 25, 2012 12:30:04 AM >>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon >>>>>>>>>>>>>>> sleep >>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> sleep until Wed Jul 25 0 >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Yunhee >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) >>>>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>>>>>> Hi YunHee, >>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>> Can you try simply putting a specific <dir name="..."> instead >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> \d{4}, etc., and >>>>>>>>>>>>>>>> then can you put a specific file name in the <file name="..">. >>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>> that works and >>>>>>>>>>>>>>>> downloads, then we know there is something up with the regex >>>>>>>>>>>>>>>> somehow. >>>>>>>>>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your >>>>>>>>>>>>>>>> file? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The following is the content of TESL2CO2.xml. >>>>>>>>>>>>>>>>> $ cat >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>>>>>>> <root> >>>>>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>>>>>>>>>>>>>> <nofiles/> >>>>>>>>>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}"> >>>>>>>>>>>>>>>>> <nodirs/> >>>>>>>>>>>>>>>>> <file >>>>>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/> >>>>>>>>>>>>>>>>> </dir> >>>>>>>>>>>>>>>>> </dirstruct> >>>>>>>>>>>>>>>>> </root> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Let me know if there is a problem in the configuration file. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Yunhee >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J) >>>>>>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>: >>>>>>>>>>>>>>>>>> Hi YunHee, >>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Chris, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I am stuck on the problem related with "pushpull" script >>>>>>>>>>>>>>>>>>> occurred >>>>>>>>>>>>>>>>>>> the following error message: >>>>>>>>>>>>>>>>>>> .... >>>>>>>>>>>>>>>>>>> INFO: Testing protocol >>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . >>>>>>>>>>>>>>>>>>> this may >>>>>>>>>>>>>>>>>>> take a few minutes . . . >>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler >>>>>>>>>>>>>>>>>>> connect >>>>>>>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov >>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>>>>>>>>>>>>>>>> username 'anonymous' >>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>>>>>>>>>>>>>> retrieveFiles >>>>>>>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>>>>>>>>> : null >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Can you do me a favor and simply run this command: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> cat >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What do you get? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is >>>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012 >>>>>>>>>>>>>>>>>>> Jul 24, 2012 4:06:00 PM >>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager >>>>>>>>>>>>>>>>>>> setAsRunning >>>>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> First of all, I would like test the pushpull framework to >>>>>>>>>>>>>>>>>>> move >>>>>>>>>>>>>>>>>>> data >>>>>>>>>>>>>>>>>>> written in NetCDF from a remote site by a network protocols >>>>>>>>>>>>>>>>>>> (like ftp, >>>>>>>>>>>>>>>>>>> http, etc). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Yep, sounds good. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I think that there is a test suite which can be used to >>>>>>>>>>>>>>>>>>> check >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> pushpull framework. >>>>>>>>>>>>>>>>>>> Can you give configuration files(a set of XML files) that I >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The examples in the resources directory that you found should >>>>>>>>>>>>>>>>>> work >>>>>>>>>>>>>>>>>> so we'll just need to figure out why they aren't working on >>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>> system. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>> Please let me know a reference implementation of pushpull >>>>>>>>>>>>>>>>>>> framework >>>>>>>>>>>>>>>>>>> that I can use to move NetCDF files. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks we'll figure this out. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>>>> Senior Computer Scientist >>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> Chris Mattmann, Ph.D. >>>>>>>>>> Senior Computer Scientist >>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>>>> Phone: +1 (818) 354-8810 >>>>>>> >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> Chris Mattmann, Ph.D. >>>>>>>> Senior Computer Scientist >>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>>>> Email: chris.a.mattm...@nasa.gov >>>>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>> >>>
