Hi YunHee, Thanks, can you try some of the other XML file configs. Maybe it's being blocked in your country, or something, but let's see if we can get some of the other XML RemoteSpec configs to work?
Cheers, Chris On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote: > Hi Chris, > > >> Try out the MODIS one there, does that work? > No, it doesn't. > I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection problem. > > $ ftp e4ftl01u.ecs.nasa.gov > ftp: connect: Connection timed out > ftp> > > The message below also say the reason why the problem was occurred. > WARNING: Error occurred while connecting to RemoteSite: alias = > 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username = > 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to > connect to server : Connection timed out > Jul 25, 2012 1:31:25 AM > org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect > INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url = > 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir = > 'null' maxConnections = '-1' in 5 seconds > Waiting . . . . . > > Thanks, > Yunhee > > 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >> Hi YunHee, >> >> OK, can you try some of the other files, e.g., the ones in: >> >> http://s.apache.org/ks0 >> >> Try out the MODIS one there, does that work? >> >> Cheers, >> Chris >> >> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote: >> >>> Hi Chris, >>> >>> I added the header of XML as follows: >>> <?xml version="1.0"?> >>> >>> Next the value of elements was changed into specific one. >>> The following is the output of the XML: >>> <?xml version="1.0"?> >>> <root> >>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>> <nofiles/> >>> <!--dir name="\d{4}\.\d{2}\.\d{2}"--> >>> <dir name="2004\.09\.20"> >>> <nodirs/> >>> <!--file >>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/--> >>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/> >>> </dir> >>> </dirstruct> >>> </root> >>> >>> Unfortunately I got the same result before: >>> ... >>> INFO: Testing protocol >>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this ma >>> y take a few minutes . . . >>> Jul 25, 2012 12:30:03 AM >>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>> protocol 'org.apache.oodt. >>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' >>> Jul 25, 2012 12:30:03 AM >>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrie >>> veFiles >>> SEVERE: Failed to finish downloading per property files >>> /home/yhkang/oodt/cas-pushpull/etc/ >>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null >>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>> sleep until Wed Jul 25 0 >>> ... >>> >>> >>> Thanks, >>> Yunhee >>> >>> >>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>> Hi YunHee, >>>> >>>> Can you try simply putting a specific <dir name="..."> instead of \d{4}, >>>> etc., and >>>> then can you put a specific file name in the <file name="..">. If that >>>> works and >>>> downloads, then we know there is something up with the regex somehow. >>>> Also can you try adding the <?xml version="1.0"?> start to your file? >>>> >>>> Cheers, >>>> Chris >>>> >>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote: >>>> >>>>> Hi Chris, >>>>> >>>>> >>>>> The following is the content of TESL2CO2.xml. >>>>> $ cat >>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>> <root> >>>>> <dirstruct starting_path="/TES/TL2CO2N.005"> >>>>> <nofiles/> >>>>> <dir name="\d{4}\.\d{2}\.\d{2}"> >>>>> <nodirs/> >>>>> <file >>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/> >>>>> </dir> >>>>> </dirstruct> >>>>> </root> >>>>> >>>>> Let me know if there is a problem in the configuration file. >>>>> >>>>> Thanks, >>>>> Yunhee >>>>> >>>>> 2012/7/24 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >>>>>> Hi YunHee, >>>>>> >>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote: >>>>>> >>>>>>> Hi Chris, >>>>>>> >>>>>>> I am stuck on the problem related with "pushpull" script occurred >>>>>>> the following error message: >>>>>>> .... >>>>>>> INFO: Testing protocol >>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this may >>>>>>> take a few minutes . . . >>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >>>>>>> username 'anonymous' >>>>>>> Jul 24, 2012 4:05:02 PM >>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >>>>>>> retrieveFiles >>>>>>> SEVERE: Failed to finish downloading per property files >>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>>> : null >>>>>> >>>>>> Can you do me a favor and simply run this command: >>>>>> >>>>>> cat >>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >>>>>> >>>>>> What do you get? >>>>>> >>>>>>> Jul 24, 2012 4:05:02 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to >>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012 >>>>>>> Jul 24, 2012 4:06:00 PM >>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning >>>>>>> INFO: Daemon with ID = 91131 was given permission to run >>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >>>>>>> >>>>>>> First of all, I would like test the pushpull framework to move data >>>>>>> written in NetCDF from a remote site by a network protocols (like ftp, >>>>>>> http, etc). >>>>>> >>>>>> Yep, sounds good. >>>>>> >>>>>>> I think that there is a test suite which can be used to check the >>>>>>> pushpull framework. >>>>>>> Can you give configuration files(a set of XML files) that I can test >>>>>>> downloading files from a FTP site (public anonymous site) ? >>>>>> >>>>>> The examples in the resources directory that you found should work >>>>>> so we'll just need to figure out why they aren't working on your system. >>>>>> >>>>>>> or >>>>>>> Please let me know a reference implementation of pushpull framework >>>>>>> that I can use to move NetCDF files. >>>>>> >>>>>> Thanks we'll figure this out. >>>>>> >>>>>> Cheers, >>>>>> Chris >>>>>> >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Chris Mattmann, Ph.D. >>>>>> Senior Computer Scientist >>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>>>> Office: 171-266B, Mailstop: 171-246 >>>>>> Email: chris.a.mattm...@nasa.gov >>>>>> WWW: http://sunset.usc.edu/~mattmann/ >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> Adjunct Assistant Professor, Computer Science Department >>>>>> University of Southern California, Los Angeles, CA 90089 USA >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>> >>>> >>>> >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Chris Mattmann, Ph.D. >>>> Senior Computer Scientist >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>>> Office: 171-266B, Mailstop: 171-246 >>>> Email: chris.a.mattm...@nasa.gov >>>> WWW: http://sunset.usc.edu/~mattmann/ >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> Adjunct Assistant Professor, Computer Science Department >>>> University of Southern California, Los Angeles, CA 90089 USA >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Senior Computer Scientist >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 171-266B, Mailstop: 171-246 >> Email: chris.a.mattm...@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Assistant Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
