Hey YunHee,
Doesn't look like the patch took this time, but don't worry about that now, i
have put the fixes into the trunk... if you just check out the latest trunk and
mvn install it (make sure you check out more than just cas-pushpull, because if
protocol-ftp doesn't get reinstalled your cas-pushpull mvn install will pick up
the last mvn installed version of the protocol-ftp 0.5 in your local maven
repository... another thing to note, you probably should utilize the
environment variables in place in the pushpull config files because if you ever
decide to move your install you are going to have to change all the paths in
your configuration files instead of just being able to change environment
variables (if you have any question about that let me know)... again if you run
into any road blocks just hit up the list and sorry about the bumpy road of
getting pushpull up and running... it should be good to go now though (it
downloads the files on my machine at least... lol).
-brian
On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote:
> Hi Brain,
>
> Unfortunately your patch didn't fix the problem. I got the same
> error message before:
> SEVERE: Failed to finish downloading per property files
> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
> : null
> java.lang.NullPointerException
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
> at
> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
> at
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
> at java.lang.Thread.run(Thread.java:662)
> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>
> First of all, I replaced CommonsNetFtpProtocol.java with new one
> that you modified as follows:
> ftp.retrieveFile(fromFile.getName(), os);
> /*
> if (ftp.retrieveFile(fromFile.getName(), os))// {
> throw new ProtocolException("Failed to download file "
> + fromFile.getName());
> // }
>
> */
> Next I rebuild a new jar file cas-protocol-ftp-0.4.jar and then
> copied it to lib directory:
> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar
> ls -al cas-protocol-ftp-0.4.jar
>
> I am wondering that there is difference between the following codes(1 and 2)
> :
> 1) ftp.retrieveFile(fromFile.getName(), os);
> 2) if (ftp.retrieveFile(fromFile.getName(), os))// {
> throw new ProtocolException("Failed to download file "
> + fromFile.getName());
> // }
>
> Will you let me know if the method get() in the
> CommonsNetFtpProtocol.java was called when pushpull was run ?
>
> Thanks,
> Yunhee
>
>
> 2012/7/27 Brian Foster <holeno...@mac.com>:
>> Hey YunHee,
>>
>> Okay the latest patch attached to the issue now (the on with todays date)
>> works as far as pushpull is concerned, however FTPClient.retrieveFile being
>> used in CommonsNetFtpProtocol is returning false for some reason... so i
>> would recommend trying the following... first unpatch the first patch from
>> your cas-pushpull and repatch with the new patch i've uploaded... then check
>> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify
>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this:
>>
>>
>> if (ftp.retrieveFile(fromFile.getName(), os))// {
>>
>> throw new ProtocolException("Failed to download file "
>>
>> + fromFile.getName());
>>
>> // }
>>
>>
>> to this:
>>
>>
>> ftp.retrieveFile(fromFile.getName(), os);
>>
>>
>> then mvn compile jar:jar and replace ftp protocol jar with that one in your
>> pushpull lib directory and give pushpull another run and see what happens
>> (validate the files downloaded to make sure they are complete)... hope this
>> helps and let me know if you need any further help!
>>
>>
>> -brian
>>
>>
>> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.k...@gmail.com> wrote:
>>
>> Hi Brian,
>>
>> I attach a tar file including etc and policy directories.
>>
>> Thanks,
>> Yunhee
>>
>> 2012/7/27 Brian Foster <holeno...@mac.com>:
>>> Hey YunHee,
>>>
>>> send me a tarball with your etc and policy directories i will spend some
>>> time today figuring out what is going on here
>>>
>>> -brian
>>>
>>>
>>> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.k...@gmail.com> wrote:
>>>
>>> Hi Brian,
>>>
>>> I just applied the patch you made according to the following steps:
>>> cd apache-oodt-0.4-src/pushpull
>>> patch -p0 < OODT-476.2012-07-25.txt
>>> mvn clean
>>> mvn install
>>> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/
>>>
>>> After I got a new cas-pushpull-0.4.jar,
>>> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar
>>> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar
>>> I ran the script pushpull.
>>>
>>>
>>> But the same error message was generated:
>>>
>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>> username 'anonymous'
>>> Jul 26, 2012 5:12:56 PM
>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>> retrieveFiles
>>> SEVERE: Failed to finish downloading per property files
>>>
>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>> : null
>>> java.lang.NullPointerException
>>> at
>>>
>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368)
>>> at
>>>
>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312)
>>> at
>>>
>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301)
>>> at
>>>
>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104)
>>> at
>>>
>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> Could you check a class name for handling ftp protocol ?
>>> I used "CommonsNetFtpProtocolFactory.class" in the jar named
>>> "cas-protocol-ftp-0.4.jar".
>>> The class file was described in the file ProtocolFactoryInfo.xml as
>>> follows:
>>> <protocol type="ftp">
>>> ....
>>> <!-- cas-protocol-ftp-0.4.jar -->
>>> <!--protocolFactory
>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/-->
>>> <protocolFactory
>>> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
>>> </protocol>
>>>
>>> Thanks,
>>> Yunhee
>>>
>>>
>>>
>>>
>>> 2012/7/26 Brian Foster <holeno...@mac.com>:
>>>> hey Yunhee,
>>>>
>>>> i just realized that your probably not on the dev list and my last email
>>>> just went to the list and didn't have you cc'ed
>>>>
>>>> try the patch attached to this issue mentioned below...
>>>>
>>>> -brian
>>>>
>>>> Begin forwarded message:
>>>>
>>>> From: Brian Foster <holeno...@mac.com>
>>>> Date: July 25, 2012 12:43:10 AM
>>>> To: dev@oodt.apache.org,dev@oodt.apache.org
>>>> Subject: Re: Problem when running pushpull
>>>>
>>>> hey Yunhee,
>>>>
>>>> Try patching your pushpull code with the fix i've attached to issue:
>>>>
>>>> https://issues.apache.org/jira/browse/OODT-476
>>>>
>>>> -brian
>>>>
>>>> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote:
>>>>
>>>> Hi Brian,
>>>>
>>>> I changed the file RetrievalSetup.java to get the stacktrace.
>>>> So I got a new cas-pushpull-0.4.jar after rebuilt it.
>>>>
>>>> The following is a message including the stacktrace with information
>>>> about excpetion propation.
>>>>
>>>> SEVERE: Failed to finish downloading per property files
>>>>
>>>>
>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>>> : null
>>>> java.lang.NullPointerException
>>>> at
>>>>
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367)
>>>> at
>>>>
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311)
>>>> at
>>>>
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300)
>>>> at
>>>>
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103)
>>>> at
>>>>
>>>>
>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
>>>> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> Please let me know what java programs I check frist.
>>>>
>>>> Thanks,
>>>>
>>>> Yunhee
>>>> 2012/7/25 Brian Foster <holeno...@mac.com>:
>>>>> YunHee,
>>>>>
>>>>> There is a NullPointerException being thrown... on line 118 of
>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change
>>>>> it
>>>>>
>>>>> rebuild and deploy and that should give you the stacktrace which will
>>>>> answer
>>>>> what is going on here.
>>>>>
>>>>> -brian
>>>>>
>>>>>
>>>>> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)"
>>>>
>>>>> <chris.a.mattm...@jpl.nasa.gov> wrote:
>>>>>
>>>>> Hi YunHee,
>>>>>
>>>>> Thanks for your mail.
>>>>>
>>>>> Let me take a look at the configs again. Does anyone else using
>>>>> PushPull have any advice for YunHee here? That error message
>>>>> seems to indicate there is a config file error somewhere. I'm wondering
>>>>> if your etc/push_pull_framework.properties has all the right values
>>>>> in there. Did you go through Cameron's guide to getting Push Pull
>>>>> running on the wiki?
>>>>>
>>>>> Cheers,
>>>>> Chris
>>>>
>>>>>
>>>>> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:
>>>>>
>>>>>> Hi Chris,
>>>>>>
>>>>>> There are some problems in the configuration files.
>>>>>> I found there is a dead link which does not connect two information
>>>>>> in two files :RemoteSpecs.xml and ExternalSources.xml.
>>>>>> For example an alias named "DewFtp" in RemoteSpecs.xml has no
>>>>>> associated source in "ExternalSources.xml.
>>>>>>
>>>>>> After I tailored those two XML files, I just tested one of
>>>>>> configurations, whose alias is PolarNcepNoaaFtp.
>>>>>> I also faced the same problem as follows:
>>>>>>
>>>>>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
>>>>>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>> username 'anonymous'
>>>>>> Jul 25, 2012 2:21:04 AM
>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>> retrieveFiles
>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>>>>>> : null
>>>>>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>>>>>> sleep
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Yunhee
>>>>>>
>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>> Hi YunHee,
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Chris
>>>>
>>>>>>>
>>>>>>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>>>>>>>
>>>>>>>> Hi Chris,
>>>>>>>>
>>>>>>>>> Thanks, can you try some of the other XML file configs.
>>>>>>>> Yes, I will try it and let you know its result.
>>>>>>>>
>>>>>>>>> Maybe it's being blocked in your country,
>>>>>>>> That site may be blocked.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Yunhee
>>>>>>>>
>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>> Hi YunHee,
>>>>>>>>>
>>>>
>>>>>>>>> Thanks, can you try some of the other XML file configs. Maybe it's
>>>>>>>>> being
>>>>>>>>> blocked in your country, or something, but let's see if we can get
>>>>>>>>> some
>>>>>>>>> of the other XML RemoteSpec configs to work?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>>>>>>>>>
>>>>>>>>>> Hi Chris,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Try out the MODIS one there, does that work?
>>>>>>>>>> No, it doesn't.
>>>>>>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection
>>>>>>>>>> problem.
>>>>>>>>>>
>>>>>>>>>> $ ftp e4ftl01u.ecs.nasa.gov
>>>>>>>>>> ftp: connect: Connection timed out
>>>>>>>>>> ftp>
>>>>>>>>>>
>>>>>>>>>> The message below also say the reason why the problem was occurred.
>>>>>>>>>> WARNING: Error occurred while connecting to RemoteSite: alias =
>>>>>>>>>> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username =
>>>>>>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>>>>>>>>>> connect to server : Connection timed out
>>>>>>>>>> Jul 25, 2012 1:31:25 AM
>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url =
>>>>>>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir =
>>>>>>>>>> 'null' maxConnections = '-1' in 5 seconds
>>>>>>>>>> Waiting . . . . .
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Yunhee
>>>>>>>>>>
>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>
>>>>
>>>>>>>>>>> OK, can you try some of the other files, e.g., the ones in:
>>>>>>>>>>>
>>>>>>>>>>> http://s.apache.org/ks0
>>>>>>>>>>>
>>>>>>>>>>> Try out the MODIS one there, does that work?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>>
>>>>>>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>
>>>>>>>>>>>> I added the header of XML as follows:
>>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>>>
>>>>>>>>>>>> Next the value of elements was changed into specific one.
>>>>>>>>>>>> The following is the output of the XML:
>>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>>> <root>
>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>>>> <nofiles/>
>>>>>>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>>>>>>>>>>>> <dir name="2004\.09\.20">
>>>>>>>>>>>> <nodirs/>
>>>>>>>>>>>> <!--file
>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>>>>>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>>>>>>>>>>>> </dir>
>>>>>>>>>>>> </dirstruct>
>>>>>>>>>>>> </root>
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately I got the same result before:
>>>>>>>>>>>> ...
>>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this
>>>>>>>>>>>> ma
>>>>>>>>>>>> y take a few minutes . . .
>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>>>>>>> protocol 'org.apache.oodt.
>>>>>>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>>>>>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>>> retrie
>>>>>>>>>>>> veFiles
>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/
>>>>>>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>>>>>>>>>>>> Jul 25, 2012 12:30:04 AM
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon
>>>>>>>>>>>> sleep
>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going
>>>>>>>>>>>> to
>>>>>>>>>>>> sleep until Wed Jul 25 0
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Yunhee
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2012/7/25 Mattmann, Chris A (388J)
>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>>>
>>>>
>>>>>>>>>>>>> Can you try simply putting a specific <dir name="..."> instead
>>>>>>>>>>>>> of
>>>>>>>>>>>>> \d{4}, etc., and
>>>>>>>>>>>>> then can you put a specific file name in the <file name="..">.
>>>>>>>>>>>>> If
>>>>>>>>>>>>> that works and
>>>>>>>>>>>>> downloads, then we know there is something up with the regex
>>>>>>>>>>>>> somehow.
>>>>>>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your
>>>>>>>>>>>>> file?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The following is the content of TESL2CO2.xml.
>>>>>>>>>>>>>> $ cat
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>> <root>
>>>>>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>>>>>> <nofiles/>
>>>>>>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}">
>>>>>>>>>>>>>> <nodirs/>
>>>>>>>>>>>>>> <file
>>>>>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>>>>>>>>>>>>>> </dir>
>>>>>>>>>>>>>> </dirstruct>
>>>>>>>>>>>>>> </root>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know if there is a problem in the configuration file.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Yunhee
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J)
>>>>>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>>>>>>> Hi YunHee,
>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am stuck on the problem related with "pushpull" script
>>>>>>>>>>>>>>>> occurred
>>>>>>>>>>>>>>>> the following error message:
>>>>>>>>>>>>>>>> ....
>>>>>>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . .
>>>>>>>>>>>>>>>> this may
>>>>>>>>>>>>>>>> take a few minutes . . .
>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>>>>>>>>>>>> username 'anonymous'
>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>>>>>>> retrieveFiles
>>>>>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>>> : null
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you do me a favor and simply run this command:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cat
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What do you get?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is
>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012
>>>>>>>>>>>>>>>> Jul 24, 2012 4:06:00 PM
>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager
>>>>>>>>>>>>>>>> setAsRunning
>>>>>>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> First of all, I would like test the pushpull framework to
>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>> written in NetCDF from a remote site by a network protocols
>>>>>>>>>>>>>>>> (like ftp,
>>>>>>>>>>>>>>>> http, etc).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yep, sounds good.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think that there is a test suite which can be used to check
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> pushpull framework.
>>>>>>>>>>>>>>>> Can you give configuration files(a set of XML files) that I
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> test
>>>>>>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The examples in the resources directory that you found should
>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>> so we'll just need to figure out why they aren't working on
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> system.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> Please let me know a reference implementation of pushpull
>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>> that I can use to move NetCDF files.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks we'll figure this out.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>> Senior Computer Scientist
>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Chris Mattmann, Ph.D.
>>>>>>> Senior Computer Scientist
>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> Phone: +1 (818) 354-8810
>>>>
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>
>>>>>
>>>>>
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Chris Mattmann, Ph.D.
>>>>> Senior Computer Scientist
>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> Office: 171-266B, Mailstop: 171-246
>>>>> Email: chris.a.mattm...@nasa.gov
>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>
