hey Yunhee,
Try patching your pushpull code with the fix i've attached to issue:
https://issues.apache.org/jira/browse/OODT-476
-brian
Try patching your pushpull code with the fix i've attached to issue:
https://issues.apache.org/jira/browse/OODT-476
-brian
On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote:
Hi Brian,
I changed the file RetrievalSetup.java to get the stacktrace.
So I got a new cas-pushpull-0.4.jar after rebuilt it.
The following is a message including the stacktrace with information
about excpetion propation.
SEVERE: Failed to finish downloading per property files
/home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
: null
java.lang.NullPointerException
at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367)
at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311)
at org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300)
at org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103)
at org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109)
at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218)
at java.lang.Thread.run(Thread.java:662)
Please let me know what java programs I check frist.
Thanks,
Yunhee
2012/7/25 Brian Foster <holeno...@mac.com>:
> YunHee,
>
> There is a NullPointerException being thrown... on line 118 of
> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change it
>
> rebuild and deploy and that should give you the stacktrace which will answer
> what is going on here.
>
> -brian
>
>
> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)"
> <chris.a.mattm...@jpl.nasa.gov> wrote:
>
> Hi YunHee,
>
> Thanks for your mail.
>
> Let me take a look at the configs again. Does anyone else using
> PushPull have any advice for YunHee here? That error message
> seems to indicate there is a config file error somewhere. I'm wondering
> if your etc/push_pull_framework.properties has all the right values
> in there. Did you go through Cameron's guide to getting Push Pull
> running on the wiki?
>
> Cheers,
> Chris
>
> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote:
>
>> Hi Chris,
>>
>> There are some problems in the configuration files.
>> I found there is a dead link which does not connect two information
>> in two files :RemoteSpecs.xml and ExternalSources.xml.
>> For example an alias named "DewFtp" in RemoteSpecs.xml has no
>> associated source in "ExternalSources.xml.
>>
>> After I tailored those two XML files, I just tested one of
>> configurations, whose alias is PolarNcepNoaaFtp.
>> I also faced the same problem as follows:
>>
>> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with
>> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>> username 'anonymous'
>> Jul 25, 2012 2:21:04 AM
>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>> retrieveFiles
>> SEVERE: Failed to finish downloading per property files
>>
>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml
>> : null
>> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>
>>
>> Thanks,
>> Yunhee
>>
>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>> Hi YunHee,
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote:
>>>
>>>> Hi Chris,
>>>>
>>>>> Thanks, can you try some of the other XML file configs.
>>>> Yes, I will try it and let you know its result.
>>>>
>>>>> Maybe it's being blocked in your country,
>>>> That site may be blocked.
>>>>
>>>>
>>>> Thanks,
>>>> Yunhee
>>>>
>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>> Hi YunHee,
>>>>>
>>>>> Thanks, can you try some of the other XML file configs. Maybe it's
>>>>> being
>>>>> blocked in your country, or something, but let's see if we can get some
>>>>> of the other XML RemoteSpec configs to work?
>>>>>
>>>>> Cheers,
>>>>> Chris
>>>>>
>>>>> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote:
>>>>>
>>>>>> Hi Chris,
>>>>>>
>>>>>>
>>>>>>> Try out the MODIS one there, does that work?
>>>>>> No, it doesn't.
>>>>>> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection
>>>>>> problem.
>>>>>>
>>>>>> $ ftp e4ftl01u.ecs.nasa.gov
>>>>>> ftp: connect: Connection timed out
>>>>>> ftp>
>>>>>>
>>>>>> The message below also say the reason why the problem was occurred.
>>>>>> WARNING: Error occurred while connecting to RemoteSite: alias =
>>>>>> 'ModisFtp' url = '' username =
>>>>>> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to
>>>>>> connect to server : Connection timed out
>>>>>> Jul 25, 2012 1:31:25 AM
>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url =""> >>>>>> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir =
>>>>>> 'null' maxConnections = '-1' in 5 seconds
>>>>>> Waiting . . . . .
>>>>>>
>>>>>> Thanks,
>>>>>> Yunhee
>>>>>>
>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>> Hi YunHee,
>>>>>>>
>>>>>>> OK, can you try some of the other files, e.g., the ones in:
>>>>>>>
>>>>>>> http://s.apache.org/ks0
>>>>>>>
>>>>>>> Try out the MODIS one there, does that work?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>>
>>>>>>> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote:
>>>>>>>
>>>>>>>> Hi Chris,
>>>>>>>>
>>>>>>>> I added the header of XML as follows:
>>>>>>>> <?xml version="1.0"?>
>>>>>>>>
>>>>>>>> Next the value of elements was changed into specific one.
>>>>>>>> The following is the output of the XML:
>>>>>>>> <?xml version="1.0"?>
>>>>>>>> <root>
>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>> <nofiles/>
>>>>>>>> <!--dir name="\d{4}\.\d{2}\.\d{2}"-->
>>>>>>>> <dir name="2004\.09\.20">
>>>>>>>> <nodirs/>
>>>>>>>> <!--file
>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/-->
>>>>>>>> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/>
>>>>>>>> </dir>
>>>>>>>> </dirstruct>
>>>>>>>> </root>
>>>>>>>>
>>>>>>>> Unfortunately I got the same result before:
>>>>>>>> ...
>>>>>>>> INFO: Testing protocol
>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this ma
>>>>>>>> y take a few minutes . . .
>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>>> protocol 'org.apache.oodt.
>>>>>>>> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous'
>>>>>>>> Jul 25, 2012 12:30:03 AM
>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup retrie
>>>>>>>> veFiles
>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/
>>>>>>>> examples/DirStructXmlParserFiles/TESL2CO2.xml : null
>>>>>>>> Jul 25, 2012 12:30:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>>>>>>>> sleep
>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to
>>>>>>>> sleep until Wed Jul 25 0
>>>>>>>> ...
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Yunhee
>>>>>>>>
>>>>>>>>
>>>>>>>> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>> Hi YunHee,
>>>>>>>>>
>>>>>>>>> Can you try simply putting a specific <dir name="..."> instead of
>>>>>>>>> \d{4}, etc., and
>>>>>>>>> then can you put a specific file name in the <file name="..">. If
>>>>>>>>> that works and
>>>>>>>>> downloads, then we know there is something up with the regex
>>>>>>>>> somehow.
>>>>>>>>> Also can you try adding the <?xml version="1.0"?> start to your
>>>>>>>>> file?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote:
>>>>>>>>>
>>>>>>>>>> Hi Chris,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The following is the content of TESL2CO2.xml.
>>>>>>>>>> $ cat
>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>> <root>
>>>>>>>>>> <dirstruct starting_path="/TES/TL2CO2N.005">
>>>>>>>>>> <nofiles/>
>>>>>>>>>> <dir name="\d{4}\.\d{2}\.\d{2}">
>>>>>>>>>> <nodirs/>
>>>>>>>>>> <file
>>>>>>>>>> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/>
>>>>>>>>>> </dir>
>>>>>>>>>> </dirstruct>
>>>>>>>>>> </root>
>>>>>>>>>>
>>>>>>>>>> Let me know if there is a problem in the configuration file.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Yunhee
>>>>>>>>>>
>>>>>>>>>> 2012/7/24 Mattmann, Chris A (388J)
>>>>>>>>>> <chris.a.mattm...@jpl.nasa.gov>:
>>>>>>>>>>> Hi YunHee,
>>>>>>>>>>>
>>>>>>>>>>> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>
>>>>>>>>>>>> I am stuck on the problem related with "pushpull" script
>>>>>>>>>>>> occurred
>>>>>>>>>>>> the following error message:
>>>>>>>>>>>> ....
>>>>>>>>>>>> INFO: Testing protocol
>>>>>>>>>>>> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . .
>>>>>>>>>>>> this may
>>>>>>>>>>>> take a few minutes . . .
>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
>>>>>>>>>>>> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with
>>>>>>>>>>>> protocol
>>>>>>>>>>>> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and
>>>>>>>>>>>> username 'anonymous'
>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup
>>>>>>>>>>>> retrieveFiles
>>>>>>>>>>>> SEVERE: Failed to finish downloading per property files
>>>>>>>>>>>>
>>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>> : null
>>>>>>>>>>>
>>>>>>>>>>> Can you do me a favor and simply run this command:
>>>>>>>>>>>
>>>>>>>>>>> cat
>>>>>>>>>>> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml
>>>>>>>>>>>
>>>>>>>>>>> What do you get?
>>>>>>>>>>>
>>>>>>>>>>>> Jul 24, 2012 4:05:02 PM
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>>>>>>>>>>>> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going
>>>>>>>>>>>> to
>>>>>>>>>>>> sleep until Tue Jul 24 16:06:00 KST 2012
>>>>>>>>>>>> Jul 24, 2012 4:06:00 PM
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning
>>>>>>>>>>>> INFO: Daemon with ID = 91131 was given permission to run
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
>>>>>>>>>>>>
>>>>>>>>>>>> First of all, I would like test the pushpull framework to move
>>>>>>>>>>>> data
>>>>>>>>>>>> written in NetCDF from a remote site by a network protocols
>>>>>>>>>>>> (like ftp,
>>>>>>>>>>>> http, etc).
>>>>>>>>>>>
>>>>>>>>>>> Yep, sounds good.
>>>>>>>>>>>
>>>>>>>>>>>> I think that there is a test suite which can be used to check
>>>>>>>>>>>> the
>>>>>>>>>>>> pushpull framework.
>>>>>>>>>>>> Can you give configuration files(a set of XML files) that I can
>>>>>>>>>>>> test
>>>>>>>>>>>> downloading files from a FTP site (public anonymous site) ?
>>>>>>>>>>>
>>>>>>>>>>> The examples in the resources directory that you found should
>>>>>>>>>>> work
>>>>>>>>>>> so we'll just need to figure out why they aren't working on your
>>>>>>>>>>> system.
>>>>>>>>>>>
>>>>>>>>>>>> or
>>>>>>>>>>>> Please let me know a reference implementation of pushpull
>>>>>>>>>>>> framework
>>>>>>>>>>>> that I can use to move NetCDF files.
>>>>>>>>>>>
>>>>>>>>>>> Thanks we'll figure this out.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>>>> Senior Computer Scientist
>>>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>>>>
>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>>>>
>>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>> Senior Computer Scientist
>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Chris Mattmann, Ph.D.
>>>>>>> Senior Computer Scientist
>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>>> Email: chris.a.mattm...@nasa.gov
>>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>
>>>>>
>>>>>
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Chris Mattmann, Ph.D.
>>>>> Senior Computer Scientist
>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> Office: 171-266B, Mailstop: 171-246
>>>>> Email: chris.a.mattm...@nasa.gov
>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattm...@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> Phone: +1 (818) 354-8810
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
