Hi Brian,
> Also, if you want to turn off filemgr communication you should only comment > out the following property: > > org.apache.oodt.cas.filemgr.url If I turned off the property that is related with filemgr mentioned above, there was no problem to download a file from remote site. I also checked the file was in the staging area: $ ls TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5 TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5.info.tmp I found an extra file named "TES-Aura_L2-CO2-Nadir_r0000002147_F06_09.he5.info.tmp". I think the extra file was generated by oodt. Please let me know what the file is. I also have a question about pushpull script. The pushpull script has been running without termination even if the file was completely downloaded. Is it working well ? > Let's go this route instead... don't worry about the mime-types.xml files... > in your push_pull_framework.properties file just modify the following > property to the following: > > org.apache.oodt.cas.pushpull.allow.only.defined.types=false I modified the value of the property mentioned above to false. It didn't give an answer to fix my problem. > does your filemgr have any products ingested? Yes it does. I tried to test that my filemgr have a text file that was ingested. $./filemgr-client.sh Aug 1, 2012 10:26:54 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties] ingestProduct: Result: fb44d9a8-db77-11e1-a6d8-abc525326b84 I used the filemgr-client.sh as follows: #!/bin/sh ./filemgr-client --url http://localhost:8000 --operation --ingestProduct --prod uctName blahProduct --productStructure Flat --productTypeName GenericFile --me tadataFile file:///home/yhkang/oodt-0.5/cas-filemgr/bin/blah.txt.met --refs file:// /home/yhkang/oodt/cas-filemgr/bin/blah.txt But I am not sure that the the file was ingested well. Because I can't get a result by the following query: ./query_tool --url http://localhost:8000 --lucene -query CAS.ProductType:GenericFile As you can see, I used 8000 as the port number of my filemgr. Let me know if there is a problem in the query. I am sorry to give you too many questions. Thanks, Yunee 2012/7/31 Brian Foster <holeno...@me.com>: > Hey YunHee, > > Let's go this route instead... don't worry about the mime-types.xml files... > in your push_pull_framework.properties file just modify the following > property to the following: > > org.apache.oodt.cas.pushpull.allow.only.defined.types=false > > That turns off the mime-type stuff... > > Also, if you want to turn off filemgr communication you should only comment > out the following property: > > org.apache.oodt.cas.filemgr.url > > However you should be fine with just turning off the mime-type stuff like > described above... another thing, does your filemgr have any products > ingested? i know there was a bug (not sure if it had been fixed yet) where > the filemgr would throw an exception if you tried to query it when there > were no products ingested (This is when using Lucene for the catalog > store)... you may want to just make sure there is a dummy file ingested > > -brian > > > On Jul 29, 2012, at 11:50 PM, YunHee Kang <yunh.k...@gmail.com> wrote: > > Hi Brain, > > I found two mime-types.xml files: > - pushpull/policy/mime-types.xml > - cas-filemgr/etc/mimet-ypes.xml > > In the directory pushpull/policy, I put the element "mine-type" like > the following into mimetypes.xml: > <mime-type type="product/tes"> > <_comment>ProductType=MyTesProductType</_comment> > <glob > pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5" > isregex="true"/> > </mime-type> > >> fill in an actual filemgr ProductType in place of >> MyTesProdectType... and the pattern should be the regex describing the >> filenames of the files that belong to the give ProductType. > > I also put the same "mime-type" into mime-types.xml in the directory > cas-filemgr/etc. > <mime-type type="product/tes"> > <_comment>ProductType=MyTesProductType</_comment> > <glob > pattern="TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5" > isregex="true"/> > </mime-type> > > Am I doing well ? > > Thanks, > Yunhee > > 2012/7/30 Brian Foster <holeno...@me.com>: >> hey YunHee, >> >> I keep forgetting to add you to the email cc... are you on the dev >> list?... >> see email below, explains your commented out filemgr url making the >> pushpull >> work >> >> -brian >> >> Begin forwarded message: >> >> From: Brian Foster <holeno...@me.com> >> Date: July 29, 2012 10:12:48 AM PDT >> To: "dev@oodt.apache.org" <dev@oodt.apache.org> >> Cc: Brian Foster <holeno...@mac.com>, "dev@oodt.apache.org" >> <dev@oodt.apache.org>, "Mattmann, Chris A (388J)" >> <chris.a.mattm...@jpl.nasa.gov> >> Subject: Re: Problem when running pushpull >> Reply-To: dev@oodt.apache.org >> >> Hey YunHee, >> >> You need to configure you mime-types.xml file in your policy directory to >> map your file names to filemgr ProductTypes... if you want to just >> download >> the files for now just comment out the filemgr >> URL property in your pushpull properties file... if you want to configure >> the mime-types.xml file, take a look at: >> >> >> https://svn.apache.org/repos/asf/oodt/trunk/pushpull/src/main/resources/policy/mimetypes.xml >> >> Here is basically how the file works: >> >> <mime-type type="product/tes"> >> <_comment>ProductType=MyTesProductType</_comment> >> <glob pattern="any valid Pattern.java regex" is regex="true"/> >> </mime-type> >> >> the mime-type "type" must have a / and is just a name being used to title >> the mapping... fill in an actual filemgr ProductType in place of >> MyTesProdectType... and the pattern should be the regex describing the >> filenames of the files that belong to the give ProductType. >> >> then make sure that >> org.apache.oodt.cas.pushpull.config.type.dectection.file >> property points at your mime-types.xml file. >> >> -Brian >> On Jul 29, 2012, at 6:45 AM, YunHee Kang <yunh.k...@gmail.com> wrote: >> >> Hi Brian, >> >> >> I checked out the latest trunk of oodt 0.5 and installed it. >> >> After I set up the file manger and pushpull framework, I ran the >> >> pushpull script. >> >> As I described below, I think that the trunk is useful to fix the >> >> problem, which was caused by handling ftp protocol. >> >> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >> >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >> >> username 'anonymous' >> >> Jul 29, 2012 10:09:30 PM >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem >> >> initializeStagingArea >> >> INFO: Preparing staging area >> /home/yhkang/oodt-0.5/cas-pushpull/staging/TESL2CO2 >> >> Jul 29, 2012 10:09:30 PM org.apache.commons.httpclient.HttpClient <clinit> >> >> FINE: Java version: 1.6.0_31 >> >> >> But I got the difference problem that is related with filemgr. >> >> The following is part of error message in the file cas-pushpull0.log : >> >> Jul 29, 2012 10:09:30 PM >> >> org.apache.oodt.cas.filemgr.ingest.StdIngester hasProduct >> >> WARNING: Unable to check for existance of product: [null]: Message: >> >> Failure writing request >> >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >> >> Failure writing request >> >> at >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.hasProduct(XmlRpcFileManagerClient.java:606) >> >> at >> >> org.apache.oodt.cas.filemgr.ingest.StdIngester.hasProduct(StdIngester.java:284) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.isAlreadyInDatabase(FileRetrievalSystem.java:254) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.addToDownloadQueue(FileRetrievalSystem.java:463) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:138) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >> >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >> >> at java.lang.Thread.run(Thread.java:662) >> >> Jul 29, 2012 10:09:30 PM >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> >> retrieveFiles >> >> SEVERE: Failed to finish downloading per property files >> >> >> /home/yhkang/oodt-0.5/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> >> : [RemoteCrawler] Failed to communicate with database : Failure >> >> writing request >> >> org.apache.oodt.cas.pushpull.exceptions.RetrievalMethodException: >> >> [RemoteCrawler] Failed to communicate with database : Failure writing >> >> request >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:150) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >> >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >> >> at java.lang.Thread.run(Thread.java:662) >> >> Jul 29, 2012 10:09:30 PM >> >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect >> >> INFO: Disconnecting protocol >> >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol >> >> Jul 29, 2012 10:09:30 PM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >> >> INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to >> >> sleep until Sun Jul 29 22:12:00 KST 2012 >> >> >> Please let me know how to fix this problem. >> >> >> Thanks, >> >> Yunhee >> >> >> >> >> 2012/7/28 Brian Foster <holeno...@me.com>: >> >> Hey YunHee, >> >> >> Doesn't look like the patch took this time, but don't worry about that >> now, >> i have put the fixes into the trunk... if you just check out the latest >> trunk and mvn install it (make sure you check out more than just >> cas-pushpull, because if protocol-ftp doesn't get reinstalled your >> cas-pushpull mvn install will pick up the last mvn installed version of >> the >> protocol-ftp 0.5 in your local maven repository... another thing to note, >> you probably should utilize the environment variables in place in the >> pushpull config files because if you ever decide to move your install you >> are going to have to change all the paths in your configuration files >> instead of just being able to change environment variables (if you have >> any >> question about that let me know)... again if you run into any road blocks >> just hit up the list and sorry about the bumpy road of getting pushpull up >> and running... it should be good to go now though (it downloads the files >> on >> my machine at least... lol). >> >> >> -brian >> >> >> On Jul 27, 2012, at 10:56 AM, YunHee Kang wrote: >> >> >> Hi Brain, >> >> >> Unfortunately your patch didn't fix the problem. I got the same >> >> error message before: >> >> SEVERE: Failed to finish downloading per property files >> >> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> >> : null >> >> java.lang.NullPointerException >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104) >> >> at >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >> >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >> >> at java.lang.Thread.run(Thread.java:662) >> >> Jul 28, 2012 2:31:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep >> >> >> First of all, I replaced CommonsNetFtpProtocol.java with new one >> >> that you modified as follows: >> >> ftp.retrieveFile(fromFile.getName(), os); >> >> /* >> >> if (ftp.retrieveFile(fromFile.getName(), os))// { >> >> throw new ProtocolException("Failed to download file " >> >> + fromFile.getName()); >> >> // } >> >> >> */ >> >> Next I rebuild a new jar file cas-protocol-ftp-0.4.jar and then >> >> copied it to lib directory: >> >> -rw-r--r-- 1 yhkang yhkang 17279 2012-07-28 02:09 cas-protocol-ftp-0.4.jar >> >> ls -al cas-protocol-ftp-0.4.jar >> >> >> I am wondering that there is difference between the following codes(1 and >> 2) : >> >> 1) ftp.retrieveFile(fromFile.getName(), os); >> >> 2) if (ftp.retrieveFile(fromFile.getName(), os))// { >> >> throw new ProtocolException("Failed to download file " >> >> + fromFile.getName()); >> >> // } >> >> >> Will you let me know if the method get() in the >> >> CommonsNetFtpProtocol.java was called when pushpull was run ? >> >> >> Thanks, >> >> Yunhee >> >> >> >> 2012/7/27 Brian Foster <holeno...@mac.com>: >> >> Hey YunHee, >> >> >> Okay the latest patch attached to the issue now (the on with todays date) >> >> works as far as pushpull is concerned, however FTPClient.retrieveFile >> being >> >> used in CommonsNetFtpProtocol is returning false for some reason... so i >> >> would recommend trying the following... first unpatch the first patch from >> >> your cas-pushpull and repatch with the new patch i've uploaded... then >> check >> >> out https://svn.apache.org/repos/asf/oodt/trunk/protocol/ftp/ and modify >> >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol line 160 from this: >> >> >> >> if (ftp.retrieveFile(fromFile.getName(), os))// { >> >> >> throw new ProtocolException("Failed to download file " >> >> >> + fromFile.getName()); >> >> >> // } >> >> >> >> to this: >> >> >> >> ftp.retrieveFile(fromFile.getName(), os); >> >> >> >> then mvn compile jar:jar and replace ftp protocol jar with that one in >> your >> >> pushpull lib directory and give pushpull another run and see what happens >> >> (validate the files downloaded to make sure they are complete)... hope >> this >> >> helps and let me know if you need any further help! >> >> >> >> -brian >> >> >> >> On Jul 26, 2012, at 03:37 PM, YunHee Kang <yunh.k...@gmail.com> wrote: >> >> >> Hi Brian, >> >> >> I attach a tar file including etc and policy directories. >> >> >> Thanks, >> >> Yunhee >> >> >> 2012/7/27 Brian Foster <holeno...@mac.com>: >> >> Hey YunHee, >> >> >> send me a tarball with your etc and policy directories i will spend some >> >> time today figuring out what is going on here >> >> >> -brian >> >> >> >> On Jul 26, 2012, at 01:26 AM, YunHee Kang <yunh.k...@gmail.com> wrote: >> >> >> Hi Brian, >> >> >> I just applied the patch you made according to the following steps: >> >> cd apache-oodt-0.4-src/pushpull >> >> patch -p0 < OODT-476.2012-07-25.txt >> >> mvn clean >> >> mvn install >> >> cp target/cas-pushpull-0.4.jar ~/oodt/cas-pushpull/lib/ >> >> >> After I got a new cas-pushpull-0.4.jar, >> >> yhkang@yunhee-kisti:~/oodt/cas-pushpull/lib$ ls -l cas-pushpull-0.4.jar >> >> -rw-r--r-- 1 yhkang yhkang 422602 2012-07-26 16:58 cas-pushpull-0.4.jar >> >> I ran the script pushpull. >> >> >> >> But the same error message was generated: >> >> >> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >> >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >> >> username 'anonymous' >> >> Jul 26, 2012 5:12:56 PM >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> >> retrieveFiles >> >> SEVERE: Failed to finish downloading per property files >> >> >> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> >> : null >> >> java.lang.NullPointerException >> >> at >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:368) >> >> at >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:312) >> >> at >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:301) >> >> at >> >> >> >> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:104) >> >> at >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >> >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >> >> at java.lang.Thread.run(Thread.java:662) >> >> >> Could you check a class name for handling ftp protocol ? >> >> I used "CommonsNetFtpProtocolFactory.class" in the jar named >> >> "cas-protocol-ftp-0.4.jar". >> >> The class file was described in the file ProtocolFactoryInfo.xml as >> >> follows: >> >> <protocol type="ftp"> >> >> .... >> >> <!-- cas-protocol-ftp-0.4.jar --> >> >> <!--protocolFactory >> >> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory"/--> >> >> <protocolFactory >> >> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/> >> >> </protocol> >> >> >> Thanks, >> >> Yunhee >> >> >> >> >> >> 2012/7/26 Brian Foster <holeno...@mac.com>: >> >> hey Yunhee, >> >> >> i just realized that your probably not on the dev list and my last email >> >> just went to the list and didn't have you cc'ed >> >> >> try the patch attached to this issue mentioned below... >> >> >> -brian >> >> >> Begin forwarded message: >> >> >> From: Brian Foster <holeno...@mac.com> >> >> Date: July 25, 2012 12:43:10 AM >> >> To: dev@oodt.apache.org,dev@oodt.apache.org >> >> Subject: Re: Problem when running pushpull >> >> >> hey Yunhee, >> >> >> Try patching your pushpull code with the fix i've attached to issue: >> >> >> https://issues.apache.org/jira/browse/OODT-476 >> >> >> -brian >> >> >> On Jul 24, 2012, at 09:24 PM, YunHee Kang <yunh.k...@gmail.com> wrote: >> >> >> Hi Brian, >> >> >> I changed the file RetrievalSetup.java to get the stacktrace. >> >> So I got a new cas-pushpull-0.4.jar after rebuilt it. >> >> >> The following is a message including the stacktrace with information >> >> about excpetion propation. >> >> >> SEVERE: Failed to finish downloading per property files >> >> >> >> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >> >> : null >> >> java.lang.NullPointerException >> >> at >> >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.validate(FileRetrievalSystem.java:367) >> >> at >> >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:311) >> >> at >> >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem.changeToDir(FileRetrievalSystem.java:300) >> >> at >> >> >> >> >> org.apache.oodt.cas.pushpull.retrievalmethod.RemoteCrawler.processPropFile(RemoteCrawler.java:103) >> >> at >> >> >> >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.retrieveFiles(RetrievalSetup.java:109) >> >> at org.apache.oodt.cas.pushpull.daemon.Daemon$1.run(Daemon.java:218) >> >> at java.lang.Thread.run(Thread.java:662) >> >> >> Please let me know what java programs I check frist. >> >> >> Thanks, >> >> >> Yunhee >> >> 2012/7/25 Brian Foster <holeno...@mac.com>: >> >> YunHee, >> >> >> There is a NullPointerException being thrown... on line 118 of >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup.java change >> >> it >> >> >> rebuild and deploy and that should give you the stacktrace which will >> >> answer >> >> what is going on here. >> >> >> -brian >> >> >> >> On Jul 24, 2012, at 10:51 AM, "Mattmann, Chris A (388J)" >> >> >> <chris.a.mattm...@jpl.nasa.gov> wrote: >> >> >> Hi YunHee, >> >> >> Thanks for your mail. >> >> >> Let me take a look at the configs again. Does anyone else using >> >> PushPull have any advice for YunHee here? That error message >> >> seems to indicate there is a config file error somewhere. I'm wondering >> >> if your etc/push_pull_framework.properties has all the right values >> >> in there. Did you go through Cameron's guide to getting Push Pull >> >> running on the wiki? >> >> >> Cheers, >> >> Chris >> >> >> >> On Jul 24, 2012, at 10:48 AM, YunHee Kang wrote: >> >> >> Hi Chris, >> >> >> There are some problems in the configuration files. >> >> I found there is a dead link which does not connect two information >> >> in two files :RemoteSpecs.xml and ExternalSources.xml. >> >> For example an alias named "DewFtp" in RemoteSpecs.xml has no >> >> associated source in "ExternalSources.xml. >> >> >> After I tailored those two XML files, I just tested one of >> >> configurations, whose alias is PolarNcepNoaaFtp. >> >> I also faced the same problem as follows: >> >> >> INFO: Successfully connected to ftp://polar.ncep.noaa.gov with >> >> protocol 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >> >> username 'anonymous' >> >> Jul 25, 2012 2:21:04 AM >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> >> retrieveFiles >> >> SEVERE: Failed to finish downloading per property files >> >> >> >> >> >> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/RTG_SST.xml >> >> : null >> >> Jul 25, 2012 2:21:04 AM org.apache.oodt.cas.pushpull.daemon.Daemon >> >> sleep >> >> >> >> Thanks, >> >> Yunhee >> >> >> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >> >> Hi YunHee, >> >> >> Thanks! >> >> >> Cheers, >> >> Chris >> >> >> >> On Jul 24, 2012, at 9:54 AM, YunHee Kang wrote: >> >> >> Hi Chris, >> >> >> Thanks, can you try some of the other XML file configs. >> >> Yes, I will try it and let you know its result. >> >> >> Maybe it's being blocked in your country, >> >> That site may be blocked. >> >> >> >> Thanks, >> >> Yunhee >> >> >> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >> >> Hi YunHee, >> >> >> >> Thanks, can you try some of the other XML file configs. Maybe it's >> >> being >> >> blocked in your country, or something, but let's see if we can get >> >> some >> >> of the other XML RemoteSpec configs to work? >> >> >> Cheers, >> >> Chris >> >> >> On Jul 24, 2012, at 9:40 AM, YunHee Kang wrote: >> >> >> Hi Chris, >> >> >> >> Try out the MODIS one there, does that work? >> >> No, it doesn't. >> >> I think that the ftp site e4ftl01u.ecs.nasa.gov has a connection >> >> problem. >> >> >> $ ftp e4ftl01u.ecs.nasa.gov >> >> ftp: connect: Connection timed out >> >> ftp> >> >> >> The message below also say the reason why the problem was occurred. >> >> WARNING: Error occurred while connecting to RemoteSite: alias = >> >> 'ModisFtp' url = 'ftp://e4ftl01u.ecs.nasa.gov' username = >> >> 'anonymous' cdTestDir = 'null' maxConnections = '-1' : Failed to >> >> connect to server : Connection timed out >> >> Jul 25, 2012 1:31:25 AM >> >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >> >> INFO: Will retry connecting to RemoteSite: alias = 'ModisFtp' url = >> >> 'ftp://e4ftl01u.ecs.nasa.gov' username = 'anonymous' cdTestDir = >> >> 'null' maxConnections = '-1' in 5 seconds >> >> Waiting . . . . . >> >> >> Thanks, >> >> Yunhee >> >> >> 2012/7/25 Mattmann, Chris A (388J) <chris.a.mattm...@jpl.nasa.gov>: >> >> Hi YunHee, >> >> >> >> OK, can you try some of the other files, e.g., the ones in: >> >> >> http://s.apache.org/ks0 >> >> >> Try out the MODIS one there, does that work? >> >> >> Cheers, >> >> Chris >> >> >> On Jul 24, 2012, at 8:38 AM, YunHee Kang wrote: >> >> >> Hi Chris, >> >> >> I added the header of XML as follows: >> >> <?xml version="1.0"?> >> >> >> Next the value of elements was changed into specific one. >> >> The following is the output of the XML: >> >> <?xml version="1.0"?> >> >> <root> >> >> <dirstruct starting_path="/TES/TL2CO2N.005"> >> >> <nofiles/> >> >> <!--dir name="\d{4}\.\d{2}\.\d{2}"--> >> >> <dir name="2004\.09\.20"> >> >> <nodirs/> >> >> <!--file >> >> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/--> >> >> <file name="TES-Aura\_L2-CO2-Nadir\_r0000002147\_F06\_09\.he5"/> >> >> </dir> >> >> </dirstruct> >> >> </root> >> >> >> Unfortunately I got the same result before: >> >> ... >> >> INFO: Testing protocol >> >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . this >> >> ma >> >> y take a few minutes . . . >> >> Jul 25, 2012 12:30:03 AM >> >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >> >> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with >> >> protocol 'org.apache.oodt. >> >> cas.protocol.ftp.CommonsNetFtpProtocol' and username 'anonymous' >> >> Jul 25, 2012 12:30:03 AM >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> >> retrie >> >> veFiles >> >> SEVERE: Failed to finish downloading per property files >> >> /home/yhkang/oodt/cas-pushpull/etc/ >> >> examples/DirStructXmlParserFiles/TESL2CO2.xml : null >> >> Jul 25, 2012 12:30:04 AM >> >> org.apache.oodt.cas.pushpull.daemon.Daemon >> >> sleep >> >> INFO: Daemon with ID = 91131 on RMI registry port 9113 is going >> >> to >> >> sleep until Wed Jul 25 0 >> >> ... >> >> >> >> Thanks, >> >> Yunhee >> >> >> >> 2012/7/25 Mattmann, Chris A (388J) >> >> <chris.a.mattm...@jpl.nasa.gov>: >> >> Hi YunHee, >> >> >> >> Can you try simply putting a specific <dir name="..."> instead >> >> of >> >> \d{4}, etc., and >> >> then can you put a specific file name in the <file name="..">. >> >> If >> >> that works and >> >> downloads, then we know there is something up with the regex >> >> somehow. >> >> Also can you try adding the <?xml version="1.0"?> start to your >> >> file? >> >> >> Cheers, >> >> Chris >> >> >> On Jul 24, 2012, at 8:16 AM, YunHee Kang wrote: >> >> >> Hi Chris, >> >> >> >> The following is the content of TESL2CO2.xml. >> >> $ cat >> >> >> >> >> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> >> <root> >> >> <dirstruct starting_path="/TES/TL2CO2N.005"> >> >> <nofiles/> >> >> <dir name="\d{4}\.\d{2}\.\d{2}"> >> >> <nodirs/> >> >> <file >> >> name="TES-Aura\_L2-CO2-Nadir\_r\d{10}\w{2}\d{2}\w\d{2}\.he5"/> >> >> </dir> >> >> </dirstruct> >> >> </root> >> >> >> Let me know if there is a problem in the configuration file. >> >> >> Thanks, >> >> Yunhee >> >> >> 2012/7/24 Mattmann, Chris A (388J) >> >> <chris.a.mattm...@jpl.nasa.gov>: >> >> Hi YunHee, >> >> >> >> On Jul 24, 2012, at 12:30 AM, YunHee Kang wrote: >> >> >> Hi Chris, >> >> >> I am stuck on the problem related with "pushpull" script >> >> occurred >> >> the following error message: >> >> .... >> >> INFO: Testing protocol >> >> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol . . . >> >> this may >> >> take a few minutes . . . >> >> Jul 24, 2012 4:05:02 PM >> >> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect >> >> INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov >> >> with >> >> protocol >> >> 'org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol' and >> >> username 'anonymous' >> >> Jul 24, 2012 4:05:02 PM >> >> org.apache.oodt.cas.pushpull.retrievalsystem.RetrievalSetup >> >> retrieveFiles >> >> SEVERE: Failed to finish downloading per property files >> >> >> >> >> >> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> >> : null >> >> >> Can you do me a favor and simply run this command: >> >> >> cat >> >> >> >> >> >> /home/yhkang/oodt/cas-pushpull/etc/examples/DirStructXmlParserFiles/TESL2CO2.xml >> >> >> What do you get? >> >> >> Jul 24, 2012 4:05:02 PM >> >> org.apache.oodt.cas.pushpull.daemon.Daemon sleep >> >> INFO: Daemon with ID = 91131 on RMI registry port 9113 is >> >> going >> >> to >> >> sleep until Tue Jul 24 16:06:00 KST 2012 >> >> Jul 24, 2012 4:06:00 PM >> >> org.apache.oodt.cas.pushpull.daemon.DaemonManager >> >> setAsRunning >> >> INFO: Daemon with ID = 91131 was given permission to run >> >> >> >> >> >> org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser >> >> >> First of all, I would like test the pushpull framework to >> >> move >> >> data >> >> written in NetCDF from a remote site by a network protocols >> >> (like ftp, >> >> http, etc). >> >> >> Yep, sounds good. >> >> >> I think that there is a test suite which can be used to check >> >> the >> >> pushpull framework. >> >> Can you give configuration files(a set of XML files) that I >> >> can >> >> test >> >> downloading files from a FTP site (public anonymous site) ? >> >> >> The examples in the resources directory that you found should >> >> work >> >> so we'll just need to figure out why they aren't working on >> >> your >> >> system. >> >> >> or >> >> Please let me know a reference implementation of pushpull >> >> framework >> >> that I can use to move NetCDF files. >> >> >> Thanks we'll figure this out. >> >> >> Cheers, >> >> Chris >> >> >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Chris Mattmann, Ph.D. >> >> Senior Computer Scientist >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >> Office: 171-266B, Mailstop: 171-246 >> >> Email: chris.a.mattm...@nasa.gov >> >> WWW: http://sunset.usc.edu/~mattmann/ >> >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Adjunct Assistant Professor, Computer Science Department >> >> University of Southern California, Los Angeles, CA 90089 USA >> >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Chris Mattmann, Ph.D. >> >> Senior Computer Scientist >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >> Office: 171-266B, Mailstop: 171-246 >> >> Email: chris.a.mattm...@nasa.gov >> >> WWW: http://sunset.usc.edu/~mattmann/ >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Adjunct Assistant Professor, Computer Science Department >> >> University of Southern California, Los Angeles, CA 90089 USA >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Chris Mattmann, Ph.D. >> >> Senior Computer Scientist >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >> Office: 171-266B, Mailstop: 171-246 >> >> Email: chris.a.mattm...@nasa.gov >> >> WWW: http://sunset.usc.edu/~mattmann/ >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Adjunct Assistant Professor, Computer Science Department >> >> University of Southern California, Los Angeles, CA 90089 USA >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Chris Mattmann, Ph.D. >> >> Senior Computer Scientist >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >> Office: 171-266B, Mailstop: 171-246 >> >> Email: chris.a.mattm...@nasa.gov >> >> WWW: http://sunset.usc.edu/~mattmann/ >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Adjunct Assistant Professor, Computer Science Department >> >> University of Southern California, Los Angeles, CA 90089 USA >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Chris Mattmann, Ph.D. >> >> Senior Computer Scientist >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >> Office: 171-266B, Mailstop: 171-246 >> >> Email: chris.a.mattm...@nasa.gov >> >> WWW: http://sunset.usc.edu/~mattmann/ >> >> Phone: +1 (818) 354-8810 >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Adjunct Assistant Professor, Computer Science Department >> >> University of Southern California, Los Angeles, CA 90089 USA >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Chris Mattmann, Ph.D. >> >> Senior Computer Scientist >> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> >> Office: 171-266B, Mailstop: 171-246 >> >> Email: chris.a.mattm...@nasa.gov >> >> WWW: http://sunset.usc.edu/~mattmann/ >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> Adjunct Assistant Professor, Computer Science Department >> >> University of Southern California, Los Angeles, CA 90089 USA >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >>
