Jenkins build is unstable: oodt-trunk #307

[Apache Jenkins Server]

See 

Jenkins build is back to stable : oodt-trunk #308

[Apache Jenkins Server]

See 

Re: question on push pull

[Mattmann, Chris A (388J)]

Hey Guys,

Just wanted to loop in on this. Yes, push pull does support download
of full directory trees. Probably the best guide to check out is Brian's
documentation on the DirFileStructXML here:

http://s.apache.org/yz

I also wrote a static user guide here:

http://s.apache.org/10Z

One thing to note too is that some of the plugins for Push Pull that
have been develop use non ALv2 compatible code, so if you want
those plugins for now (until someone writes nice shiny new ALv2
compatible versions which I would LOVE), you can find them here
at Apache Extras:

http://code.google.com/a/apache-extras.org/p/oodt-pushpull-plugins/

That you can cull information from. If you have any further specific
questions I can help.

Thanks!

Cheers,
Chris

On Feb 28, 2012, at 7:40 AM, Cameron Goodale wrote:

> Luca,
> 
> I haven't tried this exact use case within Crawler, but Crawler does
> support scp and I have used 'scp -r' to recursively download a folder and
> all content housed within.  I can only imagine ftp has a similar recursive
> option as well.
> 
> Maybe another more Crawler Savy dev can shine some light on the recursion
> use case when using Crawler.
> 
> -Cameron
> 
> P.S. When we get a final answer let's add this to the Crawler User Guide
> Wiki too as an example use case.  Glad you found the Crawler Wiki page
> useful.
> 
> On Tue, Feb 28, 2012 at 7:01 AM, Cinquini, Luca (3880) <
> luca.cinqu...@jpl.nasa.gov> wrote:
> 
>> Hi all,
>>   I have a quick question concerning the pushpull framework : is
>> there any way to transfer full directory trees, as opposed to single files
>> ? And which of the currently implemented transfer protocols would allow
>> that ? I haven't see any examples on that, though I might have missed it.
>> 
>> thanks a lot,
>> Luca
>> 
>> P.S.: Cameron, thanks for writing the push-pull user guide - it's great.
>> 
>> 
> 
> 
> -- 
> 
> Sent from a Tin Can attached to a String


++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++

Re: Continuing a workflow after restarting the workflow manager

[Mattmann, Chris A (388J)]

Hi Keith,

For whatever reason I cannot find the original email for this thread in my 
email reader, gah. So, I am going to reply to this particular version of the
thread so sorry for those looking at the mail archives that will have a hard
time finding this.

In short, the functionality that you're asking for can be supported by the 0.3
workflow manager, it's just more cumbersome to implement, and it's not
something that the config supports out of the box, so you'll have to play
with it a bit. The wengine branch [1] supports this more flexibly and more 
natively, but we're in the process (in 0.4-SNAPSHOT) of porting over
this functionality so that 0.4 (or 0.5) and beyond it'll be supported natively.
I think Paul Ramirez linked you over to OODT-215 [2] which is a good place
to check the status overall of that effort.

Right now, with 0.3 if you want to support that functionality to dynamically
restart a workflow at some point in the pipeline, here's how you can do it.
Let's assume that you have a 3 task workflow, t1, t2 and t3.

You would set up 4 workflow XML documents that contain the following
variations in those task:

w1.workflow.xml -> only runs t1
w2.workflow.xml -> only runs t2
w3.workflow.xml -> only runs t3
w4.workflow.xml -> runs t1, then t2, then t3

(note: you could also do the other permeations for t1->t2, or t2->t3, but
I'm leaving those out for brevity and for simplicity). 

Once you set those workflow XML files up, you would then add 4 events
to the events.xml file:

e1->runs w1
e2->runs w2
e3->runs w3
e4->runs w4

Then with the above set up, you could theoretically recover from a workflow
whether it's restart or not, assuming that the context of the restart is at 
the task level, and assuming that you could wire up the same metadata
context as was currently running by gleaning it from the 
WorkflowInstanceRepository.
The WorkflowInstanceRepository *does* store the current state of the 
WorkflowInstance
in terms of its:

* start/stop task ISO 8601 times
* start/stop overall workflow ISO 8601 times
* current workflow instance metadata (the dynamic "context" that tasks read 
from and write to)
* current task running

The combination of the above information is enough to recover from a failure in 
an existing
workflow task, at a coarse grained level. At a more finer grained level, to 
perform that kind
of checkpointing and recovery is kind of difficult, and really domain specific 
to the type of
workflow task that is being written. I was just discussing this with Sheryl in 
OODT-212 [3]
and one methodology that could be employed is to force WorkflowTaskInstances to 
implement
a rollback method (perhaps guided by JTA or some other Java-based transaction 
model).
That would be a change to existing users and code bases with existing compiled 
tasks and
code. However, since CAS-PGE [4] is a common mechanism for folks to run 
workflows
and integrate science algorithms, changing there could provide the necessary 
insulation
and directed path to implement a capability like OODT-212.

To suppor the current functionality of the wengine branch, in the trunk 
workflow, I was 
thinking of a simple patch mechanism that did the following:

* upon restart of the WM, interrogate the Workflow Instance Repository and find 
any 
Workflows that are still in one of the executing states (STARTED, PAUSED, 
PGE_EXEC,
etc...) and then for those, roll them back to their started state, and go 
through execution
again.
* provide the capability for resume to actually "resume" a workflow in any one 
of the
above states.

What you are seeing with resume right now and with pause is that they only work 
on 
workflows that the engine is currently tracking. Using the 
ThreadPoolWorkflowEngine
extension point only keeps tracked and queued workflows that are executing in 
memory
so long as the WM is up and running. As soon as it goes down, though the 
context and
state information *is* persisted in the WorkflowInstanceRepository, the 
ThreadPoolWorkflowEngine
loses track of the executing information in *its* area, and thus cannot pause 
or resume after
the fact. So, pause/resume right now work, but they do so only in an active 
state, not for 
checkpointing *after* a WM restart.

But, we are working to support this in the trunk, and in various places around 
the OODT
ecosystem, it is supported already. Hope that clarifies and answers your 
question. Thanks 
much!

Cheers,
Chris


[1] http://svn.apache.org/repos/asf/oodt/branches/wengine-branch
[2] https://issues.apache.org/jira/browse/OODT-215
[3] https://issues.apache.org/jira/browse/OODT-212
[4] http://svn.apache.org/repos/asf/oodt/trunk/pge

On Feb 28, 2012, at 2:23 PM, Keith Cummings wrote:

> Hi Sheryl.
> I tried using the wmgr-client command line options to pause/resume/stop 
> workflow instances as you suggested.  It worked great; thanks for 
> pointing me there.  FYI, I'm using v0.3, if that matters.
> 
> As for the repo getting wiped out when the Workflow Manager is 
> 

Re: question on push pull

[Cinquini, Luca (3880)]

Hi Cameron,
Maybe I am confused, but I was actually asking about the push-pull 
capabilities - does the crawler plug into the push-pull framework ? (sorry 
about my ignorance here). If push-pull supports scp, would you know the name of 
the protocol transfer factory to use - I haven't found one.
thanks a lot,
Luca

On Feb 28, 2012, at 8:40 AM, Cameron Goodale wrote:

> Luca,
> 
> I haven't tried this exact use case within Crawler, but Crawler does
> support scp and I have used 'scp -r' to recursively download a folder and
> all content housed within.  I can only imagine ftp has a similar recursive
> option as well.
> 
> Maybe another more Crawler Savy dev can shine some light on the recursion
> use case when using Crawler.
> 
> -Cameron
> 
> P.S. When we get a final answer let's add this to the Crawler User Guide
> Wiki too as an example use case.  Glad you found the Crawler Wiki page
> useful.
> 
> On Tue, Feb 28, 2012 at 7:01 AM, Cinquini, Luca (3880) <
> luca.cinqu...@jpl.nasa.gov> wrote:
> 
>> Hi all,
>>   I have a quick question concerning the pushpull framework : is
>> there any way to transfer full directory trees, as opposed to single files
>> ? And which of the currently implemented transfer protocols would allow
>> that ? I haven't see any examples on that, though I might have missed it.
>> 
>> thanks a lot,
>> Luca
>> 
>> P.S.: Cameron, thanks for writing the push-pull user guide - it's great.
>> 
>> 
> 
> 
> -- 
> 
> Sent from a Tin Can attached to a String

Re: Continuing a workflow after restarting the workflow manager

[Keith Cummings]

Hey Chris.
It's good to see that there are options on the table and an eventual 
path forward on this issue.  Thanks for the all the info.

-Keith

Mattmann, Chris A (388J) wrote:

Hi Keith,

For whatever reason I cannot find the original email for this thread in my 
email reader, gah. So, I am going to reply to this particular version of the

thread so sorry for those looking at the mail archives that will have a hard
time finding this.

In short, the functionality that you're asking for can be supported by the 0.3
workflow manager, it's just more cumbersome to implement, and it's not
something that the config supports out of the box, so you'll have to play
with it a bit. The wengine branch [1] supports this more flexibly and more 
natively, but we're in the process (in 0.4-SNAPSHOT) of porting over

this functionality so that 0.4 (or 0.5) and beyond it'll be supported natively.
I think Paul Ramirez linked you over to OODT-215 [2] which is a good place
to check the status overall of that effort.

Right now, with 0.3 if you want to support that functionality to dynamically
restart a workflow at some point in the pipeline, here's how you can do it.
Let's assume that you have a 3 task workflow, t1, t2 and t3.

You would set up 4 workflow XML documents that contain the following
variations in those task:

w1.workflow.xml -> only runs t1
w2.workflow.xml -> only runs t2
w3.workflow.xml -> only runs t3
w4.workflow.xml -> runs t1, then t2, then t3

(note: you could also do the other permeations for t1->t2, or t2->t3, but
I'm leaving those out for brevity and for simplicity). 


Once you set those workflow XML files up, you would then add 4 events
to the events.xml file:

e1->runs w1
e2->runs w2
e3->runs w3
e4->runs w4

Then with the above set up, you could theoretically recover from a workflow
whether it's restart or not, assuming that the context of the restart is at 
the task level, and assuming that you could wire up the same metadata

context as was currently running by gleaning it from the 
WorkflowInstanceRepository.
The WorkflowInstanceRepository *does* store the current state of the 
WorkflowInstance
in terms of its:

* start/stop task ISO 8601 times
* start/stop overall workflow ISO 8601 times
* current workflow instance metadata (the dynamic "context" that tasks read 
from and write to)
* current task running

The combination of the above information is enough to recover from a failure in 
an existing
workflow task, at a coarse grained level. At a more finer grained level, to 
perform that kind
of checkpointing and recovery is kind of difficult, and really domain specific 
to the type of
workflow task that is being written. I was just discussing this with Sheryl in 
OODT-212 [3]
and one methodology that could be employed is to force WorkflowTaskInstances to 
implement
a rollback method (perhaps guided by JTA or some other Java-based transaction 
model).
That would be a change to existing users and code bases with existing compiled 
tasks and
code. However, since CAS-PGE [4] is a common mechanism for folks to run 
workflows
and integrate science algorithms, changing there could provide the necessary 
insulation
and directed path to implement a capability like OODT-212.

To suppor the current functionality of the wengine branch, in the trunk workflow, I was 
thinking of a simple patch mechanism that did the following:


* upon restart of the WM, interrogate the Workflow Instance Repository and find any 
Workflows that are still in one of the executing states (STARTED, PAUSED, PGE_EXEC,

etc...) and then for those, roll them back to their started state, and go 
through execution
again.
* provide the capability for resume to actually "resume" a workflow in any one 
of the
above states.

What you are seeing with resume right now and with pause is that they only work on 
workflows that the engine is currently tracking. Using the ThreadPoolWorkflowEngine

extension point only keeps tracked and queued workflows that are executing in 
memory
so long as the WM is up and running. As soon as it goes down, though the 
context and
state information *is* persisted in the WorkflowInstanceRepository, the 
ThreadPoolWorkflowEngine
loses track of the executing information in *its* area, and thus cannot pause 
or resume after
the fact. So, pause/resume right now work, but they do so only in an active state, not for 
checkpointing *after* a WM restart.


But, we are working to support this in the trunk, and in various places around 
the OODT
ecosystem, it is supported already. Hope that clarifies and answers your question. Thanks 
much!


Cheers,
Chris


[1] http://svn.apache.org/repos/asf/oodt/branches/wengine-branch
[2] https://issues.apache.org/jira/browse/OODT-215
[3] https://issues.apache.org/jira/browse/OODT-212
[4] http://svn.apache.org/repos/asf/oodt/trunk/pge

On Feb 28, 2012, at 2:23 PM, Keith Cummings wrote:

  

Hi Sheryl.
I tried using the wmgr-client command line options to pause/resume/stop 
workflow instanc

Re: Review Request: OODT-361: Workflow test event fails with NPE

[Ricky Nguyen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4089/
---

(Updated 2012-02-29 19:20:28.973385)


Review request for oodt.


Changes
---

updated exception msg


Summary
---

When DateConvert.isoParse is given a null input, it throws NPE.
Solution is throw ParseException if input is null. WorkflowInstance catches 
ParseException and ignores it.


This addresses bug OODT-361.
https://issues.apache.org/jira/browse/OODT-361


Diffs (updated)
-

  trunk/commons/src/main/java/org/apache/oodt/commons/util/DateConvert.java 
1295223 

Diff: https://reviews.apache.org/r/4089/diff


Testing
---

Ran HelloWorld, GoodbyeWorld.


Thanks,

Ricky

Re: Review Request: OODT-395: SerializableMetadata.toXml() doesn't set namespace on root element

[Ricky Nguyen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4088/
---

(Updated 2012-02-29 20:02:06.646815)


Review request for oodt.


Changes
---

test checks namespace value and prefix value


Summary
---

Create metadata root element with a proper namespace and prefix.


This addresses bug OODT-395.
https://issues.apache.org/jira/browse/OODT-395


Diffs (updated)
-

  
trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/SerializableMetadata.java
 1293402 
  
trunk/metadata/src/test/org/apache/oodt/cas/metadata/TestSerializableMetadata.java
 1293402 

Diff: https://reviews.apache.org/r/4088/diff


Testing
---

Added unit test in TestSerializableMetadata#testNamespace() to demonstrate bug 
and fix.


Thanks,

Ricky

Build failed in Jenkins: oodt-trunk #312

[Apache Jenkins Server]

See 

Changes:

[bfoster] - cas-cli sub-option requirements for help printing was not using 
p:required for GroupSubOption

--
OODT-399

--
[...truncated 1563 lines...]
[INFO] Surefire report directory: 


---
 T E S T S
---
Running org.apache.oodt.cas.crawl.option.TestBooleanOptions
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[JENKINS] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: 

[INFO] Preparing javadoc:javadoc
[WARNING] Removing: javadoc from forked lifecycle, to prevent recursive 
invocation.
[INFO] No goals needed for project - skipping
[WARNING] DEPRECATED [aggregate]: since 2.5. Use the goals 
javadoc:aggregate and javadoc:test-aggregate instead.
[INFO] [javadoc:javadoc {execution: attach-javadocs}]
[INFO] [assembly:single {execution: default}]
[INFO] Reading assembly descriptor: src/main/assembly/assembly.xml
[INFO] Processing DependencySet (output=lib)
[INFO] Building tar : 

[INFO] Processing DependencySet (output=lib)
[INFO] Building zip: 

[INFO] [install:install {execution: default-install}]
[INFO] Installing 

 to 

[INFO] Installing 

 to 

[INFO] Installing 

 to 

[JENKINS] Archiving 
 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-08-56/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-08-56/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT.jar
[JENKINS] Archiving 

 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-08-56/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT-dist.tar.gz
[JENKINS] Archiving 

 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-08-56/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT-dist.zip
[INFO] 
[INFO] Building CAS Curation Interface
[INFO]task-segment: [install]
[INFO] 
[INFO] [remote-resources:process {execution: default}]
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] Copying 3 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory: 


---
 T E S T S
---
There are no tests to run.

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[JENKINS] Recording test

Build failed in Jenkins: oodt-trunk #313

[Apache Jenkins Server]

See 

Changes:

[bfoster] - Updated CHANGES.txt

--
OODT-399

--
[...truncated 1563 lines...]
[INFO] Surefire report directory: 


---
 T E S T S
---
Running org.apache.oodt.cas.crawl.option.TestBooleanOptions
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[JENKINS] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: 

[INFO] Preparing javadoc:javadoc
[WARNING] Removing: javadoc from forked lifecycle, to prevent recursive 
invocation.
[INFO] No goals needed for project - skipping
[WARNING] DEPRECATED [aggregate]: since 2.5. Use the goals 
javadoc:aggregate and javadoc:test-aggregate instead.
[INFO] [javadoc:javadoc {execution: attach-javadocs}]
[INFO] [assembly:single {execution: default}]
[INFO] Reading assembly descriptor: src/main/assembly/assembly.xml
[INFO] Processing DependencySet (output=lib)
[INFO] Building tar : 

[INFO] Processing DependencySet (output=lib)
[INFO] Building zip: 

[INFO] [install:install {execution: default-install}]
[INFO] Installing 

 to 

[INFO] Installing 

 to 

[INFO] Installing 

 to 

[JENKINS] Archiving 
 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-15-34/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-15-34/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT.jar
[JENKINS] Archiving 

 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-15-34/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT-dist.tar.gz
[JENKINS] Archiving 

 to 
/home/hudson/hudson/jobs/oodt-trunk/modules/org.apache.oodt$cas-crawler/builds/2012-03-01_02-15-34/archive/org.apache.oodt/cas-crawler/0.4-SNAPSHOT/cas-crawler-0.4-SNAPSHOT-dist.zip
[INFO] 
[INFO] Building CAS Curation Interface
[INFO]task-segment: [install]
[INFO] 
[INFO] [remote-resources:process {execution: default}]
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] Copying 3 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory: 


---
 T E S T S
---
There are no tests to run.

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[JENKINS] Recording test results
[INFO] [war:war {execution: default-war}]
[INFO] Packaging webapp

Adding Apache OODT to CodeMaps

[Sheryl John]

Hi all,

I just found out about CodeMaps ( http://www.codemaps.org/) and they've
added Tika.

Shall we add OODT to CodeMaps?

I've asked them what to do to add OODT  and Vineet just replied:

-- Forwarded message --
From: Vineet Sinha 
Date: Wed, Feb 29, 2012 at 9:21 PM
Subject: Re: Adding Apache OODT to CodeMaps
To: shery...@gmail.com
Cc: "Dev @ Architexa" 


Sheryl,
Glad to hear. To add Apache OODT to CodeMaps just login to the site and
click add project. It will get added to the queue.

If you can promise to spend 30 minutes providing some basic information for
the site - I am sure I can convince my team to get it done first thing in
the morning. :-)

Regards,
Vineet
-- 
President & CTO, Architexa - www.architexa.com
Understand & Document Code In Seconds
vin...@architexa.com :: 617.818.0548



On Thu, Mar 1, 2012 at 12:18 AM, Sheryl John  wrote:

> Hi Vineet,
>
> I've just seen CodeMaps and the Tika documentation and looks awesome!
>
> How can we add Apache OODT (http://oodt.apache.org/) to CodeMaps?
>
>
> Thanks!
> --
> Sheryl
>




-- 
-Sheryl

Re: question on push pull

[Cameron Goodale]

Luca,

Good catch.  I was typing faster than my brain was working.  I meant to say
Push Pull instead of Crawler.

Sorry for the confusion.  Chris laid it all out really well.  Guess I was
just a little to excited to answer you and goofed in my haste.

To answer your question, I am not sure where the code is in PUSHPULL (got
it right this time ;) since I haven't had reason to use it yet.

I did check the etc/examples directory in PushPull and I only found file,
ftp, and sftp examples.  If you do sort out the scp and scp -r versions it
would be great to add them to the etc/examples area.

It could be part of the Apache Extras that Chris mentioned:
http://code.google.com/a/apache-extras.org/p/oodt-pushpull-plugins/

Good Luck.



-Cameron

On Wed, Feb 29, 2012 at 10:23 AM, Cinquini, Luca (3880) <
luca.cinqu...@jpl.nasa.gov> wrote:

> Hi Cameron,
>Maybe I am confused, but I was actually asking about the push-pull
> capabilities - does the crawler plug into the push-pull framework ? (sorry
> about my ignorance here). If push-pull supports scp, would you know the
> name of the protocol transfer factory to use - I haven't found one.
> thanks a lot,
> Luca
>
> On Feb 28, 2012, at 8:40 AM, Cameron Goodale wrote:
>
> > Luca,
> >
> > I haven't tried this exact use case within Crawler, but Crawler does
> > support scp and I have used 'scp -r' to recursively download a folder and
> > all content housed within.  I can only imagine ftp has a similar
> recursive
> > option as well.
> >
> > Maybe another more Crawler Savy dev can shine some light on the recursion
> > use case when using Crawler.
> >
> > -Cameron
> >
> > P.S. When we get a final answer let's add this to the Crawler User Guide
> > Wiki too as an example use case.  Glad you found the Crawler Wiki page
> > useful.
> >
> > On Tue, Feb 28, 2012 at 7:01 AM, Cinquini, Luca (3880) <
> > luca.cinqu...@jpl.nasa.gov> wrote:
> >
> >> Hi all,
> >>   I have a quick question concerning the pushpull framework : is
> >> there any way to transfer full directory trees, as opposed to single
> files
> >> ? And which of the currently implemented transfer protocols would allow
> >> that ? I haven't see any examples on that, though I might have missed
> it.
> >>
> >> thanks a lot,
> >> Luca
> >>
> >> P.S.: Cameron, thanks for writing the push-pull user guide - it's great.
> >>
> >>
> >
> >
> > --
> >
> > Sent from a Tin Can attached to a String
>
>


-- 

Sent from a Tin Can attached to a String

Re: Adding Apache OODT to CodeMaps

[Mattmann, Chris A (388J)]

+1 to add OODT to CodeMaps -- looks useful...

Cheers,
Chris

On Feb 29, 2012, at 9:26 PM, Sheryl John wrote:

> Hi all,
> 
> I just found out about CodeMaps ( http://www.codemaps.org/) and they've
> added Tika.
> 
> Shall we add OODT to CodeMaps?
> 
> I've asked them what to do to add OODT  and Vineet just replied:
> 
> -- Forwarded message --
> From: Vineet Sinha 
> Date: Wed, Feb 29, 2012 at 9:21 PM
> Subject: Re: Adding Apache OODT to CodeMaps
> To: shery...@gmail.com
> Cc: "Dev @ Architexa" 
> 
> 
> Sheryl,
> Glad to hear. To add Apache OODT to CodeMaps just login to the site and
> click add project. It will get added to the queue.
> 
> If you can promise to spend 30 minutes providing some basic information for
> the site - I am sure I can convince my team to get it done first thing in
> the morning. :-)
> 
> Regards,
> Vineet
> -- 
> President & CTO, Architexa - www.architexa.com
> Understand & Document Code In Seconds
> vin...@architexa.com :: 617.818.0548
> 
> 
> 
> On Thu, Mar 1, 2012 at 12:18 AM, Sheryl John  wrote:
> 
>> Hi Vineet,
>> 
>> I've just seen CodeMaps and the Tika documentation and looks awesome!
>> 
>> How can we add Apache OODT (http://oodt.apache.org/) to CodeMaps?
>> 
>> 
>> Thanks!
>> --
>> Sheryl
>> 
> 
> 
> 
> 
> -- 
> -Sheryl


++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++

Re: question on push pull

[Brian Foster]

hey Luca,

Ya pushpull can download full directories... do you have a distruct XML file 
for downloading individual files?... paste it into a message on this thread and 
let me know which directory you would like to download and i can do the mods or 
help you through making the mods to the file... in short if you don't specify a 
 element in a  element then all files in that 
directory will be downloaded

-Brian

On Feb 29, 2012, at 10:12 PM, Cameron Goodale  wrote:

> Luca,
> 
> Good catch.  I was typing faster than my brain was working.  I meant to say
> Push Pull instead of Crawler.
> 
> Sorry for the confusion.  Chris laid it all out really well.  Guess I was
> just a little to excited to answer you and goofed in my haste.
> 
> To answer your question, I am not sure where the code is in PUSHPULL (got
> it right this time ;) since I haven't had reason to use it yet.
> 
> I did check the etc/examples directory in PushPull and I only found file,
> ftp, and sftp examples.  If you do sort out the scp and scp -r versions it
> would be great to add them to the etc/examples area.
> 
> It could be part of the Apache Extras that Chris mentioned:
> http://code.google.com/a/apache-extras.org/p/oodt-pushpull-plugins/
> 
> Good Luck.
> 
> 
> 
> -Cameron
> 
> On Wed, Feb 29, 2012 at 10:23 AM, Cinquini, Luca (3880) <
> luca.cinqu...@jpl.nasa.gov> wrote:
> 
>> Hi Cameron,
>>   Maybe I am confused, but I was actually asking about the push-pull
>> capabilities - does the crawler plug into the push-pull framework ? (sorry
>> about my ignorance here). If push-pull supports scp, would you know the
>> name of the protocol transfer factory to use - I haven't found one.
>> thanks a lot,
>> Luca
>> 
>> On Feb 28, 2012, at 8:40 AM, Cameron Goodale wrote:
>> 
>>> Luca,
>>> 
>>> I haven't tried this exact use case within Crawler, but Crawler does
>>> support scp and I have used 'scp -r' to recursively download a folder and
>>> all content housed within.  I can only imagine ftp has a similar
>> recursive
>>> option as well.
>>> 
>>> Maybe another more Crawler Savy dev can shine some light on the recursion
>>> use case when using Crawler.
>>> 
>>> -Cameron
>>> 
>>> P.S. When we get a final answer let's add this to the Crawler User Guide
>>> Wiki too as an example use case.  Glad you found the Crawler Wiki page
>>> useful.
>>> 
>>> On Tue, Feb 28, 2012 at 7:01 AM, Cinquini, Luca (3880) <
>>> luca.cinqu...@jpl.nasa.gov> wrote:
>>> 
 Hi all,
  I have a quick question concerning the pushpull framework : is
 there any way to transfer full directory trees, as opposed to single
>> files
 ? And which of the currently implemented transfer protocols would allow
 that ? I haven't see any examples on that, though I might have missed
>> it.
 
 thanks a lot,
 Luca
 
 P.S.: Cameron, thanks for writing the push-pull user guide - it's great.
 
 
>>> 
>>> 
>>> --
>>> 
>>> Sent from a Tin Can attached to a String
>> 
>> 
> 
> 
> -- 
> 
> Sent from a Tin Can attached to a String