On Mon, Jun 30, 2008 at 5:40 AM, Malte Marquarding <
[EMAIL PROTECTED]> wrote:
> Hi Simon,
>
> your mock up of operations looks pretty much like what I have myself.
> We are in the toddler stage of our project and looking into possible
> high-level architecture solutions. Traditionally astronomy is based on
> common frameworks ( e.g. CORBA). I tend to think that for scalability and
> to
> be able to re-use existing software in the various aspects of running the
> observations, a more heterogeneous approach as in SCA would be the better
> solution. So each component can select what implementation it suits best.
> Anyway I am looking into SOA/SCA in general and as we are interested in
> open
> source solutions naturally came across tuscany. I am interested in a bit
> more detailed discussion if you are available offline, because I really
> like
> to see us going forward with this. I can send you a whitepaper on our
> poject
> if you want.
>
> As to what is run on the supercomputer - as one of the solutions to
> workflow
> and load balancing we are looking into System S and doing some evaluation (
> I see you work for IBM so you might have heard of it ;-) ). So the native
> c++ components might not be necessary long term as System S also has a java
> API.
>
> Cheers,
> Malte.
>
>
> On Fri, Jun 27, 2008 at 11:18 PM, Simon Laws <[EMAIL PROTECTED]>
> wrote:
>
> > On Fri, Jun 27, 2008 at 1:56 AM, Malte Marquarding <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Hi Raymond,
> > >
> > > The system we envisage is roughly as follows.
> > >
> > > We are running a set of radio telescopes at remote site. The high level
> > > exposure of the various parts of system should be via service
> components.
> > > The actual implementation is specific to the problem.We have thl
> > telescope
> > > control (of the hardware). The data generated form these telescopes has
> > to
> > > be processed on a supercomputer ( hence implementation in c++), through
> a
> > > set of services like Calibration, Imaging, Analysis. It also has some
> > > control queue (running the end-to-end process, referencing the various
> > > services), archiving, logging, user access (virtual observatory)
> > components
> > > etc.
> > >
> > > I am investigating tuscany for this.
> > >
> > > Another unrelated problem I have is that I can't see (with SDOs and
> SCA
> > )
> > > how to handle the transfer of the data. The data output of the c++
> > services
> > > is tens to hundreds of Terabytes. I was thinking of having a DataMoving
> > > service encapsulating something like GridFTP. Has anyone got a
> suggestion
> > > of
> > > how to handle this in an SCA context.
> > >
> > > I can give more details if necessary.
> > >
> > > Cheers,
> > > Malte.
> > >
> > > On Fri, Jun 27, 2008 at 4:16 AM, Raymond Feng <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Can you describe the use cases you have in mind? It will help us
> better
> > > > understand what you want to achieve.
> > > >
> > > >
> > >
> >
> > Hi Malte
> >
> > The bindings we have implemented to date are intended to operate in the
> > typical SOA environment where you pass data to a component and ask it to
> do
> > something. We tend to talk in terms of XML documents which will be fine
> for
> > the control messages you need but not suitable for the telescope data
> > itself.
> >
> > To try and understand the subtleties of your scenario I'm going to take
> the
> > components you suggested and invent some operations that we might expect
> to
> > find there...
> >
> > TelescopeControl
> > PerformObservation(ObservatonParameters, ObservationId)
> > // I assume you give the telescope a job to do, i.e point at the sky
> and
> > record the results against a given ID
> > // and then callback when the task is complete
> > DataTransfer
> > Transfer(FromLocation, ToLocation)
> > // just manages the task of moving large datasets across the network.
> As
> > you say gridFtp could be a candidate here.
> > Calibration
> > Run(ObservationId)
> > GetDatasetLocation(ID)
> > Imaging
> > Run(ObservationId)
> > GetDatasetLocation(ID)
> > Analysis
> > Run(ObservationId)
> > GetDatasetLocation(ID)
> > Archive
> > GetDatasetLocation(ID)
> > Logging
> > // are you logging control messages here
> > Coordination
> > DoSometing()
> > // coordinate the activities of the application, for example, it might
> do
> > TelescopeControl.PerformObservation(someParames, "reading1")
> > // when task is complete
> > fromLocation = TelescopeControl.GetDatasetLocation("reading1")
> > toLocation = Calibration.GetDatasetLocation("reading1")
> > DataTransfer.Transfer(fromLocation, toLocation)
> > Calibration.run("reading1")
> >
> > etc.
> >
> > The Calibration, Analysis, Imaging components are a bit tricky to
> > visualize.
> > Are they closely related or stand alone? Do they always have to run in
> > sequence in the same order? What sort of infrastructure do they rely on?
> > For
> > example, you mention a supercomputer so are we talking MPI collectives
> and
> > Condor like schedulers. In which case an SCA component such as
> "Calibrate"
> > may just wrap the task of creating and submitting JSDL to the scheduler
> > rather than representing the Calibration code itself. You may even
> resort
> > to a more generic "ComputeEngine" component that allows you to
> dynamically
> > configure jobs to be run.
> >
> > Personally I would like Tuscany to be able to slot right in here so that
> > the
> > analytical components could be supported in HPC environments with
> > appropriate SCA implementation types, bindings and integration with the
> > underlying HPC and grid infrastructure. We are not really there yet. I've
> > done some work on a LoadBalancer demo that shows Tuscany running in a
> > Tomcat
> > cluster but scenarios like yours can really help us all think what
> features
> > would be appropriate.
> >
> > Regards
> >
> > Simon
> >
>
Hi Malte
Sure, feel free to get me at my gmail address or on google talk.
Regards
Simon
