Re: [Discuss] Serverless scientific computing (function as a service)

2017-06-13 Thread Strong, Dena L 
I have anecdotes rather than data, but around here it's getting to be more 
popular as more people try it out. 

Folks do need to take care to look through their grant terms to see if there's 
any specific language about region restrictions or cloud computing 
restrictions. (I have a friend who keeps burning through SSDs on an 
under-her-desk server who would love to have Amazon's resources available but 
her grant specifically prohibits cloud computing.)

One local example: Here at the University of Illinois we had a team spend 3 
months analyzing a set of data, only to discover at the end of the 3 months 
that there was a software version mismatch that made that run incompatible with 
the rest of their results. 

They came to talk to my department's cloud and virtualization team about what 
could be done, and discovered that there was a way to re-run their work in 3 
days at a really cost effective price point on Amazon Web Services with our 
team's help. (Our lead thinks he might be able to get that originally 3 month 
process down into 1 day with a little more optimization.) Our AWS team often 
serves as an intermediary to help guide researchers through navigating the 
networking, security, and optimization issues.

-Dena Strong, Technology Services, University of Illinois

-Original Message-
From: Discuss [mailto:discuss-boun...@lists.software-carpentry.org] On Behalf 
Of Peter Steinbach
Sent: Tuesday, June 13, 2017 6:38 AM
To: discuss@lists.software-carpentry.org
Subject: Re: [Discuss] Serverless scientific computing (function as a service)

Dear both,

as a side note (and my apologies for digressing), I was wondering how popular 
cloud computing for data processing at scale in an academic context is in the 
US or elsewhere?

Here in Europe, many universities run their own HPC centers where people can 
sign up to process larger amounts of data or do larger simulations or whatnot 
... mostly people here are concerned about efficiency (data connnections into 
the cloud are typically poor, VM overhead is
considerable) and security/confidentiality when putting scientific workflows 
into the cloud.
What is your take on this?

Best,
Peter


PS. I love the "serverless" metaphor. Get's rid of all the problems of 
computers. ;)

On 06/12/2017 06:02 PM, Marianne Corvellec wrote:
> Hi Justin,
>
> Thank you so much for the quick reply!
>
> I'm going to give this new package a try.
>
> Best,
> Marianne
>
> On Fri, Jun 9, 2017 at 11:20 AM, Justin Kitzes  wrote:
>> Hi Marianne,
>>
>> PyWren by Eric Jonas sounds like it's pretty similar to what you're 
>> looking for -
>>
>> http://pywren.io/
>>
>> It's a relatively new package that's still in active development, but Eric 
>> is very interested in expanding it (and has some support from the riselab at 
>> UC Berkeley to do so). I know that he's also actively looking for use cases, 
>> so I'd definitely suggest getting in touch with him if you're interested.
>>
>> Best,
>>
>> Justin
>>
>> --
>> Justin Kitzes
>> Energy and Resources Group
>> Berkeley Institute for Data Science
>> University of California, Berkeley
>>
>>> On Jun 9, 2017, at 6:51 AM, Marianne Corvellec 
>>>  wrote:
>>>
>>> Dear community,
>>>
>>> I'm curious as to whether some of you might have worked on or used 
>>> solutions such as AWS Lambda in the context of your scientific 
>>> research.
>>>
>>> If so, have you documented it in a blog post that you could share?
>>> Thanks in advance!
>>>
>>> Without even considering workflows or full-fledged projects, 
>>> wouldn't we want to be able to make a standard API call to, say, fit 
>>> a polynomial to some data?  Is anyone aware of any effort in this 
>>> direction?
>>>
>>> A friend of mine just drew my attention to this general issue, which 
>>> touches on open science and reproducible research...  In the 
>>> meantime, I'll encourage him to join this mailing list!
>>>
>>> Thank you,
>>> Marianne
>>> ___
>>> Discuss mailing list
>>> Discuss@lists.software-carpentry.org
>>> http://lists.software-carpentry.org/listinfo/discuss
>>
> ___
> Discuss mailing list
> Discuss@lists.software-carpentry.org
> http://lists.software-carpentry.org/listinfo/discuss
>
___
Discuss mailing list
Discuss@lists.software-carpentry.org
http://lists.software-carpentry.org/listinfo/discuss
___
Discuss mailing list
Discuss@lists.software-carpentry.org
http://lists.software-carpentry.org/listinfo/discuss

Re: [Discuss] Serverless scientific computing (function as a service)

2017-06-13 Thread alexsa...@gmail.com 
Hi Peter,

I wouldnt be able to use such services with clinical data. It's totally not
an option for me.
Although I've seen some talks and the performance seems quite competitive
since scalability is easy. It's true that uploading a big quantity of data
can take a considerable time and bandwith, some labs use the weekends for
data uploading. One problem may be to convince University fund managers to
pay for external computing services when they already provide HPC services.

My five cents...

On Tue, 13 Jun 2017, 13:38 Peter Steinbach,  wrote:

> Dear both,
>
> as a side note (and my apologies for digressing), I was wondering how
> popular cloud computing for data processing at scale in an academic
> context is in the US or elsewhere?
>
> Here in Europe, many universities run their own HPC centers where people
> can sign up to process larger amounts of data or do larger simulations
> or whatnot ... mostly people here are concerned about efficiency (data
> connnections into the cloud are typically poor, VM overhead is
> considerable) and security/confidentiality when putting scientific
> workflows into the cloud.
> What is your take on this?
>
> Best,
> Peter
>
>
> PS. I love the "serverless" metaphor. Get's rid of all the problems of
> computers. ;)
>
> On 06/12/2017 06:02 PM, Marianne Corvellec wrote:
> > Hi Justin,
> >
> > Thank you so much for the quick reply!
> >
> > I'm going to give this new package a try.
> >
> > Best,
> > Marianne
> >
> > On Fri, Jun 9, 2017 at 11:20 AM, Justin Kitzes 
> wrote:
> >> Hi Marianne,
> >>
> >> PyWren by Eric Jonas sounds like it's pretty similar to what you're
> looking for -
> >>
> >> http://pywren.io/
> >>
> >> It's a relatively new package that's still in active development, but
> Eric is very interested in expanding it (and has some support from the
> riselab at UC Berkeley to do so). I know that he's also actively looking
> for use cases, so I'd definitely suggest getting in touch with him if
> you're interested.
> >>
> >> Best,
> >>
> >> Justin
> >>
> >> --
> >> Justin Kitzes
> >> Energy and Resources Group
> >> Berkeley Institute for Data Science
> >> University of California, Berkeley
> >>
> >>> On Jun 9, 2017, at 6:51 AM, Marianne Corvellec <
> marianne.corvel...@gmail.com> wrote:
> >>>
> >>> Dear community,
> >>>
> >>> I'm curious as to whether some of you might have worked on or used
> >>> solutions such as AWS Lambda in the context of your scientific
> >>> research.
> >>>
> >>> If so, have you documented it in a blog post that you could share?
> >>> Thanks in advance!
> >>>
> >>> Without even considering workflows or full-fledged projects, wouldn't
> >>> we want to be able to make a standard API call to, say, fit a
> >>> polynomial to some data?  Is anyone aware of any effort in this
> >>> direction?
> >>>
> >>> A friend of mine just drew my attention to this general issue, which
> >>> touches on open science and reproducible research...  In the meantime,
> >>> I'll encourage him to join this mailing list!
> >>>
> >>> Thank you,
> >>> Marianne
> >>> ___
> >>> Discuss mailing list
> >>> Discuss@lists.software-carpentry.org
> >>> http://lists.software-carpentry.org/listinfo/discuss
> >>
> > ___
> > Discuss mailing list
> > Discuss@lists.software-carpentry.org
> > http://lists.software-carpentry.org/listinfo/discuss
> >
> ___
> Discuss mailing list
> Discuss@lists.software-carpentry.org
> http://lists.software-carpentry.org/listinfo/discuss

-- 

Sent from my phone, sorry for brevity or typos.
___
Discuss mailing list
Discuss@lists.software-carpentry.org
http://lists.software-carpentry.org/listinfo/discuss

Re: [Discuss] Serverless scientific computing (function as a service)

2017-06-13 Thread Peter Steinbach 

Dear both,

as a side note (and my apologies for digressing), I was wondering how 
popular cloud computing for data processing at scale in an academic 
context is in the US or elsewhere?


Here in Europe, many universities run their own HPC centers where people 
can sign up to process larger amounts of data or do larger simulations 
or whatnot ... mostly people here are concerned about efficiency (data 
connnections into the cloud are typically poor, VM overhead is 
considerable) and security/confidentiality when putting scientific 
workflows into the cloud.

What is your take on this?

Best,
Peter


PS. I love the "serverless" metaphor. Get's rid of all the problems of 
computers. ;)


On 06/12/2017 06:02 PM, Marianne Corvellec wrote:

Hi Justin,

Thank you so much for the quick reply!

I'm going to give this new package a try.

Best,
Marianne

On Fri, Jun 9, 2017 at 11:20 AM, Justin Kitzes  wrote:

Hi Marianne,

PyWren by Eric Jonas sounds like it's pretty similar to what you're looking for 
-

http://pywren.io/

It's a relatively new package that's still in active development, but Eric is 
very interested in expanding it (and has some support from the riselab at UC 
Berkeley to do so). I know that he's also actively looking for use cases, so 
I'd definitely suggest getting in touch with him if you're interested.

Best,

Justin

--
Justin Kitzes
Energy and Resources Group
Berkeley Institute for Data Science
University of California, Berkeley


On Jun 9, 2017, at 6:51 AM, Marianne Corvellec  
wrote:

Dear community,

I'm curious as to whether some of you might have worked on or used
solutions such as AWS Lambda in the context of your scientific
research.

If so, have you documented it in a blog post that you could share?
Thanks in advance!

Without even considering workflows or full-fledged projects, wouldn't
we want to be able to make a standard API call to, say, fit a
polynomial to some data?  Is anyone aware of any effort in this
direction?

A friend of mine just drew my attention to this general issue, which
touches on open science and reproducible research...  In the meantime,
I'll encourage him to join this mailing list!

Thank you,
Marianne
___
Discuss mailing list
Discuss@lists.software-carpentry.org
http://lists.software-carpentry.org/listinfo/discuss



___
Discuss mailing list
Discuss@lists.software-carpentry.org
http://lists.software-carpentry.org/listinfo/discuss


___
Discuss mailing list
Discuss@lists.software-carpentry.org
http://lists.software-carpentry.org/listinfo/discuss