Re: [go-nuts] Re: [urgent] need aguments to justify use of Go for a scientific application

[Jon Conradt]

Would you share the document?  It likely would save time for others when 
they have similar tasks.

Jon

On Wednesday, December 6, 2017 at 4:27:04 PM UTC-8, Christophe Meessen 
wrote:
>
> Thank you all for your help. 
>
> I assembled a sort document with the collected arguments (thanks 
> Sebastien). 
>
> It also contains short code samples. 
>
> Le 06/12/2017 à 16:17, Jesper Louis Andersen a écrit :
>
> The best point in time to switch languages are when one of the following 
> things is happening:
>
> * There is a small experiment on the side of the larger system which can 
> be handled in another language.
> * There is a large project which has failed internally in the business, it 
> is severely behind and nobody thinks it can be solved anyway.
>
> The by far best situation is when the sky is falling. This is when you can 
> propose a solution to the problem rather than a language. The solution then 
> proves itself first and then the language decision follows as part of the 
> solution.
>
> However, for your situation, analyze the requirements of the project 
> first, then make a real effort to analyze a Python solution against a Go 
> solution. If astrophysicists are likely to grab e.g., numpy to do their 
> image processing loop, it is a hard sell. OTOH, if you have to serve a 
> serious amount of image data then the concurrency of Go is a better choice 
> and so on. Quantify "how many images per minute", "how many concurrent 
> requests", "resource usage" and so on.
>
> You could also look for a split solution: process the images in Python, 
> but serve them from Go?
>
> To me, the two biggest weaknesses of Python in (industrial) production 
> settings are:
>
> * Dynamic typing and no other program modularization leads to programs 
> that are really hard to maintain, especially as they grow. A small 
> machine-learning training script is easy enough to maintain as a 
> work-sheet, but I don't particularly like it for serving said 
> machine-learning model.
> * Deployment often requires some hoop-jumping. Getting the right software 
> versions built on one machine and deployed on another is IMO living hell 
> for Python programs. My experience is that build pipelines for Python 
> systems often break in subtle ways. Go produces static binaries which are 
> much easier to handle (Other systems doing it right: Haskell, Erlang, 
> OCaml, Elixir, ...)
>
>
> On Wed, Dec 6, 2017 at 2:34 PM Christophe Meessen  > wrote:
>
>> Thanks far all the answers. Sebastien Binet is a colleague and he is 
>> indeed brilliant. 
>> pachyderm looks powerful, but also an overkill for my need. It might 
>> scare my colleagues away. 
>>
>> Le 06/12/2017 à 12:34, Henrik Johansson a écrit :
>>
>> I have a vague memory of +Rob Pike  tweeting something 
>> about astronomy or perhaps an observatory a few months ago.
>> Perhaps there was no programming involved but if so I imagine Go is safe 
>> bet.
>>
>> But building pipelines using something like Pachyderm would allow for a 
>> very polyglot "use the tool that fits in each part" approach.
>>
>>
>> ons 6 dec. 2017 kl 11:43 skrev Volker Dobler > >:
>>
>>> I know about https://go-hep.org probably Sebastien can elaborate more 
>>> if and how it is used at CERN.
>>>
>>> V.
>>>
>>>
>>> On Wednesday, 6 December 2017 10:56:01 UTC+1, Christophe Meessen wrote: 

 Hello, 

 I'm a computer scientist in charge of developing an image processing 
 pipeline for telescope images. 
 It will also have a web server and DB connection.

 The project is going through reviews by external experts, and the 
 problem I'm facing is that my proposal to use Go is about to be rejected. 

 The main opposing arguments are 
 - everybody uses python in astrophysics
 - it is very easy to find someone who knows python
 - risk that I, sole Go programmer, might become unavailable

 I would have the same arguments if I was project leader and unfamiliar 
 with Go. 

 The counter arguments I found so far are that
 - Go is simpler and safer than Python
 - I learned Go in a week-end

 The problem is that they don't convince people who don't know Go, are 
 not experienced software developers, and don't want to do the due 
 diligence. 
 It's the usual inertia to change.  

 What other arguments could I use ?

 Do you know other significant scientific experiments that have adopted 
 Go ? 



 I have found this github project. 
 https://github.com/indigo-astronomy/indigo
 INDI is a well known Python Observatory Control System. 
 INDIGO is its translation into Go. 

 I have also found SciPipe https://github.com/scipipe.
 It is a Go pipeline framework used in scientific applications. 


 -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To 

[go-nuts] Re: [urgent] need aguments to justify use of Go for a scientific application

[as]

You are responsible for all problems related to the project after proposing 
a language. Caveat emptor.

- Go has a backwards compatibility promise. Go didn't change it's APIs and 
syntax after releasing a 1.0 version of the software.
- Go has formatting standards and a powerful standard library
- Go has code refactoring tools
- Go is a new language with decades of architectural and engineering work
- Authors worke(d) Bell Labs and Google, considered subject matter experts 
in their field
- Go is simple, which is integral to large projects and their 
maintainability
- Go is not a language associated with academia (it's more practical and 
less theoretical)
- There is a chance you will be paying someone to rewrite or port the 
Python version into a Go version later* 

*I speak from experience as someone who does this for C#, Python, C++ --> 
Go. Your mileage will vary on the veracity of this statement because this 
observation is anecdotal and not statistical.




On Wednesday, December 6, 2017 at 1:56:01 AM UTC-8, Christophe Meessen 
wrote:
>
> Hello, 
>
> I'm a computer scientist in charge of developing an image processing 
> pipeline for telescope images. 
> It will also have a web server and DB connection.
>
> The project is going through reviews by external experts, and the problem 
> I'm facing is that my proposal to use Go is about to be rejected. 
>
> The main opposing arguments are 
> - everybody uses python in astrophysics
> - it is very easy to find someone who knows python
> - risk that I, sole Go programmer, might become unavailable
>
> I would have the same arguments if I was project leader and unfamiliar 
> with Go. 
>
> The counter arguments I found so far are that
> - Go is simpler and safer than Python
> - I learned Go in a week-end
>
> The problem is that they don't convince people who don't know Go, are not 
> experienced software developers, and don't want to do the due diligence. 
> It's the usual inertia to change.  
>
> What other arguments could I use ?
>
> Do you know other significant scientific experiments that have adopted Go 
> ? 
>
>
>
> I have found this github project. 
> https://github.com/indigo-astronomy/indigo
> INDI is a well known Python Observatory Control System. 
> INDIGO is its translation into Go. 
>
> I have also found SciPipe https://github.com/scipipe.
> It is a Go pipeline framework used in scientific applications. 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: [urgent] need aguments to justify use of Go for a scientific application

[Christophe Meessen]

Thank you all for your help.

I assembled a sort document with the collected arguments (thanks 
Sebastien).


It also contains short code samples.


Le 06/12/2017 à 16:17, Jesper Louis Andersen a écrit :
The best point in time to switch languages are when one of the 
following things is happening:


* There is a small experiment on the side of the larger system which 
can be handled in another language.
* There is a large project which has failed internally in the 
business, it is severely behind and nobody thinks it can be solved anyway.


The by far best situation is when the sky is falling. This is when you 
can propose a solution to the problem rather than a language. The 
solution then proves itself first and then the language decision 
follows as part of the solution.


However, for your situation, analyze the requirements of the project 
first, then make a real effort to analyze a Python solution against a 
Go solution. If astrophysicists are likely to grab e.g., numpy to do 
their image processing loop, it is a hard sell. OTOH, if you have to 
serve a serious amount of image data then the concurrency of Go is a 
better choice and so on. Quantify "how many images per minute", "how 
many concurrent requests", "resource usage" and so on.


You could also look for a split solution: process the images in 
Python, but serve them from Go?


To me, the two biggest weaknesses of Python in (industrial) production 
settings are:


* Dynamic typing and no other program modularization leads to programs 
that are really hard to maintain, especially as they grow. A small 
machine-learning training script is easy enough to maintain as a 
work-sheet, but I don't particularly like it for serving said 
machine-learning model.
* Deployment often requires some hoop-jumping. Getting the right 
software versions built on one machine and deployed on another is IMO 
living hell for Python programs. My experience is that build pipelines 
for Python systems often break in subtle ways. Go produces static 
binaries which are much easier to handle (Other systems doing it 
right: Haskell, Erlang, OCaml, Elixir, ...)



On Wed, Dec 6, 2017 at 2:34 PM Christophe Meessen 
> 
wrote:


Thanks far all the answers. Sebastien Binet is a colleague and he
is indeed brilliant.

pachyderm looks powerful, but also an overkill for my need. It
might scare my colleagues away.

Le 06/12/2017 à 12:34, Henrik Johansson a écrit :

I have a vague memory of +Rob Pike  tweeting
something about astronomy or perhaps an observatory a few months ago.
Perhaps there was no programming involved but if so I imagine Go
is safe bet.

But building pipelines using something like Pachyderm would allow
for a very polyglot "use the tool that fits in each part" approach.


ons 6 dec. 2017 kl 11:43 skrev Volker Dobler
>:

I know about https://go-hep.org probably Sebastien can
elaborate more
if and how it is used at CERN.

V.


On Wednesday, 6 December 2017 10:56:01 UTC+1, Christophe
Meessen wrote:

Hello,

I'm a computer scientist in charge of developing an image
processing pipeline for telescope images.
It will also have a web server and DB connection.

The project is going through reviews by external experts,
and the problem I'm facing is that my proposal to use Go
is about to be rejected.

The main opposing arguments are
- everybody uses python in astrophysics
- it is very easy to find someone who knows python
- risk that I, sole Go programmer, might become unavailable

I would have the same arguments if I was project leader
and unfamiliar with Go.

The counter arguments I found so far are that
- Go is simpler and safer than Python
- I learned Go in a week-end

The problem is that they don't convince people who don't
know Go, are not experienced software developers, and
don't want to do the due diligence.
It's the usual inertia to change.

What other arguments could I use ?

Do you know other significant scientific experiments that
have adopted Go ?



I have found this github project.
https://github.com/indigo-astronomy/indigo
INDI is a well known Python Observatory Control System.
INDIGO is its translation into Go.

I have also found SciPipe https://github.com/scipipe.
It is a Go pipeline framework used in scientific
applications.


-- 
You received this message because you are subscribed to the

Google Groups "golang-nuts" group.
 

Re: [go-nuts] Re: [urgent] need aguments to justify use of Go for a scientific application

[Jesper Louis Andersen]

The best point in time to switch languages are when one of the following
things is happening:

* There is a small experiment on the side of the larger system which can be
handled in another language.
* There is a large project which has failed internally in the business, it
is severely behind and nobody thinks it can be solved anyway.

The by far best situation is when the sky is falling. This is when you can
propose a solution to the problem rather than a language. The solution then
proves itself first and then the language decision follows as part of the
solution.

However, for your situation, analyze the requirements of the project first,
then make a real effort to analyze a Python solution against a Go solution.
If astrophysicists are likely to grab e.g., numpy to do their image
processing loop, it is a hard sell. OTOH, if you have to serve a serious
amount of image data then the concurrency of Go is a better choice and so
on. Quantify "how many images per minute", "how many concurrent requests",
"resource usage" and so on.

You could also look for a split solution: process the images in Python, but
serve them from Go?

To me, the two biggest weaknesses of Python in (industrial) production
settings are:

* Dynamic typing and no other program modularization leads to programs that
are really hard to maintain, especially as they grow. A small
machine-learning training script is easy enough to maintain as a
work-sheet, but I don't particularly like it for serving said
machine-learning model.
* Deployment often requires some hoop-jumping. Getting the right software
versions built on one machine and deployed on another is IMO living hell
for Python programs. My experience is that build pipelines for Python
systems often break in subtle ways. Go produces static binaries which are
much easier to handle (Other systems doing it right: Haskell, Erlang,
OCaml, Elixir, ...)


On Wed, Dec 6, 2017 at 2:34 PM Christophe Meessen <
christophe.mees...@gmail.com> wrote:

> Thanks far all the answers. Sebastien Binet is a colleague and he is
> indeed brilliant.
> pachyderm looks powerful, but also an overkill for my need. It might scare
> my colleagues away.
>
> Le 06/12/2017 à 12:34, Henrik Johansson a écrit :
>
> I have a vague memory of +Rob Pike  tweeting something
> about astronomy or perhaps an observatory a few months ago.
> Perhaps there was no programming involved but if so I imagine Go is safe
> bet.
>
> But building pipelines using something like Pachyderm would allow for a
> very polyglot "use the tool that fits in each part" approach.
>
>
> ons 6 dec. 2017 kl 11:43 skrev Volker Dobler :
>
>> I know about https://go-hep.org probably Sebastien can elaborate more
>> if and how it is used at CERN.
>>
>> V.
>>
>>
>> On Wednesday, 6 December 2017 10:56:01 UTC+1, Christophe Meessen wrote:
>>>
>>> Hello,
>>>
>>> I'm a computer scientist in charge of developing an image processing
>>> pipeline for telescope images.
>>> It will also have a web server and DB connection.
>>>
>>> The project is going through reviews by external experts, and the
>>> problem I'm facing is that my proposal to use Go is about to be rejected.
>>>
>>> The main opposing arguments are
>>> - everybody uses python in astrophysics
>>> - it is very easy to find someone who knows python
>>> - risk that I, sole Go programmer, might become unavailable
>>>
>>> I would have the same arguments if I was project leader and unfamiliar
>>> with Go.
>>>
>>> The counter arguments I found so far are that
>>> - Go is simpler and safer than Python
>>> - I learned Go in a week-end
>>>
>>> The problem is that they don't convince people who don't know Go, are
>>> not experienced software developers, and don't want to do the due
>>> diligence.
>>> It's the usual inertia to change.
>>>
>>> What other arguments could I use ?
>>>
>>> Do you know other significant scientific experiments that have adopted
>>> Go ?
>>>
>>>
>>>
>>> I have found this github project.
>>> https://github.com/indigo-astronomy/indigo
>>> INDI is a well known Python Observatory Control System.
>>> INDIGO is its translation into Go.
>>>
>>> I have also found SciPipe https://github.com/scipipe.
>>> It is a Go pipeline framework used in scientific applications.
>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
>
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/BH4tOlu6df8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
>
>
> For more options, visit 

Re: [go-nuts] Re: [urgent] need aguments to justify use of Go for a scientific application

[Christophe Meessen]

Thanks far all the answers. Sebastien Binet is a colleague and he is 
indeed brilliant.


pachyderm looks powerful, but also an overkill for my need. It might 
scare my colleagues away.


Le 06/12/2017 à 12:34, Henrik Johansson a écrit :
I have a vague memory of +Rob Pike  tweeting 
something about astronomy or perhaps an observatory a few months ago.
Perhaps there was no programming involved but if so I imagine Go is 
safe bet.


But building pipelines using something like Pachyderm would allow for 
a very polyglot "use the tool that fits in each part" approach.



ons 6 dec. 2017 kl 11:43 skrev Volker Dobler 
>:


I know about https://go-hep.org probably Sebastien can elaborate more
if and how it is used at CERN.

V.


On Wednesday, 6 December 2017 10:56:01 UTC+1, Christophe Meessen
wrote:

Hello,

I'm a computer scientist in charge of developing an image
processing pipeline for telescope images.
It will also have a web server and DB connection.

The project is going through reviews by external experts, and
the problem I'm facing is that my proposal to use Go is about
to be rejected.

The main opposing arguments are
- everybody uses python in astrophysics
- it is very easy to find someone who knows python
- risk that I, sole Go programmer, might become unavailable

I would have the same arguments if I was project leader and
unfamiliar with Go.

The counter arguments I found so far are that
- Go is simpler and safer than Python
- I learned Go in a week-end

The problem is that they don't convince people who don't know
Go, are not experienced software developers, and don't want to
do the due diligence.
It's the usual inertia to change.

What other arguments could I use ?

Do you know other significant scientific experiments that have
adopted Go ?



I have found this github project.
https://github.com/indigo-astronomy/indigo
INDI is a well known Python Observatory Control System.
INDIGO is its translation into Go.

I have also found SciPipe https://github.com/scipipe.
It is a Go pipeline framework used in scientific applications.


-- 
You received this message because you are subscribed to the Google

Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to golang-nuts+unsubscr...@googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the 
Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit 
https://groups.google.com/d/topic/golang-nuts/BH4tOlu6df8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 
golang-nuts+unsubscr...@googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: [urgent] need aguments to justify use of Go for a scientific application

[Henrik Johansson]

I have a vague memory of +Rob Pike  tweeting something about
astronomy or perhaps an observatory a few months ago.
Perhaps there was no programming involved but if so I imagine Go is safe
bet.

But building pipelines using something like Pachyderm would allow for a
very polyglot "use the tool that fits in each part" approach.


ons 6 dec. 2017 kl 11:43 skrev Volker Dobler :

> I know about https://go-hep.org probably Sebastien can elaborate more
> if and how it is used at CERN.
>
> V.
>
>
> On Wednesday, 6 December 2017 10:56:01 UTC+1, Christophe Meessen wrote:
>>
>> Hello,
>>
>> I'm a computer scientist in charge of developing an image processing
>> pipeline for telescope images.
>> It will also have a web server and DB connection.
>>
>> The project is going through reviews by external experts, and the problem
>> I'm facing is that my proposal to use Go is about to be rejected.
>>
>> The main opposing arguments are
>> - everybody uses python in astrophysics
>> - it is very easy to find someone who knows python
>> - risk that I, sole Go programmer, might become unavailable
>>
>> I would have the same arguments if I was project leader and unfamiliar
>> with Go.
>>
>> The counter arguments I found so far are that
>> - Go is simpler and safer than Python
>> - I learned Go in a week-end
>>
>> The problem is that they don't convince people who don't know Go, are not
>> experienced software developers, and don't want to do the due diligence.
>> It's the usual inertia to change.
>>
>> What other arguments could I use ?
>>
>> Do you know other significant scientific experiments that have adopted Go
>> ?
>>
>>
>>
>> I have found this github project.
>> https://github.com/indigo-astronomy/indigo
>> INDI is a well known Python Observatory Control System.
>> INDIGO is its translation into Go.
>>
>> I have also found SciPipe https://github.com/scipipe.
>> It is a Go pipeline framework used in scientific applications.
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [urgent] need aguments to justify use of Go for a scientific application

[Volker Dobler]

I know about https://go-hep.org probably Sebastien can elaborate more
if and how it is used at CERN.

V.

On Wednesday, 6 December 2017 10:56:01 UTC+1, Christophe Meessen wrote:
>
> Hello, 
>
> I'm a computer scientist in charge of developing an image processing 
> pipeline for telescope images. 
> It will also have a web server and DB connection.
>
> The project is going through reviews by external experts, and the problem 
> I'm facing is that my proposal to use Go is about to be rejected. 
>
> The main opposing arguments are 
> - everybody uses python in astrophysics
> - it is very easy to find someone who knows python
> - risk that I, sole Go programmer, might become unavailable
>
> I would have the same arguments if I was project leader and unfamiliar 
> with Go. 
>
> The counter arguments I found so far are that
> - Go is simpler and safer than Python
> - I learned Go in a week-end
>
> The problem is that they don't convince people who don't know Go, are not 
> experienced software developers, and don't want to do the due diligence. 
> It's the usual inertia to change.  
>
> What other arguments could I use ?
>
> Do you know other significant scientific experiments that have adopted Go 
> ? 
>
>
>
> I have found this github project. 
> https://github.com/indigo-astronomy/indigo
> INDI is a well known Python Observatory Control System. 
> INDIGO is its translation into Go. 
>
> I have also found SciPipe https://github.com/scipipe.
> It is a Go pipeline framework used in scientific applications. 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.