Re: [go-nuts] [urgent] need aguments to justify use of Go for a scientific application

[Robin Bartholdson]

Well, the top 1 argument is that go is the language of the future ;) 
https://erikbern.com/2017/03/15/the-eigenvector-of-why-we-moved-from-language-x-to-language-y.html
 


Some reason-to-believes that go is a good fit for data science and your use 
case:

• There is a mature and well-maintained scientific computation library: 
https://github.com/gonum/gonum  
• Data pipeline processing infrastructure is really coming of age. Some 
examples include: Apache Beam (https://github.com/apache/beam/pull/4200 
), Pachyderm 
(https://github.com/pachyderm/pachyderm 
), Fission Workflows 
(https://github.com/fission/fission-workflows/ 
) (the two latter projects are 
based on Kubernetes which happens to be written in — go).
• Other people do serious physics in go: https://go-hep.org/ 
. (Sebastien Binet can most likely tell you more :)

Also, with the C bindings it’s perfectly possible to utilise all the Fortran 
and C libs that you don’t have time to rewrite right now.

Join #data-science on the gophers slack if you want to continue the discussion.

-Robin

> On 6 Dec 2017, at 10:56, Christophe Meessen  
> wrote:
> 
> Hello, 
> 
> I'm a computer scientist in charge of developing an image processing pipeline 
> for telescope images. 
> It will also have a web server and DB connection.
> 
> The project is going through reviews by external experts, and the problem I'm 
> facing is that my proposal to use Go is about to be rejected. 
> 
> The main opposing arguments are 
> - everybody uses python in astrophysics
> - it is very easy to find someone who knows python
> - risk that I, sole Go programmer, might become unavailable
> 
> I would have the same arguments if I was project leader and unfamiliar with 
> Go. 
> 
> The counter arguments I found so far are that
> - Go is simpler and safer than Python
> - I learned Go in a week-end
> 
> The problem is that they don't convince people who don't know Go, are not 
> experienced software developers, and don't want to do the due diligence. 
> It's the usual inertia to change.  
> 
> What other arguments could I use ?
> 
> Do you know other significant scientific experiments that have adopted Go ? 
> 
> 
> 
> I have found this github project. https://github.com/indigo-astronomy/indigo
> INDI is a well known Python Observatory Control System. 
> INDIGO is its translation into Go. 
> 
> I have also found SciPipe https://github.com/scipipe.
> It is a Go pipeline framework used in scientific applications. 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [urgent] need aguments to justify use of Go for a scientific application

[Sebastien Binet]

On Wed, Dec 6, 2017 at 2:15 PM, Sebastien Binet  wrote:

> Christophe,
>
> On Wed, Dec 6, 2017 at 10:56 AM, Christophe Meessen <
> christophe.mees...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm a computer scientist in charge of developing an image processing
>> pipeline for telescope images.
>> It will also have a web server and DB connection.
>>
>> The project is going through reviews by external experts, and the problem
>> I'm facing is that my proposal to use Go is about to be rejected.
>>
>> The main opposing arguments are
>> - everybody uses python in astrophysics
>> - it is very easy to find someone who knows python
>> - risk that I, sole Go programmer, might become unavailable
>>
>> I would have the same arguments if I was project leader and unfamiliar
>> with Go.
>>
>> The counter arguments I found so far are that
>> - Go is simpler and safer than Python
>> - I learned Go in a week-end
>>
>> The problem is that they don't convince people who don't know Go, are not
>> experienced software developers, and don't want to do the due diligence.
>> It's the usual inertia to change.
>>
>> What other arguments could I use ?
>>
>> Do you know other significant scientific experiments that have adopted Go
>> ?
>>
>
> in physics, I know 3 experiments that have parts of their software written
> in Go:
> - SoLiD (they wrote their data acquisition system in Go, as well as some
> environmental sensors monitoring (on raspi3)).
>  The DAQ code isn't public but there is an IEEE presentation about it
> (Nick Ryder)
>  The monitoring code is there: https://github.com/sbin
> et-solid/solid-mon-rpi
>
> - LSST: one of the subcomponents testing system has been written in Go
> (slow control):
>  https://github.com/go-lsst/fcs-lpc-motor-ctl
>
> - EIC: there is some R&D code for the Electron-Ion collider that is being
> written in Go (and can generate code for C++, Python and Java)
>  https://github.com/decibelCooper/eicio
>

ah! and I almost forgot about CMS@LHC: they're in the middle of rewritting
their data placement + data transfer system from PERL to Go.
https://github.com/vkuznet/transfer2go
https://github.com/vkuznet/transfer2go/blob/master/docs/Transfer2go_RD_201709.pdf
https://github.com/vkuznet/dbs2go

it seems Valentin is also playing with TensorFlow and Go for CMS:
https://github.com/vkuznet/TFaaS

-s

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [urgent] need aguments to justify use of Go for a scientific application

[Sebastien Binet]

Christophe,

On Wed, Dec 6, 2017 at 10:56 AM, Christophe Meessen <
christophe.mees...@gmail.com> wrote:

> Hello,
>
> I'm a computer scientist in charge of developing an image processing
> pipeline for telescope images.
> It will also have a web server and DB connection.
>
> The project is going through reviews by external experts, and the problem
> I'm facing is that my proposal to use Go is about to be rejected.
>
> The main opposing arguments are
> - everybody uses python in astrophysics
> - it is very easy to find someone who knows python
> - risk that I, sole Go programmer, might become unavailable
>
> I would have the same arguments if I was project leader and unfamiliar
> with Go.
>
> The counter arguments I found so far are that
> - Go is simpler and safer than Python
> - I learned Go in a week-end
>
> The problem is that they don't convince people who don't know Go, are not
> experienced software developers, and don't want to do the due diligence.
> It's the usual inertia to change.
>
> What other arguments could I use ?
>
> Do you know other significant scientific experiments that have adopted Go
> ?
>

in physics, I know 3 experiments that have parts of their software written
in Go:
- SoLiD (they wrote their data acquisition system in Go, as well as some
environmental sensors monitoring (on raspi3)).
 The DAQ code isn't public but there is an IEEE presentation about it (Nick
Ryder)
 The monitoring code is there: https://github.com/sbinet-solid/solid-mon-rpi

- LSST: one of the subcomponents testing system has been written in Go
(slow control):
 https://github.com/go-lsst/fcs-lpc-motor-ctl

- EIC: there is some R&D code for the Electron-Ion collider that is being
written in Go (and can generate code for C++, Python and Java)
 https://github.com/decibelCooper/eicio

In my lab, there is also a team working with transfering HEP technologies
to the medical environment, that is using Go for their DAQ+monitoring.
(and babysteps for the analysis too)
There's an IEEE publication that has just been accepted (Busato et al.),
not sure it's public yet.
An early version of the code is there: https://github.com/
ebusato/analysis-go

You may also want to talk to Fabio Hernandez (from LSST/CC-IN2P3) about his
experience with Go.
He may have some interesting "reviewers worthy" insights.

There are also a few interesting research papers about Go:
 https://github.com/golang/go/wiki/ResearchPapers

many are CS oriented, but a few Gonum friends have also a few papers
*using* Go:
 - http://arc.aiaa.org/doi/abs/10.2514/6.2015-1287
 - https://arxiv.org/abs/1706.02419

you also probably know about my work:
 https://go-hep.org
 https://github.com/astrogo

but there is also a great library for biology (some of the "students" I had
2 weeks ago at a Go tutorial in Marseille, knew about it):
 - https://github.com/biogo/biogo
 - http://dx.doi.org/10.21105/joss.00167
 - http://biorxiv.org/content/early/2014/05/12/005033

You could also perhaps mention SpaceX (which is -apparently- using Go for
its rocket monitoring system.)

and I am in the middle of writing an aXiv preprint wrt why I think HEP
should migrate en masse to Go.
it's only in my head right now, but the seeds for that article are
contained in my ACAT-2017 presentation:
 -
http://talks.godoc.org/github.com/go-hep/talks/2017/2017-08-24-go-hep-acat/talk.slide
and the (not yet approved) accompanying paper:
- https://github.com/go-hep/talks/tree/master/2017/2017-08-24-go-hep-acat

finally, here is what I got with a Master-1 Physics student who knew a bit
of Python (and had no previous exposure to Go), in a 3 months internship:
- https://github.com/lsst-lpc/hubble-fit
a little cosmo analysis replicating results from a previous python-based
analysis involving reading a few FITS files containing a few 2220x2220
matrices that are inverted and such, and then a minimization is applied to
extract Hubble-related parameters.
it was nice to see the student learning Go in a few weeks and producing a
version that was OK, w/o much help from my end.
the final version is faster than the python (with numpy and C++ under the
hood) one, in single-threaded mode.
and ~2 faster with 2 cores.

ok, I don't know whether this is significant enough.

>From a "close to the ground" angle, typical arguments for a Go .vs. Python
are:
- static type system so: refactoring is less brittle, smaller amount of
errors discovered at runtime, better runtime performances
- even if the language is compiled and statically typed, the development
speed isn't hindered too much thanks to the type inference system and the
fast compiler (so edit-compile-run dev cycles are *comparable* to python)
- concurrency programming. no need for the "multiprocessing" and
"threading" python modules. concurrency is builtin and the language was
designed with it from the start.
- the language is simpler than python (I always show the "for i in
range(x): print(i)" snippet which involves knowing about exceptions,
coroutines and special 

Re: [go-nuts] [urgent] need aguments to justify use of Go for a scientific application

[Peter Mogensen]

On 2017-12-06 10:56, Christophe Meessen wrote:
> The main opposing arguments are
> - everybody uses python in astrophysics
> - it is very easy to find someone who knows python
> - risk that I, sole Go programmer, might become unavailable

And those are important arguments - especially if the "everybody uses"
includes that there's important libraries only available for python.

Though not for scientific experiments, I've find the best argument for
Go over Python is better concurrency with lesser memory footprint.
I have a python HTTP server where a single request can make a worker
process size increase from 65Mb to 4.3Gb and the only way to make the OS
reclaim that memory from the worker is to kill it off and spawn a new.
That's the kind of moments you think "If only this was written in Go".

But that might be totally irrelevant for a science application. Usually
"if it ain't broken, don't fix it", applies.

/Peter

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [urgent] need aguments to justify use of Go for a scientific application

['Axel Wagner' via golang-nuts]

Not to be a buzzkill, but… Maybe not having good arguments for using Go in
this case should be taken as a signal that Go isn't a good fit? As much as
I think Go would be useful for scientific applications, if we want to see
that happens, we should work on making Go better for them, not on making up
arguments for why it already is.

That being said, this seems like a classical "$boss don't want to bet
$project on $technology, how do I convince them?" question. First step:
Accept that $boss is reasonable to not put new (to them) technology in the
business-critical path. All of the arguments you mentioned seem 100%
convincing to me - even *though* I know Go much better than Python. If I'd
start an astrophysics project today, I'd probably use Python for the
sciency-computational parts, not Go. But you could try to find non-critical
applications where Go already *is* a good fit, to prove its worth and
increase your peers exposure to the language. Or you could propose to use
Go in pieces of the puzzle, where it works very well. So, e.g. Python is
great to code the computations and everything, but Go would be better
suited to write the server and interface with the DB.

On Wed, Dec 6, 2017 at 11:08 AM, Tamás Gulácsi  wrote:

> Python is terrible to maintain (lack of static typing or even compile-time
> argument count (!) check).
>
> Write down the interfaces between the pipeline steps and nodes, making
> each module replaceable.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.