[go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

This is a belated addition to my notes on the Go translation of 
reposurgeon. I'll be adding it to the revised version I post to golang-dev.

One extremely positive thing I must say before closing.  Translation from 
Python, which is a dreadful language to try to do concurrency in due to its 
global interpreter lock, really brought home to me how unobtrusively 
brilliant the Go implementation of Communicating Sequential Processes is.  
The primitives are right and the integration with the rest of the language 
is wonderfully seamless.  The Go port of reposurgeon got some very large 
speedups at an incremental-complexity cost that I found to be astonishingly 
low.  I am impressed both by the power of the CSP part of the design and 
the extreme simplicity and non-fussiness of the interface it presents.  I 
hope it will become a model for how concurrency is managed in future 
languages.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/abc51fb9-0f98-47e6-91a3-d8950953bfdd%40googlegroups.com.

[go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

On Thursday, January 30, 2020 at 2:05:17 AM UTC-5, Liam wrote:
>
> I'd suggest re-posting this on golang-dev; I almost missed it as I rarely 
> read -nuts.
>

That's a good idea.  I think what I'll do is revise it lightly in view of 
some of the feedback I've gotten here and post it there.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/152ab11e-2a6d-4ec7-83a4-4feccf66eca7%40googlegroups.com.

[go-nuts] Golang on UnixWare

[Eric Raymond]

I've received a feeler about a most interesting reposurgeon-related 
consulting engagement from the company that currently owns the historical 
Unix sources. I'm not going to go into the gory details in a public forum 
as they might be considered company confidential, but it gives me reason 
to  ask the following:

What is the procedure for porting Golang to a reasonably well-behaved, 
standards-conformant Unix variant?  UnixWare in this case. You can assume 
the target platform hosts GCC.  It it possible to use gccgo to port the 
reference compiler? If so, is this procedure documented anywhere?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/21750718-83b3-4c30-b3df-85a083961eb7%40googlegroups.com.

Re: [go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

On Monday, January 27, 2020 at 9:51:43 AM UTC-5, Philip Boampong wrote:
>
> I assumed the ugliness is about the recover logic being exposed in the 
> catch signature and in the handler function. If you change catch to 
> take the exception handler as a callback you can hide the recover 
> logic without language changes. But maybe there's something you want 
> to do that I can't see from the example.
>

You're right. I will experiment.
 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d365f80b-c41e-4d44-947c-e4b9b9a3e546%40googlegroups.com.

[go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

On Sunday, January 26, 2020 at 7:14:50 PM UTC-5, pboam...@gmail.com wrote:
>
> log.Fatal and os.Exit have the same problem. They are not "terminating 
> statements", so if you want them at the bottom of a function with result 
> parameters you have to add a panic("unreachable").
>

Excellent point.  But contemplating being able to declare library functions 
terminating - as opposed to making it a property the compiler deduces from 
knowing how panic works - opens up a bigger can of worms...

But I think it's very rare not to have one of the "success" paths at the 
> end; in 8 years it happened to me like a couple of times. Do you really 
> expect to have throw() at the bottom of functions?
>

Absolutely.  Consider a parser in which your handler function for a given 
token or subtree consists of  a bunch of if/then returns, and not matching 
one of them means you should throw upwards to an error handler.
 

> > 2. A recover() call is no longer required to be within the lexical frame 
> of a defer().
>
> Would it be less ugly like this? (with recover in the catch func.)
>
> | defer catch("recoverable", func(err error) {
> | fmt.Println("Recover:", err)
> | })
>
> Maybe I'm being dim. but I don'r understand your counter-question. Can you 
> unpack it a bit? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9aac4872-b90a-4c95-9220-04b7c8acacb8%40googlegroups.com.

Re: [go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

On Sunday, January 26, 2020 at 9:06:47 PM UTC-5, Robert Engels wrote:
>
> I think trying to write Python in Go is problematic. 
>

Of course it is. If I were starting a Go program from scratch I certainly 
wouldn't try to code as though the language were Python.

The real burden of my experience report is that *after I got past* the 
initial and somewhat herky-jerky translation, some of the expressiveness 
problems I tripped over *continued* to be expressiveness problems in 
idiomatic Go.

Perhaps this would be more obvious if you had seen some of the gripes I had 
in early drafts that I removed because I realized they were artifacts of 
Pythonic thinking. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d810807f-a1ab-4fb6-a303-439ec456c7b5%40googlegroups.com.

[go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

I spotted a typo:

On Saturday, January 25, 2020 at 3:46:19 AM UTC-5, Eric Raymond wrote:
>
> The Go translation of reposurgeon is better - more maintainable - code
> than the Python original, not just faster.  And this is because I
> rewrote or refactored as I went; as I've explained, I tried very hard
> to avoid that. It's that Go's minimalistic approach actually...works.
>

Missing "not": "And this is not because" 

The strategy of writing as literal as possible a translation of the Python 
first, 
at the cost of generating Go code that was initially clunky  and 
unidiomatic,
worked quite well.  It actually took effort and discipline to refrain from 
trying
to improve the code as it passed through translation, but I am very glad I 
expended that effort.  It kept the translation process sane and 
controllable,

I should note that a prerequisite for a translation like this is an 
excellent test 
suite.  At 22KLOC, reposurgeon now has 52 unit tests, 177 round-trip tests, 
218 end-to-end functional tests, and a miscellany of special tests that add
up to a total of 502 reporting items.  Only around 20 of these were added
during the translation itself.  All these tests are run by continuous 
integration
on every commit.

That translation phase was completed In November 2019 when the Go code
first passed the full test suite. The two months since has been sufficient 
to
polish the literalistic mock-Python it was at that time into what is now, I 
believe,
pretty clean idiomatic Go.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/760a6b1d-d3e2-4eeb-a074-19ea568b2449%40googlegroups.com.

[go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

On Saturday, January 25, 2020 at 5:43:24 AM UTC-5, Brian Candler wrote:
>
> 1. When the issue of keyword arguments has come up before, usually someone 
> suggests passing a struct as the function argument.  Did you try this?  It 
> might be worth mentioning in your analysis, if only to give an example of 
> why it wasn't a good match.
>

I'm picky about when I'll do that.  I view putting a bunch of items in a 
structure as a promise to people reading the code that the structure isn't 
just a vacuous shim but captures an interesting chunk in the ontology of 
the program.  "They're all arguments to this function" isn't interesting 
enough. 

2. I notice some potential overlap between a couple of features you 
> mention.  The first is map(f,x) for mapping a slice of arbitrary type 
> (incidentally it couldn't be called "map" for obvious reasons)  The second 
> is using generator functions as ranges.
>
> It occurs to me that if you could initialize a slice from a generator 
> function, you'd have another way to achieve your map. The generator 
> function could also do filtering, making it more like a comprehension.
>

Perhaps. But map-over=sequence seems to me like a simpler primitive idea 
than "comprehension" and thus preferable.
 

> 3. A generator function would have to maintain its own thread of execution 
> independent from the recipient - much like a goroutine.  So this might end 
> up looking very similar to a function which stuffs values down a channel, 
> which is option 3 in the linked article 
> .
>
> The problem with garbage collection needs to be dealt with, which at the 
> simplest could be that the receiver closes the channel when done. Today 
> this would cause the sender to panic, so that would need to be dealt with - 
> perhaps some sort of "soft close".
>

And that's the insight that led be to the extension I proposed. I asked 
myself what the most natural way to pass out a soft close might be.
 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/04b5ee2c-40d0-4275-a53a-eaad9828c849%40googlegroups.com.

Re: [go-nuts] Re: Experience report on a large Python-to-Go translation

[Eric Raymond]

On Saturday, January 25, 2020 at 10:04:19 AM UTC-5, Robert Engels wrote:
>
> Whenever I see a type switch it screams to me “use an interface and 
> restructure. “
>

You may be right.  On the other hand, I think I've already gone far enough 
down the  interface road to have collected most of the gains from that 
tactic.

Interfaces are fine when you have a bunch of disparate types with similar 
external interfaces and are at a stage of processing where you can 
disregard the differences.  You end up in type-switch land when the point 
is that they have *dissimilar* external interfaces but you have to cope 
anyway.

Not ideal, but sometimes necessary.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/88b8c01c-b70e-41dc-a1e7-fb412b831f50%40googlegroups.com.

[go-nuts] Experience report on a large Python-to-Go translation

[Eric Raymond]

Most of the remainder of this post is asciidoc source, because that's easy 
to quote.  If you want to look at a nicely rendered version, see

https://gitlab.com/esr/reposurgeon/blob/master/GoNotes.adoc

= Notes on the Go translation of Reposurgeon =
version 1.8, 2020-01-25

This is an experience report on a Python-to-Go translation of a
program with significant complexity, written in attempted conformance
with the Go community's practice for grounding language enhancement
requests not in it-would-be-nice-to-have abstractions but rather in a
description of real-world problems.

Reposurgeon is a program for editing version-control histories and
interconverting them among version-control systems. I spent much of
2019 moving the reposurgeon codebase from Python to Go because the
Python implementation was too slow to be useful on on really large
repositories.  The workflow it is designed to support is rapid
iterative improvement of conversion recipes by a human operator;
long test cycles disrupt this by making experimentation painful.

Subversion-to-git conversion of the Gnu Compiler Collection history,
at over 280K commits (over 1.6M individual change actions), was the
straw that broke the camel's back. Using PyPy with every optimization
on semi-custom hardware tuned for this job still yielded test
conversion times of over 9 hours, which is death on the
recipe-debugging cycle.  A test translation of the auxiliary
repocutter tool suggested that we could expect up to a 40x speedup,
which was in line with published Python vs. Go comparative benchmarks.

The problem directed the choice of Go, not the other way around.  I
seriously considered OCaml or a compiled Lisp as alternatives.  I
concluded that in either case the semantic gap between Python and
the target language was so large that translation would be
impractical. Only Go offered me any practical hope.

I did examine two automated tools for Python to Go translation, but
rejected them because I judged the generated Go code would have been a
maintainability disaster.  Thus, translation by hand.  Though at about
22% in I did write https://gitlab.com/esr/pytogo[a fast, crude,
incomplete Python-to-Go source translator] to assist the process.

The man barrier to translation was that, while at 14KLOC of Python
reposurgeon was not especially large, the code is very *dense*.  It's
a DSL that's a structure editor for attributed DAGs - algorithmically
complex, bristling with graph theory, FSMs, parsing, tricky data
structures, two robot harnesses driving other tools, and three
different operator-composition algebras.  It became 21KLOC of Go.

The algorithmic density of reposurgeon is such that it would be a
challenge to the expressiveness of any language it were implemented
in.  It makes a good test of the relative expressiveness of Python and
Go, and an effective way to audit for places where moving from Python
to Go hinders concision and readability.

The skillset I approached this problem with is: Go novice, Python and
C expert; old Unix hand; Lisp hacker of even more ancient vintage;
ex-mathematician with strength in graph theory, group theory and
combinatorics; lots of experience as a systems programmer, lots of
exposure to odd languages, and lots of domain knowledge about
version-control systems.  Adjust bias compensators accordingly.

== Expected problems that weren't ==

Semantic distance. In general, the translation gap between Python and
Go is narrower than I initially expected, especially considering the
dynamic-vs-static type-system difference.  On reflection, I think it
turns out that GC and the existence of maps as first-class types do
more to narrow that gap than the static/dynamic divergence in type
systems does to widen it.

Polymorphic lists.  The principal data structure of a repository's
representation in reposurgeon is a list of events - data structures
representing modification operations on sets of files.  The list is
necessarily polymorphic because (for example) a change commit and a
tag creation are different kinds of things.  Translating this to
static typing using interfaces proved less arduous than I had feared,
though the process revealed a documentation issue and a problem
with absence of sum types; I will return to both points.

Operator overloading.  Good style in Python, but surprisingly easy to
relinquish in Go.  I went in thinking that they'd be on my want list
for Go before I was done, but no - not even at reposurgeon's
complexity scale.

Generics.  Yes, these would have made translation easier, but the main
impact turned out to be that I had to write my own set-of-int and
set-of-string classes.  That was a surprisingly light hit.  What I
really missed was generic map-function-over-slice, which could be
handled by adding a much narrower feature.

The positive part of my summation is that hand-translation of Python
to Go even at this scale and complexity is not horrible.  It's not
*easy*, exactly, but quite doable.  It is however tim

Re: [go-nuts] Re: C++ 11 to Golang convertor

[Eric Raymond]

On Friday, January 4, 2019 at 2:41:19 PM UTC-5, robert engels wrote:
>
> I still think it would be a nearly impossible task given the C code in the 
> wild - outside of threading, the common usage of  ‘unions’  - there is no 
> way I know of to map these to a simple Go struct, or even several - you 
> need something like protobufs. So even if you could convert, you’d end up 
> with code that is probably harder to maintain/verify than the original C 
> code. 
>

I've worked on a lot of this old infrastructure code  myself, which is how 
I know that use of unions and threading simply isn't that common in it. You 
have to bear in mind that a lot of it was originally written in the last 
century, before the standards people won their war.   So for practical 
purpose ANSI pthreads didn't yet exist, though  in the NTP case (the one 
I've been elbow-deep in recently) some threading stuff was bolted on later 
to avoid stalling on DNS lookups.

As for security checking each line..yes, in an ideal world, but not 
necessary for the translation to be a worthy improvement,  Sure, we know 
the C code is leaky, but given the verification properties of both 
languages, a line-by-line translation can't make you worse off unless 
there's some huge undetected hole in the Go libraries.
That's a bet I'm willing to make.

You haven't noticed what I think is a larger problem - preprocessor 
conditionals.  That's a headache.  Still puzzling on that one.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: C++ 11 to Golang convertor

[Eric Raymond]

On Thursday, January 3, 2019 at 12:11:06 PM UTC-5, Jake Montgomery wrote:
>
> I would note that any tool that ports from C++, or even C, to Go is going 
> to produce ugly, unmaintainable, and non-idiomatic code, at best.
>

These are two different cases.  I agree that graceful C++ to Go 
transpilation is effectively impossible.

On the other hand, I believe graceful, comment-preserving C to idiomatic-Go 
transpilation is almost possible.  By 'almost' I mean that the tool would 
pass through a small enough percentage of untranslated residuals for 
corrections to be around a 5% job for a human expert. 

I've had a lot of incentive to think about this because my concerns center 
around vast masses of C infrastructure code in critical network services 
like NTP, DNS, etc.  The security and reliability consequences of unsafe 
code in that swamp are serious and it needs to be drained.  Transpilation 
to golang is, I think, the first realistic hope we've  had of doing that 
without a prohibitively high labor input. 

By possible I do not mean easy.  I've scoped the job and done a design 
sketch. I think my qualifications for writing such a transpiler are 
exceptionally good, but it would nevertheless take me a minimum of two 
years of hard work to get there.   I have put put some feelers for 
funding;  if I get to choose my next major project after NTPsec, this would 
be it.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Example of setting a struct field value via reflection requested

[Eric Raymond]

On Monday, November 5, 2018 at 6:18:27 PM UTC-5, kortschak wrote:
>
> https://play.golang.org/p/EtXlOCR1fDY 
>

Thanks, I was able to adapt that into working code.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Example of setting a struct field value via reflection requested

[Eric Raymond]

I need an equivalent of Python setattr() - that is, set a struct member by 
string name - but the only example I've found on line (SetField 
in oleiade/reflections on GitHub) panics and dies due to what looks like a 
spurious Elem() call.  I thought I could see how to fix it, but the 
resulting code has no effect,.

Would someone please point me at a working example, or post one here?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Failed iterator proposals?

[Eric Raymond]

Thank you, that was a helpful response.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Failed iterator proposals?

[Eric Raymond]

I have a  real-world case where I need Python style iterators for 
performance.  It's not good to eager-evaluate an event-list 259K items when 
you know that in the typical case you're only going to select two or three 
orders of  magnitude fewer of them to be passed to a work function.  (The 
list is of commit events in a version-control repository; the selector 
might be something like "commited within 24 hours of time t").

There's an obvious, clumsy way to do this by having a factory pass back a 
stateful iterator object:

for iterator := IteratorFactory(selector); iterator.Continue();iterator.Next
(){
   DoSomethingWith(iterator.Value())
}


What one would really like to be able to do, though, is much more concise:

for i, v := range iteratorFunction() {
 DoSomethingWith(i, v)
}

Before I post an RFE, are there rejected iterator proposals I should read 
so I don't waste anyone's time on stale ideas?


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [ANN] pygor: yet another Python to Go regurgitator

[Eric Raymond]

On Saturday, October 20, 2018 at 12:17:16 AM UTC-4, Raffaele Sena wrote:
>
> Inspired by ESR pytogo (and tired of writing python code), I went the 
> complete opposite way and came up with pygor:
>

Bravo.

I am amused by the name - puts me in mind of Discworld Igors.  And if my 
sadly departed friend Terry Pratchett were still alive so I could explain 
it to him I have no doubt he would be amused too.  He would have made a 
fine upstanding hacker if his life had gone a bit differently.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Avoiding overloading

[Eric Raymond]

I think Ian Denhardt is right to argue that fear of operator overloading is 
driving people in the generics debate to complicated, ugly workarounds that 
should not be.

I myself do not  actually want overloading as a surface feature of the 
language.  In my original "implements" proposal, I accepted it as an 
implication of using operators to define type classes, because I thought 
(and still think) it is the simplest possible way to declare contracts.

I think the simplicity of having "implements" as the only contract 
declaration is too important a win to mortgage to a feature many - 
including myself - are not actually friendly to.

Therefore, as the author of the "implements" proposal, I am declaring 
neutrality on whether an "implements" clause should declare an overload at 
all!

That is, there is a possible future in which "implements <" on type T  does 
not  make a < b legal on non-primitive type T, but is solely a typeclass 
declaration associating  T with all other types that implement <; 
semantically, a Sortable contract.

I would be OK with that future,  It keeps the important part of 
"implements" - simplest way to declare contracts - while tossing out the 
part that (not without reason) raise some hackles.

My only caution here is on interface psychology.  If we're going to go this 
route, users are, to be honest. probably going to bitch that not taking the 
tiny additional step to overloading is perverse.  We'll need a good strong 
public argument to counter that.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Proposed changes to the Go draft generics design in the light of feedback received

[Eric Raymond]

On Friday, October 19, 2018 at 1:48:36 PM UTC-4, alanfo wrote:
>
> In the light of all the feedback there's been, I've put together a 
> proposal which sticks closely to the original design and only changes what 
> most people consider needs to be changed in some way. Some recent ideas 
> which seemed plausible but which I felt had little chance of adoption have 
> been rejected.
>

Surface changes that don't tackle the fundamental unwieldiness and 
overcomplexity of the  contracts extension.

A good honest effort, Alan, but (with due respect to the people who put in 
cleverness and effort on the original overcomplication, I don't mean to 
insult them) this is lipstick on a pig.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Wednesday, October 17, 2018 at 11:12:19 AM UTC-4, Ian Denhardt wrote:
>
> My instinct here is to be conservative; we've got a boatload of use 
> cases for < and ==, a few obvious ones for +, *, / etc, and a long-tail 
> for operators after that. Eric's proposal makes it trivial to add more 
> operators to the list of allowed ones, but it's always harder to go the 
> other way. So my inclination would be to implement this for operators 
> like `<` that are kinda screaming at us, and then build some experience 
> with that. 
>

I concur fully with this.

The spirit of my proposal is to do something transparent,  minimal, and 
lighter-weight than any of the heavyweight contracts proposals, then find 
out by experience what more we need, if anything. I am directly opposing 
approaches that incur large increases in the syntactic and semantic 
complexity of the language up front at the cost of setting us all up for 
disappointment when they  turn out to be inadequate or have lurking boojums.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 4:52:54 PM UTC-4, Ian Denhardt wrote:
>
> The trouble is that (in the case where you compile using the implements 
> <, rather than throwing an error) the type signature no longer tells you 
> what the constraints actually are; this information is only in the body 
> of the function, so it's easy to accidentally change it when changing 
> something that you think is just implementation detail. 
>

Fair enough. But is there any way to have a contract system of *any* kind 
that doesn't have the identical problem? 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 4:41:53 PM UTC-4, Dave MacFarlane wrote:
>
> Now I'm confused about what problem regarding generics you're trying 
> to solve with operator overloading. If you can't assume anything about 
> the meaning of an operator other than "it returns a bool", how could a 
> programmer use them in a generic contract? Or were you suggesting 
> operator overloading *instead of* contracts that exist in the current 
> proposal? (Even in that case, how would a programmer use them in a 
> generic function without their meaning being well defined?) 
>

Reviewing this thread, I find your confusion quite reasonable.  So let me 
lay out my philosophical assumptions more carefully.

An "implements <" clause would have two different kinds of meaning.  One is 
mechanical and the only kind the compiler knows about. It declares a 
rewrite rule that means "when you encounter a < between to instances of 
type T, rewrite it as a call to the the implements method of T if one 
exists, otherwise throw a compile-time error".
 
The other kind of meaning is unknown to the compiler but shared by humans 
and library functions.  It is an implied semantics something like "this is 
a predicate testing the most natural ordering of whatever type implements 
it".

The utility of tying contracts to overloading is that we have a 
well-defined set of expectations about the *semantics* of  various 
operators based on how they behave on primitive types. But the compiler 
does not and need not know about these expectations; they're expressed as 
conventions like "don't play stupid and overload < with a method that isn't 
anticommutative".  Or "don't overload + with a noncommutative operation".

The motivation for my proposal is that I think 90% of what people want to 
want a contract system for can be expressed by conventions about what kinds 
of actual method  you bind under a particular operator.  I agree that "a << 
os.Stdout" would be an ugly mistake, but it's not the compiler's job to 
enforce what can really only be made normative by a healthy culture 
surrounding the language.

(Also, does this mean the compiler wouldn't be able to make any 
> optimization assumptions regarding whether an operation is 
> commutative/associative/distributive?) 
>

How does trying to make  such assumptions gain anything when the operator 
compiles to a method call?  Serious question; I'm not seeing it.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 4:17:36 PM UTC-4, Ian Denhardt wrote:
>
> What is this intended to buy us? Is this as Ian T suggests in lieu of 
> any kind of contract system? 


Oh , no.  You *need* a contract system (or at least my implements clause) 
so you can know if a < b has a meaning when a and b are non-primitive types.
 

> I outlined a similar thing in another thread, where you use a lexical 
> convention to distinguish type variables from concrete types, so you 
> don't need an explicit parameter list.


I will admit that this is my fallback position if deducing any-type 
arguments from the unboundedness of the type identifier is too brash for 
this crew,  That is, some kind of lexical marking of generic arguments 
seems to me greatly preferable to the Taylor/Griesemer syntax, which (with 
due respect to those two gentlemen)   I think is just plain ugly.

 

>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 4:03:16 PM UTC-4, Ian Lance Taylor wrote:
>
> It's a feasible approach.  But unless I misunderstand, testing 
> arguments at the call site means that you've discarded contracts. 
>

I don't understand that claim. A contract (in my plan, an "implements" 
clause) is how you decide whether binding a template formal to a concrete 
type at the callsite makes sense .  For example, if the generic function 
has the signature "func foo( a T, b T)" and contains the expression " a < 
b" and you bind the unspecified type T to a a concrete one based on the 
arguments passed to foo at callsite, you either compile "a < b" using the 
"implements <" method of the concrete type or throw an error.
 

> This means that a change in the generic function, which happens to 
> introduce a new use of the parameter, can cause a failure at the call 
> site, which may be far away. 


That doesn't seem to me to entail more risk than *any* change to the 
generic function would.
 

>  It's 
> possible that that is the best way 
> forward.  But it's problematic, for the reasons discussed at the "type 
> contracts" section of the design draft.  This is less of a problem in 
> a dynamic language like LISP than it is in a nominally 
> compile-time-type-safe language like Go. 
>

Can you exhibit any scenario in which, under my plan, any validity check 
has to wait until runtime?

If it helps, think of an "implements <" method as a text-rewrite rule that 
is evaluated before code generation.  Anywhere you see "a < b" and the 
operand type has an "implements <" method, the expression is effectively 
replaced by a method call.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

(Sorry about the post fragment that got away from me.)

On Tuesday, October 16, 2018 at 3:33:58 PM UTC-4, Ian Denhardt wrote:
>
>
> The alternative though is having to implement several different methods 
> with obvious implementations in terms of one another, and make sure 
> they're consistent.


If the alternative is requiring programmers to remember pre-cooked 
relationships between different operators that may or may not be 
appropriate for what he or she is trying to model, then I embrace this 
"problem".

It's up to the programmer to implement things like commutativity.  I don't 
think the language should pre-specify whether < is a partial or total 
ordering.  Any choice of this kind the language makes will inevitably turn 
out wrong a significant percentage of the time.  For proper orthogonality 
we should not specify as much as we can about each operator, but as 
*little* as we can and still have readable code.

I would say the contract for < is just "takes operands of the same 
unspecified type; returns a bool".  The fact that this will usually be used 
to express an order for sorting, and that it is anticommutative, is none of 
the compiler's business.  It's a convention that libraries will observe and 
programmers know about, but that's a different matter.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 3:33:58 PM UTC-4, Ian Denhardt wrote:
>
>
> The partial vs. total order thing brings up another issue that I think 
> needs to be kept in mind: what are the contracts for the various 
> operators? 
>

I don't undetstand rte q

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 1:23:20 PM UTC-4, Dave MacFarlane wrote:
>
> Why is the implements keyword a better solution than magic names? 
>
 
Because you get syntactic uniformity with primitive types without claiming 
namespace you don't need.  It's more minimalist.

Would there be any rules defined for relationships between operators? 
> For instance, if the "<" operator is defined does "<=" come for free 
> or does it require a separate overloading? (If so, what about ">"?) 
>

Good  heavens, no - < doesn't imply a definition of >.  That kind of spooky 
side effect would be extremely un-Go-like.

Besides, you don't want it. There are lots of uses for partial orderings.

I would say >= is defined if and only if == and >= are defined and cannot 
be separately overloaded.  But I'd be open to argument on that point.

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 12:24:59 PM UTC-4, alanfo wrote:
>
> With regard to your reply to Ian, I'm afraid I can't agree with you that 
> the type parameters shouldn't be specified 'up front'. That's just too 
> 'dynamic' for me and not very go-like. I'd imagine it would give the 
> compiler all sorts of problems and anyone reading the code would wonder 
> whether T, Node, Edge etc. were concrete types defined elsewhere in the 
> package or type parameters of a generic function.
>

So now we get to the philosophical issue.  Why and when do I *care* 
whether  I'm looking at a concrete type or type parameter? 

I maintain that normally I don't - it only matters to me what contract the 
type is expected to satisfy, and that should be clear from looking at its 
callsites.  If it isn't clear, that's a problem that heavier notation won't 
solve.

I've already thought of a compilation strategy for this. It's not a new 
problem; I had to think about a closely parallel case in Lisp around 1980.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 9:44:23 AM UTC-4, Ian Lance Taylor wrote:
>
> It's not clear to me what other aspects of various generic proposals 
> you are picking up here.


The answer is "none of them".  That wasn't my goal, because all the 
previous generics proposals I have seen struck me as overcomplicated and 
un-Go-like. What I was attempting to demonstrate was something minimal and 
clean that would handle your smoke test and the common cases that people 
who say they want generics seem to want to cover, such as your smoke test - 
min and max - and reordering a slice of elements of a type with a sortable 
contract.

 How are you imagining that we would write 
> the function that merges two channels? 


Exactly as you did, except I would omit the type parameter.  It's 
unnecessary weight.

New rule: A function on variables of undefined formal-argument types 
compiles to a template (not a textual template - we don't want identifier 
capture).  At each callsite of the function, the undefined types in the 
formals are bound to the types of the arguments in the call. If every 
operation in the resulting code is defined, the code compiles as if the 
generic function had been written with the actual argument types at the 
callsite.  If not, the compiler throws an error on each undefined 
operation. 

(Of course, operation definitions forward through "implements" clauses.)
 
What could be simpler than that?  No additional syntax, and it's easy to 
model in your head   There's precedent for it working; I'm an old LISPer 
and in that world what I just said can be summarized as "a function with a 
formal of unspecified type acts like  a hygienic macro".

Before we tackle the graph example, let's examine this new compilation rule 
for traps. Call me crazy, but I think we just got parametric polymorphism 
with zero additional complexity load on the rest of the language.  Yes, 
compiling those templates in such a way that they can be instantiated at 
callsites from the compiled version will be interesting, but there's an 
obvious trick where you substitute pass-by-pointer-reference-to-the-stack 
for pass-by-value in order to make every reference to an argument have 
constant width regardless of what its concrete type is.  You're probably 
already doing something similar for interfaces.

As the meme goes, "Prove me wrong".  If nobody does, we now have both 
parametric polymorphism and a complementary contract system at the cost of 
one (1) new keyword and (1) interaction with the rest of the design - 
overloading pointer types has to be forbidden in order not to break ==. . 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Generics with adaptors

[Eric Raymond]

On Tuesday, October 16, 2018 at 6:34:10 AM UTC-4, Patrick Smith wrote:
>
> Yet another generics discussion at 
> https://gist.github.com/pat42smith/ccf021193971f6de6fdb229d68215302
>
> This one looks at what programmers would be able to do if very basic 
> generics were added to Go, without contracts. Generic functions may not use 
> methods or operators of their type parameters.
>
> The answer is quite a lot, actually. The code would be verbose, but not 
> impossibly so.
>

Ingenious, but  I think it fails Ian Lance Taylor's smoke test. 

I also dislike the verbosity.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 6:33:26 AM UTC-4, alanfo wrote:
>
> However, I think it's important to learn the lessons of the past and not 
> follow languages such as C++ or Scala where you can overload virtually 
> anything or even make up your own operators which can result in confusing 
> or even write-only code.
>

Strongly agreed.   I too would be OK with only allowing a simple subset of 
operators to be overloaded - excluding && and ||, notably; I wrote up that 
possibility as a way of exploring the edges of the proposal. not as  a 
serious amendment.

On the other hand, I think it's important as a matter of good interface 
psychology not to be arbitrary.  That is, having introduced "implements", 
the question "why can't I overload operator foo" should always have an 
answer that is principled and specific to a given operator.  For example, 
we might disallow / because the relationship between its operand types and 
result type is not obvious by inspection.

That criterion might also be a principled way to knock out short-circuit 
use of && and || on non-booleans.  On the other hand, I don't know that I 
want to be quite that austere.  If someone said to me "I want to overload 
<- for a struct wrapping a channel type, why is that excluded," I don't 
think I'd have a satisfactory answer

I would also disallow overloading of the =, :=, <-, ..., [], {}, () and yes 
> - equality operators - as well because I believe to do otherwise would be 
> very confusing. 
>

I wouldn't exclude ==.  Even if we have to have a special exclusion for 
pointer types,  any generic/overloading system that didn't support that is 
something most people who want generics would regard as a cruel tease.

:= is not an operator at all in Go (and nor is =), unless I'm missing 
something basic.  I don't usually think of  [], {}, () as operators either, 
though one could make a case for [] as a dyadic indexing/mapping operator.  
I wouldn't be hostile to including that in the allowed set, but I wouldn't 
cry if it were excluded either.

Another possibility would be to introduce two new ordering operators, 
> perhaps >< (for ==) and <> (for !=), which *could* be overloaded though 
> the first of these would take some getting used to!
>

This is not keeping it simple. Excuse me, but I think you are now making 
the sound of someone getting lost in the weeds. ;-)

As for Eric's proposal - using an 'implements' keyword - I like it for the 
> following reasons:
>
> 1. It avoids the need to make up a long list of names for the operators 
> which folks either have to remember or look up in the spec.
>
> 2. It makes it easier to retrofit operator overloading to existing types 
> for which suitable methods have already been defined.
>
> 3. As 'implements' would only be used in contexts which don't exist at 
> present, it needn't be a 'full' keyword and would therefore be backwards 
> compatible.
>

These points had occurred to me, though I did not make them explicit in my 
proposal.  In my invention process, the idea of using operator signatures 
as implied contracts came first. I was then searching for a way to restrict 
the parse context in which new syntax might collide with identifiers in 
existing code and came up with the "implements" clause and its location 
just before a method's leading { as a way to accomplish that restriction.
  

> One area that hasn't been addressed so far is conversions. To convert to 
> say, int64, I'd suggest 'implements int64()' and to convert from int64 
> perhaps 'implements T(int64)' where T is the method's receiver type. TBH, I 
> don't particularly like the idea of user-defined conversions at all but 
> they're probably inevitable if you want to have unified generic constraints 
> and at least they'd always be explicit.
>

I am neutral on whether the contract system should support conversions.  If 
the consensus is that they're needed, fine by me; if not, also fine by me.  
I do not regard either position as a hill that "implements" should die on.
 

> Subject to all this, I think it might well be possible to get rid of 
> contracts and just use interfaces for generic constraints.
>

That was more or less my goal.  All the contract proposals I have seen 
struck me as clever but overweight.
 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Monday, October 15, 2018 at 11:59:44 PM UTC-4, Ian Denhardt wrote:
>
> I'm not in love with the inconsistency, and expect it to cause some 
> confusion with newbies, but I'd have to use it to see how big of a 
> footgun it is in practice. There are certainly worse ideas. 
>

Yes.  If that's the biggest objection anyone comes up with, I think we win 
by a long country mile  over complicated contract-declaration proposals 
where it's difficult to even see where the edge cases might be.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Tuesday, October 16, 2018 at 12:16:16 AM UTC-4, Beoran wrote:
>
>  For niw, I don't see what complelling benefits allowing operators in 
> generic contracts would bring.
>

Consider Ian Lance Taylor's smoke test for generics.  It is: can we 
implement min() and max() on a generic type. 

What he's getting at is that a generic-type system that doesn't have enough 
contract-making strength for this is going to disappoint a lot of the 
people who report wanting generics. I agree.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Monday, October 15, 2018 at 11:32:23 PM UTC-4, Eric Raymond wrote:
>
> Fair enough.  I am completely willing to discard the possibility of 
> overloading && and ||
>

A little thought showed me that this is not required.

The straightforward way to write the contract of "!" would be that it is a 
monadic function of any type returning true if the operand is the zero 
value of that type and false otherwise.  It follows that for types with 
"implements !" the expression a && b expands to this:

if !!a {return !!a} else {return !!b}

If we want to be more Pythonic and remove the requirement that it return 
bool

if !!a {return a} else {return b}

provided a and b are the same type.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

On Monday, October 15, 2018 at 11:02:18 PM UTC-4, Ian Denhardt wrote:
>
>
> There are other operators in the language that don't behave like 
> functions or methods (e.g. boolean operators like && and ||, which 
> short-circut), but the rest of them are things that don't have gobs of 
> use cases for overriding anyway, so it's easy to just say "you can't 
> override those." I don't think we can credibly call operator overloading 
> a solution if you can't abstract over ==. 
>

Fair enough.  I am completely willing to discard the possibility of 
overloading && and || and almost any other operator that doesn't behave 
like a function, because I think we get a rich enough set of contracts from 
those that do.  As I implied in my proposal, I think what people really 
want from contacts is relationals and some operator algebra.

But your objection about pointer types is sound.  And there is a simple 
solution to it. That is to forbid "implements" methods on pointer types at 
compile time.

I can't think of a use case for such overloads that isn't overly clever to 
the point of crazy.  I am more than willing to chop that off to preserve 
the simplicity and transparency of the cases where "implements" is legal.
 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Generics: an unwelcome conclusion and a proposal

[Eric Raymond]

Recent discussion of possible generics designs has forced me to a 
conclusion I'm not happy with, because it requires a feature-cluster that I 
thought I was glad  to be leaving behind in Python.  That is this:

The simplest and most effective way to solve the generics problem is to 
embrace operator overloading and the kind of magic method designations that 
go with it.

I'm still not a big fan of operator overloading as a surface feature of the 
language.  I think the arguments that it encourages overly clever 
one-liners are sound. My argument is not in favor of that. Rather, after 
reviewing all the design strawmen I have seen, I see no way to declare 
contracts for generics that is (a) as simple, and (this is a really 
important point brought out in a recent post) maintains unification with 
primitive types.

In fact, more and more as I look at the proposals that have failed to catch 
fire I see them as clever but doomed attempts to evade operator overloading 
because most people did not want that camel's nose in the tent.  As a 
matter of cold fact I don't either, but I am forced to the conclusion that 
in the portion of design space we can easily reach from Go 1, operator 
overloading and contracts are more joined at the hip than anyone - 
including me - has been willing to face up to now.  I therefore propose 
that we embrace the suck and limit the complexity load on the rest of the 
language as much as possible. 

Here's a stupid-simple system for describing generic contracts with just 
one new keyword: "implements", having  a single argument which is a token 
that may occur as an operator in expressions.   Here is what it would look 
like:

type Sortable interface { 
implements <
}

type MySortable struct {
name string
sortkey id
}

func (r MySortable) LessThan (s MySortable) bool implements < {
return r.sortkey < s.sortkey
}

Because MySortable has a method that "imnplements <", it satisfies the 
Sortable interface.

Each eligible operator has an implied generic signature.  For example is we 
use s and t as generic argument placeholders, that of < is 
s.(T).func(t T) bool.   That of + would be s.(T).func(t T) T.  It would be 
a compile-time error for an "implements"  method not to match the signature 
template of its operator as it applies to basic types.

The general insight this leverages is that every primitive-type operator 
implies a description of a contract *without adding any additional 
complexity to the language*.

Notice that we have evaded the methods themselves needing to have magic 
names.  The only declaration of contract and overloading is the 
"implements" clause.

This passes Ian's smoke test. That is, it is easy to see how to implement 
min() and max() on generics under this system.

By being able to define relationals  and + or * as a composition operator 
for algebras on user-defined types I think we solve a huge part of the 
generic contracts problem.  At the cost of adding only one new construct to 
the language, and one that is easy to describe and understand.  (Because 
the heavy lifting is done by well-established expectations about the 
behavior of primitive types.)

Perhaps I risk overreaching, for I am relatively new to the language, but 
it seems to me that the simplicity and orthogonality of this proposal are 
very much in the spirit of Go. Enough to that the side effect of 
overloading as a surface syntactic feature is - if grudgingly - forgivable.

Can it even possibly be simpler than this? What, if anything, am I missing?


 


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] pytogo 1.2 has shipped

[Eric Raymond]

I am pleased to announce the release of pytogo 1.2.

https://gitlab.com/esr/pytogo

New features in this release:

*  PEP484 type hints are now converted into Go type signatures.
* Without PEP484 hints, return types of functions are now sometimes deduced.
 * Docstrings are moved to the godoc-preferred position above the signature.
 * Python math and cmath library functions are mapped over.
 * Python shebang lines are recognized and converted to "package main".
 * Map variadic function signatures to use "..."
 * Import translation.

Of these, the most interesting to go-nuts is almost certainly the PEP484 
support. It means that function signatures can be type-annotated in context 
in Python before it is translated, substantially reducing the among of 
hand-fixing required in the generated Go code. 

While this is still not a perfect, fire-and-forget translator - and never 
will be - the ratio of output quality over LOC is, I feel, nothing short of 
astonishing.  We get what Michael Ellis described as "a good first draft" 
from just just 594 lines of Python, much of that rule tables rather than 
code!  This demonstrates very concretely that the semantic gap between 
Python and Go is narrow.

The algorithm used for import translation is, though naive, surprisingly 
effective. Python "import foo" is moved straight to Go `import "foo"` 
(actually spans of them are grouped as is idiomatic in Go) except that 
there is a table of special cases that should be dropped or translated.  
So, for example, "import sys" is dropped (the portable stuff in it is in Go 
os package) but "cmath" is mapped to "math/cmplx". Because Go throws errors 
on unrecognized or unneeded imports, cleanup of an import section after 
translation is fast and easy.

This program will undoubtedly continue to gain capability, simply because 
the overhead of adding new rules (such as new standard library module 
mappings) is very low and testing them is easy.  One current development 
focus is better translation of "in" expressions outside for loops.  It is 
also very likely that the next release will translate some module-level 
assignments in Python to Go var and const declarations.

Sometimes minimum-effort, kludgy, brute-force-ish approaches work very 
well. Python-to-idiomatic-Go translation seems to be one of them.

Not losing sight of the goal despite this yak-shaving. The reposurgeon 
translation is at 35% and my translation speed is up to more than 400 LOC 
per day, easily doubling what I was doing before. It would be faster still 
if I weren't writing a lot of unit tests that weren't there in the Python, 
and I expect it to continue to rise.  I plan to ship my experience report 
on the translation to go-nuts at somewhere around 70%.  It will be meaty.






-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Announcing pytogo. a crude but surprisingly effective Python to Go translator

[Eric Raymond]

On Thursday, October 4, 2018 at 4:26:44 AM UTC-4, Peter Waller wrote:
>
> My approach is a bit different, and works by printing the Python AST in 
> Go. There is an overlap in our philosophy of 'produce standalone code' and 
> 'help a human do the translation into idiomatic Go'. However, mine was  a 
> Saturday morning project which was never used in anger, so it is naturally 
> quite incomplete.
>
>>
>>
Oh, good.  We now have direct competition between a swarm-rule approach and 
a deep-reasoning approach. It will be very interesting to see which does 
better in the long run. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Rule-swarm attacks can outdo deep reasoning

[Eric Raymond]

On Wednesday, October 3, 2018 at 4:05:30 PM UTC-4, Michael Ellis wrote:
>
> Nice writeup. Thanks.  Based on today's xkcd , I 
> think Randall Munroe must have read it, too ;-)
>

Seems likely.  Randall and I have been friendly.
 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Rule-swarm attacks can outdo deep reasoning

[Eric Raymond]

Is promised in the thread on pytogo, I have blogged on the general topic of 
rule-swarm attacks in the domain of language transformation.

http://esr.ibiblio.org/?p=8153

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Announcing pytogo. a crude but surprisingly effective Python to Go translator

[Eric Raymond]

I've fixed the bug in handling of multiline strings.

1.1 also translate commion for-loop patterns to Go, including traversing 
lists and dictionaries and doing enumerates.

On Sunday, September 30, 2018 at 6:02:21 PM UTC-4, Michael Ellis wrote:
>
>  I'd love to make use of pytogo my own project. I've got a little over 20K 
> lines that still need translation.
>

Go to it.   By the time we've done that and my 14KLOC this thing ought to 
be in pretty good shape.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Announcing pytogo. a crude but surprisingly effective Python to Go translator

[Eric Raymond]

Thanks, I'll fix the bug and add that to my test loads.
>
>
I discovered earlier today that an incautious rule steps on lambda 
expressions and some kinds of slice indexing.  The risks of not using a 
parser...

Until I ship 1.1 (shortly, probably tomiorrow), I recommend filtering your 
file in small sections, avoiding things like multiline strings.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Announcing pytogo. a crude but surprisingly effective Python to Go translator

[Eric Raymond]

I have mentioned before on this list that I am engaged in source 
translation of a largeish Python prgram - 14KLOC - to Go.  I found the 
automated AST-based translation tools already available unsatisfactory; 
they don't produce a quality of Go code I would consider maintainable.  So 
I wrote my own using crude, old-school regexp-bashing.

This turned out to be surprisingly effective.  The following is from the 
manual page:

* Opens Python block scopes with { and closes them with }.

* Changes docstrings to comments.

* Changes comments from #-led to //-led

* Translates Python boolean constants True, False to Go true, false

* Translates Python "in range" to Go ":= range",  You will need to fix
  up the comma and range clauses by hand.

* Maps Python single-quote literals to Go double-quote literals.

* Translates Python logical connectives and or not to Go && || !.

* Changes Python None to Go nil.

* Changes "def " at the left margin to "func ".

* Common cases of various Python string library methods - capitalize(), 
count(), endswith(), find(), join(), lower(), lstrip(), rfind(), replace(), 
rstrip(), split(), startswith(), split(), upper() - are translated to Go 
string library calls.

* Some common Python standard library calls, notably from os and 
os.filepath, are translated into Go equivalents. The result is not  
guaranteed to be perfectly correct; a Python call that throws a signal on 
failure may map into a Go function with an error return. But it will always 
be a step in the right direction, and the Go compiler will respond with 
indicative error messages.

* The append() method of Python is translated to a call to Go's append 
intrinsic.

* Python's "del foo[bar]" is translated to Go "delete(foo, bar)". Note that 
this can false-match on a list del with a numeric key; this
  will throw a compilation error in Go and you can hand-fix it.

* Trailing line-continuation backslashes are removed.

* Python r-prefix string literals become Go backtick literals

* Python try/finally/catch become pseudo-statements with delimited block 
scopes.  You will need to fix these up by hand.

* Single-line % expressions with a string-literal left hand and either 
tuple or single-variable right hands are translated into fmt.Sprintf 
calls.  You will need to translate %-cookies by hand; %r -> %v or %q is the 
usual trouble spot.

* Changes multiline strings to backtick literals. You'll need to fix up 
backslash escapes and %-cookies yourself.

* Many, but not necessarily all, cases where Go ":=" assignment can replace 
a plain "=" assignment in Python are translated.

This doesn't do a complete job, of course.  But it takes a lot of the pain 
out of hand-translation, and it does preserve comments and structure.  
Probably the automated block scope closing is the biggest single win; 
that's a tedious, fiddly job by hand and mistakes are all too easy.

Future releases will translate a larger subset of Python standard library 
calls.

The repository is at https://gitlab.com/esr/pytogo and the project home 
page at http://www.catb.org/esr/pytogo/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Interaction of signals with defer

[Eric Raymond]

MR and CLA submitted.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Interaction of signals with defer

[Eric Raymond]

On Tuesday, September 18, 2018 at 6:14:54 PM UTC-4, Ian Lance Taylor wrote:
>
> >  Under "Default behavior of signals in Go programs" in the Signals 
> package 
> > documentation, the one sentence "Defer hooks are *not* fired after 
> signal 
> > receipt." would head off potential confusion on this issue. 
>
> Well, maybe.  I agree that that would help if you come with a certain 
> vision of how signals work.  But I don't know how many people come in 
> with that vision. 



It's a plausible mistake from any old Unix hand, I think.

 Maybe we should start by saying something like 
> "signals do not interrupt the execution of Go programs." 
>

Agreed.  I'll go figure out the mechanics for that MR.
 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Interaction of signals with defer

[Eric Raymond]

On Tuesday, September 18, 2018 at 4:10:52 PM UTC-4, Ian Lance Taylor wrote:
>
> Signals in Go never interrupt functions. 
>

Thank you, that was an admirably clear reply. 

I think this needs to be documented more prominently.  Can I file an MR 
against the Go documentation? Web search is not telling me where that 
repository is.  I understand it nmay be privare.

If you would prefer, I can open a documentation issue on it.

 Under "Default behavior of signals in Go programs" in the Signals package 
documentation, the one sentence "Defer hooks are *not* fired after signal 
receipt." would head off potential confusion on this issue. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Interaction of signals with defer

[Eric Raymond]

If interruption of a function with defers is interrupted by an uncaught 
signal (say, ^C from the keyboard) do its defer hooks fire? What about 
defer hooks in functions up the call stack?  If the interrupt happens 
during a defer hook's execution, do the remaining hooks in the implied LIFO 
queue in the current function fire?

Diligent web searches have failed to produce an answer.  Is this 
undocumented?  Unspecified?

Suppose I catch the signal, turning it into a channel notification. Very 
well; now I can call a handler, a goroutine waiting on a notification 
channel.  But it seems I no longer get the effect of
aborting out of the context where the signal was received unless I litter 
the code with notification checks.  Is this the design intent?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Gitlab CI loses its marbles attempting go get.

[Eric Raymond]

On Monday, September 10, 2018 at 5:56:23 PM UTC-4, Uli Kunitz wrote:
>
> Use 
>
> GOPATH= go get *-u* golang.org/x/crypto/ssh/terminal
>
> The package golang.org/x/sys/unix is probably old and requires updating. 
> I have no problems on my machine to go get -u 
> golang.org/x/crypto/ssh/terminal. So it is definitely not broken.
>

Doesn't change the behavior:

 GOPATH=/builds/esr/reposurgeon go get -u golang.org/x/crypto/ssh/terminal
# golang.org/x/sys/unix
src/golang.org/x/sys/unix/env_unix.go:30: undefined: syscall.Unsetenv
src/golang.org/x/sys/unix/ioctl.go:18: undefined: runtime.KeepAlive
src/golang.org/x/sys/unix/ioctl.go:28: undefined: runtime.KeepAlive
Makefile:62: recipe for target 'gosetup' failed
make: *** [gosetup] Error 2
ERROR: Job failed: exit code 1

The get -u command works on my desktop, so this is undoubtedly some 
weirdness about the Docker einvironment involved.  I was hoing someone had 
seen it before and knew a workaround.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Gitlab CI loses its marbles attempting go get.

[Eric Raymond]

Anyone seen behavior like this before?

GOPATH=/builds/esr/reposurgeon go get golang.org/x/crypto/ssh/terminal
# golang.org/x/sys/unix
src/golang.org/x/sys/unix/env_unix.go:30: undefined: syscall.Unsetenv
src/golang.org/x/sys/unix/ioctl.go:18: undefined: runtime.KeepAlive
src/golang.org/x/sys/unix/ioctl.go:28: undefined: runtime.KeepAlive
Makefile:62: recipe for target 'gosetup' failed
make: *** [gosetup] Error 2
ERROR: Job failed: exit code 1

This is a Gitlab docker image trying to fetch Go dependencies and failing.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reflecting on an outer struct from a method of an inner one.

[Eric Raymond]

On Thursday, September 6, 2018 at 9:45:37 PM UTC-4, Nigel Tao wrote:
>
> Asking about embedding and reflection sounds like an XY problem 
> (http://xyproblem.info/). Can you elaborate on what "walk through the 
> user-defined methods of a Cmd-like interpreter instance" is? 
>

Sure.  My reposurgeon program uses the Python Cmd class:

https://docs.python.org/2/library/cmd.html

The Go translation needs an emulation of Cmd, which my talented and 
hardworking apprentice Ian Bruene has just shipped an 0.1 version of:

https://gitlab.com/ianbruene/Kommandant

The main remaining issue with his implementation is that he has to 
explicitly register command handlers, linking them to names. This is an 
obviously error-prone pain in the ass, and I want to make it work more like 
the Python original.  In that original, you declare a class that inherits 
from Cmd.  Cmd then walks through your interpreter class picking out method 
names beginning with "doX" and binding them to be fired by the command 
string X.

Obviously we can't use inheritance in the normal sense here, but I was 
hoping that if an embedded instance of Cmd could discover the struct 
instance it's part of, a similar thing could be pulled off.  Alas, your 
answer that this can't be done is unsurprising and I was half expecting 
it.  Calling one function to pass in the outer type beats having to declare 
all the command handlers individually

...provided, that is, that it's possible to get from the member name to a 
function pointer that can be called like a closure.  Does the reflection 
system support such a call-indirect method?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Reflecting on an outer struct from a method of an inner one.

[Eric Raymond]

This question is intended to assist implementation of a Go workalike for 
the Python Cmd library.  This workalike will be made publicly available,

The attached program lists

Inner
id

as expected - that is, the fieldnames of the Outer structure. But 
ListFields is a method of Inner.

In the attached program, I would like to be able to remove the &x argument 
from ListFields.  That is, I want the Inner code to be able to reflect on 
the Outer instance without having to be fed the address of the Outer 
instance explicitly.  I specifically do *not* want to move the method to 
Outer - reflection  from within Inner is the point of the exercise.

Is there a way for a struct instance to get its own address through the 
runtime? 

The application would be to walk through the user-defined methods of a 
Cmd-like interpreter instance.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


innerclass.go
Description: Binary data

Re: [go-nuts] Space-optimization question

[Eric Raymond]

On Wednesday, September 5, 2018 at 10:38:16 AM UTC-4, Jan Mercl wrote:
>
> On Wed, Sep 5, 2018 at 4:23 PM Eric Raymond  > wrote:
>
> > Matters in my case because the deserialization of a repository history 
> can contain hundreds of thousands of constants like "M", "D", "R", and "C"
> > representing fast-export stream file operation types. I could intern 
> them explicitly but it avoid code clutter and some lookup time if I don't 
> have to.
>
> Maybe I'm misunderstanding something, but the above quoted seems to talk 
> about string values obtained at runtime. The linker coalescing string 
> literals (compile time constants) discussed before does and cannot apply to 
> string values not known at compile time.
>

No, these are actually compile-time constants used in constructor 
functions  that stuff them into instance structures.  In C they'd be 
#define macros.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Space-optimization question

[Eric Raymond]

On Wednesday, September 5, 2018 at 10:12:51 AM UTC-4, Jan Mercl wrote:
>
> On Wed, Sep 5, 2018 at 4:00 PM Eric Raymond  > wrote:
>
> > If I have multiple occurrences of a string constant in source code - 
> say, "M" - can I count on the compiler to create one static instance and 
> pass references to it everywhere? 
>
> It's unspecified, so strictly speaking you cannot. (Also, everything in Go 
> is passed by value). However, all well known Go compilers do put text 
> literals into the text segment and one can reasonably expect the linker to 
> coalesce them. The implementation of a string variable is a two worded 
> struct having a pointer to the string value and the length of the string 
> value. Only this struct is being passed around (by value), the actual value 
> the struct points to never is
>

Thanks, that is pretty much the exact answer I was expecting  given the 
immutability of strings.

Matters in my case because the deserialization of a repository history can 
contain hundreds of thousands of constants like "M", "D", "R", and "C" 
representing fast-export stream file operation types. I could intern them 
explicitly but it avoid code clutter and some lookup time if I don't have 
to.

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Space-optimization question

[Eric Raymond]

If I have multiple occurrences of a string constant in source code - say, 
"M" - can I count on the compiler to create one static instance and pass 
references to it everywhere?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Tuesday, September 4, 2018 at 7:13:50 AM UTC-4, Jan Mercl wrote:
>
> I'm worried the suggested editing pass could make the docs better for 
> newbies and less useful for non-newbies in the same time. Maybe the docs 
> for beginner's should be the edited, but separate _version_ of the current 
> docs.
>

Dangerous.  Remember the Single Point of Truth rule.

No, it could be done right so that the outside-in view adds value without 
cruftifying.   I know this because my brain automatically composes those 
edits as soon as I have the knowledge to do it.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Friday, August 31, 2018 at 6:55:28 AM UTC-4, ohir wrote:
>
> On Thu, 30 Aug 2018 18:41:48 -0700 (PDT) 
> Eric Raymond > wrote: 
>
> > Oh, actually, I have one other issue; how to translate virtual methods 
> in a 
> > Python base class into Go's object model. 
>
> Inheritance based domain models do not translate directly to go's 
> composition 
> based ones. Forget "virtual". Forget inheritance. 
>

Most of your advice is helpful.  This is not.  Helpful advice would begin 
with a topic sentence  something like "Here is how to use composition to 
emulate superclass behavior."

Remember, I'm not writing code de novo.  I'm translating a large existing 
codebase. The distance I can wander away from its present 
factoring/organization without triggering a dangerous complexity explosion 
is very limited.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Friday, August 31, 2018 at 12:06:44 AM UTC-4, Nigel Tao wrote:
I'm only guessing, but I think Ian's "which docs did you look at" was 
referring to your previous "I was unable to extract this enlightenment 
from the documentation, despite sweating over it pretty hard"

Well, that could be.  I grovelled through a lot of the official Go 
documentation, also third-party stuff I found with search engines.  And 
yes, it did include reading "The Laws of Reflection". Which didn't 
enlighten me, but did adequately prepare me for the short explanation 
upthread that did, so there's that.

It's difficult for me to be specific about how much coverage I achieved, 
because "the documentation", outside of the library pages, is kind of a 
poorly organized blob.   Many individual parts are excellent, but it is 
difficult to know where to find things until one is already familiar enough 
with the territory that finding things has fallen down one's list of 
concerns.

There's also a pervasive problem with the style.  The Go documentation is 
insular and hieratic.  It tends to explain things very well if you already 
inhabit the mindset of a Go programmer, but to not be very good at 
providing an entry into that mindset to people who do not already inhabit 
it. I've seen this movie before, it's a common problem in complex software 
with documentation written by its developers from the inside out.

For me personally this is a relatively minor problem - not my first rodeo 
of this kind nor even my dozenth, I have broad experience and know how to 
adapt and persist. For most newbies it's going to be a serious blocker. I 
can recommend a fix: the team could hire a tech writer with an outside-in 
view (and no prior knowledge of Go) to do a serious editing pass on the 
documentation.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Some background on reposurgeon and my Go translation problem

[Eric Raymond]

On Friday, August 31, 2018 at 4:30:08 AM UTC-4, Jan Mercl wrote:
>
> In case you haven't heard it before, Google was thinking on the same lines 
> and released Grumpy last year: 
> https://opensource.googleblog.com/2017/01/grumpy-go-running-python.html. 
> I never used the tool and it may possibly not even support your large 
> Python program(s). But maybe it could be useful one way or another.
>

 I did experiment with that.  Unfortunately, the source code it produced 
when I ran it on my stuff was an unmaintainable mess. Possibly correct, I 
don't know - but I couldn't trust it because I can't read it. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Some background on reposurgeon and my Go translation problem

[Eric Raymond]

There's been enough interest here in the technical questions I've been 
raising recently that a bit of a backgrounder seems in order.

Back in 2010 I noticed that git's fast-import stream format opened up some 
possibilities its designers probably hadn't anticipated. Their original 
motivation was to make it easy to write exporters from other 
version-control systems.  Their offer was this: here's a flat-file format 
that can serialize the entire state of a git repository.  If you can dump 
your repository state from random version control system X in this format, 
we can reanimate the history in git.

This was a very clever idea, and lots of people proceeded to write 
exporters on this model.  A few VCS implementers, noticing that this 
implied a vast  one way traffic of user attention away from them and 
towards git, wrote importers that could consume fast-export streams from 
git back *to *random version-control system X.  I noticed that the effect 
was to turn git stream dumps into a de-facto exchange standard for 
version-control histories.

One of my quirks is that I like thinking about version-control systems and 
the tools around them.  I had been noticing for years that most repository 
conversion tools are pretty bad in a specific way.  They tend to try to 
over-automate the process, producing brute-force conversions full of crufty 
artifacts and minor defects around the places where the data models of 
source and target systems don't quite match.  There were, at the time, no 
tools fit for a human to fix these problems.

Reposurgeon implements - in Python - a domain-specific language for 
describing surgical operation on repositories.  It can be run in batch mode 
or as an interactive structure editor for doing forensics on damaged 
repository metadata.  It works by calling a front end to get a stream dump 
of the repository you want to edit, deserializing the dump into an 
attributed graph, supporting a full repertoire of operations on that graph, 
and then writing the result out as a stream dump fed to an importer for the 
target system.

The target system can be the same as the source.  Or a different one.  
There are front-end/back-end pairs to support RCS, CVS, Subversion, git, 
mercurial, darcs, monotone,  bitkeeper, bzr, and src. Not all of these 
combinations are well-tested, but moves from CVS, Subversion, git, bzr, and 
mercurial to git or mercurial are pretty solid.

My conjecture that a human-driven tool with good exploratory capabilities 
would produce higher-quality history translations than fully-automated 
converters rapidly proved correct, rather dramatically so in fact. 
Reposurgeon has been the key tool for a great many high-complexity, 
high-risk history translations.  Probably the most consequential single 
success was moving the history of GNU Emacs from bzr to git, cleaning up a 
lot of ancient cruft from RCS and CVS along the way.

Below the size of GCC, which IIRC was around 40K commits, Python gave me 
reasonable turnaround times. This is important if you need to do a lot of 
exploration to find a good conversion recipe, which you always do with 
these large old histories.  But there was an adverse-selection effect.  The 
average size of the histories people wanted by to convert kept increasing. 
Eventually I ended up designing a semi-custom PC optimized for the job 
load, with high CPU-to-memory bandwidth and beefy primary caches to enable 
it to handle large working sets - graph-theory problems gigabytes wide.  
Its fans call it the Great Beast, and three years after it was built you 
still can't spec a machine that performs better from COTS parts.

(That may change soon. The guy who actually put togeether the Beast for me 
is contemplating an upgrade based on the Cascade Lake chipset due out from 
Intel in Q4.  His plan is to build one for me and another for Linus 
Torvalds. The clever fellow has sponsors lined up a block long to be in the 
build video.)

Then came GCC.  The GCC repository is over 259K commits.  It brings the 
Great Beast to its knees. Minimum 9-hour times for test conversions, which 
is intolerable. I concluded that Python doesn't just cut it at this scale.  
I then shopped for a new language pretty carefully before choosing Go.  
Compiled Lisp was a contender, and I even briefly considered OCaml.  Go won 
in part because the semantic distance from Python is not all that large, 
considering the whole late-binding issue.

Python reposurgeon is about 14KLOC  of code.  In six days I've translated 
about 11% of it, building unit tests as I go. (There's already a very 
strong set of functional and end-to-end tests. There has to be; I wouldn't 
dare modify it otherwise. To say it's become algorithmically dense enough 
to be tricky to change would be to wallow in understatement...)

However, this is only my second Go project.  My first was a finger 
exercise, a re-implementation of David A. Wheeler's sloccount tool.  So I'm 
in a weird place where I'm tran

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Thursday, August 30, 2018 at 8:31:44 PM UTC-4, Ian Lance Taylor wrote:
>
> > That was helpful.  I feel like I have a more solid grasp  now, though 
> I'm a 
> > bit worried about what details might have changed.  Is there any chance 
> of 
> > getting the author to update this and add it to the official Go 
> > documentation? 
>
> Conceptually speaking, nothing has changed. 
>
> Which docs did you look at?  Just wondering what could be updated. 
>

This:  https://research.swtch.com/interfaces 

It did look rather like the abstractions had not changed, but I don't have 
the expertise to be sure.

I'm successfully writing type switches on my Events now, so that is 
significant progress. I expect the  rest of the translation to be tedious 
but not conceptually difficult. All 13KLOC of it,  ouch.

Oh, actually, I have one other issue; how to translate virtual methods in a 
Python base class into Go's object model.  I'll start another thread about 
that.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Thursday, August 30, 2018 at 7:25:10 PM UTC-4, Nigel Tao wrote:
>
> On Fri, Aug 31, 2018 at 4:18 AM Burak Serdar  > wrote: 
> > If b is an interface to a *Blob, what's stored in the slice is {Type: 
> > *Blob, Value: pointer to the object}. A copy of this interface value 
> > is also in the map. So you have two copies of the interface value, 
> > both pointing to the same Blob object. 
>
> Yes, the details might have changed slightly since 2009, but 
> https://research.swtch.com/interfaces might help grok the mental model 
> for Go interfaces. 
>

That was helpful.  I feel like I have a more solid grasp  now, though I'm a 
bit worried about what details might have changed.  Is there any chance of 
getting the author to update this and add it to the official Go 
documentation?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Thursday, August 30, 2018 at 3:00:18 PM UTC-4, Burak Serdar wrote:
>
> If you're porting from another language, you're likely to encounter 
> this situation soon, if not already. 
>

Right, and because Python code  often makes aggressive use of runtime 
polymorphism it's *particularly *likely to come from that direction.

I don't think my case is at all a strange outlier. Maybe there ought to be 
a FAQ entry on this addressed to Python programmers?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Thursday, August 30, 2018 at 2:18:38 PM UTC-4, Burak Serdar wrote:
>
> If b is an interface to a *Blob, what's stored in the slice is {Type: 
> *Blob, Value: pointer to the object}. A copy of this interface value 
> is also in the map. So you have two copies of the interface value, 
> both pointing to the same Blob object. 
>

Thank you!  That is the answer I was looking for!

I note that I was unable to extract this enlightenment from the  
documentation, despite sweating over it pretty hard.

Either I'm stupid or this is a documentation/tutorial issue that could 
stand to be addressed better.  And I don't think I'm stupid.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

>
> If Event is your interface and EventImpl is your struct, what you need 
> is a map[string]Event, not map[string]*Event. Make sure you put 
> &EvenImpl{} to your slices and maps, not EventImpl{} 
>
>
Here'a what is confusing me.  Yes, I can write

 _mark_to_object  map[string]Event


and

b.repo._mark_to_object[mark] = b

for b a pointer to Blob, with Blob satisfying Event.  This compiles without 
error. 

But what actually happens here? Is the new value a pointer to the same Blob 
instance that is in the main list, or does the code dereference b and put a 
*copy* of the blob in the map?  The first behavior (reference) is what I 
want.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Reference to type satisfying a specified interface

[Eric Raymond]

On Thursday, August 30, 2018 at 1:32:18 PM UTC-4, Akram Ahmad wrote:
>
> Your work on porting *reposurgeon* (from Python into Go) for improved 
> performance sounds very cool, Eric (BTW, your work in, and musings on, Lisp 
> and UNIX have been a huge influence on my evolution as a programmer. Kudos 
> for that!
>

Thanks for the kind words.

I have rather been forced into this. In cooperation with the GCC devteam, 
I'm trying to move their repository from Subversion to git.  It's so 
freaking big and Python is so slow that test conversions are taking more 
than 9 hours, with occasional OOMs because the  working set is so huge.  I 
have got to reduce that cycle time, it is making forensics on the 
malformations in the history effectively impossible.  

Go seems to be a tool that fits the job - translation of a smaller 
auxiliary program suggests I ca n reasonably expect a 40x speed increase -  
but this is my first major project in it and I have not yet mastered the 
language.

Once I do, I expect it to replace C for all my new projects that would 
otherwise have been C.  Also anything I would havew written in Python that 
needs to perform at machine speeds rather than human speed.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

On Thursday, August 30, 2018 at 1:14:25 PM UTC-4, Jan Mercl wrote:
>
> On Thu, Aug 30, 2018, 18:57 Eric Raymond > 
> wrote:
>
>> I'm trying to translate this to Go in a type-safe way. To do this, I need 
>> to be able to write two declarations: "Slice of pointers to objects 
>> satisfying the Event interface" 
>>
>
> []*Event, but quite probably []Event is what is really needed.
>
> and "map of string keys to pointers to objects satisfying the Event 
>> interface".  
>>
>
> map[string]*Event, but once again, my bet is on map[string]Event.
>
> My attempts so far have yielded very cryptic error messages and no success.
>>
>
> It would probably help your case if you could provide some self-contained 
> example reproducing the errors and post a link to the Go Playground.
>

That's hard.  The structures are intertwingled parts of ab only partly 
translated 14KLOC program  But I can isolate some key declarations.  If 
Event is my interface type, and I write

events []Event
_mark_to_object  map[string]*Event

as member declarations in the Repository class, and b is a pointer to a 
just-allocated instance of a Blob object satisfying the Event interface, 
and  I write this

 b.repo._mark_to_object[mark] = b

I get this message:

cannot use b (type *Blob) as type *Event in assignment:
*Event is pointer to interface, not interface

Note that the map values really do need to be pointers, because I need the 
map to refer to the mutable data in the event list, not a local copy of it 
in the map.  What's stumping me is how to communicate this to the compiler.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Reference to type satisfying a specified interface

[Eric Raymond]

I apologize if this seems like an elementary question, but web searches and 
reading the Go documentation is not turning up an answer.

I'm in the process of translating reposurgeon, an editor for 
version-control histories, from Python into Go for improved performance, 
.The central data structure is Repository, which consists mainly of a list 
of events such as commits and tags.  In Python, I use the language's late 
binding and simply declare a list of containing multiple object types named 
Commit, Tag, etc.  There are also a number of lookup maps pointing at 
objects in the main event list.  Because objects in Pytron are passed by 
reference everything is fairly straightforward.

I'm trying to translate this to Go in a type-safe way. To do this, I need 
to be able to write two declarations: "Slice of pointers to objects 
satisfying the Event interface" and "map of string keys to pointers to 
objects satisfying the Event interface".  My attempts so far have yielded 
very cryptic error messages and no success.

Is it actually possible to restrict the main list's polymorphism in this 
way and avoid casts?






 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Unexpected behavior of %+02d in Sprintf() and friends

[Eric Raymond]

On Wednesday, August 29, 2018 at 7:57:23 AM UTC-4, Borman, Paul wrote:
>
> Is it possible you used the format string “+%02d” vs “%+02d”?  The first 
> will give you the +00 you expected while the second is +0, as discussed.
>

Alas, no.  Here's the line from my test program:

fmt.Printf("%s: %+02d %02d\n", runtime.Version(), 0, 0)

In the original context where I saw the behavior, I was generating [+-]hhmm 
strings corresponding to timezone offsets. The sign in the output could not 
be fixed in the format; it had to be derived from the offset.




-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] More trouble with date formatting

[Eric Raymond]

The attached program emits 

go1.10.1: value '2010-10-27 18:43:32 + +' error 

on my system.  Note the duplicated zone field. Is this expected?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


gobug2.go
Description: Binary data

[go-nuts] Re: Unexpected behavior of %+02d in Sprintf() and friends

[Eric Raymond]

On Tuesday, August 28, 2018 at 4:49:02 PM UTC-4, peterGo wrote:
>
> "Width is specified by an optional decimal number immediately preceding 
> the verb. If absent, the width is whatever is necessary to represent the 
> value. "
>
> https://golang.org/pkg/fmt/
>
> Width is two.
>

Thanks for the clarification.  May I recommend appending to that sentence 
"(including the leading sign, if any, even if the sign is forced by a + 
modifier)"?

I still think i saw apparently inconsistent behavior.  I'm still writing 
unit tests involving date conversions, so I'm going to try to reproduce 
this and turn in an actionable report.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Unexpected behavior of %+02d in Sprintf() and friends

[Eric Raymond]

Under Go 1.10.1, feeding an 0 value to a %+02d specifier sometimes yields  
"+0", not "+00". The attached tiny Go program may reproduce this behavior.  
I say "may" because I first observed it in a series of unit tests of date 
format conversions - in different format strings %+02d expanded 
differently.  I haven't found a pattern to this, or I'd report it. On my 
system this program, at least, has repeatable behavior.

If this behavior were consistent, I'm not sure it would be a bug. It's 
possible that the sign is supposed to be counted as part of the number 
width; if so, it's an interesting question whether this is the right thing 
when explicit sign is forced by +.  The documentation is unclear.

The apparent inconsistency worries me.  There may be some state in the 
form,at-interpretation code that is not always tracked correctly.

In accordance with the Contribution Guidelines, I'm tossing  the question 
out here for a sanity check before throwing it on the bugtracker.  Have 
there been any similar reports?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


gobug1.go
Description: Binary data