Re: Type checking expressions

[Robin Palotai]

I wondered if
https://downloads.haskell.org/~ghc/7.8.1/docs/html/users_guide/defer-type-errors.html
could help.

I tried to click around in the GHC 8.2 tree of
http://stuff.codereview.me/ghc/#ghc/compiler/typecheck/TcErrors.hs?corpus=ghc-8.2.1-rc2=120,
but it seems deferring type errors just reports via a different means, and
doesn't change the type-checking behavior. But correct me if I'm wrong.

2018-03-05 17:11 GMT+01:00 Simon Peyton Jones via ghc-devs <
ghc-devs@haskell.org>:

> Always cc ghc-devs!   Bottle-necking on me may well yield a slow
> response!   Or even Haskell-café.
>
>
>
> What is the type of (\x -> [x,y])?   Where y is in scope with type y::a.
> Presumably something like   a -> [a]?  Or is it forall a. a -> [a]?  And
> would your answer change if you had just (\x -> [x,x])?
>
>
>
> Generalisation is tricky, and for terms with non-closed types it is hard
> to know what you need in your use-case.  A type like ‘a’ might be a very
> fine answer!
>
>
>
> A lot depends on precisely what you are trying to do.
>
>
>
> Simon
>
>
>
> *From:* Peter Podlovics [mailto:peter.d.podlov...@gmail.com]
> *Sent:* 05 March 2018 14:54
> *To:* Simon Peyton Jones 
> *Subject:* Re: Type checking expressions
>
>
>
> My main concern with that approach is that it might not give the correct
> type. For example the hsPatType function only gives unconstrained types, so
> it is incorrect for any numeric literal, since it gives "a" instead of "Num
> a => a".
>
> So the question is whether it is possible to retrieve the context of the
> type variables as well. Also this problem may arise in the case of
> expressions as well, that is why I scrapped that approach and tried to type
> check the AST with the TcM monad directly, but without any success.
>
> Could you give me any leads on how to solve this problem?
>
> Thanks in advance,
>
> Peter
>
>
>
> On Mon, Mar 5, 2018 at 9:44 AM, Simon Peyton Jones 
> wrote:
>
> Peter
>
>
>
> My goal is to determine the type of every expression, pattern etc. in the
> syntax tree
>
>
>
> After type checking is complete, the syntax tree is liberally annotated
> with types.
>
>
>
> We do not yet have a function
>
> hsExprType :: HsExpr Id -> Type
>
> but we do have
>
> TcHsTyn.hsPatType :: Pat GhcTc -> Type
>
> and you or someone could readily make an equivalent for HsExpr.
>
>
>
> Most type errors are reported by adding an error constraint, but still
> returning an annotated tree.
>
> Some, I’m afraid, are still done in the old way, by throwing an exception
> – so you don’t get back an annotated tree in that case.  But they are
> relatively rare.
>
>
>
> Others must have wanted something like this…
>
>
>
> Simon
>
>
>
> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Peter
> Podlovics
> *Sent:* 02 March 2018 12:05
> *To:* ghc-devs@haskell.org
> *Subject:* Fwd: Type checking expressions
>
>
>
> Hello everyone,
>
> I would like to ask for some advice regarding the type checker part of GHC.
> My goal is to determine the type of every expression, pattern etc. in the
> syntax tree. Currently the compiler doesn't store this information, so I
> have
> to type check manually. One important aspect is that the program may be
> ill-typed,
> but I still want to extract as much information as possible.
>
> I tried using local type checking functions (eg.: tcInferSigma), but
> whenever
> I used it on an expression that had some "out-of-scope" names in it, it
> failed.
>
> > f xs = length xs
>
> The reason was that xs was not in the local environment.
>
> My question is: how could I provide the necessary local environment for
> these
> type checking functions? Also in the general case, is it possible to
> somehow
> annotate each expression with its type during the type checking?
>
> The motivation for this is that I want to implement a tool that
> automatically
> corrects ill-typed programs based heuristics. For that I need to know the
> types
> of certain AST nodes.
>
> Peter Podlovics
>
>
>
>
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Extracting representation from GHC

[Robin Palotai]

See also
https://github.com/google/haskell-indexer/blob/master/haskell-indexer-backend-ghc/src/Language/Haskell/Indexer/Backend/GhcApiSupport.hs
for an as-complete GHC API based setup as I could get. The comments
indicate possible deficiencies.

The test
https://github.com/google/haskell-indexer/blob/master/haskell-indexer-backend-ghc/tests/Language/Haskell/Indexer/Backend/Ghc/Test/BasicTestBase.hs#L179
shows some cases that are covered (for example testTemplateHaskellCodeExecFFI
).

In practice one can run AST extraction with HscNoLink + HscInterpreted for
most targets, but for hairy ones (FFI invoked from TemplateHaskell in
certain ways, for examples) that will fail.

2018-01-19 19:46 GMT+01:00 Robin Palotai <palotai.ro...@gmail.com>:

> See some additions inline.
> BR
> Robin
>
> 2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs <
> ghc-devs@haskell.org>:
>
>> |  To do this, my idea is to instruct GHC with a compilation flag to give
>> |  out its internal representation of the source code.
>>
>> Why can't you just use GHC as a library, and ask it to parse and
>> typecheck the module and then look at what it gives you.
>>
>> Last time I checked (GHC 8.2, for haskell-indexer), using the library is
> not equivalent to using GHC's Main. GHC's Main does tremendous amount of
> magic with flag parsing and state setup, and doesn't expose all the
> functionality for libraries to do the same.
>
> AFAIR we saw two possible ways to get the AST out from a complicated setup
> (FFI, objects, packages, ...):
> 1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more
> limited at the time - the gist in the below trac entry mentions that even
> the Frontend plugin didn't do everything Main does).
> 2) Refactor GHC Main and expose all the functionality to GHC API.
>
> I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's
> slightly related.
>
> By the way, you can click around http://stuff.codereview.me/ghc/#ghc/ghc/
> Main.hs?corpus=ghc-8.2.1-rc2 in the 'main' function to see all
> the magic.
>
> Others are more used to the GHC API than me, though.
>>
>> S
>>
>> |  -Original Message-
>> |  From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of
>> |  Németh Boldizsár
>> |  Sent: 19 January 2018 09:35
>> |  To: ghc-devs@haskell.org
>> |  Subject: Extracting representation from GHC
>> |
>> |  Dear GHC Developers,
>> |
>> |  I would like to ask your opinion on my ideas to make it easier to use
>> |  development tools with GHC.
>> |
>> |  In the past when working on a Haskell refactoring tool I relied on
>> |  using the GHC API for parsing and type checking Haskell sources. I
>> |  extracted the representation and performed analysis and transformation
>> |  on it as it was needed. However using the refactorer would be easier
>> |  if it could work with build tools.
>> |
>> |  To do this, my idea is to instruct GHC with a compilation flag to give
>> |  out its internal representation of the source code. Most build tools
>> |  let the user to configure the GHC flags so the refactoring tool would
>> |  be usable in any build infrastructure. I'm thinking of using the pre-
>> |  existing plugin architecture and adding two new fields to the Plugin
>> |  datastructure. One would be called with the parsed representation
>> |  (HsParsedModule) when parsing succeeds, another with the result of the
>> |  type checking (TcGblEnv) when type checking is finished.
>> |
>> |  What do you think about this solution?
>> |
>> |  Boldizsár
>> |
>> |  (ps: My first idea was using frontend plugins, but I could not access
>> |  the representation from there and --frontend flag changed GHC
>> |  compilation mode.)
>> |
>> |  ___
>> |  ghc-devs mailing list
>> |  ghc-devs@haskell.org
>> |  https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h
>> |  askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
>> |  devs=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5
>> |  5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471
>> <(513)%20536-0471>
>> |  87=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D=0
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Extracting representation from GHC

[Robin Palotai]

See some additions inline.
BR
Robin

2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs <
ghc-devs@haskell.org>:

> |  To do this, my idea is to instruct GHC with a compilation flag to give
> |  out its internal representation of the source code.
>
> Why can't you just use GHC as a library, and ask it to parse and typecheck
> the module and then look at what it gives you.
>
> Last time I checked (GHC 8.2, for haskell-indexer), using the library is
not equivalent to using GHC's Main. GHC's Main does tremendous amount of
magic with flag parsing and state setup, and doesn't expose all the
functionality for libraries to do the same.

AFAIR we saw two possible ways to get the AST out from a complicated setup
(FFI, objects, packages, ...):
1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more
limited at the time - the gist in the below trac entry mentions that even
the Frontend plugin didn't do everything Main does).
2) Refactor GHC Main and expose all the functionality to GHC API.

I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's
slightly related.

By the way, you can click around
http://stuff.codereview.me/ghc/#ghc/ghc/Main.hs?corpus=ghc-8.2.1-rc2
in the 'main' function to see all the magic.

Others are more used to the GHC API than me, though.
>
> S
>
> |  -Original Message-
> |  From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of
> |  Németh Boldizsár
> |  Sent: 19 January 2018 09:35
> |  To: ghc-devs@haskell.org
> |  Subject: Extracting representation from GHC
> |
> |  Dear GHC Developers,
> |
> |  I would like to ask your opinion on my ideas to make it easier to use
> |  development tools with GHC.
> |
> |  In the past when working on a Haskell refactoring tool I relied on
> |  using the GHC API for parsing and type checking Haskell sources. I
> |  extracted the representation and performed analysis and transformation
> |  on it as it was needed. However using the refactorer would be easier
> |  if it could work with build tools.
> |
> |  To do this, my idea is to instruct GHC with a compilation flag to give
> |  out its internal representation of the source code. Most build tools
> |  let the user to configure the GHC flags so the refactoring tool would
> |  be usable in any build infrastructure. I'm thinking of using the pre-
> |  existing plugin architecture and adding two new fields to the Plugin
> |  datastructure. One would be called with the parsed representation
> |  (HsParsedModule) when parsing succeeds, another with the result of the
> |  type checking (TcGblEnv) when type checking is finished.
> |
> |  What do you think about this solution?
> |
> |  Boldizsár
> |
> |  (ps: My first idea was using frontend plugins, but I could not access
> |  the representation from there and --frontend flag changed GHC
> |  compilation mode.)
> |
> |  ___
> |  ghc-devs mailing list
> |  ghc-devs@haskell.org
> |  https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h
> |  askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
> |  devs=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5
> |  5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471
> |  87=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D=0
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Determine instance method from class method callsite

[Robin Palotai]

About "Var" - that is just a Name with additional info, but happens in the
typechecked tree (according to API docs).

2017-10-05 18:00 GMT+02:00 Simon Peyton Jones <simo...@microsoft.com>:

> well the HsWrapper might have something much more complicated than a
> DFunId.  It might need to construct the dictionary for Eq [a] from the DFun
> for Eq [a] and a dictionary for Eq a.
>
>
>
> it would be easy to resolve the Var of the instance method.
>
> I don’t know what “the var of the instance method” is.
>
>
>
> I feel I’m missing the point.  maybe others can help?
>
>
>
> Simon
>
>
>
> *From:* Robin Palotai [mailto:palotai.ro...@gmail.com]
> *Sent:* 05 October 2017 11:17
> *To:* Simon Peyton Jones <simo...@microsoft.com>
> *Cc:* GHC developers <ghc-devs@haskell.org>; haskell <
> haskell-c...@haskell.org>
> *Subject:* Re: Determine instance method from class method callsite
>
>
>
> Hello Simon - I outlined the approach in https://github.com/google/
> haskell-indexer/issues/73
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgoogle%2Fhaskell-indexer%2Fissues%2F73=02%7C01%7Csimonpj%40microsoft.com%7C6798607dd45a4865d5ef08d50bda3727%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636427954212256346=HkxeXUYlbyGJnHu2%2BM17Z5aE7XXAZCWCnWwGLMV1%2BEQ%3D=0>
> .
> TLDR is
> 1) at callsite, indeed the HsWrapper needs to be analysed to get the
> instance DFunId
>
> 2) at instance declaration site, we need to again take note of the DFunId.
>
> It is slightly more complicated than I expected the API. I would have
> expected that from having the Var of a class method, and the  of
> the instance (here DFunId), it would be easy to resolve the Var of the
> instance method.
>
> But it seems there's no direct way, one has to build a lookup table from
> the instance methods' (DFunId + plain stringy method name), and look that
> up from the callsite.
>
> Or I might have missed a way to deconstruct / query a DFunId for the
> method Vars.
>
> I'll put augmenting the commentary on my mental TODO list :)
>
>
>
> 2017-10-05 11:58 GMT+02:00 Simon Peyton Jones <simo...@microsoft.com>:
>
> Did you get a reply?
>
>
>
> I’m not 100% certain of your question, but consider the
>
>
>
> bar = show 
>
> for some expression e.   In the input to the type type checker the syntax
> tree for the RHS will be something like
>
> HsApp (HsVar “show”) 
>
>
>
> (The “show” isn’t really a string, it’s the Name for the class method.)
>
>
>
> After typechecking the syntax tree is augmented (or “elaborated”) with
> type and dictionary application.  So in concrete form it might look like
>
> bar = show @Foo dShowFoo 
>
>
>
> Because show :: forall a. Show a => a -> String, so show is apply to the
> type of its argument, and then to the dictionary.
>
>
>
> In HsSyn this part is done with a HsWrapper See TcEvidence.HsWrapper.  The
> elaborated syntax tree look like
>
>
>
> HsApp (HsWrap  (HsVar “show”))
>
>   
>
>
>
> The  part expresses the type and dictionary application. In this
> case it’ll look like
>
> WpEvApp dShowFoo (WpTyApp Foo WpHole)
>
>
>
> See the notes with `HsWrapper` in TcEvidence.
>
>
>
> Does that help?
>
>
>
> It would be great to augment the https://ghc.haskell.org/trac/
> ghc/wiki/Commentary with this sort of info (insofar as it doesn’t have it
> already).  If you augment I can review.  Email is quickly lost.
>
>
>
> Simon
>
> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Robin
> Palotai
> *Sent:* 19 September 2017 06:39
> *To:* GHC developers <ghc-devs@haskell.org>; haskell <
> haskell-c...@haskell.org>
> *Subject:* Determine instance method from class method callsite
>
>
>
> Sorry, I messed up subject and mailing list. Copying to both list now
> after the mistake (wanted only ghc-devs for specificity).
>
>
>
> Thanks!
>
>
>
> 2017-09-19 7:36 GMT+02:00 Robin Palotai <palotai.ro...@gmail.com>:
>
> Hello GHC devs,
>
>
>
> Before inventing the wheel, want to check if there is a GHC API way to
> look up the (fully) resolved instance method from a class method.
>
>
>
> For example, given a code
>
>
>
> data Foo Int deriving Show
>
>
>
> bar = show (Foo 3)
>
>
>
> when inspecting the Typechecked AST for bar's show call, I would like to
> get to the Name / Id of 'show' of the 'Show' typeclass.
>
>
>
> I believe I 

Re: Determine instance method from class method callsite

[Robin Palotai]

Hello Simon - I outlined the approach in
https://github.com/google/haskell-indexer/issues/73.
TLDR is
1) at callsite, indeed the HsWrapper needs to be analysed to get the
instance DFunId
2) at instance declaration site, we need to again take note of the DFunId.

It is slightly more complicated than I expected the API. I would have
expected that from having the Var of a class method, and the  of
the instance (here DFunId), it would be easy to resolve the Var of the
instance method.

But it seems there's no direct way, one has to build a lookup table from
the instance methods' (DFunId + plain stringy method name), and look that
up from the callsite.

Or I might have missed a way to deconstruct / query a DFunId for the method
Vars.

I'll put augmenting the commentary on my mental TODO list :)

2017-10-05 11:58 GMT+02:00 Simon Peyton Jones <simo...@microsoft.com>:

> Did you get a reply?
>
>
>
> I’m not 100% certain of your question, but consider the
>
>
>
> bar = show 
>
> for some expression e.   In the input to the type type checker the syntax
> tree for the RHS will be something like
>
> HsApp (HsVar “show”) 
>
>
>
> (The “show” isn’t really a string, it’s the Name for the class method.)
>
>
>
> After typechecking the syntax tree is augmented (or “elaborated”) with
> type and dictionary application.  So in concrete form it might look like
>
> bar = show @Foo dShowFoo 
>
>
>
> Because show :: forall a. Show a => a -> String, so show is apply to the
> type of its argument, and then to the dictionary.
>
>
>
> In HsSyn this part is done with a HsWrapper See TcEvidence.HsWrapper.  The
> elaborated syntax tree look like
>
>
>
> HsApp (HsWrap  (HsVar “show”))
>
>   
>
>
>
> The  part expresses the type and dictionary application. In this
> case it’ll look like
>
> WpEvApp dShowFoo (WpTyApp Foo WpHole)
>
>
>
> See the notes with `HsWrapper` in TcEvidence.
>
>
>
> Does that help?
>
>
>
> It would be great to augment the https://ghc.haskell.org/trac/
> ghc/wiki/Commentary with this sort of info (insofar as it doesn’t have it
> already).  If you augment I can review.  Email is quickly lost.
>
>
>
> Simon
>
> *From:* ghc-devs [mailto:ghc-devs-boun...@haskell.org] *On Behalf Of *Robin
> Palotai
> *Sent:* 19 September 2017 06:39
> *To:* GHC developers <ghc-devs@haskell.org>; haskell <
> haskell-c...@haskell.org>
> *Subject:* Determine instance method from class method callsite
>
>
>
> Sorry, I messed up subject and mailing list. Copying to both list now
> after the mistake (wanted only ghc-devs for specificity).
>
>
>
> Thanks!
>
>
>
> 2017-09-19 7:36 GMT+02:00 Robin Palotai <palotai.ro...@gmail.com>:
>
> Hello GHC devs,
>
>
>
> Before inventing the wheel, want to check if there is a GHC API way to
> look up the (fully) resolved instance method from a class method.
>
>
>
> For example, given a code
>
>
>
> data Foo Int deriving Show
>
>
>
> bar = show (Foo 3)
>
>
>
> when inspecting the Typechecked AST for bar's show call, I would like to
> get to the Name / Id of 'show' of the 'Show' typeclass.
>
>
>
> I believe I could use splitHsSigmaTy on the HsType of the function call to
> get the context, and then evaluate the HsWrapper somehow to find out what
> instance dictionary is applied to the class restriction in the context, and
> then look up the instance method from the dictionary..
>
>
>
> Two questions:
>
>
>
> 1) Is there maybe functionality for this?
>
>
>
> 2) If not, is there any guarantee about the constraint order in the
> context, at the method call? So I could more easily determine which
> constraint's application to look for..
>
>
>
> Any hints welcome && Thank you!
>
> Robin
>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Determine instance method from class method callsite

[Robin Palotai]

My conclusion so far: there's no royal way. One can get the instance
dictionary DFunId pretty easy, but then access to the Typechecked AST of
the instance declaration is really needed to find all the method bindings
$c... (that get applied when constructing the dictionary $d...).

2017-09-19 7:38 GMT+02:00 Robin Palotai <palotai.ro...@gmail.com>:

> Sorry, I messed up subject and mailing list. Copying to both list now
> after the mistake (wanted only ghc-devs for specificity).
>
> Thanks!
>
> 2017-09-19 7:36 GMT+02:00 Robin Palotai <palotai.ro...@gmail.com>:
>
>> Hello GHC devs,
>>
>> Before inventing the wheel, want to check if there is a GHC API way to
>> look up the (fully) resolved instance method from a class method.
>>
>> For example, given a code
>>
>> data Foo Int deriving Show
>>
>> bar = show (Foo 3)
>>
>> when inspecting the Typechecked AST for bar's show call, I would like to
>> get to the Name / Id of 'show' of the 'Show' typeclass.
>>
>> I believe I could use splitHsSigmaTy on the HsType of the function call
>> to get the context, and then evaluate the HsWrapper somehow to find out
>> what instance dictionary is applied to the class restriction in the
>> context, and then look up the instance method from the dictionary..
>>
>> Two questions:
>>
>> 1) Is there maybe functionality for this?
>>
>> 2) If not, is there any guarantee about the constraint order in the
>> context, at the method call? So I could more easily determine which
>> constraint's application to look for..
>>
>> Any hints welcome && Thank you!
>> Robin
>>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Determine instance method from class method callsite

[Robin Palotai]

Sorry, I messed up subject and mailing list. Copying to both list now after
the mistake (wanted only ghc-devs for specificity).

Thanks!

2017-09-19 7:36 GMT+02:00 Robin Palotai <palotai.ro...@gmail.com>:

> Hello GHC devs,
>
> Before inventing the wheel, want to check if there is a GHC API way to
> look up the (fully) resolved instance method from a class method.
>
> For example, given a code
>
> data Foo Int deriving Show
>
> bar = show (Foo 3)
>
> when inspecting the Typechecked AST for bar's show call, I would like to
> get to the Name / Id of 'show' of the 'Show' typeclass.
>
> I believe I could use splitHsSigmaTy on the HsType of the function call to
> get the context, and then evaluate the HsWrapper somehow to find out what
> instance dictionary is applied to the class restriction in the context, and
> then look up the instance method from the dictionary..
>
> Two questions:
>
> 1) Is there maybe functionality for this?
>
> 2) If not, is there any guarantee about the constraint order in the
> context, at the method call? So I could more easily determine which
> constraint's application to look for..
>
> Any hints welcome && Thank you!
> Robin
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: ANN: Overlay Hackage Package Index for GHC HEAD

[Robin Palotai]

Sounds amazing - could you add the procedure you described to the "stock"
multi-GHC Travis example [1]?
It already has a "HEAD" env (see "env: BUILD=cabal GHCVER=head"...), but
breaks [2] due to the package problems you mentioned.

[1]:
https://raw.githubusercontent.com/commercialhaskell/stack/master/doc/travis-complex.yml
[2]: for haskell-indexer:
https://travis-ci.org/robinp/haskell-indexer/builds/276778630

2017-09-18 14:31 GMT+02:00 Matthew Pickering :

> Something like this is definitely useful for testing.
>
> When building GHC HEAD I override the ghcHEAD derivation on nixpkgs to
> the right commit I want to use and then can similarly specify which
> patches and versions of packages to use by modifying the
> 'configuration-ghc-head.nix' file.
> This is quite a bit more flexible than just patch files as I can point
> to specific commits in git repos etc.
>
> This repo will definitely be useful for me with this workflow as well.
>
> Thanks,
>
> Matt
>
> On Mon, Sep 18, 2017 at 1:04 PM, Herbert Valerio Riedel
>  wrote:
> > Hi GHC devs,
> >
> > A long-standing common problem when testing/dogfooding GHC HEAD is that
> > at some point during the development cycle more and more packages from
> > Hackage will run into build failures.
> >
> > Obviously, constantly nagging upstream maintainers to release quickfixes
> > for unreleased GHC HEAD snapshots which will likely break again a few
> > weeks later (as things are generally in flux until freeze comes into
> > effect) does not scale and only introduces a latency/coordination
> > bottleneck, and on top of it ultimately results in spamming the primary
> > Hackage Package index with releases (which has non-negligible negative
> > impact/costs of its own on the Hackage infrastructure, and thus ought to
> > be minimised).
> >
> > OTOH, we need the ability to easily test, debug, profile, and prototype
> > changes to GHC HEAD while things are still in motion, and case in point,
> > if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
> > run into a dozen or so of packages not building with GHC HEAD.
> >
> > To this end, I've finally found time to work on a side-project (related
> > to matrix.hackage.haskell.org) which implements a scheme tailored to
> > `cabal new-build`, which is inspired by how Eta copes with a very
> > related issue (they rely on it for stable versions of the compiler);
> > i.e., they maintain a set of patches at
> >
> >   https://github.com/typelead/eta-hackage/tree/master/patches
> >
> > which fix up existing Hackage packages to work with the Eta compiler.
> >
> >
> > And this gave me the idea to use a similar scheme for GHC HEAD:
> >
> >   https://github.com/hvr/head.hackage/tree/master/patches
> >
> > This folder already contains several of patches (which mostly originate
> > from Ryan, Ben and myself) to packages which I needed to patch in order
> > to build popular Hackage packages & tools.
> >
> > The main difference is how those patches are applied; Eta uses a
> > modified `cabal` which they renamed to `etlas` which is checks
> > availability of .patch & .cabal files in the GitHub repo linked above;
> >
> > Whereas for GHC HEAD with `cabal new-build` a different scheme makes
> > more sense: we simply generate an add-on Hackage repo, and use the
> > existing `cabal` facilities (e.g. multi-repo support or the nix-style
> > package store which makes sure that unofficially patched packages don't
> > contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
> > repo containing fixed up packages.
> >
> >
> >
> > I've tried to describe how to use the HEAD.hackage add-on repo in the
> > README at
> >
> >   https://github.com/hvr/head.hackage#how-to-use
> >
> >
> > And finally, here's a practical example of how you can use it to build
> > e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
> > build your project of choice; please refer to
> >
> >   http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html
> >
> > to learn more about how to describe your project via  `cabal.project`
> > files):
> >
> >
> > 0.) This assumes you have a recent cabal 2.1 snapshot built from Git
> >
> > 1.) create & `cd` into a new work-folder
> >
> > 2.) invoke `head.hackage.sh update` to update the HEAD.hackage index
> cache
> >
> > 3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
> > configuration which locally activates the HEAD.hackage overlay repo
> >
> > 4.) If needed, edit the cabal.project file and change where GHC
> > HEAD can be found (the script currently assumes GHC HEAD is
> > installed from my Ubuntu PPA), e.g.
> >
> >   with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2
> >
> > or you can add something like `optional-packages: deps/*/*.cabal`
> > to have cabal pick up package source-trees unpacked in the deps/
> > folder, or you can inject ghc-options, relax upper bounds via
> > 

Re: Crossreferenced GHC 8.0.2

[Robin Palotai]

Sorry for the delay - added it, see for example
http://stuff.codereview.me/#../logs/Control.Monad.Skeleton.core?corpus=core-kythe=41
.

Note that the Haskell sources (reachable by following the 'generates'
reverse edge from the Core) seem to have the empty corpus (for example
http://stuff.codereview.me/#F.hs?corpus=1).

(Also, one of the example entries had a serialization error, omitted that
one).

Looks interesting!

2017-08-17 19:22 GMT+02:00 Matthew Pickering <matthewtpicker...@gmail.com>:

> Hi all,
>
> If anyone is interested in this and using it further I have been
> working on two improvements.
>
> 1. A cross-referencer in the same style for the output of -ddump-simpl
> (which also links to the source code which produced the core).
>
> 2. A nix function which builds and references all dependencies.
>
> They are both found in my core-kythe repo with instructions about how
> to use them.
>
> https://github.com/mpickering/core-kythe
>
> You will need nix in order to use either of them but if you do it
> should be straightforward (albeit long in the first case!)
>
> I don't have a server to post examples to but Robin said he would when
> he got the chance.
>
> Matt
>
>
> On Tue, Jul 4, 2017 at 6:44 AM, Robin Palotai <palotai.ro...@gmail.com>
> wrote:
> > FYI I added GHC 8.2.1-rc2 source to the index. Please tell if some source
> > you would be interested in is obviously missing. Thanks!
> >
> > 2017-06-30 22:41 GMT+02:00 Robin Palotai <palotai.ro...@gmail.com>:
> >>
> >> Hello Matthew,
> >>
> >> Please see inline
> >>
> >> 2017-06-30 11:57 GMT+02:00 Matthew Pickering
> >> <matthewtpicker...@gmail.com>:
> >>>
> >>> Hi Robin,
> >>>
> >>> This looks really useful for developers.
> >>>
> >>> 1. Would it be possible to provide a script which allows developers to
> >>> build this index for themselves easily?
> >>
> >>
> >> First, build and install the `ghc_kythe_wrapper` (instructions at
> >> https://github.com/google/haskell-indexer).
> >>
> >> Then build GHC with `make`, and capture the build log. Here's the hacky
> >> script I used:
> >> https://gist.github.com/robinp/222cf3a39cc19178ec8691522056d7fe
> >>
> >> It filters the log and replaces GHC calls to call the wrapper, which
> emits
> >> Kythe entries.
> >>
> >> Finally run `serve.sh` of the repo to postprocess and serve the entries
> >> through HTTP.
> >>
> >> This is all pretty new, so feedback or questions welcome. If the method
> >> distills, could try to write a more formal guide.
> >>
> >>>
> >>> 2. Is it possible to use this tool to detect dead code? Functions
> >>> which are not used anywhere in the compiler.
> >>
> >>
> >> We'll get there eventually, but for now the emitted data is not
> >> fine-grained enough. The main missing piece is recording what are
> exported
> >> entities of a module (Kythe schema discussion in progress). Without
> this,
> >> unused locals (which anyway surface with -Wall) would be presented and
> >> noisy.
> >>
> >> Also, one would probably need to postprocess the data a bit for this,
> like
> >> loading into a graph database or other ways.
> >>
> >> Did you see https://github.com/ndmitchell/weeder by the way? Might
> work.
> >>
> >>>
> >>> 3. How are you pretty printing the output whilst retaining the source
> >>> formatting? I had a quick look at the source but I couldn't see where
> >>> the output was being produced.
> >>>
> >>
> >> The haskell-indexer-frontend-kythe emits Kythe (http://kythe.io) schema
> >> data. We just export the source offsets, and it's the Kythe
> postprocessing /
> >> serving pipeline that does all the formatting.
> >>
> >>>
> >>> Cheers,
> >>>
> >>> Matt
> >>>
> >>> On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai <
> palotai.ro...@gmail.com>
> >>> wrote:
> >>> > Hello GHC devs,
> >>> >
> >>> > I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I
> >>> > find
> >>> > myself reading GHC source from time to time while working on the
> >>> > indexer,
> >>> > and partly since it's fun.
> >>> >
> >>> > First, here you can click around [2] and find where beloved

Re: Compilation time

[Robin Palotai]

Note that Travis CI time is a sum of all operations, including
fetching/saving caches etc.

Opening the build log and looking for the compilation might be more
revealing. For example, 7.10 seems ~10 seconds, 8.2.1 seems ~50 seconds.

There are two issues I can see with this, that should be addressed by
benchmarking:

1) The Travis CI build hosts might show varying performance across reruns
due to shared machines.

2) Tuning the RTS params when invoking GHC as a compiler can yield
significant benefits, at least on larger compilations (such as -A128m).

It would probably be doable to fetch a set of packages from *ackage and
build with various compilers under controlled circumstances?

Robin

2017-07-04 9:16 GMT+02:00 Deest, Gaël :

> Hi all,
>
> As you are probably well aware, GHC performance has been a growing concern
> over the last few years. Many Haskell programmers complain that build time
> has significantly increased over the last few releases. However, to our
> knowledge, there isn't much data available to substantiate this claim and
> the severity of these problems is not well known.
>
> That's why we would like to bring some anecdotal evidence to your
> attention that seems to indicate really major performance regressions. We
> stumbled upon the CI of the data-reify package, which is built against all
> GHC releases since 2010 :
>
> https://travis-ci.org/ku-fpg/data-reify
>
> tl;dr: Build time has gone from 1 min 32s for GHC 7.0 to 4 min 35s for GHC
> 8.2. The 8.2 release alone seems to have increased compilation time by
> almost 2 minutes, with the current development branch bringing only minor
> performance improvements.
>
> Of course, this single data point is not sufficient to establish how
> severe and widespread these problems are. More data could probably be
> gathered from other packages. However, it certainly matches our
> (subjective) experience and we felt important to report it to you.
>
> Regards,
>
> --
> Gaël Deest
> Tweag I/O
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Crossreferenced GHC 8.0.2

[Robin Palotai]

FYI I added GHC 8.2.1-rc2 source to the index. Please tell if some source
you would be interested in is obviously missing. Thanks!

2017-06-30 22:41 GMT+02:00 Robin Palotai <palotai.ro...@gmail.com>:

> Hello Matthew,
>
> Please see inline
>
> 2017-06-30 11:57 GMT+02:00 Matthew Pickering <matthewtpicker...@gmail.com>
> :
>
>> Hi Robin,
>>
>> This looks really useful for developers.
>>
>> 1. Would it be possible to provide a script which allows developers to
>> build this index for themselves easily?
>>
>
> First, build and install the `ghc_kythe_wrapper` (instructions at
> https://github.com/google/haskell-indexer).
>
> Then build GHC with `make`, and capture the build log. Here's the hacky
> script I used: https://gist.github.com/robinp/
> 222cf3a39cc19178ec8691522056d7fe
>
> It filters the log and replaces GHC calls to call the wrapper, which emits
> Kythe entries.
>
> Finally run `serve.sh` of the repo to postprocess and serve the entries
> through HTTP.
>
> This is all pretty new, so feedback or questions welcome. If the method
> distills, could try to write a more formal guide.
>
>
>> 2. Is it possible to use this tool to detect dead code? Functions
>> which are not used anywhere in the compiler.
>>
>
> We'll get there eventually, but for now the emitted data is not
> fine-grained enough. The main missing piece is recording what are exported
> entities of a module (Kythe schema discussion in progress). Without this,
> unused locals (which anyway surface with -Wall) would be presented and
> noisy.
>
> Also, one would probably need to postprocess the data a bit for this, like
> loading into a graph database or other ways.
>
> Did you see https://github.com/ndmitchell/weeder by the way? Might work.
>
>
>> 3. How are you pretty printing the output whilst retaining the source
>> formatting? I had a quick look at the source but I couldn't see where
>> the output was being produced.
>>
>>
> The haskell-indexer-frontend-kythe emits Kythe (http://kythe.io) schema
> data. We just export the source offsets, and it's the Kythe postprocessing
> / serving pipeline that does all the formatting.
>
>
>> Cheers,
>>
>> Matt
>>
>> On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai <palotai.ro...@gmail.com>
>> wrote:
>> > Hello GHC devs,
>> >
>> > I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I
>> find
>> > myself reading GHC source from time to time while working on the
>> indexer,
>> > and partly since it's fun.
>> >
>> > First, here you can click around [2] and find where beloved functions
>> are
>> > called from:
>> > http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?co
>> rpus
>> > (scroll down a bit, imports are not linked yet).
>> >
>> > Second, the way I indexed was pretty simple. I took the output of
>> `make`,
>> > replaced the ghc used with the ghc_kythe_wrapper, and filtered the lines
>> > which included '-c', since I noticed that those duplicate previous large
>> > compile lines. This only indexes the stage1 compilation AFAIU.
>> >
>> > Feel free to suggest a better way to tap into the compilations to get
>> > everything properly indexed (and possibly only once).
>> >
>> > Any comments welcome!
>> > Robin
>> >
>> > [1]: https://github.com/google/haskell-indexer
>> >
>> > [2]: TLDR UI quirks:
>> >   - Click the :: in top-left to navigate file tree
>> >   - Ctrl-Click (on linux) to go directly to definition (otherwise click
>> > stuff from bottom pane)
>> >   - Bottom pane often hides content, close it if stuck.
>> >
>> > ___
>> > ghc-devs mailing list
>> > ghc-devs@haskell.org
>> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>> >
>>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Crossreferenced GHC 8.0.2

[Robin Palotai]

Hello Matthew,

Please see inline

2017-06-30 11:57 GMT+02:00 Matthew Pickering <matthewtpicker...@gmail.com>:

> Hi Robin,
>
> This looks really useful for developers.
>
> 1. Would it be possible to provide a script which allows developers to
> build this index for themselves easily?
>

First, build and install the `ghc_kythe_wrapper` (instructions at
https://github.com/google/haskell-indexer).

Then build GHC with `make`, and capture the build log. Here's the hacky
script I used:
https://gist.github.com/robinp/222cf3a39cc19178ec8691522056d7fe

It filters the log and replaces GHC calls to call the wrapper, which emits
Kythe entries.

Finally run `serve.sh` of the repo to postprocess and serve the entries
through HTTP.

This is all pretty new, so feedback or questions welcome. If the method
distills, could try to write a more formal guide.


> 2. Is it possible to use this tool to detect dead code? Functions
> which are not used anywhere in the compiler.
>

We'll get there eventually, but for now the emitted data is not
fine-grained enough. The main missing piece is recording what are exported
entities of a module (Kythe schema discussion in progress). Without this,
unused locals (which anyway surface with -Wall) would be presented and
noisy.

Also, one would probably need to postprocess the data a bit for this, like
loading into a graph database or other ways.

Did you see https://github.com/ndmitchell/weeder by the way? Might work.


> 3. How are you pretty printing the output whilst retaining the source
> formatting? I had a quick look at the source but I couldn't see where
> the output was being produced.
>
>
The haskell-indexer-frontend-kythe emits Kythe (http://kythe.io) schema
data. We just export the source offsets, and it's the Kythe postprocessing
/ serving pipeline that does all the formatting.


> Cheers,
>
> Matt
>
> On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai <palotai.ro...@gmail.com>
> wrote:
> > Hello GHC devs,
> >
> > I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I find
> > myself reading GHC source from time to time while working on the indexer,
> > and partly since it's fun.
> >
> > First, here you can click around [2] and find where beloved functions are
> > called from:
> > http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?
> corpus
> > (scroll down a bit, imports are not linked yet).
> >
> > Second, the way I indexed was pretty simple. I took the output of `make`,
> > replaced the ghc used with the ghc_kythe_wrapper, and filtered the lines
> > which included '-c', since I noticed that those duplicate previous large
> > compile lines. This only indexes the stage1 compilation AFAIU.
> >
> > Feel free to suggest a better way to tap into the compilations to get
> > everything properly indexed (and possibly only once).
> >
> > Any comments welcome!
> > Robin
> >
> > [1]: https://github.com/google/haskell-indexer
> >
> > [2]: TLDR UI quirks:
> >   - Click the :: in top-left to navigate file tree
> >   - Ctrl-Click (on linux) to go directly to definition (otherwise click
> > stuff from bottom pane)
> >   - Bottom pane often hides content, close it if stuck.
> >
> > ___
> > ghc-devs mailing list
> > ghc-devs@haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> >
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Crossreferenced GHC 8.0.2

[Robin Palotai]

Hello GHC devs,

I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I find
myself reading GHC source from time to time while working on the indexer,
and partly since it's fun.

First, here you can click around [2] and find where beloved functions are
called from:
http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?corpus
(scroll down a bit, imports are not linked yet).

Second, the way I indexed was pretty simple. I took the output of `make`,
replaced the ghc used with the ghc_kythe_wrapper, and filtered the lines
which included '-c', since I noticed that those duplicate previous large
compile lines. This only indexes the stage1 compilation AFAIU.

Feel free to suggest a better way to tap into the compilations to get
everything properly indexed (and possibly only once).

Any comments welcome!
Robin

[1]: https://github.com/google/haskell-indexer

[2]: TLDR UI quirks:
  - Click the :: in top-left to navigate file tree
  - Ctrl-Click (on linux) to go directly to definition (otherwise click
stuff from bottom pane)
  - Bottom pane often hides content, close it if stuck.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs