Re: Extracting representation from GHC

[Robin Palotai]

See also
https://github.com/google/haskell-indexer/blob/master/haskell-indexer-backend-ghc/src/Language/Haskell/Indexer/Backend/GhcApiSupport.hs
for an as-complete GHC API based setup as I could get. The comments
indicate possible deficiencies.

The test
https://github.com/google/haskell-indexer/blob/master/haskell-indexer-backend-ghc/tests/Language/Haskell/Indexer/Backend/Ghc/Test/BasicTestBase.hs#L179
shows some cases that are covered (for example testTemplateHaskellCodeExecFFI
).

In practice one can run AST extraction with HscNoLink + HscInterpreted for
most targets, but for hairy ones (FFI invoked from TemplateHaskell in
certain ways, for examples) that will fail.

2018-01-19 19:46 GMT+01:00 Robin Palotai :

> See some additions inline.
> BR
> Robin
>
> 2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs <
> ghc-devs@haskell.org>:
>
>> |  To do this, my idea is to instruct GHC with a compilation flag to give
>> |  out its internal representation of the source code.
>>
>> Why can't you just use GHC as a library, and ask it to parse and
>> typecheck the module and then look at what it gives you.
>>
>> Last time I checked (GHC 8.2, for haskell-indexer), using the library is
> not equivalent to using GHC's Main. GHC's Main does tremendous amount of
> magic with flag parsing and state setup, and doesn't expose all the
> functionality for libraries to do the same.
>
> AFAIR we saw two possible ways to get the AST out from a complicated setup
> (FFI, objects, packages, ...):
> 1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more
> limited at the time - the gist in the below trac entry mentions that even
> the Frontend plugin didn't do everything Main does).
> 2) Refactor GHC Main and expose all the functionality to GHC API.
>
> I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's
> slightly related.
>
> By the way, you can click around http://stuff.codereview.me/ghc/#ghc/ghc/
> Main.hs?corpus=ghc-8.2.1-rc2 in the 'main' function to see all
> the magic.
>
> Others are more used to the GHC API than me, though.
>>
>> S
>>
>> |  -Original Message-
>> |  From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of
>> |  Németh Boldizsár
>> |  Sent: 19 January 2018 09:35
>> |  To: ghc-devs@haskell.org
>> |  Subject: Extracting representation from GHC
>> |
>> |  Dear GHC Developers,
>> |
>> |  I would like to ask your opinion on my ideas to make it easier to use
>> |  development tools with GHC.
>> |
>> |  In the past when working on a Haskell refactoring tool I relied on
>> |  using the GHC API for parsing and type checking Haskell sources. I
>> |  extracted the representation and performed analysis and transformation
>> |  on it as it was needed. However using the refactorer would be easier
>> |  if it could work with build tools.
>> |
>> |  To do this, my idea is to instruct GHC with a compilation flag to give
>> |  out its internal representation of the source code. Most build tools
>> |  let the user to configure the GHC flags so the refactoring tool would
>> |  be usable in any build infrastructure. I'm thinking of using the pre-
>> |  existing plugin architecture and adding two new fields to the Plugin
>> |  datastructure. One would be called with the parsed representation
>> |  (HsParsedModule) when parsing succeeds, another with the result of the
>> |  type checking (TcGblEnv) when type checking is finished.
>> |
>> |  What do you think about this solution?
>> |
>> |  Boldizsár
>> |
>> |  (ps: My first idea was using frontend plugins, but I could not access
>> |  the representation from there and --frontend flag changed GHC
>> |  compilation mode.)
>> |
>> |  ___
>> |  ghc-devs mailing list
>> |  ghc-devs@haskell.org
>> |  https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h
>> |  askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
>> |  devs=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5
>> |  5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471
>> <(513)%20536-0471>
>> |  87=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D=0
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Extracting representation from GHC

[Robin Palotai]

See some additions inline.
BR
Robin

2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs <
ghc-devs@haskell.org>:

> |  To do this, my idea is to instruct GHC with a compilation flag to give
> |  out its internal representation of the source code.
>
> Why can't you just use GHC as a library, and ask it to parse and typecheck
> the module and then look at what it gives you.
>
> Last time I checked (GHC 8.2, for haskell-indexer), using the library is
not equivalent to using GHC's Main. GHC's Main does tremendous amount of
magic with flag parsing and state setup, and doesn't expose all the
functionality for libraries to do the same.

AFAIR we saw two possible ways to get the AST out from a complicated setup
(FFI, objects, packages, ...):
1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more
limited at the time - the gist in the below trac entry mentions that even
the Frontend plugin didn't do everything Main does).
2) Refactor GHC Main and expose all the functionality to GHC API.

I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's
slightly related.

By the way, you can click around
http://stuff.codereview.me/ghc/#ghc/ghc/Main.hs?corpus=ghc-8.2.1-rc2
in the 'main' function to see all the magic.

Others are more used to the GHC API than me, though.
>
> S
>
> |  -Original Message-
> |  From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of
> |  Németh Boldizsár
> |  Sent: 19 January 2018 09:35
> |  To: ghc-devs@haskell.org
> |  Subject: Extracting representation from GHC
> |
> |  Dear GHC Developers,
> |
> |  I would like to ask your opinion on my ideas to make it easier to use
> |  development tools with GHC.
> |
> |  In the past when working on a Haskell refactoring tool I relied on
> |  using the GHC API for parsing and type checking Haskell sources. I
> |  extracted the representation and performed analysis and transformation
> |  on it as it was needed. However using the refactorer would be easier
> |  if it could work with build tools.
> |
> |  To do this, my idea is to instruct GHC with a compilation flag to give
> |  out its internal representation of the source code. Most build tools
> |  let the user to configure the GHC flags so the refactoring tool would
> |  be usable in any build infrastructure. I'm thinking of using the pre-
> |  existing plugin architecture and adding two new fields to the Plugin
> |  datastructure. One would be called with the parsed representation
> |  (HsParsedModule) when parsing succeeds, another with the result of the
> |  type checking (TcGblEnv) when type checking is finished.
> |
> |  What do you think about this solution?
> |
> |  Boldizsár
> |
> |  (ps: My first idea was using frontend plugins, but I could not access
> |  the representation from there and --frontend flag changed GHC
> |  compilation mode.)
> |
> |  ___
> |  ghc-devs mailing list
> |  ghc-devs@haskell.org
> |  https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h
> |  askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
> |  devs=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5
> |  5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471
> |  87=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D=0
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: Why is EvTerm limited?

[Simon Peyton Jones via ghc-devs]

|  What would break if we had
|  
|   | EvExpr CoreExpr
|  
|  as an additional constructor there?

This has come up before.  I think that'd be a solid win. 

In fact, eliminate all the existing evidence constructors with "smart 
constructors" that produce an EvExpr.  That'd mean moving stuff from the 
desugarer into these smart constructors, but that's ok.

I /think/ I didn't do that initially only because there were very few forms and 
it mean that there was no CoreExpr stuff in the type checker. But as we add 
more forms that decision looks and less good.

You'd need to add zonkCoreExpr in place of zonkEvTerm.

evVarsOfTerm is called quite a bit; you might want to cache the result in the 
EvExpr constructor.

Make a ticket and execute?

Simon

|  -Original Message-
|  From: Glasgow-haskell-users [mailto:glasgow-haskell-users-
|  boun...@haskell.org] On Behalf Of Joachim Breitner
|  Sent: 19 January 2018 14:01
|  To: Glasgow-Haskell-Users users 
|  Subject: Why is EvTerm limited?
|  
|  Hi,
|  
|  I had some funky idea where a type checker plugin would have to
|  synthesize code for a custom-solved instances on the fly. But it seems
|  that does not work because EvTerm is less expressive than Core
|  (especially, no lambdas):
|  https://na01.safelinks.protection.outlook.com/?url=https:%2F%2Fdownloa
|  ds.haskell.org%2F~ghc%2F8.2.2%2Fdocs%2Fhtml%2Flibraries%2Fghc-
|  8.2.2%2FTcEvidence.html%23t:EvTerm=02%7C01%7Csimonpj%40microsoft.
|  com%7C513ff7ae83914913225008d55f452dec%7C72f988bf86f141af91ab2d7cd011d
|  b47%7C1%7C0%7C636519673089385423=kFkUugVn02Nfu4QXJ6dkVwtx8KWFrTM
|  fWcVEiwf6KyI%3D=0
|  
|  What would break if we had
|  
|   | EvExpr CoreExpr
|  
|  as an additional constructor there?
|  
|  Cheers,
|  Joachim
|  
|  --
|  Joachim “nomeata” Breitner
|m...@joachim-breitner.de
|  
|  https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.j
|  oachim-
|  breitner.de%2F=02%7C01%7Csimonpj%40microsoft.com%7C513ff7ae839149
|  13225008d55f452dec%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636519
|  673089385423=Vh4BvbeEVUBIntKcf3XEseOzwUTx2RHPuANTY328dpM%3D
|  rved=0
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Extracting representation from GHC

[Shao Cheng]

Hi,

IIRC you can already use hscFrontendHook in the DynFlags hooks to retrieve
TcGblEnv, and with a little bit of work, also HsParsedModule.

Regards,
Shao Cheng

On Fri, Jan 19, 2018, 5:41 PM Matthew Pickering 
wrote:

> I have too wanted this in the past and made a post to a similar effect
> on the mailing list 6 months ago.
>
> https://mail.haskell.org/pipermail/ghc-devs/2017-July/014427.html
>
> It references this proposal for a similar feature.
>
> https://ghc.haskell.org/trac/ghc/wiki/FrontendPluginsProposal#no1
>
> If you would be glad to implement it then feel free to add me as a
> reviewer.
>
> Cheers,
>
> Matt
>
> On Fri, Jan 19, 2018 at 9:35 AM, Németh Boldizsár  wrote:
> > Dear GHC Developers,
> >
> > I would like to ask your opinion on my ideas to make it easier to use
> > development tools with GHC.
> >
> > In the past when working on a Haskell refactoring tool I relied on using
> the
> > GHC API for parsing and type checking Haskell sources. I extracted the
> > representation and performed analysis and transformation on it as it was
> > needed. However using the refactorer would be easier if it could work
> with
> > build tools.
> >
> > To do this, my idea is to instruct GHC with a compilation flag to give
> out
> > its internal representation of the source code. Most build tools let the
> > user to configure the GHC flags so the refactoring tool would be usable
> in
> > any build infrastructure. I'm thinking of using the pre-existing plugin
> > architecture and adding two new fields to the Plugin datastructure. One
> > would be called with the parsed representation (HsParsedModule) when
> parsing
> > succeeds, another with the result of the type checking (TcGblEnv) when
> type
> > checking is finished.
> >
> > What do you think about this solution?
> >
> > Boldizsár
> >
> > (ps: My first idea was using frontend plugins, but I could not access the
> > representation from there and --frontend flag changed GHC compilation
> mode.)
> >
> > ___
> > ghc-devs mailing list
> > ghc-devs@haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Extracting representation from GHC

[Németh Boldizsár]

Dear GHC Developers,

I would like to ask your opinion on my ideas to make it easier to use 
development tools with GHC.


In the past when working on a Haskell refactoring tool I relied on using 
the GHC API for parsing and type checking Haskell sources. I extracted 
the representation and performed analysis and transformation on it as it 
was needed. However using the refactorer would be easier if it could 
work with build tools.


To do this, my idea is to instruct GHC with a compilation flag to give 
out its internal representation of the source code. Most build tools let 
the user to configure the GHC flags so the refactoring tool would be 
usable in any build infrastructure. I'm thinking of using the 
pre-existing plugin architecture and adding two new fields to the Plugin 
datastructure. One would be called with the parsed representation 
(HsParsedModule) when parsing succeeds, another with the result of the 
type checking (TcGblEnv) when type checking is finished.


What do you think about this solution?

Boldizsár

(ps: My first idea was using frontend plugins, but I could not access 
the representation from there and --frontend flag changed GHC 
compilation mode.)


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs