RE: TTG hsSyn for Batch and Interactive Parsing

[Simon Peyton Jones via ghc-devs]

If we are always going to generate a parse tree with annotations from the 
parser, let’s not generate two!

I’m fine with always generating the annotations, but we just need to check that 
it doesn’t have insupportable costs.

Simon

From: Alan & Kim Zimmerman <alan.z...@gmail.com>
Sent: 18 May 2018 15:31
To: Simon Peyton Jones <simo...@microsoft.com>
Cc: ghc-devs <ghc-devs@haskell.org>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing



On 18 May 2018 at 16:13, Simon Peyton Jones 
<simo...@microsoft.com<mailto:simo...@microsoft.com>> wrote:

We can’t make the result type depend on DynFlags! (Yet)(

parser :: DynFlags -> String

  -> HsSyn (GhcPass (Parsed (if …

then WithApiAnnotations

else WihoutsApiAnnotations)

We could conceptually have

  parser :: DynFlags -> String -> Either (HsSyn (Parsed WithApiAnnotations)) 
(HsSyn (Parsed NoApiAnnotations))

The main point is that the next phase can make use of either of the variants.

And it may be simplest to just always use the annotations, the ParsedSource is 
normally discarded after renaming anyway.

Alan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: TTG hsSyn for Batch and Interactive Parsing

[Alan & Kim Zimmerman]

On 18 May 2018 at 16:13, Simon Peyton Jones  wrote:

>
>
> We can’t make the result type depend on DynFlags! (Yet)(
>
> *parser :: DynFlags -> String *
>
> *  -> HsSyn (GhcPass (Parsed (if … *
>
> *then WithApiAnnotations*
>
> *else WihoutsApiAnnotations)*
>
>
We could conceptually have

  parser :: DynFlags -> String -> Either (HsSyn (Parsed
WithApiAnnotations)) (HsSyn (Parsed NoApiAnnotations))

The main point is that the next phase can make use of either of the
variants.

And it may be simplest to just always use the annotations, the ParsedSource
is normally discarded after renaming anyway.

Alan
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: TTG hsSyn for Batch and Interactive Parsing

[Simon Peyton Jones via ghc-devs]

This sounds fine.  But I still don’t understand

There would still be a single parser definition in Parser.y, which would make 
use of functions to add the additional info to the generated source tree, which 
would be NOPs if the information was not being kept. This is similar to what 
happens at present with the Api Annotations.

What is the type of the parser?  Does it produce

parser :: String -> HsSyn (GhcPass (Parsed WithApiAnnotation)
or

parser :: String -> HsSyn (GhcPass (Parsed WithoutApiAnnotation)
?

We can’t make the result type depend on DynFlags! (Yet)(

parser :: DynFlags -> String

  -> HsSyn (GhcPass (Parsed (if …

then WithApiAnnotations

else WihoutsApiAnnotations)

So I’m puzzled.

Nomenclature.  I’d say “NoApiAnnotations” rather than “WithoutApiAnnotations”.

Also: do we have data to show that it’s not OK to always keep API annotations.  
 That would be simpler, wouldn’t it.

Incidentally the Haddock stuff, decorations of type (Maybe LHsDocString) 
somehow belong in this world too, don’t they?

SImon


From: Alan & Kim Zimmerman <alan.z...@gmail.com>
Sent: 10 May 2018 22:35
To: Simon Peyton Jones <simo...@microsoft.com>
Cc: ghc-devs <ghc-devs@haskell.org>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

I have updated the Wiki with the clearer names, and noted that a single parser 
definition would still be used, as at present, but would only keep the extra 
info if it was requested to.

The naming around interactive and batch is to anticipate a further step I would 
like to take, to make the parser fully incremental, in the sense that it would 
process as input the prior parse tree and a list of changes to the source, and 
then generate a fresh parse tree, with the changed nodes marked.  This mode 
would be tightly coupled to an external too like haskell-ide-engine, to manage 
the bookkeeping around this.

My thinking for this is to use the approach presented in the paper "Efficient 
and Flexible Incremental Parsing" by Wagner and Graham[1].
The plan is to modify `happy`, so that we can reuse the existing GHC Parser.y 
with minor modifications. This is the same approach as used in the library 
tree-sitter[2], which is a very active project on github.
WIP is at [3], but it is very early stage.

Regards
  Alan


[1] 
https://pdfs.semanticscholar.org/4d22/fab95c78b3c23fa9dff88fb82976edc213c2.pdf<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpdfs.semanticscholar.org%2F4d22%2Ffab95c78b3c23fa9dff88fb82976edc213c2.pdf=02%7C01%7Csimonpj%40microsoft.com%7C186cfaf7a10a4663ce2f08d5b6bdf0f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636615849263176553=itxDiaFAuZY4MH1oZDKOudKLNcsqrm7cywA9vt4Y1DI%3D=0>
[2] 
https://github.com/tree-sitter/tree-sitter<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftree-sitter%2Ftree-sitter=02%7C01%7Csimonpj%40microsoft.com%7C186cfaf7a10a4663ce2f08d5b6bdf0f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636615849263176553=HmYE2FMO97ouuAR1vQHPfaEooI1oxtcU4aNWNg3E9T8%3D=0>
[3] 
https://github.com/alanz/happy/tree/incremental<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Falanz%2Fhappy%2Ftree%2Fincremental=02%7C01%7Csimonpj%40microsoft.com%7C186cfaf7a10a4663ce2f08d5b6bdf0f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636615849263332819=3RcrS%2FMH53aENrS%2FFmdKt%2Fc9JUjIeEq6h5xugF7HmhU%3D=0>


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: TTG hsSyn for Batch and Interactive Parsing

[Alan & Kim Zimmerman]

I have updated the Wiki with the clearer names, and noted that a single
parser definition would still be used, as at present, but would only keep
the extra info if it was requested to.

The naming around interactive and batch is to anticipate a further step I
would like to take, to make the parser fully incremental, in the sense that
it would process as input the prior parse tree and a list of changes to the
source, and then generate a fresh parse tree, with the changed nodes
marked.  This mode would be tightly coupled to an external too like
haskell-ide-engine, to manage the bookkeeping around this.

My thinking for this is to use the approach presented in the paper
"Efficient and Flexible Incremental Parsing" by Wagner and Graham[1].
The plan is to modify `happy`, so that we can reuse the existing GHC
Parser.y with minor modifications. This is the same approach as used in the
library tree-sitter[2], which is a very active project on github.

WIP is at [3], but it is very early stage.

Regards
  Alan


[1] https://pdfs.semanticscholar.org/4d22/fab95c78b3c23fa9dff88fb82976ed
c213c2.pdf
[2] https://github.com/tree-sitter/tree-sitter
[3] https://github.com/alanz/happy/tree/incremental
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: TTG hsSyn for Batch and Interactive Parsing

[Simon Peyton Jones via ghc-devs]

Thanks.  But I still don’t see the connection with “interactive”.  Why should 
maintaining API annotations have anything to do with interactivity?


Maybe

  data Process = WithApiAnnotations | WithoutApiAnnotations



I understand you want two different variants of the syntax tree, but I don’t 
understand what functions might produce or consume them.



In particular does the parser produce (HsSyn (GhcPs WithApiAnnotations)) or 
without?



Simon






From: Alan & Kim Zimmerman <alan.z...@gmail.com>
Sent: 09 May 2018 21:12
To: Simon Peyton Jones <simo...@microsoft.com>
Cc: ghc-devs <ghc-devs@haskell.org>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

I have updated the Wiki.

On 9 May 2018 at 10:15, Simon Peyton Jones 
<simo...@microsoft.com<mailto:simo...@microsoft.com>> wrote:
Thanks.

I am absolutely behind this objective:
I propose to move the API Annotations to where they belong, inside the AST.
Indeed I thought that was always part of the TTG plan.

But I don’t understand what this has to do with interactive vs batch parsing.  
Why don’t you unconditionally retain API-annotation info?  How would GhcPs be 
used differently to GhcPsI?

You might want to answer by clarifying on the wiki page, so that it is a 
persistent record of the design debugged in dialogue by email.

Simon

From: Alan & Kim Zimmerman <alan.z...@gmail.com<mailto:alan.z...@gmail.com>>
Sent: 08 May 2018 21:02
To: Simon Peyton Jones <simo...@microsoft.com<mailto:simo...@microsoft.com>>
Cc: ghc-devs <ghc-devs@haskell.org<mailto:ghc-devs@haskell.org>>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

I have started a wiki page at 
https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/IdeSupport

On 8 May 2018 at 10:54, Simon Peyton Jones 
<simo...@microsoft.com<mailto:simo...@microsoft.com>> wrote:
At first blush, “running the parser in two modes” and “changing the Pass” type 
don’t match up in my mind.  One seems quite local (how to run the parser).  The 
other seems more pervasive.

Can you say more about your proposed design, perhaps even on a wiki page?

Simon

From: ghc-devs 
<ghc-devs-boun...@haskell.org<mailto:ghc-devs-boun...@haskell.org>> On Behalf 
Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <ghc-devs@haskell.org<mailto:ghc-devs@haskell.org>>
Subject: TTG hsSyn for Batch and Interactive Parsing

I want to be able to run the GHC parser in one of two modes, batch which 
functions as now, and interactive which will (eventually) be incremental.
In addition, the hsSyn AST for each will have different TTG[1] annotations, so 
that it can better support IDE usage.
I think this can be done by changing the types in HsExtension to introduce a 
'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
 deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)
We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)
I have attached a simple proof of concept file, which emulates parsing and 
renaming.
Is this an appropriate approach to take?
Alan

[1] Trees That Grow 
https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow


___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: TTG hsSyn for Batch and Interactive Parsing

[Alan & Kim Zimmerman]

I have updated the Wiki.

On 9 May 2018 at 10:15, Simon Peyton Jones <simo...@microsoft.com> wrote:

> Thanks.
>
>
>
> I am absolutely behind this objective:
>
> I propose to move the API Annotations to where they belong, inside the AST.
>
> Indeed I thought that was always part of the TTG plan.
>
>
>
> *But I don’t understand what this has to do with interactive vs batch
> parsing.  *Why don’t you unconditionally retain API-annotation info?  How
> would GhcPs be used differently to GhcPsI?
>
>
>
> You might want to answer by clarifying on the wiki page, so that it is a
> persistent record of the design debugged in dialogue by email.
>
>
>
> Simon
>
>
>
> *From:* Alan & Kim Zimmerman <alan.z...@gmail.com>
> *Sent:* 08 May 2018 21:02
> *To:* Simon Peyton Jones <simo...@microsoft.com>
> *Cc:* ghc-devs <ghc-devs@haskell.org>
> *Subject:* Re: TTG hsSyn for Batch and Interactive Parsing
>
>
>
> I have started a wiki page at https://ghc.haskell.org/trac/ghc/wiki/
> ImplementingTreesThatGrow/IdeSupport
>
>
>
> On 8 May 2018 at 10:54, Simon Peyton Jones <simo...@microsoft.com> wrote:
>
> At first blush, “running the parser in two modes” and “changing the Pass”
> type don’t match up in my mind.  One seems quite local (how to run the
> parser).  The other seems more pervasive.
>
>
>
> Can you say more about your proposed design, perhaps even on a wiki page?
>
>
>
> Simon
>
>
>
> *From:* ghc-devs <ghc-devs-boun...@haskell.org> *On Behalf Of *Alan & Kim
> Zimmerman
> *Sent:* 07 May 2018 16:17
> *To:* ghc-devs <ghc-devs@haskell.org>
> *Subject:* TTG hsSyn for Batch and Interactive Parsing
>
>
>
> I want to be able to run the GHC parser in one of two modes, batch which
> functions as now, and interactive which will (eventually) be incremental.
>
> In addition, the hsSyn AST for each will have different TTG[1]
> annotations, so that it can better support IDE usage.
>
> I think this can be done by changing the types in HsExtension to introduce
> a 'Process'  type as follows
>
> data Pass = Parsed Process | Renamed | Typechecked
>  deriving (Data)
>
> data Process = Batch | Interactive
>   deriving (Show, Data)
>
> We then rename the pass synonyms so that batch is the default
>
> type GhcPs   = GhcPass ('Parsed 'Batch)
> type GhcPsI  = GhcPass ('Parsed 'Interactive)
>
> I have attached a simple proof of concept file, which emulates parsing and
> renaming.
>
> Is this an appropriate approach to take?
>
> Alan
>
>
>
> [1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/
> ImplementingTreesThatGrow
>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: TTG hsSyn for Batch and Interactive Parsing

[Simon Peyton Jones via ghc-devs]

Thanks.

I am absolutely behind this objective:
I propose to move the API Annotations to where they belong, inside the AST.
Indeed I thought that was always part of the TTG plan.

But I don’t understand what this has to do with interactive vs batch parsing.  
Why don’t you unconditionally retain API-annotation info?  How would GhcPs be 
used differently to GhcPsI?

You might want to answer by clarifying on the wiki page, so that it is a 
persistent record of the design debugged in dialogue by email.

Simon

From: Alan & Kim Zimmerman <alan.z...@gmail.com>
Sent: 08 May 2018 21:02
To: Simon Peyton Jones <simo...@microsoft.com>
Cc: ghc-devs <ghc-devs@haskell.org>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

I have started a wiki page at 
https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/IdeSupport

On 8 May 2018 at 10:54, Simon Peyton Jones 
<simo...@microsoft.com<mailto:simo...@microsoft.com>> wrote:
At first blush, “running the parser in two modes” and “changing the Pass” type 
don’t match up in my mind.  One seems quite local (how to run the parser).  The 
other seems more pervasive.

Can you say more about your proposed design, perhaps even on a wiki page?

Simon

From: ghc-devs 
<ghc-devs-boun...@haskell.org<mailto:ghc-devs-boun...@haskell.org>> On Behalf 
Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <ghc-devs@haskell.org<mailto:ghc-devs@haskell.org>>
Subject: TTG hsSyn for Batch and Interactive Parsing

I want to be able to run the GHC parser in one of two modes, batch which 
functions as now, and interactive which will (eventually) be incremental.
In addition, the hsSyn AST for each will have different TTG[1] annotations, so 
that it can better support IDE usage.
I think this can be done by changing the types in HsExtension to introduce a 
'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
 deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)
We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)
I have attached a simple proof of concept file, which emulates parsing and 
renaming.
Is this an appropriate approach to take?
Alan

[1] Trees That Grow 
https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: TTG hsSyn for Batch and Interactive Parsing

[Alan & Kim Zimmerman]

I have started a wiki page at
https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/IdeSupport

On 8 May 2018 at 10:54, Simon Peyton Jones <simo...@microsoft.com> wrote:

> At first blush, “running the parser in two modes” and “changing the Pass”
> type don’t match up in my mind.  One seems quite local (how to run the
> parser).  The other seems more pervasive.
>
>
>
> Can you say more about your proposed design, perhaps even on a wiki page?
>
>
>
> Simon
>
>
>
> *From:* ghc-devs <ghc-devs-boun...@haskell.org> *On Behalf Of *Alan & Kim
> Zimmerman
> *Sent:* 07 May 2018 16:17
> *To:* ghc-devs <ghc-devs@haskell.org>
> *Subject:* TTG hsSyn for Batch and Interactive Parsing
>
>
>
> I want to be able to run the GHC parser in one of two modes, batch which
> functions as now, and interactive which will (eventually) be incremental.
>
> In addition, the hsSyn AST for each will have different TTG[1]
> annotations, so that it can better support IDE usage.
>
> I think this can be done by changing the types in HsExtension to introduce
> a 'Process'  type as follows
>
> data Pass = Parsed Process | Renamed | Typechecked
>  deriving (Data)
>
> data Process = Batch | Interactive
>   deriving (Show, Data)
>
> We then rename the pass synonyms so that batch is the default
>
> type GhcPs   = GhcPass ('Parsed 'Batch)
> type GhcPsI  = GhcPass ('Parsed 'Interactive)
>
> I have attached a simple proof of concept file, which emulates parsing and
> renaming.
>
> Is this an appropriate approach to take?
>
> Alan
>
>
>
> [1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/
> ImplementingTreesThatGrow
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

RE: TTG hsSyn for Batch and Interactive Parsing

[Simon Peyton Jones via ghc-devs]

At first blush, “running the parser in two modes” and “changing the Pass” type 
don’t match up in my mind.  One seems quite local (how to run the parser).  The 
other seems more pervasive.

Can you say more about your proposed design, perhaps even on a wiki page?

Simon

From: ghc-devs <ghc-devs-boun...@haskell.org> On Behalf Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <ghc-devs@haskell.org>
Subject: TTG hsSyn for Batch and Interactive Parsing

I want to be able to run the GHC parser in one of two modes, batch which 
functions as now, and interactive which will (eventually) be incremental.
In addition, the hsSyn AST for each will have different TTG[1] annotations, so 
that it can better support IDE usage.
I think this can be done by changing the types in HsExtension to introduce a 
'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
 deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)
We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)
I have attached a simple proof of concept file, which emulates parsing and 
renaming.
Is this an appropriate approach to take?
Alan

[1] Trees That Grow 
https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

TTG hsSyn for Batch and Interactive Parsing

[Alan & Kim Zimmerman]

I want to be able to run the GHC parser in one of two modes, batch which
functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations,
so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce
a 'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
 deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and
renaming.

Is this an appropriate approach to take?

Alan


[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/
ImplementingTreesThatGrow
{-# LANGUAGE DeriveDataTypeable #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE StandaloneDeriving #-}
-- Next two needed for the Show instances
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE UndecidableInstances #-}

import Data.Data hiding ( Fixity )

main :: IO ()
main = putStrLn "hello"


data NoExt = NoExt
  deriving (Data,Eq,Ord,Show)

-- | Used as a data type index for the hsSyn AST
data GhcPass (c :: Pass)
deriving instance Eq (GhcPass c)
deriving instance Typeable c => Data (GhcPass c)

data Pass = Parsed Process | Renamed | Typechecked
 deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

-- Type synonyms as a shorthand for tagging
type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)
type GhcRn   = GhcPass 'Renamed
type GhcTc   = GhcPass 'Typechecked
type GhcTcId = GhcTc

-- | Maps the "normal" id type for a given pass
type family IdP p
type instance IdP (GhcPass ('Parsed _)) = RdrName
type instance IdP GhcRn = Name
type instance IdP GhcTc = Id

data RdrName = RdrName String deriving Show
data Name= Name Stringderiving Show
data Id  = Id String  deriving Show

-- -


data AST p = AST (XAST p) [Decl p]

deriving instance (Show (XAST p),Show (XD p), Show (IdP p)) => Show (AST p)

type family   XAST p
type instance XAST GhcPs  = NoExt
type instance XAST GhcPsI = Int
type instance XAST GhcRn  = NoExt
type instance XAST GhcTc  = NoExt

data Decl p = D (XD p) (IdP p)
deriving instance (Show (XD p), Show (IdP p)) => Show (Decl p)

type family   XD p
type instance XD GhcPs  = NoExt
type instance XD GhcPsI = Int
type instance XD GhcRn  = NoExt
type instance XD GhcTc  = NoExt

-- -

parseBatch :: String -> AST GhcPs
parseBatch str = AST NoExt [D NoExt (RdrName str)]

parseInteractive :: String -> AST GhcPsI
parseInteractive str = AST 1 [D 2 (RdrName str)]

rename :: AST (GhcPass ('Parsed p)) -> AST GhcRn
rename (AST _ ds) = AST NoExt ds'
  where
ds' = map rn ds
rn (D _ (RdrName s)) = (D NoExt (Name s))

i s = rename $ parseInteractive s
b s = rename $ parseBatch s
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs