Re: Crossreferenced GHC 8.0.2

2017-08-29 Thread Niklas Hambüchen 
On 17/08/17 18:22, Matthew Pickering wrote:
> 2. A nix function which builds and references all dependencies.

Very nice, it worked out of the box for me! (Almost, small issue with
https://github.com/mpickering/core-kythe/issues/10.)

I like how you can just add package names to a file and everything
magically works without setup effort.

Niklas
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Crossreferenced GHC 8.0.2

2017-08-20 Thread Robin Palotai 
Sorry for the delay - added it, see for example
http://stuff.codereview.me/#../logs/Control.Monad.Skeleton.core?corpus=core-kythe=41
.

Note that the Haskell sources (reachable by following the 'generates'
reverse edge from the Core) seem to have the empty corpus (for example
http://stuff.codereview.me/#F.hs?corpus=1).

(Also, one of the example entries had a serialization error, omitted that
one).

Looks interesting!

2017-08-17 19:22 GMT+02:00 Matthew Pickering :

> Hi all,
>
> If anyone is interested in this and using it further I have been
> working on two improvements.
>
> 1. A cross-referencer in the same style for the output of -ddump-simpl
> (which also links to the source code which produced the core).
>
> 2. A nix function which builds and references all dependencies.
>
> They are both found in my core-kythe repo with instructions about how
> to use them.
>
> https://github.com/mpickering/core-kythe
>
> You will need nix in order to use either of them but if you do it
> should be straightforward (albeit long in the first case!)
>
> I don't have a server to post examples to but Robin said he would when
> he got the chance.
>
> Matt
>
>
> On Tue, Jul 4, 2017 at 6:44 AM, Robin Palotai 
> wrote:
> > FYI I added GHC 8.2.1-rc2 source to the index. Please tell if some source
> > you would be interested in is obviously missing. Thanks!
> >
> > 2017-06-30 22:41 GMT+02:00 Robin Palotai :
> >>
> >> Hello Matthew,
> >>
> >> Please see inline
> >>
> >> 2017-06-30 11:57 GMT+02:00 Matthew Pickering
> >> :
> >>>
> >>> Hi Robin,
> >>>
> >>> This looks really useful for developers.
> >>>
> >>> 1. Would it be possible to provide a script which allows developers to
> >>> build this index for themselves easily?
> >>
> >>
> >> First, build and install the `ghc_kythe_wrapper` (instructions at
> >> https://github.com/google/haskell-indexer).
> >>
> >> Then build GHC with `make`, and capture the build log. Here's the hacky
> >> script I used:
> >> https://gist.github.com/robinp/222cf3a39cc19178ec8691522056d7fe
> >>
> >> It filters the log and replaces GHC calls to call the wrapper, which
> emits
> >> Kythe entries.
> >>
> >> Finally run `serve.sh` of the repo to postprocess and serve the entries
> >> through HTTP.
> >>
> >> This is all pretty new, so feedback or questions welcome. If the method
> >> distills, could try to write a more formal guide.
> >>
> >>>
> >>> 2. Is it possible to use this tool to detect dead code? Functions
> >>> which are not used anywhere in the compiler.
> >>
> >>
> >> We'll get there eventually, but for now the emitted data is not
> >> fine-grained enough. The main missing piece is recording what are
> exported
> >> entities of a module (Kythe schema discussion in progress). Without
> this,
> >> unused locals (which anyway surface with -Wall) would be presented and
> >> noisy.
> >>
> >> Also, one would probably need to postprocess the data a bit for this,
> like
> >> loading into a graph database or other ways.
> >>
> >> Did you see https://github.com/ndmitchell/weeder by the way? Might
> work.
> >>
> >>>
> >>> 3. How are you pretty printing the output whilst retaining the source
> >>> formatting? I had a quick look at the source but I couldn't see where
> >>> the output was being produced.
> >>>
> >>
> >> The haskell-indexer-frontend-kythe emits Kythe (http://kythe.io) schema
> >> data. We just export the source offsets, and it's the Kythe
> postprocessing /
> >> serving pipeline that does all the formatting.
> >>
> >>>
> >>> Cheers,
> >>>
> >>> Matt
> >>>
> >>> On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai <
> palotai.ro...@gmail.com>
> >>> wrote:
> >>> > Hello GHC devs,
> >>> >
> >>> > I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I
> >>> > find
> >>> > myself reading GHC source from time to time while working on the
> >>> > indexer,
> >>> > and partly since it's fun.
> >>> >
> >>> > First, here you can click around [2] and find where beloved functions
> >>> > are
> >>> > called from:
> >>> >
> >>> > http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?
> corpus
> >>> > (scroll down a bit, imports are not linked yet).
> >>> >
> >>> > Second, the way I indexed was pretty simple. I took the output of
> >>> > `make`,
> >>> > replaced the ghc used with the ghc_kythe_wrapper, and filtered the
> >>> > lines
> >>> > which included '-c', since I noticed that those duplicate previous
> >>> > large
> >>> > compile lines. This only indexes the stage1 compilation AFAIU.
> >>> >
> >>> > Feel free to suggest a better way to tap into the compilations to get
> >>> > everything properly indexed (and possibly only once).
> >>> >
> >>> > Any comments welcome!
> >>> > Robin
> >>> >
> >>> > [1]: https://github.com/google/haskell-indexer
> >>> >
> >>> > [2]: TLDR UI quirks:
> >>> >   - Click the :: in top-left to navigate file tree
> >>> >   - Ctrl-Click (on linux) to go directly

Re: Crossreferenced GHC 8.0.2

2017-08-17 Thread Matthew Pickering 
Hi all,

If anyone is interested in this and using it further I have been
working on two improvements.

1. A cross-referencer in the same style for the output of -ddump-simpl
(which also links to the source code which produced the core).

2. A nix function which builds and references all dependencies.

They are both found in my core-kythe repo with instructions about how
to use them.

https://github.com/mpickering/core-kythe

You will need nix in order to use either of them but if you do it
should be straightforward (albeit long in the first case!)

I don't have a server to post examples to but Robin said he would when
he got the chance.

Matt


On Tue, Jul 4, 2017 at 6:44 AM, Robin Palotai  wrote:
> FYI I added GHC 8.2.1-rc2 source to the index. Please tell if some source
> you would be interested in is obviously missing. Thanks!
>
> 2017-06-30 22:41 GMT+02:00 Robin Palotai :
>>
>> Hello Matthew,
>>
>> Please see inline
>>
>> 2017-06-30 11:57 GMT+02:00 Matthew Pickering
>> :
>>>
>>> Hi Robin,
>>>
>>> This looks really useful for developers.
>>>
>>> 1. Would it be possible to provide a script which allows developers to
>>> build this index for themselves easily?
>>
>>
>> First, build and install the `ghc_kythe_wrapper` (instructions at
>> https://github.com/google/haskell-indexer).
>>
>> Then build GHC with `make`, and capture the build log. Here's the hacky
>> script I used:
>> https://gist.github.com/robinp/222cf3a39cc19178ec8691522056d7fe
>>
>> It filters the log and replaces GHC calls to call the wrapper, which emits
>> Kythe entries.
>>
>> Finally run `serve.sh` of the repo to postprocess and serve the entries
>> through HTTP.
>>
>> This is all pretty new, so feedback or questions welcome. If the method
>> distills, could try to write a more formal guide.
>>
>>>
>>> 2. Is it possible to use this tool to detect dead code? Functions
>>> which are not used anywhere in the compiler.
>>
>>
>> We'll get there eventually, but for now the emitted data is not
>> fine-grained enough. The main missing piece is recording what are exported
>> entities of a module (Kythe schema discussion in progress). Without this,
>> unused locals (which anyway surface with -Wall) would be presented and
>> noisy.
>>
>> Also, one would probably need to postprocess the data a bit for this, like
>> loading into a graph database or other ways.
>>
>> Did you see https://github.com/ndmitchell/weeder by the way? Might work.
>>
>>>
>>> 3. How are you pretty printing the output whilst retaining the source
>>> formatting? I had a quick look at the source but I couldn't see where
>>> the output was being produced.
>>>
>>
>> The haskell-indexer-frontend-kythe emits Kythe (http://kythe.io) schema
>> data. We just export the source offsets, and it's the Kythe postprocessing /
>> serving pipeline that does all the formatting.
>>
>>>
>>> Cheers,
>>>
>>> Matt
>>>
>>> On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai 
>>> wrote:
>>> > Hello GHC devs,
>>> >
>>> > I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I
>>> > find
>>> > myself reading GHC source from time to time while working on the
>>> > indexer,
>>> > and partly since it's fun.
>>> >
>>> > First, here you can click around [2] and find where beloved functions
>>> > are
>>> > called from:
>>> >
>>> > http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?corpus
>>> > (scroll down a bit, imports are not linked yet).
>>> >
>>> > Second, the way I indexed was pretty simple. I took the output of
>>> > `make`,
>>> > replaced the ghc used with the ghc_kythe_wrapper, and filtered the
>>> > lines
>>> > which included '-c', since I noticed that those duplicate previous
>>> > large
>>> > compile lines. This only indexes the stage1 compilation AFAIU.
>>> >
>>> > Feel free to suggest a better way to tap into the compilations to get
>>> > everything properly indexed (and possibly only once).
>>> >
>>> > Any comments welcome!
>>> > Robin
>>> >
>>> > [1]: https://github.com/google/haskell-indexer
>>> >
>>> > [2]: TLDR UI quirks:
>>> >   - Click the :: in top-left to navigate file tree
>>> >   - Ctrl-Click (on linux) to go directly to definition (otherwise click
>>> > stuff from bottom pane)
>>> >   - Bottom pane often hides content, close it if stuck.
>>> >
>>> > ___
>>> > ghc-devs mailing list
>>> > ghc-devs@haskell.org
>>> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>> >
>>
>>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Crossreferenced GHC 8.0.2

2017-07-03 Thread Robin Palotai 
FYI I added GHC 8.2.1-rc2 source to the index. Please tell if some source
you would be interested in is obviously missing. Thanks!

2017-06-30 22:41 GMT+02:00 Robin Palotai :

> Hello Matthew,
>
> Please see inline
>
> 2017-06-30 11:57 GMT+02:00 Matthew Pickering 
> :
>
>> Hi Robin,
>>
>> This looks really useful for developers.
>>
>> 1. Would it be possible to provide a script which allows developers to
>> build this index for themselves easily?
>>
>
> First, build and install the `ghc_kythe_wrapper` (instructions at
> https://github.com/google/haskell-indexer).
>
> Then build GHC with `make`, and capture the build log. Here's the hacky
> script I used: https://gist.github.com/robinp/
> 222cf3a39cc19178ec8691522056d7fe
>
> It filters the log and replaces GHC calls to call the wrapper, which emits
> Kythe entries.
>
> Finally run `serve.sh` of the repo to postprocess and serve the entries
> through HTTP.
>
> This is all pretty new, so feedback or questions welcome. If the method
> distills, could try to write a more formal guide.
>
>
>> 2. Is it possible to use this tool to detect dead code? Functions
>> which are not used anywhere in the compiler.
>>
>
> We'll get there eventually, but for now the emitted data is not
> fine-grained enough. The main missing piece is recording what are exported
> entities of a module (Kythe schema discussion in progress). Without this,
> unused locals (which anyway surface with -Wall) would be presented and
> noisy.
>
> Also, one would probably need to postprocess the data a bit for this, like
> loading into a graph database or other ways.
>
> Did you see https://github.com/ndmitchell/weeder by the way? Might work.
>
>
>> 3. How are you pretty printing the output whilst retaining the source
>> formatting? I had a quick look at the source but I couldn't see where
>> the output was being produced.
>>
>>
> The haskell-indexer-frontend-kythe emits Kythe (http://kythe.io) schema
> data. We just export the source offsets, and it's the Kythe postprocessing
> / serving pipeline that does all the formatting.
>
>
>> Cheers,
>>
>> Matt
>>
>> On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai 
>> wrote:
>> > Hello GHC devs,
>> >
>> > I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I
>> find
>> > myself reading GHC source from time to time while working on the
>> indexer,
>> > and partly since it's fun.
>> >
>> > First, here you can click around [2] and find where beloved functions
>> are
>> > called from:
>> > http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?co
>> rpus
>> > (scroll down a bit, imports are not linked yet).
>> >
>> > Second, the way I indexed was pretty simple. I took the output of
>> `make`,
>> > replaced the ghc used with the ghc_kythe_wrapper, and filtered the lines
>> > which included '-c', since I noticed that those duplicate previous large
>> > compile lines. This only indexes the stage1 compilation AFAIU.
>> >
>> > Feel free to suggest a better way to tap into the compilations to get
>> > everything properly indexed (and possibly only once).
>> >
>> > Any comments welcome!
>> > Robin
>> >
>> > [1]: https://github.com/google/haskell-indexer
>> >
>> > [2]: TLDR UI quirks:
>> >   - Click the :: in top-left to navigate file tree
>> >   - Ctrl-Click (on linux) to go directly to definition (otherwise click
>> > stuff from bottom pane)
>> >   - Bottom pane often hides content, close it if stuck.
>> >
>> > ___
>> > ghc-devs mailing list
>> > ghc-devs@haskell.org
>> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>> >
>>
>
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Crossreferenced GHC 8.0.2

2017-06-30 Thread Robin Palotai 
Hello Matthew,

Please see inline

2017-06-30 11:57 GMT+02:00 Matthew Pickering :

> Hi Robin,
>
> This looks really useful for developers.
>
> 1. Would it be possible to provide a script which allows developers to
> build this index for themselves easily?
>

First, build and install the `ghc_kythe_wrapper` (instructions at
https://github.com/google/haskell-indexer).

Then build GHC with `make`, and capture the build log. Here's the hacky
script I used:
https://gist.github.com/robinp/222cf3a39cc19178ec8691522056d7fe

It filters the log and replaces GHC calls to call the wrapper, which emits
Kythe entries.

Finally run `serve.sh` of the repo to postprocess and serve the entries
through HTTP.

This is all pretty new, so feedback or questions welcome. If the method
distills, could try to write a more formal guide.


> 2. Is it possible to use this tool to detect dead code? Functions
> which are not used anywhere in the compiler.
>

We'll get there eventually, but for now the emitted data is not
fine-grained enough. The main missing piece is recording what are exported
entities of a module (Kythe schema discussion in progress). Without this,
unused locals (which anyway surface with -Wall) would be presented and
noisy.

Also, one would probably need to postprocess the data a bit for this, like
loading into a graph database or other ways.

Did you see https://github.com/ndmitchell/weeder by the way? Might work.


> 3. How are you pretty printing the output whilst retaining the source
> formatting? I had a quick look at the source but I couldn't see where
> the output was being produced.
>
>
The haskell-indexer-frontend-kythe emits Kythe (http://kythe.io) schema
data. We just export the source offsets, and it's the Kythe postprocessing
/ serving pipeline that does all the formatting.


> Cheers,
>
> Matt
>
> On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai 
> wrote:
> > Hello GHC devs,
> >
> > I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I find
> > myself reading GHC source from time to time while working on the indexer,
> > and partly since it's fun.
> >
> > First, here you can click around [2] and find where beloved functions are
> > called from:
> > http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?
> corpus
> > (scroll down a bit, imports are not linked yet).
> >
> > Second, the way I indexed was pretty simple. I took the output of `make`,
> > replaced the ghc used with the ghc_kythe_wrapper, and filtered the lines
> > which included '-c', since I noticed that those duplicate previous large
> > compile lines. This only indexes the stage1 compilation AFAIU.
> >
> > Feel free to suggest a better way to tap into the compilations to get
> > everything properly indexed (and possibly only once).
> >
> > Any comments welcome!
> > Robin
> >
> > [1]: https://github.com/google/haskell-indexer
> >
> > [2]: TLDR UI quirks:
> >   - Click the :: in top-left to navigate file tree
> >   - Ctrl-Click (on linux) to go directly to definition (otherwise click
> > stuff from bottom pane)
> >   - Bottom pane often hides content, close it if stuck.
> >
> > ___
> > ghc-devs mailing list
> > ghc-devs@haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> >
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Crossreferenced GHC 8.0.2

2017-06-30 Thread Matthew Pickering 
Hi Robin,

This looks really useful for developers.

1. Would it be possible to provide a script which allows developers to
build this index for themselves easily?
2. Is it possible to use this tool to detect dead code? Functions
which are not used anywhere in the compiler.
3. How are you pretty printing the output whilst retaining the source
formatting? I had a quick look at the source but I couldn't see where
the output was being produced.

Cheers,

Matt

On Fri, Jun 30, 2017 at 8:55 AM, Robin Palotai  wrote:
> Hello GHC devs,
>
> I ran haskell-indexer [1] on the GHC 8.0.2 tarball, partly because I find
> myself reading GHC source from time to time while working on the indexer,
> and partly since it's fun.
>
> First, here you can click around [2] and find where beloved functions are
> called from:
> http://stuff.codereview.me/#ghc/compiler/hsSyn/HsBinds.hs?corpus
> (scroll down a bit, imports are not linked yet).
>
> Second, the way I indexed was pretty simple. I took the output of `make`,
> replaced the ghc used with the ghc_kythe_wrapper, and filtered the lines
> which included '-c', since I noticed that those duplicate previous large
> compile lines. This only indexes the stage1 compilation AFAIU.
>
> Feel free to suggest a better way to tap into the compilations to get
> everything properly indexed (and possibly only once).
>
> Any comments welcome!
> Robin
>
> [1]: https://github.com/google/haskell-indexer
>
> [2]: TLDR UI quirks:
>   - Click the :: in top-left to navigate file tree
>   - Ctrl-Click (on linux) to go directly to definition (otherwise click
> stuff from bottom pane)
>   - Bottom pane often hides content, close it if stuck.
>
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Crossreferenced GHC 8.0.2

2017-06-30 Thread Niklas Hambüchen 
Hey Robin,

I find that super useful, thanks!

I hope some day we'll get to the stage where for any Haskell code I can
easily discover all inputs, like the Java world has in their IDEs for
decades already.

Niklas

On 30/06/17 09:55, Robin Palotai wrote:
> First, here you can click around [2] and find where beloved functions
> are called from
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs