Re: Why and How the External STG interpreter is Useful (Online presentation, Dec 2, Friday, 17:00 UTC)

[YueCompl via ghc-devs]

This sounds pretty exciting! 

Can we expect a full fledged stepping debugger integrated with IDEs via 
https://github.com/phoityne/haskell-debug-adapter 
 ? 
https://github.com/phoityne/ghci-dap  is 
still quite limited feature-wise.

> On 2021-12-02, at 22:52, Csaba Hruska  wrote:
> 
> Hello,
> 
> Today I'll do a presentation about the external stg interpreter.
> If you are interested please join and ask questions.
> https://skillsmatter.com/meetups/13654-haskell-stg-interp 
> 
> 
> Regards,
> Csaba Hruska
> 
> Abstract:
> Haskell: Why and How the External STG Interpreter is Useful
> 
> The external STG interpreter is a from scratch implementation of the STG 
> machine in Haskell. Currently it supports almost all GHC primops and RTS 
> features. It can run real world Haskell programs that were compiled with GHC 
> Whole Program Compiler (GHC-WPC). GHC-WPC is a GHC fork that exports the 
> whole program STG IR.
> 
> The external STG interpreter is an excellent tool to study the runtime 
> behaviour of Haskell programs, i.e. it can run/interpret GHC or Pandoc. The 
> implementation of the interpreter is in plain simple Haskell, so it makes 
> compiler backend and tooling development approachable for everyone. It 
> already has a programmable debugger which supports step-by-step evaluation, 
> breakpoints and execution region based inspection. It also can export the 
> whole program memory state and call-graphs to files for further 
> investigation. These features make it easy to find a memory leak or to 
> identify a performance bottleneck in a large real world Haskell application.
> 
> https://github.com/grin-compiler/ghc-whole-program-compiler-project 
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Potential improvement in compacting GC

[YueCompl via ghc-devs]

Greetings!

I'd like to take this opportunity to ask you experts about feasibility / 
technology-readiness of distributed GC, that I've recently been pondering with 
the idea to have a distributed GC managing shared heap across multiple server 
nodes, those inter-connected through fast ethernet. 

I'm implementing an array database system for internal use, each array is 
constrained to flat address space so no GC is required within one. But the 
number of arrays are unexpectedly large as my use cases becoming more apparent. 
Managing relations between those arrays (i.e. meta data) has appeared as far 
beyond the computational capacity of a single physical PC server (even powerful 
ones in today's market). We are working around this limitation by restricting 
meta data to be mappable to directory structure of underlying filesystem, but 
it's quite limiting from business perspective. 

We use FUSE filesystem driver to handle fine grained data coherence control 
over these many small arrays, to be accessed by many concurrent client 
machines,  by exposing a large virtual file for mmap on each client, to 
efficiently leverage os kernel pages as good shared cache storage, backing 
multiple processes running on each client os.

I'd imagine shared heap with https://hackage.haskell.org/package/compact 
 and a GC to have structural meta 
data managed likewise, so each client can mmap the entire shared heap, but only 
load relevant kernel pages on demand of the data nodes as it explores the 
information graph. This way the native programming language assumes the role of 
Data Manipulation Language as well, and Query Language can be simply some 
optimized data structures wrapped with accessing lib functions, more 
importantly the database system will have great horizontal scalability. And the 
whole architecture will have little technical debt as it seems so far.

So how feasible it is that you see? I'm aware that [compact] is still some 
experimental, and not even portable across GHC compilations yet, but it seems 
solvable with enough effort put in.

Thanks,
Compl


> On 2021-07-14, at 16:12, Simon Peyton Jones via ghc-devs 
>  wrote:
> 
> Thanks Omer
> 
> I had an interesting conversation with Steve Blackburn, the brains behind
> the MMTk memory management toolkit recently
> https://www.mmtk.io/
> 
> MMTk is designed to be a re-usable, open-source garbage collector, 
> specifically
> designed to be usable with lots of languages. In principle this is a great
> idea: GC is such a big field that no runtime (GHC's included) can ever devote
> enough effort to GC to do a really state of the art job.  It makes sense for
> one bunch of people to stellar GC and another bunch to simply reuse their
> work.
> 
> Of course, the interface between the GC and the mutator, scheduler, etc
> is particularly intimate.  Teasing them apart in GHC would be a significant
> task, and success would not be guaranteed.
> 
> But Steve is interested in working on this, with help from our end, perhaps
> initially with a student (or volunteer) project or two.
> 
> If it worked, it'd be cool.  
> 
> Here's a talk about MMTk: https://www.youtube.com/watch?v=3L6XEVaYAmU
> 
> Simon
> 
> 
> |  -Original Message-
> |  From: ghc-devs  On Behalf Of Ömer Sinan
> |  Agacan
> |  Sent: 14 July 2021 07:27
> |  To: ghc-devs 
> |  Subject: Re: Potential improvement in compacting GC
> |  
> |  Two other ideas that should improve GHC's compacting GC much more
> |  significantly. I've implemented both of these in another project and
> |  the results were great. In a GC benchmark (mutator does almost no work
> |  other than allocating data using a bump allocator), first one reduced
> |  Wasm instructions executed by 14%, second one 19.8%.
> |  
> |  Both of these ideas require pushing object headers to the mark stack
> |  with the objects, which means larger mark stacks. This is the only
> |  downside of these algorithms.
> |  
> |  - Instead of marking and then threading in the next pass, mark phase
> |  threads
> |all fields when pushing the fields to the mark stack. We still need
> |  two other
> |passes: one to unthread headers, another to move the objects. (we
> |  can't do
> |both in one pass, let me know if you're curious and I can elaborate)
> |  
> |This has the same number of passes as the current implementation,
> |  but it only
> |visits object fields once. Currently, we visit fields once when
> |  marking, to
> |mark fields, then again in `update_fwd`. This implementation does
> |  both in one
> |pass over the fields. `update_fwd` does not visit fields.
> |  
> |This reduced Wasm instructions executed by 14% in my benchmark.
> |  
> |  - Marking phase threads backwards pointers (ignores forwards
> |  pointers). Then we
> |do one pass instead of two: for a marked object, unthread it (update
> |forwards pointers to the object's new location), move it to its new

Re: What's the modern way to apply a polymorphic function to a Dynamic value in GHC 8.8 and onwards?

[YueCompl via ghc-devs]

A followup wish I have:

```hs
case io `eqTypeRep` typeRep @IO of
  Just HRefl -> Dynamic TypeRep <$> monotypedAct
  Nothing -> naAlt -- not an IO action
```

The `Just HRefl` part as in above remains hard to understand for me, I had 
glanced it in doc of the 'Type.Reflection' module earlier, but had no chance to 
figure out the usage of `eqTypeRep` to be like this, at least on my own. The 
community is very helpful in this regards, in leading me to it. But may there 
can be better surface syntax / usage hints that more intuitive, i.e. costing 
less effort to reach the solution? I anticipate improvements but apparently 
lack expertise for progress, I tried `Just {}` and it won't compile already...

I mean, things are already great as far, well, maybe the learning experience 
can be made even better.

Best,
Compl


> On 2021-04-13, at 22:07, YueCompl via ghc-devs  wrote:
> 
> After struggled this far, I decide that I can neither trivially understand 
> `pattern TypeRep`, nor the `withTypeable` at core. But this is what really 
> amazing with Haskell, GHC and the community here - I can get my job done even 
> without full understanding of what's going on under the hood, so long as the 
> compiler says it's okay! The warning has gone due to unknown reason after I 
> refactored the code a bit, surprisingly but well, I feel safe and comfort to 
> use it now.
> 
> Thanks to Erik, Vlad and Jaro again for your help.
> 
> u/Iceland_jack made a ticket to [add pattern TypeRep to 
> Type.Reflection](https://gitlab.haskell.org/ghc/ghc/-/issues/19691 
> <https://gitlab.haskell.org/ghc/ghc/-/issues/19691>) and appears it's very 
> welcomed. Though I don't expect it get shipped very soon or even could be 
> back ported to GHC 8.8, so I end up with this shim:
> 
> (there `PolyKinds` appears some unusual to be put into my `.cabal` due to its 
> syntax change can break some of my existing code)
> 
> ```hs
> {-# LANGUAGE PolyKinds #-}
> 
> module Dyn.Shim
>   ( pattern TypeRep,
> dynPerformIO,
> dynPerformSTM,
> dynContSTM,
>   )
> where
> 
> import Control.Concurrent.STM (STM)
> import Data.Dynamic (Dynamic (..), Typeable)
> import Type.Reflection
>   ( TypeRep,
> eqTypeRep,
> typeRep,
> withTypeable,
> pattern App,
> type (:~~:) (HRefl),
>   )
> import Prelude
> 
> data TypeableInstance a where
>   TypeableInstance :: Typeable a => TypeableInstance a
> 
> typeableInstance :: TypeRep a -> TypeableInstance a
> typeableInstance tr = withTypeable tr TypeableInstance
> 
> {- ORMOLU_DISABLE -}
> 
> -- | Shim for the proposed one at:
> --   https://gitlab.haskell.org/ghc/ghc/-/issues/19691 
> <https://gitlab.haskell.org/ghc/ghc/-/issues/19691>
> pattern TypeRep :: forall k (a :: k). () => Typeable a => TypeRep a
> pattern TypeRep <- (typeableInstance -> TypeableInstance)
>   where TypeRep = typeRep
> 
> {- ORMOLU_ENABLE -}
> 
> -- | Perform a polymorphic IO action which is wrapped in a 'Dynamic'
> --
> -- The specified 'naAlt' action will be performed instead, if the wrapped
> -- computation is not applicable, i.e. not really an IO action.
> dynPerformIO :: IO Dynamic -> Dynamic -> IO Dynamic
> dynPerformIO naAlt (Dynamic trAct monotypedAct) = case trAct of
>   App io TypeRep ->
> case io `eqTypeRep` typeRep @IO of
>   Just HRefl -> Dynamic TypeRep <$> monotypedAct
>   Nothing -> naAlt -- not an IO action
>   _ -> naAlt -- not even a poly-type
> 
> -- | Perform a polymorphic STM action which is wrapped in a 'Dynamic'
> --
> -- The specified 'naAlt' action will be performed instead, if the wrapped
> -- computation is not applicable, i.e. not really an STM action.
> dynPerformSTM :: STM Dynamic -> Dynamic -> STM Dynamic
> dynPerformSTM naAlt (Dynamic trAct monotypedAct) = case trAct of
>   App io TypeRep ->
> case io `eqTypeRep` typeRep @STM of
>   Just HRefl -> Dynamic TypeRep <$> monotypedAct
>   Nothing -> naAlt -- not an STM action
>   _ -> naAlt -- not even a poly-type
> 
> -- | Perform a polymorphic STM action which is wrapped in a 'Dynamic'
> --
> -- The specified 'naAlt' action will be performed instead, if the wrapped
> -- computation is not applicable, i.e. not really an STM action.
> dynContSTM :: STM () -> Dynamic -> (Dynamic -> STM ()) -> STM ()
> dynContSTM naAlt (Dynamic trAct monotypedAct) !exit = case trAct of
>   App io TypeRep ->
> case io `eqTypeRep` typeRep @STM of
>   Just HRefl -> exit . Dynamic TypeRep =<< monotypedAct
>   Nothing -> n

Re: What's the modern way to apply a polymorphic function to a Dynamic value in GHC 8.8 and onwards?

[YueCompl via ghc-devs]

After struggled this far, I decide that I can neither trivially understand 
`pattern TypeRep`, nor the `withTypeable` at core. But this is what really 
amazing with Haskell, GHC and the community here - I can get my job done even 
without full understanding of what's going on under the hood, so long as the 
compiler says it's okay! The warning has gone due to unknown reason after I 
refactored the code a bit, surprisingly but well, I feel safe and comfort to 
use it now.

Thanks to Erik, Vlad and Jaro again for your help.

u/Iceland_jack made a ticket to [add pattern TypeRep to 
Type.Reflection](https://gitlab.haskell.org/ghc/ghc/-/issues/19691) and appears 
it's very welcomed. Though I don't expect it get shipped very soon or even 
could be back ported to GHC 8.8, so I end up with this shim:

(there `PolyKinds` appears some unusual to be put into my `.cabal` due to its 
syntax change can break some of my existing code)

```hs
{-# LANGUAGE PolyKinds #-}

module Dyn.Shim
  ( pattern TypeRep,
dynPerformIO,
dynPerformSTM,
dynContSTM,
  )
where

import Control.Concurrent.STM (STM)
import Data.Dynamic (Dynamic (..), Typeable)
import Type.Reflection
  ( TypeRep,
eqTypeRep,
typeRep,
withTypeable,
pattern App,
type (:~~:) (HRefl),
  )
import Prelude

data TypeableInstance a where
  TypeableInstance :: Typeable a => TypeableInstance a

typeableInstance :: TypeRep a -> TypeableInstance a
typeableInstance tr = withTypeable tr TypeableInstance

{- ORMOLU_DISABLE -}

-- | Shim for the proposed one at:
--   https://gitlab.haskell.org/ghc/ghc/-/issues/19691
pattern TypeRep :: forall k (a :: k). () => Typeable a => TypeRep a
pattern TypeRep <- (typeableInstance -> TypeableInstance)
  where TypeRep = typeRep

{- ORMOLU_ENABLE -}

-- | Perform a polymorphic IO action which is wrapped in a 'Dynamic'
--
-- The specified 'naAlt' action will be performed instead, if the wrapped
-- computation is not applicable, i.e. not really an IO action.
dynPerformIO :: IO Dynamic -> Dynamic -> IO Dynamic
dynPerformIO naAlt (Dynamic trAct monotypedAct) = case trAct of
  App io TypeRep ->
case io `eqTypeRep` typeRep @IO of
  Just HRefl -> Dynamic TypeRep <$> monotypedAct
  Nothing -> naAlt -- not an IO action
  _ -> naAlt -- not even a poly-type

-- | Perform a polymorphic STM action which is wrapped in a 'Dynamic'
--
-- The specified 'naAlt' action will be performed instead, if the wrapped
-- computation is not applicable, i.e. not really an STM action.
dynPerformSTM :: STM Dynamic -> Dynamic -> STM Dynamic
dynPerformSTM naAlt (Dynamic trAct monotypedAct) = case trAct of
  App io TypeRep ->
case io `eqTypeRep` typeRep @STM of
  Just HRefl -> Dynamic TypeRep <$> monotypedAct
  Nothing -> naAlt -- not an STM action
  _ -> naAlt -- not even a poly-type

-- | Perform a polymorphic STM action which is wrapped in a 'Dynamic'
--
-- The specified 'naAlt' action will be performed instead, if the wrapped
-- computation is not applicable, i.e. not really an STM action.
dynContSTM :: STM () -> Dynamic -> (Dynamic -> STM ()) -> STM ()
dynContSTM naAlt (Dynamic trAct monotypedAct) !exit = case trAct of
  App io TypeRep ->
case io `eqTypeRep` typeRep @STM of
  Just HRefl -> exit . Dynamic TypeRep =<< monotypedAct
  Nothing -> naAlt -- not an STM action
  _ -> naAlt -- not even a poly-type

```

And my test case being a little more complex than the very first example, might 
be easier for others to grasp the usage, it runs like this:

```console
λ> import PoC.DynPoly
λ> testDynHold 
First got Nothing
Then got Just 3
λ> 
```

With the code:

```hs
module PoC.DynPoly where

import Control.Monad (void)
import Data.Dynamic (Dynamic (..), fromDynamic, toDyn)
import Data.IORef (modifyIORef', newIORef, readIORef, writeIORef)
import Dyn.Shim
import Type.Reflection (eqTypeRep, typeRep, pattern App, type (:~~:) (HRefl))
import Prelude

dynHoldEvent :: Dynamic -> Dynamic
dynHoldEvent (Dynamic trEvs monotypedEvs) =
  case trEvs of
App trEs TypeRep ->
  case trEs `eqTypeRep` typeRep @EventSink of
Just HRefl -> Dynamic TypeRep (holdEvent monotypedEvs)
Nothing -> error "not an EventSink" -- to be handled properly
_ -> error "even not a poly-type" -- to be handled properly
  where
holdEvent :: forall a. EventSink a -> IO (TimeSeries a)
holdEvent !evs = do
  !holder <- newIORef Nothing
  listenEvents evs $ writeIORef holder . Just
  return $ TimeSeries $ readIORef holder

data EventSink a = EventSink
  { listenEvents :: (a -> IO ()) -> IO (),
publishEvent :: a -> IO ()
  }

newtype TimeSeries a = TimeSeries {readTimeSeries :: IO (Maybe a)}

newEventSink :: forall a. IO (EventSink a)
newEventSink = do
  !listeners <- newIORef []
  let listen listener = modifyIORef' listeners (listener :)
  publish a = readIORef listeners >>= void . mapM ($ a)
  return $ EventSink listen publish

testDynHold :: IO ()
testDynHold = do
  (evs :: EventSink Int) <- newEventS

Re: What's the modern way to apply a polymorphic function to a Dynamic value in GHC 8.8 and onwards?

[YueCompl via ghc-devs]

Oh, forgot to mention that there is a warning I also don't understand by far:

```log
src/PoC/DynPoly.hs:40:3: warning: [-Woverlapping-patterns]
Pattern match has inaccessible right hand side
In a case alternative: Dynamic (App eventSink TypeRep) evs' -> ...
   |
40 |   Dynamic (App eventSink TypeRep) evs' ->
   |   ^^^...
```
I need to work out some extra stuff to test the solution in real case, 
meanwhile this warning seems worrying ...

> On 2021-04-13, at 00:27, YueCompl via ghc-devs  wrote:
> 
> Thanks Erik,
> 
> With the help from Iceland_jack <https://www.reddit.com/user/Iceland_jack> 
> via /r/haskell <https://www.reddit.com/r/haskell> , I end up with a working 
> solution like this:
> 
> ```hs
> data TypeableInstance a where
>   -- data TypeableInstance :: forall k. k -> Type where
>   TypeableInstance :: Typeable a => TypeableInstance a
> 
> typeableInstance :: forall (k :: Type) (a :: k). TypeRep a -> 
> TypeableInstance a
> typeableInstance typeRep' = withTypeable typeRep' TypeableInstance
> 
> pattern TypeRep :: forall k (a :: k). () => Typeable a => TypeRep a
> pattern TypeRep <-
>   (typeableInstance -> TypeableInstance)
>   where
> TypeRep = typeRep
> 
> holdEvent :: Dynamic -> Dynamic
> holdEvent !devs = case devs of
>   Dynamic (App eventSink TypeRep) evs' ->
> case eqTypeRep (typeRep @EventSink) eventSink of
>   Just HRefl -> Dynamic TypeRep (hcHoldEvent evs')
>   Nothing -> error "not an EventSink" -- to be handled properly
>   _ -> error "even not a poly-type" -- to be handled properly
>   where
> hcHoldEvent :: forall a. EventSink a -> IO (TimeSeries a)
> hcHoldEvent !evs = do
>   !holder <- newIORef Nothing
>   listenEvents evs $ writeIORef holder . Just
>   return $ TimeSeries $ readIORef holder
> 
> data EventSink a = EventSink
>   { listenEvents :: (a -> IO ()) -> IO (),
> closeStream :: IO ()
>   }
> 
> instance Functor EventSink where
>   fmap = undefined
> 
> newtype TimeSeries a = TimeSeries {readTimeSeries :: IO (Maybe a)}
> 
> instance Functor TimeSeries where
>   fmap = undefined
> 
> ```
> 
> I'm still wrapping my head around it, for how the `pattern TypeRep` works in 
> this case.
> 
> Or you think there exists a solution without using such a pattern?
> 
> My function (hcHoldEvent) is polymorphic so not eligible to be wrapped as a 
> Dynamic in the first place, or there also some way to specialize it at 
> runtime? That'll be another interesting tool.
> 
> Thanks with regards,
> Compl
> 
>> On 2021-04-12, at 22:50, Erik Hesselink > <mailto:hessel...@gmail.com>> wrote:
>> 
>> Your function is not `forall a. a -> f a`, as in your initial example, but 
>> requires its argument to be an `EventSink`. The value you unwrap from the 
>> `Dynamic` is any existential type, not necessarily an `EventSink`. You'll 
>> have to compare the TypeReps (with something like `eqTypeRep`[1], or wrap 
>> your function in a `Dynamic` and use `dynApply` [2], which does the 
>> comparison for you.
>> 
>> Cheers,
>> 
>> Erik
>> 
>> [1] 
>> https://hackage.haskell.org/package/base-4.15.0.0/docs/Type-Reflection.html#v:eqTypeRep
>>  
>> <https://hackage.haskell.org/package/base-4.15.0.0/docs/Type-Reflection.html#v:eqTypeRep>
>> [2] 
>> https://hackage.haskell.org/package/base-4.15.0.0/docs/Data-Dynamic.html#v:dynApply
>>  
>> <https://hackage.haskell.org/package/base-4.15.0.0/docs/Data-Dynamic.html#v:dynApply>
>> On Mon, 12 Apr 2021 at 16:15, YueCompl via ghc-devs > <mailto:ghc-devs@haskell.org>> wrote:
>> Thanks to Vlad and Jaro, your solution of `apD` compiles, I think it should 
>> work.
>> 
>> But unfortunately my real case is a little different / more complex, a MWE 
>> appears like this:
>> 
>> ```hs
>> holdEvent :: Dynamic -> Dynamic
>> holdEvent (Dynamic t evs') =
>>   withTypeable t $ Dynamic typeRep (hcHoldEvent evs')
>>   where
>> hcHoldEvent :: forall a. EventSink a -> IO (TimeSeries a)
>> hcHoldEvent !evs = do
>>   !holder <- newIORef Nothing
>>   listenEvents evs $ writeIORef holder . Just
>>   return $ TimeSeries $ readIORef holder
>> 
>> data EventSink a = EventSink
>>   { listenEvents :: (a -> IO ()) -> IO (),
>> closeStream :: IO ()
>>   }
>> 
>> instance Functor EventSink where
>>   fmap = undefined
>> 
>&

Re: What's the modern way to apply a polymorphic function to a Dynamic value in GHC 8.8 and onwards?

[YueCompl via ghc-devs]

Thanks Erik,

With the help from Iceland_jack <https://www.reddit.com/user/Iceland_jack> via 
/r/haskell <https://www.reddit.com/r/haskell> , I end up with a working 
solution like this:

```hs
data TypeableInstance a where
  -- data TypeableInstance :: forall k. k -> Type where
  TypeableInstance :: Typeable a => TypeableInstance a

typeableInstance :: forall (k :: Type) (a :: k). TypeRep a -> TypeableInstance a
typeableInstance typeRep' = withTypeable typeRep' TypeableInstance

pattern TypeRep :: forall k (a :: k). () => Typeable a => TypeRep a
pattern TypeRep <-
  (typeableInstance -> TypeableInstance)
  where
TypeRep = typeRep

holdEvent :: Dynamic -> Dynamic
holdEvent !devs = case devs of
  Dynamic (App eventSink TypeRep) evs' ->
case eqTypeRep (typeRep @EventSink) eventSink of
  Just HRefl -> Dynamic TypeRep (hcHoldEvent evs')
  Nothing -> error "not an EventSink" -- to be handled properly
  _ -> error "even not a poly-type" -- to be handled properly
  where
hcHoldEvent :: forall a. EventSink a -> IO (TimeSeries a)
hcHoldEvent !evs = do
  !holder <- newIORef Nothing
  listenEvents evs $ writeIORef holder . Just
  return $ TimeSeries $ readIORef holder

data EventSink a = EventSink
  { listenEvents :: (a -> IO ()) -> IO (),
closeStream :: IO ()
  }

instance Functor EventSink where
  fmap = undefined

newtype TimeSeries a = TimeSeries {readTimeSeries :: IO (Maybe a)}

instance Functor TimeSeries where
  fmap = undefined

```

I'm still wrapping my head around it, for how the `pattern TypeRep` works in 
this case.

Or you think there exists a solution without using such a pattern?

My function (hcHoldEvent) is polymorphic so not eligible to be wrapped as a 
Dynamic in the first place, or there also some way to specialize it at runtime? 
That'll be another interesting tool.

Thanks with regards,
Compl

> On 2021-04-12, at 22:50, Erik Hesselink  wrote:
> 
> Your function is not `forall a. a -> f a`, as in your initial example, but 
> requires its argument to be an `EventSink`. The value you unwrap from the 
> `Dynamic` is any existential type, not necessarily an `EventSink`. You'll 
> have to compare the TypeReps (with something like `eqTypeRep`[1], or wrap 
> your function in a `Dynamic` and use `dynApply` [2], which does the 
> comparison for you.
> 
> Cheers,
> 
> Erik
> 
> [1] 
> https://hackage.haskell.org/package/base-4.15.0.0/docs/Type-Reflection.html#v:eqTypeRep
>  
> <https://hackage.haskell.org/package/base-4.15.0.0/docs/Type-Reflection.html#v:eqTypeRep>
> [2] 
> https://hackage.haskell.org/package/base-4.15.0.0/docs/Data-Dynamic.html#v:dynApply
>  
> <https://hackage.haskell.org/package/base-4.15.0.0/docs/Data-Dynamic.html#v:dynApply>
> On Mon, 12 Apr 2021 at 16:15, YueCompl via ghc-devs  <mailto:ghc-devs@haskell.org>> wrote:
> Thanks to Vlad and Jaro, your solution of `apD` compiles, I think it should 
> work.
> 
> But unfortunately my real case is a little different / more complex, a MWE 
> appears like this:
> 
> ```hs
> holdEvent :: Dynamic -> Dynamic
> holdEvent (Dynamic t evs') =
>   withTypeable t $ Dynamic typeRep (hcHoldEvent evs')
>   where
> hcHoldEvent :: forall a. EventSink a -> IO (TimeSeries a)
> hcHoldEvent !evs = do
>   !holder <- newIORef Nothing
>   listenEvents evs $ writeIORef holder . Just
>   return $ TimeSeries $ readIORef holder
> 
> data EventSink a = EventSink
>   { listenEvents :: (a -> IO ()) -> IO (),
> closeStream :: IO ()
>   }
> 
> instance Functor EventSink where
>   fmap = undefined
> 
> newtype TimeSeries a = TimeSeries {readTimeSeries :: IO (Maybe a)}
> 
> instance Functor TimeSeries where
>   fmap = undefined
> 
> ```
> 
> Now I'm clueless how to use the `withTypeable` trick to apply my polymorphic 
> `hcHoldEvent` to `Dynamic`, naively written as in above, the error is:
> 
> ```log
> src/PoC/DynPoly.hs:20:49: error:
> • Couldn't match expected type ‘EventSink a0’ with actual type ‘a’
>   ‘a’ is a rigid type variable bound by
> a pattern with constructor:
>   Dynamic :: forall a.
>  base-4.13.0.0:Data.Typeable.Internal.TypeRep a -> a -> 
> Dynamic,
> in an equation for ‘holdEvent’
> at src/PoC/DynPoly.hs:19:12-25
> • In the first argument of ‘hcHoldEvent’, namely ‘evs'’
>   In the second argument of ‘Dynamic’, namely ‘(hcHoldEvent evs')’
>   In the second argument of ‘($)’, namely
> ‘Dynamic typeRep (hcHoldEvent evs')’
> • Relevant bindings include
> evs' :: a (bound at src/PoC/DynPoly.hs:

Re: What's the modern way to apply a polymorphic function to a Dynamic value in GHC 8.8 and onwards?

[YueCompl via ghc-devs]

Thanks to Vlad and Jaro, your solution of `apD` compiles, I think it should 
work.

But unfortunately my real case is a little different / more complex, a MWE 
appears like this:

```hs
holdEvent :: Dynamic -> Dynamic
holdEvent (Dynamic t evs') =
  withTypeable t $ Dynamic typeRep (hcHoldEvent evs')
  where
hcHoldEvent :: forall a. EventSink a -> IO (TimeSeries a)
hcHoldEvent !evs = do
  !holder <- newIORef Nothing
  listenEvents evs $ writeIORef holder . Just
  return $ TimeSeries $ readIORef holder

data EventSink a = EventSink
  { listenEvents :: (a -> IO ()) -> IO (),
closeStream :: IO ()
  }

instance Functor EventSink where
  fmap = undefined

newtype TimeSeries a = TimeSeries {readTimeSeries :: IO (Maybe a)}

instance Functor TimeSeries where
  fmap = undefined

```

Now I'm clueless how to use the `withTypeable` trick to apply my polymorphic 
`hcHoldEvent` to `Dynamic`, naively written as in above, the error is:

```log
src/PoC/DynPoly.hs:20:49: error:
• Couldn't match expected type ‘EventSink a0’ with actual type ‘a’
  ‘a’ is a rigid type variable bound by
a pattern with constructor:
  Dynamic :: forall a.
 base-4.13.0.0:Data.Typeable.Internal.TypeRep a -> a -> 
Dynamic,
in an equation for ‘holdEvent’
at src/PoC/DynPoly.hs:19:12-25
• In the first argument of ‘hcHoldEvent’, namely ‘evs'’
  In the second argument of ‘Dynamic’, namely ‘(hcHoldEvent evs')’
  In the second argument of ‘($)’, namely
‘Dynamic typeRep (hcHoldEvent evs')’
• Relevant bindings include
evs' :: a (bound at src/PoC/DynPoly.hs:19:22)
t :: base-4.13.0.0:Data.Typeable.Internal.TypeRep a
  (bound at src/PoC/DynPoly.hs:19:20)
   |
20 |   withTypeable t $ Dynamic typeRep (hcHoldEvent evs')
   | 

```

Thanks with best regards,
Compl


> On 2021-04-12, at 22:04, Jaro Reinders  wrote:
> 
> I have no experience in this area, but this compiles:
> 
> ```
> {-# LANGUAGE RankNTypes, ScopedTypeVariables #-}
> import Type.Reflection
> import Data.Dynamic
> 
> appD :: forall f. Typeable f => (forall a. a -> f a) -> Dynamic -> Dynamic
> appD f (Dynamic rep (x :: a)) = withTypeable rep (toDyn (f x))
> ```
> 
> Cheers,
> 
> Jaro


> On 2021-04-12, at 21:06, Vladislav Zavialov  wrote:
> 
> Would something like this work for you?
> 
>  import Type.Reflection
>  import Data.Dynamic
> 
>  apD :: Typeable f => (forall a. a -> f a) -> Dynamic -> Dynamic
>  apD f (Dynamic t a) = withTypeable t $ Dynamic typeRep (f a)
> 
> - Vlad
> 
>> On 12 Apr 2021, at 14:34, YueCompl via ghc-devs  wrote:
>> 
>> Dear Cafe and GHC devs,
>> 
>> 
>> There used to be a "principled way with pattern match on the constructor":
>> 
>> ```hs
>> data Dynamic where
>> Dynamic :: Typeable a => a -> Dynamic
>> 
>> apD :: Typeable f => (forall a. a -> f a) -> Dynamic -> Dynamic
>> apD f (Dynamic a) = Dynamic $ f a
>> ```
>> Source: 
>> https://www.reddit.com/r/haskell/comments/2kdcca/q_how_to_apply_a_polymorphic_function_to_a/
>> 
>> 
>> But now with GHC 8.8 as in my case, `Dynamic` constructor has changed its 
>> signature to: 
>> 
>> ```hs
>> Dynamic :: forall a. TypeRep a -> a -> Dynamic
>> ```
>> 
>> Which renders the `apD` not working anymore. 
>> 
>> 
>> And it seems missing dependencies now for an older solution Edward KMETT 
>> provides:
>> 
>> ```hs
>> apD :: forall f. Typeable1 f => (forall a. a -> f a) -> Dynamic -> Dynamic
>> apD f a = dynApp df a
>> where t = dynTypeRep a
>>   df = reify (mkFunTy t (typeOf1 (undefined :: f ()) `mkAppTy` t)) $ 
>> \(_ :: Proxy s) -> toDyn (WithRep f :: WithRep s (() -> f 
>> ()))
>> ```
>> Source: 
>> https://stackoverflow.com/questions/10889682/how-to-apply-a-polymorphic-function-to-a-dynamic-value
>> 
>> 
>> So, how can I do that nowadays?
>> 
>> Thanks,
>> Compl
>> 
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

What's the modern way to apply a polymorphic function to a Dynamic value in GHC 8.8 and onwards?

[YueCompl via ghc-devs]

Dear Cafe and GHC devs,


There used to be a "principled way with pattern match on the constructor":

```hs
data Dynamic where
  Dynamic :: Typeable a => a -> Dynamic

apD :: Typeable f => (forall a. a -> f a) -> Dynamic -> Dynamic
apD f (Dynamic a) = Dynamic $ f a
```
Source: 
https://www.reddit.com/r/haskell/comments/2kdcca/q_how_to_apply_a_polymorphic_function_to_a/


But now with GHC 8.8 as in my case, `Dynamic` constructor has changed its 
signature to: 

```hs
Dynamic :: forall a. TypeRep a -> a -> Dynamic
```

Which renders the `apD` not working anymore. 


And it seems missing dependencies now for an older solution Edward KMETT 
provides:

```hs
apD :: forall f. Typeable1 f => (forall a. a -> f a) -> Dynamic -> Dynamic
apD f a = dynApp df a
  where t = dynTypeRep a
df = reify (mkFunTy t (typeOf1 (undefined :: f ()) `mkAppTy` t)) $ 
  \(_ :: Proxy s) -> toDyn (WithRep f :: WithRep s (() -> f ()))
```
Source: 
https://stackoverflow.com/questions/10889682/how-to-apply-a-polymorphic-function-to-a-dynamic-value


So, how can I do that nowadays?

Thanks,
Compl

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: How to ensure optimization for large immutable vectors to be shared w.r.t. Referential Transparency

[YueCompl via ghc-devs]

On a second thought, maybe GHCi's silence is a bad thing here? Maybe it should 
complain loudly as GHC does?

```hs
λ> :set -package vector
package flags have changed, resetting and loading new packages...
λ> 
λ> import Prelude
λ> 
λ> import qualified Data.Vector.Storable as VS
λ> 
λ> :{
λ| 
λ| newtype SomeVector = SomeVector (VS.Vector Int)
λ| 
λ| isSameVector :: SomeVector -> SomeVector -> Bool
λ| isSameVector (SomeVector x) (SomeVector y) = 
λ|   x'offset == y'offset && x'fp == y'fp
λ|  where
λ|   (x'fp, x'offset, _x'len) = VS.unsafeToForeignPtr x
λ|   (y'fp, y'offset, _y'len) = VS.unsafeToForeignPtr y
λ| 
λ| :}
λ> 
λ> let (v :: VS.Vector Int) = VS.fromList [3,2,5] in isSameVector (SomeVector 
v) (SomeVector v)
True
λ> 
λ> 
λ> :set -XMonomorphismRestriction
λ> 
λ> let v = VS.fromList [3,2,5] in isSameVector (SomeVector v) (SomeVector v)
True
λ> 
λ> :set -XNoMonomorphismRestriction
λ> 
λ> let v = VS.fromList [3,2,5] in isSameVector (SomeVector v) (SomeVector v)
False
λ> 
```

Further more, my intuition about GHC's type inference here is proved wrong by 
it, right. But I still think that with a single piece of `let-in` construct, 
types are better to be inferred as specific as possible, then the result would 
not be affected by some extension's semantics modification. Here v's type can 
obviously be inferred to `VS.Vector Int` according to its usage in the 
`SomeVector` data constructor, I wonder why GHC is not doing this?


> On 2021-04-06, at 22:19, YueCompl via ghc-devs  wrote:
> 
> Thanks very much for the diagnostic and explanation!
> 
> I was wrong in assuming the `in isSameVector (SomeVector v) (SomeVector v)` 
> part is enough to have type of v in `let !v = VS.fromList [3,2,5]` inferred 
> as monomorphic, totally unaware about "NoMonomorphismRestriction" before, 
> I've learned it today :D
> 
>> On 2021-04-06, at 21:51, Viktor Dukhovni  wrote:
>> 
>> On Tue, Apr 06, 2021 at 07:12:51PM +0800, YueCompl via ghc-devs wrote:
>> 
>>> λ> import Control.Monad.ST
>>> λ> import qualified Data.Vector.Storable as VS
>>> λ> 
>>> λ> :{
>>> λ| 
>>> λ| newtype SomeVector = SomeVector (VS.Vector Int)
>>> λ| 
>>> λ| isSameVector :: SomeVector -> SomeVector -> Bool
>>> λ| isSameVector (SomeVector !x) (SomeVector !y) = runST $ do
>>> λ|   mx@(VS.MVector !x'offset !x'fp) <- VS.unsafeThaw x
>>> λ|   my@(VS.MVector !y'offset !y'fp) <- VS.unsafeThaw y
>>> λ|   _ <- VS.unsafeFreeze mx
>>> λ|   _ <- VS.unsafeFreeze my
>>> λ|   return $ x'offset == y'offset && x'fp == y'fp
>>> λ| 
>>> λ| :}
>>> λ> 
>>> λ> let !v = VS.fromList [3,2,5] in isSameVector (SomeVector v) (SomeVector 
>>> v)
>>> False
>>> λ> 
>>> λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSameVector v v
>>> True
>> 
>> In GHCi, but not in compiled programs, by default the
>> `NoMonomorphismRestriction` extension is enabled.  If I compile your
>> code with that restriction, I can reproduce your results (the values are
>> not shared).
>> 
>> If I either skip the extension, or add an explicit type annotation to
>> for the vector, then the values are shared.
>> 
>>   {-# LANGUAGE BangPatterns #-}
>>   {-# LANGUAGE NoMonomorphismRestriction #-}
>>   import Control.Monad.ST
>>   import qualified Data.Vector.Storable as VS
>> 
>>   newtype SomeVector = SomeVector (VS.Vector Int)
>> 
>>   isSameVector :: SomeVector -> SomeVector -> Bool
>>   isSameVector (SomeVector !x) (SomeVector !y) = runST $ do
>> mx@(VS.MVector !x'offset !x'fp) <- VS.unsafeThaw x
>> my@(VS.MVector !y'offset !y'fp) <- VS.unsafeThaw y
>> _ <- VS.unsafeFreeze mx
>> _ <- VS.unsafeFreeze my
>> return $ x'offset == y'offset && x'fp == y'fp
>> 
>>   main :: IO ()
>>   main =
>>   let !v = VS.fromList [0..1023] -- :: VS.Vector Int
>>in print $ isSameVector (SomeVector v) (SomeVector v)
>> 
>> Since newtypes are always strict in their argument, I don't think the
>> BangPattern does what you'd like it to do, it just makes "main" strict
>> in v.  As defined with `NoMonomorphismRestriction` v is a polymorphic
>> function, and I guess it is specialised at the call site.
>> 
>> -- 
>>   Viktor.
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: How to ensure optimization for large immutable vectors to be shared w.r.t. Referential Transparency

[YueCompl via ghc-devs]

Thanks very much for the diagnostic and explanation!

I was wrong in assuming the `in isSameVector (SomeVector v) (SomeVector v)` 
part is enough to have type of v in `let !v = VS.fromList [3,2,5]` inferred as 
monomorphic, totally unaware about "NoMonomorphismRestriction" before, I've 
learned it today :D

> On 2021-04-06, at 21:51, Viktor Dukhovni  wrote:
> 
> On Tue, Apr 06, 2021 at 07:12:51PM +0800, YueCompl via ghc-devs wrote:
> 
>> λ> import Control.Monad.ST
>> λ> import qualified Data.Vector.Storable as VS
>> λ> 
>> λ> :{
>> λ| 
>> λ| newtype SomeVector = SomeVector (VS.Vector Int)
>> λ| 
>> λ| isSameVector :: SomeVector -> SomeVector -> Bool
>> λ| isSameVector (SomeVector !x) (SomeVector !y) = runST $ do
>> λ|   mx@(VS.MVector !x'offset !x'fp) <- VS.unsafeThaw x
>> λ|   my@(VS.MVector !y'offset !y'fp) <- VS.unsafeThaw y
>> λ|   _ <- VS.unsafeFreeze mx
>> λ|   _ <- VS.unsafeFreeze my
>> λ|   return $ x'offset == y'offset && x'fp == y'fp
>> λ| 
>> λ| :}
>> λ> 
>> λ> let !v = VS.fromList [3,2,5] in isSameVector (SomeVector v) (SomeVector v)
>> False
>> λ> 
>> λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSameVector v v
>> True
> 
> In GHCi, but not in compiled programs, by default the
> `NoMonomorphismRestriction` extension is enabled.  If I compile your
> code with that restriction, I can reproduce your results (the values are
> not shared).
> 
> If I either skip the extension, or add an explicit type annotation to
> for the vector, then the values are shared.
> 
>{-# LANGUAGE BangPatterns #-}
>{-# LANGUAGE NoMonomorphismRestriction #-}
>import Control.Monad.ST
>import qualified Data.Vector.Storable as VS
> 
>newtype SomeVector = SomeVector (VS.Vector Int)
> 
>isSameVector :: SomeVector -> SomeVector -> Bool
>isSameVector (SomeVector !x) (SomeVector !y) = runST $ do
>  mx@(VS.MVector !x'offset !x'fp) <- VS.unsafeThaw x
>  my@(VS.MVector !y'offset !y'fp) <- VS.unsafeThaw y
>  _ <- VS.unsafeFreeze mx
>  _ <- VS.unsafeFreeze my
>  return $ x'offset == y'offset && x'fp == y'fp
> 
>main :: IO ()
>main =
>let !v = VS.fromList [0..1023] -- :: VS.Vector Int
> in print $ isSameVector (SomeVector v) (SomeVector v)
> 
> Since newtypes are always strict in their argument, I don't think the
> BangPattern does what you'd like it to do, it just makes "main" strict
> in v.  As defined with `NoMonomorphismRestriction` v is a polymorphic
> function, and I guess it is specialised at the call site.
> 
> -- 
>Viktor.
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: [Haskell-cafe] How to ensure optimization for large immutable vectors to be shared w.r.t. Referential Transparency

[YueCompl via ghc-devs]

Appears it'd work as expected when the immutable vector is originally created 
from foreign ptr, I think it'll work for my cases. 
(Though it's still strangely unexpected for ad hoc immutable vectors unshared 
when wrapped.)

```hs
λ> :set -XBangPatterns
λ> 
λ> :set -package vector
package flags have changed, resetting and loading new packages...
λ> 
λ> import Prelude
λ> 
λ> import Foreign.Marshal.Alloc
λ> import Foreign.Storable
λ> import Foreign.ForeignPtr
λ> 
λ> import qualified Data.Vector.Storable as VS
λ> 
λ> :{
λ| 
λ| data SomeVector = SomeVector (VS.Vector Int)
λ| 
λ| isSameVector :: Storable a => VS.Vector a -> VS.Vector a -> Bool
λ| isSameVector !x !y = 
λ|   x'offset == y'offset && x'fp == y'fp
λ|  where
λ|   (x'fp, x'offset, _x'len) = VS.unsafeToForeignPtr x
λ|   (y'fp, y'offset, _y'len) = VS.unsafeToForeignPtr y
λ| 
λ| isSame :: SomeVector -> SomeVector -> Bool
λ| isSame (SomeVector !x) (SomeVector !y) = isSameVector x y
λ| 
λ| :}
λ> 
λ> let !v = VS.fromList [3,2,5] in isSame (SomeVector v) (SomeVector v)
False
λ> 
λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSame v v
True
λ> 
λ> 
λ> (fp :: ForeignPtr Int) <- mallocBytes 256 >>= newForeignPtr_
λ> let !v = VS.unsafeFromForeignPtr fp 0 32
λ| 
λ> isSame (SomeVector v) (SomeVector v)
True
λ> 
```

> On 2021-04-06, at 20:50, Henning Thielemann  
> wrote:
> 
> 
> On Tue, 6 Apr 2021, YueCompl wrote:
> 
>> ```hsλ> let !v = VS.fromList [3,2,5] in isSameVector (SomeVector v) 
>> (SomeVector v)
>> False
>> λ> 
>> λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSameVector v v
>> True
>> λ> 
>> ```
> 
> Then I have no idea.
> 
> Maybe ghc-heap-view/ghc-vis can reveal the mystery.
> 
> https://github.com/nomeata/haskell-bytes-bobkonf2021

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: [Haskell-cafe] How to ensure optimization for large immutable vectors to be shared w.r.t. Referential Transparency

[YueCompl via ghc-devs]

```hs
λ> let !v = VS.fromList [3,2,5] in isSameVector (SomeVector v) (SomeVector v)
False
λ> 
λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSameVector v v
True
λ> 
```

> On 2021-04-06, at 20:34, Henning Thielemann  
> wrote:
> 
> 
> On Tue, 6 Apr 2021, YueCompl wrote:
> 
>> Thanks, it's a good idea. Unfortunately the result remains, making me even 
>> more curious.
>> ```hs
>> λ> :set -XBangPatterns
>> λ> 
>> λ> :set -package vector
>> package flags have changed, resetting and loading new packages...
>> λ> 
>> λ> import Prelude
>> λ> 
>> λ> import Control.Monad.ST
>> λ> import qualified Data.Vector.Storable as VS
>> λ> 
>> λ> :{
>> λ| 
>> λ| newtype SomeVector = SomeVector (VS.Vector Int)
>> λ| 
>> λ| isSameVector :: SomeVector -> SomeVector -> Bool
>> λ| isSameVector (SomeVector !x) (SomeVector !y) = 
>> λ|   x'offset == y'offset && x'fp == y'fp
>> λ|  where
>> λ|   (x'fp, x'offset, _x'len) = VS.unsafeToForeignPtr x
>> λ|   (y'fp, y'offset, _y'len) = VS.unsafeToForeignPtr y
>> λ| :}
>> λ> 
>> λ> let !v = VS.fromList [5..20] in isSameVector (SomeVector v) 
>> (SomeVector v)
>> False
> 
> What happens for [3,2,5]?
> 
> 
>> λ> 
>> λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSameVector v v
>> True
>> λ> 
>> ```

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: [Haskell-cafe] How to ensure optimization for large immutable vectors to be shared w.r.t. Referential Transparency

[YueCompl via ghc-devs]

Thanks, it's a good idea. Unfortunately the result remains, making me even more 
curious.

```hs
λ> :set -XBangPatterns
λ> 
λ> :set -package vector
package flags have changed, resetting and loading new packages...
λ> 
λ> import Prelude
λ> 
λ> import Control.Monad.ST
λ> import qualified Data.Vector.Storable as VS
λ> 
λ> :{
λ| 
λ| newtype SomeVector = SomeVector (VS.Vector Int)
λ| 
λ| isSameVector :: SomeVector -> SomeVector -> Bool
λ| isSameVector (SomeVector !x) (SomeVector !y) = 
λ|   x'offset == y'offset && x'fp == y'fp
λ|  where
λ|   (x'fp, x'offset, _x'len) = VS.unsafeToForeignPtr x
λ|   (y'fp, y'offset, _y'len) = VS.unsafeToForeignPtr y
λ| :}
λ> 
λ> let !v = VS.fromList [5..20] in isSameVector (SomeVector v) (SomeVector 
v)
False
λ> 
λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSameVector v v
True
λ> 
```

> On 2021-04-06, at 20:00, Henning Thielemann  
> wrote:
> 
> 
> On Tue, 6 Apr 2021, YueCompl via Haskell-Cafe wrote:
> 
>> In an attempt to determine whether two immutable vectors can be treated as 
>> the same one to enable specific optimizations for that case, I tried to use 
>> ST to determine their respective backing foreign ptrs for comparison. But 
>> appears it can be copied when wrapped in a newtype, I wonder why it is the 
>> case, and how to avoid the copy?
> 
> You compare the ForeignPtrs of the mutable vectors. What about comparing the 
> ForeignPtrs of the original immutable vectors?

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

How to ensure optimization for large immutable vectors to be shared w.r.t. Referential Transparency

[YueCompl via ghc-devs]

Hello Cafe and respected GHC Devs,

I would like to ensure some immutable vectors (can be quite large) are always 
shared instead of copied, and I think that should be straight forward w.r.t. 
referential transparency we enjoy.

In an attempt to determine whether two immutable vectors can be treated as the 
same one to enable specific optimizations for that case, I tried to use ST to 
determine their respective backing foreign ptrs for comparison. But appears it 
can be copied when wrapped in a newtype, I wonder why it is the case, and how 
to avoid the copy?

Here's my minimum reproducible snippet:

```hs

λ> :set -XBangPatterns
λ> 
λ> :set -package vector
package flags have changed, resetting and loading new packages...
λ> 
λ> import Prelude
λ> 
λ> import Control.Monad.ST
λ> import qualified Data.Vector.Storable as VS
λ> 
λ> :{
λ| 
λ| newtype SomeVector = SomeVector (VS.Vector Int)
λ| 
λ| isSameVector :: SomeVector -> SomeVector -> Bool
λ| isSameVector (SomeVector !x) (SomeVector !y) = runST $ do
λ|   mx@(VS.MVector !x'offset !x'fp) <- VS.unsafeThaw x
λ|   my@(VS.MVector !y'offset !y'fp) <- VS.unsafeThaw y
λ|   _ <- VS.unsafeFreeze mx
λ|   _ <- VS.unsafeFreeze my
λ|   return $ x'offset == y'offset && x'fp == y'fp
λ| 
λ| :}
λ> 
λ> let !v = VS.fromList [3,2,5] in isSameVector (SomeVector v) (SomeVector v)
False
λ> 
λ> let !v = SomeVector (VS.fromList [3,2,5]) in isSameVector v v
True
λ> 

```

Thanks with best regards,
Compl

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

How to get dwarf stack trace of all live threads (HECs) on demand (e.g. SIGQUIT)?

[YueCompl via ghc-devs]

Dear GHC devs,

I recently encountered 100% ~ 400% CPU hogging situations to trouble-shoot, I 
managed to build and use a dwarf enabled GHC-8.10.3 on Ubuntu 18.04 and 20.04 
for that, then I find it seemingly dumps only the main thread's stack trace 
when I press Ctrl+\, but my program is a concurrent language server that will 
fork a new thread to handle each LSP request, so I really need to see what is 
going on those threads when CPU get hogged, as the situation is not trivially 
reproducible, only occurs occasionally after some time of active IDE activity.

And I can't figure out how to do that from the user guide, so help please!

Thanks & best regards,
Compl

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for larg

[YueCompl via ghc-devs]

And I have some uneducated guess also in 
https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132559.html 
<https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132559.html>

> And per my understanding, GHC's GC doesn't seek free segments within a heap, 
> it instead will copy all live objects to a new heap after then swap the new 
> heap to be the live one, so I assume memory (address space) fragmentation 
> doesn't make much trouble for a GHC process, as for other runtimes.

> I suspect the difficulty resides in the detection of circular/cyclic 
> circumstances wrt live data structures within the old heap, especially the 
> circles form with arbitrary number of pointers of indirection. If the GC has 
> to perform some dict lookup to decide if an object has been copied to new 
> heap, that's O(n*log(n)) complexity in best case, where n is number of live 
> objects in the heap.

> To efficiently copy circular structures, one optimization I can imagine is to 
> have a `new ptr` field in every heap object, then in copying another object 
> with a pointer to one object, the `new ptr` can be read out and if not nil, 
> assign the pointer field on another object' in the new heap to that value and 
> it's done; or copy one object' to the new heap, and update the field on one 
> object in the old heap pointing to the new heap. But I don't know details of 
> GHC GC and can't imagine even feasibility of this technique. And even the new 
> nonmoving GC may have similar difficulty to jump out of a circle when 
> following pointers.

FYI

> On 2020-07-30, at 21:53, YueCompl via ghc-devs  wrote:
> 
> Dear GHC Devs,
> 
> I realize I should seek your helps regarding my current situation.
> 
> TL;DR the most serious suffering is: After the heap loaded with many TVars 
> with cyclic structures, GC will dominate my CPU utilization with little 
> business progressing.
> 
> Nonmoving GC `-xn` with 8.10.1 helps, but the ceiling is not high enough for 
> my case, obvious performance degrade starts at about ~350MB RSS, and falls 
> unusable as RSS approaching 1GB. While I expect a GHC compiled process to 
> serve 20~30GB in-mem data practically okay.
> 
> https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html 
> <https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html> 
> contains most interesting conversations.
> 
> I found 
> https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-memory.html
>  
> <https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-memory.html>
>  much relevant, they managed to keep 100GB per instance, but switching to 
> immutable data structures to reside within compact regions is not immediately 
> feasible for my case, as the schema has to be re-designed, though I have my 
> colleges started evaluating that option.
> 
> Best regards,
> Compl
> 
> 
>> Begin forwarded message:
>> 
>> From: YueCompl via Haskell-Cafe > <mailto:haskell-c...@haskell.org>>
>> Subject: [Haskell-cafe] For STM to practically serve large in-mem datasets 
>> with cyclic structures WAS: STM friendly TreeMap (or similar with range scan 
>> api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM 
>> threads, with hot TVar updates?
>> Date: 2020-07-30 at 21:28:31 GMT+8
>> To: Haskell Cafe mailto:haskell-c...@haskell.org>>
>> Reply-To: YueCompl mailto:compl@icloud.com>>
>> 
>> For the record, overhead of STM over IO (or other means where manual 
>> composition of transactions needed) based concurrency control, is a price 
>> I'm willing to pay in my use case, as it's not machine-performance critical 
>> in distributing input data + parameters to a cluster of worker nodes, and 
>> collecting their results into permanent storage or a data pipeline. But to 
>> keep professionally crafting well synced, race-free scheduling code is 
>> barely affordable by my org, as shape of datasets, relationship between them 
>> and algorithms processing them are varying at fast paces, we have 
>> difficulty, or lack the willingness, to hire some workforce specifically to 
>> keep each new data pipeline race free, it has to be, but better at cost of 
>> machine-hours instead of human head counts. 
>> 
>> While easily compositing stm code, wrapped in scriptable procedures, will 
>> enable our analysts to author the scheduling scripts without too much 
>> concerns. Then our programmers can focus on performance critical parts of 
>> the data processing code, like optimization of tight-loops.
>> 
>> Only if not in the tight loops, I think it

Fwd: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for lar

[YueCompl via ghc-devs]

Dear GHC Devs,

I realize I should seek your helps regarding my current situation.

TL;DR the most serious suffering is: After the heap loaded with many TVars with 
cyclic structures, GC will dominate my CPU utilization with little business 
progressing.

Nonmoving GC `-xn` with 8.10.1 helps, but the ceiling is not high enough for my 
case, obvious performance degrade starts at about ~350MB RSS, and falls 
unusable as RSS approaching 1GB. While I expect a GHC compiled process to serve 
20~30GB in-mem data practically okay.

https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html 
 
contains most interesting conversations.

I found 
https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-memory.html
 

 much relevant, they managed to keep 100GB per instance, but switching to 
immutable data structures to reside within compact regions is not immediately 
feasible for my case, as the schema has to be re-designed, though I have my 
colleges started evaluating that option.

Best regards,
Compl


> Begin forwarded message:
> 
> From: YueCompl via Haskell-Cafe 
> Subject: [Haskell-cafe] For STM to practically serve large in-mem datasets 
> with cyclic structures WAS: STM friendly TreeMap (or similar with range scan 
> api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM 
> threads, with hot TVar updates?
> Date: 2020-07-30 at 21:28:31 GMT+8
> To: Haskell Cafe 
> Reply-To: YueCompl 
> 
> For the record, overhead of STM over IO (or other means where manual 
> composition of transactions needed) based concurrency control, is a price I'm 
> willing to pay in my use case, as it's not machine-performance critical in 
> distributing input data + parameters to a cluster of worker nodes, and 
> collecting their results into permanent storage or a data pipeline. But to 
> keep professionally crafting well synced, race-free scheduling code is barely 
> affordable by my org, as shape of datasets, relationship between them and 
> algorithms processing them are varying at fast paces, we have difficulty, or 
> lack the willingness, to hire some workforce specifically to keep each new 
> data pipeline race free, it has to be, but better at cost of machine-hours 
> instead of human head counts. 
> 
> While easily compositing stm code, wrapped in scriptable procedures, will 
> enable our analysts to author the scheduling scripts without too much 
> concerns. Then our programmers can focus on performance critical parts of the 
> data processing code, like optimization of tight-loops.
> 
> Only if not in the tight loops, I think it's acceptable by us, that up to 2~3 
> order of magnitude slower for an stm solution compared to its best rivals, as 
> long as it's scalable. For a (maybe cheating) example, if fully optimized 
> code can return result in 10 ms after an analyst clicked a button, we don't 
> bother if unoptimized stm script needs 10 second, so long as the result is 
> correct.
> 
> In a philosophic thinking, I heard that AT&T had UNIX specifically designed 
> for their Control panel, while their Data panel runs separate software (and 
> on separate hardware obviously), while modern systems have powerful CPUs 
> tempting us to squeeze more performance out of it, and SIMD instructions make 
> it even more tempting, I think we'd better resist it when programming 
> something belong to the Control panel per se, but do it in programming 
> something belong to the Data panel. And appears Data panel programs are being 
> shifted to GPUs nowadays, which feels right.
> 
> Regards,
> Compl
> 
> 
>> On 2020-07-30, at 20:10, YueCompl via Haskell-Cafe > > wrote:
>> 
>> Hi Peter,
>> 
>> Great to hear from you! 
>> 
>> For the record tskiplist (and stm-containers together) did improve my 
>> situation a great lot with respect to scalability at concurrency/parallelism!
>> 
>> I'm still far from the stage to squeeze last drops of performance, currently 
>> I'm just making sure performance wise concerns be reasonable during my PoC 
>> in correctness and ergonomics of my HPC architecture (an in-memory graph + 
>> out-of-core (mmap) array DBMS powered computation cluster, with shared 
>> storage), and after parallelism appears acceptable, I seemingly suffer from 
>> serious GC issue at up scaling on process working memory size. I'm 
>> suspecting it's because of the added more TVars and/or aggressive circular 
>> structures of them in my case, and can not find a way to overcome it by far.
>> 
>> Thanks for your detailed information!
>> 
>> Best regards,
>> Compl
>> 
>> 
>>> On 2020-07-30, at 19:19, Peter Robinson >> > wrote:
>>> 
>>> Hi Compl,
>>> >+ This package provides a proof-of-concept implementation of a skip list 
>>> >in STM
>>> 
>>> This has to mean something but I can't fi