date:20120920

Re: Why is Bag's Data instance "broken"?

2012-09-20 Thread José Pedro Magalhães

On Wed, Aug 29, 2012 at 12:01 PM, Philip Holzenspies
wrote:

> Dear GHCers,
>
> I'm performing traversals over GHC-API results (HsSyn et al). For this
> purpose, I'm using SYB generics.
>
> I found that I couldn't use "ext1Q" for a function with type "Data x =>
> Bag x -> String", i.e. that this function was never applied. The source of
> Bag's instance of the Data class seems to explain why:
>
>
> instance Data a => Data (Bag a) where
>   gfoldl k z b = z listToBag `k` bagToList b -- traverse abstract type
> abstractly
>   toConstr _   = abstractConstr $ "Bag("++show (typeOf (undefined::a))++")"
>   gunfold _ _  = error "gunfold"
>   dataTypeOf _ = mkNoRepType "Bag"
>

Btw, where is this instance defined?


Thanks,
Pedro


>
>
> Is there a rationale to not allow gunfolds and to keep toConstr abstract?
> More to the point for my needs, is there a reason to not allow dataCast1
> casting of Bags?
>
> Regards,
> Philip
> ___
> Glasgow-haskell-users mailing list
> Glasgow-haskell-users@haskell.org
> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: Why is Bag's Data instance "broken"?

2012-09-20 Thread Philip Holzenspies

On 20 Sep 2012, at 09:40, José Pedro Magalhães wrote:

instance Data a => Data (Bag a) where
  gfoldl k z b = z listToBag `k` bagToList b -- traverse abstract type 
abstractly
  toConstr _   = abstractConstr $ "Bag("++show (typeOf (undefined::a))++")"
  gunfold _ _  = error "gunfold"
  dataTypeOf _ = mkNoRepType "Bag"

Btw, where is this instance defined?

GHCROOT/compiler/utils/Bag.lhs, lines 266-270 (current git-repo HEAD and many 
versions previously).

Ph.
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: Why is Bag's Data instance "broken"?

2012-09-20 Thread Edward Kmett

Note: It was probably built with an eye towards how Data.Map and the like
performed abstraction. However, This isn't necessary to protect the
invariants of a bag.

The constructors exposed via Data do not have to be the actual constructors
of the data type. With this you can quotient out the portions of the
structure you don't want the user to be able to inspect.

See the libraries@ proposal that I put in 3-4 weeks ago (which will have
just passed) to fix all the broken Data instances for containers by using
virtual constructors such as 'fromList', (which incidentally led to Milan
finding huge space and time improvements in fromList).

Effectively allowing the user to use the 'listToBag' as a "constructor"
loses no information violates no invariants, and prevents code written for
uniplate, SYB, etc. from having to crash, panic or give up upon the sight
of a mkNoRepType.

My reaction for years to the sight of a mkNoRepType and undefined gunfold
has been to hang my head. Now I just fix them.

-Edward

On Wed, Aug 29, 2012 at 7:11 AM, José Pedro Magalhães  wrote:

> Hi Philip,
>
> On Wed, Aug 29, 2012 at 12:01 PM, Philip Holzenspies <
> p...@st-andrews.ac.uk> wrote:
>
>> Dear GHCers,
>>
>> I'm performing traversals over GHC-API results (HsSyn et al). For this
>> purpose, I'm using SYB generics.
>>
>> I found that I couldn't use "ext1Q" for a function with type "Data x =>
>> Bag x -> String", i.e. that this function was never applied. The source of
>> Bag's instance of the Data class seems to explain why:
>>
>>
>> instance Data a => Data (Bag a) where
>>   gfoldl k z b = z listToBag `k` bagToList b -- traverse abstract type
>> abstractly
>>   toConstr _   = abstractConstr $ "Bag("++show (typeOf
>> (undefined::a))++")"
>>   gunfold _ _  = error "gunfold"
>>   dataTypeOf _ = mkNoRepType "Bag"
>>
>>
>> Is there a rationale to not allow gunfolds and to keep toConstr abstract?
>
>
> As far as I understand, this is to keep `Bag` itself abstract, preventing
> users from inspecting its internals.
>
>
>> More to the point for my needs, is there a reason to not allow dataCast1
>> casting of Bags?
>>
>
> That is a separate issue; I believe this instance is just missing a
> `dataCast1 = gcast1` line.
> All datatypes of kind `* -> *` should have such a definition.
>
> (Having a look at Data.Data, I guess the same applies to `Ptr a` and
> `ForeignPtr a`.
> And `Array a b` seems to be missing the `dataCast2` method. I propose
> fixing all of these.)
>
>
> Cheers,
> Pedro
>
>
>>
>> Regards,
>> Philip
>> ___
>> Glasgow-haskell-users mailing list
>> Glasgow-haskell-users@haskell.org
>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>>
>
>
> ___
> Glasgow-haskell-users mailing list
> Glasgow-haskell-users@haskell.org
> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
>
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: Why is Bag's Data instance "broken"?

2012-09-20 Thread José Pedro Magalhães

Right now I was just planning to fix the missing dataCast1 from Bag, and
the rest from
Data.Data (see http://hackage.haskell.org/trac/ghc/ticket/7256). I think
those are just
a bug, unrelated to the abstraction story, no?


Cheers,
Pedro

On Thu, Sep 20, 2012 at 12:19 PM, Edward Kmett  wrote:

> Note: It was probably built with an eye towards how Data.Map and the like
> performed abstraction. However, This isn't necessary to protect the
> invariants of a bag.
>
> The constructors exposed via Data do not have to be the actual
> constructors of the data type. With this you can quotient out the portions
> of the structure you don't want the user to be able to inspect.
>
> See the libraries@ proposal that I put in 3-4 weeks ago (which will have
> just passed) to fix all the broken Data instances for containers by using
> virtual constructors such as 'fromList', (which incidentally led to Milan
> finding huge space and time improvements in fromList).
>
> Effectively allowing the user to use the 'listToBag' as a "constructor"
> loses no information violates no invariants, and prevents code written for
> uniplate, SYB, etc. from having to crash, panic or give up upon the sight
> of a mkNoRepType.
>
> My reaction for years to the sight of a mkNoRepType and undefined gunfold
> has been to hang my head. Now I just fix them.
>
> -Edward
>
> On Wed, Aug 29, 2012 at 7:11 AM, José Pedro Magalhães wrote:
>
>> Hi Philip,
>>
>> On Wed, Aug 29, 2012 at 12:01 PM, Philip Holzenspies <
>> p...@st-andrews.ac.uk> wrote:
>>
>>> Dear GHCers,
>>>
>>> I'm performing traversals over GHC-API results (HsSyn et al). For this
>>> purpose, I'm using SYB generics.
>>>
>>> I found that I couldn't use "ext1Q" for a function with type "Data x =>
>>> Bag x -> String", i.e. that this function was never applied. The source of
>>> Bag's instance of the Data class seems to explain why:
>>>
>>>
>>> instance Data a => Data (Bag a) where
>>>   gfoldl k z b = z listToBag `k` bagToList b -- traverse abstract type
>>> abstractly
>>>   toConstr _   = abstractConstr $ "Bag("++show (typeOf
>>> (undefined::a))++")"
>>>   gunfold _ _  = error "gunfold"
>>>   dataTypeOf _ = mkNoRepType "Bag"
>>>
>>>
>>> Is there a rationale to not allow gunfolds and to keep toConstr abstract?
>>
>>
>> As far as I understand, this is to keep `Bag` itself abstract, preventing
>> users from inspecting its internals.
>>
>>
>>> More to the point for my needs, is there a reason to not allow dataCast1
>>> casting of Bags?
>>>
>>
>> That is a separate issue; I believe this instance is just missing a
>> `dataCast1 = gcast1` line.
>> All datatypes of kind `* -> *` should have such a definition.
>>
>> (Having a look at Data.Data, I guess the same applies to `Ptr a` and
>> `ForeignPtr a`.
>> And `Array a b` seems to be missing the `dataCast2` method. I propose
>> fixing all of these.)
>>
>>
>> Cheers,
>> Pedro
>>
>>
>>>
>>> Regards,
>>> Philip
>>> ___
>>> Glasgow-haskell-users mailing list
>>> Glasgow-haskell-users@haskell.org
>>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>>>
>>
>>
>> ___
>> Glasgow-haskell-users mailing list
>> Glasgow-haskell-users@haskell.org
>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>>
>>
>
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: Why is Bag's Data instance "broken"?

2012-09-20 Thread Edward Kmett

The missing dataCast1 is just a bug, yes. I suppose someone who uses Bag
and cares can submit something about fixing gunfold.

On Thu, Sep 20, 2012 at 7:22 AM, José Pedro Magalhães  wrote:

> Right now I was just planning to fix the missing dataCast1 from Bag, and
> the rest from
> Data.Data (see http://hackage.haskell.org/trac/ghc/ticket/7256). I think
> those are just
> a bug, unrelated to the abstraction story, no?
>
>
> Cheers,
> Pedro
>
>
> On Thu, Sep 20, 2012 at 12:19 PM, Edward Kmett  wrote:
>
>> Note: It was probably built with an eye towards how Data.Map and the like
>> performed abstraction. However, This isn't necessary to protect the
>> invariants of a bag.
>>
>> The constructors exposed via Data do not have to be the actual
>> constructors of the data type. With this you can quotient out the portions
>> of the structure you don't want the user to be able to inspect.
>>
>> See the libraries@ proposal that I put in 3-4 weeks ago (which will have
>> just passed) to fix all the broken Data instances for containers by using
>> virtual constructors such as 'fromList', (which incidentally led to Milan
>> finding huge space and time improvements in fromList).
>>
>> Effectively allowing the user to use the 'listToBag' as a "constructor"
>> loses no information violates no invariants, and prevents code written for
>> uniplate, SYB, etc. from having to crash, panic or give up upon the sight
>> of a mkNoRepType.
>>
>> My reaction for years to the sight of a mkNoRepType and undefined gunfold
>> has been to hang my head. Now I just fix them.
>>
>> -Edward
>>
>> On Wed, Aug 29, 2012 at 7:11 AM, José Pedro Magalhães wrote:
>>
>>> Hi Philip,
>>>
>>> On Wed, Aug 29, 2012 at 12:01 PM, Philip Holzenspies <
>>> p...@st-andrews.ac.uk> wrote:
>>>
 Dear GHCers,

 I'm performing traversals over GHC-API results (HsSyn et al). For this
 purpose, I'm using SYB generics.

 I found that I couldn't use "ext1Q" for a function with type "Data x =>
 Bag x -> String", i.e. that this function was never applied. The source of
 Bag's instance of the Data class seems to explain why:


 instance Data a => Data (Bag a) where
   gfoldl k z b = z listToBag `k` bagToList b -- traverse abstract type
 abstractly
   toConstr _   = abstractConstr $ "Bag("++show (typeOf
 (undefined::a))++")"
   gunfold _ _  = error "gunfold"
   dataTypeOf _ = mkNoRepType "Bag"


 Is there a rationale to not allow gunfolds and to keep toConstr
 abstract?
>>>
>>>
>>> As far as I understand, this is to keep `Bag` itself abstract,
>>> preventing users from inspecting its internals.
>>>
>>>
 More to the point for my needs, is there a reason to not allow
 dataCast1 casting of Bags?

>>>
>>> That is a separate issue; I believe this instance is just missing a
>>> `dataCast1 = gcast1` line.
>>> All datatypes of kind `* -> *` should have such a definition.
>>>
>>> (Having a look at Data.Data, I guess the same applies to `Ptr a` and
>>> `ForeignPtr a`.
>>> And `Array a b` seems to be missing the `dataCast2` method. I propose
>>> fixing all of these.)
>>>
>>>
>>> Cheers,
>>> Pedro
>>>
>>>

 Regards,
 Philip
 ___
 Glasgow-haskell-users mailing list
 Glasgow-haskell-users@haskell.org
 http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

>>>
>>>
>>> ___
>>> Glasgow-haskell-users mailing list
>>> Glasgow-haskell-users@haskell.org
>>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>>>
>>>
>>
>
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: How do I build GHC 7.6 from source?

2012-09-20 Thread Simon Marlow

On 19/09/2012 02:15, Iavor Diatchki wrote:

  > exactly what git's submodule machinery does, so it seems pointless to

 > implement the functionality which is already there with a standard
 > interface.  Thoughts?

http://hackage.haskell.org/trac/ghc/wiki/DarcsConversion#Theperspectiveonsubmodules

I have seen this.  Our custom "fingerprint" solution has the exact same
drawbacks (because it does the exact same thing as sub-modules), and in
addition it has the drawback of
   1. being a custom non-standard solution,
   2. it is not obvious where to find the "fingerprint" associated with
a particular branch (which is what lead to my question in the first place).

Well, it doesn't quite have the same drawbacks as submodules, because 
our solution places a burden only on someone who wants to recover a 
particular repository state, rather than on everyone doing development.

I think it's worth keeping an eye on submodules in case they fix the 
gotchas in the UI, but at the moment it looks like we'd have a lot of 
confused developers, lost work and accidental breakages due to people 
not understanding how submodules work or forgetting to jump through the 
correct hoops.

I'm not saying fingerprints are a good solution, obviously they only 
solve a part of the problem, but the current tooling for submodules 
leaves a lot to be desired.

Cheers,
Simon

___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: How do I build GHC 7.6 from source?

2012-09-20 Thread Claus Reinke

Well, it doesn't quite have the same drawbacks as submodules, because our solution places a burden 
only on someone who wants to recover a particular repository state, rather than on everyone doing 
development.


I think it's worth keeping an eye on submodules in case they fix the gotchas in the UI, but at the 
moment it looks like we'd have a lot of confused developers, lost work and accidental breakages 
due to people not understanding how submodules work or forgetting to jump through the correct 
hoops.


I'm not saying fingerprints are a good solution, obviously they only solve a part of the problem, 
but the current tooling for submodules leaves a lot to be desired.


Since you already have fingerprinting in place, have you looked
into 'git subtree' for the remaining functionality? Used to be an
addon module, has recently been merged into git (contrib/).

http://apenwarr.ca/log/?m=200904#02

https://github.com/gitster/git/blob/634392b26275fe5436c0ea131bc89b46476aa4ae/contrib/subtree/git-subtree.txt

https://github.com/apenwarr/git-subtree

I don't have further information or recommendations, just wanted
to pass on this alternative - it tends to come up when googling for
'git submodule' issues, and most reports on it have been favorable
(no idea whether it would work for the size and complexity of the
ghc&friends repo landscape).

Claus



___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: How do I build GHC 7.6 from source?

2012-09-20 Thread Iavor Diatchki

Hello,

perhaps we should have a well-defined place in the repo where we keep the
finger-prints associated with tags and branches in the main repo?
This would make it a lot easier to get to a fully defined
previous/different state.

On this note, could someone send the link to the 7.6 fingerprint?  Ian said
that it is somewhere in the nightly build logs but I don't where to look.

-Iavor



On Thu, Sep 20, 2012 at 7:20 AM, Simon Marlow  wrote:

> On 19/09/2012 02:15, Iavor Diatchki wrote:
>
>> exactly what git's submodule machinery does, so it seems pointless to
>>
>>  > implement the functionality which is already there with a standard
>>  > interface.  Thoughts?
>>
>> http://hackage.haskell.org/**trac/ghc/wiki/DarcsConversion#**
>> Theperspectiveonsubmodules
>>
>>
>> I have seen this.  Our custom "fingerprint" solution has the exact same
>> drawbacks (because it does the exact same thing as sub-modules), and in
>> addition it has the drawback of
>>1. being a custom non-standard solution,
>>2. it is not obvious where to find the "fingerprint" associated with
>> a particular branch (which is what lead to my question in the first
>> place).
>>
>
>
> Well, it doesn't quite have the same drawbacks as submodules, because our
> solution places a burden only on someone who wants to recover a particular
> repository state, rather than on everyone doing development.
>
> I think it's worth keeping an eye on submodules in case they fix the
> gotchas in the UI, but at the moment it looks like we'd have a lot of
> confused developers, lost work and accidental breakages due to people not
> understanding how submodules work or forgetting to jump through the correct
> hoops.
>
> I'm not saying fingerprints are a good solution, obviously they only solve
> a part of the problem, but the current tooling for submodules leaves a lot
> to be desired.
>
> Cheers,
> Simon
>
>
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

memory fragmentation with ghc-7.6.1

2012-09-20 Thread John Lato

Hello,

We've noticed that some applications exhibit significantly worse
memory usage when compiled with ghc-7.6.1 compared to ghc-7.4, leading
to out of memory errors in some cases.  Running one app with +RTS -s,
I see this:

ghc-7.4
 525,451,699,736 bytes allocated in the heap
  53,404,833,048 bytes copied during GC
  39,097,600 bytes maximum residency (2439 sample(s))
   1,547,040 bytes maximum slop
 628 MB total memory in use (0 MB lost due to fragmentation)

ghc-7.6
512,535,907,752 bytes allocated in the heap
  53,327,184,712 bytes copied during GC
  40,038,584 bytes maximum residency (2391 sample(s))
   1,456,472 bytes maximum slop
3414 MB total memory in use (2744 MB lost due to fragmentation)

The total memory in use (consistent with 'top's output) is much higher
when built with ghc-7.6, due entirely to fragmentation.

I've filed a bug report
(http://hackage.haskell.org/trac/ghc/ticket/7257,
http://hpaste.org/74987), but I was wondering if anyone else has
noticed this?  I'm not entirely sure what's triggering this behavior
(some applications work fine), although I suspect it has to do with
allocation of pinned memory.

John L.

___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: memory fragmentation with ghc-7.6.1

2012-09-20 Thread Carter Schonwald

So the problem is only with the data structures on the heap that are pinned
in place to play nice with C?

I'd be curious to understand the change too, though per se pinned memory (a
la storable or or bytestring) will by definition cause memory fragmentation
in a gc'd lang as a rule,  (or at least one like Haskell).
-Carter

On Thu, Sep 20, 2012 at 8:59 PM, John Lato  wrote:

> Hello,
>
> We've noticed that some applications exhibit significantly worse
> memory usage when compiled with ghc-7.6.1 compared to ghc-7.4, leading
> to out of memory errors in some cases.  Running one app with +RTS -s,
> I see this:
>
> ghc-7.4
>  525,451,699,736 bytes allocated in the heap
>   53,404,833,048 bytes copied during GC
>   39,097,600 bytes maximum residency (2439 sample(s))
>1,547,040 bytes maximum slop
>  628 MB total memory in use (0 MB lost due to fragmentation)
>
> ghc-7.6
> 512,535,907,752 bytes allocated in the heap
>   53,327,184,712 bytes copied during GC
>   40,038,584 bytes maximum residency (2391 sample(s))
>1,456,472 bytes maximum slop
> 3414 MB total memory in use (2744 MB lost due to fragmentation)
>
> The total memory in use (consistent with 'top's output) is much higher
> when built with ghc-7.6, due entirely to fragmentation.
>
> I've filed a bug report
> (http://hackage.haskell.org/trac/ghc/ticket/7257,
> http://hpaste.org/74987), but I was wondering if anyone else has
> noticed this?  I'm not entirely sure what's triggering this behavior
> (some applications work fine), although I suspect it has to do with
> allocation of pinned memory.
>
> John L.
>
> ___
> Glasgow-haskell-users mailing list
> Glasgow-haskell-users@haskell.org
> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: memory fragmentation with ghc-7.6.1

2012-09-20 Thread John Lato

Yes, that's my current understanding.  I see this with ByteString and
Data.Vector.Storable, but not
Data.Vector/Data.Vector.Unboxed/Data.Text.  As ByteStrings are pretty
widely used for IO, I expected that somebody else would have
experienced this too.

I would expect some memory fragmentation with pinned memory, but the
change from ghc-7.4 to ghc-7.6 is rather extreme (no fragmentation to
several GB).

John L.

On Fri, Sep 21, 2012 at 10:53 AM, Carter Schonwald
 wrote:
> So the problem is only with the data structures on the heap that are pinned
> in place to play nice with C?
>
> I'd be curious to understand the change too, though per se pinned memory (a
> la storable or or bytestring) will by definition cause memory fragmentation
> in a gc'd lang as a rule,  (or at least one like Haskell).
> -Carter
>
> On Thu, Sep 20, 2012 at 8:59 PM, John Lato  wrote:
>>
>> Hello,
>>
>> We've noticed that some applications exhibit significantly worse
>> memory usage when compiled with ghc-7.6.1 compared to ghc-7.4, leading
>> to out of memory errors in some cases.  Running one app with +RTS -s,
>> I see this:
>>
>> ghc-7.4
>>  525,451,699,736 bytes allocated in the heap
>>   53,404,833,048 bytes copied during GC
>>   39,097,600 bytes maximum residency (2439 sample(s))
>>1,547,040 bytes maximum slop
>>  628 MB total memory in use (0 MB lost due to fragmentation)
>>
>> ghc-7.6
>> 512,535,907,752 bytes allocated in the heap
>>   53,327,184,712 bytes copied during GC
>>   40,038,584 bytes maximum residency (2391 sample(s))
>>1,456,472 bytes maximum slop
>> 3414 MB total memory in use (2744 MB lost due to
>> fragmentation)
>>
>> The total memory in use (consistent with 'top's output) is much higher
>> when built with ghc-7.6, due entirely to fragmentation.
>>
>> I've filed a bug report
>> (http://hackage.haskell.org/trac/ghc/ticket/7257,
>> http://hpaste.org/74987), but I was wondering if anyone else has
>> noticed this?  I'm not entirely sure what's triggering this behavior
>> (some applications work fine), although I suspect it has to do with
>> allocation of pinned memory.
>>
>> John L.
>>
>> ___
>> Glasgow-haskell-users mailing list
>> Glasgow-haskell-users@haskell.org
>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
>

___
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

Re: Why is Bag's Data instance "broken"?

Re: Why is Bag's Data instance "broken"?

Re: Why is Bag's Data instance "broken"?

Re: Why is Bag's Data instance "broken"?

Re: Why is Bag's Data instance "broken"?

Re: How do I build GHC 7.6 from source?

Re: How do I build GHC 7.6 from source?

Re: How do I build GHC 7.6 from source?

memory fragmentation with ghc-7.6.1

Re: memory fragmentation with ghc-7.6.1

Re: memory fragmentation with ghc-7.6.1

11 matches

Site Navigation

Mail list logo

Footer information