[Python-Dev] How can we use 48bit pointer safely?

2018-03-29 Thread INADA Naoki 
Hi,

As far as I know, most amd64 and arm64 systems use only 48bit address spaces.
(except [1])

[1] 
https://software.intel.com/sites/default/files/managed/2b/80/5-level_paging_white_paper.pdf

It means there are some chance to compact some data structures.
I point two examples below.

My question is; can we use 48bit pointer safely?
It depends on CPU architecture & OS memory map.
Maybe, configure option which is available on only (amd64, amd64) *
(Linux, Windows, macOS)?


# Possible optimizations by 48bit pointer

## PyASCIIObject

[snip]
unsigned int ready:1;
/* Padding to ensure that PyUnicode_DATA() is always aligned to
   4 bytes (see issue #19537 on m68k). */
unsigned int :24;
} state;
wchar_t *wstr;  /* wchar_t representation (null-terminated) */
} PyASCIIObject;

Currently, state is 8bit + 24bit padding.  I think we can pack state and wstr
in 64bit.

## PyDictKeyEntry

typedef struct {
/* Cached hash code of me_key. */
Py_hash_t me_hash;
PyObject *me_key;
PyObject *me_value; /* This field is only meaningful for combined tables */
} PyDictKeyEntry;

There are chance to compact it: Use only 32bit for hash and 48bit*2
for key and value.  CompactEntry may be 16byte instead of 24byte.


Regards,
-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How can we use 48bit pointer safely?

2018-03-30 Thread Serhiy Storchaka 

30.03.18 09:28, INADA Naoki пише:

As far as I know, most amd64 and arm64 systems use only 48bit address spaces.
(except [1])

[1] 
https://software.intel.com/sites/default/files/managed/2b/80/5-level_paging_white_paper.pdf

It means there are some chance to compact some data structures.
I point two examples below.

My question is; can we use 48bit pointer safely?
It depends on CPU architecture & OS memory map.
Maybe, configure option which is available on only (amd64, amd64) *
(Linux, Windows, macOS)?


If the size be the main problem, we could use these 8 bit for encoding 
the type and the size of the value for some types, and even encode the 
value itself for some types in other 48 bits. For example 48 bit 
integers, 0-, 1- and 2-character Unicode strings, ASCII strings up to 6 
characters (and even longer if use base 64 encodings for ASCII 
identifiers), singletons like None, True, False, Ellipsis, 
NotImplementes could be encoded in 64 bit word without using additional 
memory.


But this would significantly complicate and slow down the code.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How can we use 48bit pointer safely?

2018-03-30 Thread Antoine Pitrou 
On Fri, 30 Mar 2018 15:28:47 +0900
INADA Naoki  wrote:
> Hi,
> 
> As far as I know, most amd64 and arm64 systems use only 48bit address spaces.
> (except [1])
> 
> [1] 
> https://software.intel.com/sites/default/files/managed/2b/80/5-level_paging_white_paper.pdf
> 
> It means there are some chance to compact some data structures.

As that paper shows, effective virtual address width tends to increase
over time to accomodate growing needs.  Bigger systems like IBM POWER
sytems may already have larger virtual address spaces.  So we can't
safely assume that bits 48-63 are available for us.

Another issue is the cost of the associated bit-twiddling.  It will all
depend how often it needs to be done.  Note that pointers can be
"negative", i.e. some of them will have all 1s in their upper bits, and
you need to reproduce that when reconstituting the original pointer.

A safer alternative is to use the *lower* bits of pointers.  The bottom
3 bits are always available for storing ancillary information, since
typically all heap-allocated data will be at least 8-bytes aligned
(probably 16-bytes aligned on 64-bit processes). However, you also get
less bits :-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How can we use 48bit pointer safely?

2018-03-30 Thread Ronald Oussoren 



On Mar 30, 2018, at 08:31 AM, INADA Naoki  wrote:

Hi,

As far as I know, most amd64 and arm64 systems use only 48bit address spaces.
(except [1])

[1] 
https://software.intel.com/sites/default/files/managed/2b/80/5-level_paging_white_paper.pdf

It means there are some chance to compact some data structures.
I point two examples below.

My question is; can we use 48bit pointer safely?

Not really, at least some CPUs can also address more memory than that. See 
 which talks about Linux support for 57-bit 
virtual addresses and 52-bit physical addresses. 

Ronald
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How can we use 48bit pointer safely?

2018-03-30 Thread Joao S. O. Bueno 
Not only that, but afaik  Linux could simply raise that 57bit virtual
to 64bit virtual without previous
warning on any version change.

On 30 March 2018 at 08:55, Ronald Oussoren  wrote:
>
>
> On Mar 30, 2018, at 08:31 AM, INADA Naoki  wrote:
>
> Hi,
>
> As far as I know, most amd64 and arm64 systems use only 48bit address
> spaces.
> (except [1])
>
> [1]
> https://software.intel.com/sites/default/files/managed/2b/80/5-level_paging_white_paper.pdf
>
> It means there are some chance to compact some data structures.
> I point two examples below.
>
> My question is; can we use 48bit pointer safely?
>
>
> Not really, at least some CPUs can also address more memory than that. See
>  which talks about Linux support for
> 57-bit virtual addresses and 52-bit physical addresses.
>
> Ronald
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How can we use 48bit pointer safely?

2018-03-30 Thread Ronald Oussoren 



On Mar 30, 2018, at 03:11 PM, "Joao S. O. Bueno"  wrote:

Not only that, but afaik Linux could simply raise that 57bit virtual
to 64bit virtual without previous
warning on any version change.

The change from 48-bit to 57-bit virtual addresses was not done without any 
warning because that would have broken too much code (IIRC due to at least some 
JS environments assuming 48bit pointers).

Ronald ___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How can we use 48bit pointer safely?

2018-03-30 Thread Ronald Oussoren 



On Mar 30, 2018, at 01:40 PM, Antoine Pitrou  wrote:


A safer alternative is to use the *lower* bits of pointers. The bottom
3 bits are always available for storing ancillary information, since
typically all heap-allocated data will be at least 8-bytes aligned
(probably 16-bytes aligned on 64-bit processes). However, you also get
less bits :-)

The lower bits are more interesting to use. I'm still hoping to find some time 
to experiment with tagged pointers some day, that could be interesting w.r.t. 
performance and memory use (at the cost of being ABI incompatible). 

Ronald
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com