[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Marco Sulla 
On Thu, 30 Jul 2020 at 23:42, Steven D'Aprano  wrote:

> Of course anything is possible in theory, but I think that would require
> shifting the existing keys and values in the array, and updating the
> entries in the hash table to point to their key in the new position, and
> probably other complications I haven't thought of.
>

Also list must shift the internal array. You don't have to update the
hashtable. For example, when you pop an item, the key is not removed from
the hashtable completely, but it's marked as dummy.

I think the main problem here is set, since set is not ordered. In theory,
dict views could be an "ordered" set, so you can do:

d.keys()[5]
d.values()[1:]
d.items()[3] = "omg"

In any case, if people want to propose turning dicts into fully-fledged
> reorderable, slicable sequences as well as mappings, they will probably
> need a PEP :-)
>

I'm just brainstorming. Not sure about the real use-case.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EF3FRDGV5DPTEIFY2DVRTAMEUUSOEFV7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Guido van Rossum 
On Thu, Jul 30, 2020 at 22:23 Greg Ewing 
wrote:

> On 31/07/20 4:04 am, Christopher Barker wrote:
> > (Side note: why ARE the views provided by methods, rather than
> > properties?)
>
> Probably because they construct objects, and are therefore
> somewhat more expensive than is usually expected for an
> attribute access.


Also for compatibility (even if imperfect) with Python 2. It’s water over
the dam now.

—Guido

-- 
--Guido (mobile)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RPAYXPAB24V7EDINJLHIW4FVHR4DL7FM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Greg Ewing 

On 31/07/20 4:04 am, Christopher Barker wrote:
(Side note: why ARE the views provided by methods, rather than 
properties?


Probably because they construct objects, and are therefore
somewhat more expensive than is usually expected for an
attribute access.

--
Greg
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2SJXX4PDFTEC7YSZVGGWNCXSOGLUXK2S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Steven D'Aprano 
On Thu, Jul 30, 2020 at 06:17:24PM -0400, Wes Turner wrote:

> What new syntax do you propose?

I don't propose new syntax. If this functionality is needed, regular 
method syntax will do the job without misleading people to th8ink that 
dicts are sequences that ought to support the full sequence API or the 
full set of list methods. One or the other of these methods should be 
sufficient for the uses I've seen:

dict.getkey(index)  # return key
dict.getitem(index) # return (key, value)

I'm not even convinced that there is a compelling use-case for this 
functionality at all, but one or both of the above seem more than 
sufficient. Once you have the key, you can do everything else.

If there is a need to get the insertion index of a key, that could also 
be a method, but let's hold off adding that unless there is a good 
reason.


-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PKKO4XFUC5BIQZKXMWM2VJGO4GDELYAW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Wes Turner 
What new syntax do you propose?

On Thu, Jul 30, 2020 at 5:55 PM Steven D'Aprano  wrote:

> On Thu, Jul 30, 2020 at 05:03:58PM -0400, Wes Turner wrote:
>
> > Would these be the *non-mutating* methods desired of insertion-ordered
> > dicts?
> >
> > .iloc[sli:ce]
> > .iloc[int]
> > .iloc[[list,]]
> > .iloc[callable]
> > .iloc[bitmask]
> >
> > .index(key)
>
> No.
>
> "iloc" sounds like a new Apple product to keep strangers out of your
> home, or maybe something to do with iterators.
>
> Both slicing and "[list,]" arguments can be easily performed using a
> comprehension.
>
> I have no idea what calling it with a callable or bitmask is intended to
> do, or why they would be useful, but bitmasks are usually ints, so how
> would it distinguish between the item at index 3 and the item with
> bitmask 3?
>
> Is there a use-case for retrieving the insertion position of a key, or
> are we just adding it because we can?
>
>
> --
> Steven
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/XWQZD45EE6MY2NKV3UYN7GZXYMYZPO2E/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LPG74OMOAWWKO7BBTWUJPIVSNIBNKRRV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Steven D'Aprano 
On Thu, Jul 30, 2020 at 05:03:58PM -0400, Wes Turner wrote:

> Would these be the *non-mutating* methods desired of insertion-ordered
> dicts?
> 
> .iloc[sli:ce]
> .iloc[int]
> .iloc[[list,]]
> .iloc[callable]
> .iloc[bitmask]
> 
> .index(key)

No.

"iloc" sounds like a new Apple product to keep strangers out of your 
home, or maybe something to do with iterators.

Both slicing and "[list,]" arguments can be easily performed using a 
comprehension.

I have no idea what calling it with a callable or bitmask is intended to 
do, or why they would be useful, but bitmasks are usually ints, so how 
would it distinguish between the item at index 3 and the item with 
bitmask 3?

Is there a use-case for retrieving the insertion position of a key, or 
are we just adding it because we can?


-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XWQZD45EE6MY2NKV3UYN7GZXYMYZPO2E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Steven D'Aprano 
On Thu, Jul 30, 2020 at 09:35:31PM +0200, Marco Sulla wrote:
> On Thu, 30 Jul 2020 at 19:24, Steven D'Aprano  wrote:
> 
> > You can't insert a key in a specific position. If I have this dict:
> >
> > mydict = {'a': 1, 'c': 3, 'd': 4, 'e': 5}
> >
> > I can't insert 'b':2 between keys 'a' and 'c', except by creating a new
> > dict.
> >
> 
> Not sure about this. In C code, dicts are a hashtable and an array of
> items. In theory, nothing prevents you from inserting a new key in a
> specific position of the key array instead of at the end.

Of course anything is possible in theory, but I think that would require 
shifting the existing keys and values in the array, and updating the 
entries in the hash table to point to their key in the new position, and 
probably other complications I haven't thought of.

Of course it is possible to have a data structure with O(1) insertions 
and updates by key, and O(1) insertions and updates by index, and fully 
reorderable keys. But it probably won't be small and lean and fast. Just 
because something takes constant time doesn't mean it will be a *fast* 
constant time.

In any case, if people want to propose turning dicts into fully-fledged 
reorderable, slicable sequences as well as mappings, they will probably 
need a PEP :-)


-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NLT46M65Y5326AETV6NFEXJONCQZCC65/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Wes Turner 
Are you suggesting that dict views would become subscriptable?

In [3]: %doctest_mode
Exception reporting mode: Plain
Doctest mode is: ON
>>> x = dict(a=2, b=3, c=4)
>>> x.items()[3]
Traceback (most recent call last):
  File "", line 1, in 
x.items()[3]
TypeError: 'dict_items' object is not subscriptable

On Thu, Jul 30, 2020, 12:05 PM Christopher Barker 
wrote:

> On Wed, Jul 29, 2020 at 7:23 AM Wes Turner  wrote:
>
>> .iloc[] is the Pandas function for accessing by integer-location:
>>
>> https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html
>>
>
> I'm not sure I quite get the analogy here, but ...
>
> >>> odict.iloc[3]
>>
>>
>> Would this be the only way to access only item 4 from an odict of length
>> greater than 4 with slice notation for dict views generated from selection
>> by integer-location?
>>
>
> are suggesting that we add another attribute to dicts to provide "index"
> access? I'm not sure the advantage of that, as we already have the dict
> views, so I guess it's nice not to have the type the parentheses:
>
> odict.items()[3] IS a bit klunkier than odict.iloc[3]
>
> (Side note: why ARE the views provided by methods, rather than properties?
> But I digress ...)
>
>
>> >>> odict[3:4]
>>
>
> that would be possible.
>
> What does this do? Confusing to a beginner:
>>
>> >>> odict[3,]
>>
>
> no more confusing than it is for Sequences, is it?
>
> In [4]: l[3,]
>
> ---
> TypeError Traceback (most recent call last)
>  in 
> > 1 l[3,]
>
> TypeError: list indices must be integers or slices, not tuple
>
> But yes, if we allowed dicts to be indexable with slices, they still
> couldn't be indexed by single indexes (unless that value happened to be a
> key) so there would be no way to get a single item by index, as a length-1
> slice would presumably return a dict with one item.
>
> So yeah, if indexing, slicing were to happen, it would have to be in the
> views.
>
> But this does bring up that there are a lot of places where slice notation
> could be used outside of [] -- and that would be handy for use cases like
> pandas, even if not for dicts.
>
> -CHB
>
> --
> Christopher Barker, PhD
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TH54BGX66NQ2VECS3XO33ADY3GRTX5NU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Wes Turner 
On Thu, Jul 30, 2020, 3:37 PM Marco Sulla 
wrote:

> On Thu, 30 Jul 2020 at 19:24, Steven D'Aprano  wrote:
>
>> You can't insert a key in a specific position. If I have this dict:
>>
>> mydict = {'a': 1, 'c': 3, 'd': 4, 'e': 5}
>>
>> I can't insert 'b':2 between keys 'a' and 'c', except by creating a new
>> dict.
>>
>
> Not sure about this. In C code, dicts are a hashtable and an array of
> items. In theory, nothing prevents you from inserting a new key in a
> specific position of the key array instead of at the end.
>

Nothing but the cost of shifting successive elements by 1 and sometimes
copying the entire array to a new, larger array.

Would these be the *non-mutating* methods desired of insertion-ordered
dicts?

.iloc[sli:ce]
.iloc[int]
.iloc[[list,]]
.iloc[callable]
.iloc[bitmask]

.index(key)



___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/PX6OBI4FFJXLLTWWSA5O7SKCD3L6KMWP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TIQKQ3WP2NMC6MIESMOBCW2SAHXUP74T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Marco Sulla 
On Thu, 30 Jul 2020 at 19:24, Steven D'Aprano  wrote:

> You can't insert a key in a specific position. If I have this dict:
>
> mydict = {'a': 1, 'c': 3, 'd': 4, 'e': 5}
>
> I can't insert 'b':2 between keys 'a' and 'c', except by creating a new
> dict.
>

Not sure about this. In C code, dicts are a hashtable and an array of
items. In theory, nothing prevents you from inserting a new key in a
specific position of the key array instead of at the end.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PX6OBI4FFJXLLTWWSA5O7SKCD3L6KMWP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Steven D'Aprano 
On Thu, Jul 30, 2020 at 08:52:34AM -0700, Christopher Barker wrote:

> The point being that dicts are now ordered, so maybe it makes sense to add
> some operations that make it natural and easy to work with them in an
> ordered way -- you know, like slicing :-)

Dicts are not sequences and cannot support ordering operations like 
insert, sort, prepend, reverse or slice assignment. They merely preserve 
insertion order.

You can't insert a key in a specific position. If I have this dict:

mydict = {'a': 1, 'c': 3, 'd': 4, 'e': 5}

I can't insert 'b':2 between keys 'a' and 'c', except by creating a new 
dict. Nor can I sort the keys, or reverse them, except by putting them 
into a list and sorting or reversing the list, and only then putting 
them into a dict.

We could read the key and/or value at index N, that is straight forward 
if we can decide on a colour of the bike shed. But I don't believe that 
there is an straight forward way of replacing the key at index N, or of 
inserting keys, or swapping them, or directly changing their order.

Dicts are, sort of, like a stack: you can "insert" (append) new keys at 
the end, but not in the middle.

So let's please stop over-generalising by saying dicts are "ordered" 
without the qualifier that they only preserve insertion order, just like 
collections.OrderedDict.

The minimal API we could support would be a getter that returns the key 
at index N. Once you have the key, you can get or set the value. So all 
we really need is that one single getter. Anything else is gravy.

dict.getkey(n)  # returns the key at index n by insertion order

I wouldn't object to:

dict.getitem(n)  # returns (key, value) at index n

since it's hardly more trouble to return both the key and the value at 
the same time. But more than that seems difficult to justify, and 
supporting slices seems like overkill. You can always get them with a 
comprehension:

[mydict.getkey(n) for n in range(3, 49, 7)]


-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ICROBIPAJEXZUV3J7U2QTTCGUL7HIODU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread David Mertz 
On Thu, Jul 30, 2020 at 11:55 AM Christopher Barker 
wrote:

>  > smaller_dict = dict(islice(large_dict.items(), 0, 255))
> well, beauty is in the eye of the beholder, of course. But really, you
> think it's pretty to import itertools, then make a function call, for
> what's very much a slicing operation?
>

In your post that introduced that,that is *exactly* what I thought of when
you first described the issue, before I read down to your solution.  It
took a fraction of a second to think of for me.  It *is* slightly verbose.
What do you think about this in current Python:

>>> large = {i:i**2 for i in range(1000)}
>>> import sys
>>> from itertools import islice
>>> class IterSlicer:
... def __getitem__(self, what):
... it, sl = what
... return islice(it, sl.start or 0, sl.stop or sys.maxsize,
sl.step or 1)
...
>>> IS = IterSlicer()
>>> dict(IS[large.items(), 3:10:2])
{3: 9, 5: 25, 7: 49, 9: 81}
>>> from itertools import count
>>> set(IS[count(), 10:100:9])
{64, 37, 73, 10, 46, 82, 19, 55, 91, 28}

Potentially IterSlicer, or IS, could live in itertools (or more likely
more-itertools) just to provide a short way to use slice notation with an
arbitrary iterable... not limited to dict.items().


-- 
The dead increasingly dominate and strangle both the living and the
not-yet born.  Vampiric capital and undead corporate persons abuse
the lives and control the thoughts of homo faber. Ideas, once born,
become abortifacients against new conceptions.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OEO7XAM4FJM2742M4AWJVEEHBGIKZPJV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

2020-07-30 Thread Barry Scott 


> On 30 Jul 2020, at 11:55, Vinay Sharma via Python-ideas 
>  wrote:
> 
> 
> 
>> On 28-Jul-2020, at 5:19 AM, Robert Collins > > wrote:
>> 
>> On Mon, 27 Jul 2020 at 23:24, Vinay Sharma > > wrote:
>>> 
>>> Hi, Thanks for replying.
>>> 
 One thing that is worth thinking about is the safety of the API that
 is put together. A memory segment plus a separate detached semaphore
 or mutex can be used to build a safe API, but is not itself a safe
 API.
>>> 
>>> Agreed. That’s why I am more inclined to the second solution that I 
>>> mentioned.
>> 
>> The second approach isn't clearly specified yet: is 'sync' in the name
>> implying a mutex, an RW lock, or dependent on pointers to atomic types
>> (which then becomes a portability challenge in some cases). The C++
>> atomics documentation you linked to documents a similar but
>> differently named set of methods, so you'll need to clarify the
>> difference you intend.
> 
> Python has support for atomic types, I guess: 
> Atomic Int: 
> https://github.com/python/cpython/blob/master/Include/internal/pycore_atomic.h#L80
>  
> 
> Atomic Store: 
> https://github.com/python/cpython/blob/master/Include/internal/pycore_atomic.h#L94
>  
> 
> 
> And, these methods don’t use any locks, they are just atomic operations.
> So, my approach was to lock the whole shared memory segment at once, and to 
> do that we can store an integer at the beginning of every shared memory 
> segment, which will denote whether this segment is locked (1), or unlocked 
> (0), and atomic operations can be used to update this integer ( 0 -> 1) lock, 
> (1 -> 0) unlock. Although, `wait` function will have to be implemented like 
> in semaphores, which will wait until the segment is free (becomes 0).

Surely you need locks and semaphores that work between processes? Both unix and 
Windows have these primitives.

The atomics are great for lockless changing of single ints, but anything more 
complex needs locks and semaphores.

Surely you do not want to be implementing your own locks with the OS support 
that works well with OS scheduling?

Barry



> 
>> > > For instance, we could have an object
>> representing a memory range that
 doesn't offer read/write at all, but allows:
 - either one process write access over the range
 - or any number of readers read access over the range
 - allows subdividing the range (so that you can e.g. end one write
 lock and keep another)
>>> 
>>> Where will this memory object be stored ?
>> 
>> There are a few options. The most obvious one given that bookkeeping
>> data is required, is to build a separate layer offering this
>> functionality, which uses the now batteries-included SHM facilities as
>> part of its implementation, but doesn't directly surface it.
>> 
> Can you please elaborate more on this ? 
> I understand that shared memory will be used to store ranges and whether they 
> are being locked/unlocked, etc. But if multiple process can update this data, 
> then we will also have to think about the synchronisation of this 
> book-keeping data.
> 
> So, I guess you mean to say that all processes will be allotted shared memory 
> using a separate API/layer, which will take care of book-keeping, and since 
> this separate API/layer will be only responsible for book-keeping, there will 
> be no need to synchronise book-keeping data.
> 
> But, then the question arises how will unrelated processes communicate with 
> this layer/API to request shared memory.
> 
> One way could be that a separate process managing this book-keeping could be 
> created, and other process will request access/lock/unlock using this 
> separate process.
> 
> And the communication between between this layer (separate process) and the 
> other processes (using shared memory) will be using some form of IPC.
> 
>>> Locking a particular range instead of the whole memory segment will be 
>>> relatively efficient because processes using different ranges can write 
>>> simultaneously.
>>> 
>>> Since, this object will also be shared across multiple processes, there 
>>> must be a safe way to update it.
>> 
>> There's a lot of prior art on named locks of various sorts, I'd
>> personally be inclined to give the things a name that can be used
>> across different processes in some form and bootstrap from there.
>> 
>>> Any thoughts on that ?
>>> 
 On 27-Jul-2020, at 3:50 PM, Robert Collins >>> > wrote:
 
 On Sun, 26 Jul 2020 at 19:11, Vinay Sharma via Python-ideas
 mailto:python-ideas@python.org>> wrote:
> 
> Problem:
> Currently, let’s say I create a shared_memory segment using 
> mulitprocessing.shared_memory.SharedMemory in Process 1 and open the same 
> in Process

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Christopher Barker 
On Wed, Jul 29, 2020 at 7:23 AM Wes Turner  wrote:

> .iloc[] is the Pandas function for accessing by integer-location:
>
> https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html
>

I'm not sure I quite get the analogy here, but ...

>>> odict.iloc[3]
>
>
> Would this be the only way to access only item 4 from an odict of length
> greater than 4 with slice notation for dict views generated from selection
> by integer-location?
>

are suggesting that we add another attribute to dicts to provide "index"
access? I'm not sure the advantage of that, as we already have the dict
views, so I guess it's nice not to have the type the parentheses:

odict.items()[3] IS a bit klunkier than odict.iloc[3]

(Side note: why ARE the views provided by methods, rather than properties?
But I digress ...)


> >>> odict[3:4]
>

that would be possible.

What does this do? Confusing to a beginner:
>
> >>> odict[3,]
>

no more confusing than it is for Sequences, is it?

In [4]: l[3,]

---
TypeError Traceback (most recent call last)
 in 
> 1 l[3,]

TypeError: list indices must be integers or slices, not tuple

But yes, if we allowed dicts to be indexable with slices, they still
couldn't be indexed by single indexes (unless that value happened to be a
key) so there would be no way to get a single item by index, as a length-1
slice would presumably return a dict with one item.

So yeah, if indexing, slicing were to happen, it would have to be in the
views.

But this does bring up that there are a lot of places where slice notation
could be used outside of [] -- and that would be handy for use cases like
pandas, even if not for dicts.

-CHB

-- 
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZSVYRCBPH2OQWKWHD4363C4EVDOXS2JD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Access (ordered) dict by index; insert slice

2020-07-30 Thread Christopher Barker 
On Tue, Jul 28, 2020 at 10:26 PM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Christopher Barker writes:
>  > from itertools import islice
>  >
>  > smaller_dict = dict(islice(large_dict.items(), 0, 255))
>  >
>  > which works, and isn't doing any unnecessary copying, but it's pretty
>  > darn ugly, as far as I'm concerned.
>
> In your application, I think that's just pretty, myself.


well, beauty is in the eye of the beholder, of course. But really, you
think it's pretty to import itertools, then make a function call, for
what's very much a slicing operation?

Python has been shifting away from a sequence-focused to an iteration
focused approach for a while -- and while I do see the benefits, I also see
the drawbacks. And it seems to be coupled with what might be called a shift
to a functional style

This reminds me a bit of a thread on this list a couple years(?) back,
about a built in "group by" operation. It petered out, but one point of
disagreement was whether folks wanted an API that was more "functional",
e.g. map-like, or comprehension based. Comprehensions are functional, and
at least generator comprehensions (expressions) are all about iterables,
but there is a different style as to whether you are more using expressions
or passing functions around. I personally far prefer the comprehension
style, but was surprised by how many folks prefered the "functional" style.

This is not exactly the same, but it "feels" the similar to me -- I really
prefer things like expressions and slice notation to a pile of nested
function calls.

Think about it -- if Python had started out being built around iterables,
and itertools had been there from the beginning, would Sequences simply be
iterables that happened to have a length? And would people be expressing
that:

itertools.islice(a_list, 0, 100, 2)

was a pretty way to get a portion of a list -- why would we want "slicing"
notation at all?

That's being dramatic, but to bring this back around, the original title of
this thread is:"Access (ordered) dict by ..."

The point being that dicts are now ordered, so maybe it makes sense to add
some operations that make it natural and easy to work with them in an
ordered way -- you know, like slicing :-)

I don't expect this to be accepted, folks didn't seem to like the idea
much, and it's not a very big deal, as we've seen this is only the second
use case anyone has come up with (the other being selecting a random item
out of a dict).

But it did strike me when I had the use-case.

And I'm well aware of islice, and yet it still took me a LOT longer to
write that code than it would have to do a slice :-)

that's missing is slice notation.  But that's probably not hard to
> implement in terms of the islice function, just completely redundant.
>

I don't follow here -- the slice notation is very important here -- I'm not
sure what you mean by "implement in terms of the islice function", but that
does bring up another point:

islice does not provide a way to do the equivalent of negative indexes --
because it's designed to work with iterables that don't have a length. But
if you ARE using it on a Sequence -- that's a missing feature.

-CHB



-- 
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KAFHZT3A2UJVYPKJ5Y6Y7HAYFJZVMEO4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

2020-07-30 Thread Vinay Sharma via Python-ideas 


> On 28-Jul-2020, at 5:19 AM, Robert Collins  wrote:
> 
> On Mon, 27 Jul 2020 at 23:24, Vinay Sharma  wrote:
>> 
>> Hi, Thanks for replying.
>> 
>>> One thing that is worth thinking about is the safety of the API that
>>> is put together. A memory segment plus a separate detached semaphore
>>> or mutex can be used to build a safe API, but is not itself a safe
>>> API.
>> 
>> Agreed. That’s why I am more inclined to the second solution that I 
>> mentioned.
> 
> The second approach isn't clearly specified yet: is 'sync' in the name
> implying a mutex, an RW lock, or dependent on pointers to atomic types
> (which then becomes a portability challenge in some cases). The C++
> atomics documentation you linked to documents a similar but
> differently named set of methods, so you'll need to clarify the
> difference you intend.

Python has support for atomic types, I guess: 
Atomic Int: 
https://github.com/python/cpython/blob/master/Include/internal/pycore_atomic.h#L80
Atomic Store: 
https://github.com/python/cpython/blob/master/Include/internal/pycore_atomic.h#L94
 


And, these methods don’t use any locks, they are just atomic operations.
So, my approach was to lock the whole shared memory segment at once, and to do 
that we can store an integer at the beginning of every shared memory segment, 
which will denote whether this segment is locked (1), or unlocked (0), and 
atomic operations can be used to update this integer ( 0 -> 1) lock, (1 -> 0) 
unlock. Although, `wait` function will have to be implemented like in 
semaphores, which will wait until the segment is free (becomes 0).

> > > For instance, we could have an object
> representing a memory range that
>>> doesn't offer read/write at all, but allows:
>>> - either one process write access over the range
>>> - or any number of readers read access over the range
>>> - allows subdividing the range (so that you can e.g. end one write
>>> lock and keep another)
>> 
>> Where will this memory object be stored ?
> 
> There are a few options. The most obvious one given that bookkeeping
> data is required, is to build a separate layer offering this
> functionality, which uses the now batteries-included SHM facilities as
> part of its implementation, but doesn't directly surface it.
> 
Can you please elaborate more on this ? 
I understand that shared memory will be used to store ranges and whether they 
are being locked/unlocked, etc. But if multiple process can update this data, 
then we will also have to think about the synchronisation of this book-keeping 
data.

So, I guess you mean to say that all processes will be allotted shared memory 
using a separate API/layer, which will take care of book-keeping, and since 
this separate API/layer will be only responsible for book-keeping, there will 
be no need to synchronise book-keeping data.

But, then the question arises how will unrelated processes communicate with 
this layer/API to request shared memory.

One way could be that a separate process managing this book-keeping could be 
created, and other process will request access/lock/unlock using this separate 
process.

And the communication between between this layer (separate process) and the 
other processes (using shared memory) will be using some form of IPC.

>> Locking a particular range instead of the whole memory segment will be 
>> relatively efficient because processes using different ranges can write 
>> simultaneously.
>> 
>> Since, this object will also be shared across multiple processes, there must 
>> be a safe way to update it.
> 
> There's a lot of prior art on named locks of various sorts, I'd
> personally be inclined to give the things a name that can be used
> across different processes in some form and bootstrap from there.
> 
>> Any thoughts on that ?
>> 
>>> On 27-Jul-2020, at 3:50 PM, Robert Collins  
>>> wrote:
>>> 
>>> On Sun, 26 Jul 2020 at 19:11, Vinay Sharma via Python-ideas
>>>  wrote:
 
 Problem:
 Currently, let’s say I create a shared_memory segment using 
 mulitprocessing.shared_memory.SharedMemory in Process 1 and open the same 
 in Process 2.
 Then, I try to write some data to the shared memory segment using both the 
 processes, so for me to prevent any race condition (data corruption), 
 either these operations must be atomic, or I should be able to lock / 
 unlock shared memory segment, which I cannot at the moment.
 
 I earlier posted a solution to this problem, which received positive 
 response, but there weren’t many responses to it, despite the fact this 
 problem makes shared_memory practically unusable if there are simultaneous 
 writes.
 So, the purpose of this post is to have discussion about the solution of 
 the same.
>>> 
>>> One thing that is worth thinking about is the safety of the API that
>>> is put together. A memory segment plus a separate detached