Re: importlib changes from py3.7 to py3.8

[Eko palypse]

Hello Greg,
thank you for your answer. I have checked it twice and the file is really
empty.
I even opened it with a hex editor - no content.
BUT in order to create a bug ticket I tried to make a mcve and I didn't
succeed.
I copied the function (load_plugins) 1 to 1, but outside my program it
works as it should.
So it seems that something inside my application is causing this problem.
Hmm ... Fortunately this is a pure python application so debugging should
lead to enlightenment. :-)

Thx
Eren

Am So., 20. Sept. 2020 um 02:06 Uhr schrieb Greg Ewing <
greg.ew...@canterbury.ac.nz>:

> On 19/09/20 7:51 am, Eko palypse wrote:
> > ValueError: source code string cannot contain null bytes
> >
> > Any idea what happened here?
>
> > Seems I've missed that __init__.py's aren't allowed to be empty anymore.
> > Adding a single # solved the issue.
>
> I just tried importing a package with an empty __init__.py in
> Python 3.8 and there was no problem:
>
> % ls -l empty
> total 0
> -rw-r--r--  1 greg  staff  0 20 Sep 12:00 __init__.py
> % python3
> Python 3.8.2 (default, Mar 23 2020, 11:36:18)
> [Clang 8.1.0 (clang-802.0.42)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import empty
>  >>>
>
> It looks more like the file in question wasn't completely empty,
> but had a null character in it for some reason. I expect that if
> you re-saved it as truly empty it would work. If not, you should
> probably report it as a bug.
>
> --
> Greg
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: importlib changes from py3.7 to py3.8

[Eko palypse]

Seems I've missed that __init__.py's aren't allowed to be empty anymore.
Adding a single # solved the issue.

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

importlib changes from py3.7 to py3.8

[Eko palypse]

With python3.7 I'm able to use

_module = importlib.import_module('app.plugins.plugin_dir.plugin_name')

succesfully but with python 3.8 this results in 

  File "D:\ProgramData\Python\Python38_64\lib\importlib\__init__.py", line 127, 
in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 961, in _find_and_load_unlocked
  File "", line 219, in _call_with_frames_removed
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 961, in _find_and_load_unlocked
  File "", line 219, in _call_with_frames_removed
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 975, in _find_and_load_unlocked
  File "", line 671, in _load_unlocked
  File "", line 779, in exec_module
  File "", line 916, in get_code
  File "", line 846, in source_to_code
  File "", line 219, in _call_with_frames_removed
ValueError: source code string cannot contain null bytes

Any idea what happened here?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Embedded python: How to debug code in an isolated way

[Eko palypse]

Thank you very much for your interest in my little problem.

> When the app calls into python does the event loop of the gui block?

Yes, the cpp app calls a callback function from the embedded python
interpreter synchronously.


> When you print does that trigger the event loop to run in a nested call
back into the cpp?

I can't say for sure but I don't think so. At least my script or the
embedded interpreter doesn't do that.


>  Maybe you need to run python in its own thread and Marshall in and out
of the gui main thread with the event loop?

I've checked by using GetCurrentThreadId
<https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-getcurrentthreadid>
that the registered callback function and debugger run in the same thread
but it is different to the UI thread.
Do you think I have to run the debugger in its own thread? I'll give it a
try anyways. :-)

Thx
Eren

Am So., 23. Aug. 2020 um 22:55 Uhr schrieb Barry :

>
>
> > On 22 Aug 2020, at 20:53, Eko palypse  wrote:
> >
> > Hello,
> >
> > background info first. On windows, python3.8.5
> >
> > A cpp app has an embedded python interpreter which allows to
> modify/enhance the cpp app
> > by providing objects to manipulate the cpp app and callbacks to act on
> certain events,
> > like fileopen, fileclose, updateui ... which are send by the cpp app.
> > The embedded python interpreter acts more or less like the standard
> python REPL.
> > If you run a script with code like
> >
> > def test(i):
> >print(i*2)
> >
> > test(4)
> >
> > then the test function is now globally available as long as the cpp app
> is alive.
> >
> > I've written a debugger based on bdb and encountered a situation which I
> haven't found out how to handle correctly yet.
> >
> > Let's assume I've registered a callback when something changes in the UI.
> > In that case an object function gets called which highlights certain
> aspects in the UI.
> >
> > Now if I run a script via this debugger it works as long as I don't use
> the same object function.
> > If I do use it, then the stack gets corrupted(?).
>
> When the app calls into python does the event loop of the gui block?
> When you print does that trigger the event loop to run in a nested call
> back into the cpp?
>
> Maybe you need to run python in its own thread and Marshall in and out of
> the gui main thread with the event loop?
>
> Barry
>
> > For example it look like this
> >
> > stack:
> > ,
> 580
> > ', line 1, code >, 1
> >  code >, 8
> >
> > now if I enter the function which call this object function this gets add
> >
> >  add_match>, 5
> >
> > I step through the code and suddenly the stack looks like this
> >
> > ,
> 580
> > ', line 1, code >, 1
> >  code >, 8
> >  code >, 154
> >  code paint_it>, 102)]
> >
> > EnhanceAnyLexer is neither part of the test_pydebugger script nor part
> of the debugger.
> > It is used to register a callback on the updateui event.
> >
> > So the question is, what do I need to read/learn/understand in order to
> solve this issue?
> > Or in other words, how can I debug my script in an isolated environment.
> > I can't manipulate the __main__.dict as this seems to have general
> impact on registered callbacks as well.
> > Any ideas?
> >
> > Thank you
> > Eren
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Embedded python: How to debug code in an isolated way

[Eko palypse]

Thx for your tip/suggestion.

> If In Doubt, Print It Out!

That's the current situation and that's usually enough, but then there's
this situation
where it gets annoying because you realize that the print wouldn't make
more sense at this point
but at that point and that's where a debugger is just nicer, I think.

Eren

Am Sa., 22. Aug. 2020 um 22:59 Uhr schrieb Chris Angelico :

> On Sun, Aug 23, 2020 at 5:51 AM Eko palypse  wrote:
> > So the question is, what do I need to read/learn/understand in order to
> solve this issue?
> > Or in other words, how can I debug my script in an isolated environment.
>
> I'd go for the old standby - IIDPIO: If In Doubt, Print It Out!
> Instead of trying to use a debug harness, just run your code normally,
> and print out whatever you think might be of interest. If you don't
> have a console, well, that would be the first thing to do - you
> *always* need a console.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Embedded python: How to debug code in an isolated way

[Eko palypse]

Hello,

background info first. On windows, python3.8.5

A cpp app has an embedded python interpreter which allows to modify/enhance the 
cpp app
by providing objects to manipulate the cpp app and callbacks to act on certain 
events, 
like fileopen, fileclose, updateui ... which are send by the cpp app.
The embedded python interpreter acts more or less like the standard python REPL.
If you run a script with code like

def test(i):
print(i*2)

test(4)

then the test function is now globally available as long as the cpp app is 
alive.

I've written a debugger based on bdb and encountered a situation which I 
haven't found out how to handle correctly yet.

Let's assume I've registered a callback when something changes in the UI.
In that case an object function gets called which highlights certain aspects in 
the UI.

Now if I run a script via this debugger it works as long as I don't use the 
same object function.
If I do use it, then the stack gets corrupted(?).

For example it look like this

stack:
, 580
', line 1, code >, 1
>, 8

now if I enter the function which call this object function this gets add

, 5

I step through the code and suddenly the stack looks like this

, 580
', line 1, code >, 1
>, 8
>, 154
, 102)]

EnhanceAnyLexer is neither part of the test_pydebugger script nor part of the 
debugger.
It is used to register a callback on the updateui event.

So the question is, what do I need to read/learn/understand in order to solve 
this issue?
Or in other words, how can I debug my script in an isolated environment.
I can't manipulate the __main__.dict as this seems to have general impact on 
registered callbacks as well.
Any ideas?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Windows and Subclassing - 'OverflowError': int too long to convert

[Eko palypse]

Hello Eryk,
thank you for your interest in my problem and you've nailed it, the problem
was solved by putting all Windows API definitions in a separate module and
making sure that I only use WinDLL.
I would never have thought of that, because I only used WinDLL in this
module.
But a PostMessage in another module, which I created with windll, seems to
have confused things a bit and now that I have merged this and other
modules into one, I don't see the problem anymore.
Here is the desired excerpt. (user32 is now import from win32api, where all
other definitions are now also)

def new_scintilla2_proc(self, hWnd, msg, wParam, lParam):
if self.__WndProc(hWnd, msg, wParam, lParam):
return 0

print(f'{hWnd=}, {msg=}, {wParam=}, {lParam=}')
return user32.CallWindowProcW(self.prev_scintilla2_wnd_proc, hWnd,
msg, wParam, lParam)


def __WndProc(self, hWnd, msg, wParam, lParam):
event_handler = self._event_handler.get(msg, None)
if event_handler:
return event_handler(msg, wParam, lParam)
return False

Once again, thank you
Eren

Am Mi., 29. Juli 2020 um 16:26 Uhr schrieb Eryk Sun :

> On 7/28/20, Eko palypse  wrote:
> >
> > Now when I get this error the message I receive in this situation is
> always
> > like this.
> >
> > hWnd=197364, msg=20, wParam=*18446744072652653190*, lParam=0
> >
> > Traceback (most recent call last):
> >   File "_ctypes/callbacks.c", line 237, in 'calling callback function'
> >   File "", line 121, in new_scintilla2_proc
> > ctypes.ArgumentError: argument 4: : int too long
> to
> > convert
>
> Can you provide a more complete example of how you're calling
> CallWindowProcW in new_scintilla2_proc? It seems like a function
> pointer is being called that doesn't have argtypes defined. For
> example:
>
> >>> user32 = ctypes.WinDLL('user32', use_last_error=True)
> >>> user32.CallWindowProcW(None, None, 0x14, 18446744072652653190, 0)
> Traceback (most recent call last):
>   File "", line 1, in 
> ctypes.ArgumentError: argument 4: : int too
> long to convert
>
> Perhaps in new_scintilla2_proc you're using a different instance of
> ctypes.WinDLL, or you switched to using ctypes.windll.user32. I
> recommend avoiding the global ctypes.LibraryLoader instances --
> ctypes.cdll, ctypes.windll, and ctypes.oledll -- because using them
> leads to conflicts with other libraries, but you do need to use a
> single instance of ctypes.WinDLL for your user32 function prototypes,
> imported from a common module.
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Windows and Subclassing - 'OverflowError': int too long to convert

[Eko palypse]

Hello,
I am subclassing a scintilla window and it happens every now and then
that I get an OverflowError. I've logged the message and could narrow down
that it
is a single windows message which causes this trouble. It's WM_ERASEBKGND.

Now the confusing part for me is the following.
According to MS WPARAM
 is
an UINT_PTR which is defined as

#if defined(_WIN64) typedef unsigned __int64 UINT_PTR; #else typedef
unsigned int UINT_PTR; #endif

wintypes.WPARAM is ctypes.c_ulonglong and is of size 8 on my system.
I'm on a x64 Windows - so far as good, right?

Now when I get this error the message I receive in this situation is always
like this.

hWnd=197364, msg=20, wParam=*18446744072652653190*, lParam=0

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 237, in 'calling callback function'
  File "", line 121, in new_scintilla2_proc
ctypes.ArgumentError: argument 4: : int too long to
convert

I really doubt that 18446744072652653190 is a valid device context but who
knows.
The question is, why do I even get this OverflowError?

If I do wintypes.WPARAM(18446744072652653190) I don't receive an exception
and it is a valid 64bit value, isn't it?

The code I'm using is this

user32 = ctypes.WinDLL('user32', use_last_error=True)

LRESULT = wintypes.LPARAM
WND_PROC = ctypes.WINFUNCTYPE(LRESULT,
  wintypes.HWND,
  wintypes.UINT,
  wintypes.WPARAM,
  wintypes.LPARAM)

user32.CallWindowProcW.restype = LRESULT
user32.CallWindowProcW.argtypes = [WND_PROC,
   wintypes.HWND,
   wintypes.UINT,
   wintypes.WPARAM,
   wintypes.LPARAM]


Any idea what might have caused this?

Thank you for reading
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Threading module and embedded python

[Eko palypse]

Thank you for your help.
I made a minimum example which just creates the dialog window.
This runs, using the standard python37.exe without a problem.
But this also runs within the embedded interpreter slightly different
then my previous code.
In my previous code I used the hwnd from the C++ app

  DialogBoxIndirectParam(None,
(ctypes.c_ubyte *
len(self._array)).from_buffer_copy(self._array),
self.parent_hwnd,
DIALOGPROC(self._dlgproc),
0)

now I'm using None.
This still does NOT show the dialog when it is called,
BUT the C++ app keeps responsive and if I close it, then the dialog
appears. (???)

> Maybe in the embedded version you do not have an HWND that is usable, can
NULL be used?

The HWND I normally use should be correct as I have to use it in various
other calls
with SendMessageW and those methods works.

>  Is there a SHOW arg that you need to pass?

No.


Here the minimal code, just in case one is interested.

import ctypes
from ctypes import wintypes
import platform
from threading import Thread

user32 = ctypes.WinDLL('user32')

LRESULT = wintypes.LPARAM
DIALOGPROC = ctypes.WINFUNCTYPE(LRESULT,
wintypes.HWND,
wintypes.UINT,
wintypes.WPARAM,
wintypes.LPARAM)

INT_PTR = wintypes.INT if platform.architecture()[0] == '32bit' else
wintypes.LARGE_INTEGER

DialogBoxIndirectParam = user32.DialogBoxIndirectParamW
DialogBoxIndirectParam.restype = wintypes.HWND
DialogBoxIndirectParam.argtypes = [wintypes.HINSTANCE,
   ctypes.POINTER(ctypes.c_ubyte),
   wintypes.HWND,
   DIALOGPROC,
   wintypes.LPARAM]

GetDlgItem = user32.GetDlgItem
GetDlgItem.restype = wintypes.HWND
GetDlgItem.argtypes = [wintypes.HWND, wintypes.INT]

EndDialog = user32.EndDialog
EndDialog.restype = wintypes.BOOL
EndDialog.argtypes = [wintypes.HWND, INT_PTR]


def align_struct(tmp):
''' align control structure to dword size '''
dword_size = ctypes.sizeof(wintypes.DWORD)
align = dword_size - (len(tmp) % dword_size)
if align < dword_size:
tmp += bytearray(align)
return tmp


class DlgWindow(Thread):
''' Implements a threaddialog template window '''

def __init__(self):
super().__init__()

self._array = bytearray()  # DLGTEMPLATEEX structure buffer
self._array += wintypes.WORD(1)  # dlgVer
self._array += wintypes.WORD(0x)  # signature
self._array += wintypes.DWORD(0)  # helpID
self._array += wintypes.DWORD(0)  # exStyle
#  WS_POPUP | WS_BORDER | WS_SYSMENU |
WS_CAPTION | DS_MODALFRAME | DS_SETFONT | DS_CENTER
self._array += wintypes.DWORD(0x8000 | 0x0080 | 0x0008
| 0x00C0 | 0x80 | 0x40 | 0x800)  # style
self._array += wintypes.WORD(0)  # cDlgItems
self._array += wintypes.SHORT(0)  # x
self._array += wintypes.SHORT(0)  # y
self._array += wintypes.SHORT(200)  # cx
self._array += wintypes.SHORT(200)  # cy
self._array += wintypes.WORD(0)  # menu
self._array += wintypes.WORD(0)  # windowClass
self._array += ctypes.create_unicode_buffer('Test Dialog')  # title
self._array += wintypes.WORD(9)  # pointsize
self._array += wintypes.WORD(400)  # weight
self._array += wintypes.BYTE(0)  # italic
self._array += wintypes.BYTE(0)  # charset
self._array += ctypes.create_unicode_buffer('MS Shell Dlg')  #
typeface


def _dlgproc(self, hwnd, msg, wparam, lparam):
print(hwnd, msg, wparam, lparam)
if msg == 16:  # WM_CLOSE
EndDialog(hwnd, 0)
return 1

elif msg == 272:  # WM_INITDIALOG
return 1

return 0


def run(self):
''' create the dialog window '''
self._array = align_struct(self._array)
DialogBoxIndirectParam(None,
   (ctypes.c_ubyte *
len(self._array)).from_buffer_copy(self._array),
   None,
   DIALOGPROC(self._dlgproc),
   0)


def test_window():
dlg = DlgWindow()
dlg.start()


if __name__ == '__main__':
test_window()

Thanks
Eren

Am Do., 16. Apr. 2020 um 18:33 Uhr schrieb Barry Scott <
ba...@barrys-emacs.org>:

>
>
> > On 16 Apr 2020, at 14:55, Eko palypse  wrote:
> >
> > Barry, sorry for sending you a private message yesterday, was not
> intended.
> >
> > No, I only have rudimentary knowledge of C++,
> &g

Re: Threading module and embedded python

[Eko palypse]

Barry, sorry for sending you a private message yesterday, was not intended.

No, I only have rudimentary knowledge of C++,
but it has been developing since I started using Cython.
I haven't done any stack analysis yet but hey, there's always a first time.
I think my stubbornness could be of help here :-)

Visual Studio reports that the last location of the thread is in _ctypes.pyd
and the call stack window shows that the last execution is
user32.dll!InternalDailogBox().

Call Stack








*user32.dll!InternalDialogBox()user32.dll!DialogBoxIndirectParamAorW()user32.dll!DialogBoxIndirectParamW()_ctypes.pyd!07fee7fc17e3()_ctypes.pyd!07fee7fbfee3()_ctypes.pyd!07fee7fbb4c5()_ctypes.pyd!07fee7fbc019()_ctypes.pyd!07fee7fb6dfa()python37.dll!_PyObject_FastCallKeywords(_object
* callable=0x02fa8c78, _object * const * stack=0x05261c78,
__int64 nargs=5, _object * kwnames=0x)*


The thread is neither suspended nor does it have any different status than
the main thread
which loops through its main event queue at this point.

Thank you
Eren


Am Mi., 15. Apr. 2020 um 22:57 Uhr schrieb Barry :

>
>
> > On 15 Apr 2020, at 21:18, Eko palypse  wrote:
> >
> > Thank you for your suggestion. I will give it a try.
> >
> >> What is the "stuck" thread doing? waiting for a lock?
> >
> > No, it should open a dialog created with DialogBoxIndirectParamW
> > <
> https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-dialogboxindirectparamw
> >
>
> I assume you are a C++ developer and can look at the stack of the thread.
> What is the thread doing? Is it in python code? Is it in windows code?
>
> Barry
>
> > .
> >
> > Eren
> >
> >> Am Mi., 15. Apr. 2020 um 20:12 Uhr schrieb Barry Scott <
> >> ba...@barrys-emacs.org>:
> >>
> >>
> >>
> >>>> On 15 Apr 2020, at 13:30, Eko palypse  wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> the following happens on Windows7 x64 and Python37 x64
> >>>
> >>> I have a plugin DLL for a C++ application in which Python37 is
> embedded.
> >>> The plugin itself works, except when I want to use the threading
> module.
> >>>
> >>> If I start a Python script in my plugin which uses the threading module
> >>> I can verify via ProcessExplorer that the thread is started,
> >>> but it doesn't do anything (??) and the c++ application doesn't really
> >> do anything anymore either.
> >>>
> >>> Only when I stop the C++ Applikation, the thread becomes active for a
> >> short time.
> >>> Verified with logging module over time print-outs.
> >>>
> >>> Apparently I did not understand everything about threads and embedded
> >> python.
> >>>
> >>> Any idea what I'm doing wrong?
> >>
> >> This is what I typically do.
> >>
> >> Make sure that you have installed the Python debug files.
> >> Now you can use the visual C++ debugger to attach to the process and
> >> look at what the threads are doing.
> >>
> >> I always have the python source code on hand to read as well.
> >>
> >> This should give you a clue.
> >>
> >> What is the "stuck" thread doing? waiting for a lock?
> >>
> >> Barry
> >>
> >>
> >>
> >>
> >>>
> >>>
> >>> The whole thing is initialized by the DllMain routine.
> >>>
> >>>
> >>> BOOL APIENTRY DllMain( HANDLE hModule,
> >>>  DWORD  reasonForCall,
> >>>  LPVOID /* lpReserved */ )
> >>> {
> >>>   switch ( reasonForCall )
> >>>   {
> >>>   case DLL_PROCESS_ATTACH:
> >>>   if (!Py_IsInitialized())
> >>>   {
> >>>   PyImport_AppendInittab("Npp", &PyInit_Npp);
> >>>   Py_InitializeEx(0);
> >>>   PyEval_InitThreads();  //<- this shouldn't be needed as I
> >> understand that it is called by Py_InitializeEx anyway
> >>>   }
> >>>   PyImport_ImportModule("Npp");
> >>>   break;
> >>>   case DLL_PROCESS_DETACH:
> >>>   Py_Finalize();
> >>>   break;
> >>>
> >>>   case DLL_THREAD_ATTACH:
> >>>   break;
> >>>
> >>>   

Re: Threading module and embedded python

[Eko palypse]

Thank you for your suggestion. I will give it a try.

> What is the "stuck" thread doing? waiting for a lock?

No, it should open a dialog created with DialogBoxIndirectParamW
<https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-dialogboxindirectparamw>
.

Eren

Am Mi., 15. Apr. 2020 um 20:12 Uhr schrieb Barry Scott <
ba...@barrys-emacs.org>:

>
>
> > On 15 Apr 2020, at 13:30, Eko palypse  wrote:
> >
> > Hi everyone,
> >
> > the following happens on Windows7 x64 and Python37 x64
> >
> > I have a plugin DLL for a C++ application in which Python37 is embedded.
> > The plugin itself works, except when I want to use the threading module.
> >
> > If I start a Python script in my plugin which uses the threading module
> > I can verify via ProcessExplorer that the thread is started,
> > but it doesn't do anything (??) and the c++ application doesn't really
> do anything anymore either.
> >
> > Only when I stop the C++ Applikation, the thread becomes active for a
> short time.
> > Verified with logging module over time print-outs.
> >
> > Apparently I did not understand everything about threads and embedded
> python.
> >
> > Any idea what I'm doing wrong?
>
> This is what I typically do.
>
> Make sure that you have installed the Python debug files.
> Now you can use the visual C++ debugger to attach to the process and
> look at what the threads are doing.
>
> I always have the python source code on hand to read as well.
>
> This should give you a clue.
>
> What is the "stuck" thread doing? waiting for a lock?
>
> Barry
>
>
>
>
> >
> >
> > The whole thing is initialized by the DllMain routine.
> >
> >
> > BOOL APIENTRY DllMain( HANDLE hModule,
> >   DWORD  reasonForCall,
> >   LPVOID /* lpReserved */ )
> > {
> >switch ( reasonForCall )
> >{
> >case DLL_PROCESS_ATTACH:
> >if (!Py_IsInitialized())
> >{
> >PyImport_AppendInittab("Npp", &PyInit_Npp);
> >Py_InitializeEx(0);
> >PyEval_InitThreads();  //<- this shouldn't be needed as I
> understand that it is called by Py_InitializeEx anyway
> >}
> >PyImport_ImportModule("Npp");
> >break;
> >case DLL_PROCESS_DETACH:
> >Py_Finalize();
> >break;
> >
> >case DLL_THREAD_ATTACH:
> >break;
> >
> >case DLL_THREAD_DETACH:
> >break;
> >}
> >
> >return TRUE;
> > }
> >
> > and the code in the plugin which executes the python scripts is this
> >
> > cdef void run_code():
> >try:
> >global_dict = globals()
> >if '__name__' not in global_dict or global_dict['__name__'] !=
> '__main__':
> >global_dict.update({"__name__": "__main__",})
> >exec(compile(editor.getText(), '', 'exec'), global_dict)
> >
> >except Exception:
> >MessageBoxW(nppData._nppHandle,
> >traceback.format_exc(),
> >'RUN CODE EXCEPTION',
> >0)
> >
> > I don't know if this is important, but the DLL is generated by Cython.
> >
> > Thank you for reading and stay healthy
> >
> > Eren
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Threading module and embedded python

[Eko palypse]

Hi everyone,

the following happens on Windows7 x64 and Python37 x64

I have a plugin DLL for a C++ application in which Python37 is embedded.
The plugin itself works, except when I want to use the threading module.

If I start a Python script in my plugin which uses the threading module
I can verify via ProcessExplorer that the thread is started,
but it doesn't do anything (??) and the c++ application doesn't really do 
anything anymore either.

Only when I stop the C++ Applikation, the thread becomes active for a short 
time.
Verified with logging module over time print-outs.

Apparently I did not understand everything about threads and embedded python.

Any idea what I'm doing wrong?


The whole thing is initialized by the DllMain routine.


BOOL APIENTRY DllMain( HANDLE hModule,
   DWORD  reasonForCall,
   LPVOID /* lpReserved */ )
{
switch ( reasonForCall )
{
case DLL_PROCESS_ATTACH:
if (!Py_IsInitialized())
{
PyImport_AppendInittab("Npp", &PyInit_Npp);
Py_InitializeEx(0);
PyEval_InitThreads();  //<- this shouldn't be needed as I 
understand that it is called by Py_InitializeEx anyway
}
PyImport_ImportModule("Npp");
break;
case DLL_PROCESS_DETACH:
Py_Finalize();
break;

case DLL_THREAD_ATTACH:
break;

case DLL_THREAD_DETACH:
break;
}

return TRUE;
}

and the code in the plugin which executes the python scripts is this

cdef void run_code():
try:
global_dict = globals()
if '__name__' not in global_dict or global_dict['__name__'] != 
'__main__':
global_dict.update({"__name__": "__main__",})
exec(compile(editor.getText(), '', 'exec'), global_dict)

except Exception:
MessageBoxW(nppData._nppHandle,
traceback.format_exc(),
'RUN CODE EXCEPTION',
0)  

I don't know if this is important, but the DLL is generated by Cython.

Thank you for reading and stay healthy

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Load python from different plugin dlls

[Eko palypse]

Thanks for the information.
My test looks like this right now.
I have two plugins which, when loaded separately, work.
But when both are loaded, I get AccessVioletion messages from python37.dll.
The init code for both dlls is this:

cpp_init_code = f'''#include 
#include "PluginInterface.h"

PyMODINIT_FUNC PyInit_{PLUGIN_NAME}(void);

BOOL APIENTRY DllMain( HANDLE hModule,
   DWORD reasonForCall,
   LPVOID /* lpReserved */ )
{{
switch ( reasonForCall )
{{
case DLL_PROCESS_ATTACH:
PyImport_AppendInittab("{PLUGIN_NAME}", &PyInit_{PLUGIN_NAME});
Py_InitializeEx(Py_IsInitialized() ? 0 : 1);
PyImport_ImportModule("{PLUGIN_NAME}");
break;
case DLL_PROCESS_DETACH:
Py_Finalize();
break;

case DLL_THREAD_ATTACH:
break;

case DLL_THREAD_DETACH:
break;
}}

return TRUE;
}}'''

This is written to a plugininit.cpp file and then built via setup.py with
distutil, cython and VS2017.

According to the docs  PyImport_AppendInittab should be called before
Py_Initialize
but I can't call  Py_Initialize the second time as it would remove what was
initialized by the first dll.
And using PyImport_ImportModule on the second dll only does not work either.
So that is were I'm trapped.

Thank you
Eren

Am Do., 13. Feb. 2020 um 07:26 Uhr schrieb R.Wieser :

> Eko,
>
> > which needs also access to python3.dll but cannot load it itself as it
> has
> > been already loaded by plugin1
> >
> > Is such a scenario actually possible?
>
> Yes.
>
> Normally a DLL can be loaded in as many processes (and threads thereof) as
> you like.
>
> However, it is possible that the DLLs initialisation contains code to
> check
> if something it needs is available*, and if not directly exit.
>
> *like if it can access certain I/O - if the first DLL instance "takes
> posession" the second DLL instance wil fail.  Than again, this can also
> happen when using two totally different plugin DLLs.
>
> tl;dr:
> Yes, but not likely.
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Load python from different plugin dlls

[Eko palypse]

Assuming there is an CppApp which allows extending its functionality by adding 
plugins. Now let's assume there are plugin developer which use cython to create 
such a plugins.

How can one check if there has been already a plugin loaded which itself loaded 
a python interpreter?

And if this is possible, what needs to be done that a second plugin can be used 
with the python interpreter loaded from the first plugin?

To summarize:
CppApp loads pluign1 (which loads python3.dll) ok
CppApp loads pluign2 (which needs also access to python3.dll but cannot load it 
itself as it has been already loaded by plugin1)

Is such a scenario actually possible?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can i stop this Infinite While Loop - Python

[Eko palypse]

>From what I understand you want to give the user the possibility to try 10
times or enter -1 to end the script, right?
If so, you need to check if item is -1 or total tries are already 10 and in
such a case break the loop.
No need for an else branch.
Note, input returns an str object but you compare against an integer  ->
item != -1
This   total += int(item)   should be raised by one as you don't know what
item actually is, it might be a text

Eren

Am Mi., 30. Okt. 2019 um 13:01 Uhr schrieb ferzan saglam <
ferzan...@gmail.com>:

> I have tried many ways to stop the infinite loop but my program doesn't
> seem to stop after 10 items! It keeps going...!)
>
> ---
> total = 0
> while True:
>
>   print('Cost of item')
>   item = input()
>
>   if item != -1:
> total += int(item)
>   else:
> break
>
> print(total)
> ---
>
>
> The output I am aiming for is for the code to stop asking 'Cost of item'
> after 10 inputs or until -1 is typed, but the code repeats itself on an
> infinite loop. I cannot seem to stop the infinite loop.
>
> Thanks for any help in advance.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3, regular expression and bytes text

[Eko palypse]

Am Montag, 14. Oktober 2019 13:56:09 UTC+2 schrieb Chris Angelico:
> 
> (My apologies for saying this in reply to an unrelated post, but I
> also don't see those posts, so it's not easy to reply to them.)
> 
> ChrisA

Nothing to apologize and thank you for clarification,
I was already checking my settings to see if I could have set
some kind of filtering which I wasn't aware.

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3, regular expression and bytes text

[Eko palypse]

Am Sonntag, 13. Oktober 2019 21:20:26 UTC+2 schrieb moi:
> [Do not know why I spent hours with this...]
> 
> To answer you question. 
> Yes, I confirm.
> It seems that as soon as one works with bytes and when
> a char is encoded in more than 1 byte, "re" goes into
> troubles.
> 

First, sorry for answering so late but I double checked my mail inbox and there 
is no notification that you have answered, whereas I got a notification for 
each of the other posts. Not sure what happened.
Thx for confirmation.

> Something else.
> I do not like to use "\w". One can rapidly be confronted with
> annoying boundary limits. A while loop can be much more "fine
> grained" and it works with everything, bytes, encoded chars,
> code points...

As the whole idea is, maybe I should use "was" by now, 
to build a regex tester script, it seems to be logical to use
regular expressions, isn't it. :-)

Thx
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3, regular expression and bytes text

[Eko palypse]

First of all many thanks to everyone for the active participation.

@Chris Angelico
I think I understand what you illustrated with the byte example,
makes sense. As it was developed for 8-bit encoding only, 
it cannot be used for mulitbyte encoding.

@Richard Damon and @MRAB
thank you very much for the information too, very much appreciated.

I think I understand what you all mean but I am not sure
how to put this all together.

Maybe a little bit more information about what I wanted to do.

Using notepad++ and scintilla. 
Scintilla passes a readonly pointer with SCI_GETCHARACTERPOINTER
of the current buffer to me.
The problem is that the buffer can have all possible encodings.
cp1251, cp1252, utf8, ucs-2 ... but scintilla informs me about
which encoding is currently used.

I wanted to realize a regular expression tester with Python3,
and mark the text that has been matched by regular expressions.

After testing to treat everything as python3 str I found out that
the positions of the matched text are not correctly reported.
E.g say, if I want to find the word "Ärger", assumed encoded in utf8, with
the regex \w+. 
If I decode it, it would return the length of 5, whereas it is of length 6 
within the document, so marking the match would be wrong, wouldn't it?

I understand the reason of the difference.

If I use the builtin find dialog of notepad++, which uses internally the 
boost::regex engine, I can use \w+ to find the word.

So that's where I'm stuck at the moment. 
How can I find and mark those matches correctly.
Wrapping boost:regex with ICU support?

Thx
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3, regular expression and bytes text

[Eko palypse]

> You cannot. First, \w in re.LOCALE works only when the text is encoded 
> with the locale encoding (cp1252 in your case). Second, re.LOCALE 
> supports only 8-bit charsets. So even if you set the utf-8 locale, it 
> would not help.
> 
> Regular expressions with re.LOCALE are slow. It may be more efficient to 
> decode text and use Unicode regular expression.

Thank you, I guess I'm convinced to always decode everything (re pattern and 
text) to utf8 internally and then do the re search but then I would need to 
figure out the correct position, hmm - some ongoing investigation needed, I 
guess.

Thx
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3, regular expression and bytes text

[Eko palypse]

Thank you very much for your answer.

> You have to be able to match bytes, not strings.

May I ask you to elaborate on this, sorry non-native English speaker.
The buffer I receive is a byte-like buffer.

> I don't think you'll be able to 100% reliably match bytes in this way.
> You're asking it to make analysis of multiple bytes and to interpret
> them according to which character they would represent if decoded from
> UTF-8.
> 
> My recommendation: Even if your buffer is multiple gigabytes, just
> decode it anyway. Maybe you can decode your buffer in chunks, but
> otherwise, just bite the bullet and do the decode. You may be
> pleasantly surprised at how little you suffer as a result; Python is
> quite decent at memory management, and even if you DO get pushed into
> the swapper by this, it's still likely to be faster than trying to
> code around all the possible problems that come from mismatching your
> text search.
> 
> ChrisA

That's what I was afraid of. 
It would be nice if the "world" could commit itself to one standard, 
but I'm afraid that won't happen in my life anymore, I guess. :-(

Thx
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

python3, regular expression and bytes text

[Eko palypse]

What needs to be set in order to be able to use a re search within
utf8 encoded bytes?


My test, being on a windows PC with cp1252 setup, looks like this

import re
import locale


cp1252 = 'Ärger im Paradies'.encode('cp1252')
utf8 = 'Ärger im Paradies'.encode('utf-8')

print('cp1252:', cp1252)
print('utf8  :', utf8)
print('-'*80)
print("search for 'Ärger'.encode('cp1252') in cp1252 encoded text")
for m in re.finditer('Ärger'.encode('cp1252'), cp1252):
print(m)

print('-'*80)
print("search for 'Ärger'.encode('') in utf8 encoded text")
for m in re.finditer('Ärger'.encode(), utf8):
print(m)


print('-'*80)
print("search for '\\w+'.encode('cp1252') in cp1252 encoded text")
for m in re.finditer('\\w+'.encode('cp1252'), cp1252):
print(m)

print('-'*80)
print("search for '\\w+'.encode('') in utf8 encoded text")
for m in re.finditer('\\w+'.encode(), utf8):
print(m)

locale.setlocale(locale.LC_ALL, '')
print('-'*80)
print("search for '\\w+'.encode('cp1252') using re.LOCALE in cp1252 encoded 
text")
for m in re.finditer('\\w+'.encode('cp1252'), cp1252, re.LOCALE):
print(m)

print('-'*80)
print("search for '\\w+'.encode('') using ??? in utf8 encoded text")
for m in re.finditer('\\w+'.encode(), utf8):
print(m)



if you run this you will get something like



cp1252: b'\xc4rger im Paradies'
utf8  : b'\xc3\x84rger im Paradies'

search for 'Ärger'.encode('cp1252') in cp1252 encoded text


search for 'Ärger'.encode('') in utf8 encoded text




these two are ok BUT the result for \w+ shows a difference


search for '\w+'.encode('cp1252') in cp1252 encoded text




search for '\w+'.encode('') in utf8 encoded text






it doesn't find the Ä, which from documentation point of view is expected
and a hint to use locale is given, so let's do it and the results are


search for '\w+'.encode('cp1252') using re.LOCALE in cp1252 encoded text






works for cp1252 BUT does not work for utf8


search for '\w+'.encode('') using ??? in utf8 encoded text





So how can I make it work with utf8 encoded text?
Note, decoding it to a string isn't preferred as this would mean
allocating the bytes buffer a 2nd time and it might be that a 
buffer is several 100MBs, even GBs.

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Synchronous and Asynchronous callbacks

[Eko palypse]

Am Montag, 30. September 2019 11:46:43 UTC+2 schrieb Barry Scott:
> > On 29 Sep 2019, at 21:41, Eko palypse  wrote:
> > 
> > Am Sonntag, 29. September 2019 19:18:32 UTC+2 schrieb Barry Scott:
> >>> On 29 Sep 2019, at 14:14, Eko palypse  wrote:
> >>> 
> >>> Unfortunately, I can't make all callbacks synchronous or asynchronous 
> >>> because this has negative effects on the application in one way or 
> >>> another.
> >> 
> >> Surely you can make all callbacks async?
> >> You do not have to have them wait, they can complete their work in one 
> >> call.
> >> 
> >> sync == async-that-does-not-await
> >> 
> >> Barry
> > 
> > Thank you for your answer but I'm sorry I don't get it. 
> > What do you mean by I don't have to have them wait?
> > 
> > Let me explain my understanding, with some pseudo code, how async callbacks
> > works and please keep in mind that I'm a novice when it comes to OO designs
> > and basic software design strategies/techniques, so I might be totally
> > wrong about what I'm doing.
> 
> When you said async I thought you meant the specific programming model in 
> Python
> that uses "async def", "await" etc. You are using threads.
> 
> The problem you are facing is that the UI must not become unresponsive.
> Further a background thread cannot call any UI framework functions as the UI
> framework can only be called on the main thread.
> 
> You can run fast event handers on the UI thread but slow running handlers
> must be run on another thread to allow the UI to be responsive.
> 
> What I do is have the event handler for a long running task take the 
> responsibility
> to queue the task onto a background thread. While the task is running it 
> sends status
> to the UI thread so that UI elements are updated (progress bars, etc).
> 
> If you are using PyQt5 you might find the code I used for SCM Workbench 
> interesting.
> 
> I can write event handlers that look like this:
> 
> @thread_switcher
> def eventHandler( self, ... ):
>   # starts in the foreground (UI thread)
>   self.status.update('Working...')
> 
>   yield self.switchToBackground
>   # now on the background thread
>   self.doSlowTask()
> 
>   yield self.switchToForeground
>   # now in the foreground
>   self.status.update('Done')
> 
> The code is in:
> 
> https://github.com/barry-scott/scm-workbench/blob/master/Source/Common/wb_background_thread.py
> 
> @thread_switcher adds an attribute to the function so the rest of the
> code knows this is an event handler that needs to use a background thread.
> 
> When I add an event handler I call through a function (wrapWithThreadSwitcher)
> that wraps only functions that have been marked with @thread_switcher.
> 
> The rest of the code is the machinery to handle moving between threads via 
> the generator.
> 
> Barry
> 
> > 
> > An SYNCHRONOUS callback is like this. Event/notification gets fired and
> > notification handler calls all registered methods one after the other.
> > Something like
> > 
> > if notification.list_of_methods:
> >for method in list_of_methods:
> >method()
> > 
> > whereas an ASYNCHRONOUS callback is more like this. Event/notification gets
> > fired and notification handler informs another thread that the event has
> > been fired and all registered async callback methods should get executed.
> > So something like this
> > 
> > class EventThread(threading.Thread):
> >def __init__(...):
> >super()...
> >self.event = threading.Event()
> >self.kill = False
> >...
> >def run(self):
> >while True:
> >self.event.wait()
> >self.event.clear()
> >if not self.kill:
> >for method in self.list_of_methods:
> >method()
> > 
> > et = EventThread()
> > if notification.list_of_methods:
> >et.event.set()  # run all async methods
> >for method in list_of_methods:  # run sync methods
> >method())
> > 
> > 
> > So if there is no synchronous callback for that notification then the 
> > notification handler would just set the event and return. 
> > The EventThread would then call one registered callback after the other.
> > Right?
> > 
> > Using this approach does sound fine from UI point of view as there is
> > minimal impact and UI keeps responsive 

Re: Synchronous and Asynchronous callbacks

[Eko palypse]

Am Sonntag, 29. September 2019 19:18:32 UTC+2 schrieb Barry Scott:
> > On 29 Sep 2019, at 14:14, Eko palypse  wrote:
> > 
> > Unfortunately, I can't make all callbacks synchronous or asynchronous 
> > because this has negative effects on the application in one way or another.
> 
> Surely you can make all callbacks async?
> You do not have to have them wait, they can complete their work in one call.
> 
> sync == async-that-does-not-await
> 
> Barry

Thank you for your answer but I'm sorry I don't get it. 
What do you mean by I don't have to have them wait?

Let me explain my understanding, with some pseudo code, how async callbacks
works and please keep in mind that I'm a novice when it comes to OO designs
and basic software design strategies/techniques, so I might be totally
wrong about what I'm doing.

An SYNCHRONOUS callback is like this. Event/notification gets fired and
notification handler calls all registered methods one after the other.
Something like

if notification.list_of_methods:
for method in list_of_methods:
method()

whereas an ASYNCHRONOUS callback is more like this. Event/notification gets
fired and notification handler informs another thread that the event has
been fired and all registered async callback methods should get executed.
So something like this

class EventThread(threading.Thread):
def __init__(...):
super()...
self.event = threading.Event()
self.kill = False
...
def run(self):
while True:
self.event.wait()
self.event.clear()
if not self.kill:
for method in self.list_of_methods:
method()

et = EventThread()
if notification.list_of_methods:
et.event.set()  # run all async methods
for method in list_of_methods:  # run sync methods
method()


So if there is no synchronous callback for that notification then the 
notification handler would just set the event and return. 
The EventThread would then call one registered callback after the other.
Right?

Using this approach does sound fine from UI point of view as there is
minimal impact and UI keeps responsive but from code execution point of view
it sound much more complicated. What if one async callback is slow or buggy?
This could lead to unpredictable behavior where as in a synchronous execution 
you immediately know what is causing it. 
That was the reason I decided not to offer async callbacks until I 
discovered that other parts, like UI modifications aren't working properly.
And there are notification which must be called in a synchronous way, like
the modification event. If you want to intercept a modification safely then it 
can only be done within a synchronous callback. 

Does this makes sense to you? Or do I have wrong understanding or, again, an
wrong assumption how this works?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Synchronous and Asynchronous callbacks

[Eko palypse]

Am Sonntag, 29. September 2019 01:02:48 UTC+2 schrieb Paul Rubin:
> Eko palypse  writes:
> > Two dicts, one for sync and the other for async registrations?
> 
> Why not make all the callbacks asynchronous?  Clients that don't want to
> handle a callback immediately can keep it pending until they are ready,
> perhaps in a queue.
> 
> Whether this can work of course depends on how the application is put
> together, but the pattern of a bunch of asynchronous tasks communicating
> through fifos tends to work pretty well.  I'm used to doing it with
> threads, though those can be expensive if you use a lot of them.  I
> don't know whether async/await unavoidably complicates things.

Thank you very much for your answer.
Unfortunately, I can't make all callbacks synchronous or asynchronous because 
this has negative effects on the application in one way or another.

I also came across the asyncio methods yesterday, 
Whoa, what can I say, I haven't really understood yet.
My first impression is that either the whole library works asynchronously or 
not.
Still too many Bohemian villages (not sure if this is also common in English - 
it should say there are too many things I don't understand).

But as I said, thank you very much for the time and your thoughts you have 
given about my problem, very much appreciated.

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Synchronous and Asynchronous callbacks

[Eko palypse]

Hello, I'm looking for a solution to a problem which is described best as an 
observer pattern issue I guess.
The scenario is the following, client code can register a callback for one or a 
list of notifications. 
This is currently solved by having a dictionary with the possible notifications 
as the keys and lists of methods being the values. 
Notification sent, triggers the list of method to be called. Nothing fancy.

Now, where it gets trickier is that the client should be able to register 
synchronous and asynchronous callbacks because some of the actions can't be 
done in a synchronous callback like updating the statusbar of the UI as the 
main app would overwrite it.

I've read that this isn't a good idea of having synchronous and asynchronous 
callbacks but to be honest, I don't see how this could be solved otherwise, 
hence the question what to do in such a case?

And as I currently think that there is no other solution then to provide sync 
and async callback registration possibility how would a solution look like? 
Two dicts, one for sync and the other for async registrations?
That would mean that each notification handler needs to execute one dict after 
the other.
In an ideal world, the notification handler wouldn't know about the difference 
at all and would call its list of methods(objects?)

Any tips for me?

Just to be sure as I don't know whether this is relevant,
each notification handler provides a different set of arguments to the 
registered callback functions, hence the multiple notification handlers.

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unable to start Python with Windows 7

[Eko palypse]

>What is strange is that I can run Python, but only with a short program !

Hi Crhistian,
hard to tell, what might help is if you can copy/paste code which does
not run (an example should be as small as possible).
In addition, isn't there any output you get when the program doesn't run?
A traceback etc... and how do you run those programs, by double clicking
.py files via cmd like python myfile.py, via an editor ...

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

Am Donnerstag, 19. September 2019 20:24:49 UTC+2 schrieb Peter Otten:
> Eko palypse wrote:
> 
> > Am Donnerstag, 19. September 2019 18:31:43 UTC+2 schrieb Peter Otten:
> >> Eko palypse wrote:
> >> 
> >> > No, I have to correct myself
> >> > 
> >> > x = 5
> >> > def f1():
> >> > exec("x = x + 1; print('f1 in:', x)")
> >> > return x
> >> > print('f1 out', f1())
> >> > 
> >> > results in the same, for me confusing, results.
> >> > 
> >> > f1 in: 6
> >> > f1 out 5
> >> 
> >> Inside a function exec assignments go to a *copy* of the local namespace.
> >> Also LOAD_NAME is used to look up names. Therefore you can read and then
> >> shade a global name with its local namesake.
> >> 
> >> Inside a function the namespace is determined statically. As f1() has no
> >> assignment to x (code inside exec(...) is not considered) x is looked up
> >> in directly the global namespace using LOAD_GLOBAL.
> >> 
> >> If you want to access the local namespace used by exec() you have to
> >> provide one explicitly:
> >> 
> >> >>> x = 5
> >> >>> def f():
> >> ... ns = {}
> >> ... exec("x += 1", globals(), ns)
> >> ... return ns["x"]
> >> ...
> >> >>> f()
> >> 6
> >> >>> x
> >> 5
> >> 
> >> By the way, in Python 2 where exec was a statement the local namespace is
> >> shared:
> >> 
> >> >>> x = 5
> >> >>> def f():
> >> ... exec "x += 1"
> >> ... return x
> >> ...
> >> >>> f()
> >> 6
> >> >>> x
> >> 5
> > 
> > Sorry, missed that.
> > Thank you, may I ask you how I could have come myself
> > to that explanation? What do I have to read to get that understanding?
> > Hopefully you don't say read the C code, because that is something
> > I tried but failed miserably.
> 
> https://docs.python.org/3/library/functions.html#exec
> https://docs.python.org/3/reference/executionmodel.html#naming-and-binding
> 
> (I had to google for the second link.)
> 
> I usually experiment with code and the disassembler. I find its output quite 
> readable, 
> 
> >>> def f(): x += 1
> ... 
> >>> import dis
> >>> dis.dis(f)
>   1   0 LOAD_FAST0 (x)
>   3 LOAD_CONST   1 (1)
>   6 INPLACE_ADD
>   7 STORE_FAST   0 (x)
>  10 LOAD_CONST   0 (None)
>  13 RETURN_VALUE
> >>> dis.dis(compile("x += 1", "", "exec"))
>   1   0 LOAD_NAME0 (x)
>   3 LOAD_CONST   0 (1)
>   6 INPLACE_ADD
>   7 STORE_NAME   0 (x)
>  10 LOAD_CONST   1 (None)
>  13 RETURN_VALUE
> 
> and you can limit yourself to small snippets.

Thank you very much, really, very much appreciated.
Normally I do use docs.python.org as my first resource to look 
for an answer to my questions but I'm not always convinced that
my understanding/interpretation is correct.
Then I research on the web to see if there are similar questions and
try to understand these explanations but ...
and in this particular case I wasn't even able to find an answer. 

The dis module is nice to see differences but I assume more useful
to software developers than to hobby programmers like me.
At least not in the current state of knowledge I have.
I was under the impression that I have a good understanding of the 
python language before I started my current project but now ...
Enough whining :-)

Once again,
Thank you very much
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

Am Donnerstag, 19. September 2019 18:31:43 UTC+2 schrieb Peter Otten:
> Eko palypse wrote:
> 
> > No, I have to correct myself
> > 
> > x = 5
> > def f1():
> > exec("x = x + 1; print('f1 in:', x)")
> > return x
> > print('f1 out', f1())
> > 
> > results in the same, for me confusing, results.
> > 
> > f1 in: 6
> > f1 out 5
> 
> Inside a function exec assignments go to a *copy* of the local namespace.
> Also LOAD_NAME is used to look up names. Therefore you can read and then 
> shade a global name with its local namesake.
> 
> Inside a function the namespace is determined statically. As f1() has no 
> assignment to x (code inside exec(...) is not considered) x is looked up in 
> directly the global namespace using LOAD_GLOBAL.
> 
> If you want to access the local namespace used by exec() you have to provide 
> one explicitly:
> 
> >>> x = 5
> >>> def f():
> ... ns = {}
> ... exec("x += 1", globals(), ns)
> ... return ns["x"]
> ... 
> >>> f()
> 6
> >>> x
> 5
> 
> By the way, in Python 2 where exec was a statement the local namespace is 
> shared:
> 
> >>> x = 5
> >>> def f():
> ... exec "x += 1"
> ... return x
> ... 
> >>> f()
> 6
> >>> x
> 5

Sorry, missed that. 
Thank you, may I ask you how I could have come myself
to that explanation? What do I have to read to get that understanding?
Hopefully you don't say read the C code, because that is something
I tried but failed miserably.

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

First thank you for all the answers, very much appreciated.
I assume the root cause might be explained by the zen of python as well.

If the implementation is hard to explain, it's a bad idea.

Maybe I need to rethink my implementation :-)

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unable to start Python with Windows 7

[Eko palypse]

Am Donnerstag, 19. September 2019 17:52:48 UTC+2 schrieb cdoa...@orange.com:
> Hi,
> I am no more able to start Python from Windows 7 environment.
> I have the following message :
> "The Application was unable to start correctly, (0xC142). Click OK to 
> close the application"
> 
> Do you have any idea where the problem is ?
> 
> Bests regards,
> 

A shot in the dark, you do have Win7 32bit and tried to start python 64bit
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

No, I have to correct myself

x = 5
def f1():
exec("x = x + 1; print('f1 in:', x)")
return x
print('f1 out', f1())

results in the same, for me confusing, results.

f1 in: 6
f1 out 5

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

Am Donnerstag, 19. September 2019 12:56:59 UTC+2 schrieb Peter Otten:
> Eko palypse wrote:
> 
> >> Then it should be clear that the name 'test01' is put into globals(), if
> >> load_module() doesn't throw an exception. No sharing or nesting of
> >> namespaces takes place.
> > 
> > Thank you too for your answer. Ok, that means that in every case when exec
> > imports something it has its own global namespace, right?
> > Is there a way to manipulate this namespace before any code from that
> > module gets executed?
> 
> Create a module object, preload its namespace, then exec the module's code, 
> like:
> 
> $ python3
> Python 3.4.3 (default, Nov 12 2018, 22:25:49) 
> [GCC 4.8.4] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys, types
> >>> module = types.ModuleType("module")
> >>> module.x = 42
> >>> exec("print(x)\ndef f(): return x * x", module.__dict__)
> 42
> >>> sys.modules["module"] = module
> >>> import module
> >>> module.f()
> 1764
> 
> The importlib probably has an official way, but I've yet to wrap my head 
> around that part of Python.

Thank you, I'm currently investigating importlib and read that __builtins__
might be another alternative.
My ultimate goal would be to have objects available without the need to import 
them, regardless whether used in a script directly or used in an imported 
module.

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

Am Donnerstag, 19. September 2019 12:45:35 UTC+2 schrieb Richard Damon:
> On 9/19/19 6:16 AM, Eko palypse wrote:
> >> In all cases, if the optional parts are omitted, the code is executed in 
> >> the current scope. ...
> >>
> >>
> >> You can see from it that "globals" is optional.
> >> And that, if "globals" is missing, then
> >> "exec" is executed in the current scope ("f1" in your case).
> > Thank you for your answer, and that is exactly what confuses me?
> > Where does x come from? If I only would read x then I would understand why
> > it can be found/read but I alter it and as such I either have to provide the
> > info that this is a global variable, declare it inside of f1 or provide 
> > the globals dict to exec. But I don't do any of it. Why is exec able to use
> > the global x?
> >
> > Eren
> 
> I think the issue is that x += 1 isn't exactly like x = x + 1, and this
> is one case that shows it. x = x + 1 is an assignment to the symbol x,
> which makes x a local, and thus the read becomes an undefined symbol. x
> += 1 is different, it isn't a plain assignment so doesn't create the
> local. The read of x is inherently tied to the writing of x so x stays
> referring to the global.
> 
> -- 
> Richard Damon

Thank you that would never have come to my mind.
I thought +=1 is just syntactic sugar which clearly isn't.
If I do the regular x = x + 1 then I do get the expected exception.

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

> Then it should be clear that the name 'test01' is put into globals(), if 
> load_module() doesn't throw an exception. No sharing or nesting of 
> namespaces takes place.

Thank you too for your answer. Ok, that means that in every case when exec
imports something it has its own global namespace, right?
Is there a way to manipulate this namespace before any code from that module
gets executed? 

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and globals and locals ...

[Eko palypse]

> In all cases, if the optional parts are omitted, the code is executed in the 
> current scope. ...
> 
> 
> You can see from it that "globals" is optional.
> And that, if "globals" is missing, then
> "exec" is executed in the current scope ("f1" in your case).

Thank you for your answer, and that is exactly what confuses me?
Where does x come from? If I only would read x then I would understand why
it can be found/read but I alter it and as such I either have to provide the
info that this is a global variable, declare it inside of f1 or provide 
the globals dict to exec. But I don't do any of it. Why is exec able to use
the global x?

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

exec and globals and locals ...

[Eko palypse]

Why does f1 work? I've expected an exception as no global dict has been 
provided, and why does throw f3 an exception if it does, more or less, the same 
as f1?

x += 5
def f1():
exec("x += 1; print('f1 in:', x)")
return x
print('f1 out', f1())

# result => f1 in: 6
# result => f1 out 5


x = 5
def f2():
exec("x += 1; print('f2 in:', x)", globals())
return x
print('f2 out', f2())

# result => f2 in: 6
# result => f2 out 6


exec('import test01', globals())
print('f3 out', x)

# result exception, expected but because f1 didn't throw an exception 
# I'm confused. module test01 has only this two lines
x += 1
print('f3 in:', x)

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python3.7 singleton is not unique anymore

[Eko palypse]

Shame on me :-)
Thank you very much Ethan.
-- 
https://mail.python.org/mailman/listinfo/python-list

Python3.7 singleton is not unique anymore

[Eko palypse]

Using the following code in Python3.7 doesn't make FOO to be singleton anymore.

import sys

class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, 
**kwargs)
return cls._instances[cls]

if sys.version_info.major == 2:  
class FOO():
__metaclass__ = Singleton
def __init__(self):
self.id = None

def setid(self, id):
self.id = id

def getid(self):
return self.id
else:
class FOO():
def __init__(self, metaclass=Singleton):
self.id = None

def setid(self, id):
self.id = id

def getid(self):
return self.id


x1 = FOO()
x1.setid(1)
x2 = FOO()
x2.setid(2)
x = x1
y = FOO()

print(x1, x1.id)
print(x2, x2.id)
print(x, x.id)
print(y, y.id)

python2 results in something like

<__main__.FOO object at 0x07D9C978>, 2
<__main__.FOO object at 0x07D9C978>, 2
<__main__.FOO object at 0x07D9C978>, 2
<__main__.FOO object at 0x07D9C978>, 2

whereas python3.7 result in
<__main__.FOO object at 0x072C2EB8> 1
<__main__.FOO object at 0x07357080> 2
<__main__.FOO object at 0x072C2EB8> 1
<__main__.FOO object at 0x072C2160> None

What did I miss?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Metaclasses and classproperties

[Eko palypse]

Thank you for your thoughts.
I'm trying to migrating en existing python2 api, 
which has been build using boost::python with pure python3 code. 
All of the py2 classes do have these additional properties names and values.
You are right, using dict(Ordinal.__members__) for names
could have been used as well but I thought, 
if someone else wants to change something it might be easier
to understand if both properties look the same, maybe a wrong assumption.
(something I seem to do often :-)

Eren

-- 
https://mail.python.org/mailman/listinfo/python-list

Metaclasses and classproperties

[Eko palypse]

I'm fairly new when it comes to metaclass programming and therefore the
question whether the following makes sense or not.

The goal is to have two additional class properties which return a
dictionary name:class_attribute and value:class_attribute for an IntEnum class
and after reading about it I came to the conclusion that the following code
might do what I want, and it does do the job BUT does it make sense also?

Meaning, my ultimate goal would be to be able to decide myself
if this is a good or bad idea doing this, so what do I need to 
read/understand in order to achieve such a goal.


from enum import EnumMeta, IntEnum

class EnhancedIntEnum(EnumMeta):
@property
def names(cls):
return {k: v for k, v in cls.__members__.items()}

@property
def values(cls):
return {v.value: v for k, v in cls.__members__.items()}


class Ordinal(IntEnum, metaclass=EnhancedIntEnum):
NORTH = 0
SOUTH = 1
EAST = 2
WEST = 3

print(Ordinal.names)
print(Ordinal.values)

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Color representation as rgb, int and hex

[Eko palypse]

> No, constructing a bytes literal from hex digits implies that they
> follow the sequence in the string of digits. It's nothing to do with
> the endinanness of integers.
> ChrisA

> Why should it imply that? You're asking it to create some bytes
> from a string of hex digits -- no mention of integers. The obvious
> thing to do is to put the bytes in the order they apper in the
> string.
> Greg

@Greg, ChrisA
Agreed, as I said this came into my mind first as I saw the 
name of the method but your arguments make sense, absolutely.

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Color representation as rgb, int and hex

[Eko palypse]

> ChrisA

You are correct, but, sorry for not being clear what confused me.
I assumed it would use the sys.byteorder but I guess this is simply a
AssumedError exception.  :-)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Color representation as rgb, int and hex

[Eko palypse]

> You haven't said whether your machine is big-endian or little-endian.

Correct, it is little but I'm wondering why this isn't taking into account.
I thought a method called fromhex would imply that bytes for an integer
should be created and as that it would use the proper byte order to create it.
But it seems that it treats the string literally.
Isn't that confusing?

Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Color representation as rgb, int and hex

[Eko palypse]

I'm confused about the following

import sys
print(tuple(bytes.fromhex('282C34')))
print(tuple((0x282C34).to_bytes(3, byteorder=sys.byteorder)))

which results in 

(40, 44, 52)
(52, 44, 40)

on my machine. Shouldn't I expect the same result?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

How to annotate an IntEnum value

[Eko palypse]

Suppose the following

from enum import IntEnum

class ENDOFLINE(IntEnum):
CRLF = 0
CR = 1
LF = 2


def get_eol() -> ??:
return ENDOFLINE.CRLF 

def set_eol(eol_value) -> None:
pass


How should the return value from get_eol be annotated? As ENDOFLINE?
The same question for set_eol function, is it assumed to see the following 
declaration?

def set_eol(value: ENDOFLINE) -> None:
pass

I've tried to see what pep484 suggests but either I haven't understood it
or it doesn't go into detail in such a case.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: a,b = 2,3 and [a,b] = [2,3]

[Eko palypse]

Am Montag, 2. September 2019 00:49:05 UTC+2 schrieb Hongyi Zhao:
> Hi,
> 
> What's differences:
> 
> a,b = 2,3 and [a,b] = [2,3]
> 
> Regards

In this example the result is the same but the second one
builds, internally, an additional list, therefore isn't as sufficient
as the first one.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using exec with embedded python interpreter 3.7

[Eko palypse]

@MRAB, I'm building a notepad++ plugin which can execute the
written code and if one writes help(os) it gets executed via
exec(editor.getText()) and output redirected to the plugin console window.
Sorry to you as well as I have also replied to you directly.
Thank you
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using exec with embedded python interpreter 3.7

[Eko palypse]

Just saw, that I replied to you directly instead to python list, sorry.
That did it, changed encoding from function to property and now I'm able to 
call help(object)
Thank you.
-- 
https://mail.python.org/mailman/listinfo/python-list

Using exec with embedded python interpreter 3.7

[Eko palypse]

I've already sent this through mail yesterday but it doesn't appear here, maybe 
because of the help word in the content. Please execute in case it appears a 
second time.


Hello,
I'm creating a notepad++ plugin which hosts an embedded python interpreter by 
using cffi to build the dll.
So far so good. One obstacle I've found is that I'm not able to use 
exec(help(object)) in order to get the wanted info from the object.

The error I get is:

Traceback (most recent call last):
  File "", line 131, in run_code
  File "", line 1, in 
  File "D:\...\Python\Python37_64\Lib\_sitebuiltins.py", line 103, in __call__
return pydoc.help(*args, **kwds)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1895, in __call__
self.help(request)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1954, in help
else: doc(request, 'Help on %s:', output=self._output)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1674, in doc
pager(render_doc(thing, title, forceload))
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1451, in pager
pager(text)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1576, in plainpager
sys.stdout.write(plain(_escape_stdout(text)))
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1528, in _escape_stdout
return text.encode(encoding, 'backslashreplace').decode(encoding)
TypeError: encode() argument 1 must be str, not method

If I encase the object with quotes then I do get the info that no documentation
is found for that object.

1. It doesn't feel right to encase the object with quotes. I don't have to do 
this using the standard python interpreter shell. What did I miss?

2. If it is needed to use the encased way, why don't it show the help info?
The object class contains the module, class and functions doc strings, what did 
I miss here?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list

Using exec with embedded python interpreter 3.7

[Eko palypse]

Hello,
I'm creating a notepad++ plugin which hosts an embedded python interpreter
by using cffi to build the dll.
So far so good. One obstacle I've found is that I'm not able to use
exec(help(object)) in order to get the wanted info from the object.

The error I get is:

Traceback (most recent call last):
  File "", line 131, in run_code
  File "", line 1, in 
  File "D:\...\Python\Python37_64\Lib\_sitebuiltins.py", line 103, in
__call__
return pydoc.help(*args, **kwds)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1895, in __call__
self.help(request)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1954, in help
else: doc(request, 'Help on %s:', output=self._output)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1674, in doc
pager(render_doc(thing, title, forceload))
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1451, in pager
pager(text)
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1576, in plainpager
sys.stdout.write(plain(_escape_stdout(text)))
  File "D:\...\Python\Python37_64\Lib\pydoc.py", line 1528, in
_escape_stdout
return text.encode(encoding, 'backslashreplace').decode(encoding)
TypeError: encode() argument 1 must be str, not method

If I encase the object with quotes then I do get the info that no
documentation
is found for that object.

1. It doesn't feel right to encase the object with quotes. I don't have to
do this using the standard python interpreter shell. What did I miss?

2. If it is needed to use the encased way, why don't it show the help info?
The object class contains the module, class and functions doc strings, what
did I miss here?

Thank you
Eren
-- 
https://mail.python.org/mailman/listinfo/python-list