Re: Python3 extension help needed

[matkuki]

Thanks jlp765!

Tried **GC_disableMarkAndSweep()** at the beginning and 
**GC_enableMarkAndSweep()** at the end of the procedure, still it throws the 
segmentation fault.

Then I tried **GC_disable()** and **GC_enable()** \+ **GC_fullCollect()** and 
it works!

Re: Python3 extension help needed

[jlp765]

Try disabling 
[MarkAndSweep](https://nim-lang.org/docs/system.html#GC_disableMarkAndSweep) 
temporarily.

Re: Python3 extension help needed

[matkuki]

Araq, a little help please.

I have a Nim procedure that takes text and a highlighting function from Python, 
then parses and highlights it. This works great but when parsing large pieces 
of text something with the Nim garbage collector causes a segmentation fault in 
Python. I know it has something to do with the GC, because the segmentation 
fault goes away if I compile the module with **\--gc:none**. But if I do this, 
the memory of the running application constantly rises, as you would expect.

Is there a solution for this without disabling the GC?

Re: Python3 extension help needed

[matkuki]

Yep, Araq, you were right 

It's an implementation issue that has nothing to do with Nim. Seems that the 
Raspberry PI 3 has a bit of a hard time processing long strings.

Thanks.

Re: Python3 extension help needed

[Araq]

No, that's strange. Why would it do that? 

Re: Python3 extension help needed

[matkuki]

Ok, so I managed to write an extension by directly using the **Python.h** 
header. This works great on Windows x86/x64 and Linux x86/x64, but not on Linux 
ARM x86 (Raspberry PI 3). By doesn't work I mean it compiles and executes, but 
**EVERY** call to the Nim compiled module (**.so** shared library) has about a 
one second delay where the CPU usage of one core jumps to 100%.

Is this a known issue on ARM? 

Re: Python3 extension help needed

[matkuki]

Hey,

The conclusion from the python3 IRC discussion:

> Debian and Debian based distributions do not load the Python 3 shared library 
> into memory, so it is necessary to link statically with the **Python.h** 
> header.

Lesson learned. 

Re: Python3 extension help needed

[matkuki]

@Araq,

Fiddled around in the generated C file in the **nimcache** directory by adding 
code to load the library **libpython3.5m.so.1.0** in C directly and it throws 
the same error!

The added code (added to the generated **PyInit_nim_module** function) is: 


N_CDECL(..., PyInit_nim_module)(void) {
  NimMain();
  nimln_(171, "nim_module.nim");
  void* handle;
  nimln_(900, "nim_module.nim");
  handle = dlopen("libpython3.5m.so.1.0", RTLD_LAZY);
  nimln_(901, "nim_module.nim");
  void (*init)(void) = dlsym(handle, "Py_Initialize");
  nimln_(902, "nim_module.nim");
  (*init)(); <- ERROR
  nimln_(903, "nim_module.nim");
  dlclose(handle);
  /*
  Rest of the code
  */
}


Do you perhaps know why this error occurs when invoking the **dlsym** loaded 
function?

I know this question would probably be better placed in a python forum, but I'm 
hoping someone knows something about the problem!

Re: Python3 extension help needed

[matkuki]

After further testing and help from jasonrbriggs, I found out that the error 
appears on all Debian Linux distributions. It works on Windows and Arch Linux, 
but on Ubuntu MATE (ARM), Lubuntu (x86) and Raspbian (ARM) it doesn't. Tried 
Python versions from 3.4 to 3.6, Nim was at the latest github version.

Even this simple example (already shown above) like this throws an error: 


# The imported Python3 initialization function
proc initialize*(){.cdecl, importc: "Py_Initialize" dynlib: 
"libpython3.YOUR_VERSIONm.so.1.0".}

# The module's init function
proc PyInit_nim_module() {.cdecl, exportc.} =
  {.emit: """NimMain();""".}
  initialize() # <--- ERROR


Anyone know how the Python 3 installaion Debian distributions differs from 
other distributions or other OS'es?