Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
On 8/27/2012 2:45 PM, Christoph Gohlke wrote: > On 8/27/2012 10:58 AM, Stuart Mentzer wrote: >> On 8/27/2012 1:22 PM, Christoph Gohlke wrote: >>> On 8/27/2012 9:42 AM, Antonio Valentino wrote: Hi Stuart, Il 27/08/2012 17:43, Stuart Mentzer ha scritto: > Hello, > > I upgraded to PyTables 2.4.0 and I was "freezing" an application on > Windows with PyInstaller. The frozen app fails at this new find_library > call in __init__.py: > > if not ctypes.util.find_library('hdf5dll.dll'): > raise ImportError('Could not load "hdf5dll.dll", please > ensure' + > ' that it can be found in the system path') > > PyInstaller correctly places this DLL in the same directory as the > application .exe where standard Windows DLL search logic will find it. > Apparently the find_library doesn't do that in a frozen application. That > is a big problem. I had to comment this code out to get a working frozen > app. > > That code was added in revision e9f6919. > It is mainly a sanity check added under request of one of our users: https://github.com/PyTables/PyTables/pull/146 > This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both > PyInstaller 1.5.1 and 2.0. > > Should I file a bug report? Any easy work-around? > > Thanks, > Stuart > Yes please file a pull request with your patch. It would be nice to preserve the sanity check in standard case so, maybe, a good solution could be adding some check on sys.frozen or something like that. Thank you >>> Hello, >>> >>> As a workaround for frozen distributions, try to add the sys.executable >>> directory to os.environ['PATH'] before importing tables. >>> >>> Ctypes only tries to find a library in the os.environ['PATH'] >>> directories, not the current directory or the sys.executable directory >>> as one could expect. >>> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48 >>> >>> As a workaround, for distributions that place the HDF5 and other DLLs in >>> the tables package directory, tables.__init__.py adds the tables package >>> directory to os.environ['PATH']. This also makes sure that the DLLs are >>> found when loading the hdf5Extension.pyd and other C extension modules >>> (another common problem). The use of __file__ to get the tables >>> directory should better be wrapped in a try..except statement. >>> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 >>> >>> Christoph >> Hi Christoph, >> >> Thanks for the info/suggestions. It might be nice to add your comments to the >> Issue #177 I created. >> >> I was aware that altering the PATH is a work-around. Patching PyTables is >> cleaner >> and seems like the proper fix, and I think we agree that there is a problem >> here >> that should be addressed. Maybe the PyTables test suite should even include a >> frozen application test. >> >> Thanks, >> Stuart >> > Hi Stuart, > > I'll try to work on a patch tonight. It's probably better to use Ctypes > LoadLibrary instead of find_library because that makes sure all HDF5 > dependencies are found, it (supposedly) searches the Windows DLL search > path not just os.environ['PATH'], and (supposedly) takes into account > libraries already loaded into the process. > > Christoph Great. I'll be happy to test a patch in the frozen context. Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
On 8/27/2012 10:58 AM, Stuart Mentzer wrote: > On 8/27/2012 1:22 PM, Christoph Gohlke wrote: >> On 8/27/2012 9:42 AM, Antonio Valentino wrote: >>> Hi Stuart, >>> >>> Il 27/08/2012 17:43, Stuart Mentzer ha scritto: Hello, I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py: if not ctypes.util.find_library('hdf5dll.dll'): raise ImportError('Could not load "hdf5dll.dll", please ensure' + ' that it can be found in the system path') PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app. That code was added in revision e9f6919. >>> It is mainly a sanity check added under request of one of our users: >>> https://github.com/PyTables/PyTables/pull/146 >>> >>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0. Should I file a bug report? Any easy work-around? Thanks, Stuart >>> Yes please file a pull request with your patch. >>> It would be nice to preserve the sanity check in standard case so, >>> maybe, a good solution could be adding some check on sys.frozen or >>> something like that. >>> >>> Thank you >>> >> Hello, >> >> As a workaround for frozen distributions, try to add the sys.executable >> directory to os.environ['PATH'] before importing tables. >> >> Ctypes only tries to find a library in the os.environ['PATH'] >> directories, not the current directory or the sys.executable directory >> as one could expect. >> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48 >> >> As a workaround, for distributions that place the HDF5 and other DLLs in >> the tables package directory, tables.__init__.py adds the tables package >> directory to os.environ['PATH']. This also makes sure that the DLLs are >> found when loading the hdf5Extension.pyd and other C extension modules >> (another common problem). The use of __file__ to get the tables >> directory should better be wrapped in a try..except statement. >> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 >> >> Christoph > Hi Christoph, > > Thanks for the info/suggestions. It might be nice to add your comments to the > Issue #177 I created. > > I was aware that altering the PATH is a work-around. Patching PyTables is > cleaner > and seems like the proper fix, and I think we agree that there is a problem > here > that should be addressed. Maybe the PyTables test suite should even include a > frozen application test. > > Thanks, > Stuart > Hi Stuart, I'll try to work on a patch tonight. It's probably better to use Ctypes LoadLibrary instead of find_library because that makes sure all HDF5 dependencies are found, it (supposedly) searches the Windows DLL search path not just os.environ['PATH'], and (supposedly) takes into account libraries already loaded into the process. Christoph -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
On 8/27/2012 1:52 PM, Antonio Valentino wrote: > Hi Christoph, > > thank you very much for your hints. > > Il 27/08/2012 19:22, Christoph Gohlke ha scritto: >> On 8/27/2012 9:42 AM, Antonio Valentino wrote: >>> Hi Stuart, >>> >>> Il 27/08/2012 17:43, Stuart Mentzer ha scritto: Hello, I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py: if not ctypes.util.find_library('hdf5dll.dll'): raise ImportError('Could not load "hdf5dll.dll", please ensure' + ' that it can be found in the system path') PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app. That code was added in revision e9f6919. >>> It is mainly a sanity check added under request of one of our users: >>> https://github.com/PyTables/PyTables/pull/146 >>> >>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0. Should I file a bug report? Any easy work-around? Thanks, Stuart >>> Yes please file a pull request with your patch. >>> It would be nice to preserve the sanity check in standard case so, >>> maybe, a good solution could be adding some check on sys.frozen or >>> something like that. >>> >>> Thank you >>> >> Hello, >> >> As a workaround for frozen distributions, try to add the sys.executable >> directory to os.environ['PATH'] before importing tables. >> > Christoph, I suppose it can also be done at this point too: > https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 > Isn't it? > > Please Stuart, can you try this fix as well. Can't try it right now but I can't see why adding sys.excutable to the PATH at that point wouldn't "work" as an alternative to skipping the sanity check when frozen. My broader concern is whether fiddling with the PATH is the best or really the correct fix. If you need to exactly replicate Windows DLL search rules (which, just to make life fun, vary depending on the Windows version and the state of the SafeDllSearchMode*) then you should do that rather than hacking the PATH. For example, what if there is another hdf5dll.dll already on the PATH? In that case you'd better put sys.executable in the front of the PATH, but that might break something else. Maybe 99.9% of the time the PATH hack works but it can also fail. I think the only proper fix is to make find_library follow the Windows rules exactly. Short of that, it is not safe to use it when the library might be in different places, such as frozen vs non-frozen applications. [*] http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586%28v=vs.85%29.aspx >> Ctypes only tries to find a library in the os.environ['PATH'] >> directories, not the current directory or the sys.executable directory >> as one could expect. >> http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48 >> >> As a workaround, for distributions that place the HDF5 and other DLLs in >> the tables package directory, tables.__init__.py adds the tables package >> directory to os.environ['PATH']. This also makes sure that the DLLs are >> found when loading the hdf5Extension.pyd and other C extension modules >> (another common problem). The use of __file__ to get the tables >> directory should better be wrapped in a try..except statement. >> https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 >> >> Christoph > Thanks, I'll fix it ASAP. > > P.S.: please Christoph, do you have some hint for gh-175 [1]? > There is something we can do in PyTables? > > [1] https://github.com/PyTables/PyTables/issues/175 Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
On 8/27/2012 1:22 PM, Christoph Gohlke wrote: > On 8/27/2012 9:42 AM, Antonio Valentino wrote: >> Hi Stuart, >> >> Il 27/08/2012 17:43, Stuart Mentzer ha scritto: >>> Hello, >>> >>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows >>> with PyInstaller. The frozen app fails at this new find_library call in >>> __init__.py: >>> >>> if not ctypes.util.find_library('hdf5dll.dll'): >>> raise ImportError('Could not load "hdf5dll.dll", please ensure' >>> + >>> ' that it can be found in the system path') >>> >>> PyInstaller correctly places this DLL in the same directory as the >>> application .exe where standard Windows DLL search logic will find it. >>> Apparently the find_library doesn't do that in a frozen application. That >>> is a big problem. I had to comment this code out to get a working frozen >>> app. >>> >>> That code was added in revision e9f6919. >>> >> It is mainly a sanity check added under request of one of our users: >> https://github.com/PyTables/PyTables/pull/146 >> >> >>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both >>> PyInstaller 1.5.1 and 2.0. >>> >>> Should I file a bug report? Any easy work-around? >>> >>> Thanks, >>> Stuart >>> >> Yes please file a pull request with your patch. >> It would be nice to preserve the sanity check in standard case so, >> maybe, a good solution could be adding some check on sys.frozen or >> something like that. >> >> Thank you >> > Hello, > > As a workaround for frozen distributions, try to add the sys.executable > directory to os.environ['PATH'] before importing tables. > > Ctypes only tries to find a library in the os.environ['PATH'] > directories, not the current directory or the sys.executable directory > as one could expect. > http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48 > > As a workaround, for distributions that place the HDF5 and other DLLs in > the tables package directory, tables.__init__.py adds the tables package > directory to os.environ['PATH']. This also makes sure that the DLLs are > found when loading the hdf5Extension.pyd and other C extension modules > (another common problem). The use of __file__ to get the tables > directory should better be wrapped in a try..except statement. > https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 > > Christoph Hi Christoph, Thanks for the info/suggestions. It might be nice to add your comments to the Issue #177 I created. I was aware that altering the PATH is a work-around. Patching PyTables is cleaner and seems like the proper fix, and I think we agree that there is a problem here that should be addressed. Maybe the PyTables test suite should even include a frozen application test. Thanks, Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
Hi Christoph, thank you very much for your hints. Il 27/08/2012 19:22, Christoph Gohlke ha scritto: > On 8/27/2012 9:42 AM, Antonio Valentino wrote: >> Hi Stuart, >> >> Il 27/08/2012 17:43, Stuart Mentzer ha scritto: >>> Hello, >>> >>> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows >>> with PyInstaller. The frozen app fails at this new find_library call in >>> __init__.py: >>> >>> if not ctypes.util.find_library('hdf5dll.dll'): >>> raise ImportError('Could not load "hdf5dll.dll", please ensure' >>> + >>> ' that it can be found in the system path') >>> >>> PyInstaller correctly places this DLL in the same directory as the >>> application .exe where standard Windows DLL search logic will find it. >>> Apparently the find_library doesn't do that in a frozen application. That >>> is a big problem. I had to comment this code out to get a working frozen >>> app. >>> >>> That code was added in revision e9f6919. >>> >> >> It is mainly a sanity check added under request of one of our users: >> https://github.com/PyTables/PyTables/pull/146 >> >> >>> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both >>> PyInstaller 1.5.1 and 2.0. >>> >>> Should I file a bug report? Any easy work-around? >>> >>> Thanks, >>> Stuart >>> >> >> Yes please file a pull request with your patch. >> It would be nice to preserve the sanity check in standard case so, >> maybe, a good solution could be adding some check on sys.frozen or >> something like that. >> >> Thank you >> > > Hello, > > As a workaround for frozen distributions, try to add the sys.executable > directory to os.environ['PATH'] before importing tables. > Christoph, I suppose it can also be done at this point too: https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 Isn't it? Please Stuart, can you try this fix as well. > Ctypes only tries to find a library in the os.environ['PATH'] > directories, not the current directory or the sys.executable directory > as one could expect. > http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48 > > As a workaround, for distributions that place the HDF5 and other DLLs in > the tables package directory, tables.__init__.py adds the tables package > directory to os.environ['PATH']. This also makes sure that the DLLs are > found when loading the hdf5Extension.pyd and other C extension modules > (another common problem). The use of __file__ to get the tables > directory should better be wrapped in a try..except statement. > https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 > > Christoph Thanks, I'll fix it ASAP. P.S.: please Christoph, do you have some hint for gh-175 [1]? There is something we can do in PyTables? [1] https://github.com/PyTables/PyTables/issues/175 -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
> Hi Stuart, > > Il 27/08/2012 17:43, Stuart Mentzer ha scritto: >> Hello, >> >> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows >> with PyInstaller. The frozen app fails at this new find_library call in >> __init__.py: >> >>if not ctypes.util.find_library('hdf5dll.dll'): >>raise ImportError('Could not load "hdf5dll.dll", please ensure' + >>' that it can be found in the system path') >> >> PyInstaller correctly places this DLL in the same directory as the >> application .exe where standard Windows DLL search logic will find it. >> Apparently the find_library doesn't do that in a frozen application. That is >> a big problem. I had to comment this code out to get a working frozen app. >> >> That code was added in revision e9f6919. >> > It is mainly a sanity check added under request of one of our users: > https://github.com/PyTables/PyTables/pull/146 > > >> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both >> PyInstaller 1.5.1 and 2.0. >> >> Should I file a bug report? Any easy work-around? >> >> Thanks, >> Stuart >> > Yes please file a pull request with your patch. > It would be nice to preserve the sanity check in standard case so, > maybe, a good solution could be adding some check on sys.frozen or > something like that. > > Thank you Antonio, Thanks for the quick response. To save time I did the report as a simple Issue (https://github.com/PyTables/PyTables/issues/177) showing the patch code that preserves and with a reference to #146. I hope that suffices. Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
On 8/27/2012 9:42 AM, Antonio Valentino wrote: > Hi Stuart, > > Il 27/08/2012 17:43, Stuart Mentzer ha scritto: >> Hello, >> >> I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows >> with PyInstaller. The frozen app fails at this new find_library call in >> __init__.py: >> >>if not ctypes.util.find_library('hdf5dll.dll'): >>raise ImportError('Could not load "hdf5dll.dll", please ensure' + >>' that it can be found in the system path') >> >> PyInstaller correctly places this DLL in the same directory as the >> application .exe where standard Windows DLL search logic will find it. >> Apparently the find_library doesn't do that in a frozen application. That is >> a big problem. I had to comment this code out to get a working frozen app. >> >> That code was added in revision e9f6919. >> > > It is mainly a sanity check added under request of one of our users: > https://github.com/PyTables/PyTables/pull/146 > > >> This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both >> PyInstaller 1.5.1 and 2.0. >> >> Should I file a bug report? Any easy work-around? >> >> Thanks, >> Stuart >> > > Yes please file a pull request with your patch. > It would be nice to preserve the sanity check in standard case so, > maybe, a good solution could be adding some check on sys.frozen or > something like that. > > Thank you > Hello, As a workaround for frozen distributions, try to add the sys.executable directory to os.environ['PATH'] before importing tables. Ctypes only tries to find a library in the os.environ['PATH'] directories, not the current directory or the sys.executable directory as one could expect. http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48 As a workaround, for distributions that place the HDF5 and other DLLs in the tables package directory, tables.__init__.py adds the tables package directory to os.environ['PATH']. This also makes sure that the DLLs are found when loading the hdf5Extension.pyd and other C extension modules (another common problem). The use of __file__ to get the tables directory should better be wrapped in a try..except statement. https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24 Christoph -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
Hi Stuart, Il 27/08/2012 17:43, Stuart Mentzer ha scritto: > Hello, > > I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows > with PyInstaller. The frozen app fails at this new find_library call in > __init__.py: > > if not ctypes.util.find_library('hdf5dll.dll'): > raise ImportError('Could not load "hdf5dll.dll", please ensure' + > ' that it can be found in the system path') > > PyInstaller correctly places this DLL in the same directory as the > application .exe where standard Windows DLL search logic will find it. > Apparently the find_library doesn't do that in a frozen application. That is > a big problem. I had to comment this code out to get a working frozen app. > > That code was added in revision e9f6919. > It is mainly a sanity check added under request of one of our users: https://github.com/PyTables/PyTables/pull/146 > This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both > PyInstaller 1.5.1 and 2.0. > > Should I file a bug report? Any easy work-around? > > Thanks, > Stuart > Yes please file a pull request with your patch. It would be nice to preserve the sanity check in standard case so, maybe, a good solution could be adding some check on sys.frozen or something like that. Thank you -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
Hello, I upgraded to PyTables 2.4.0 and I was "freezing" an application on Windows with PyInstaller. The frozen app fails at this new find_library call in __init__.py: if not ctypes.util.find_library('hdf5dll.dll'): raise ImportError('Could not load "hdf5dll.dll", please ensure' + ' that it can be found in the system path') PyInstaller correctly places this DLL in the same directory as the application .exe where standard Windows DLL search logic will find it. Apparently the find_library doesn't do that in a frozen application. That is a big problem. I had to comment this code out to get a working frozen app. That code was added in revision e9f6919. This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both PyInstaller 1.5.1 and 2.0. Should I file a bug report? Any easy work-around? Thanks, Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users