Re: When will os.remove fail?
On 13/03/2017 02:51, Steve D'Aprano wrote: On Mon, 13 Mar 2017 05:45 am, Alain Ketterlin wrote: Steve D'Aprano writes: [...] It seems that os.remove on Linux will force the delete even if the file is read-only or unreadable, provided you own the file. Your permissions on the file do not really matters. It's all about your permissions on the parent directory (removing a file is really modifying the parent dir). Actually, it is even slightly more complicated. Here is an excerpt of the unlink(2) call (which does the job) listing common error cases: Granted, I am a bit behind in the discussion - and I know nothing about how Windows manages this since DOS 3.3 - there it also called unlink(). rm is the command we run. The system call it uses to remove a file is unlink(). unlink() removes the "link" of the name in the directory to the inode and lowers the count of the number of links to the file (a hard link is an additional directory link to the inode). To modify the inode (link counter) you need to own the file or have (super user) elevated privelidges. To remove the directory link you need to have write privilidge on the (special file type) directory. On UNIX/Linux when the link count is zero the inode and the filesystem blocks it provides access to are cleared if no process is holding the file open. This means it is easy to unlink() (not remove!) an open file. The inode is active and can be used "normally". FYI, the classic mktemp() library routine on UNIX would create a file, open (better create) the file, unlink the file, and then return file descriptor. Regardless of how the program ended (crash or normal) the temp file would be cleaned up on exit. I would be guessing - but FAT and/or NTFS may not be written to clean up on a file with no "links" - and this is why Windows behavior seems to be more restrictive than POSIX/LINUX. Thanks for the detailed explanation, and to everyone else who corrected my misunderstanding. -- https://mail.python.org/mailman/listinfo/python-list
Re: best way to ensure './' is at beginning of sys.path?
On 04-Feb-17 02:07, Cameron Simpson wrote: On 03Feb2017 17:21, Wildman wrote: On Sat, 04 Feb 2017 09:25:42 +1100, Cameron Simpson wrote: Also, what you describe with rc.local wouldn't work anyway, even if it had ben what was asked. Of course, you are correct. I don't know where my head was. I think my tongue got in front of my eye teeth and I could not see what I was saying. :-) If anyone is interested the correct way is to add this to /etc/profile (at the bottom): PATH=$PATH:./ export PATH Indeed. But this should usually be accompanied by the advice that this isn't a good idea. Having one's commands at the mercy of whatever directory one is standing in is a recipe for being subverted. While the risk is lessened by having "." at the end of the search path, that just means the attacker (== other user of this system) has to name their trojan horse commands after typing errors, such as the ever popular "gerp" search programme. Even with Python I'd feel it is better to not have "." in the sys.path; I'd be happier with a full path to a preferred source of modules. (Yes, I know Python likes to search the current directory for modules, notmy favourite feature.) This sound like something that could be hidden, read moved, if not removed - to a feature only available if built with something resembling a --dev-build flag (Python3.7 of course, as there may be earlier projects that depend on it - certainly officially deprecate in all other Python3 releases - there never being a Python2.8 a deprecate in Python2.7 is pointless). Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Conventions and requirements for a python module
On 11/10/2016 17:30, Michael Torrie wrote: On 10/11/2016 08:29 AM, Michael Felt wrote: From reading the python source, and other projects I am looking to patch I see that there is often a file __init__.py, sometimes empty (only comments), sometimes not. I have tried looking in what I hope are the "regular" places such as: https://docs.python.org, readthedocs (it took 454 seconds to build something - what did it build, and where do I get it? I was assuming it was "latest documentation" and I even tried the old, no-longer maintained, wiki. A search for __init__.py, except for on https://packaging.python.org/ that talks about a setup command to extract the version from __init__.py - I have not been able to find anything on "standards" for packages that are more than a single .py file. Probably, I am not looking correctly - however, I do hope someone also notices that finding this is not straight forward for a novice in python. Had I lacked curiosity I would have given up, moved on. Instead - this email. Yes all this is in the docs: https://docs.python.org/3/tutorial/modules.html#packages Thanks! Also there are links that show up when searching google for "python packages": http://www.learnpython.org/en/Modules_and_Packages http://docs.python-guide.org/en/latest/writing/structure/#packages And again! Packages are convenient for organizing code. Typically I put code into __init__.py to organize the package's symbols. In other words whatever symbols get imported when the package is imported are defined here. Sometimes I'll import things from sub-modules from within __init__.py and name and organize them. Sometimes submodules are left on their own to be imported separately. Though the notation "from foo import *" is discouraged, if a programmer is wont to do that, the package can limit the damage somewhat by defining the __all__ variable. As I said, probably not looking correctly - looking under the wrong baskets. Tutorials is not where I usually look first - I like them better after I have done some basic reading to confirm and/or correct my expectations. -- https://mail.python.org/mailman/listinfo/python-list
Conventions and requirements for a python module
From reading the python source, and other projects I am looking to patch I see that there is often a file __init__.py, sometimes empty (only comments), sometimes not. I have tried looking in what I hope are the "regular" places such as: https://docs.python.org, readthedocs (it took 454 seconds to build something - what did it build, and where do I get it? I was assuming it was "latest documentation" and I even tried the old, no-longer maintained, wiki. A search for __init__.py, except for on https://packaging.python.org/ that talks about a setup command to extract the version from __init__.py - I have not been able to find anything on "standards" for packages that are more than a single .py file. Probably, I am not looking correctly - however, I do hope someone also notices that finding this is not straight forward for a novice in python. Had I lacked curiosity I would have given up, moved on. Instead - this email. In short, I would rather try to "read the manual" and ask questions when I get stuck than ask "stupid" questions. That only makes me look bad ;) (e.g., in my other thread the idea of using subclasses was made frequently. That was something I was considering, but what they are, how they are specified, what they achieve rather than guessing that it means - intent some related class xxx: lines. re: project setup: when to use _filename.py - when to use filename.py; what is the difference/significance of a sub-directory when "masking" complexity, better implementation details, is the goal (i.e., there is the API and there are the nitty-gritty implementation details). The key thing I have learned is that I must not assume that python is anything like what I already know/knew. Assume python "does it differently" and mistakes maybe fewer. Thanks for your time and pointers (to links, if not direct advice) Michael -- https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
On 04-Oct-16 04:48, eryk sun wrote:
On Mon, Oct 3, 2016 at 9:27 PM, Michael Felt wrote:
int perfstat_subsystem_total(
perfstat_id_t *name,
perfstat_subsystem_total_t *userbuff,
int sizeof_struct,
int desired_number);
...
+79 class cpu_total:
+80 def __init__(self):
+81 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
Thanks for the model below - a lot to think about. Some was expected,
some is a gift!
Python is "a lot to learn". This helps a great deal.
Move loading the library and defining function prototypes to either
the module or class body. Also, don't use dunder names. The convention
for a private attribute is a single underscore. For type safety,
define each function's argtypes (and restype where required, the
default is c_int). For a more Pythonic interface, define a ctypes
errcheck function that encapsulates raising an appropriate exception
when a C call fails. For example:
import ctypes
# If the following types are used generally, define them at
# the module level, else define them in the class namespace
# that uses them.
perfstat = ctypes.CDLL("libperfstat.a(shr_64.o)")
class perfstat_id_t(ctypes.Structure):
pass
IDENTIFIER_LENGTH = 64
class time64_t(ctypes._SimpleCData):
_type_ = ctypes.c_int64._type_
time_t = time64_t
class perfstat_cpu_total_t(ctypes.Structure):
_fields_ = (("ncpus", ctypes.c_int),
("ncpus_cfg", ctypes.c_int),
("description", ctypes.c_char * IDENTIFIER_LENGTH),
("buffer1", ctypes.c_ulonglong * 15),
("lbolt", time_t),
("loadavg", ctypes.c_ulonglong * 3),
("buffer2", ctypes.c_ulonglong * 29),
("ncpus_high", ctypes.c_int),
("puser", ctypes.c_ulonglong),
("psys",ctypes.c_ulonglong),
("pidle", ctypes.c_ulonglong),
("pwait", ctypes.c_ulonglong),
("buffer3", ctypes.c_ulonglong * 12))
def _perfstat_errcheck(result, func, args):
if result == -1:
# get error info and
raise SomeException()
return args
class CPUTotal(object):
# These may be defined here or just referenced here.
_perfstat = perfstat
_perfstat_id_t = perfstat_id_t
_perfstat_cpu_total_t = perfstat_cpu_total_t
_cpu_total = _perfstat.perfstat_cpu_total
_cpu_total.errcheck = _perfstat_errcheck
_cpu_total.argtypes = (
ctypes.POINTER(_perfstat_id_t),
ctypes.POINTER(_perfstat_cpu_total_t),
ctypes.c_int, ctypes.c_int)
def __init__(self):
self._tcpu = self._perfstat_cpu_total_t()
self._cpu_total(None,
ctypes.byref(self._tcpu),
ctypes.sizeof(self._tcpu), 1)
--
https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
Never said thank you - so, thanks! What I need to do was add the .v at the end so I was accessing the value of the structure. Unlilke Linux, AIX - for reasons unknown to all, they have the time_t definition that is specific to the ABI size, at least for these performance libraries that probably originated before 64-bit was a concern. So my guess is that the dual size in the struct, depending on the application size (not the kernel) is to maintain compatibility with 32-bit applications that had been built against/on a 32-bit kernel. So, lucky for me it did not work intiallly - because it made me pause and understand better what I had written. And now the real thankyou for the detail you shared! M On 03-Oct-16 17:53, eryk sun wrote: On Mon, Oct 3, 2016 at 2:35 PM, Michael Felt wrote: On 02-Oct-16 23:44, eryk sun wrote: On Sun, Oct 2, 2016 at 5:50 PM, Michael Felt wrote: b) what I am not understanding - as the basic documentation shows FOO.value as the way to set/get the value of a _field_ You may be surprised when accessing simple-type fields such as c_int and c_char_p. These simple types have getter and setter functions that get called automatically whenever the type is used as a function result, array index, or struct field. For example: OK - so lucky me - it "does not work" like earlier examples because I am referencing, generally, c_ulonglong - and these do not have a automatic getter/setter function? If not, how do I go about making one (preferably without needing to right a "method" for each and every _field_ in a class. No, c_ulonglong is a simple (fundamental) type, which behaves just like other simple types such as c_int or c_char_p. On platform's with a 64-bit long, c_ulonglong is an alias for c_ulong (i.e. type "L"). On the other hand, on 64-bit Windows, c_ulonglong is an unsigned quad word (i.e. type "Q"). All simple types subclass ctypes._SimpleCData and define a `_type_` code such as "c" for c_char. On 64-bit Linux the simple types are defined as follows: ?: c_bool c: c_char z: c_char_p u: c_wchar Z: c_wchar_p P: c_void_p b: c_int8, c_byte B: c_uint8, c_ubyte h: c_int16, c_short H: c_uint16, c_ushort i: c_int32, c_int I: c_uint32, c_uint l: c_int64, c_long, c_longlong, c_ssize_t L: c_uint64, c_ulong, c_ulonglong, c_size_t f: c_float d: c_double g: c_longdouble For convenience, simple types are automatically converted when accessed as a function result, struct field, or array index. As I mentioned previously, the only way around this behavior is to use a subclass. A subclass doesn't get automatically converted because it might define custom methods and attributes that need to be preserved. I'd alias the type instead of defining a struct, e.g. `time_t = c_long`. This preserves automatic conversion of the simple type. The reason for the not using alias is because a) I was trying to be more inline with the text of the include file. I will have to check the sizeof c_long (i.e., sizeof(long) in both 32 and 64-bit modes I don't follow. Currently you wrap a c_int or c_long in a struct when you could just use those types directly. You have to check the pointer size, but then it's up to you what assumptions to make about the target platform's integer sizes. Currently on a 64-bit system you're assuming a Unix-style LP64 data model [1], in which a long is 64-bit. That should be fine if you're writing Unix-specific code that doesn't care about LLP64 Windows systems. Wrapping the type in a struct provides more type safety, but if I wanted that I'd create my own simple subclass. For example, assuming time_t should be a signed integer that's the same size as a pointer: class time_t(ctypes._SimpleCData): if ctypes.sizeof(ctypes.c_void_p) == ctypes.sizeof(ctypes.c_longlong): _type_ = ctypes.c_longlong._type_ elif ctypes.sizeof(ctypes.c_void_p) == ctypes.sizeof(ctypes.c_long): _type_ = ctypes.c_long._type_ elif ctypes.sizeof(ctypes.c_void_p) == ctypes.sizeof(ctypes.c_int): _type_ = ctypes.c_int._type_ # else raise AttributeError for missing _type_ [1]: https://en.wikipedia.org/wiki/64-bit_computing#64-bit_data_models -- https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
On 05-Oct-16 22:29, Emile van Sebille wrote:
Thanks for the reply!
After a shirt coffeebreak - back into the fray - and I found the following:
+76 class cpu_total:
+77 def __init__(self):
+78 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
+79 prototype = CFUNCTYPE(c_int, c_void_p, c_void_p, c_int,
c_int)
error #2 - see below
+80 args = (1, "name", None), (2, "buff", None), (1, "size",
0), (1, "count", 1)
error #1. paramater type 2 (the buffer might be where data is being put,
but for the call, the pointer is INPUT)
+81 cpu_total = prototype(("perfstat_cpu_total",
__perfstat__), args)
+82
+83 buff = perfstat_cpu_total_t()
+84 cpu_total(buff=buff, size=sizeof(buff))
+85
The error #2 is #2, because only after I corrected the inputs did I get
Type Exceptions. I had hoped that c_void_p was a "typeless" type, mainly
the right size, but the ctypes interface is more particular when you use
the CFUNCTYPE() - so now I have my boilerplate for what I hope
can be a tutorial for interfacing - by hand - with complex functions,
i.e., using one class as the data aka typedef defintion, and a second
class to work
on it.
In template form:
class perfstat_xxx_t:
_fields_ = ( ... )
class perfstat_xxx:
def init(self):
_perflib = ctypes.CDLL("libperfstat.a(shr_64.o)") # AIX member
in an archive
_fn_xxx = _perflib.xxx
# ALL AIX perfstat routines have 4 arguments: (name, buff,
sizeof_buff, count)
# the buff is a pointer to where the information will be stored
_fn = CFUNCTYPE(c_int, c_void_p, POINTER(xxx_t), c_int, c_int)
_args = (1, "name", None), (1, "buff", None), (1, "size", 0),
(1, "count", 1)
_xxx = _fn(("xxx", _fn_xxx), _args)
_buff = perfstat_cpu_total_t()
_xxx(buff=_buff, size=sizeof(_buff))
self._v = _buff
So, the next question/test will be to move the _fields_ into the
"second" class definition. I just need to find what "self" attribute
that is.
--
https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
On 02-Oct-16 19:50, Michael Felt wrote:
I am trying to understand the documentation re: ctypes and interfacing
with existing libraries.
I am reading the documentation, and also other sites that have largely
just copied the documentation - as well as the "free chapter" at
O'Reilly (Python Cookbook).
Using O'Reilly and the standard documentation I now have these two
excerpts. The second def works, the first does not.
Someone helping me understand what I am not reading properly in this bit
of documentation - is much appreciated.
from ctypes import c_int, WINFUNCTYPE, windll
from ctypes.wintypes import HWND, LPCSTR, UINT
prototype = WINFUNCTYPE(c_int, HWND, LPCSTR, LPCSTR, UINT)
paramflags = (1, "hwnd", 0), (1, "text", "Hi"), (1, "caption", None), (1,
"flags", 0)
MessageBox = prototype(("MessageBoxA", windll.user32), paramflags)
The MessageBox foreign function can now be called in these ways:
MessageBox()
MessageBox(text="Spam, spam, spam")
MessageBox(flags=2, text="foo bar")
Note that MessageBox() may give two arguments.
My code: note both def init() are meant to do the same thing, just different
call syntax.
+76 class cpu_total:
+77 def __init__(self):
+78 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
+79 prototype = CFUNCTYPE(c_int, c_void_p, c_void_p, c_int, c_int)
+80 args = (1, "name", None), (2, "buff", None), (1, "size", 0), (1,
"count", 1)
+81 cpu_total = prototype(("perfstat_cpu_total", __perfstat__),
args)
+82
+83 buff = perfstat_cpu_total_t()
+84 cpu_total(buff=buff, size=sizeof(buff))
+85
+86 class perfstat_cpu_total:
+87 def __init__(self):
+88 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
+89 cpu_total = __perfstat__.perfstat_cpu_total
+90
+91 _perfstat_cpu_total = perfstat_cpu_total_t()
+92 ret = cpu_total(None,
+93 byref(_perfstat_cpu_total),
+94 sizeof(perfstat_cpu_total_t), 1)
+95 self.boot = _perfstat_cpu_total.lbolt.v / 100
+96 self.ncpus = _perfstat_cpu_total.ncpus
+97 self._v = _perfstat_cpu_total
When I call the second I can print values and even call functions, reload the
_v, etc..
The first class fails with this message:
a = cpu_total()
Traceback (most recent call last):
File "", line 135, in
File "", line 84, in __init__
TypeError: call takes exactly 1 arguments (2 given)
Thanks for your attention.
--
https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
On 02-Oct-16 19:50, Michael Felt wrote:
class perfstat_cpu_total_t(Structure):
"""
typedef struct { /* global cpu information */
int ncpus;/* number of active logical
processors */
int ncpus_cfg; /* number of configured processors */
char description[IDENTIFIER_LENGTH]; /* processor description
(type/official name) */
u_longlong_t buffer1[15]; /* several variables being
skipped for now */
time_t lbolt; /* number of ticks since last
reboot */
u_longlong_t loadavg[3]; /* (1<number of runnables processes during the last
1, 5 and 15 minutes.*/
u_longlong_t buffer2[29]; /* several variables being
skipped for now */
int ncpus_high; /* index of highest processor online */
u_longlong_t puser; /* raw number of physical processor
tics in user mode */
u_longlong_t psys;/* raw number of physical processor
tics in system mode */
u_longlong_t pidle; /* raw number of physical processor
tics idle */
u_longlong_t pwait; /* raw number of physical processor
tics waiting for I/O */
u_longlong_t buffer2[29]; /* several variables being
skipped for now */
} perfstat_cpu_total_t;
"""
_fields_ = [
("ncpus", c_int),
("ncpus_cfg", c_int),
("description", c_char * IDENTIFIER_LENGTH),
("buffer1", c_ulonglong * 15),
("lbolt", time_t),
("loadavg", c_ulonglong * 3),
("buffer2", c_ulonglong * 29),
("ncpus_high", c_int),
("puser", c_ulonglong),
("psys", c_ulonglong),
("pidle", c_ulonglong),
("pwait", c_ulonglong),
("buffer3", c_ulonglong * 12)
]
Perhaps I should explain what I want to accomplish.
a) use ctypes.CDLL to open libperfstat.a(shr_64.o)
b) call a number for functions, among them
The prototype template (in C) is:
The common signature used by all of the global interfaces is as follows:
int perfstat_subsystem_total(
perfstat_id_t *name,
perfstat_subsystem_total_t *userbuff,
int sizeof_struct,
int desired_number);
The usage of the parameters for all of the interfaces is as follows:
perfstat_id_t *name Reserved for future use, should be NULL
perfstat_subsystem_total_t *userbuff A pointer to a memory area with
enough space for the returned structure
int sizeof_struct Should be set to sizeof(perfstat_subsystem_t)
int desired_number Reserved for future use, must be set to 0 or 1
The return value will be -1 in case of errors. Otherwise, the number of
structures copied is returned. This
is always 1.
So, in C the call looks something like this - for cpu_total
perfstat_cpu_total(NULL, (struct perfstat_cpu_total_t *) &xxx,
sizeof(perfstat_cpu_total_t), 1)
I am struggling with the declarations needed to make the call
I have several iterations - the most simple case (no CFUNCTYPE declaration)
gives
+79 class cpu_total:
+80 def __init__(self):
+81 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
+82 xxx = perfstat_cpu_total_t
+83 cpu_total = __perfstat__.perfstat_cpu_total
+84 ret = cpu_total(None, xxx, sizeof(xxx), 1)
+85 print xxx.ncpus
+86 print xxx.lbolt.v
+87
+88 a = cpu_total()
!cat test_1.py | python
Traceback (most recent call last):
File "", line 88, in
File "", line 84, in __init__
ctypes.ArgumentError: argument 2: : Don't know how
to convert parameter 2
I have tried some experiments with CFUNCTYPE - basically
prototype = CFUNCTYPE(c_int, c_void_p, c_int, c_int, c_int)
but I do not understand how (I think some of the details are missing from the
example) how the following works:
I have also tried:
+80 def __init__(self):
+81 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
+82 xxx = perfstat_cpu_total_t
+83 xptr = POINTER(xxx)
+84 x2 = cast(xptr,POINTER(c_void_p))
+85 cpu_total = __perfstat__.perfstat_cpu_total
+86 ret = cpu_total(None, x2, sizeof(xxx), 1)
+87 print xxx.ncpus
+88 print xxx.lbolt.v
+89
+90 a = cpu_total()
+91 print sizeof(a)
!cat test_1.py | python
Traceback (most recent call last):
File "", line 90, in
File "", line 84, in __init__
File "/opt/lib/python2.7/ctypes/__init__.py", line 509, in cast
return _cast(obj, obj, typ)
ctypes.ArgumentError: argument 1: : wrong type
+79 class cpu_total:
+80 def __init__(self):
+81 __perfstat__ = CDLL("libperfstat.a(shr_64.o)")
+82 xxx = perfstat_cpu_total_t
+83 xptr = POINTER(xxx)
+84 cpu_total = __perfstat__.perfstat_cpu_total
Re: working with ctypes and complex data structures
On 03-Oct-16 16:35, Michael Felt wrote:
I'd alias the type instead of defining a struct, e.g. `time_t =
c_long`. This preserves automatic conversion of the simple type.
The reason for the not using alias is because a) I was trying to be
more inline with the text of the include file.
I will have to check the sizeof c_long (i.e., sizeof(long) in both 32
and 64-bit modes
Also, sys.maxsize is based on the platform's size_t type. That's
generally the same size as a pointer, but C doesn't require this.
Instead use sizeof(c_void_p), which will be 8 on a 64-bit platform and
4 on a 32-bit platform.
I had checked this before - I shall have to try a 32-bit python to see
what _ctypes is doing with c_long on AIX - is it 4, or is it 8.
michael@x071:[/data/prj/python/cloud-init/aix-cloud-init-0.7.8.1/new]cat
xxx.c
#include
time_t
lbolt()
{
perfstat_cpu_total_t cpu_totals;
void *p1, *p2;
printf("int_size:%d\n", sizeof(int));
printf("long_size:%d\n", sizeof(long));
printf("timet_size:%d\n", sizeof(time_t));
printf("void_ptr_size:%d\n", sizeof(void *));
printf("lbolt_size:%d\n", sizeof(cpu_totals.lbolt));
p1 = &cpu_totals;
p2 = &cpu_totals.lbolt;
printf("lbolt offset:%d\n", p2 - p1);
}
main()
{
lbolt();
}
michael@x071:[/data/prj/python/cloud-init/aix-cloud-init-0.7.8.1/new]cc
-q32 >
int_size:4
long_size:4
timet_size:4
void_ptr_size:4
lbolt_size:4
lbolt offset:192
michael@x071:[/data/prj/python/cloud-init/aix-cloud-init-0.7.8.1/new]cc
-q64 >
int_size:4
long_size:8
timet_size:8
void_ptr_size:8
lbolt_size:8
lbolt offset:192
I think 4 and 8 because of problems I have with this block when not in
the right "mode", i.e. SIZEOF_LONG changes
#if LONG_BIT != 8 * SIZEOF_LONG
/* 04-Oct-2000 LONG_BIT is apparently (mis)defined as 64 on some recent
* 32-bit platforms using gcc. We try to catch that here at compile-time
* rather than waiting for integer multiplication to trigger bogus
* overflows.
*/
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc
config?)."
#endif
--
https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
On 02-Oct-16 23:44, eryk sun wrote:
On Sun, Oct 2, 2016 at 5:50 PM, Michael Felt wrote:
a) where is documentation on "CField"'s?
It's undocumented.
So I do not feel so bad about not finding anything :)
A CField is a data descriptor that accesses a
struct field with the given type, size, and offset. Like most
descriptors, it's meant to be accessed as an attribute of an instance,
not as an attribute of the type.
When accessed as an attribute of a struct type, the value returned is
the CField descriptor itself. One reason to reference the descriptor
is for its "offset" attribute, since there's no offsetof() in ctypes.
At least in this regard it should be documented.
b) what I am not understanding - as the basic documentation shows FOO.value
as the way to set/get the value of a _field_
You may be surprised when accessing simple-type fields such as c_int
and c_char_p. These simple types have getter and setter functions that
get called automatically whenever the type is used as a function
result, array index, or struct field. For example:
OK - so lucky me - it "does not work" like earlier examples because I am
referencing, generally, c_ulonglong - and these do not have a automatic
getter/setter function? If not, how do I go about making one (preferably
without needing to right a "method" for each and every _field_ in a class.
class Foo(ctypes.Structure):
_fields_ = (('v', ctypes.c_int),)
>>> foo = Foo()
>>> foo.v = 5
>>> foo.v
5
You can use a subclass to avoid the getfunc's automatic conversion. For example;
class my_int(ctypes.c_int):
pass
class Bar(ctypes.Structure):
_fields_ = (('v', my_int),)
>>> bar = Bar()
>>> bar.v = 5
>>> bar.v
>>> bar.v.value
5
I'll try it also with my c_int/c_uint fields - maybe these just work. If
so, again - how do I get a c_ulonglong?
class time_t(Structure):
...
if (maxsize > 2**32):
_fields_ = [("v", c_long)]
else:
_fields_ = [("v", c_int)]
I'd alias the type instead of defining a struct, e.g. `time_t =
c_long`. This preserves automatic conversion of the simple type.
The reason for the not using alias is because a) I was trying to be more
inline with the text of the include file.
I will have to check the sizeof c_long (i.e., sizeof(long) in both 32
and 64-bit modes
Also, sys.maxsize is based on the platform's size_t type. That's
generally the same size as a pointer, but C doesn't require this.
Instead use sizeof(c_void_p), which will be 8 on a 64-bit platform and
4 on a 32-bit platform.
Thanks!
Also, while it's unrelated to your current problem, bear in mind that
int and long are both always 32-bit on Windows. c_int64 is a
cross-platform 64-bit integer.
--
https://mail.python.org/mailman/listinfo/python-list
Re: working with ctypes and complex data structures
On 02-Oct-16 23:44, eryk sun wrote: On Sun, Oct 2, 2016 at 5:50 PM, Michael Felt wrote: > >a) where is documentation on "CField"'s? I will reply more later - just a quick thanks. Not using maxsize will be good, also in a different patch - also specific to AIX. This "thing" I am working is to: a) document stuff (for a novice like myself) b) have a better way to get system low-level statistics, comparable to Linux, where additions have been made to libc to get them. -- https://mail.python.org/mailman/listinfo/python-list
Re: Merecurial and Python-2.7.x, Python-3.Y
On 02/10/2016 19:01, Ned Batchelder wrote: On Saturday, October 1, 2016 at 9:25:57 PM UTC-4, Michael Felt wrote: Finally, I got to where I understood what needed to be done to get both Mercurial built - and the the new SSL requirements met. So, running: # hg clone https://hg.python.org/cpython works. What is the next step to getting Python-2.7 AND Python-3.7 so I can submit patches against both versions and/or other versions? BTW, only very few changes are being accepted for 2.7.x these days, you might want to check ahead of time before putting in much work on that branch. Yes, I understand that - I have been working for months on something - that brings in - imho - something that should have been there years ago (AIX support for ctypes CDLL_LoadLibrary and ctypes.util.find_library) Not new functionality - just the equivalent of what has been customized for Linux, Solaris, Windows and Mac/OS (aka darwin). Hence I mentioned 2.7 and 3.7 - as my patch has been held back even for 3.6. I know it was not, maybe even is not, perfect - but the base functionality has not changed since March (so I am a bit frustrated with the difficulties of getting anything accepted). Part of that is because my patch was not submitted in "hg" form. So I am trying to remove that obstacle. (p.s. My email account settings do not support newsgroups - so someone else will need to forward it, if so desired) --Ned. -- https://mail.python.org/mailman/listinfo/python-list
working with ctypes and complex data structures
I am trying to understand the documentation re: ctypes and interfacing
with existing libraries.
I am reading the documentation, and also other sites that have largely
just copied the documentation - as well as the "free chapter" at
O'Reilly (Python Cookbook).
I am missing anything on CFields.
My test (template) is:
"""Internal ctypes definitions for AIX libperfstat"""
from sys import maxsize
from ctypes import Structure, Union
from ctypes import c_bool, c_char, c_wchar, c_byte, c_ubyte, c_short,
c_ushort
from ctypes import c_int, c_uint, c_long, c_ulong, c_longlong, c_ulonglong
from ctypes import c_float, c_double, c_longdouble
from ctypes import c_char_p, c_wchar_p, c_void_p
from ctypes import sizeof
IDENTIFIER_LENGTH = 64
class time_t(Structure):
"""
#ifndef _TIME_T
#define _TIME_T
#ifdef _LINUX_SOURCE_COMPAT
typedef long inttime_t;
#else /* !_LINUX_SOURCE_COMPAT */
#ifdef __64BIT__
typedef longtime_t;
#else
typedef int time_t;
#endif
#endif /* !_LINUX_SOURCE_COMPAT */
#endif
"""
if (maxsize > 2**32):
_fields_ = [("v", c_long)]
else:
_fields_ = [("v", c_int)]
class perfstat_cpu_total_t(Structure):
"""
typedef struct { /* global cpu information */
int ncpus;/* number of active logical processors */
int ncpus_cfg; /* number of configured processors */
char description[IDENTIFIER_LENGTH]; /* processor description
(type/official name) */
u_longlong_t buffer1[15]; /* several variables being skipped
for now */
time_t lbolt; /* number of ticks since last reboot */
u_longlong_t loadavg[3]; /* (1
Re: Merecurial and Python-2.7.x, Python-3.Y
On 02/10/2016 08:34, Terry Reedy wrote: No. By default, the working directory for a clone is for the default branch. Any clone can be 'updated' to any branch. I have each share clone updated to a different branch, so I never update to a different branch. -- This answers my underlying question - start for the 'base' and then 'update' to a specific branch. Guess I just have to experiment to see what works best for "I am inherently lazy". p.s. The OS is AIX, but for hg I would not think that is a an issue. -- https://mail.python.org/mailman/listinfo/python-list
Merecurial and Python-2.7.x, Python-3.Y
Finally, I got to where I understood what needed to be done to get both Mercurial built - and the the new SSL requirements met. So, running: # hg clone https://hg.python.org/cpython works. What is the next step to getting Python-2.7 AND Python-3.7 so I can submit patches against both versions and/or other versions? Is it going to be a command like in the developer guide (that references Python3.5)? Does this create a new directory, or just undo a lot of things that was just cloned - or was 3.5 the development branch when the guide was last updated? Will I need to clone (as above) several times - one for each version I want to test against - and then run a second command -- OR -- is there an hg clone command to get a specific version (If that is in the guide - my apologies, as I missed it. Thanks for hints and patience! M -- https://mail.python.org/mailman/listinfo/python-list
AIX build and (potentially missing modules
I have been packaging python for AIX - and wanting minimal dependancies I have been ignoring the final messages from make. Personally, I do not see any real harm in the missing *audio "bits", but how terrible are the other missing "bits" for normal python programs? Many thanks for feedback! ... building dbm using ndbm INFO: Can't locate Tcl/Tk libs and/or headers Python build finished, but the necessary bits to build these modules were not found: _bsddb _curses_panel _sqlite3 _tkinter bsddb185 dl gdbm imageoplinuxaudiodev ossaudiodevreadline spwd sunaudiodev -- https://mail.python.org/mailman/listinfo/python-list
Re: (Initial) Questions about sysconfig.*
Thanks for the quick reply. My mailer lost it (rather remembered it
originiated as a reply to something else, and stored it there) - :)
I would reply between the text below - but, mainly thanks for showing
the use of 'inspect' - will be extremely handy as I learn enough to make
a (local) patch.
Michael
p.s. the >>> prompts make it look very colorful in my mail program!
On 2016-03-04 11:22, Peter Otten wrote:
Michael Felt wrote:
First, a simple one:
sysconfig.is_python_build()
Return True if the current Python installation was built from source.
sysconfig.is_python_build()
False
Now, not earth shattering, but I did build this from source - so can
someone help me with understanding why Python says no?
Looking into the source of a self-compilee Python version:
$ python3.6
Python 3.6.0a0 (default:c092148a1b55+, Mar 1 2016, 15:26:05)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
import sysconfig, inspect
print(inspect.getsource(sysconfig.is_python_build))
def is_python_build(check_home=False):
if check_home and _sys_home:
return _is_python_source_dir(_sys_home)
return _is_python_source_dir(_PROJECT_BASE)
print(inspect.getsource(sysconfig._is_python_source_dir))
def _is_python_source_dir(d):
for fn in ("Setup.dist", "Setup.local"):
if os.path.isfile(os.path.join(d, "Modules", fn)):
return True
return False
sysconfig._PROJECT_BASE
'/usr/local/bin'
So this effectively looks for the file /usr/local/bin/Modules/Setup.dist
or /usr/local/bin/Modules/Setup.local which of course fails. You have to
invoke the interpreter in the directory where it was built:
$ /wherever/you/unpacked/and/compiled/python
Python 3.6.0a0 (default:c092148a1b55+, Mar 1 2016, 15:26:05)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
import sysconfig
sysconfig.is_python_build()
True
And a last question (more about the configure phase I expect)
...() returns, among other things:
'LIBRARY_OBJS': '\\',
'MODULE_OBJS': '\\',
'PARSER_HEADERS': '\\',
'PGENSRCS': '\\ \\',
'PYTHON_OBJS': '\\',
'QUICKTESTOPTS': '-l -x test_subprocess test_io test_lib2to3 \\',
'SHLIB_EXT': '".so"',
Why are any of these using '\\' for anything on 'posix'?
Is that data actually used anywhere? I suspect this is a parsing mishap and
these are actually line continuation indicators. In the Makefile I find (e.
g.)
PYTHON_OBJS=\
Python/_warnings.o \
Python/Python-ast.o \
Python/asdl.o \
[snip]
--
https://mail.python.org/mailman/listinfo/python-list
(Initial) Questions about sysconfig.*
First, a simple one:
sysconfig.is_python_build()
Return True if the current Python installation was built from source.
sysconfig.is_python_build()
False
Now, not earth shattering, but I did build this from source - so can someone
help me with understanding why Python says no?
FYI: These are accurate: (reformated for legibility, where needed)
sysconfig.get_paths()
{'platinclude': '/opt/include/python2.7',
'platstdlib': '/opt/lib/python2.7',
'platlib': '/opt/lib/python2.7/site-packages',
'purelib': '/opt/lib/python2.7/site-packages',
'stdlib': '/opt/lib/python2.7',
'scripts': '/opt/bin',
'include': '/opt/include/python2.7',
'data': '/opt'
}
sysconfig.get_platform()
'aix-5.3'
And a last question (more about the configure phase I expect)
...() returns, among other things:
'LIBRARY_OBJS': '\\',
'MODULE_OBJS': '\\',
'PARSER_HEADERS': '\\',
'PGENSRCS': '\\ \\',
'PYTHON_OBJS': '\\',
'QUICKTESTOPTS': '-l -x test_subprocess test_io test_lib2to3 \\',
'SHLIB_EXT': '".so"',
Why are any of these using '\\' for anything on 'posix'?
And, as a DLL object on AIX (in the same sense as linux) is both 'libNAME.so'
when available as a file AND libNAME.a(anything.so) when a MEMBER in an archive.
Or are there variables there I should ignore (on 'posix')?
And, having read a bit - is there a PEP on these variables, meaning, and usage?
Thanks for your assistance!
Michael
--
https://mail.python.org/mailman/listinfo/python-list
