Re: What make function with huge list so slow

2019-08-25 Thread Windson Yang
> Figure out how much memory fib_dp is holding on to right before it returns
> the answer.  fib(40) is a _big_ number!  And so is fib(39), and
> fib(38), and fib(37), etc.  By the time you're done, you're holding
> on to quite a huge pile of storage here.  Depending on how much physical
> memory you have, you much actually be swapping before you're done.
>
> --
> Alan Bawden
> --
>
T
Thank you Alan. You are right! we use too much memeoy to save the answer in
the dp list. Now I get it, thank you so much.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: What make function with huge list so slow

2019-08-24 Thread Windson Yang
'I'm just running them in succession and seeing how long they'. The full
code looks like this, this is only an example.py here. and I run 'time
python3 example.py' for each function.

def fib_dp(n):
dp = [0] * (n+1)
if n <= 1:
return n
dp[0], dp[1] = 0, 1
for i in range(2, n+1):
dp[i] = dp[i-1] + dp[i-2]
return dp[-1]

def fib_dp2(n):
# add dp here just for calculating memory allocation time
dp = [0] * (n+1)
if n <= 1:
return n
pre, now = 0, 1
for i in range(2, (n+1)):
pre, now = now, pre+now
return now


# run the function only once
# fib_dp(40) # took more than 60s
# fib_dp2(40) # took about 2s
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: What make function with huge list so slow

2019-08-24 Thread Windson Yang
Thank you, Chris. I tried your suggestions. I don't think that is the
reason, fib_dp_look() and fib_dp_set() which also allocation a big list can
return in 2s.

Chris Angelico  于2019年8月25日周日 上午11:27写道:

> On Sun, Aug 25, 2019 at 12:56 PM Windson Yang  wrote:
> >
> > I have two functions to calculate Fibonacci numbers. fib_dp use a list to
> > store the calculated number. fib_dp2 just use two variables.
> >
> > def fib_dp(n):
> > if n <= 1:
> > return n
> > dp = [0] * (n+1)
> > dp[0], dp[1] = 0, 1
> > for i in range(2, n+1):
> > dp[i] = dp[i-1] + dp[i-2]
> > return dp[-1]
> >
> > def fib_dp2(n):
> > if n <= 1:
> > return n
> > pre, now = 0, 1
> > for i in range(2, (n+1)):
> > pre, now = now, pre+now
> > return now
> >
> > Theoretically, both of their time complexity should be O(n). However,
> when
> > the input n is so big (like 40), fib_dp2(40) can calculate it in
> 2s
> > but fib_dp(40) takes *more than 60s* (python3.7.3 and macOS 10.14.6).
> > Why?
>
> Memory allocation can take a long time. Try grabbing the line
> initializing dp and slapping that into the body of dp2, and then
> compare their times; that might make all the difference.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


What make function with huge list so slow

2019-08-24 Thread Windson Yang
I have two functions to calculate Fibonacci numbers. fib_dp use a list to
store the calculated number. fib_dp2 just use two variables.

def fib_dp(n):
if n <= 1:
return n
dp = [0] * (n+1)
dp[0], dp[1] = 0, 1
for i in range(2, n+1):
dp[i] = dp[i-1] + dp[i-2]
return dp[-1]

def fib_dp2(n):
if n <= 1:
return n
pre, now = 0, 1
for i in range(2, (n+1)):
pre, now = now, pre+now
return now

Theoretically, both of their time complexity should be O(n). However, when
the input n is so big (like 40), fib_dp2(40) can calculate it in 2s
but fib_dp(40) takes *more than 60s* (python3.7.3 and macOS 10.14.6).
Why?. At first, I guess the reasons are

1. It took too much time looking up the value in the list (the worse case
should be O(1) according to the document). However, the below function
fib_dp_tem(40) can finish in 2s, so looking up value should not be the
bottleneck.

def fib_dp_look(n):
if n <= 1:
return n
dp = [0] * (n+1)
dp[0], dp[1] = 0, 1
for i in range(2, n+1):
# change dp[i] to tem, this function is not correct now but it
return dp[-1] in 2s
tem = dp[i-1] + dp[i-2]
return dp[-1]

2. It took too much time setting up the value in the list (the worse case
should be O(1) according to the document). Again, the below function
fib_dp_set(40) can finish in 2s, so setting value should not be the
bottleneck too.

def fib_dp_set(n):
if n <= 1:
return n
dp = [0] * (n+1)
dp[0], dp[1] = 0, 1
for i in range(2, n+1):
 # this function is not correct now but it return dp[-1] in 2s
dp[i-1] = i
dp[i-2] = i + 1
return dp[-1]

3. python use some kind of cache for 'pre', 'now' variable, (like 'register
variable' in C, but I'm not sure how it work in CPython)

I also tried to use the dis module but with no luck. Any reason to make
fib_dp so much slower than fib_dp2?  Please let me know, thank you.

Bests,

Windson
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How should we use global variables correctly?

2019-08-23 Thread Windson Yang
Thank you all. I agreed with Frank that

> It would make sense to use the 'global' keyword if you have a module
with various functions, several of which refer to 'foo', but only one of
which changes the value of 'foo'.

I also found an example in cpython/lib/gettext.py, only 'textdomain
function' can change '_current_domain', other functions just refer to it.
So, it will be not evil or to use 'global' keyword correctly when there is
only one function can change its value?

Cameron Simpson  于2019年8月23日周五 下午3:15写道:

> On 23Aug2019 09:07, Frank Millman  wrote:
> >On 2019-08-23 8:43 AM, Windson Yang wrote:
> >>In class.py
> >>
> >>  class Example:
> >> def __init__(self):
> >> self.foo = 1
> >> def bar()
> >> return self.foo + 1
> >>
> >>Expect the syntax, why using class variable self.foo would be better (or
> >>more common)? I think the 'global' here is relative, foo is global in
> >>global.py and self.foo is global in Example class. If the global.py is
> >>short and clean enough (didn't have a lot of other class), they are
> pretty
> >>much the same. Or I missed something?
> >>
> >
> >One difference is that you could have many instances of Example, each
> >with its own value of 'foo', whereas with a global 'foo' there can
> >only be one value of 'foo' for the module.
>
> But that is an _instance_ attribute. Which is actually what Windson Yang
> made.
>
> A class attribute is bound to the class, not an instance. The
> terminology is important.
>
> Cheers,
> Cameron Simpson 
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How should we use global variables correctly?

2019-08-22 Thread Windson Yang
I also want to know what is the difference between "using 'global
variables' in a py module" and "using a variable in class". For example:

In global.py:

foo = 1
def bar():
global foo
return foo + 1

In class.py

 class Example:
def __init__(self):
self.foo = 1
def bar()
return self.foo + 1

Expect the syntax, why using class variable self.foo would be better (or
more common)? I think the 'global' here is relative, foo is global in
global.py and self.foo is global in Example class. If the global.py is
short and clean enough (didn't have a lot of other class), they are pretty
much the same. Or I missed something?




Chris Angelico  于2019年8月23日周五 上午9:34写道:

> On Fri, Aug 23, 2019 at 11:24 AM Windson Yang  wrote:
> >
> > Thank you all for the great explanation, I still trying to find some good
> > example to use 'global', In CPython, I found an example use 'global' in
> > cpython/Lib/zipfile.py
> >
> > _crctable = None
> > def _gen_crc(crc):
> > for j in range(8):
> > if crc & 1:
> > crc = (crc >> 1) ^ 0xEDB88320
> > else:
> > crc >>= 1
> > return crc
> >
> > def _ZipDecrypter(pwd):
> > key0 = 305419896
> > key1 = 591751049
> > key2 = 878082192
> >
> > global _crctable
> > if _crctable is None:
> > _crctable = list(map(_gen_crc, range(256)))
> > crctable = _crctable
> >
> > _crctable only been used in the _ZipDecrypter function. IIUC, the code
> can
> > be refactored to
> >
> > def _gen_crc(crc):
> > ...stay the same
> >
> > def _ZipDecrypter(pwd, _crctable=list(map(_gen_crc, range(256:
> > key0 = 305419896
> > key1 = 591751049
> > key2 = 878082192
> >crctable = _crctable
> >
> > Which avoid using 'global' keyword. Why we are not doing this? I guess
> the
> > reason we use 'global' here because we don't want to  create `_crctable =
> > list(map(_gen_crc, range(256)))`  every time when we run '_ZipDecrypter'
> > function. So we kinda cache _crctable with 'global', am I right?
>
> It's a cache that is made ONLY when it's first needed. If you put it
> in the function header, it has to be created eagerly as soon as the
> module is imported.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: How should we use global variables correctly?

2019-08-22 Thread Windson Yang
Thank you all for the great explanation, I still trying to find some good
example to use 'global', In CPython, I found an example use 'global' in
cpython/Lib/zipfile.py

_crctable = None
def _gen_crc(crc):
for j in range(8):
if crc & 1:
crc = (crc >> 1) ^ 0xEDB88320
else:
crc >>= 1
return crc

def _ZipDecrypter(pwd):
key0 = 305419896
key1 = 591751049
key2 = 878082192

global _crctable
if _crctable is None:
_crctable = list(map(_gen_crc, range(256)))
crctable = _crctable

_crctable only been used in the _ZipDecrypter function. IIUC, the code can
be refactored to

def _gen_crc(crc):
...stay the same

def _ZipDecrypter(pwd, _crctable=list(map(_gen_crc, range(256:
key0 = 305419896
key1 = 591751049
key2 = 878082192
   crctable = _crctable

Which avoid using 'global' keyword. Why we are not doing this? I guess the
reason we use 'global' here because we don't want to  create `_crctable =
list(map(_gen_crc, range(256)))`  every time when we run '_ZipDecrypter'
function. So we kinda cache _crctable with 'global', am I right?
-- 
https://mail.python.org/mailman/listinfo/python-list


How should we use global variables correctly?

2019-08-22 Thread Windson Yang
I can 'feel' that global variables are evil. I also read lots of articles
proves that (http://wiki.c2.com/?GlobalVariablesAreBad). However, I found
CPython Lib use quite a lot of `global` keyword. So how should we use
`global` keyword correctly? IIUC, it's fine that we use `global` keyword
inside the Lib since most of the time the user just import the lib and call
the API. In any other situation, we should avoid using it. Am I right?
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: fopen() and open() in cpython

2019-08-14 Thread Windson Yang
Thank you so much for the answer, now it makes sense :D

eryk sun  于2019年8月15日周四 上午12:27写道:

> On 8/13/19, Windson Yang  wrote:
> > After my investigation, I found Since Python maintains its own buffer
> when
> > read/write files, the build-in python open() function will call the
> open()
> > system call instead of calling standard io fopen() for caching.  So when
> we
> > read/write a file in Python, it would not call fopen(), fopen() only use
> > for Python itself but not for python user. Am I correct?
>
> Python 2 I/O wraps C FILE streams (i.e. fopen, fclose, fread, fwrite,
> fgets). Python 3 has its own I/O stack (raw, buffered, text) that aims
> to be more reliably cross-platform than C FILE streams. Python 3 still
> uses FILE streams internally in some cases (e.g. to read pyvenv.cfg at
> startup).
>
> FYI in Windows open() or _wopen() is a C runtime library function, not
> a system function. It calls the Windows API function CreateFile, which
> calls the NT system function, NtCreateFile. It's similarly layered for
> all calls, e.g. read() calls ReadFile or ReadConsoleW, which calls
> NtReadFile or NtDeviceIoControlFile (ReadConsoleW).
>
-- 
https://mail.python.org/mailman/listinfo/python-list


fopen() and open() in cpython

2019-08-13 Thread Windson Yang
After my investigation, I found Since Python maintains its own buffer when
read/write files, the build-in python open() function will call the open()
system call instead of calling standard io fopen() for caching.  So when we
read/write a file in Python, it would not call fopen(), fopen() only use
for Python itself but not for python user. Am I correct?
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Understand workflow about reading and writing files in Python

2019-06-24 Thread Windson Yang
DL Neil  于2019年6月24日周一 上午11:18写道:

> Yes, better to reply to list - others may 'jump in'...
>
>
> On 20/06/19 5:37 PM, Windson Yang wrote:
> > Thank you so much for you review DL Neil, it really helps :D. However,
> > there are some parts still confused me, I replyed as below.
>
> It's not a particularly easy topic...
>
>
> > DL Neil  > <mailto:pythonl...@danceswithmice.info>> 于2019年6月19日周三 下午2:03写道:
> >
> > I've not gone 'back' to refer to any ComSc theory on
> buffer-management.
> > Perhaps you might benefit from such?
> >
> > I just take a crash course on it so I want to know if I understand the
> > details correctly :D
>
> ...there are so many ways one can mess-up!
>
>
> > I like your use of the word "shift", so I'll continue to use it.
> >
> > There are three separate units of data to consider - each of which
> > could
> > be called a "buffer". To avoid confusing (myself) I'll only call the
> > 'middle one' that:
> > 1 the unit of data 'coming' from the data-source
> > 2 the "buffer" you are implementing
> > 3 the unit of data 'going' out to a data-destination.
> >
> > Just to make it clear, when we use `f.write('abc')` in python, (1) means
> > 'abc', (2) means the buffer handle by Python (by default 8kb), (2) means
> > the file *f* we are writing to, right?
>
> Sorry, this is my typo, (3) means the file *f* we are writing to, right?


> No! (sorry) f.write() is an output operation, thus nr3.
>
> "f" is not a "buffer handle" but a "file handle" or more accurately a
> "file object".
>
> When we:
>
> one_input = f.read( NRbytes )
>
> (ignoring EOF/short file and other exceptions) that many bytes will
> 'appear' in our program labelled as "one_input".
>
> However, the OpSys may have read considerably more data, depending upon
> the device(s) involved, the application, etc; eg if we ask for 2 bytes
> the operating system will read a much larger block (or applicable unit)
> of data from a disk drive.
>
> The same applies in reverse, with f.write( NRbytes/byte-object ), until
> we flush or close the file.
>
> Those situations account for nr1 and nr3. In the usual case, we have no
> control over the size of these buffers - and it is best not to meddle!
>
> I agreed with you.

Hence:-
>
> > 1 and 3 may be dictated to you, eg hardware or file specifications,
> > code
> > requirements, etc.
> >
> > So, data is shifted into the (2) buffer in a unit-size decided by
> (1) -
> > in most use-cases each incoming unit will be the same size, but
> > remember
> > that the last 'unit' may/not be full-size. Similarly, data shifted
> out
> > from the (2) buffer to (3).
> >
> > The size of (1) is likely not that of (3) - otherwise why use a
> > "buffer"? The size of (2) must be larger than (1) and larger than
> (2) -
> > for reasons already illustrated.
> >
> > Is this a typo? (2) larger than (1) larger than (2)?
>
> Correct - well spotted! nr2 > nr1 and nr2 > nr3
>

When we run 'f.write(100', I understand why nr2 (by defaut 8kb) > nr1
(100), but I'm not sure why nr2 > nr3 (file object) here?

>
>
> > I recall learning how to use buffers with a series of hand-drawn
> block
> > diagrams. Recommend you try similarly!
>
> Try this!
>
>
> > Now, let's add a few critiques, as requested (interposed below):-
> >
> >
> > On 19/06/19 3:53 PM, Windson Yang wrote:t
> >  > I'm trying to understand the workflow of how Python read/writes
> > data with
> >  > buffer. I will be appreciated if someone can review it.
> >  >
> >  > ### Read n data
> >
> > - may need more than one read operation if the size of (3) "demands"
> > more data than the size of (1)/one "read".
> >
> >
> > Looks like the size of len of one read() depends on
> >
> https://github.com/python/cpython/blob/master/Modules/_io/bufferedio.c#L1655
>  ?
>
>
> You decide how many bytes should be read. That's how much will be
> transferred from the OpSys' I/O into the Python program's space. With
> the major exception, that if there is no (more) data available, it is
> defined as an exception (EOF = end of file) or if there are fewer bytes
> of data

Re: Understand workflow about reading and writing files in Python

2019-06-24 Thread Windson Yang
When you said "C-runtime buffered I/O", are you talking about Standard I/O
in C (FILE * object)? AFAIN, In CPython, we didn't use Standard I/O, right?

Dennis Lee Bieber  于2019年6月25日周二 上午12:48写道:

> On Mon, 24 Jun 2019 15:18:26 +1200, DL Neil
>  declaimed the following:
>
>
> >
> >However, the OpSys may have read considerably more data, depending upon
> >the device(s) involved, the application, etc; eg if we ask for 2 bytes
> >the operating system will read a much larger block (or applicable unit)
> >of data from a disk drive.
> >
>
> Depending upon implementation, there could be layers of buffers
> involved...
>
> Python application requests, say, 50 bytes using a "buffered I/O"
> file
> object. That file object may invoke a C-runtime buffered I/O call that
> requests whatever the C-runtime buffer-size is -- say a 512 byte sector.
> That request goes to a low-level device driver for a file system/device
> that does I/O in 4KB clusters. So the first read results in the OS reading
> 4KB into a buffer, and passing 512 bytes to the C-call, which then returns
> 50 bytes to the Python layer.
>
> The second read for 50 bytes is satisfied from the remaining bytes
> in
> the C-runtime sector buffer. The 11th read of 50 bytes will get 12 bytes
> from the sector, and then the C-runtime has to request the next sector from
> the OS, which is satisfied from the file system cluster buffer. After the
> 8th sector is processed, the next request results in the OS going to disk
> for the next cluster.
>
>
> --
> Wulfraed Dennis Lee Bieber AF6VN
> wlfr...@ix.netcom.com
> http://wlfraed.microdiversity.freeddns.org/
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Fwd: Understand workflow about reading and writing files in Python

2019-06-23 Thread Windson Yang
Thank you so much for you review DL Neil, it really helps :D. However,
there are some parts still confused me, I replyed as below.

DL Neil  于2019年6月19日周三 下午2:03写道:

> I've not gone 'back' to refer to any ComSc theory on buffer-management.
> Perhaps you might benefit from such?
>
> I just take a crash course on it so I want to know if I understand the
details correctly :D


> I like your use of the word "shift", so I'll continue to use it.
>
> There are three separate units of data to consider - each of which could
> be called a "buffer". To avoid confusing (myself) I'll only call the
> 'middle one' that:
> 1 the unit of data 'coming' from the data-source
> 2 the "buffer" you are implementing
> 3 the unit of data 'going' out to a data-destination.
>
> Just to make it clear, when we use `f.write('abc')` in python, (1) means
'abc', (2) means the buffer handle by Python (by default 8kb), (2) means
the file *f* we are writing to, right?

1 and 3 may be dictated to you, eg hardware or file specifications, code
> requirements, etc.
>
> So, data is shifted into the (2) buffer in a unit-size decided by (1) -
> in most use-cases each incoming unit will be the same size, but remember
> that the last 'unit' may/not be full-size. Similarly, data shifted out
> from the (2) buffer to (3).
>
> The size of (1) is likely not that of (3) - otherwise why use a
> "buffer"? The size of (2) must be larger than (1) and larger than (2) -
> for reasons already illustrated.
>

Is this a typo? (2) larger than (1) larger than (2)?

>
> I recall learning how to use buffers with a series of hand-drawn block
> diagrams. Recommend you try similarly!
>
>
> Now, let's add a few critiques, as requested (interposed below):-
>
>
> On 19/06/19 3:53 PM, Windson Yang wrote:t
> > I'm trying to understand the workflow of how Python read/writes data with
> > buffer. I will be appreciated if someone can review it.
> >
> > ### Read n data
>
> - may need more than one read operation if the size of (3) "demands"
> more data than the size of (1)/one "read".
>

Looks like the size of len of one read() depends on
https://github.com/python/cpython/blob/master/Modules/_io/bufferedio.c#L1655
 ?

>
> > 1. If the data already in the buffer, return data
>
> - this a data-transfer of size (3)
>
> For extra credit/an unnecessary complication (but probable speed-up!):
> * if the data-remaining is less than size (3) consider a read-ahead
> mechanism
>
> > 2. If the data not in the buffer:
>
> - if buffer's data-len < size (3)
>
> >  1. copy all the current data from the buffer
>
> * if "buffer" is my (2), then no-op
>

I don't understand your point here, when we read data we would copy some
data from the current buffer from python, right? (
https://github.com/python/cpython/blob/master/Modules/_io/bufferedio.c#L1638),
we use `out` (which point to res) to store the data here.

>
> >  2. create a new buffer object, fill the new buffer with raw read
> which
> > read data from disk.
>
> * this becomes: perform read operation and append incoming data (size
> (1)) to "buffer" - hence why "buffer" is larger than (1), by definition.
> NB if size (1) is smaller than size (3), multiple read operations may be
> necessary. Thus a read-loop!?
>
> Yes, you are right, here is a while loop (
https://github.com/python/cpython/blob/master/Modules/_io/bufferedio.c#L1652
)

>
> >  3. concat the data in the old buffer and new buffer.
>
> = now no-op. Hopefully the description of 'three buffers' removes this
> confusion of/between buffers.
>
>  I don't get it. When we call the function like seek(0) then read(1000),
we can still use the data from buffer from python, right?

>
> >  4. return the data
>
> * make the above steps into a while-loop and there won't be a separate
> step here (it is the existing step 1!)
>
>
> * build all of the above into a function/method, so that the 'mainline'
> only has to say 'give me data'!
>
>
> > ### Write n data
> > 1. If data small enough to fill into the buffer, write data to the buffer
>
> =yes, the data coming from source (1), which in this case is 'your' code
> may/not be sufficient to fill the output size (3). So, load it into the
> "buffer" (2).
>
> > 2. If data can't fill into the buffer
> >  1. flush the data in the buffer
>
> =This statement seems to suggest that if there is already some data in
> the buffer, it will b

Understand workflow about reading and writing files in Python

2019-06-18 Thread Windson Yang
I'm trying to understand the workflow of how Python read/writes data with
buffer. I will be appreciated if someone can review it.

### Read n data
1. If the data already in the buffer, return data
2. If the data not in the buffer:
1. copy all the current data from the buffer
2. create a new buffer object, fill the new buffer with raw read which
read data from disk.
3. concat the data in the old buffer and new buffer.
4. return the data

### Write n data
1. If data small enough to fill into the buffer, write data to the buffer
2. If data can't fill into the buffer
1. flush the data in the buffer
1. If succeed:
1. create a new buffer object.
2. fill the new buffer with data return from raw write
2. If failed:
1. Shifting the buffer to make room for writing data to the
buffer
2. Buffer as much writing data as possible (may raise
BlockingIOError)
2. return the data
-- 
https://mail.python.org/mailman/listinfo/python-list


Understand workflow about reading and writing files in Python

2019-06-17 Thread Windson Yang
I'm trying to understand the workflow of how python read/writes files with
buffer. I drew a diagram for it.  I will be appreciated if someone can
review the diagram :D


[image: 屏幕快照 2019-06-15 下午12.50.57.png]
-- 
https://mail.python.org/mailman/listinfo/python-list


Questions about the IO modules and C-api

2019-06-02 Thread Windson Yang
I have some questions about the IO modules.

1. My script:

f = open('myfile, 'a+b')
f.close()

I added a printf statement at the beginning of _io_open_impl
,
the output is:

_io_open
_io_open
_io_open
_io_open
_io_open

Why this function has been called more than one time? I expect this
function only let us call the `open()` system call once.

2. I'm not familiar with the C, How the c-api like
PyObject_CallMethodObjArgs(self->raw,
_PyIO_str_write, memobj, NULL)

 works?

I guess this function will finally call the `write()` system call but I
don't know why it would work. I found `self->raw` is an empty PyObject and
`_PyIO_str_write` is a global variable which is NULL.  Why an empty
PyObject have a write method? Why we didn't just use `write()` system call
directly?

Thank you so much :D

Best,

Windson
-- 
https://mail.python.org/mailman/listinfo/python-list


write function call _io_BufferedWriter_write_impl twice?

2019-05-29 Thread Windson Yang
My script looks like this:

f = open('myfile', 'a+b')
f.write(b'abcde')

And I also add a `printf` statement in the _io_BufferedWriter_write_impl

function.

static PyObject * _io_BufferedWriter_write_impl(buffered *self,
Py_buffer *buffer)
/*[clinic end generated code: output=7f8d1365759bfc6b
input=dd87dd85fc7f8850]*/
{
printf("call write_impl\n");
PyObject *res = NULL;
Py_ssize_t written, avail, remaining;
Py_off_t offset;
   ...

After I compiled then run my script. I found _io_BufferedWriter_write_impl

had
been called twice which I expected only once. The second time it changed
self->pos to an unexpected value too.
-- 
https://mail.python.org/mailman/listinfo/python-list


Duplicate function in thread_pthread.h

2019-05-27 Thread Windson Yang
When I try to understand the code about the thread. I found
the thread_pthread.h file has some duplicate functions. Like
`PyThread_free_lock`, `PyThread_release_lock`,
`PyThread_acquire_lock_timed`. IIUC, C doesn't support function overload.
So why we have functions with the same name and args?
-- 
https://mail.python.org/mailman/listinfo/python-list


CPython compiled failed in macOS

2019-05-21 Thread Windson Yang
version: macOS 10.14.4, Apple LLVM version 10.0.1 (clang-1001.0.46.4).

I cloned the CPython source code from GitHub then compiled it which used to
work quite well. However, I messed up my terminal a few days ago for
installing gdb. Now when I try to compile the latest CPython source code
with ./configure --with-pydebug && make -j I always get the error messages:

ld: warning: ld: warning: ignoring file libpython3.8d.a, file was
built for archive which is not the architecture being linked (x86_64):
libpython3.8d.aignoring file libpython3.8d.a, file was built for
archive which is not the architecture being linked (x86_64):
libpython3.8d.a
Undefined symbols for architecture x86_64:
  "__Py_UnixMain", referenced from:
  _main in python.o
ld: symbol(s) not found for architecture x86_64Undefined symbols for
architecture x86_64:
  "_PyEval_InitThreads", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  "_PyEval_ReleaseThread", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  "_PyEval_RestoreThread", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  _test_bpo20891 in _testembed.o
  "_PyEval_SaveThread", referenced from:
  _test_bpo20891 in _testembed.o
  "_PyGILState_Check", referenced from:
  _bpo20891_thread in _testembed.o
  "_PyGILState_Ensure", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  _bpo20891_thread in _testembed.o
  "_PyGILState_Release", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  _bpo20891_thread in _testembed.o
  "_PyInterpreterState_GetID", referenced from:
  _print_subinterp in _testembed.o
  "_PyMem_RawFree", referenced from:
  _test_pre_initialization_api in _testembed.o
  "_PyRun_SimpleStringFlags", referenced from:
  _test_pre_initialization_api in _testembed.o
  _test_pre_initialization_sys_options in _testembed.o
  _test_init_main in _testembed.o
  _check_stdio_details in _testembed.o
  _print_subinterp in _testembed.o
  _dump_config in _testembed.o
  "_PySys_AddWarnOption", referenced from:
  _test_pre_initialization_sys_options in _testembed.o
  "_PySys_AddXOption", referenced from:
  _test_pre_initialization_sys_options in _testembed.o
  "_PySys_ResetWarnOptions", referenced from:
  _test_pre_initialization_sys_options in _testembed.o
  "_PyThreadState_Get", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  _print_subinterp in _testembed.o
  "_PyThreadState_Swap", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  "_PyThread_acquire_lock", referenced from:
  _test_bpo20891 in _testembed.o
  "_PyThread_allocate_lock", referenced from:
  _test_bpo20891 in _testembed.o
  "_PyThread_exit_thread", referenced from:
  _bpo20891_thread in _testembed.o
  "_PyThread_free_lock", referenced from:
  _test_bpo20891 in _testembed.o
  "_PyThread_release_lock", referenced from:
  _bpo20891_thread in _testembed.o
  "_PyThread_start_new_thread", referenced from:
  _test_bpo20891 in _testembed.o
  "_Py_BytesWarningFlag", referenced from:
  _test_init_global_config in _testembed.o
  _test_init_from_config in _testembed.o
  _set_all_global_config_variables in _testembed.o
  "_Py_DebugFlag", referenced from:
  _set_all_global_config_variables in _testembed.o
  "_Py_DecodeLocale", referenced from:
  _test_pre_initialization_api in _testembed.o
  "_Py_DontWriteBytecodeFlag", referenced from:
  _test_init_global_config in _testembed.o
  _test_init_from_config in _testembed.o
  _set_all_global_config_variables in _testembed.o
  _check_init_python_config in _testembed.o
  "_Py_EndInterpreter", referenced from:
  _test_repeated_init_and_subinterpreters in _testembed.o
  "_Py_Finalize", referenced from:
  _test_forced_io_encoding in _testembed.o
  _test_repeated_init_and_subinterpreters in _testembed.o
  _test_pre_initialization_api in _testembed.o
  _test_pre_initialization_sys_options in _testembed.o
  _test_initialize_twice in _testembed.o
  _test_initialize_pymain in _testembed.o
  _test_init_default_config in _testembed.o
  ...
  "_Py_FrozenFlag", referenced from:
  _test_init_global_config in _testembed.o
  _test_init_from_config in _testembed.o
  _set_all_global_config_variables in _testembed.o
  _check_init_python_config in _testembed.o
  "_Py_IgnoreEnvironmentFlag", referenced from:
  _test_init_env in _testembed.o
  _test_init_env_dev_mode in _testembed.o
  _test_init_env_dev_mode_alloc in _testembed.o
  _set_all_global_config_variables in _testembed.o
  _check_init_python_config in _testembed.o
  "_Py_Initialize", referenced from:
  _test_forced_io_encoding in _testembed.o
  _test_pre_initialization_api in _testembed.o
  _test_initialize_twice in _testembed.o
  _test