Re: I'd like to use "semantic indentation"
On 30 September 2017 at 21:12, Stefan Ram wrote: > I would like to write source code similar to: > > country( 'USA' ) > state( 'Alabama' ) > town( 'Abbeville' ) > town( 'Addison' ) > state( 'Arizona' ) > town( 'Apache Junction' ) > town( 'Avondale ) > town( 'Benson' ) > > you can't get off quite as clean without sth like macropy.. but you can get close with custom contextmanagers. with country( 'USA' ): with state( 'Alabama' ): town( 'Abbeville' ) town( 'Addison' ) of course you need to write the contextmanager yourself.. and need to decide what the topmost level contextmanager will operate on unless you want to do sth .. .probably quite nasty -- https://mail.python.org/mailman/listinfo/python-list
Re: How to concatenate strings in different lists
On 23 November 2016 at 19:40, Daiyue Weng wrote: > Hi, I am wondering how to concatenate corresponding strings in two lists, > assuming that the lists are of same length, e.g. > > val_1 = ['a', 'b', 'c'] > val_2 = ['A', 'B', 'C'] > > # concatenate corresponding strings in 'val_1' and 'val_2' > # and put results in another list 'val' so that > # val = ['aA', 'bB', 'cC'] > > cheers > -- > https://mail.python.org/mailman/listinfo/python-list one way to do it: val = [''.join(pair) for pair in zip(val_1, val2)] -- https://mail.python.org/mailman/listinfo/python-list
Re: why this code loop forever after a draw a rectangle
On 16 September 2016 at 14:24, meInvent bbird wrote:
> im = img.copy()
> cntcounter = 0
> for cnt in contours:
> epsilon = 0.1*cv2.arcLength(cnt,True)
> approx = cv2.approxPolyDP(cnt,epsilon,True)
> #peri = cv2.arcLength(cnt, True)
> #approx = cv2.approxPolyDP(c, 0.5 * peri, True)
> #print("len(approx)="+str(len(approx)))
> if len(approx) == 4:
> print("approx=" + str(approx))
> cntcounter = cntcounter + 1
> print("here1")
> x,y,w,h = cv2.boundingRect(cnt)
> print("here2")
> while im is None:
> time.sleep(1)
> if im is not None:
> print("here3")
> im = cv2.rectangle(im.copy(), (x,y), (x+w, y+h), (0,255,0), 2)
> #im = cv2.line(im,(x,y),(x+w,y),(0,255,0),2)
> #im = cv2.line(im,(x+w,y),(x+w,y+h),(0,255,0),2)
> #im = cv2.line(im,(x,y+h),(x+w,y+h),(0,255,0),2)
> #im = cv2.line(im,(x,y),(x,y+h),(0,255,0),2)
>
>
> cv2.imwrite(r'C:\Users\tester\Documents\masda.png',im)
> --
> https://mail.python.org/mailman/listinfo/python-list
not sure but.. this bit reads really suspicious:
while im is None:
time.sleep(1)
if im is ever None then how will it ever become not None? unless there
is some other thread at work i can't really see this happening.
and if there is some other thread at work then there is probably some
better solution than sleep()
--
https://mail.python.org/mailman/listinfo/python-list
Re: Alternatives to XML?
On 26 August 2016 at 17:58, Joonas Liik wrote:
> On 26 August 2016 at 16:10, Frank Millman wrote:
>> "Joonas Liik" wrote in message
>> news:cab1gnpqnjdenaa-gzgt0tbcvwjakngd3yroixgyy+mim7fw...@mail.gmail.com...
>>
>>> On 26 August 2016 at 08:22, Frank Millman wrote:
>>> >
>>> > So this is my conversion routine -
>>> >
>>> > lines = string.split('"') # split on attributes
>>> > for pos, line in enumerate(lines):
>>> >if pos%2: # every 2nd line is an attribute
>>> >lines[pos] = line.replace('<', '<').replace('>', '>')
>>> > return '"'.join(lines)
>>> >
>>>
>>> or.. you could just escape all & as & before escaping the > and <,
>>> and do the reverse on decode
>>>
>>
>> Thanks, Joonas, but I have not quite grasped that.
>>
>> Would you mind explaining how it would work?
>>
>> Just to confirm that we are talking about the same thing -
>>
>> This is not allowed - '' [A]
>>
>>>>> import xml.etree.ElementTree as etree
>>>>> x = ''
>>>>> y = etree.fromstring(x)
>>
>> Traceback (most recent call last):
>> File "", line 1, in
>> File
>> "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\xml\etree\ElementTree.py",
>> line 1320, in XML
>>parser.feed(text)
>> xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1,
>> column 17
>>
>> You have to escape it like this - ''
>> [B]
>>
>>>>> x = ''
>>>>> y = etree.fromstring(x)
>>>>> y.find('fld').get('name')
>>
>> ''
>>>>>
>>>>>
>>
>> I want to convert the string from [B] to [A] for editing, and then back to
>> [B] before saving.
>>
>> Thanks
>>
>> Frank
>>
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>
> something like.. (untested)
>
> def escape(untrusted_string):
> ''' Use on the user provided strings to render them inert for storage
> escaping & ensures that the user cant type sth like '>' in
> source and have it magically decode as '>'
> '''
> return untrusted_string.replace("&","&").replace("<",
> "<").replace(">", ">")
>
> def unescape(escaped_string):
> '''Once the user string is retreived from storage use this
> function to restore it to its original form'''
> return escaped_string.replace("<","<").replace(">",
> ">").replace("&", "&")
>
> i should note tho that this example is very ad-hoc, i'm no xml expert
> just know a bit about xml entities.
> if you decide to go this route there are probably some much better
> tested functions out there to escape text for storage in xml
> documents.
you might want to un-wrap that before testing tho.. no idea why my
messages get mutilated like that :(
(sent using gmail, maybe somebody can comment on that?)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Alternatives to XML?
On 26 August 2016 at 16:10, Frank Millman wrote:
> "Joonas Liik" wrote in message
> news:cab1gnpqnjdenaa-gzgt0tbcvwjakngd3yroixgyy+mim7fw...@mail.gmail.com...
>
>> On 26 August 2016 at 08:22, Frank Millman wrote:
>> >
>> > So this is my conversion routine -
>> >
>> > lines = string.split('"') # split on attributes
>> > for pos, line in enumerate(lines):
>> >if pos%2: # every 2nd line is an attribute
>> >lines[pos] = line.replace('<', '<').replace('>', '>')
>> > return '"'.join(lines)
>> >
>>
>> or.. you could just escape all & as & before escaping the > and <,
>> and do the reverse on decode
>>
>
> Thanks, Joonas, but I have not quite grasped that.
>
> Would you mind explaining how it would work?
>
> Just to confirm that we are talking about the same thing -
>
> This is not allowed - '' [A]
>
>>>> import xml.etree.ElementTree as etree
>>>> x = ''
>>>> y = etree.fromstring(x)
>
> Traceback (most recent call last):
> File "", line 1, in
> File
> "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\xml\etree\ElementTree.py",
> line 1320, in XML
>parser.feed(text)
> xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1,
> column 17
>
> You have to escape it like this - ''
> [B]
>
>>>> x = ''
>>>> y = etree.fromstring(x)
>>>> y.find('fld').get('name')
>
> ''
>>>>
>>>>
>
> I want to convert the string from [B] to [A] for editing, and then back to
> [B] before saving.
>
> Thanks
>
> Frank
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
something like.. (untested)
def escape(untrusted_string):
''' Use on the user provided strings to render them inert for storage
escaping & ensures that the user cant type sth like '>' in
source and have it magically decode as '>'
'''
return untrusted_string.replace("&","&").replace("<",
"<").replace(">", ">")
def unescape(escaped_string):
'''Once the user string is retreived from storage use this
function to restore it to its original form'''
return escaped_string.replace("<","<").replace(">",
">").replace("&", "&")
i should note tho that this example is very ad-hoc, i'm no xml expert
just know a bit about xml entities.
if you decide to go this route there are probably some much better
tested functions out there to escape text for storage in xml
documents.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Alternatives to XML?
On 26 August 2016 at 08:22, Frank Millman wrote: > "Peter Otten" wrote in message news:npn25e$s5n$1...@blaine.gmane.org... > > Frank Millman wrote: > >>> As you have to keep the "<", why bother? >> >> >> If you mean why don't I convert the '<' to '<', the answer is that I do >> - I just omitted to say so. However, explicit is better than implicit :-) > > >> Doesn't that make the XML document invalid or changes it in an >> irreversible way? How would you know whether > > > "" > > started out as > > "" > > or > > "" > > ? > > I cheat ;-) > > It is *my* XML, and I know that I only use the offending characters inside > attributes, and attributes are the only place where double-quote marks are > allowed. > > So this is my conversion routine - > > lines = string.split('"') # split on attributes > for pos, line in enumerate(lines): >if pos%2: # every 2nd line is an attribute >lines[pos] = line.replace('<', '<').replace('>', '>') > return '"'.join(lines) > > Frank > > > -- > https://mail.python.org/mailman/listinfo/python-list or.. you could just escape all & as & before escaping the > and <, and do the reverse on decode -- https://mail.python.org/mailman/listinfo/python-list
Re: Compression of random binary data
On 11 July 2016 at 20:52, wrote: > What kind of statistic law or mathematical conjecture or is it even a > physical law is violated by compression of random binary data? > > I only know that Shanon theorised it could not be done, but were there any > proof? Compression relies on some items in the dataset being more frequent than others, if you have some dataset that is completely random it would be hard to compress as most items have very similar number of occurrances. > What is to say that you can not do it if the symbolic representation is > richer than the symbolic represenatation of the dataset. > > Isn't it a fact that the set of squareroots actually depict numbers in a > shorter way than their actual representation. A square root may be smaller numerically than a number but it definitely is not smaller in terms of entropy. lets try to compress the number 2 for instance using square roots. sqrt(2) = 1.4142 the square root actually takes more space in this case even tho it is a smaller number. so having the square root would have negative compression in this case. with some rounding back and forth we can probably get around the fact that sqrt(2) would take an infinite amout of memory to accurately represent but that neccesarily means restricting the values we are possible of encoding. for sqrt(2) to not have worse space consumprion than the number 2 itself we basically have to trow away precision so sqrt(2) ~= 1 now i challenge you to get that 2 back out of that 1.. -- https://mail.python.org/mailman/listinfo/python-list
Re: Proposal: named return values through dict initialization and unpacking
On 26 June 2016 at 18:28, Ari Freund via Python-list wrote: > Thanks everybody. There seems to be a lot of resistance to dict unpacking, > in addition to the problem with my proposed shorthand dict() initialization > syntax pointed out by Steven D'Aprano, so I won't be pursuing this. > -- > https://mail.python.org/mailman/listinfo/python-list something like: a, b, c = my_magic_reordering_func(mydict, ["alpha", "beta", "charlie"]) ..is not too far off, altho it is a little repetitive is is trivial to write in current python and reaches the goal of insulating you from some degree of change in the return value. can also be extended to work with namedtuple for example (which you probably should already be using if you want to return so many arguments that this is a problem, returning several arguments is already returning a tuple under the hood afterall) -- https://mail.python.org/mailman/listinfo/python-list
Re: how to python to use virtual memory?
On 26 June 2016 at 04:47, Ho Yeung Lee wrote: > what is the command or code to write to use virtual memory if i use extra > 20 GB from hard disk as memory, means from 70GB memory to 90GB memory > and left 10GB for file? > > Michael Torrie於 2016年6月25日星期六 UTC+8上午11時00分36秒寫道： >> On 06/24/2016 08:44 PM, Dennis Lee Bieber wrote: >> > I don't know how Linux handles swap disk -- Windows normally sets the >> > swap space to ~2X physical memory (for small RAM -- my 12GB system has a >> > 12GB swap and suggests 18GB). >> >> Linux typically uses a user-set swap partition. The old rule of thumb >> was to make the swap partition 2x the size of RAM. Now, though, for most >> installations with lots of RAM, 1:1 is often used. >> >> However, if the OP's program really requires 70 to 100 GB of space, >> relying on the virtual memory system to do this (64-bit only of course) >> is a mistake. The system will simply thrash itself to death on any OS >> at that level of over-commit. If he has that much data, he needs to >> employ techniques for working with the data directly on disk himself. I >> highly doubt these big data sets that large companies work rely simply >> on the OS to manage it! > > -- > https://mail.python.org/mailman/listinfo/python-list You really should avoid relying on the OS trick of pretending you have more ram than you actually do. this can easily cause several magnitudes worth of slowdown. if you succeed i bet the next post to this list will be titled "why is my program taking forever to run". in stead try to partition the work in to chunks and only keep the truly necessary working set loaded at a time. For example if you load data from an xml or json file it will likely include more data than you need to do your calculations. perhaps it is possible to throw some of it away or at least remove it for the time of the memory intensive step. there are ways to allow your program to use more memory ofc. but this may run in to OS limits (like the aforementioned 32 bit windows issue) and will likely incur heavy performance penalties. you should avoid it if at all possible. even if you do you will likely benefit from adapting your algorithm. -- https://mail.python.org/mailman/listinfo/python-list
Re: Method Chaining
On 18 June 2016 at 23:47, Ethan Furman wrote: > On 06/18/2016 07:05 AM, Joonas Liik wrote: >> >> On 18 June 2016 at 15:04, Pete Forman wrote: > > >>> with obj: >>> .a = 1# equivalent to obj.a = 1 >>> .total = .total + 1 # obj.total = obj.total + 1 >> >> >> the leading dot does not resolve the ambiguity that arises from: >> >> with ob_a: >> with ob_b: >> .attr_c = 42 # which object are we modifying right now? > > > The innermost one. Why would it be anything else? > > -- > ~Ethan~ > > -- > https://mail.python.org/mailman/listinfo/python-list What if ob_b does not have attribute attr_c but ob_a does? This may be simple for a computer to solve - try looking it up on ob_b and fall back to ob_a if ob_b has no such attr.. but it is hard(er) to reason about for a human. You need to know about the inner structure of ob_b. not an issue for the person who writes it at first (unless 2 months have passed ) but a real pain if you are seeing that bit of code for the first time or are not intimately familiar with what ob_b is. Not unsolvable ofc, but makes it easier for obscure bugs to hide. -- https://mail.python.org/mailman/listinfo/python-list
Re: Method Chaining
On 18 June 2016 at 15:04, Pete Forman wrote: > Rustom Mody writes: > >> On Friday, June 17, 2016 at 2:58:19 PM UTC+5:30, Steven D'Aprano wrote: >>> On Fri, 17 Jun 2016 06:13 pm, Ned Batchelder wrote: >>> >>> > To me, it's a toss-up. The chained version is nice in that it >>> > removes the repetition of "g". But the unchained version is more >>> > explicit, and avoids the awkward parenthesis. >>> > >>> > I think I would lean toward the unchained version. Clearly tastes >>> > can differ. >>> >>> Indeed. For what it's worth, I'm ever-so-slightly leaning towards >>> Lawrence's taste here. >> >> More than 'slightly' out here! >> One thing about python OOP that irritates me is the 'self.' clutter. >> With a Pascal/VB style with-statement its naturally taken care of >> >> Yeah I know there is this FAQ: >> https://docs.python.org/2/faq/design.html#why-doesn-t-python-have-a-with-statement-for-attribute-assignments >> >> I consider it bogus if we allow with to mean something like: >> https://msdn.microsoft.com/en-us/library/wc500chb.aspx > > One subtle difference between your two citations is that VB uses a > leading dot. Might that lessening of ambiguity enable a future Python to > allow this? > > class Foo: > def .set(a): # equivalent to def set(self, a): > .a = a# equivalent to self.a = a > > Unless it is in a with statement > > with obj: > .a = 1# equivalent to obj.a = 1 > .total = .total + 1 # obj.total = obj.total + 1 > > -- > Pete Forman > -- > https://mail.python.org/mailman/listinfo/python-list the leading dot does not resolve the ambiguity that arises from: with ob_a: with ob_b: .attr_c = 42 # which object are we modifying right now? also refer to "javascript the bad parts" about all the edege cases that python would surely face. also with is allready used for context managers.. -- https://mail.python.org/mailman/listinfo/python-list
Re: how to extract a variable as parameter which has index using by a for loop?
On 8 June 2016 at 12:20, meInvent bbird wrote: > just extract b[i][0:1] and b[i][1:2] out of for loop > > but i depend on for loop > > def node(mmm, A, B): > H2 = [MM[mmm][A+B] for i in range(len(b))] > return H2 > > node(5, b[i][0:1], b[i][1:2]) > > it is not convenient to disclose in detail > just expect to discuss from the view of programming > > On Wednesday, June 8, 2016 at 4:56:56 PM UTC+8, Steven D'Aprano wrote: >> On Wednesday 08 June 2016 17:31, meInvent bbird wrote: >> >> > b = [str(i)+str(j)+str(k) for i in range(m) for j in range(m) for k in >> > range(m)] >> > b[21][0:1]+b[21][1:2] >> > b[21][1:2]+b[21][2:3] >> > b[21][0:1]+b[21][2:3] >> > >> > >> > originally, >> > >> > mmm = 5 >> > H2 = [MM[mmm][b[i][0:1]+b[i][1:2]] for i in range(len(b))] >> >> This is a mess. I don't understand what you are trying to do. You have these >> variable names that don't mean anything, like "b" and "H2", and others which >> aren't defined, like MM. I don't understand what you are trying to >> accomplish, >> or the purpose of your code. >> >> >> > how to extract b[i][0:1] and b[i][1:2] as parameters? >> >> I don't understand the question. >> >> >> > def node(mmm, b[i][0:1], b[i][1:2]): >> > H2 = [MM[mmm][A+B] for i in range(len(b))] >> > return H2 >> > >> > node(5, b[i][0:1], b[i][1:2]) >> >> Explain what node() is supposed to do, in English. Don't write any code yet. >> What is its purpose? What does it return? What arguments does it need to take >> in order to perform its purpose? >> >> >> -- >> Steve > -- > https://mail.python.org/mailman/listinfo/python-list for a start.. a few things to improve readability.. (as well as perf maybe) b[i][0:1]+b[i][1:2] == b[i][0]+b[i][1] possibly dependant of on the data type of b[i] possibly.. b[i][0]+b[i][1] == b[i][0:2] also.. >> > b[21][0:1]+b[21][1:2] >> > b[21][1:2]+b[21][2:3] >> > b[21][0:1]+b[21][2:3] this is.. really confusing.. usually when you add things you would assign the result to something or use it in another way.. so this code looks like it is totally useless. it might not be 100% useless but if it is.. it is such horrible code u rly need to burn it with fire . since u use b[21] 6 times here you should just extract it as a separate variable eg.. b21 = b[21] A = b21[0:1]+b21[1:2] B = b21[1:2]+b21[2:3] C = b21[0:1]+b21[2:3] that is ofc if now this is rly interesting.. i think there is a big misunderstanding about what function arguments are.. def node(mmm, b[i][0:1], b[i][1:2]): H2 = [MM[mmm][A+B] for i in range(len(b))] return H2 node(5, b[i][0:1], b[i][1:2]) if i can guess your intent correctly.. what you actually want is sth like def node(mmm, b01, b12): # b01 = b[i][0:1]; b12 = b[i][1:2] # since you obviously omitted some code you might want to take in some other params as well.. like the variable ´b´ for example. # i hope this is at least a little helpful, it really is hard to give constructive feedback if you cant even post the actual code nor can you give any info about what you are attempting to do H2 = [MM[mmm][A+B] for i in range(len(b))] return H2 node(5, b[i][0:1], b[i][1:2]) also it helps to give meaningful names to things, i guarantee next week even you will be having trouble reading this code... /now what the heck was that MM thing.. i wonder.. i could totally fix it if i only named it sth decent../ -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3 sort() problem
> > I know that sort() returns None, but I guess that it would be returned x > that was sorted. Why so? if it returned a sorted list it might lead some people to believe it did not modify the oridinal list which would lead to a ton of confusion for new users. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most pythonic way of rotating a circular list to a canonical point
I have this feeling that you would get a lot more useful anwsers if you were to describe your actual problem in stead of what you think the solution is. There might be other, better solutions but since we know so little about what you are doing we will likely never find them by just guessing.. -- https://mail.python.org/mailman/listinfo/python-list
Re: Possibly Pythonic Tail Call Optimization (TCO/TRE)
On 16 July 2015 at 21:58, Steven D'Aprano wrote: > On Fri, 17 Jul 2015 03:34 am, Joonas Liik wrote: > >> Now i admit that it is possible to have infinite recursion but it is >> also possiblew to have infinite loops. and we don't kill your code >> after 1000 iterations of a while loop so why should we treat recursion >> any differently? > > Because a while loop which repeats to many times is annoying but harmless, > but a function that recurses too many times will blow up the stack and > cause a seg fault, possibly executing arbitrary memory as code. You want > malware and viruses to take over your system? That's how you get malware > and viruses to take over your system. That's just a buggy implementation, there are ways to extend the stack that nears its capacity, safely. > >> Having a user defined maximum stack limit might be a good idea, eg if >> my stack takes up 100MB its prolly broke, but it should be my duty as >> a programmer to specify such a limit, not have it inflicted upon me >> (worse, in a manner that cannot be changed!). > > You mean sys.getrecursionlimit() and sys.setrecursionlimit()? > ... and that buggy implementation means that when you sys.setrecursionlimit() you will overflow the stack and crash because the recursion limit is an aritificial safeguard and the underlying c buffer is not adjusted accordingly or at least so it would seem. https://docs.python.org/2/library/sys.html#sys.setrecursionlimit so as per the docs the programmer has no real control over how much stack his program can have. all you can say is "let me ignore the safeguard a little longer, i hope i wont crash the program" that is not the same as "can i please have a stack depth of 2.. You are giving the programmer a choice between "run out of stack and crash" and "mutilate interpreter internals and crash or zero out the hard drive" this is not a real choice.. -- https://mail.python.org/mailman/listinfo/python-list
Re: Possibly Pythonic Tail Call Optimization (TCO/TRE)
On 16 July 2015 at 20:49, Chris Angelico wrote: > > This sounds like a denial-of-service attack. If you can state that no > reasonable document will ever have more than 100 levels of nesting, > then you can equally state that cutting the parser off with a tidy > exception if it exceeds 100 levels is safe. > This particular example does have that kind of smell.. my bad for being careless with examples. what if its not a ddos tho, maybe its just strange data? how about you run some genetic algorithm to optimise something, and you store a log of your progress as a tree structure. then you have a script to traverse that tree for some reason, maybe to analyse the history and optimise the parameters of the search in the future. when the problem is complex it might well require thousands of steps to get to the desired outcome.. but over time the log grows longer and longer. at first as the script is written it's probably tested on some rather small logs, but they grow over time so eventually your program will implode on completely reasonable input. notice that the tree grows at a constant rate with generations rather than ~log(n) so it will not reach a natural limit other than finding a satisfying solution. whether that limit be 1k or 8k there is no single limit that is reasonable for all use cases and the range you can vary it is rather restricted.. (notice: this example has the issue with the genetic algorithm being potentially expensive to run so it might not reach the 1k limit, but that does not mean there are not other problems that share this property. what I want to convey here is that not all tree traversal has a natural depth limit and there are cases where it will be hit even with completely natural inputs) > > Partly because infinite recursion requires infinite storage too. If it > truly is tail calls, then you can turn it into a while loop and not > have the limit; otherwise, you run the risk of blowing out your RAM. A valid argument. tho 100MB stack space vs infinite stack space is still very much distinct. For a long running program it might not even be a big issue if some of the stack (the very bottom) is swapped to disk as the top will be nice and warm in the cache. and yes such a program is not exactly common but just because it uses a lot of memory does not mean it has "frozen". it is the job of the programmer to say how much heap his program can use and it is also his job to say how much stack space is acceptable. also: def much_memory_1(str): return munch_memory_1(str+"munch much memory!") much_memory_1(str) --vs-- def much_memory_2(str): tmp = str[:] while True: tmp +="munch much memory!" return tmp # will never reach this much_memory_2(str) >> Having a user defined maximum stack limit might be a good idea, eg if >> my stack takes up 100MB its prolly broke, but it should be my duty as >> a programmer to specify such a limit, not have it inflicted upon me >> (worse, in a manner that cannot be changed!). > > Actually, it is up to you. There's a default, but you can change it. > def recurse(n): return n and recurse(n-1) > ... recurse(998) > 0 recurse(999) > (throws RuntimeError) sys.getrecursionlimit() > 1000 sys.setrecursionlimit(5) recurse(5) > Traceback (most recent call last): > File "", line 1, in > File "", line 1, in recurse > File "", line 1, in recurse > File "", line 1, in recurse > File "", line 1, in recurse > RuntimeError: maximum recursion depth exceeded sys.setrecursionlimit(5000) recurse(5000) > (throws RuntimeError with a gigantic traceback) sys.setrecursionlimit(5) recurse(5) > Segmentation fault ..as i recall reading from a certain stackoverflow post the limit was about 8000 and possibly varying.. -- https://mail.python.org/mailman/listinfo/python-list
Re: Possibly Pythonic Tail Call Optimization (TCO/TRE)
On 16 July 2015 at 20:03, Chris Angelico wrote: > > The trouble with that is that it can quickly run you out memory when > you accidentally trigger infinite recursion. A classic example is a > simple wrapper function... > > def print(msg): > print(ctime()+" "+msg) > > With the recursion limit at its default of 1000, this can't do more > than 1000 iterations before the system detects that it's stuck. With > an infinite stack, this could destroy your system memory and then > finally terminate with MemoryError... if you're lucky. If you're not, > the whole system could get wedged before this gets killed. (Imagine, > for instance, a computer with a large hard disk devoted to virtual > memory. Long before that's exhausted, the system will be practically > unusable, because every little operation will require going back to > the disk.) > That all sounds reasonable. However that can be looked another way. Soppose you have some code that traverses some tree, a strange imbalanced tree (say from some xml) It is, semantically at least, a reasonable aproach to process such a structure with some recursive function. Lets say we have a function that counts all elements in a document for example. and recursively traverses the element tree. Because most xml is relatively flat (let's assume its rare to find more than 100 levels of nesting say) this function would perform perfectly well for most cases. however if some guy sent you an xml document with 1000 levels of nesting your program would crash. Now suddenly you have perfectly good functioning code that randomly crashes. because of some arbitrary limit. it most distinctly reminds me of a certain programming language that kills your thread after 3 operations because you are obviously-in-an-infinite-loop. it leaves a very bad taste in my mouth. That 30k limit (much less lines of source code ofc) is the reason you need nasty hacks to do stuff like implement BigInteger. That 1k stack limit is the limit you cant use perfectly good code just because your input data has some weird quirk. This puts python on par with jass2 and this deeply saddens me. Now i admit that it is possible to have infinite recursion but it is also possiblew to have infinite loops. and we don't kill your code after 1000 iterations of a while loop so why should we treat recursion any differently? Having a user defined maximum stack limit might be a good idea, eg if my stack takes up 100MB its prolly broke, but it should be my duty as a programmer to specify such a limit, not have it inflicted upon me (worse, in a manner that cannot be changed!). -- https://mail.python.org/mailman/listinfo/python-list
Re: Possibly Pythonic Tail Call Optimization (TCO/TRE)
Wouldn't it be possible to have like a dynamically sized stack so that you can grow it endlessly with some acceptable overhead.. That would pretty much take care of the stack-overflow argument without many painful side effects on the semantics at least.. -- https://mail.python.org/mailman/listinfo/python-list
Re: windows and file names > 256 bytes
It sounds to me more like it is possible to use long file names on windows but it is a pain and in python, on windows it is basically impossible. So shouldn't it be possible to maniulate these files with extended names.. I mean even if you had to use some special function to ask for long names it would still be better than no support at all. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pure Python Data Mangling or Encrypting
Personally, i have had AVG give at least 2 false positives (fyi one of them was like python2.6) as long as antivirus software can give so many false positives i would thing preventing your AV from nuking someone elses data is a reasonable thing. -- https://mail.python.org/mailman/listinfo/python-list
Re: JSON Object to CSV File Troubleshooting
On 21 June 2015 at 17:38, Sahlusar wrote:
>
> [snip]
> I do agree with you Denis that this is an unconventional approach. I was
> wondering then that perhaps I should add additional functionality at the XML
> to JSON step? So far, with JSON objects without nested lists (as values) I
> have been successful with this (the following is rather lengthy):
> [snip]
> ##JSON sample:
>
> data2 = {
> "OTF": "0",
> "F": "False",
> "F": {
> "Int32": ["0",
> "0",
> "0",
> "0"]
> },
> [snip]
> "PBDS": {
> "DateTime": ["1/1/0001 12:00:00 AM",
> "1/1/0001 12:00:00 AM",
> "1/1/0001 12:00:00 AM",
> "1/1/0001 12:00:00 AM"]
> },
> "PBDS": {
> "Double": ["0",
> "0",
> "0"]
> },
> "SCS": {
> "String": ["1",
> "2"]
> }
> }
>
> The result:
and compare those closely now
>
> {'D_B': ['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0'],
> 'F_Int32': ['0',
> '0',
> '0',
> '0'],
> 'OTF': '0',
> 'PBDS_Double': ['0', '0', '0', '0', '0', '0', '0', '0'],
> 'SCS_String': ['1', '2']}
>
Notice in the original text you have 2 entries under the name F and
later 2 entiries under the name PBDS. in the result you are missing
the first entry of each.
you say you have succeeded in generating json, unless you meant to
throw away huge swafts of data i would say... nope..
[snip]
>
> I know that this is alot of sequential steps. I am wondering if I could
> insert or conditionally pass these functions when originally parsing the XML,
> so that the JSON is formatted for more recursive reading of the JSON
> dictionary and then writing to CSV? I welcome constructive feedback for
> refactoring
theres things you could do to fix up the generated json .. tho really,
stop generating json when you need to generate csv.
you are winning nothing. you are losing.. well pretty much .. a little
of everything .. by doing this
there are fundemental properties of xml and json you fail to grasp,
you are touting code claiming that it works when the output it
produces is horribly deformed :(
In xml for instance this is valid:
1
.. and so is this:
1
2
a naive translatio n of the first might yield
{"a":
{"b":1}
}
but this will not work with the second example, it would emit
{"a":
{"b":1,"b":2}
}
which really means
{"a":
{"b":2}
}
if you insist on emitting json as an intermediate step you need to
take care of these inconsistencies somehow.
you need to decide which behaviour you want and be explicit about it.
is it desireable that the last entry overrites the previous one? (you
have this now, i doubt this is what you want)
would you like some mergine behaviour? (some config file might work
well with this, or not)
would you like to have every entry be a list? (this is simple, but you
will end up with a lot of junk like {a:[{b:[1]}]}
do you wrap some things in list but not others?
and the conversion from json to CSV has similar issues ofc.
--
https://mail.python.org/mailman/listinfo/python-list
Re: JSON Object to CSV Question
this.. might not throw an eror, but you have 2 keys with the same name
"F", and 1 of them will probably be disgarded..., you have data
corruption even before you try to process it.
{
"F": "False",
"F": {
"Int32": ["0",
"0",
"0"]
},
}
you mentioned Excel at one point.
perhaps you could mock up what you'd like your finished data to look
like in a spreadsheet (google docs for instance, since thats easy to
link to) and reference there.
just having a list of headers doesnt say much about the data format you want.
"client wants csv" hmm..they want "csv" or they want "csv that fists
this very particular description that fits our special decoder or the
like" ?
do you know how the client will use this data. could that info be used
to simplify the output to some degree?
and finally..
the client gives you malformed xml??
I'm very sorry to hear that. also does the client know they are
emitting invalid xml?
is it rly xml?
is it valid in another language? (html is much more lenient for
instance, an html parser might be able to gleam more meaning)
by what definition is it malformed? is it outright structuralyl
broken, does it fail to meet some schema?
does it fail to meet some expectation you have
for some reason ("client said it has these properties")
you also mentioned you use JSON because it maps nicely to python
dicts.. this is true ofc.. but why not just read that in to a python
dict in the first place?
> DB1 : 0, DB2: 0, DB3: 0 etc. and F1: 0, F1: 0. DB1, DB2 would be the headers
> and the 0s as values in the CSV file.
DB1 etc seems ok at first glance however...
say there are 2 nested items and each of them have a DB property which
is an array, you will have name collisions.
you need more thought in to naming the headers at the very least.
if this is meant for a spreadsheet.. then you will end up with 2 very
very very long rows, it will NOT be readable by any stretch of the
imagination.. do you want this.
i'm afraid you'll essentially end up with a translation that looks sth like
{A:{b:"c",q:"w"}}
===
"A.b", "A.q"
"c", "w"
if you just want key-value pairs there are better options out
there..besides csv..
--
https://mail.python.org/mailman/listinfo/python-list
Re: JSON Object to CSV Question
You say you are taking this from an xml file and want to get a CSV file.. Why are you making an intermediate JSON file? Why do you need the CSV output? Could you perhaps be better off using another format? Your data seems to be a quite deeply nested hierarchical structure and doesn't seem to suit the simple CSV format very well.. (i see several layers of nested arrays for one).. how to you plan to map these nested structures to the CSV format? And your example json bit.. the JSON you posted in your last post seems quite different from some of the first ones you posted. You cant expect to give people 5% of a malformed simplified example and to get anything useful (much less usable) back. And finally, the original xml? mock at least? -- https://mail.python.org/mailman/listinfo/python-list
Re: [RELEASED] Python 3.5.0b2 is now available
Perhaps its just me, but it seems to me that this release is mighty picky about annotations. More specifically everything is fine in the interactive interpreter but the same code won't fly when run as a file. example: def some_function(my_arg:"my random annotation")->"my random return type annotation": pass ..will fail with a syntax error when run from a file. -- https://mail.python.org/mailman/listinfo/python-list
Re: Multiple thread program problem
You think "(f)" makes a tuple, but it does not. the parentesis is not the tuple constructor, the comma is try: t=thread.start_new_thread(proc,(f,)) -- https://mail.python.org/mailman/listinfo/python-list
Re: Please help on this sorted function
my_dict = {1: 'D', 2: 'B', 3: 'A', 4: 'E', 5: 'B'}
# dict.items() returns an iterator that returns pairs of (key, value) pairs
# the key argument to sorted tells sorted what to sort by,
operator.itemgetter is a factory function , itemgetter(1)== lambda
iterable: iterable[1]
sorted_dict = sorted(my_dict.items(), key=itemgetter(1))
# at this moment sorted dict is a generator of key-value tuples in the
right order
sorted_dict = OrderedDict(sorted_dict) # turn the generator in to an actual
dict.
# notice: regular dicts are NOT ORDERED, you need a special type of dict to
preserve the order, hence OrderedDict
--
https://mail.python.org/mailman/listinfo/python-list
Re: Please help on this sorted function
>>> ff=sorted({1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'})
>>> ff
[1, 2, 3, 4, 5]
sorted({1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'}) is equivalent to
sorted(iter({1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'}))
and iter(dict) iterates over the dict keys, so when you do
iter({1: 'D', 2: 'B', 3: 'B', 4: 'E', 5: 'A'}) you essentially get
[1,2,3,4,5]
and sorted([1,2,3,4,5]) returns [1,2,3,4,5]
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python is not bad ;-)
Balancing of trees is kind of irrelevant when "tree" means "search space" no? And you definitely dont need to keep the entire tree in memory at the same time. By cropping unsuitable branches early (and not keeping the entire tree in memory) it is quite easy to have more than 1000 of call stack and still have reasonable preformance. (some/many nodes have 0 or 1 children) Also should not-running-out-of-call-stack really be the main reason to balance trees? That sounds like an optimisation to me .. On 2 May 2015 at 18:45, Ian Kelly wrote: > On Sat, May 2, 2015 at 5:42 AM, Marko Rauhamaa wrote: > > Christian Gollwitzer : > > > >> That's why I still think it is a microoptimization, which helps only > >> in some specific cases. > > > > It isn't done for performance. It's done to avoid a stack overflow > > exception. > > If your tree is balanced, then the number of items you would need to > have to get a stack overflow exception would be approximately 2 ** > 1000, which you can't possibly hope to fit into memory. > > If your tree is unbalanced and you're getting a stack overflow > exception, then maybe you should think about balancing it. > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Python is not bad ;-)
I agree, stack overflow is literally the main issue that ive run in to (tree traversal) I've yet to refactor recursion to iterative for speed, but i have done so to avoid hitting the stack size limit. Tree traversal and similar problems in particular lend themselves well to recursion and are not quite as trivial to do in an iterative fashion. I've also found myself constructing an ad-hoc trampoline at least once just to sneak past the stack limit. (probably very evil but it worked and it wasn't nearly as ugly as it sounds like so ill live with it..) On 2 May 2015 at 14:42, Marko Rauhamaa wrote: > Christian Gollwitzer : > > > That's why I still think it is a microoptimization, which helps only > > in some specific cases. > > It isn't done for performance. It's done to avoid a stack overflow > exception. > > > Marko > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
