Re: ANN: Sarge, a library wrapping the subprocess module, has been released.

[Anh Hai Trinh]

> Objection! Does the defense REALLY expect this court to believe that
> he can testify as to how MOST members of the Python community would or
> would not favor bash over Python? And IF they do in fact prefer bash,
> is this display of haughty arrogance nothing more than a hastily
> stuffed straw-man presented to protect his own ego?

Double objection! Relevance. The point is that the OP created another language 
that is neither Python nor Bash.

> And why do you need to voice such strong opinions of disdain in an
> announcement thread? Testing the integrity of a module (or module)
> author is fine so long as we are respectful whilst doing so. However,
> i must take exception with your crass attitude.

My respectful opinion is that the OP's approach is fundamentally flawed. There 
are many platform-specific issues when forking and threading are fused. My 
benign intent was to warn others about unsolved problems and 
scratching-your-head situations.

Obviously, the OP can always choose to continue his direction at his own 
discretion.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: Sarge, a library wrapping the subprocess module, has been released.

[Anh Hai Trinh]

On Monday, February 13, 2012 3:13:17 AM UTC+7, Vinay Sajip wrote:
> On Feb 12, 3:35 pm, Anh Hai Trinh  wrote:
> 
> > I think most users like to use Python, or they'd use Bash. I think people 
> > prefer not another language that is different from both, and having little 
> > benefits. My own opinion of course.
> >
> 
> I have looked at pbs and clom: they Pythonify calls to external
> programs by making spawning those look like function calls. There's
> nothing wrong with that, it's just a matter of taste. I find that e.g.
> 
> wc(ls("/etc", "-1"), "-l")
> 
> is not as readable as
> 
> call(“ls /etc –1 | wc –l”)

I don't disagree with it. But the solution is really easy, just call 'sh' and 
pass it a string!

>>> from extproc import sh
>>> n = int(sh(“ls /etc –1 | wc –l”))

No parser needed written!

Yes there is a danger of argument parsing and globs and all that. But people 
are aware of it. With string parsing, ambiguity is always there. Even when you 
have a BNF grammar, people easily make mistakes.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: Sarge, a library wrapping the subprocess module, has been released.

[Anh Hai Trinh]

> For a careful impl of fork-exec with threads, see 
> http://golang.org/src/pkg/syscall/exec_unix.go

I forgot to mention that this impl is indeed "correct" only because you cannot 
start thread or call fork() directly in the Go language, other than use 
goroutines and the ForkExec() function implemented there. So all that locking 
is internal.

If you use threads and call fork(), you'll almost guaranteed to face with 
deadlocks. Perhaps not in a particular piece of code, but some others. Perhaps 
not on your laptop, but on the production machine with different kernels. Like 
most race conditions, they will eventually show up.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: Sarge, a library wrapping the subprocess module, has been released.

[Anh Hai Trinh]

> It's not hard for the user

I think most users like to use Python, or they'd use Bash. I think people 
prefer not another language that is different from both, and having little 
benefits. My own opinion of course.

Re. threads & fork(): 
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

For a careful impl of fork-exec with threads, see 
http://golang.org/src/pkg/syscall/exec_unix.go

> By that token, disasters await if you ever use threads, unless you know what 
> you're doing

So don't, this package is mainly a fork-exec-wait library providing shell-like 
functionalities. Just use fork().

>  BTW extproc is nice, but I wanted to push the envelope a little :-)

Hmm, if the extra "envelop" is the async code with threads that may deadlock, I 
would say "thanks but no thanks" :p

I do think that IO redirection is much nicer with extproc.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: Sarge, a library wrapping the subprocess module, has been released.

[Anh Hai Trinh]

Having written something with similar purpose (https://github.com/aht/extproc), 
here are my comments:

* Having command parsed from a string is complicated. Why not just have an OOP 
API to construct commands? extproc does this, but you opted to write a 
recursive descent parser. I'm sure it's fun but I think simple is better than 
complex. Most users would prefer not to deal with Python, not another language.

* Using threads and fork()ing process does not play nice together unless 
extreme care is taken. Disasters await. For a shell-like library, I would 
recommend its users to never use threads (so that those who do otherwise know 
what they are in for).
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Writing an assembler in Python

[Anh Hai Trinh]

On Feb 23, 10:08 am, Lawrence D'Oliveiro  wrote:
>
> Let me suggest an alternative approach: use Python itself as the assembler.
> Call routines in your library to output the code. That way you have a
> language more powerful than any assembler.
>
> See  for an example.

SyntaxError: Non-matching "#end if" in crosscode8.py:345
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.

[Anh Hai Trinh]

On Feb 23, 1:03 pm, "Alf P. Steinbach"  wrote:
>
> Uhm, Paganini...
>
> As I understand it he invented the "destroy your instruments on stage". :-)
>
> Cheers,
>
> - Alf (off-topic)

You probably meant Franz Liszt, who regularly broke piano strings.
Paganini was also a "rock-star" virtuoso but he did not destroy any
Guarnerius or Stradivarius violins in his possession (at least not to
anyone's knowledge :)

As for functional programming, different people take it to mean
different things. For some, simply using first-class functions
qualifies as functional programming.  Others require their functions
to be pure so that their call graphs can be automatically reduced and
their results can be lazily evaluated.  If you takes the former view,
most Python programmers already do functional programming :p

--aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.

[Anh Hai Trinh]

On Feb 19, 1:44 pm, Steve Howell  wrote:
>
> > def coroutine(co):
> >    def _inner(*args, **kwargs):
> >        gen = co(*args, **kwargs)
> >        gen.next()
> >        return gen
> >    return _inner
>
> > def squares_and_cubes(lst, target):
> >    for n in lst:
> >        target.send((n * n, n * n * n))
>
> > @coroutine
> > def reject_bad_values(target):
> >    while True:
> >        square, cube = (yield)
> >        if not (square == 25 or cube == 64):
> >            target.send((square, cube))
>
> > @coroutine
> > def cubes_only(target):
> >    while True:
> >        square, cube = (yield)
> >        target.send(cube)
>
> > @coroutine
> > def print_results():
> >    while True:
> >        print (yield)
>
> > squares_and_cubes(range(10),
> >        reject_bad_values(
> >            cubes_only(
> >                print_results()
> >                )
> >            )
> >        )
>
> Wow!  It took me a while to get my head around it, but that's pretty
> cool.


This pipeline idea has actually been implemented further, see .

  from stream import map, filter, cut
  range(10) >> map(lambda x: [x**2, x**3]) >> filter(lambda t: t[0]!
=25 and t[1]!=64) >> cut[1] >> list
  [0, 1, 8, 27, 216, 343, 512, 729]


--
aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Overcoming python performance penalty for multicore CPU

[Anh Hai Trinh]

On Feb 4, 10:46 am, John Nagle  wrote:
>
>     There's enough intercommunication between the threads working on
> a single site that it's a pain to do them as subprocesses. And I
> definitely don't want to launch subprocesses for each page; the
> Python load time would be worse than the actual work.  The
> subprocess module assumes you're willing to launch a subprocess
> for each transaction.

You could perhaps use a process pool inside each domain worker to work
on the pages?  There is multiprocessing.Pool and other
implementations.

For examples, in this library, you can s/ThreadPool/ProcessPool/g and
this example would work: .

If you want to DIY, with multiprocessing.Lock/Pipe/Queue, I don't
understand why it would be more of a pain to write your threads as
processes.


// aht
http://blog.onideas.ws
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is python not good enough?

[Anh Hai Trinh]

On Jan 18, 6:03 pm, Phlip  wrote:
> On Jan 12, 7:09 am, ikuta liu  wrote:
>
> > Go language try to merge low level, hight level and browser language.
>
> Go uses := for assignment.

Except that it doesn't. := is a declaration.

s := "foo"

is short for

var s string = "foo"


Cheers,

aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: iterators and views of lists

[Anh Hai Trinh]

On Dec 20, 12:04 am, Anh Hai Trinh  wrote:

> chain:
>
>   sorted(itertools.chain(listagent(x)[::2], listagent(y)[-1:1:-2]))
>   [0, 4, 8, 12, 13, 15, 16, 17, 19]
>
> zip:
>
>   sorted(itertools.izip(listagent(z)[1::3], listagent(x)[2::3]))
>   [(452, 16), (758, 4), (898, 10)]

I think I mis-interpret Andrei's slides. I think what he meant to sort
a chain of slices is such that to move the smaller elements into the
first-given slices in-place, thus moving items around whatever lists
underlying those slices.

And for zip, I think he meant sort the first slice, but moving in-
place the items referred by the others in lock-step.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: iterators and views of lists

[Anh Hai Trinh]

On Dec 19, 5:47 am, Bearophile  wrote:

> It seems you have missed my post, so here it is, more explicitly:
>
> http://www.boostcon.com/site-media/var/sphene/sphwiki/attachment/2009...


Interestingly, my `listagent` can be used as a lazy iterator and thus
using itertools we can compose them just like Andrei's `range`.

the stage:

  x = range(0, 20, 2); x
  [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

  y = range(10, 20); y
  [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

  z = [996, 758, 670, 236, 898, 337, 442, 452, 490, 547]

  from listagent import listagent
  import itertools

chain:

  sorted(itertools.chain(listagent(x)[::2], listagent(y)[-1:1:-2]))
  [0, 4, 8, 12, 13, 15, 16, 17, 19]

zip:

  sorted(itertools.izip(listagent(z)[1::3], listagent(x)[2::3]))
  [(452, 16), (758, 4), (898, 10)]

stride: slicing an agent returns another one, those lazy agents!

  python -m timeit -s'from listagent import listagent' 'list(listagent
(range(100))[1:-1:5][::7][1:-1:42])'
  10 loops, best of 3: 55.5 msec per loop

  python -m timeit 'range(100)[1:-1:5][::7][1:-1:42]'
  10 loops, best of 3: 280 msec per loop

radial: not implemented


`listagent` is implemented in pure Python, of course. If implemented
in C, there is surprisingly little difference between a slicing
`agent` like this, or a listiterator, since both will have to keep a
pointer to an memory index of in the original list: the listiterator
advances this pointer when we call next(), whereas the agent keeps a
pointer to the starting element of the slice and return basically
(PyObject *) *(p+(i*step)) on __getitem__[i]. Mutability is a just
matter of exposing this pointer to Python code in a nice way.

aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: iterators and views of lists

[Anh Hai Trinh]

On Dec 18, 3:07 am, Brendan Miller  wrote:

> Well, it doesn't really need to be any slower than a normal list. You
> only need to use index and do extra additions because it's in python.
> However, if listagent were written in C, you would just have a pointer
> into the contents of the original list, and the length, which is all
> that list itself has.

You're right, I was thinking in Python instead of C. So the
translations in that case is really straight forward, using just
pointer arithmetic.

aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: iterators and views of lists

[Anh Hai Trinh]

> I have a couple of thoughts:
> 1. Since [:] by convention already creates a copy, it might violate
> people's expectations if that syntax were used.

Indeed, listagent returns self on __getitem__[:]. What I meant was
this:

  x = [0, 1, 2, 3, 4, 5, 6, 7]
  a = listagent(x)[::2]
  a[:] = listagent(x)[::-2]

And we get x = [7, 1, 5, 3, 3, 5, 1, 7], the copying happens in-place,
of course.


> 2. I'd give the listagent the mutable sequence interface

Done!  I put the code in a repository here for those who might be
interested:
.

In retrospect, the Python gurus here was right though. Copy, modify
then replace is good enough, if not better since items are just
pointers instead of values. For reversing, you need to translate all
the indices (which cost exactly one addition and one multiplication
per index). Is that cheaper than copying all the pointers to a new
list?  For sorting, you definitely need to construct a lookup table
since the sort algorithm needs to look over the indices multiple
times, which means you are using two pointer indirections per index.
Is that cheaper than just copying all the pointers to a new list? Even
if there is any benefit at all, you'll need to implement listagent in
C to squeeze it out.

However, using listagents is faster if you just want a few items out
of the slice. And it's cute.

Peace,
aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: iterators and views of lists

[Anh Hai Trinh]

On Dec 16, 2:48 pm, Brendan Miller  wrote:

> No, that's what I'm getting at... Most of the existing mutating
> algorithms in python (sort, reverse) operate over entire collections,
> not partial collections delimited by indexes... which would be really
> awkward anyway.

Ok it can be done! The code is here: .

  >>> from listagent import listagent
  >>> x = [22, 7, 2, -5, 8, 4]
  >>> listagent(x)[1:].sort()
  >>> x
  [22, -5, 2, 4, 7, 8]
  >>> listagent(x)[::2].reverse()
  >>> x
  [7, -5, 2, 4, 22, 8]

Basically the agent refers to the original list only by "address
translation", and indeed made no copy of anything whatever! I
implemented Shell sort but I suppose others are possible.

The implementation is incomplete, for now you cannot do slice
assignment, i.e.

  a = listagent(x)[::-2]
  a[1:] = [4, 2]

but it should be easy to implement, and I'm too sleepy.

Peace,
aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: iterators and views of lists

[Anh Hai Trinh]

On Dec 16, 10:39 am, Brendan Miller  wrote:
> I was trying to reimplement some of the c++ library of generic
> algorithms in c++ in python, but I was finding that this is
> problematic to do this in a generic way because there isn't any
> equivalent of c++'s forward iterators, random access iterators, etc.
> i.e. all python iterators are just input iterators that can't mutate
> the sequence they iterate over nor move backwards or by an arbitrary
> offset.

You might be interested in this library .

You can easily create arbitrary "slice", for example

  i = mylist >> takei(primes())

will return an iterator over the items of mylist with a prime number
index, given that primes() return an iterator over prime numbers.

I'm not familiar with forward/random-access/bidirectional iterators.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Seek support for new slice syntax PEP.

[Anh Hai Trinh]

>         > from numpy import s_
>         > s_[1:2:3]
>         slice(1, 2, 3)
>         > s_[1:2:3, ..., 4:5]
>         (slice(1, 2, 3), Ellipsis, slice(4, 5, None))
>
> Or would it be possible to define "slice" itself so that it implements
> __getitem__ and __getslice__?


Indeed!

Python 2.6.4 (r264:75706, Oct 27 2009, 06:25:13)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> class slice(object):
...  @staticmethod
...  def __getitem__(sliceobj):
...return sliceobj

>>> slice = slice()

>>> slice[:]
slice(None, None, None)

>>> slice[1::-1]
slice(1, None, -1)

>>> range(10).__getitem__(slice[::2])
[0, 2, 4, 6, 8]


aht
-- 
http://mail.python.org/mailman/listinfo/python-list

which pi formula is given in the decimal module documentation?

[Anh Hai Trinh]

I'm just curious which formula for pi is given here: ?

def pi():
"""Compute Pi to the current precision.

>>> print pi()
3.141592653589793238462643383

"""
getcontext().prec += 2  # extra digits for intermediate steps
three = Decimal(3)  # substitute "three=3.0" for regular
floats
lasts, t, s, n, na, d, da = 0, three, 3, 1, 0, 0, 24
while s != lasts:
lasts = s
n, na = n+na, na+8
d, da = d+da, da+32
t = (t * n) / d
s += t
getcontext().prec -= 2
return +s   # unary plus applies the new precision


It looks like an infinite series with term `t`, where`n` = (2k-1)^2
and `d` = d = 4k(4k+2) for k = 1... Does it have a name?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A different take on finding primes

[Anh Hai Trinh]

> 1) google list of prime numbers
> 2) see "Prime numbers list" in the results (number 3 in the results)
> 3) click link that leads towww.prime-numbers.org
>
> I found 455042511 prime numbers in approx 15 seconds.

Not bad at all. How about using http://www.sagemath.org/ (written in
Python).

sage: time primes_first_n(10^7);
CPU times: user 4.36 s, sys: 2.43 s, total: 6.79 s
Wall time: 6.88 s

That used 3G of RAM, you could certainly go higher if you have more
memory.

aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: substituting list comprehensions for map()

[Anh Hai Trinh]

> Yes, just about any ‘map()’ operation has a corresponding list
> comprehension. (Does anyone know of a counter-example, a ‘map()’
> operation that doesn't have a correspondingly simple list
> comprehension?)

Try turning this into a list comprehension:

  vectorsum = lambda *args: map(sum, zip(*args))

  vectorsum([1,2], [3,4], [5,6])
->[9, 12]
  vectorsum([1,2], [3,4], [5,6], [7,8])
->[16, 20]

Peace,

aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: substituting list comprehensions for map()

[Anh Hai Trinh]

> Try turning this into a list comprehension:
>
>   vectorsum = lambda *args: map(sum, zip(*args))
>
>   vectorsum([1,2], [3,4], [5,6])
> ->[9, 12]
>   vectorsum([1,2], [3,4], [5,6], [7,8])
> ->[16, 20]

Nvm, it's actually easy:
  vectorsum = lambda *args: [sum(i) for i in zip(*args)]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: substituting list comprehensions for map()

[Anh Hai Trinh]

> On the other hand, list comps using an if clause can't be written as pure
> maps. You can do this:
>
> [func(x) for x in seq if cond(x)]
>
> filter(cond, map(func, seq))
>
> but the second version may use much more temporary memory if seq is huge
> and cond very rarely true.

You could use ifilter, imap there to reduce memory.


> And I don't think you could write this as a single map:
>
> [func(a, b) for a in seq1 for b in seq2]

Oh you surely could:

  seq1, seq2 = [1,2,3], [4,5,6]

  [a+b for a in seq1 for b in seq2]
->[5, 6, 7, 6, 7, 8, 7, 8, 9]

  from itertools import product
  map(sum, product(seq1, seq2))
->[5, 6, 7, 6, 7, 8, 7, 8, 9]


Peace,

aht
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: How about adding slice notation to iterators/generators?

[Anh Hai Trinh]

I've written something that is better than you could've imagine.

Get it here: 

It works with anything iterable, no need to alter anything.

  from itertools import count
  from stream import item
  c = count()
  c >> item[1:10:2]
->[1, 3, 5, 7, 9]
  c >> item[:5]
->[10, 11, 12, 13, 14]

There is a ton more you could do with that library, i.e. piping & lazy-
evaluation.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dataflow programming in Python

[Anh Hai Trinh]

> Does it have any advantage to generator comprehension?
>
> import re
> mm = mapmethod('strip')        # Is mapmethod something in the stdlib?
> pat = re.compile('[Pp]attern')
> result = (mm(line) for line in open('log') if pat.search(line))
>
> which is also lazy

Generator expression accomplishes the same thing, the main advantage I
think is that of better notation. You can clearly see each of the
processing steps which is usually lost in the syntaxes. This is the
simplest example here.

And strip is a method of string, so you would say `(line.strip() for
line in open('log') if pat.search(line))`.
-- 
http://mail.python.org/mailman/listinfo/python-list

Dataflow programming in Python

[Anh Hai Trinh]

Hello all,

I just want to share with you something that I've worked on recently.
It is a library which implements streams -- generalized iterators with
a pipelining mechanism and lazy-evaluation to enable data-flow
programming in Python.

The idea is to be able to take the output of a function that turn an
iterable into another iterable and plug that as the input of another
such function. While you can already do some of this using function
composition, this package provides an elegant notation for it by
overloading the '>>' operator.

To give a simple example of string processing, here we grep the lines
matching some regex, strip them and accumulate to a list:

> import re
> result = open('log').xreadlines() >> filter(re.compile('[Pp]attern').search) 
> >> mapmethod('strip') >> list

This approach focuses the programming on processing streams of data,
step by step. A pipeline usually starts with a generator, or anything
iterable, then passes through a number of processors. Multiple streams
can be branched and combined. Finally, the output is fed to an
accumulator, which can be any function of one iterable argument.

Another advantage is that the values are lazily computed, i.e. only
when the accumulator needs to have it.

Homepage:
http://trinhhaianh.com/stream.py/
-- 
http://mail.python.org/mailman/listinfo/python-list