Hello everyone,

I started making a small change to support generators to Python UDFs.

That is, now a Python function can use the 'yield' statement to return
a generator. This is useful for defining functions that can return a
large number of elements and also makes it possible to write more
'pythonic' code.

The code is quite simple (I only changed a couple of lines) and also
includes a case test. Feel free to have a look at in on my Github
repository:

https://github.com/cavorite/pig/compare/py-return-iters

If you think that it might be useful, I can submit a new ticket in
Jira along with a patch.

I also worked on a similar feature for converting bags to iterators
(instead of lists), when they are passed as arguments to Python UDFs.
In principle, it should use less memory (since the bag is not copied
to a list), but it is not backwards compatible with existing UDFs. I
can also create a Jira ticket proposing this feature if you're
interested.

Best,

Juan Manuel Caicedo

Reply via email to