[ 
https://issues.apache.org/jira/browse/PIG-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717635#action_12717635
 ] 

Ankur commented on PIG-821:
---------------------------

Ok, So I tried writing an NTILE UDF that accepts 
1. Number of tiles 
2. A bag of sorted tuples

The problem with that is it is essentially a serial process instead of parallel 
as one would expect. So I am not sure if an NTILE operation can be done 
efficiently via a UDF. An efficient NTILE operation over sorted dataset should 
1. Partition the sorted data into the number of tiles requested
2. Preserve the ordering in each tile.
3. Have each tile contain exactly the number of elements as per ntile logic.

There is a total ordering partitioner in hadoop - 
http://issues.apache.org/jira/browse/HADOOP-3019 that effects total ordering of 
output data. However it cannot strictly enforce the number of elements 
contained in each part output which is a necessary condition to comply with 
NTILE logic.

Any thoughts?


> simulate NTILE(n) , rank() functionality in pig
> -----------------------------------------------
>
>                 Key: PIG-821
>                 URL: https://issues.apache.org/jira/browse/PIG-821
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>         Environment: mithril gold -gateway 4000
>            Reporter: Rekha
>             Fix For: 0.2.0
>
>
> Hi,
> I came across a job which has some processing which I cant seem to get easily 
> over-the-counter from pig.
> These are NTILE() /rank() operations available in oracle.
> While I am trying to write a UDF, that is not working out too well for me 
> yet.. :(
> I have a ntile(n) over (partititon by x, y, z order by a desc, b desc) 
> operation to be done in pig scripts.
> Is there a default function in pig scripting which can do this?
> For example, lets consider a simple example at 
> http://download.oracle.com/docs/cd/B14117_01/server.101/b10759/functions091.htm
> So here, how would we ideally substitute NTILE() with? any pig counterpart 
> function/udf?
> SELECT last_name, salary, NTILE(4) OVER (ORDER BY salary DESC) 
>    AS quartile FROM employees
>    WHERE department_id = 100;
>  
> LAST_NAME                     SALARY   QUARTILE
> ------------------------- ---------- ----------
> Greenberg                      12000          1
> Faviet                          9000          1
> Chen                            8200          2
> Urman                           7800          2
> Sciarra                         7700          3
> Popp                            6900          4
>  
> In real case, i have ntile over multiple columns, so ideal way to find 
> histograms/boundary/spitting out the bucket number is needed.
> Similarly a pig function is required for rank() over(partition by a,b,c order 
> by d desc) as e
> Please let me know soon.
> Thanks & Regards,
> /Rekha

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to