Cheolsoo Park created PIG-4329:
----------------------------------
Summary: Fetch optimization should be disabled when limit is not
pushed up
Key: PIG-4329
URL: https://issues.apache.org/jira/browse/PIG-4329
Project: Pig
Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park
Fix For: 0.15.0
Although PIG-4135 disable fetch optimization when there is no limit in the
plan, that doesn't solve the problem completely. In fact, fetch optimization
should be still disabled if limit is not pushed up. Consider the following
query-
{code}
random_lists = load 'prodhive.schakraborty.search_server_denorm_impressions'
using DseStorage();
random_lists = filter random_lists by entity_section=='random');
random_lists = limit random_lists 10;
dump random_lists;
{code}
Because the {{filter by}} blocks limit from being pushed up, POLoad actually
scans the full table. In this case, fetch optimization makes the job extremely
slow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
