[
https://issues.apache.org/jira/browse/HCATALOG-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472918#comment-13472918
]
Feng Peng commented on HCATALOG-453:
------------------------------------
Tested on our cluster, reduced one job.xml from 31MB to 1MB.
There is no unit test for this class but it is used in other tests like
TestHCatInputFormat. Please let me know if I need to add a unit test for the
class.
> HCatalog queries fail due to exceeding max jobconf size
> -------------------------------------------------------
>
> Key: HCATALOG-453
> URL: https://issues.apache.org/jira/browse/HCATALOG-453
> Project: HCatalog
> Issue Type: Bug
> Affects Versions: 0.2, 0.4, 0.5
> Reporter: Feng Peng
> Assignee: Feng Peng
> Attachments: HCATALOG-453.patch
>
>
> The following script fails because exceeding max jobconf size:
> {noformat}
> raw_data = LOAD 'db.table' using org.apache.hcatalog.pig.HCatLoader();
> filtered_data = FILTER raw_data BY (part_dt>='20120528T000000Z') and
> (part_dt<='20120624T230000Z');
> dump filtered_data;
> {noformat}
> Stacktrace:
> {noformat}
> 2012-07-18 00:06:24,067 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 6017: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.io.IOException: Exceeded max jobconf size: 10523042 limit: 5242880
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3766)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> Caused by: java.io.IOException: Exceeded max jobconf size: 10523042 limit:
> 5242880
> at
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:406)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3764)
> ... 10 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
