[
https://issues.apache.org/jira/browse/HIVE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036758#comment-14036758
]
Navis commented on HIVE-7218:
-----------------------------
If we cannot change the limit of total size of environment variables, we can
walk-around this problem by specifying only needed ones for the script (I don't
think 'mapred.input.dir' could be useful information in the script). For
example,
{noformat}
hive>set X=someX;
hive>set Y=someY;
hive>SELECT TRANSFORM('echo ${X:-x} ${Y:-y} ${Z:-z}') USING 'sh' AS x,y,z WITH
('Z'='someZ', 'Y')
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' FROM src tablesample (1 rows);
x someY someZ
{noformat}
In the above query, by specifying WITH ('Z'='someZ', 'Y'), script accepts only
two environment variables 'Z=someZ' (which is specified explicitly) and
'Y=someY' (which is from HiveConf).
> java.io.IOException: error=7, Argument list too long
> ----------------------------------------------------
>
> Key: HIVE-7218
> URL: https://issues.apache.org/jira/browse/HIVE-7218
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0,
> 0.12.0, 0.13.0, 0.13.1
> Reporter: Ryan Harris
> Assignee: Navis
> Attachments: HIVE-7218.1.patch.txt
>
>
> HIVE-2372 was originally created in response to this error message, however
> that patch was merely a work-around to handle the condition where
> mapred.input.dir is too long.
> Any other environment variable that is "too long" for the host OS will still
> cause a job failure.
> In my case:
> While creating a table with a large number of columns, a large hive variable
> is temporarily created using SET, the variable contains the columns and
> column descriptions.
> A CREATE TABLE statement then successfully uses that large variable.
> After successfully creating the table the hive script attempts to load data
> into the table using a TRANSFORM script, triggering the error:
> java.io.IOException: error=7, Argument list too long
> Since the variable is no longer used after the table is created, the hive
> script was updated to SET the large variable to empty.
> After setting the variable empty the second statement in the hive script ran
> fine.
> Hive should more gracefully notify the user as to the cause of the problem
> and offer a configurable approach for automatically handling the condition.
> In this case, originally identifying the cause of the issue was somewhat
> confusing since the portion of the hive script that referenced the long
> variable ran successfully, and the portion of the script that failed didn't
> even use/reference the variable that was causing that portion to fail.
> Since HIVE-2372 has already been "Fixed" this JIRA re-opens the issue since
> the original issue was worked around, not resolved...
--
This message was sent by Atlassian JIRA
(v6.2#6252)
