[jira] [Created] (HIVE-2834) Diff masking it too aggressive in index_bitmap*.q and index_compact*.q tests
Diff masking it too aggressive in index_bitmap*.q and index_compact*.q tests Key: HIVE-2834 URL: https://issues.apache.org/jira/browse/HIVE-2834 Project: Hive Issue Type: Bug Components: Tests Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[ https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220637#comment-13220637 ] Carl Steinbach commented on HIVE-2833: -- @Enis: Modifying the metastore configuration properties in data/conf/hive-site.xml will probably work for most tests. > Fix test failures caused by HIVE-2716 > - > > Key: HIVE-2833 > URL: https://issues.apache.org/jira/browse/HIVE-2833 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Enis Soztutar > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[ https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220597#comment-13220597 ] Enis Soztutar commented on HIVE-2833: - I can work on it, if we can get a reproducible test, or a list of tests which are failing. Is there an easy way to run the tests on trunk with mysql? > Fix test failures caused by HIVE-2716 > - > > Key: HIVE-2833 > URL: https://issues.apache.org/jira/browse/HIVE-2833 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Enis Soztutar > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[ https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220586#comment-13220586 ] Ashutosh Chauhan commented on HIVE-2833: @Namit, It passes all the tests on the trunk. Do, you have a reproducible testcase for this? > Fix test failures caused by HIVE-2716 > - > > Key: HIVE-2833 > URL: https://issues.apache.org/jira/browse/HIVE-2833 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Enis Soztutar > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[ https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220557#comment-13220557 ] Namit Jain commented on HIVE-2833: -- Can you please work on this as a very high priority ? We need a fix asap for this. > Fix test failures caused by HIVE-2716 > - > > Key: HIVE-2833 > URL: https://issues.apache.org/jira/browse/HIVE-2833 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Enis Soztutar > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[
https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220553#comment-13220553
]
Carl Steinbach commented on HIVE-2833:
--
{quote}
@Ashutosh, it broke facebook's internal branch.
Did you test it with mysql db - I haven't debugged it, but reverting this fixes
the problem.
{quote}
> Fix test failures caused by HIVE-2716
> -
>
> Key: HIVE-2833
> URL: https://issues.apache.org/jira/browse/HIVE-2833
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Carl Steinbach
>Assignee: Enis Soztutar
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2833) Fix test failures caused by HIVE-2716
Fix test failures caused by HIVE-2716 - Key: HIVE-2833 URL: https://issues.apache.org/jira/browse/HIVE-2833 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach Assignee: Enis Soztutar -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2716) Move retry logic in HiveMetaStore to a separe class
[ https://issues.apache.org/jira/browse/HIVE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2716. -- Resolution: Fixed Hadoop Flags: Reviewed I created HIVE-2833 to cover the task of fixing the test failures caused by this patch. > Move retry logic in HiveMetaStore to a separe class > --- > > Key: HIVE-2716 > URL: https://issues.apache.org/jira/browse/HIVE-2716 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 0.9.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.9.0 > > Attachments: HIVE-2716.D1227.1.patch, HIVE-2716.D1227.2.patch, > HIVE-2716.D1227.3.patch, HIVE-2716.D1227.4.patch, HIVE-2716.patch > > > In HIVE-1219, method retrying for raw store operation are introduced to > handle jdo operations more robustly. However, the abstraction for the > RawStore operations can be moved to a separate class implementing RawStore, > which should clean up the code base for HiveMetaStore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2716) Move retry logic in HiveMetaStore to a separe class
[ https://issues.apache.org/jira/browse/HIVE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220550#comment-13220550 ] Namit Jain commented on HIVE-2716: -- @Ashutosh, it broke facebook's internal branch. Did you test it with mysql db - I haven't debugged it, but reverting this fixes the problem. > Move retry logic in HiveMetaStore to a separe class > --- > > Key: HIVE-2716 > URL: https://issues.apache.org/jira/browse/HIVE-2716 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 0.9.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.9.0 > > Attachments: HIVE-2716.D1227.1.patch, HIVE-2716.D1227.2.patch, > HIVE-2716.D1227.3.patch, HIVE-2716.D1227.4.patch, HIVE-2716.patch > > > In HIVE-1219, method retrying for raw store operation are introduced to > handle jdo operations more robustly. However, the abstraction for the > RawStore operations can be moved to a separate class implementing RawStore, > which should clean up the code base for HiveMetaStore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-2716) Move retry logic in HiveMetaStore to a separe class
[ https://issues.apache.org/jira/browse/HIVE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain reopened HIVE-2716: -- > Move retry logic in HiveMetaStore to a separe class > --- > > Key: HIVE-2716 > URL: https://issues.apache.org/jira/browse/HIVE-2716 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 0.9.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.9.0 > > Attachments: HIVE-2716.D1227.1.patch, HIVE-2716.D1227.2.patch, > HIVE-2716.D1227.3.patch, HIVE-2716.D1227.4.patch, HIVE-2716.patch > > > In HIVE-1219, method retrying for raw store operation are introduced to > handle jdo operations more robustly. However, the abstraction for the > RawStore operations can be moved to a separate class implementing RawStore, > which should clean up the code base for HiveMetaStore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2827) Implement nullsafe equi-join
[
https://issues.apache.org/jira/browse/HIVE-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220546#comment-13220546
]
Phabricator commented on HIVE-2827:
---
cwsteinbach has requested changes to the revision "HIVE-2827 [jira] Implement
nullsafe equi-join".
Looks good overall. I will test and commit once the requested changes have
been applied. Thanks.
INLINE COMMENTS
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:412 I wanted to see
this included in HIVE-2810 since it sounded like it would take some time to
implement the fix for JOINs, and in the meantime I didn't want people to
accidentally use <=> in JOINs. However, since this patch implements correct
JOIN handling for <=>, I don't think it makes sense to provide the option of
disabling this. Please remove.
ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java:50 Spelling:
s/jon/join/
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1339 Please
remove.
ql/src/test/queries/clientpositive/join_nullsafe.q:4 please reference the
path using the appropriate test system property, e.g.
"${system:test.src.data.dir}/files/in8.txt"
REVISION DETAIL
https://reviews.facebook.net/D1971
BRANCH
DPAL-873
> Implement nullsafe equi-join
>
>
> Key: HIVE-2827
> URL: https://issues.apache.org/jira/browse/HIVE-2827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Environment: ubuntu 10.04
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-2827.D1971.1.patch, HIVE-2827.D1971.2.patch,
> HIVE-2827.D1971.3.patch, HIVE-2827.D1971.4.patch
>
>
> was part of HIVE-2810, but separated because it affected more classes than
> expected.
> {noformat}
> SELECT * FROM a JOIN b ON a.key <=> b.key
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2827) Implement nullsafe equi-join
[
https://issues.apache.org/jira/browse/HIVE-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-2827:
-
Status: Open (was: Patch Available)
> Implement nullsafe equi-join
>
>
> Key: HIVE-2827
> URL: https://issues.apache.org/jira/browse/HIVE-2827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Environment: ubuntu 10.04
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-2827.D1971.1.patch, HIVE-2827.D1971.2.patch,
> HIVE-2827.D1971.3.patch, HIVE-2827.D1971.4.patch
>
>
> was part of HIVE-2810, but separated because it affected more classes than
> expected.
> {noformat}
> SELECT * FROM a JOIN b ON a.key <=> b.key
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2822) Add JSON output to the hive ddl commands
[
https://issues.apache.org/jira/browse/HIVE-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-2822:
-
Attachment: hive-json-01-branch0-8.patch
A version of the patch that applies cleanly to branch 0.8-r2
> Add JSON output to the hive ddl commands
>
>
> Key: HIVE-2822
> URL: https://issues.apache.org/jira/browse/HIVE-2822
> Project: Hive
> Issue Type: Improvement
>Reporter: Chris Dean
> Attachments: hive-json-01-branch0-8.patch, hive-json-01.patch
>
>
> The goal is to have an option to produce JSON output of the DDL commands that
> is easily machine parseable.
> For example, "desc my_table" currently gives
> {noformat}
> idbigint
> user string
> {noformat}
> and we want to allow a json output:
> {noformat}
> {
> "columns": [
> {"name": "id", "type": "bigint"},
> {"name": "user", "type": "string"}
> ]
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2278) Support archiving for multiple partitions if the table is partitioned by multiple columns
[ https://issues.apache.org/jira/browse/HIVE-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220527#comment-13220527 ] Zhenxiao Luo commented on HIVE-2278: Hi, May I ask a question about archive_corrupt.rc? As my understanding, this is an RCFile, with corrupted content. In archive_corrupt.q, it is trying to load this file into table tstsrcpart. Is this expected to be succeeded? Or, we should expect a failure like: Failed with exception Wrong file format. Please check the file's format. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask Specifically, what is inside archive_corrupt.rc? How is it corrupted? Thanks Zhenxiao > Support archiving for multiple partitions if the table is partitioned by > multiple columns > - > > Key: HIVE-2278 > URL: https://issues.apache.org/jira/browse/HIVE-2278 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Marcin Kurczych > Fix For: 0.8.0 > > Attachments: HIVE-2278.10.patch, HIVE-2278.11.patch, > HIVE-2278.12.patch, HIVE-2278.2.patch, HIVE-2278.3.patch, HIVE-2278.4.patch, > HIVE-2278.5.patch, HIVE-2278.5.patch, HIVE-2278.6.patch, HIVE-2278.6.patch, > HIVE-2278.7.patch, HIVE-2278.8.patch, HIVE-2278.9.patch, archive_corrupt.rc, > hive.2278.1.patch > > > If a table is partitioned by ds,hr > it should be possible to archive all the files in ds to reduce the number of > files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2810: - Status: Open (was: Patch Available) @Navis: Please update the patch with the request test changes. I will test and commit once this done. Thanks! > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch, HIVE-2810.D1791.2.patch, > HIVE-2810.D1791.3.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220504#comment-13220504 ] Phabricator commented on HIVE-2810: --- cwsteinbach has commented on the revision "HIVE-2810 [jira] Implement NULL-safe equality operator <=>". INLINE COMMENTS ql/src/test/queries/clientpositive/udf_equal.q:7 Please add NULL comparisons for '=': NULL=NULL true=NULL NULL=true false=NULL NULL=false ql/src/test/queries/clientpositive/udf_equal.q:12 Please add the missing comparisons from above that were used to test '='. I realize that '<=>' is implemented in terms of '=', but the tests should treat both implementations like black boxes. Thanks. REVISION DETAIL https://reviews.facebook.net/D1791 BRANCH DPAL-843 > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch, HIVE-2810.D1791.2.patch, > HIVE-2810.D1791.3.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: 'arc diff' failing with "Invalid or missing field 'Test Plan': You must provide a test plan."
I don't know if there is a way to disable it. But, then I don't know much
about phabricator/arc infra.
On Thu, Mar 1, 2012 at 15:15, Carl Steinbach wrote:
> Thanks for the tip. Is there any way to disable this behavior?
>
> On Thu, Mar 1, 2012 at 2:56 PM, Ashutosh Chauhan >wrote:
>
> > Hi Carl,
> >
> > Include in your git commit message following line
> > Test Plan: .
> >
> > It is looking for string "Test Plan" in your commit message and fails if
> > cant find one.
> >
> > Hope it helps,
> > Ashutosh
> >
> > On Thu, Mar 1, 2012 at 14:45, Carl Steinbach wrote:
> >
> > > Hey,
> > >
> > > Today I started getting the following error when I try to create a
> > > phabricator review request using arc:
> > >
> > > % arc diff --jira HIVE-2831
> > >
> > > Exception:
> > > Invalid or missing field 'Test Plan': You must provide a test plan.
> > > (Run with --trace for a full exception trace.)
> > >
> > >
> > > Here's the complete trace:
> > >
> > > % arc --trace diff --jira HIVE-2831
> > > Loading phutil library 'arc_jira_lib' from
> > > '/Users/carl/Work/repos/hive4/.arc_jira_lib'...
> > > >>> [0] conduit.connect()
> > > <<< [0] 318,295 us
> > > >>> [1] $ (cd '/Users/carl/Work/repos/hive4'; git rev-parse
> > > --show-cdup)
> > > <<< [1] 14,662 us
> > > >>> [2] $ (cd '/Users/carl/Work/repos/hive4/'; git rev-parse
> > > --verify HEAD^)
> > > <<< [2] 16,343 us
> > > >>> [3] $ (cd '/Users/carl/Work/repos/hive4/'; git log
> > > --first-parent --format=medium 'HEAD^'..HEAD)
> > > <<< [3] 15,040 us
> > > >>> [4] differential.parsecommitmessage()
> > > <<< [4] 547,222 us
> > >
> > > Fatal error: Uncaught exception
> > > 'ArcanistDifferentialCommitMessageParserException' with message
> 'Invalid
> > or
> > > missing field 'Test Plan': You must provide a test plan.' in
> > >
> > >
> >
> /Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
> > > Stack trace:
> > > #0
> > >
> > >
> >
> /Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
> > >
> > >
> >
> ArcanistDifferentialCommitMessage->pullDataFromConduit(Object(ConduitClient))
> > > #1
> > >
> > >
> >
> /Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(364):
> > > ArcJIRAConfiguration->willRunDiffWorkflow()
> > > #2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
> > > ArcJIRAConfiguration->willRunWorkflow('diff',
> > Object(ArcanistDiffWorkflow))
> > > #3 {main}
> > > thrown in
> > >
> > >
> >
> /Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
> > > on line 88
> > >
> > >
> > > Anyone know what's going on here?
> > >
> > > Thanks.
> > >
> > > Carl
> > >
> >
>
Re: 'arc diff' failing with "Invalid or missing field 'Test Plan': You must provide a test plan."
Thanks for the tip. Is there any way to disable this behavior?
On Thu, Mar 1, 2012 at 2:56 PM, Ashutosh Chauhan wrote:
> Hi Carl,
>
> Include in your git commit message following line
> Test Plan: .
>
> It is looking for string "Test Plan" in your commit message and fails if
> cant find one.
>
> Hope it helps,
> Ashutosh
>
> On Thu, Mar 1, 2012 at 14:45, Carl Steinbach wrote:
>
> > Hey,
> >
> > Today I started getting the following error when I try to create a
> > phabricator review request using arc:
> >
> > % arc diff --jira HIVE-2831
> >
> > Exception:
> > Invalid or missing field 'Test Plan': You must provide a test plan.
> > (Run with --trace for a full exception trace.)
> >
> >
> > Here's the complete trace:
> >
> > % arc --trace diff --jira HIVE-2831
> > Loading phutil library 'arc_jira_lib' from
> > '/Users/carl/Work/repos/hive4/.arc_jira_lib'...
> > >>> [0] conduit.connect()
> > <<< [0] 318,295 us
> > >>> [1] $ (cd '/Users/carl/Work/repos/hive4'; git rev-parse
> > --show-cdup)
> > <<< [1] 14,662 us
> > >>> [2] $ (cd '/Users/carl/Work/repos/hive4/'; git rev-parse
> > --verify HEAD^)
> > <<< [2] 16,343 us
> > >>> [3] $ (cd '/Users/carl/Work/repos/hive4/'; git log
> > --first-parent --format=medium 'HEAD^'..HEAD)
> > <<< [3] 15,040 us
> > >>> [4] differential.parsecommitmessage()
> > <<< [4] 547,222 us
> >
> > Fatal error: Uncaught exception
> > 'ArcanistDifferentialCommitMessageParserException' with message 'Invalid
> or
> > missing field 'Test Plan': You must provide a test plan.' in
> >
> >
> /Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
> > Stack trace:
> > #0
> >
> >
> /Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
> >
> >
> ArcanistDifferentialCommitMessage->pullDataFromConduit(Object(ConduitClient))
> > #1
> >
> >
> /Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(364):
> > ArcJIRAConfiguration->willRunDiffWorkflow()
> > #2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
> > ArcJIRAConfiguration->willRunWorkflow('diff',
> Object(ArcanistDiffWorkflow))
> > #3 {main}
> > thrown in
> >
> >
> /Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
> > on line 88
> >
> >
> > Anyone know what's going on here?
> >
> > Thanks.
> >
> > Carl
> >
>
Re: 'arc diff' failing with "Invalid or missing field 'Test Plan': You must provide a test plan."
Hi Carl,
Include in your git commit message following line
Test Plan: .
It is looking for string "Test Plan" in your commit message and fails if
cant find one.
Hope it helps,
Ashutosh
On Thu, Mar 1, 2012 at 14:45, Carl Steinbach wrote:
> Hey,
>
> Today I started getting the following error when I try to create a
> phabricator review request using arc:
>
> % arc diff --jira HIVE-2831
>
> Exception:
> Invalid or missing field 'Test Plan': You must provide a test plan.
> (Run with --trace for a full exception trace.)
>
>
> Here's the complete trace:
>
> % arc --trace diff --jira HIVE-2831
> Loading phutil library 'arc_jira_lib' from
> '/Users/carl/Work/repos/hive4/.arc_jira_lib'...
> >>> [0] conduit.connect()
> <<< [0] 318,295 us
> >>> [1] $ (cd '/Users/carl/Work/repos/hive4'; git rev-parse
> --show-cdup)
> <<< [1] 14,662 us
> >>> [2] $ (cd '/Users/carl/Work/repos/hive4/'; git rev-parse
> --verify HEAD^)
> <<< [2] 16,343 us
> >>> [3] $ (cd '/Users/carl/Work/repos/hive4/'; git log
> --first-parent --format=medium 'HEAD^'..HEAD)
> <<< [3] 15,040 us
> >>> [4] differential.parsecommitmessage()
> <<< [4] 547,222 us
>
> Fatal error: Uncaught exception
> 'ArcanistDifferentialCommitMessageParserException' with message 'Invalid or
> missing field 'Test Plan': You must provide a test plan.' in
>
> /Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
> Stack trace:
> #0
>
> /Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
>
> ArcanistDifferentialCommitMessage->pullDataFromConduit(Object(ConduitClient))
> #1
>
> /Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(364):
> ArcJIRAConfiguration->willRunDiffWorkflow()
> #2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
> ArcJIRAConfiguration->willRunWorkflow('diff', Object(ArcanistDiffWorkflow))
> #3 {main}
> thrown in
>
> /Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
> on line 88
>
>
> Anyone know what's going on here?
>
> Thanks.
>
> Carl
>
[jira] [Commented] (HIVE-2831) TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220452#comment-13220452 ] Carl Steinbach commented on HIVE-2831: -- @Ed: I think in the past we have handled problems like this by updating the list of strings that diff should ignore. The patch I just attached does the same. > TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23 > > > Key: HIVE-2831 > URL: https://issues.apache.org/jira/browse/HIVE-2831 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2831.1.patch.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2831) TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2831: - Attachment: HIVE-2831.1.patch.txt > TestContribCliDriver.dboutput and TestCliDriver.input45 fail on 0.23 > > > Key: HIVE-2831 > URL: https://issues.apache.org/jira/browse/HIVE-2831 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2831.1.patch.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
'arc diff' failing with "Invalid or missing field 'Test Plan': You must provide a test plan."
Hey,
Today I started getting the following error when I try to create a
phabricator review request using arc:
% arc diff --jira HIVE-2831
Exception:
Invalid or missing field 'Test Plan': You must provide a test plan.
(Run with --trace for a full exception trace.)
Here's the complete trace:
% arc --trace diff --jira HIVE-2831
Loading phutil library 'arc_jira_lib' from
'/Users/carl/Work/repos/hive4/.arc_jira_lib'...
>>> [0] conduit.connect()
<<< [0] 318,295 us
>>> [1] $ (cd '/Users/carl/Work/repos/hive4'; git rev-parse
--show-cdup)
<<< [1] 14,662 us
>>> [2] $ (cd '/Users/carl/Work/repos/hive4/'; git rev-parse
--verify HEAD^)
<<< [2] 16,343 us
>>> [3] $ (cd '/Users/carl/Work/repos/hive4/'; git log
--first-parent --format=medium 'HEAD^'..HEAD)
<<< [3] 15,040 us
>>> [4] differential.parsecommitmessage()
<<< [4] 547,222 us
Fatal error: Uncaught exception
'ArcanistDifferentialCommitMessageParserException' with message 'Invalid or
missing field 'Test Plan': You must provide a test plan.' in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php:88
Stack trace:
#0
/Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(88):
ArcanistDifferentialCommitMessage->pullDataFromConduit(Object(ConduitClient))
#1
/Users/carl/Work/repos/hive4/.arc_jira_lib/arcanist/ArcJIRAConfiguration.php(364):
ArcJIRAConfiguration->willRunDiffWorkflow()
#2 /Users/carl/.local/pkg/arcanist/scripts/arcanist.php(264):
ArcJIRAConfiguration->willRunWorkflow('diff', Object(ArcanistDiffWorkflow))
#3 {main}
thrown in
/Users/carl/.local/pkg/arcanist/src/differential/commitmessage/ArcanistDifferentialCommitMessage.php
on line 88
Anyone know what's going on here?
Thanks.
Carl
[jira] [Updated] (HIVE-192) DUPLICATE: Add TIMESTAMP column type for thrift dynamic_type
[ https://issues.apache.org/jira/browse/HIVE-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-192: Summary: DUPLICATE: Add TIMESTAMP column type for thrift dynamic_type (was: Add TIMESTAMP column type for thrift dynamic_type) This is a duplicate of HIVE-2272. The work contained in this ticket was never committed, and it should have been resolved as a duplicate. > DUPLICATE: Add TIMESTAMP column type for thrift dynamic_type > > > Key: HIVE-192 > URL: https://issues.apache.org/jira/browse/HIVE-192 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Johan Oskarsson >Assignee: Shyam Sundar Sarkar > Labels: SQL > Fix For: 0.8.0 > > Attachments: Hive-192.patch.txt, TIMESTAMP_specification.txt, > create_2.q.txt > > > create table something2 (test timestamp); > ERROR: DDL specifying type timestamp which has not been defined > java.lang.RuntimeException: specifying type timestamp which has not been > defined > at > org.apache.hadoop.hive.serde2.dynamic_type.thrift_grammar.FieldType(thrift_grammar.java:1879) > at > org.apache.hadoop.hive.serde2.dynamic_type.thrift_grammar.Field(thrift_grammar.java:1545) > at > org.apache.hadoop.hive.serde2.dynamic_type.thrift_grammar.FieldList(thrift_grammar.java:1501) > at > org.apache.hadoop.hive.serde2.dynamic_type.thrift_grammar.Struct(thrift_grammar.java:1171) > at > org.apache.hadoop.hive.serde2.dynamic_type.thrift_grammar.TypeDefinition(thrift_grammar.java:497) > at > org.apache.hadoop.hive.serde2.dynamic_type.thrift_grammar.Definition(thrift_grammar.java:439) > at > org.apache.hadoop.hive.serde2.dynamic_type.thrift_grammar.Start(thrift_grammar.java:101) > at > org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.initialize(DynamicSerDe.java:97) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:180) > at org.apache.hadoop.hive.ql.metadata.Table.initSerDe(Table.java:141) > at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:202) > at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:641) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:98) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:215) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:305) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2832) Cache error messages for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2832: Assignee: Kevin Wilfong Status: Patch Available (was: Open) > Cache error messages for additional logging > --- > > Key: HIVE-2832 > URL: https://issues.apache.org/jira/browse/HIVE-2832 > Project: Hive > Issue Type: Improvement > Components: Logging >Reporter: Kevin Wilfong >Assignee: Kevin Wilfong > Attachments: HIVE-2832.D2025.1.patch > > > It would be good if we could cache logs written to SessionState.err so that > they could be exposed to hooks for additional logging. This would allow > logging of error messages with the queries that failed in a central location. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2832) Cache error messages for additional logging
[ https://issues.apache.org/jira/browse/HIVE-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2832: -- Attachment: HIVE-2832.D2025.1.patch kevinwilfong requested code review of "HIVE-2832 [jira] Cache error messages for additional logging". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2832 Created a subclass of PrintStream which stores output written to its println method in a variable (in particular this will store all output written to the LogHelper class's printError method. I also created a new variable info in SessionState which, if set, will be used by LogHelper's printInfo method instead of SessionState.err, as logs written using this method aren't as significant. It would be good if we could cache logs written to SessionState.err so that they could be exposed to hooks for additional logging. This would allow logging of error messages with the queries that failed in a central location. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2025 AFFECTED FILES common/src/java/org/apache/hadoop/hive/common/io/CachingPrintStream.java cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java ql/src/test/results/clientnegative/cachingprintstream.q.out ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyCachingPrintStreamHook.java ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ql/src/test/queries/clientnegative/cachingprintstream.q ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4341/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Cache error messages for additional logging > --- > > Key: HIVE-2832 > URL: https://issues.apache.org/jira/browse/HIVE-2832 > Project: Hive > Issue Type: Improvement > Components: Logging >Reporter: Kevin Wilfong > Attachments: HIVE-2832.D2025.1.patch > > > It would be good if we could cache logs written to SessionState.err so that > they could be exposed to hooks for additional logging. This would allow > logging of error messages with the queries that failed in a central location. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2832) Cache error messages for additional logging
Cache error messages for additional logging --- Key: HIVE-2832 URL: https://issues.apache.org/jira/browse/HIVE-2832 Project: Hive Issue Type: Improvement Components: Logging Reporter: Kevin Wilfong It would be good if we could cache logs written to SessionState.err so that they could be exposed to hooks for additional logging. This would allow logging of error messages with the queries that failed in a central location. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1286 - Still Failing
Changes for Build #1280 Changes for Build #1281 Changes for Build #1282 Changes for Build #1283 Changes for Build #1284 [namit] HIVE-2825 Concatenating a partition does not inherit location from table (Kevin Wilfong via namit) Changes for Build #1285 Changes for Build #1286 No tests ran. The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1286) Status: Still Failing Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1286/ to view the results.
[jira] [Commented] (HIVE-2372) java.io.IOException: error=7, Argument list too long
[
https://issues.apache.org/jira/browse/HIVE-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219946#comment-13219946
]
Sergey Tryuber commented on HIVE-2372:
--
Edward, I've named my option hive.script.operator.truncate.env (instead of
hive.scriptoperator.truncate.env as you proposed), to be analogous of
hive.script.operator.id.env.var. Actually, I have a question, why do we have
hive-default.xml.template if all options (their names and default values) are
hardcoded in ConfVar?
Also, as I wrote in previous posts, there was some Hive tests failures in my
"ant test" before. I just want to share this information with other developers:
this problem was solved by changing locale in my system from Russian to EN. So,
I have all tests successfully completed.
And the last notice. Edward, could you add *.iml files to gitignore. I use Idea
as IDE and it creates all those files.
> java.io.IOException: error=7, Argument list too long
>
>
> Key: HIVE-2372
> URL: https://issues.apache.org/jira/browse/HIVE-2372
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Sergey Tryuber
>Priority: Critical
> Attachments: HIVE-2372.1.patch.txt, HIVE-2372.2.patch.txt
>
>
> I execute a huge query on a table with a lot of 2-level partitions. There is
> a perl reducer in my query. Maps worked ok, but every reducer fails with the
> following exception:
> 2011-08-11 04:58:29,865 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> Executing [/usr/bin/perl, , ]
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> tablename=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> partname=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> alias=null
> 2011-08-11 04:58:29,935 FATAL ExecReducer:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":129390185139228,"reducesinkkey1":"8AF163CA6F"},"value":{"_col0":"8AF163CA6F","_col1":"2011-07-27
>
> 22:48:52","_col2":129390185139228,"_col3":2006,"_col4":4100,"_col5":"10017388=6","_col6":1063,"_col7":"NULL","_col8":"address.com","_col9":"NULL","_col10":"NULL"},"alias":0}
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot
> initialize ScriptOperator
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:320)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
> ... 7 more
> Caused by: java.io.IOException: Cannot run program "/usr/bin/perl":
> java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:279)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list
> too long
> at java.lang.UNIXProcess.(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 16 more
> It seems to me, I found the cause. ScriptOperator.java puts a lot of configs
> as environment variables to the child reduce process. One of variables is
> mapred.input.dir, which in my case more than 150KB. There are a huge amount
> of input directories in this variable. In short, the problem is that Linux
> (up to 2.6.23 kernel version) limits summary size of environment variables
> for child processe
[jira] [Updated] (HIVE-2372) java.io.IOException: error=7, Argument list too long
[
https://issues.apache.org/jira/browse/HIVE-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Tryuber updated HIVE-2372:
-
Attachment: HIVE-2372.2.patch.txt
Patch that has hive.script.operator.truncate.env option that enables truncation.
> java.io.IOException: error=7, Argument list too long
>
>
> Key: HIVE-2372
> URL: https://issues.apache.org/jira/browse/HIVE-2372
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Sergey Tryuber
>Priority: Critical
> Attachments: HIVE-2372.1.patch.txt, HIVE-2372.2.patch.txt
>
>
> I execute a huge query on a table with a lot of 2-level partitions. There is
> a perl reducer in my query. Maps worked ok, but every reducer fails with the
> following exception:
> 2011-08-11 04:58:29,865 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> Executing [/usr/bin/perl, , ]
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> tablename=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> partname=null
> 2011-08-11 04:58:29,866 INFO org.apache.hadoop.hive.ql.exec.ScriptOperator:
> alias=null
> 2011-08-11 04:58:29,935 FATAL ExecReducer:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> {"key":{"reducesinkkey0":129390185139228,"reducesinkkey1":"8AF163CA6F"},"value":{"_col0":"8AF163CA6F","_col1":"2011-07-27
>
> 22:48:52","_col2":129390185139228,"_col3":2006,"_col4":4100,"_col5":"10017388=6","_col6":1063,"_col7":"NULL","_col8":"address.com","_col9":"NULL","_col10":"NULL"},"alias":0}
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:468)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot
> initialize ScriptOperator
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:320)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
> at
> org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
> ... 7 more
> Caused by: java.io.IOException: Cannot run program "/usr/bin/perl":
> java.io.IOException: error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at
> org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:279)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException: error=7, Argument list
> too long
> at java.lang.UNIXProcess.(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 16 more
> It seems to me, I found the cause. ScriptOperator.java puts a lot of configs
> as environment variables to the child reduce process. One of variables is
> mapred.input.dir, which in my case more than 150KB. There are a huge amount
> of input directories in this variable. In short, the problem is that Linux
> (up to 2.6.23 kernel version) limits summary size of environment variables
> for child processes to 132KB. This problem could be solved by upgrading the
> kernel. But strings limitations still be 132KB per string in environment
> variable. So such huge variable doesn't work even on my home computer
> (2.6.32). You can read more information on
> (http://www.kernel.org/doc/man-pages/online/pages/man2/execve.2.html).
> For now all our work has been stopped because of this problem and I can't
> find the solution. The only solution, which seems to me more reasonable is to
> get rid of this variable in reducers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://
