[
https://issues.apache.org/jira/browse/HIVE-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-2866:
------------------------------
Attachment: HIVE-2866.D2277.1.patch
kevinwilfong requested code review of "HIVE-2866 [jira] Cache local map reduce
job errors for additional logging".
Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-2866
The error stream from subprocesses where local map reduce jobs are run are
now passed through a Caching PrintStream. If the job fails, the output written
to this print stream are stored in the SessionState in a map from stage ID to a
list of strings, which represents the lines of output written to the error
stream. This way, these can be logged along with the failed query in some
central location for easier debugging.
Using the CachingPrintStream we are storing errors that occur locally in
Hive, but because local map reduce jobs are run in a separate JVM we are not
storing that occur for these. We can use this same construct to store errors
written to the subprocesses error stream. This way, when we log failed
queries, these will give us a decent idea of why those queries failed.
See related issues:
https://issues.apache.org/jira/browse/HIVE-2832
https://issues.apache.org/jira/browse/HIVE-2858
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D2277
AFFECTED FILES
data/scripts/cat_error.py
common/src/java/org/apache/hadoop/hive/common/io/CachingPrintStream.java
ql/src/test/results/clientnegative/local_mapred_error_cache.q.out
ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifySessionStateLocalErrorsHook.java
ql/src/test/queries/clientnegative/local_mapred_error_cache.q
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java
ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/5061/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Cache local map reduce job errors for additional logging
> --------------------------------------------------------
>
> Key: HIVE-2866
> URL: https://issues.apache.org/jira/browse/HIVE-2866
> Project: Hive
> Issue Type: Improvement
> Components: Logging
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2866.D2277.1.patch
>
>
> Using the CachingPrintStream we are storing errors that occur locally in
> Hive, but because local map reduce jobs are run in a separate JVM we are not
> storing that occur for these. We can use this same construct to store errors
> written to the subprocesses error stream. This way, when we log failed
> queries, these will give us a decent idea of why those queries failed.
> See related issues:
> https://issues.apache.org/jira/browse/HIVE-2832
> https://issues.apache.org/jira/browse/HIVE-2858
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
