[jira] [Created] (HIVE-16946) Information Schema Improvements
Gunther Hagleitner created HIVE-16946: - Summary: Information Schema Improvements Key: HIVE-16946 URL: https://issues.apache.org/jira/browse/HIVE-16946 Project: Hive Issue Type: Improvement Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Collection of requested enhancements and fixes for the info schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-16657) Replace conf variable with checkDatabaseProduct-like in JDBCStorageHandler
Gunther Hagleitner created HIVE-16657: - Summary: Replace conf variable with checkDatabaseProduct-like in JDBCStorageHandler Key: HIVE-16657 URL: https://issues.apache.org/jira/browse/HIVE-16657 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16631) Non-native tables should not enter stats state "accurate"
Gunther Hagleitner created HIVE-16631: - Summary: Non-native tables should not enter stats state "accurate" Key: HIVE-16631 URL: https://issues.apache.org/jira/browse/HIVE-16631 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner With non-native tables Hive is usually not the only system managing records. It's quite common to use XYZStorageHandler point to an existing data source. In these cases we can erroneously set the state of the statistics to accurate, which leads us to answer queries incorrectly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16630) Make connection pool pluggable in JdbcStorageHandler
Gunther Hagleitner created HIVE-16630: - Summary: Make connection pool pluggable in JdbcStorageHandler Key: HIVE-16630 URL: https://issues.apache.org/jira/browse/HIVE-16630 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Currently JdbcStorageHandler can only use DBCP. Would be nice to have that pluggable. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15873) Remove Windows-specific code
Gunther Hagleitner created HIVE-15873: - Summary: Remove Windows-specific code Key: HIVE-15873 URL: https://issues.apache.org/jira/browse/HIVE-15873 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner I know a lot of work has gone initially into supporting UT, runtime, etc on Windows, but this code seems to have been rotting. There have been no updates to the windows specific test files, or any new code to keep the new features compatible. We're also not running the tests or builds on windows. That is really an impediment to keepting that code healthy. The code is sprinkled all over the codebase. Makes it hard to maintain. I think we're better off removing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15791) Remove unused ant files
Gunther Hagleitner created HIVE-15791: - Summary: Remove unused ant files Key: HIVE-15791 URL: https://issues.apache.org/jira/browse/HIVE-15791 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15790) Remove unused beeline golden files
Gunther Hagleitner created HIVE-15790: - Summary: Remove unused beeline golden files Key: HIVE-15790 URL: https://issues.apache.org/jira/browse/HIVE-15790 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15549) Better naming of Tez edges
Gunther Hagleitner created HIVE-15549: - Summary: Better naming of Tez edges Key: HIVE-15549 URL: https://issues.apache.org/jira/browse/HIVE-15549 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Do the following renames: CUSTOM_EDGE -> CO_PARTITION_EDGE CUSTOM_SIMPLE_EDGE -> PARTITION_EDGE SIMPLE_EDGE -> SORT_PARTITION_EDGE Because that's what those edges actually do. Also rename Map/Reduce to just Vertex . These vertices haven't mapped or reduced in a long time. The names are leftover items from MR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
Gunther Hagleitner created HIVE-14117: - Summary: HS2 UI: List of recent queries shows most recent query last Key: HIVE-14117 URL: https://issues.apache.org/jira/browse/HIVE-14117 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13823) Remove unnecessary log line in common join operator
Gunther Hagleitner created HIVE-13823: - Summary: Remove unnecessary log line in common join operator Key: HIVE-13823 URL: https://issues.apache.org/jira/browse/HIVE-13823 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 2.1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13467) Show llap info on hs2 ui when available
Gunther Hagleitner created HIVE-13467: - Summary: Show llap info on hs2 ui when available Key: HIVE-13467 URL: https://issues.apache.org/jira/browse/HIVE-13467 Project: Hive Issue Type: Bug Components: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner When llap is on and hs2 is configured with access to an llap cluster, HS2 UI should show some status of the daemons and provide a mechanism to click through to their respective UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13417) Some vector operators return "OP" as name
Gunther Hagleitner created HIVE-13417: - Summary: Some vector operators return "OP" as name Key: HIVE-13417 URL: https://issues.apache.org/jira/browse/HIVE-13417 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Select/Group by/Filter/etc need to return the same name whether they are the regular or the vector operators. If they don't the regular path matching in our optimizer code doesn't work on them. >From the code it looks an attempt was made to follow this - unfortunately >getOperatorName is static and polymorphism doesn't work on these functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13162) Fixes for LlapDump and FileSinkoperator
Gunther Hagleitner created HIVE-13162: - Summary: Fixes for LlapDump and FileSinkoperator Key: HIVE-13162 URL: https://issues.apache.org/jira/browse/HIVE-13162 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: llap -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13133) Create initial InputFormat + record readers/writers
Gunther Hagleitner created HIVE-13133: - Summary: Create initial InputFormat + record readers/writers Key: HIVE-13133 URL: https://issues.apache.org/jira/browse/HIVE-13133 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12357) Allow user to set tez job name
Gunther Hagleitner created HIVE-12357: - Summary: Allow user to set tez job name Key: HIVE-12357 URL: https://issues.apache.org/jira/browse/HIVE-12357 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Need something like mapred.job.name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11727) Hive on Tez through Oozie: Some queries fail with fnf exception
Gunther Hagleitner created HIVE-11727:
-
Summary: Hive on Tez through Oozie: Some queries fail with fnf
exception
Key: HIVE-11727
URL: https://issues.apache.org/jira/browse/HIVE-11727
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
When we read back row containers from disk, a misconfiguration causes us to
look for a non-existing file.
{noformat}
Caused by: java.io.FileNotFoundException: File
file:/grid/0/hadoop/yarn/local/usercache/appcache/application_1440685000561_0028/container_e26_1440685000561_0028_01_05/container_tokens
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:608)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:821)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:598)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:414)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:169)
... 31 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10957) QueryPlan's start time is incorrect in certain cases
Gunther Hagleitner created HIVE-10957: - Summary: QueryPlan's start time is incorrect in certain cases Key: HIVE-10957 URL: https://issues.apache.org/jira/browse/HIVE-10957 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Wei Zheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10853) Create ExplainTask in ATS hook through ExplainWork
Gunther Hagleitner created HIVE-10853: - Summary: Create ExplainTask in ATS hook through ExplainWork Key: HIVE-10853 URL: https://issues.apache.org/jira/browse/HIVE-10853 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Right now ExplainTask is created directly. That's fragile and can lead to stuff like: HIVE-10829 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10284) enable container reuse for grace hash join
Gunther Hagleitner created HIVE-10284: - Summary: enable container reuse for grace hash join Key: HIVE-10284 URL: https://issues.apache.org/jira/browse/HIVE-10284 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10274) Send context and description to tez via dag info
Gunther Hagleitner created HIVE-10274: - Summary: Send context and description to tez via dag info Key: HIVE-10274 URL: https://issues.apache.org/jira/browse/HIVE-10274 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-10274.1.patch tez has a way to specify context and description (which is shown in the ui) for each dag. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10203) Move fragment execution onto a thread pool
Gunther Hagleitner created HIVE-10203: - Summary: Move fragment execution onto a thread pool Key: HIVE-10203 URL: https://issues.apache.org/jira/browse/HIVE-10203 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Currently we use a lot of thread local magic to make things work in llap. MRLegacyInput makes sure we use the same thread in input and processor. The execution spawns a new thread per fragment and kills it afterwards. That's being done because of Tez/MapredContext, ExecContext and IOContext. These need to be shared for a single fragment but not across. Way to clean this up is probably by creating a HiveInput that can handle the config + state sharing more efficiently than generic Tez code. cc [~sseth] [~gopalv] [~vikram.dixit] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10078) Optionally allow logging of records processed in fixed intervals
Gunther Hagleitner created HIVE-10078: - Summary: Optionally allow logging of records processed in fixed intervals Key: HIVE-10078 URL: https://issues.apache.org/jira/browse/HIVE-10078 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Tasks today log progress (records in/records out) on an exponential scale (1, 10, 100, ...). Sometimes it's helpful to be able to switch to fixed interval. That can help debugging certain issues that look like a hang, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10005) remove some unnecessary branches from the inner loop
Gunther Hagleitner created HIVE-10005: - Summary: remove some unnecessary branches from the inner loop Key: HIVE-10005 URL: https://issues.apache.org/jira/browse/HIVE-10005 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-10005.1.patch Operator.forward is doing too much. There's no reason to do the done checking per row and update it inline. It's much more efficient to just do that when the event that completes an operator happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10002) fix yarn service registry not found in ut problem
Gunther Hagleitner created HIVE-10002: - Summary: fix yarn service registry not found in ut problem Key: HIVE-10002 URL: https://issues.apache.org/jira/browse/HIVE-10002 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10003) MiniTez ut fail with missing configs
Gunther Hagleitner created HIVE-10003: - Summary: MiniTez ut fail with missing configs Key: HIVE-10003 URL: https://issues.apache.org/jira/browse/HIVE-10003 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gopal V -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10004) yarn service registry should be shim'd
Gunther Hagleitner created HIVE-10004: - Summary: yarn service registry should be shim'd Key: HIVE-10004 URL: https://issues.apache.org/jira/browse/HIVE-10004 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gopal V -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10000) 10000 whoooohooo
Gunther Hagleitner created HIVE-1: - Summary: 1 whhooo Key: HIVE-1 URL: https://issues.apache.org/jira/browse/HIVE-1 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9984) JoinReorder's getOutputSize is exponential
Gunther Hagleitner created HIVE-9984: Summary: JoinReorder's getOutputSize is exponential Key: HIVE-9984 URL: https://issues.apache.org/jira/browse/HIVE-9984 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Found by [~mmokhtar]. Causes major issues in large plans (50+ joins). Simple fix would be to memoize the recursion. There should also be a flag to switch this opt off. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9971) Clean up operator class
Gunther Hagleitner created HIVE-9971: Summary: Clean up operator class Key: HIVE-9971 URL: https://issues.apache.org/jira/browse/HIVE-9971 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner This is mostly cleanup although it does enhance the pipeline in one respect. It introduces asyn init for operators and uses it for hash table loading where desired. There's a bunch of weird code associated with the operator class: - initialize isn't recursive, rather initializeOp is supposed to call initializeChildren. That has led to bugs in the past. - setExecContext and passExecContext. Both are recursive, but passExecContext calls setExecContext and then recurses again. Boo. - lots of (getChildren() != null) although that can't happen anymore - TezCacheAccess is a hack. We should just leave init of inputs up to the operator that needs it. - Need some sanity checks that make sure that operators were all initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9915) Allow specifying file format for managed tables
Gunther Hagleitner created HIVE-9915: Summary: Allow specifying file format for managed tables Key: HIVE-9915 URL: https://issues.apache.org/jira/browse/HIVE-9915 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner We already allow setting a system wide default format. In some cases it's useful though to specify this only for managed tables, or distinguish external and managed via two variables. You might want to set a more efficient (than text) format for managed tables, but leave external to text (as they often are log files etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9891) LLAP: disable plan caching
Gunther Hagleitner created HIVE-9891: Summary: LLAP: disable plan caching Key: HIVE-9891 URL: https://issues.apache.org/jira/browse/HIVE-9891 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Can't share the same plan objects in LLAP as they are used concurrently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9850) documentation for llap
Gunther Hagleitner created HIVE-9850: Summary: documentation for llap Key: HIVE-9850 URL: https://issues.apache.org/jira/browse/HIVE-9850 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Live long and write documentation. This is the jira for all doc related things in llap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9849) LLAP: add multi threaded object registry
Gunther Hagleitner created HIVE-9849: Summary: LLAP: add multi threaded object registry Key: HIVE-9849 URL: https://issues.apache.org/jira/browse/HIVE-9849 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9837) LLAP: Decision to use llap or uber is being lost in some reducers
Gunther Hagleitner created HIVE-9837: Summary: LLAP: Decision to use llap or uber is being lost in some reducers Key: HIVE-9837 URL: https://issues.apache.org/jira/browse/HIVE-9837 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9810) LLAP: prep object registry for multi threading
Gunther Hagleitner created HIVE-9810: Summary: LLAP: prep object registry for multi threading Key: HIVE-9810 URL: https://issues.apache.org/jira/browse/HIVE-9810 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner The object registry relies on the fact that only one thread at a time is active in a container. With llap that's not the case. There's multiple threads that will try to generate the same cache object at the time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9801) LLAP: need counter for cache hit ratio
Gunther Hagleitner created HIVE-9801: Summary: LLAP: need counter for cache hit ratio Key: HIVE-9801 URL: https://issues.apache.org/jira/browse/HIVE-9801 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9776) LLAP: add simple way to determine wether you're running in daemon
Gunther Hagleitner created HIVE-9776: Summary: LLAP: add simple way to determine wether you're running in daemon Key: HIVE-9776 URL: https://issues.apache.org/jira/browse/HIVE-9776 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9778) LLAP: add config var for uber mode
Gunther Hagleitner created HIVE-9778: Summary: LLAP: add config var for uber mode Key: HIVE-9778 URL: https://issues.apache.org/jira/browse/HIVE-9778 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9756) LLAP: use log4j 2 for llap
Gunther Hagleitner created HIVE-9756: Summary: LLAP: use log4j 2 for llap Key: HIVE-9756 URL: https://issues.apache.org/jira/browse/HIVE-9756 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gopal V For the INFO logging, we'll need to use the log4j-jcl 2.x upgrade-path to get throughput friendly logging. http://logging.apache.org/log4j/2.0/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9750) LLAP: avoid log locks in operators
Gunther Hagleitner created HIVE-9750: Summary: LLAP: avoid log locks in operators Key: HIVE-9750 URL: https://issues.apache.org/jira/browse/HIVE-9750 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Basically wrap all LOG.xx calls in isLogXXXEnabled to avoid unnecessary locks on these calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9762) LLAP: hoop up decider + dag utils
Gunther Hagleitner created HIVE-9762: Summary: LLAP: hoop up decider + dag utils Key: HIVE-9762 URL: https://issues.apache.org/jira/browse/HIVE-9762 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9765) LLAP: uber mode where applicable
Gunther Hagleitner created HIVE-9765: Summary: LLAP: uber mode where applicable Key: HIVE-9765 URL: https://issues.apache.org/jira/browse/HIVE-9765 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner when task is small enough and it's the last reduce stage we can just run it in the AM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9694) LLAP: add check for udfs/udafs to llapdecider
[ https://issues.apache.org/jira/browse/HIVE-9694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9694: - Attachment: HIVE-9694.1.patch LLAP: add check for udfs/udafs to llapdecider - Key: HIVE-9694 URL: https://issues.apache.org/jira/browse/HIVE-9694 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9694.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9694) LLAP: add check for udfs/udafs to llapdecider
[ https://issues.apache.org/jira/browse/HIVE-9694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-9694. -- Resolution: Fixed Committed to branch. LLAP: add check for udfs/udafs to llapdecider - Key: HIVE-9694 URL: https://issues.apache.org/jira/browse/HIVE-9694 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9694.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9694) add check for udfs/udafs to llapdecider
Gunther Hagleitner created HIVE-9694: Summary: add check for udfs/udafs to llapdecider Key: HIVE-9694 URL: https://issues.apache.org/jira/browse/HIVE-9694 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9694) LLAP: add check for udfs/udafs to llapdecider
[ https://issues.apache.org/jira/browse/HIVE-9694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9694: - Summary: LLAP: add check for udfs/udafs to llapdecider (was: add check for udfs/udafs to llapdecider) LLAP: add check for udfs/udafs to llapdecider - Key: HIVE-9694 URL: https://issues.apache.org/jira/browse/HIVE-9694 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9635) LLAP: I'm the decider
[ https://issues.apache.org/jira/browse/HIVE-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-9635. -- Resolution: Fixed Committed to branch. LLAP: I'm the decider - Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9635.1.patch, HIVE-9635.2.patch https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9673) Set operationhandle in ATS entities for lookups
[ https://issues.apache.org/jira/browse/HIVE-9673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9673: - Issue Type: Improvement (was: Bug) Set operationhandle in ATS entities for lookups --- Key: HIVE-9673 URL: https://issues.apache.org/jira/browse/HIVE-9673 Project: Hive Issue Type: Improvement Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 1.2.0 Attachments: HIVE-9673.1.patch, HIVE-9673.2.patch Yarn App Timeline Server (ATS) users can find their query using hive query-id. However, query id is available only through the logs at the moment. Thrift api users such as Hue have another unique id for queries, which the operation handle contains (TExecuteStatementResp.TOperationHandle.THandleIdentifier.guid) . Adding the operationhandle guid to ATS will enable such thrift users to get information from ATS for the queries that they have spawned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9673) Set operationhandle in ATS entities for lookups
[ https://issues.apache.org/jira/browse/HIVE-9673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9673: - Fix Version/s: 1.2.0 Set operationhandle in ATS entities for lookups --- Key: HIVE-9673 URL: https://issues.apache.org/jira/browse/HIVE-9673 Project: Hive Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 1.2.0 Attachments: HIVE-9673.1.patch, HIVE-9673.2.patch Yarn App Timeline Server (ATS) users can find their query using hive query-id. However, query id is available only through the logs at the moment. Thrift api users such as Hue have another unique id for queries, which the operation handle contains (TExecuteStatementResp.TOperationHandle.THandleIdentifier.guid) . Adding the operationhandle guid to ATS will enable such thrift users to get information from ATS for the queries that they have spawned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9673) Set operationhandle in ATS entities for lookups
[ https://issues.apache.org/jira/browse/HIVE-9673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9673: - Resolution: Fixed Status: Resolved (was: Patch Available) committed to trunk. thanks [~thejas]! Set operationhandle in ATS entities for lookups --- Key: HIVE-9673 URL: https://issues.apache.org/jira/browse/HIVE-9673 Project: Hive Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-9673.1.patch, HIVE-9673.2.patch Yarn App Timeline Server (ATS) users can find their query using hive query-id. However, query id is available only through the logs at the moment. Thrift api users such as Hue have another unique id for queries, which the operation handle contains (TExecuteStatementResp.TOperationHandle.THandleIdentifier.guid) . Adding the operationhandle guid to ATS will enable such thrift users to get information from ATS for the queries that they have spawned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9683) Hive metastore thrift client connections hang indefinitely
[ https://issues.apache.org/jira/browse/HIVE-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320486#comment-14320486 ] Gunther Hagleitner commented on HIVE-9683: -- [~vikram.dixit] ok for 1.0 branch? Hive metastore thrift client connections hang indefinitely -- Key: HIVE-9683 URL: https://issues.apache.org/jira/browse/HIVE-9683 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 1.0.0, 1.0.1 Reporter: Gopal V Assignee: Gopal V Priority: Minor Fix For: 1.0.1 Attachments: HIVE-9683.1.patch THRIFT-2788 fixed network-partition problems that affect Thrift client connections. Since hive-1.0 is on thrift-0.9.0 which is affected by the bug, a workaround can be applied to prevent indefinite connection hangs during net-splits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9635) LLAP: I'm the decider
[ https://issues.apache.org/jira/browse/HIVE-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318807#comment-14318807 ] Gunther Hagleitner commented on HIVE-9635: -- If you like, not necessary though. I do need to finish it though. LLAP: I'm the decider - Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9635.1.patch https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9665) Parallel move task optimization causes race condition
[
https://issues.apache.org/jira/browse/HIVE-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-9665:
-
Status: Patch Available (was: Open)
Parallel move task optimization causes race condition
-
Key: HIVE-9665
URL: https://issues.apache.org/jira/browse/HIVE-9665
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Critical
Attachments: HIVE-9665.1.patch
The change in HIVE-8042 doesn't actually work. Running it at scale produces
race conditions which lead to broken thrift messages and OOMs. E.g.:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
[jira] [Commented] (HIVE-9673) Set operationhandle in ATS entities for lookups
[
https://issues.apache.org/jira/browse/HIVE-9673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319360#comment-14319360
]
Gunther Hagleitner commented on HIVE-9673:
--
Looks good - but how do you know that none of the following objs are going to
be null (handle/id/...)?
{noformat}
+ String guid64 =
Base64.encodeBase64URLSafeString(getHandle().getHandleIdentifier()
+ .toTHandleIdentifier().getGuid()).trim();
{noformat}
Set operationhandle in ATS entities for lookups
---
Key: HIVE-9673
URL: https://issues.apache.org/jira/browse/HIVE-9673
Project: Hive
Issue Type: Bug
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-9673.1.patch, HIVE-9673.2.patch
Yarn App Timeline Server (ATS) users can find their query using hive query-id.
However, query id is available only through the logs at the moment.
Thrift api users such as Hue have another unique id for queries, which the
operation handle contains
(TExecuteStatementResp.TOperationHandle.THandleIdentifier.guid) . Adding the
operationhandle guid to ATS will enable such thrift users to get information
from ATS for the queries that they have spawned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9683) Hive metastore thrift client connections hang indefinitely
[ https://issues.apache.org/jira/browse/HIVE-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319584#comment-14319584 ] Gunther Hagleitner commented on HIVE-9683: -- +1 Hive metastore thrift client connections hang indefinitely -- Key: HIVE-9683 URL: https://issues.apache.org/jira/browse/HIVE-9683 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 1.0.0, 1.0.1 Reporter: Gopal V Assignee: Gopal V Priority: Minor Fix For: 1.0.1 Attachments: HIVE-9683.1.patch THRIFT-2788 fixed network-partition problems that affect Thrift client connections. Since hive-1.0 is on thrift-0.9.0 which is affected by the bug, a workaround can be applied to prevent indefinite connection hangs during net-splits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9665) Parallel move task optimization causes race condition
[
https://issues.apache.org/jira/browse/HIVE-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-9665:
-
Resolution: Fixed
Fix Version/s: 1.2.0
Status: Resolved (was: Patch Available)
Parallel move task optimization causes race condition
-
Key: HIVE-9665
URL: https://issues.apache.org/jira/browse/HIVE-9665
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Critical
Fix For: 1.2.0
Attachments: HIVE-9665.1.patch
The change in HIVE-8042 doesn't actually work. Running it at scale produces
race conditions which lead to broken thrift messages and OOMs. E.g.:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
[jira] [Commented] (HIVE-9665) Parallel move task optimization causes race condition
[
https://issues.apache.org/jira/browse/HIVE-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319473#comment-14319473
]
Gunther Hagleitner commented on HIVE-9665:
--
Committed to trunk. Thanks [~thejas].
Parallel move task optimization causes race condition
-
Key: HIVE-9665
URL: https://issues.apache.org/jira/browse/HIVE-9665
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Critical
Fix For: 1.2.0
Attachments: HIVE-9665.1.patch
The change in HIVE-8042 doesn't actually work. Running it at scale produces
race conditions which lead to broken thrift messages and OOMs. E.g.:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
[jira] [Commented] (HIVE-9673) Set operationhandle in ATS entities for lookups
[ https://issues.apache.org/jira/browse/HIVE-9673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319507#comment-14319507 ] Gunther Hagleitner commented on HIVE-9673: -- +1 Set operationhandle in ATS entities for lookups --- Key: HIVE-9673 URL: https://issues.apache.org/jira/browse/HIVE-9673 Project: Hive Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-9673.1.patch, HIVE-9673.2.patch Yarn App Timeline Server (ATS) users can find their query using hive query-id. However, query id is available only through the logs at the moment. Thrift api users such as Hue have another unique id for queries, which the operation handle contains (TExecuteStatementResp.TOperationHandle.THandleIdentifier.guid) . Adding the operationhandle guid to ATS will enable such thrift users to get information from ATS for the queries that they have spawned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9635) LLAP: I'm the decider
[ https://issues.apache.org/jira/browse/HIVE-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9635: - Attachment: HIVE-9635.2.patch LLAP: I'm the decider - Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9635.1.patch, HIVE-9635.2.patch https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9665) Parallel move task optimization causes race condition
Gunther Hagleitner created HIVE-9665:
Summary: Parallel move task optimization causes race condition
Key: HIVE-9665
URL: https://issues.apache.org/jira/browse/HIVE-9665
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Critical
The change in HIVE-8042 doesn't actually work. Running it at scale produces
race conditions which lead to broken thrift messages and OOMs. E.g.:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
[jira] [Updated] (HIVE-9665) Parallel move task optimization causes race condition
[
https://issues.apache.org/jira/browse/HIVE-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-9665:
-
Attachment: HIVE-9665.1.patch
Parallel move task optimization causes race condition
-
Key: HIVE-9665
URL: https://issues.apache.org/jira/browse/HIVE-9665
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Critical
The change in HIVE-8042 doesn't actually work. Running it at scale produces
race conditions which lead to broken thrift messages and OOMs. E.g.:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
[jira] [Updated] (HIVE-9665) Parallel move task optimization causes race condition
[
https://issues.apache.org/jira/browse/HIVE-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-9665:
-
Attachment: (was: HIVE-9665.1.patch)
Parallel move task optimization causes race condition
-
Key: HIVE-9665
URL: https://issues.apache.org/jira/browse/HIVE-9665
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Critical
The change in HIVE-8042 doesn't actually work. Running it at scale produces
race conditions which lead to broken thrift messages and OOMs. E.g.:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
[jira] [Updated] (HIVE-9665) Parallel move task optimization causes race condition
[
https://issues.apache.org/jira/browse/HIVE-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-9665:
-
Attachment: HIVE-9665.1.patch
Parallel move task optimization causes race condition
-
Key: HIVE-9665
URL: https://issues.apache.org/jira/browse/HIVE-9665
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Critical
Attachments: HIVE-9665.1.patch
The change in HIVE-8042 doesn't actually work. Running it at scale produces
race conditions which lead to broken thrift messages and OOMs. E.g.:
{noformat}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1091)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy9.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1064)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1019)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1006)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:250)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:353)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:215)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1122)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1108)
at
[jira] [Updated] (HIVE-9635) LLAP: I'm the decider
[ https://issues.apache.org/jira/browse/HIVE-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9635: - Attachment: HIVE-9635.1.patch first draft - doesn't handle udfs yet. LLAP: I'm the decider - Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9635.1.patch https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9635) LLAP: I'm the decider
Gunther Hagleitner created HIVE-9635: Summary: LLAP: I'm the decider Key: HIVE-9635 URL: https://issues.apache.org/jira/browse/HIVE-9635 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner https://www.youtube.com/watch?v=r8VbzrZ9yHQ Physical optimizer to choose what to run inside/outside llap. Tests first whether user code has to be shipped then if the specific query fragment is suitable to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9614) Encrypt mapjoin tables
[ https://issues.apache.org/jira/browse/HIVE-9614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314621#comment-14314621 ] Gunther Hagleitner commented on HIVE-9614: -- 3 months sprints? i believe that's called a marathon. Encrypt mapjoin tables -- Key: HIVE-9614 URL: https://issues.apache.org/jira/browse/HIVE-9614 Project: Hive Issue Type: Sub-task Reporter: Brock Noland When performing a MapJoin we store data which is put in the distributed cache which is stored on local disk. Ideally we would encrypt these tables to the same degree that the they are encrypted in HDFS or find some other way to ensure they are encrypted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9529) alter table .. concatenate under Tez mode should create TezTask
[ https://issues.apache.org/jira/browse/HIVE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302091#comment-14302091 ] Gunther Hagleitner commented on HIVE-9529: -- +1 alter table .. concatenate under Tez mode should create TezTask - Key: HIVE-9529 URL: https://issues.apache.org/jira/browse/HIVE-9529 Project: Hive Issue Type: Bug Affects Versions: 1.0.0, 1.2.0, 1.1.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Attachments: HIVE-9529.1.patch, HIVE-9529.2.patch alter table .. concatenate DDL command creates MR task by default. When hive cli is launched with execution engine as tez, the scheduling of the MR task for file merging could be delayed until tez session expiration. This happens because YARN will not have capacity to launch another AppMaster for MR task. We should create tez task to overcome this. When the execution engine is tez TezTask will be created else MRTask will be created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9506) LLAP: Add an execution daemon
[ https://issues.apache.org/jira/browse/HIVE-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-9506. -- Resolution: Fixed Committed to branch. Thanks [~sseth]! LLAP: Add an execution daemon - Key: HIVE-9506 URL: https://issues.apache.org/jira/browse/HIVE-9506 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-9506.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9188) BloomFilter in ORC row group index
[ https://issues.apache.org/jira/browse/HIVE-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298021#comment-14298021 ] Gunther Hagleitner commented on HIVE-9188: -- should look at test failures (probably unrelated). Otherwise: +1 BloomFilter in ORC row group index -- Key: HIVE-9188 URL: https://issues.apache.org/jira/browse/HIVE-9188 Project: Hive Issue Type: New Feature Components: File Formats Affects Versions: 0.15.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Labels: orcfile Attachments: HIVE-9188.1.patch, HIVE-9188.2.patch, HIVE-9188.3.patch, HIVE-9188.4.patch, HIVE-9188.5.patch BloomFilters are well known probabilistic data structure for set membership checking. We can use bloom filters in ORC index for better row group pruning. Currently, ORC row group index uses min/max statistics to eliminate row groups (stripes as well) that do not satisfy predicate condition specified in the query. But in some cases, the efficiency of min/max based elimination is not optimal (unsorted columns with wide range of entries). Bloom filters can be an effective and efficient alternative for row group/split elimination for point queries or queries with IN clause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8307) null character in columns.comments schema property breaks jobconf.xml
[ https://issues.apache.org/jira/browse/HIVE-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296469#comment-14296469 ] Gunther Hagleitner commented on HIVE-8307: -- Oh yeah, I remember. But presumably you could have a non-printable or null byte in a column comment too (parser allows it, i think). So escaping might be safer yet. Also, since we use a Properties object it seems forgivable to assume that the object is serializable. Might be less of a gotcha if we change it, no? null character in columns.comments schema property breaks jobconf.xml - Key: HIVE-8307 URL: https://issues.apache.org/jira/browse/HIVE-8307 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0, 0.13.1 Reporter: Carl Laird It would appear that the fix for https://issues.apache.org/jira/browse/HIVE-6681 is causing the null character to show up in job config xml files: I get the following when trying to insert into an elasticsearch backed table: [Fatal Error] :336:51: Character reference # 14/06/17 14:40:11 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # Exception in thread main java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1263) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1129) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1063) at org.apache.hadoop.conf.Configuration.get(Configuration.java:416) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:604) at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:1273) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:667) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:251) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:300) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1181) ... 11 more Execution failed with exit status: 1 Line 336 of jobconf.xml: propertynamecolumns.comments/namevalue#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;/value/property See https://groups.google.com/forum/#!msg/mongodb-user/lKbha0SzMP8/jvE8ZrJom4AJ for more discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8307) null character in columns.comments schema property breaks jobconf.xml
[ https://issues.apache.org/jira/browse/HIVE-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295634#comment-14295634 ] Gunther Hagleitner commented on HIVE-8307: -- Meant to say [~ashutoshc] null character in columns.comments schema property breaks jobconf.xml - Key: HIVE-8307 URL: https://issues.apache.org/jira/browse/HIVE-8307 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0, 0.13.1 Reporter: Carl Laird It would appear that the fix for https://issues.apache.org/jira/browse/HIVE-6681 is causing the null character to show up in job config xml files: I get the following when trying to insert into an elasticsearch backed table: [Fatal Error] :336:51: Character reference # 14/06/17 14:40:11 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # Exception in thread main java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1263) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1129) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1063) at org.apache.hadoop.conf.Configuration.get(Configuration.java:416) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:604) at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:1273) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:667) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:251) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:300) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1181) ... 11 more Execution failed with exit status: 1 Line 336 of jobconf.xml: propertynamecolumns.comments/namevalue#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;/value/property See https://groups.google.com/forum/#!msg/mongodb-user/lKbha0SzMP8/jvE8ZrJom4AJ for more discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8307) null character in columns.comments schema property breaks jobconf.xml
[ https://issues.apache.org/jira/browse/HIVE-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295633#comment-14295633 ] Gunther Hagleitner commented on HIVE-8307: -- It's true that it's fixable on their end, but the null byte in column.comments is ugly too. @Ashutosh - I think you put that in, what was that needed for? null character in columns.comments schema property breaks jobconf.xml - Key: HIVE-8307 URL: https://issues.apache.org/jira/browse/HIVE-8307 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0, 0.13.1 Reporter: Carl Laird It would appear that the fix for https://issues.apache.org/jira/browse/HIVE-6681 is causing the null character to show up in job config xml files: I get the following when trying to insert into an elasticsearch backed table: [Fatal Error] :336:51: Character reference # 14/06/17 14:40:11 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # Exception in thread main java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1263) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1129) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1063) at org.apache.hadoop.conf.Configuration.get(Configuration.java:416) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:604) at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:1273) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:667) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.xml.sax.SAXParseException; lineNumber: 336; columnNumber: 51; Character reference # at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:251) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:300) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1181) ... 11 more Execution failed with exit status: 1 Line 336 of jobconf.xml: propertynamecolumns.comments/namevalue#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;#0;/value/property See https://groups.google.com/forum/#!msg/mongodb-user/lKbha0SzMP8/jvE8ZrJom4AJ for more discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9461) LLAP: Enable local mode tests on tez to facilitate llap testing
[ https://issues.apache.org/jira/browse/HIVE-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-9461. -- Resolution: Fixed LLAP: Enable local mode tests on tez to facilitate llap testing --- Key: HIVE-9461 URL: https://issues.apache.org/jira/browse/HIVE-9461 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9461.1.patch local mode tez tests will help the testing of multiple fragments running at the same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9460) LLAP: Fix some static vars in the operator pipeline
[ https://issues.apache.org/jira/browse/HIVE-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-9460. -- Resolution: Fixed LLAP: Fix some static vars in the operator pipeline --- Key: HIVE-9460 URL: https://issues.apache.org/jira/browse/HIVE-9460 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9460.1.patch There are a few static vars left in the operator pipeline. Can't have those with multi-threaded execution... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9461) LLAP: Enable local mode tests on tez to facilitate llap testing
Gunther Hagleitner created HIVE-9461: Summary: LLAP: Enable local mode tests on tez to facilitate llap testing Key: HIVE-9461 URL: https://issues.apache.org/jira/browse/HIVE-9461 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner local mode tez tests will help the testing of multiple fragments running at the same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9461) LLAP: Enable local mode tests on tez to facilitate llap testing
[ https://issues.apache.org/jira/browse/HIVE-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9461: - Attachment: (was: HIVE-9460.1.patch) LLAP: Enable local mode tests on tez to facilitate llap testing --- Key: HIVE-9461 URL: https://issues.apache.org/jira/browse/HIVE-9461 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9461.1.patch local mode tez tests will help the testing of multiple fragments running at the same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9461) LLAP: Enable local mode tests on tez to facilitate llap testing
[ https://issues.apache.org/jira/browse/HIVE-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9461: - Attachment: HIVE-9461.1.patch LLAP: Enable local mode tests on tez to facilitate llap testing --- Key: HIVE-9461 URL: https://issues.apache.org/jira/browse/HIVE-9461 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9461.1.patch local mode tez tests will help the testing of multiple fragments running at the same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9461) LLAP: Enable local mode tests on tez to facilitate llap testing
[ https://issues.apache.org/jira/browse/HIVE-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9461: - Attachment: HIVE-9460.1.patch LLAP: Enable local mode tests on tez to facilitate llap testing --- Key: HIVE-9461 URL: https://issues.apache.org/jira/browse/HIVE-9461 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9460.1.patch local mode tez tests will help the testing of multiple fragments running at the same time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9460) LLAP: Fix some static vars in the operator pipeline
Gunther Hagleitner created HIVE-9460: Summary: LLAP: Fix some static vars in the operator pipeline Key: HIVE-9460 URL: https://issues.apache.org/jira/browse/HIVE-9460 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner There are a few static vars left in the operator pipeline. Can't have those with multi-threaded execution... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9460) LLAP: Fix some static vars in the operator pipeline
[ https://issues.apache.org/jira/browse/HIVE-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9460: - Attachment: HIVE-9460.1.patch LLAP: Fix some static vars in the operator pipeline --- Key: HIVE-9460 URL: https://issues.apache.org/jira/browse/HIVE-9460 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9460.1.patch There are a few static vars left in the operator pipeline. Can't have those with multi-threaded execution... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9401) Backport: Fastpath for limited fetches from unpartitioned tables
[
https://issues.apache.org/jira/browse/HIVE-9401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283212#comment-14283212
]
Gunther Hagleitner commented on HIVE-9401:
--
+1
Backport: Fastpath for limited fetches from unpartitioned tables
Key: HIVE-9401
URL: https://issues.apache.org/jira/browse/HIVE-9401
Project: Hive
Issue Type: Bug
Components: Physical Optimizer
Affects Versions: 0.14.0, 0.15.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor
Labels: Backport
Fix For: 0.14.1
Attachments: HIVE-9401.1-branch-0.14.patch
Currently, for flat tables, the threshold check is applicable for a query
like {{select * from lineitem limit 1;}}.
This is not necessary as without a filter clause, this can be executed
entirely via FetchTask. Running a cluster task is redundant for this case.
This fastpath is applicable for partitioned tables already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9406) Some TestMiniTezCliDriver tests not returning results after two hours
[ https://issues.apache.org/jira/browse/HIVE-9406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner reassigned HIVE-9406: Assignee: Gunther Hagleitner Some TestMiniTezCliDriver tests not returning results after two hours - Key: HIVE-9406 URL: https://issues.apache.org/jira/browse/HIVE-9406 Project: Hive Issue Type: Bug Components: Tests, Tez Reporter: Brock Noland Assignee: Gunther Hagleitner Priority: Blocker See here: https://issues.apache.org/jira/browse/HIVE-9402?focusedCommentId=14281690page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14281690 and here: https://issues.apache.org/jira/browse/HIVE-9264?focusedCommentId=14281319page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14281319 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9406) Some TestMiniTezCliDriver tests not returning results after two hours
[ https://issues.apache.org/jira/browse/HIVE-9406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282881#comment-14282881 ] Gunther Hagleitner commented on HIVE-9406: -- Yeah, I think monitoring makes sense. I don't know if anything has changed but since build 2015 everything seems fine. Before that we've had the 2+ hours build, presumably because of hanging tests. Please let me know if you see it again. Some TestMiniTezCliDriver tests not returning results after two hours - Key: HIVE-9406 URL: https://issues.apache.org/jira/browse/HIVE-9406 Project: Hive Issue Type: Bug Components: Tests, Tez Reporter: Brock Noland Priority: Blocker See here: https://issues.apache.org/jira/browse/HIVE-9402?focusedCommentId=14281690page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14281690 and here: https://issues.apache.org/jira/browse/HIVE-9264?focusedCommentId=14281319page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14281319 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7313) Allow in-memory/ssd session-level temp-tables
[ https://issues.apache.org/jira/browse/HIVE-7313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276157#comment-14276157 ] Gunther Hagleitner commented on HIVE-7313: -- Looks good to me. +1. (pending tests). I think eventually it'd be good to expose this flag on the create tmp table sql. So you can through sql choose the storage medium. Maybe a follow up? Allow in-memory/ssd session-level temp-tables - Key: HIVE-7313 URL: https://issues.apache.org/jira/browse/HIVE-7313 Project: Hive Issue Type: Improvement Components: Tez Affects Versions: 0.14.0 Reporter: Gopal V Assignee: Gopal V Labels: InMemory, Performance Attachments: HIVE-7313.1.patch, HIVE-7313.2.patch With HDFS storage policies implementation, temporary tables can be written with different storage/reliability policies. In-session temporary tables can be targetted at both SSD and memory storage policies, with fallbacks onto the disk and the associated reliability trade-offs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9038) Join tests fail on Tez
[
https://issues.apache.org/jira/browse/HIVE-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-9038:
-
Assignee: Vikram Dixit K
Join tests fail on Tez
--
Key: HIVE-9038
URL: https://issues.apache.org/jira/browse/HIVE-9038
Project: Hive
Issue Type: Bug
Components: Tests, Tez
Reporter: Ashutosh Chauhan
Assignee: Vikram Dixit K
Tez doesn't run all tests. But, if you run them, following tests fail with
runt time exception pointing to bugs.
{{auto_join21.q,auto_join29.q,auto_join30.q
,auto_join_filters.q,auto_join_nulls.q}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9205) Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist
[ https://issues.apache.org/jira/browse/HIVE-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261486#comment-14261486 ] Gunther Hagleitner commented on HIVE-9205: -- +1 Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist --- Key: HIVE-9205 URL: https://issues.apache.org/jira/browse/HIVE-9205 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0, 0.15.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.15.0, 0.14.1 Attachments: HIVE-9205.1.patch, HIVE-9205.2.patch The common deployment scenario is to install the packages and start services. Creating the /user/user directory is currently an extra step during manual installation. In case the user tries to bring up the hive shell with tez enabled, this would result in an exception. The solution is to change the default install directory to /tmp (so that we have the permissions to create the directory /tmp/user) and create the /tmp/user directory if it did not exist earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9155) HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql
[ https://issues.apache.org/jira/browse/HIVE-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256572#comment-14256572 ] Gunther Hagleitner commented on HIVE-9155: -- +1 for 0.14.1 HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql -- Key: HIVE-9155 URL: https://issues.apache.org/jira/browse/HIVE-9155 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.14.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 0.15.0 Attachments: HIVE-9155.2.patch, HIVE-9155.3.patch, HIVE-9155.patch HL_LAST_HEARTBEAT and HL_ACQUIRED_AT should use bigint which is 8 bytes and big enough for timestamp.hive-schema-0.15.0.mssql.sql is already correct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8888) Mapjoin with LateralViewJoin generates wrong plan in Tez
[
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228138#comment-14228138
]
Gunther Hagleitner commented on HIVE-:
--
failures are unrelated.
Mapjoin with LateralViewJoin generates wrong plan in Tez
Key: HIVE-
URL: https://issues.apache.org/jira/browse/HIVE-
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0
Reporter: Prasanth J
Assignee: Prasanth J
Fix For: 0.14.1
Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch,
HIVE-.4.patch, HIVE-.5.patch
Queries like these
{code}
with sub1 as
(select aid, avalue from expod1 lateral view explode(av) avs as avalue ),
sub2 as
(select bid, bvalue from expod2 lateral view explode(bv) bvs as bvalue)
select sub1.aid, sub1.avalue, sub2.bvalue
from sub1,sub2
where sub1.aid=sub2.bid;
{code}
generates twice the number of rows in Tez when compared to MR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8964) Some TestMiniTezCliDriver tests taking two hours
[
https://issues.apache.org/jira/browse/HIVE-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228139#comment-14228139
]
Gunther Hagleitner commented on HIVE-8964:
--
Alright with the Tez guys' help figured out that the planner was producing a
cyclic graph. I've re-opened HIVE- and added a new patch. [~brocknoland]
when the new patch in HIVE-, can you re-enable the test on the build
machine? (or did you disable in the source somewhere?)
Some TestMiniTezCliDriver tests taking two hours
Key: HIVE-8964
URL: https://issues.apache.org/jira/browse/HIVE-8964
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Gunther Hagleitner
Priority: Blocker
The test {{TestMiniTezCliDriver}} with the following query files:
vectorization_16.q,mapjoin_mapjoin.q,groupby2.q,lvj_mapjoin.q,vectorization_5.q,vectorization_pushdown.q,orc_merge_incompat1.q,cbo_gby.q,vectorization_4.q,auto_join0.q,cross_product_check_1.q,vectorization_not.q,update_where_no_match.q,ctas.q,cbo_udf_udaf.q
is timing out after two hours severely delaying the Hive precommits
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1898/failed/TestMiniTezCliDriver-vectorization_16.q-mapjoin_mapjoin.q-groupby2.q-and-12-more/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8990) mapjoin_mapjoin.q is failing on Tez (missed golden file update)
[ https://issues.apache.org/jira/browse/HIVE-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8990: - Attachment: HIVE-8990.1.patch mapjoin_mapjoin.q is failing on Tez (missed golden file update) --- Key: HIVE-8990 URL: https://issues.apache.org/jira/browse/HIVE-8990 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8990.1.patch mapjoin_mapjoin.q was updated (SORT_BEFORE_DIFF). However, since the tez test were stuck the accompanying update to the golden file was missed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8990) mapjoin_mapjoin.q is failing on Tez (missed golden file update)
[ https://issues.apache.org/jira/browse/HIVE-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8990: - Status: Patch Available (was: Open) mapjoin_mapjoin.q is failing on Tez (missed golden file update) --- Key: HIVE-8990 URL: https://issues.apache.org/jira/browse/HIVE-8990 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-8990.1.patch mapjoin_mapjoin.q was updated (SORT_BEFORE_DIFF). However, since the tez test were stuck the accompanying update to the golden file was missed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8990) mapjoin_mapjoin.q is failing on Tez (missed golden file update)
Gunther Hagleitner created HIVE-8990: Summary: mapjoin_mapjoin.q is failing on Tez (missed golden file update) Key: HIVE-8990 URL: https://issues.apache.org/jira/browse/HIVE-8990 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner mapjoin_mapjoin.q was updated (SORT_BEFORE_DIFF). However, since the tez test were stuck the accompanying update to the golden file was missed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8888) Mapjoin with LateralViewJoin generates wrong plan in Tez
[
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228485#comment-14228485
]
Gunther Hagleitner commented on HIVE-:
--
[~brocknoland] I have verified that lvj_mapjoin works locally. Thanks.
Mapjoin with LateralViewJoin generates wrong plan in Tez
Key: HIVE-
URL: https://issues.apache.org/jira/browse/HIVE-
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0
Reporter: Prasanth J
Assignee: Prasanth J
Fix For: 0.14.1
Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch,
HIVE-.4.patch, HIVE-.5.patch
Queries like these
{code}
with sub1 as
(select aid, avalue from expod1 lateral view explode(av) avs as avalue ),
sub2 as
(select bid, bvalue from expod2 lateral view explode(bv) bvs as bvalue)
select sub1.aid, sub1.avalue, sub2.bvalue
from sub1,sub2
where sub1.aid=sub2.bid;
{code}
generates twice the number of rows in Tez when compared to MR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8964) Some TestMiniTezCliDriver tests taking two hours
[
https://issues.apache.org/jira/browse/HIVE-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228019#comment-14228019
]
Gunther Hagleitner commented on HIVE-8964:
--
Yes, it's definitely lvj_mapjoin.q. I can repro the hang - no idea why yet.
Some TestMiniTezCliDriver tests taking two hours
Key: HIVE-8964
URL: https://issues.apache.org/jira/browse/HIVE-8964
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Gunther Hagleitner
Priority: Blocker
The test {{TestMiniTezCliDriver}} with the following query files:
vectorization_16.q,mapjoin_mapjoin.q,groupby2.q,lvj_mapjoin.q,vectorization_5.q,vectorization_pushdown.q,orc_merge_incompat1.q,cbo_gby.q,vectorization_4.q,auto_join0.q,cross_product_check_1.q,vectorization_not.q,update_where_no_match.q,ctas.q,cbo_udf_udaf.q
is timing out after two hours severely delaying the Hive precommits
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1898/failed/TestMiniTezCliDriver-vectorization_16.q-mapjoin_mapjoin.q-groupby2.q-and-12-more/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8964) Some TestMiniTezCliDriver tests taking two hours
[
https://issues.apache.org/jira/browse/HIVE-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228026#comment-14228026
]
Gunther Hagleitner commented on HIVE-8964:
--
Happens with tez 0.6.0 also. Same as [~brocknoland] only Exception I see is:
{noformat}
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on
/Users/ghagleitner/Projects/hive-trunk2/itests/qtest/target/tmp/scratchdir/ghagleitner/_tez_session_dir/dc4fca20-4a39-4452-9\
75a-467bda4947ca/.tez/application_1417137410462_0001/recovery/1/summary (inode
16430): File does not exist. Holder DFSClient_NONMAPREDUCE_1900574341_1 does
not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3083)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3170)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3140)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:665)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:499)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
[jira] [Reopened] (HIVE-8888) Mapjoin with LateralViewJoin generates wrong plan in Tez
[
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner reopened HIVE-:
--
The original patch doesn't actually work. The problem is that we still collect
the mj during visiting the nodes. We need to clear that before bailing out of
work generation.
Mapjoin with LateralViewJoin generates wrong plan in Tez
Key: HIVE-
URL: https://issues.apache.org/jira/browse/HIVE-
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0
Reporter: Prasanth J
Assignee: Prasanth J
Fix For: 0.14.1
Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch,
HIVE-.4.patch
Queries like these
{code}
with sub1 as
(select aid, avalue from expod1 lateral view explode(av) avs as avalue ),
sub2 as
(select bid, bvalue from expod2 lateral view explode(bv) bvs as bvalue)
select sub1.aid, sub1.avalue, sub2.bvalue
from sub1,sub2
where sub1.aid=sub2.bid;
{code}
generates twice the number of rows in Tez when compared to MR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8888) Mapjoin with LateralViewJoin generates wrong plan in Tez
[
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-:
-
Attachment: HIVE-.5.patch
Mapjoin with LateralViewJoin generates wrong plan in Tez
Key: HIVE-
URL: https://issues.apache.org/jira/browse/HIVE-
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0
Reporter: Prasanth J
Assignee: Prasanth J
Fix For: 0.14.1
Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch,
HIVE-.4.patch, HIVE-.5.patch
Queries like these
{code}
with sub1 as
(select aid, avalue from expod1 lateral view explode(av) avs as avalue ),
sub2 as
(select bid, bvalue from expod2 lateral view explode(bv) bvs as bvalue)
select sub1.aid, sub1.avalue, sub2.bvalue
from sub1,sub2
where sub1.aid=sub2.bid;
{code}
generates twice the number of rows in Tez when compared to MR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8888) Mapjoin with LateralViewJoin generates wrong plan in Tez
[
https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-:
-
Status: Patch Available (was: Reopened)
Mapjoin with LateralViewJoin generates wrong plan in Tez
Key: HIVE-
URL: https://issues.apache.org/jira/browse/HIVE-
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1, 0.13.0, 0.14.0, 0.15.0
Reporter: Prasanth J
Assignee: Prasanth J
Fix For: 0.14.1
Attachments: HIVE-.1.patch, HIVE-.2.patch, HIVE-.3.patch,
HIVE-.4.patch, HIVE-.5.patch
Queries like these
{code}
with sub1 as
(select aid, avalue from expod1 lateral view explode(av) avs as avalue ),
sub2 as
(select bid, bvalue from expod2 lateral view explode(bv) bvs as bvalue)
select sub1.aid, sub1.avalue, sub2.bvalue
from sub1,sub2
where sub1.aid=sub2.bid;
{code}
generates twice the number of rows in Tez when compared to MR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted
[ https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227045#comment-14227045 ] Gunther Hagleitner commented on HIVE-8966: -- +1 for 0.14.1 Delta files created by hive hcatalog streaming cannot be compacted -- Key: HIVE-8966 URL: https://issues.apache.org/jira/browse/HIVE-8966 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.14.0 Environment: hive Reporter: Jihong Liu Assignee: Alan Gates Priority: Critical hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where n is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction. Did a test, after removed the bucket_n_flush_length file, then the alter table partition compact finished successfully. If don't delete that file, nothing will be compacted. This is probably a very severity bug. Both 0.13 and 0.14 have this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8880) non-synchronized access to split list in OrcInputFormat
[ https://issues.apache.org/jira/browse/HIVE-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227048#comment-14227048 ] Gunther Hagleitner commented on HIVE-8880: -- +1, also for 0.14.1 non-synchronized access to split list in OrcInputFormat --- Key: HIVE-8880 URL: https://issues.apache.org/jira/browse/HIVE-8880 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.14.1 Attachments: HIVE-8880.patch When adding delta files to the list of orc splits access to the list is not synchronized though it is shared across threads. All other additions to the list are synchronized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
