[jira] [Updated] (HIVE-5677) Beeline warns about unavailable files if HIVE_OPTS is set
[ https://issues.apache.org/jira/browse/HIVE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5677: Description: NO PRECOMMIT TESTS This is similar to HIVE-5085. Beeline complains about files not existing if HIVE_OPTS are set. In the Beeline commandline sh as well, we should see if setting HIVE_OPTS to '' makes sense. was: This is similar to HIVE-5085. Beeline complains about files not existing if HIVE_OPTS are set. In the Beeline commandline sh as well, we should see if setting HIVE_OPTS to '' makes sense. Beeline warns about unavailable files if HIVE_OPTS is set - Key: HIVE-5677 URL: https://issues.apache.org/jira/browse/HIVE-5677 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.12.0 Reporter: Sushanth Sowmyan Attachments: HIVE-5677.1.patch.txt NO PRECOMMIT TESTS This is similar to HIVE-5085. Beeline complains about files not existing if HIVE_OPTS are set. In the Beeline commandline sh as well, we should see if setting HIVE_OPTS to '' makes sense. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HIVE-5677) Beeline warns about unavailable files if HIVE_OPTS is set
[ https://issues.apache.org/jira/browse/HIVE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis reassigned HIVE-5677: --- Assignee: Navis Beeline warns about unavailable files if HIVE_OPTS is set - Key: HIVE-5677 URL: https://issues.apache.org/jira/browse/HIVE-5677 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.12.0 Reporter: Sushanth Sowmyan Assignee: Navis Attachments: HIVE-5677.1.patch.txt NO PRECOMMIT TESTS This is similar to HIVE-5085. Beeline complains about files not existing if HIVE_OPTS are set. In the Beeline commandline sh as well, we should see if setting HIVE_OPTS to '' makes sense. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5677) Beeline warns about unavailable files if HIVE_OPTS is set
[ https://issues.apache.org/jira/browse/HIVE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5677: Attachment: HIVE-5677.1.patch.txt Beeline warns about unavailable files if HIVE_OPTS is set - Key: HIVE-5677 URL: https://issues.apache.org/jira/browse/HIVE-5677 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.12.0 Reporter: Sushanth Sowmyan Attachments: HIVE-5677.1.patch.txt NO PRECOMMIT TESTS This is similar to HIVE-5085. Beeline complains about files not existing if HIVE_OPTS are set. In the Beeline commandline sh as well, we should see if setting HIVE_OPTS to '' makes sense. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5677) Beeline warns about unavailable files if HIVE_OPTS is set
[ https://issues.apache.org/jira/browse/HIVE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5677: Status: Patch Available (was: Open) Beeline warns about unavailable files if HIVE_OPTS is set - Key: HIVE-5677 URL: https://issues.apache.org/jira/browse/HIVE-5677 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.12.0 Reporter: Sushanth Sowmyan Assignee: Navis Attachments: HIVE-5677.1.patch.txt NO PRECOMMIT TESTS This is similar to HIVE-5085. Beeline complains about files not existing if HIVE_OPTS are set. In the Beeline commandline sh as well, we should see if setting HIVE_OPTS to '' makes sense. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5729) Beeline displays version as ???? after mavenization
[ https://issues.apache.org/jira/browse/HIVE-5729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820147#comment-13820147 ] Brock Noland commented on HIVE-5729: +1 Beeline displays version as after mavenization --- Key: HIVE-5729 URL: https://issues.apache.org/jira/browse/HIVE-5729 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.13.0 Reporter: Szehon Ho Assignee: Navis Attachments: HIVE-5729.1.patch.txt NO PRECOMMIT TESTS In Beeline.java, method getApplicationTitle(), it looks to the Beeline class's package to find version information. However, MANIFESTs are not included in Beeline jar after mavenization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5732) HiveServer2: Duplicated new OperationManager in SessionManager
[
https://issues.apache.org/jira/browse/HIVE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820150#comment-13820150
]
Brock Noland commented on HIVE-5732:
+1
HiveServer2: Duplicated new OperationManager in SessionManager
--
Key: HIVE-5732
URL: https://issues.apache.org/jira/browse/HIVE-5732
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Schubert Zhang
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5732.1.patch.txt
{noformat}
public class SessionManager extends CompositeService {
private static final Log LOG = LogFactory.getLog(CompositeService.class);
private HiveConf hiveConf;
private final MapSessionHandle, HiveSession handleToSession = new
HashMapSessionHandle, HiveSession();
private OperationManager operationManager = new OperationManager();
private static final Object sessionMapLock = new Object();
private ExecutorService backgroundOperationPool;
public SessionManager() {
super(SessionManager);
}
@Override
public synchronized void init(HiveConf hiveConf) {
this.hiveConf = hiveConf;
operationManager = new OperationManager();
int backgroundPoolSize =
hiveConf.getIntVar(ConfVars.HIVE_SERVER2_ASYNC_EXEC_THREADS);
LOG.info(HiveServer2: Async execution pool size + backgroundPoolSize);
backgroundOperationPool =
Executors.newFixedThreadPool(backgroundPoolSize);
addService(operationManager);
super.init(hiveConf);
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5732) HiveServer2: Duplicated new OperationManager in SessionManager
[
https://issues.apache.org/jira/browse/HIVE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-5732:
---
Description:
{noformat}
public class SessionManager extends CompositeService {
private static final Log LOG = LogFactory.getLog(CompositeService.class);
private HiveConf hiveConf;
private final MapSessionHandle, HiveSession handleToSession = new
HashMapSessionHandle, HiveSession();
private OperationManager operationManager = new OperationManager();
private static final Object sessionMapLock = new Object();
private ExecutorService backgroundOperationPool;
public SessionManager() {
super(SessionManager);
}
@Override
public synchronized void init(HiveConf hiveConf) {
this.hiveConf = hiveConf;
operationManager = new OperationManager();
int backgroundPoolSize =
hiveConf.getIntVar(ConfVars.HIVE_SERVER2_ASYNC_EXEC_THREADS);
LOG.info(HiveServer2: Async execution pool size + backgroundPoolSize);
backgroundOperationPool = Executors.newFixedThreadPool(backgroundPoolSize);
addService(operationManager);
super.init(hiveConf);
}
{noformat}
was:
public class SessionManager extends CompositeService {
private static final Log LOG = LogFactory.getLog(CompositeService.class);
private HiveConf hiveConf;
private final MapSessionHandle, HiveSession handleToSession = new
HashMapSessionHandle, HiveSession();
private OperationManager operationManager = new OperationManager();
private static final Object sessionMapLock = new Object();
private ExecutorService backgroundOperationPool;
public SessionManager() {
super(SessionManager);
}
@Override
public synchronized void init(HiveConf hiveConf) {
this.hiveConf = hiveConf;
operationManager = new OperationManager();
int backgroundPoolSize =
hiveConf.getIntVar(ConfVars.HIVE_SERVER2_ASYNC_EXEC_THREADS);
LOG.info(HiveServer2: Async execution pool size + backgroundPoolSize);
backgroundOperationPool = Executors.newFixedThreadPool(backgroundPoolSize);
addService(operationManager);
super.init(hiveConf);
}
HiveServer2: Duplicated new OperationManager in SessionManager
--
Key: HIVE-5732
URL: https://issues.apache.org/jira/browse/HIVE-5732
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Schubert Zhang
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5732.1.patch.txt
{noformat}
public class SessionManager extends CompositeService {
private static final Log LOG = LogFactory.getLog(CompositeService.class);
private HiveConf hiveConf;
private final MapSessionHandle, HiveSession handleToSession = new
HashMapSessionHandle, HiveSession();
private OperationManager operationManager = new OperationManager();
private static final Object sessionMapLock = new Object();
private ExecutorService backgroundOperationPool;
public SessionManager() {
super(SessionManager);
}
@Override
public synchronized void init(HiveConf hiveConf) {
this.hiveConf = hiveConf;
operationManager = new OperationManager();
int backgroundPoolSize =
hiveConf.getIntVar(ConfVars.HIVE_SERVER2_ASYNC_EXEC_THREADS);
LOG.info(HiveServer2: Async execution pool size + backgroundPoolSize);
backgroundOperationPool =
Executors.newFixedThreadPool(backgroundPoolSize);
addService(operationManager);
super.init(hiveConf);
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HIVE-5783) Native Parquet Support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Coffey reassigned HIVE-5783: --- Assignee: Justin Coffey Native Parquet Support in Hive -- Key: HIVE-5783 URL: https://issues.apache.org/jira/browse/HIVE-5783 Project: Hive Issue Type: New Feature Reporter: Justin Coffey Assignee: Justin Coffey Priority: Minor Problem Statement: Hive would be easier to use if it had native Parquet support. Our organization, Criteo, uses Hive extensively. Therefore we built the Parquet Hive integration and would like to now contribute that integration to Hive. About Parquet: Parquet is a columnar storage format for Hadoop and integrates with many Hadoop ecosystem tools such as Thrift, Avro, Hadoop MapReduce, Cascading, Pig, Drill, Crunch, and Hive. Pig, Crunch, and Drill all contain native Parquet integration. Changes Details: Parquet was built with dependency management in mind and therefore only a single Parquet jar will be added as a dependency. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5783) Native Parquet Support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820168#comment-13820168 ] Justin Coffey commented on HIVE-5783: - Thanks [~cwsteinbach] and [~ehans]. Regarding vectorization support the parquet team will review ASAP! Native Parquet Support in Hive -- Key: HIVE-5783 URL: https://issues.apache.org/jira/browse/HIVE-5783 Project: Hive Issue Type: New Feature Reporter: Justin Coffey Assignee: Justin Coffey Priority: Minor Problem Statement: Hive would be easier to use if it had native Parquet support. Our organization, Criteo, uses Hive extensively. Therefore we built the Parquet Hive integration and would like to now contribute that integration to Hive. About Parquet: Parquet is a columnar storage format for Hadoop and integrates with many Hadoop ecosystem tools such as Thrift, Avro, Hadoop MapReduce, Cascading, Pig, Drill, Crunch, and Hive. Pig, Crunch, and Drill all contain native Parquet integration. Changes Details: Parquet was built with dependency management in mind and therefore only a single Parquet jar will be added as a dependency. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5677) Beeline warns about unavailable files if HIVE_OPTS is set
[ https://issues.apache.org/jira/browse/HIVE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820188#comment-13820188 ] Xuefu Zhang commented on HIVE-5677: --- Just curious, with the changes in the patch, are we still able to debug beeline process remotely? Beeline warns about unavailable files if HIVE_OPTS is set - Key: HIVE-5677 URL: https://issues.apache.org/jira/browse/HIVE-5677 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.12.0 Reporter: Sushanth Sowmyan Assignee: Navis Attachments: HIVE-5677.1.patch.txt NO PRECOMMIT TESTS This is similar to HIVE-5085. Beeline complains about files not existing if HIVE_OPTS are set. In the Beeline commandline sh as well, we should see if setting HIVE_OPTS to '' makes sense. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5801) Support for reader/writer of ORC format for R environment
Michael Hausenblas created HIVE-5801: Summary: Support for reader/writer of ORC format for R environment Key: HIVE-5801 URL: https://issues.apache.org/jira/browse/HIVE-5801 Project: Hive Issue Type: Improvement Reporter: Michael Hausenblas Priority: Minor It would be great if the ORC format would directly be accessible from R [1], that is, providing reader/writer for it. [1] http://www.r-project.org/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5801) Support for reader/writer of ORC format for R environment
[ https://issues.apache.org/jira/browse/HIVE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820282#comment-13820282 ] Carter Shanklin commented on HIVE-5801: --- Hi [~mhausenblas], would it make sense to integrate at the HCatalog layer? It's not as optimized as reading directly but does abstract underlying format. Support for reader/writer of ORC format for R environment - Key: HIVE-5801 URL: https://issues.apache.org/jira/browse/HIVE-5801 Project: Hive Issue Type: Improvement Reporter: Michael Hausenblas Priority: Minor It would be great if the ORC format would directly be accessible from R [1], that is, providing reader/writer for it. [1] http://www.r-project.org/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations
[ https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-5356: -- Attachment: HIVE-5356.6.patch Patch #6 is equivalent to #5 with some minor refactoring. Move arithmatic UDFs to generic UDF implementations --- Key: HIVE-5356 URL: https://issues.apache.org/jira/browse/HIVE-5356 Project: Hive Issue Type: Task Components: UDF Affects Versions: 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Fix For: 0.13.0 Attachments: HIVE-5356.1.patch, HIVE-5356.2.patch, HIVE-5356.3.patch, HIVE-5356.4.patch, HIVE-5356.5.patch, HIVE-5356.6.patch, HIVE-5356.patch Currently, all of the arithmetic operators, such as add/sub/mult/div, are implemented as old-style UDFs and java reflection is used to determine the return type TypeInfos/ObjectInspectors, based on the return type of the evaluate() method chosen for the expression. This works fine for types that don't have type params. Hive decimal type participates in these operations just like int or double. Different from double or int, however, decimal has precision and scale, which cannot be determined by just looking at the return type (decimal) of the UDF evaluate() method, even though the operands have certain precision/scale. With the default of decimal without precision/scale, then (10, 0) will be the type params. This is certainly not desirable. To solve this problem, all of the arithmetic operators would need to be implemented as GenericUDFs, which allow returning ObjectInspector during the initialize() method. The object inspectors returned can carry type params, from which the exact return type can be determined. It's worth mentioning that, for user UDF implemented in non-generic way, if the return type of the chosen evaluate() method is decimal, the return type actually has (10,0) as precision/scale, which might not be desirable. This needs to be documented. This JIRA will cover minus, plus, divide, multiply, mod, and pmod, to limit the scope of review. The remaining ones will be covered under HIVE-5706. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5632) Eliminate splits based on SARGs using stripe statistics in ORC
[ https://issues.apache.org/jira/browse/HIVE-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5632: - Status: Open (was: Patch Available) Eliminate splits based on SARGs using stripe statistics in ORC -- Key: HIVE-5632 URL: https://issues.apache.org/jira/browse/HIVE-5632 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-5632.1.patch.txt, HIVE-5632.2.patch.txt, HIVE-5632.3.patch.txt, HIVE-5632.4.patch, HIVE-5632.5.patch.txt, orc_split_elim.orc HIVE-5562 provides stripe level statistics in ORC. Stripe level statistics combined with predicate pushdown in ORC (HIVE-4246) can be used to eliminate the stripes (thereby splits) that doesn't satisfy the predicate condition. This can greatly reduce unnecessary reads. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5632) Eliminate splits based on SARGs using stripe statistics in ORC
[ https://issues.apache.org/jira/browse/HIVE-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5632: - Status: Patch Available (was: Open) Eliminate splits based on SARGs using stripe statistics in ORC -- Key: HIVE-5632 URL: https://issues.apache.org/jira/browse/HIVE-5632 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-5632.1.patch.txt, HIVE-5632.2.patch.txt, HIVE-5632.3.patch.txt, HIVE-5632.4.patch, HIVE-5632.5.patch, HIVE-5632.5.patch.txt, orc_split_elim.orc HIVE-5562 provides stripe level statistics in ORC. Stripe level statistics combined with predicate pushdown in ORC (HIVE-4246) can be used to eliminate the stripes (thereby splits) that doesn't satisfy the predicate condition. This can greatly reduce unnecessary reads. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5632) Eliminate splits based on SARGs using stripe statistics in ORC
[ https://issues.apache.org/jira/browse/HIVE-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5632: - Attachment: HIVE-5632.5.patch Reupload .5 for precommit test. Eliminate splits based on SARGs using stripe statistics in ORC -- Key: HIVE-5632 URL: https://issues.apache.org/jira/browse/HIVE-5632 Project: Hive Issue Type: Improvement Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: orcfile Attachments: HIVE-5632.1.patch.txt, HIVE-5632.2.patch.txt, HIVE-5632.3.patch.txt, HIVE-5632.4.patch, HIVE-5632.5.patch, HIVE-5632.5.patch.txt, orc_split_elim.orc HIVE-5562 provides stripe level statistics in ORC. Stripe level statistics combined with predicate pushdown in ORC (HIVE-4246) can be used to eliminate the stripes (thereby splits) that doesn't satisfy the predicate condition. This can greatly reduce unnecessary reads. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5217) Add long polling to asynchronous execution in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820372#comment-13820372 ] Carl Steinbach commented on HIVE-5217: -- [~vaibhavgumashta] Is the patch ready for review? Add long polling to asynchronous execution in HiveServer2 - Key: HIVE-5217 URL: https://issues.apache.org/jira/browse/HIVE-5217 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-5217.2.patch, HIVE-5217.3.patch, HIVE-5217.D12801.2.patch, HIVE-5217.D12801.3.patch, HIVE-5217.D12801.4.patch, HIVE-5217.D12801.5.patch, HIVE-5217.D12801.6.patch [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support for async execution in HS2. The client gets an operation handle which it can poll to check on the operation status. However, the polling frequency is entirely left to the client which can be resource inefficient. Long polling will solve this, by blocking the client request to check the operation status for a configurable amount of time (a new HS2 config) if the data is not available, but responding immediately if the data is available. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5732) HiveServer2: Duplicated new OperationManager in SessionManager
[
https://issues.apache.org/jira/browse/HIVE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820380#comment-13820380
]
Hive QA commented on HIVE-5732:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12613329/HIVE-5732.1.patch.txt
{color:green}SUCCESS:{color} +1 4602 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/244/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/244/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12613329
HiveServer2: Duplicated new OperationManager in SessionManager
--
Key: HIVE-5732
URL: https://issues.apache.org/jira/browse/HIVE-5732
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Schubert Zhang
Assignee: Navis
Priority: Trivial
Attachments: HIVE-5732.1.patch.txt
{noformat}
public class SessionManager extends CompositeService {
private static final Log LOG = LogFactory.getLog(CompositeService.class);
private HiveConf hiveConf;
private final MapSessionHandle, HiveSession handleToSession = new
HashMapSessionHandle, HiveSession();
private OperationManager operationManager = new OperationManager();
private static final Object sessionMapLock = new Object();
private ExecutorService backgroundOperationPool;
public SessionManager() {
super(SessionManager);
}
@Override
public synchronized void init(HiveConf hiveConf) {
this.hiveConf = hiveConf;
operationManager = new OperationManager();
int backgroundPoolSize =
hiveConf.getIntVar(ConfVars.HIVE_SERVER2_ASYNC_EXEC_THREADS);
LOG.info(HiveServer2: Async execution pool size + backgroundPoolSize);
backgroundOperationPool =
Executors.newFixedThreadPool(backgroundPoolSize);
addService(operationManager);
super.init(hiveConf);
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-4723) DDLSemanticAnalyzer.addTablePartsOutputs eats several exceptions
[
https://issues.apache.org/jira/browse/HIVE-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-4723:
Attachment: HIVE-4723.3.patch
Attaching the same patch again to trigger testing. I believe the pre-commit
test of the previous patch got lost in the queue the build moved to bigtop.
DDLSemanticAnalyzer.addTablePartsOutputs eats several exceptions
Key: HIVE-4723
URL: https://issues.apache.org/jira/browse/HIVE-4723
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Szehon Ho
Attachments: HIVE-4723.1.patch, HIVE-4723.2.patch, HIVE-4723.3.patch,
HIVE-4723.patch
I accidently tried to archive a partition on a non-partitioned table. The
error message was bad, hive ate an exception, and NPE'ed.
{noformat}
2013-06-09 16:36:12,628 ERROR parse.DDLSemanticAnalyzer
(DDLSemanticAnalyzer.java:addTablePartsOutputs(2899)) - Got HiveException
during obtaining list of partitions
2013-06-09 16:36:12,628 ERROR ql.Driver (SessionState.java:printError(383)) -
FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.addTablePartsOutputs(DDLSemanticAnalyzer.java:2912)
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.addTablePartsOutputs(DDLSemanticAnalyzer.java:2877)
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTableArchive(DDLSemanticAnalyzer.java:2730)
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:316)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:277)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-4116) Can't use views using map datatype.
[
https://issues.apache.org/jira/browse/HIVE-4116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820405#comment-13820405
]
Hive QA commented on HIVE-4116:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12613330/HIVE-4116.2.patch.txt
{color:green}SUCCESS:{color} +1 4602 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/246/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/246/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12613330
Can't use views using map datatype.
---
Key: HIVE-4116
URL: https://issues.apache.org/jira/browse/HIVE-4116
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1, 0.10.0, 0.11.0
Reporter: Karel Vervaeke
Assignee: Navis
Attachments: D12975.1.patch, HIVE-4116.2.patch.txt
Executing the following
{noformat}
DROP TABLE IF EXISTS `items`;
CREATE TABLE IF NOT EXISTS `items` (id INT, name STRING, info
MAPSTRING,STRING) PARTITIONED BY (ds STRING);
DROP VIEW IF EXISTS `priceview`;
CREATE VIEW `priceview` AS
SELECT
`items`.`id`,
`items`.info['price']
FROM
`items`
;
select * from `priceview`;
{noformat}
Produces the following error:
{noformat}
karel@tomato:~/tmp$ $HIVE_HOME/bin/hive -f hivebug.sql
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Logging initialized using configuration in
jar:file:/home/karel/opt/hive-0.10.0-bin/lib/hive-common-0.10.0.jar!/hive-log4j.properties
Hive history file=/tmp/karel/hive_job_log_karel_201303051117_945318761.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/karel/opt/hadoop-2.0.0-mr1-cdh4.0.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/karel/opt/hive-0.10.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
OK
Time taken: 5.449 seconds
OK
Time taken: 0.303 seconds
OK
Time taken: 0.131 seconds
OK
Time taken: 0.206 seconds
FAILED: SemanticException line 3:22 mismatched input '.' expecting FROM near
'`items`' in from clause
in definition of VIEW priceview [
SELECT
`items`.`id`,
`items``items`.`info`info['price']
FROM
`default`.`items`
] used as priceview at Line 3:14
{noformat}
Unless I'm not using the right syntax, I would expect this simple example to
work. I have tried some variations (quotes, no quotes, ...), to no avail.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-4388) Upgrade HBase to 0.96
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820414#comment-13820414 ] Brock Noland commented on HIVE-4388: This is blocking HIVE-5755 Fix hadoop2 execution environment. It'd be great if someone could review! Thanks!! Upgrade HBase to 0.96 - Key: HIVE-4388 URL: https://issues.apache.org/jira/browse/HIVE-4388 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Gunther Hagleitner Assignee: Brock Noland Attachments: HIVE-4388-wip.txt, HIVE-4388.10.patch, HIVE-4388.11.patch, HIVE-4388.12.patch, HIVE-4388.13.patch, HIVE-4388.14.patch, HIVE-4388.15.patch, HIVE-4388.15.patch, HIVE-4388.16.patch, HIVE-4388.17.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch Currently we're building by default against 0.92. When you run against hadoop 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5614) Subquery support: allow subquery expressions in having clause
[ https://issues.apache.org/jira/browse/HIVE-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-5614: Attachment: HIVE-5614.1.patch Subquery support: allow subquery expressions in having clause - Key: HIVE-5614 URL: https://issues.apache.org/jira/browse/HIVE-5614 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-5614.1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5614) Subquery support: allow subquery expressions in having clause
[ https://issues.apache.org/jira/browse/HIVE-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820417#comment-13820417 ] Harish Butani commented on HIVE-5614: - uploaded preliminary patch. Includes changes for HIVE-3107 Subquery support: allow subquery expressions in having clause - Key: HIVE-5614 URL: https://issues.apache.org/jira/browse/HIVE-5614 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-5614.1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho reassigned HIVE-5717: --- Assignee: Szehon Ho Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4116) Can't use views using map datatype.
[
https://issues.apache.org/jira/browse/HIVE-4116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4116:
---
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Navis!
Can't use views using map datatype.
---
Key: HIVE-4116
URL: https://issues.apache.org/jira/browse/HIVE-4116
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1, 0.10.0, 0.11.0
Reporter: Karel Vervaeke
Assignee: Navis
Fix For: 0.13.0
Attachments: D12975.1.patch, HIVE-4116.2.patch.txt
Executing the following
{noformat}
DROP TABLE IF EXISTS `items`;
CREATE TABLE IF NOT EXISTS `items` (id INT, name STRING, info
MAPSTRING,STRING) PARTITIONED BY (ds STRING);
DROP VIEW IF EXISTS `priceview`;
CREATE VIEW `priceview` AS
SELECT
`items`.`id`,
`items`.info['price']
FROM
`items`
;
select * from `priceview`;
{noformat}
Produces the following error:
{noformat}
karel@tomato:~/tmp$ $HIVE_HOME/bin/hive -f hivebug.sql
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Logging initialized using configuration in
jar:file:/home/karel/opt/hive-0.10.0-bin/lib/hive-common-0.10.0.jar!/hive-log4j.properties
Hive history file=/tmp/karel/hive_job_log_karel_201303051117_945318761.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/karel/opt/hadoop-2.0.0-mr1-cdh4.0.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/karel/opt/hive-0.10.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
OK
Time taken: 5.449 seconds
OK
Time taken: 0.303 seconds
OK
Time taken: 0.131 seconds
OK
Time taken: 0.206 seconds
FAILED: SemanticException line 3:22 mismatched input '.' expecting FROM near
'`items`' in from clause
in definition of VIEW priceview [
SELECT
`items`.`id`,
`items``items`.`info`info['price']
FROM
`default`.`items`
] used as priceview at Line 3:14
{noformat}
Unless I'm not using the right syntax, I would expect this simple example to
work. I have tried some variations (quotes, no quotes, ...), to no avail.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HIVE-5741) Hcatalog needs to be added to the binary tar
[ https://issues.apache.org/jira/browse/HIVE-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland reassigned HIVE-5741: -- Assignee: Brock Noland Hcatalog needs to be added to the binary tar Key: HIVE-5741 URL: https://issues.apache.org/jira/browse/HIVE-5741 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5217) Add long polling to asynchronous execution in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820460#comment-13820460 ] Vaibhav Gumashta commented on HIVE-5217: [~cwsteinbach] Yes, you can take a look now. Thanks. Add long polling to asynchronous execution in HiveServer2 - Key: HIVE-5217 URL: https://issues.apache.org/jira/browse/HIVE-5217 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-5217.2.patch, HIVE-5217.3.patch, HIVE-5217.D12801.2.patch, HIVE-5217.D12801.3.patch, HIVE-5217.D12801.4.patch, HIVE-5217.D12801.5.patch, HIVE-5217.D12801.6.patch [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support for async execution in HS2. The client gets an operation handle which it can poll to check on the operation status. However, the polling frequency is entirely left to the client which can be resource inefficient. Long polling will solve this, by blocking the client request to check the operation status for a configurable amount of time (a new HS2 config) if the data is not available, but responding immediately if the data is available. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820458#comment-13820458
]
Hive QA commented on HIVE-5799:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12613327/HIVE-5799.1.patch.txt
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 4537 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.src.test.TestBeeLineWithArgs.org.apache.hive.beeline.src.test.TestBeeLineWithArgs
org.apache.hive.jdbc.TestJdbcDriver2.org.apache.hive.jdbc.TestJdbcDriver2
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnection
org.apache.hive.jdbc.TestSSL.testConnectionMismatch
org.apache.hive.jdbc.TestSSL.testInvalidConfig
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithURL
org.apache.hive.jdbc.TestSSL.testSSLFetch
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
org.apache.hive.service.auth.TestCustomAuthentication.org.apache.hive.service.auth.TestCustomAuthentication
org.apache.hive.service.auth.TestPlainSaslHelper.testDoAsSetting
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService
org.apache.hive.service.cli.TestScratchDir.testLocalScratchDirs
org.apache.hive.service.cli.TestScratchDir.testResourceDirs
org.apache.hive.service.cli.TestScratchDir.testScratchDirs
org.apache.hive.service.cli.session.TestSessionHooks.testSessionHook
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.org.apache.hive.service.cli.thrift.TestThriftHttpCLIService
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/247/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/247/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12613327
session/operation timeout for hiveserver2
-
Key: HIVE-5799
URL: https://issues.apache.org/jira/browse/HIVE-5799
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-5799.1.patch.txt
Need some timeout facility for preventing resource leakages from instable or
bad clients.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5218) datanucleus does not work with MS SQLServer in Hive metastore
[
https://issues.apache.org/jira/browse/HIVE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-5218:
Status: Open (was: Patch Available)
datanucleus does not work with MS SQLServer in Hive metastore
-
Key: HIVE-5218
URL: https://issues.apache.org/jira/browse/HIVE-5218
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: shanyu zhao
Assignee: shanyu zhao
Fix For: 0.13.0
Attachments:
0001-HIVE-5218-datanucleus-does-not-work-with-SQLServer-i.patch,
HIVE-5218-trunk.patch, HIVE-5218-v2.patch, HIVE-5218.2.patch, HIVE-5218.patch
HIVE-3632 upgraded datanucleus version to 3.2.x, however, this version of
datanucleus doesn't work with SQLServer as the metastore. The problem is that
datanucleus tries to use fully qualified object name to find a table in the
database but couldn't find it.
If I downgrade the version to HIVE-2084, SQLServer works fine.
It could be a bug in datanucleus.
This is the detailed exception I'm getting when using datanucleus 3.2.x with
SQL Server:
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTa
sk. MetaException(message:javax.jdo.JDOException: Exception thrown calling
table
.exists() for a2ee36af45e9f46c19e995bfd2d9b5fd1hivemetastore..SEQUENCE_TABLE
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusExc
eption(NucleusJDOHelper.java:596)
at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPe
rsistenceManager.java:732)
…
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawS
tore.java:111)
at $Proxy0.createTable(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
e_core(HiveMetaStore.java:1071)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
e_with_environment_context(HiveMetaStore.java:1104)
…
at $Proxy11.create_table_with_environment_context(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6417)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6401)
NestedThrowablesStackTrace:
com.microsoft.sqlserver.jdbc.SQLServerException: There is already an object
name
d 'SEQUENCE_TABLE' in the database.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError
(SQLServerException.java:197)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServ
erStatement.java:1493)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQ
LServerStatement.java:775)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute
(SQLServerStatement.java:676)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4615)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLSe
rverConnection.java:1400)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLSer
verStatement.java:179)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLS
erverStatement.java:154)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStat
ement.java:649)
at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:300)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(A
bstractTable.java:760)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementLi
st(AbstractTable.java:711)
at
org.datanucleus.store.rdbms.table.AbstractTable.create(AbstractTable.
java:425)
at
org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.
java:488)
at
org.datanucleus.store.rdbms.valuegenerator.TableGenerator.repositoryE
xists(TableGenerator.java:242)
at
org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obt
ainGenerationBlock(AbstractRDBMSGenerator.java:86)
at
org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerati
onBlock(AbstractGenerator.java:197)
at
org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractG
enerator.java:105)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGene
rator(RDBMSStoreManager.java:2019)
at
org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractS
toreManager.java:1385)
at
org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl
[jira] [Updated] (HIVE-5218) datanucleus does not work with MS SQLServer in Hive metastore
[
https://issues.apache.org/jira/browse/HIVE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-5218:
Status: Patch Available (was: Open)
datanucleus does not work with MS SQLServer in Hive metastore
-
Key: HIVE-5218
URL: https://issues.apache.org/jira/browse/HIVE-5218
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: shanyu zhao
Assignee: shanyu zhao
Fix For: 0.13.0
Attachments:
0001-HIVE-5218-datanucleus-does-not-work-with-SQLServer-i.patch,
HIVE-5218-trunk.patch, HIVE-5218-trunk.patch, HIVE-5218-v2.patch,
HIVE-5218.2.patch, HIVE-5218.patch
HIVE-3632 upgraded datanucleus version to 3.2.x, however, this version of
datanucleus doesn't work with SQLServer as the metastore. The problem is that
datanucleus tries to use fully qualified object name to find a table in the
database but couldn't find it.
If I downgrade the version to HIVE-2084, SQLServer works fine.
It could be a bug in datanucleus.
This is the detailed exception I'm getting when using datanucleus 3.2.x with
SQL Server:
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTa
sk. MetaException(message:javax.jdo.JDOException: Exception thrown calling
table
.exists() for a2ee36af45e9f46c19e995bfd2d9b5fd1hivemetastore..SEQUENCE_TABLE
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusExc
eption(NucleusJDOHelper.java:596)
at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPe
rsistenceManager.java:732)
…
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawS
tore.java:111)
at $Proxy0.createTable(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
e_core(HiveMetaStore.java:1071)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
e_with_environment_context(HiveMetaStore.java:1104)
…
at $Proxy11.create_table_with_environment_context(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6417)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6401)
NestedThrowablesStackTrace:
com.microsoft.sqlserver.jdbc.SQLServerException: There is already an object
name
d 'SEQUENCE_TABLE' in the database.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError
(SQLServerException.java:197)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServ
erStatement.java:1493)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQ
LServerStatement.java:775)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute
(SQLServerStatement.java:676)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4615)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLSe
rverConnection.java:1400)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLSer
verStatement.java:179)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLS
erverStatement.java:154)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStat
ement.java:649)
at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:300)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(A
bstractTable.java:760)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementLi
st(AbstractTable.java:711)
at
org.datanucleus.store.rdbms.table.AbstractTable.create(AbstractTable.
java:425)
at
org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.
java:488)
at
org.datanucleus.store.rdbms.valuegenerator.TableGenerator.repositoryE
xists(TableGenerator.java:242)
at
org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obt
ainGenerationBlock(AbstractRDBMSGenerator.java:86)
at
org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerati
onBlock(AbstractGenerator.java:197)
at
org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractG
enerator.java:105)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGene
rator(RDBMSStoreManager.java:2019)
at
org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractS
toreManager.java:1385)
at
[jira] [Updated] (HIVE-5218) datanucleus does not work with MS SQLServer in Hive metastore
[
https://issues.apache.org/jira/browse/HIVE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-5218:
Attachment: HIVE-5218-trunk.patch
reattaching patch to run tests
datanucleus does not work with MS SQLServer in Hive metastore
-
Key: HIVE-5218
URL: https://issues.apache.org/jira/browse/HIVE-5218
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.12.0
Reporter: shanyu zhao
Assignee: shanyu zhao
Fix For: 0.13.0
Attachments:
0001-HIVE-5218-datanucleus-does-not-work-with-SQLServer-i.patch,
HIVE-5218-trunk.patch, HIVE-5218-trunk.patch, HIVE-5218-v2.patch,
HIVE-5218.2.patch, HIVE-5218.patch
HIVE-3632 upgraded datanucleus version to 3.2.x, however, this version of
datanucleus doesn't work with SQLServer as the metastore. The problem is that
datanucleus tries to use fully qualified object name to find a table in the
database but couldn't find it.
If I downgrade the version to HIVE-2084, SQLServer works fine.
It could be a bug in datanucleus.
This is the detailed exception I'm getting when using datanucleus 3.2.x with
SQL Server:
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTa
sk. MetaException(message:javax.jdo.JDOException: Exception thrown calling
table
.exists() for a2ee36af45e9f46c19e995bfd2d9b5fd1hivemetastore..SEQUENCE_TABLE
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusExc
eption(NucleusJDOHelper.java:596)
at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPe
rsistenceManager.java:732)
…
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawS
tore.java:111)
at $Proxy0.createTable(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
e_core(HiveMetaStore.java:1071)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
e_with_environment_context(HiveMetaStore.java:1104)
…
at $Proxy11.create_table_with_environment_context(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6417)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6401)
NestedThrowablesStackTrace:
com.microsoft.sqlserver.jdbc.SQLServerException: There is already an object
name
d 'SEQUENCE_TABLE' in the database.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError
(SQLServerException.java:197)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServ
erStatement.java:1493)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQ
LServerStatement.java:775)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute
(SQLServerStatement.java:676)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4615)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLSe
rverConnection.java:1400)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLSer
verStatement.java:179)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLS
erverStatement.java:154)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStat
ement.java:649)
at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:300)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(A
bstractTable.java:760)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementLi
st(AbstractTable.java:711)
at
org.datanucleus.store.rdbms.table.AbstractTable.create(AbstractTable.
java:425)
at
org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.
java:488)
at
org.datanucleus.store.rdbms.valuegenerator.TableGenerator.repositoryE
xists(TableGenerator.java:242)
at
org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obt
ainGenerationBlock(AbstractRDBMSGenerator.java:86)
at
org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerati
onBlock(AbstractGenerator.java:197)
at
org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractG
enerator.java:105)
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGene
rator(RDBMSStoreManager.java:2019)
at
org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractS
toreManager.java:1385)
at
[jira] [Commented] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820467#comment-13820467 ] Szehon Ho commented on HIVE-5717: - I understand we need to generate the javadocs and package them with the packaging/src.tar, but I am wondering what is the requirement on source jars? Where should they be bundled, if at all? Thanks. Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5717) Generate javadoc and source jars
[
https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820496#comment-13820496
]
Brock Noland commented on HIVE-5717:
I don't think this touches packaging at all. What we want to do is generate a
-javadoc.jar and -sources.jar for each maven artifact. Those new artifect
should be attached to the project so they publish to
~/.m2/repository/org/apache/hive
For example, notice how mrunit
(http://repo1.maven.org/maven2/org/apache/mrunit/mrunit/1.0.0/) has:
{noformat}
mrunit-1.0.0.jar
mrunit-1.0.0-sources.jar
mrunit-1.0.0-javadoc.jar
{noformat}
we want that exact same thing for every artifact in the project. Additionally
generating the javadoc can be very slow so we want to do that in the dist
profile.
Generate javadoc and source jars
Key: HIVE-5717
URL: https://issues.apache.org/jira/browse/HIVE-5717
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Szehon Ho
We should be generating both javadoc and source jars.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820498#comment-13820498 ] Brock Noland commented on HIVE-5717: bq. I don't think this touches packaging at all. Meaning do not worry about including them in packaging at this point. We'll do that later. Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820500#comment-13820500 ] Brock Noland commented on HIVE-5717: mrunit is doing some funky hadoop1 and hadoop2 classifier stuff. Whirr is probably a better example: http://repo1.maven.org/maven2/org/apache/whirr/whirr-core/0.8.2/ Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5802) Hive UDF pow limits the second input to integer when the first input is a decimal
Xuefu Zhang created HIVE-5802:
-
Summary: Hive UDF pow limits the second input to integer when the
first input is a decimal
Key: HIVE-5802
URL: https://issues.apache.org/jira/browse/HIVE-5802
Project: Hive
Issue Type: Improvement
Components: Types, UDF
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
For example,
{code}
hive desc test;
OK
i int None
b boolean None
d double None
s string None
dec decimal(5,2)None
hive explain select pow(dec, 2.5) from test;
FAILED: SemanticException [Error 10014]: Line 1:15 Wrong arguments '2.5': No
matching method for class org.apache.hadoop.hive.ql.udf.UDFPower with
(decimal(5,2), double). Possible choices: _FUNC_(decimal(65,30), int)
_FUNC_(double, double) _FUNC_(double, int)
{code}
This seems too restrictive. MySQL, on the other hand, supports fractional power
as the second input.
{code}
mysql select pow(d, 2.5) from test;
mysql desc test;
+---+--+--+-+-+---+
| Field | Type | Null | Key | Default | Extra |
+---+--+--+-+-+---+
| i | int(11) | YES | | NULL| |
| b | tinyint(1) | YES | | NULL| |
| d | double | YES | | NULL| |
| s | varchar(5) | YES | | NULL| |
| dd| decimal(5,2) | YES | | NULL| |
+---+--+--+-+-+---+
5 rows in set (0.00 sec)
mysql select pow(dd, 2.5) from test;
+---+
| pow(dd, 2.5) |
+---+
| 2570.215713318881 |
+---+
1 row in set (0.00 sec)
{code}
Hive should also support this.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations
[
https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820521#comment-13820521
]
Hive QA commented on HIVE-5356:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12613404/HIVE-5356.6.patch
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 4644 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_num_op_type_conv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_coalesce
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_in
org.apache.hadoop.hive.ql.exec.TestFunctionRegistry.testCommonClassComparison
org.apache.hadoop.hive.ql.exec.TestFunctionRegistry.testGetMethodInternal
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input8
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFPosMod.testDecimalPosModDecimal
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/249/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/249/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12613404
Move arithmatic UDFs to generic UDF implementations
---
Key: HIVE-5356
URL: https://issues.apache.org/jira/browse/HIVE-5356
Project: Hive
Issue Type: Task
Components: UDF
Affects Versions: 0.11.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.13.0
Attachments: HIVE-5356.1.patch, HIVE-5356.2.patch, HIVE-5356.3.patch,
HIVE-5356.4.patch, HIVE-5356.5.patch, HIVE-5356.6.patch, HIVE-5356.patch
Currently, all of the arithmetic operators, such as add/sub/mult/div, are
implemented as old-style UDFs and java reflection is used to determine the
return type TypeInfos/ObjectInspectors, based on the return type of the
evaluate() method chosen for the expression. This works fine for types that
don't have type params.
Hive decimal type participates in these operations just like int or double.
Different from double or int, however, decimal has precision and scale, which
cannot be determined by just looking at the return type (decimal) of the UDF
evaluate() method, even though the operands have certain precision/scale.
With the default of decimal without precision/scale, then (10, 0) will be
the type params. This is certainly not desirable.
To solve this problem, all of the arithmetic operators would need to be
implemented as GenericUDFs, which allow returning ObjectInspector during the
initialize() method. The object inspectors returned can carry type params,
from which the exact return type can be determined.
It's worth mentioning that, for user UDF implemented in non-generic way, if
the return type of the chosen evaluate() method is decimal, the return type
actually has (10,0) as precision/scale, which might not be desirable. This
needs to be documented.
This JIRA will cover minus, plus, divide, multiply, mod, and pmod, to limit
the scope of review. The remaining ones will be covered under HIVE-5706.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-2055) Hive HBase Integration issue
[ https://issues.apache.org/jira/browse/HIVE-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820558#comment-13820558 ] Nick Dimiduk commented on HIVE-2055: My above link has turned stale. I'm referring to FileUtil#createJarWithClassPath. Calling this method to construct a classpath jar every time a script calls $(hbase classpath-min) sounds time-consuming and will leave temp jars orphaned on the FS. Another idea is to use maven to generate a dependency list that omits Hadoop and other runtime jars? Any friends at BigTop who could give advice on this approach? (cc [~rvs], [~mackrorysd]) Yet another option is to maintain the list manually on the HBase side. We already sort of do with in TableMapReduceUtils#addDependencyJars. This is probably the simplest but most fragile and least future-proof option. Hive HBase Integration issue Key: HIVE-2055 URL: https://issues.apache.org/jira/browse/HIVE-2055 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.10.0 Reporter: sajith v Attachments: 0001-HIVE-2055-include-hbase-dependencies-in-launch-scrip.patch, HIVE-2055.patch Created an external table in hive , which points to the HBase table. When tried to query a column using the column name in select clause got the following exception : ( java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat), errorCode:12, SQLState:42000) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5557) Push down qualifying Where clause predicates as join conditions
[ https://issues.apache.org/jira/browse/HIVE-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-5557: Attachment: HIVE-5557.5.patch Push down qualifying Where clause predicates as join conditions --- Key: HIVE-5557 URL: https://issues.apache.org/jira/browse/HIVE-5557 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-5557.1.patch, HIVE-5557.2.patch, HIVE-5557.3.patch, HIVE-5557.4.patch, HIVE-5557.5.patch See details in HIVE- -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5557) Push down qualifying Where clause predicates as join conditions
[ https://issues.apache.org/jira/browse/HIVE-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-5557: Status: Open (was: Patch Available) Push down qualifying Where clause predicates as join conditions --- Key: HIVE-5557 URL: https://issues.apache.org/jira/browse/HIVE-5557 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-5557.1.patch, HIVE-5557.2.patch, HIVE-5557.3.patch, HIVE-5557.4.patch, HIVE-5557.5.patch See details in HIVE- -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5557) Push down qualifying Where clause predicates as join conditions
[ https://issues.apache.org/jira/browse/HIVE-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Butani updated HIVE-5557: Status: Patch Available (was: Open) Push down qualifying Where clause predicates as join conditions --- Key: HIVE-5557 URL: https://issues.apache.org/jira/browse/HIVE-5557 Project: Hive Issue Type: Sub-task Components: Query Processor Reporter: Harish Butani Assignee: Harish Butani Attachments: HIVE-5557.1.patch, HIVE-5557.2.patch, HIVE-5557.3.patch, HIVE-5557.4.patch, HIVE-5557.5.patch See details in HIVE- -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-2055) Hive should add HBase classpath dependencies when available
[ https://issues.apache.org/jira/browse/HIVE-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HIVE-2055: --- Summary: Hive should add HBase classpath dependencies when available (was: Hive HBase Integration issue) Hive should add HBase classpath dependencies when available --- Key: HIVE-2055 URL: https://issues.apache.org/jira/browse/HIVE-2055 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.10.0 Reporter: sajith v Attachments: 0001-HIVE-2055-include-hbase-dependencies-in-launch-scrip.patch, HIVE-2055.patch Created an external table in hive , which points to the HBase table. When tried to query a column using the column name in select clause got the following exception : ( java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat), errorCode:12, SQLState:42000) -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15435: Add long polling to asynchronous execution in HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15435/#review28749 --- service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java https://reviews.apache.org/r/15435/#comment55768 Should probably use a long and HiveConf.getLongVar instead of ints. service/src/test/org/apache/hive/service/cli/CLIServiceTest.java https://reviews.apache.org/r/15435/#comment55770 Please try to avoid using concrete collection types on the LHS or assignments or in method parameter lists, i.e. use MapString, String instead of HashMapString, String. service/src/test/org/apache/hive/service/cli/CLIServiceTest.java https://reviews.apache.org/r/15435/#comment55742 Please remove TABs service/src/test/org/apache/hive/service/cli/CLIServiceTest.java https://reviews.apache.org/r/15435/#comment55767 Replace hive.server2.long.polling.timeout with HiveConf.ConfVars.HIVE_SERVER2_LONG_POLLING_TIMEOUT.varname. service/src/test/org/apache/hive/service/cli/CLIServiceTest.java https://reviews.apache.org/r/15435/#comment55764 Please use a switch statement here. service/src/test/org/apache/hive/service/cli/CLIServiceTest.java https://reviews.apache.org/r/15435/#comment55769 Formatting. - Carl Steinbach On Nov. 12, 2013, 2:16 a.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15435/ --- (Updated Nov. 12, 2013, 2:16 a.m.) Review request for hive, Carl Steinbach and Thejas Nair. Bugs: HIVE-5217 https://issues.apache.org/jira/browse/HIVE-5217 Repository: hive-git Description --- Add long polling to asynchronous execution in HiveServer2 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4f32390 conf/hive-default.xml.template fe7141e service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 4ee1b74 service/src/test/org/apache/hive/service/cli/CLIServiceTest.java cd9d99a Diff: https://reviews.apache.org/r/15435/diff/ Testing --- Thanks, Vaibhav Gumashta
[jira] [Updated] (HIVE-5564) Need to accomodate table decimal columns that were defined prior to HIVE-3976
[
https://issues.apache.org/jira/browse/HIVE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-5564:
--
Attachment: (was: HIVE-5564.4.patch)
Need to accomodate table decimal columns that were defined prior to HIVE-3976
-
Key: HIVE-5564
URL: https://issues.apache.org/jira/browse/HIVE-5564
Project: Hive
Issue Type: Task
Components: Types
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.13.0
Attachments: HIVE-5564.1.patch, HIVE-5564.2.patch, HIVE-5564.3.patch,
HIVE-5564.patch
With HIVE-3976, decimal columns are stored with precision/scale, such as
decimal(17,5), as the type name. However, such columns defined in hive prior
to HIVE-3976 have a name as decimal. Those columns need to continue to work
with a precision/scale as (10,0), per the functional doc. With patch in
HIVE-3976, we may get the following error message in such case:
{code}
0: jdbc:hive2://localhost:1 desc dec;
Error: Error while processing statement: FAILED: RuntimeException Decimal
type is specified without length: decimal:int (state=42000,code=4)
{code}
This issue will be addressed in this JIRA as a follow-up task.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5564) Need to accomodate table decimal columns that were defined prior to HIVE-3976
[
https://issues.apache.org/jira/browse/HIVE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-5564:
--
Attachment: HIVE-5564.4.patch
Reload the same patch to kick off test.
Need to accomodate table decimal columns that were defined prior to HIVE-3976
-
Key: HIVE-5564
URL: https://issues.apache.org/jira/browse/HIVE-5564
Project: Hive
Issue Type: Task
Components: Types
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Fix For: 0.13.0
Attachments: HIVE-5564.1.patch, HIVE-5564.2.patch, HIVE-5564.3.patch,
HIVE-5564.4.patch, HIVE-5564.patch
With HIVE-3976, decimal columns are stored with precision/scale, such as
decimal(17,5), as the type name. However, such columns defined in hive prior
to HIVE-3976 have a name as decimal. Those columns need to continue to work
with a precision/scale as (10,0), per the functional doc. With patch in
HIVE-3976, we may get the following error message in such case:
{code}
0: jdbc:hive2://localhost:1 desc dec;
Error: Error while processing statement: FAILED: RuntimeException Decimal
type is specified without length: decimal:int (state=42000,code=4)
{code}
This issue will be addressed in this JIRA as a follow-up task.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5217) Add long polling to asynchronous execution in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820574#comment-13820574 ] Carl Steinbach commented on HIVE-5217: -- I left some more comments on reviewboard. Are you planning to move the long polling timeout logic from SQLOperation.getState() to CLIService? Add long polling to asynchronous execution in HiveServer2 - Key: HIVE-5217 URL: https://issues.apache.org/jira/browse/HIVE-5217 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-5217.2.patch, HIVE-5217.3.patch, HIVE-5217.D12801.2.patch, HIVE-5217.D12801.3.patch, HIVE-5217.D12801.4.patch, HIVE-5217.D12801.5.patch, HIVE-5217.D12801.6.patch [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support for async execution in HS2. The client gets an operation handle which it can poll to check on the operation status. However, the polling frequency is entirely left to the client which can be resource inefficient. Long polling will solve this, by blocking the client request to check the operation status for a configurable amount of time (a new HS2 config) if the data is not available, but responding immediately if the data is available. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-5717: Attachment: HIVE-5717.patch Attaching first cut. Any thoughts on a separate profile for this build step? I feel dist profile is quite useful as a tool for developers to verify their changes in the final product, like what the old /build/dist library used to do in ant. But whereas it used to bring total build time from 45s to just over 1 min before, now it takes it to almost 4 mins on my machine. Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho Attachments: HIVE-5717.patch We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
Review Request 15462: HIVE-5717 Generate javadoc and source jars
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15462/ --- Review request for hive. Bugs: HIVE-5717 https://issues.apache.org/jira/browse/HIVE-5717 Repository: hive-git Description --- Add maven javadoc and sources goal to the hive build. Diffs - pom.xml 05d887f Diff: https://reviews.apache.org/r/15462/diff/ Testing --- Thanks, Szehon Ho
[jira] [Commented] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820595#comment-13820595 ] Brock Noland commented on HIVE-5717: perhaps we should have separate profiles called javadocs and sources It's only important when we are doing a release or publishing a snapshot. Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho Attachments: HIVE-5717.patch We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HIVE-5803) Support CTAS from a non-avro table to an avro table
[ https://issues.apache.org/jira/browse/HIVE-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam reassigned HIVE-5803: --- Assignee: Carl Steinbach Support CTAS from a non-avro table to an avro table --- Key: HIVE-5803 URL: https://issues.apache.org/jira/browse/HIVE-5803 Project: Hive Issue Type: Task Reporter: Mohammad Kamrul Islam Assignee: Carl Steinbach Hive currently does not work with HQL like : CREATE TABLE AVRO-BASE-TABLE as SELECT * from NON_AVRO_TABLE; Actual it works successfully. But when I run SELECT * from AVRO-BASED-TABLE .. it fails. This JIRA depends on HIVE-3159 that translates TypeInfo to Avro schema. Findings so far: CTAS uses internal column names (in place of using the column names provided in select) when crating the AVRO data file. In other words, avro data file has column names in this form of: _col0, _col1 where as table column names are different. I tested with the following test cases and it failed: - verify 1) can create table using create table as select from non-avro table 2) LOAD avro data into new table and read data from the new table CREATE TABLE simple_kv_txt (key STRING, value STRING) STORED AS TEXTFILE; DESCRIBE simple_kv_txt; LOAD DATA LOCAL INPATH '../data/files/kv1.txt' INTO TABLE simple_kv_txt; SELECT * FROM simple_kv_txt ORDER BY KEY; CREATE TABLE copy_doctors ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' as SELECT key as key, value as value FROM simple_kv_txt; DESCRIBE copy_doctors; SELECT * FROM copy_doctors; -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5803) Support CTAS from a non-avro table to an avro table
Mohammad Kamrul Islam created HIVE-5803: --- Summary: Support CTAS from a non-avro table to an avro table Key: HIVE-5803 URL: https://issues.apache.org/jira/browse/HIVE-5803 Project: Hive Issue Type: Task Reporter: Mohammad Kamrul Islam Hive currently does not work with HQL like : CREATE TABLE AVRO-BASE-TABLE as SELECT * from NON_AVRO_TABLE; Actual it works successfully. But when I run SELECT * from AVRO-BASED-TABLE .. it fails. This JIRA depends on HIVE-3159 that translates TypeInfo to Avro schema. Findings so far: CTAS uses internal column names (in place of using the column names provided in select) when crating the AVRO data file. In other words, avro data file has column names in this form of: _col0, _col1 where as table column names are different. I tested with the following test cases and it failed: - verify 1) can create table using create table as select from non-avro table 2) LOAD avro data into new table and read data from the new table CREATE TABLE simple_kv_txt (key STRING, value STRING) STORED AS TEXTFILE; DESCRIBE simple_kv_txt; LOAD DATA LOCAL INPATH '../data/files/kv1.txt' INTO TABLE simple_kv_txt; SELECT * FROM simple_kv_txt ORDER BY KEY; CREATE TABLE copy_doctors ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' as SELECT key as key, value as value FROM simple_kv_txt; DESCRIBE copy_doctors; SELECT * FROM copy_doctors; -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15151: Better error reporting by async threads in HiveServer2
On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/if/TCLIService.thrift, line 395 https://reviews.apache.org/r/15151/diff/3/?file=381909#file381909line395 Please reuse TStatus instead of adding a new struct. My understanding is that TStatus encapsulates the status of an RPC request (as pointed out by Prasad earlier). It has a required TStatusCode field as well which corresponds to the status of an RPC call. Would it not be better to expand TGetOperationStatusResp with sqlState, errorCode and errorMessage? On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: itests/hive-unit/src/test/java/org/apache/hive/service/cli/CLIServiceTest.java, line 1 https://reviews.apache.org/r/15151/diff/3/?file=381906#file381906line1 Does this patch make any changes to CLIServiceTest or ThriftCLIServiceTest, or does it just move these files from service/ to itests/ ? If it does make changes can we move the files in a different patch? It moves these files, but also adds test for this feature + some minor refactoring. On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/if/TCLIService.thrift, line 50 https://reviews.apache.org/r/15151/diff/3/?file=381909#file381909line50 Need to add HIVE_CLI_SERVICE_PROTOCOL_V5 and update any references to HIVE_CLI_SERVICE_PROTOCOL_V4. Done On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/src/java/org/apache/hive/service/cli/CLIService.java, line 274 https://reviews.apache.org/r/15151/diff/3/?file=381910#file381910line274 Please push this logic into OperationManager.getOperationStatus() Done On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/if/TCLIService.thrift, line 917 https://reviews.apache.org/r/15151/diff/3/?file=381909#file381909line917 This modification will break compatibility between upversion/downversion clients and servers since it modifies the type of an existing field. It's possible to avoid this problem by instead adding a new TStatus operationStatus field. Sorry about the slip, will expand TGetOperationStatusResp with a new optional field(s). Would appreciate your thoughts on my reply in the previous comment regarding TStatus. On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/src/java/org/apache/hive/service/cli/operation/OperationManager.java, line 147 https://reviews.apache.org/r/15151/diff/3/?file=381918#file381918line147 s/getOperationRunException/getOperationException/ Done On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java, line 71 https://reviews.apache.org/r/15151/diff/3/?file=381919#file381919line71 s/runException/operationException/ Done On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/src/java/org/apache/hive/service/cli/operation/Operation.java, line 73 https://reviews.apache.org/r/15151/diff/3/?file=381917#file381917line73 Would it make sense to replace getState() and getRunException() with a getStatus() method that returns an object wrapping the operationState and operationException? If not, please change the name of getRunException to getException(), and add a comment explaining under what conditions this method will return a non-null value. Done On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java, line 324 https://reviews.apache.org/r/15151/diff/3/?file=381919#file381919line324 May as well just push this into Operation. Done On Nov. 11, 2013, 7:52 p.m., Carl Steinbach wrote: service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java, line 307 https://reviews.apache.org/r/15151/diff/3/?file=381921#file381921line307 Todo: set the status information. Could you elaborate this a bit more? Thanks for the feedback. - Vaibhav --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15151/#review28677 --- On Nov. 11, 2013, 7:23 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15151/ --- (Updated Nov. 11, 2013, 7:23 p.m.) Review request for hive, Prasad Mujumdar and Thejas Nair. Bugs: HIVE-5230 https://issues.apache.org/jira/browse/HIVE-5230 Repository: hive-git Description --- [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support for async execution in HS2. When a background thread gets an error, currently the client can only poll for the operation state and also the error with its stacktrace is logged. However, it
Re: Review Request 15151: Better error reporting by async threads in HiveServer2
On Nov. 11, 2013, 7:54 p.m., Thejas Nair wrote: service/if/TCLIService.thrift, line 917 https://reviews.apache.org/r/15151/diff/4/?file=382149#file382149line917 This will not be backward compatible. It would be better to add a new optional field in the response, that has the additional error information. Fixed. Thanks for pointing out! - Vaibhav --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15151/#review28683 --- On Nov. 11, 2013, 7:23 p.m., Vaibhav Gumashta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15151/ --- (Updated Nov. 11, 2013, 7:23 p.m.) Review request for hive, Prasad Mujumdar and Thejas Nair. Bugs: HIVE-5230 https://issues.apache.org/jira/browse/HIVE-5230 Repository: hive-git Description --- [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support for async execution in HS2. When a background thread gets an error, currently the client can only poll for the operation state and also the error with its stacktrace is logged. However, it will be useful to provide a richer error response like thrift API does with TStatus (which is constructed while building a Thrift response object). Diffs - itests/hive-unit/src/test/java/org/apache/hive/service/cli/CLIServiceTest.java PRE-CREATION itests/hive-unit/src/test/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java fce19bf service/if/TCLIService.thrift 1f49445 service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 service/src/java/org/apache/hive/service/cli/HiveSQLException.java 74e8b94 service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 service/src/java/org/apache/hive/service/cli/OperationState.java 1ec6bd1 service/src/java/org/apache/hive/service/cli/OperationStatus.java PRE-CREATION service/src/java/org/apache/hive/service/cli/operation/Operation.java 6f4b8dc service/src/java/org/apache/hive/service/cli/operation/OperationManager.java bcdb67f service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 4ee1b74 service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 9df110e service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f service/src/test/org/apache/hive/service/cli/CLIServiceTest.java cd9d99a service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java ff7166d Diff: https://reviews.apache.org/r/15151/diff/ Testing --- Thanks, Vaibhav Gumashta
Re: Review Request 15462: HIVE-5717 Generate javadoc and source jars
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15462/ --- (Updated Nov. 12, 2013, 11:05 p.m.) Review request for hive. Changes --- Using separate profiles for these build goals, to not affect dist profile's build time. Bugs: HIVE-5717 https://issues.apache.org/jira/browse/HIVE-5717 Repository: hive-git Description --- Add maven javadoc and sources goal to the hive build. Diffs (updated) - pom.xml 05d887f Diff: https://reviews.apache.org/r/15462/diff/ Testing --- Thanks, Szehon Ho
Re: Review Request 11925: Hive-3159 Update AvroSerde to determine schema of new tables
On July 29, 2013, 5:41 p.m., Jakob Homan wrote: There is still no text covering a map-reduce job on an already existing, non-Avro table into an avro table. ie, create a text table, populate it, run a CTAS to manipulate the data into an Avro table. Mohammad Islam wrote: In general, Hive creates internal column names such as col0, col1 etc. Due to this, I didn't able to copy non-avro data to avro data and run select SQL. Only option is to change the current behavior to reuse the provided column names. Separate JIRA regarding this could be a choice. Jakob Homan wrote: Wouldn't select * or using the new column names (they're named deterministically) work? This is a major test since otherwise we're missing the most important code path... ie have a text file c1, c2, c3 create table t1 load data into t1 from text file create table a1 as select c3, c2 where c2 = foo order by c3; select * from a1; describe extended a1; And verify in the q file's result that the table is avro and that the correct rows and columns got converted. I agree that we need to support this use case as well. Current patch works on two other uses case. Created a separate JIRA to handle this use case that I suspect is a slightly different issue. New JIRA: https://issues.apache.org/jira/browse/HIVE-5803 - Mohammad --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11925/#review24149 --- On Aug. 8, 2013, 12:24 a.m., Mohammad Islam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11925/ --- (Updated Aug. 8, 2013, 12:24 a.m.) Review request for hive, Ashutosh Chauhan and Jakob Homan. Bugs: HIVE-3159 https://issues.apache.org/jira/browse/HIVE-3159 Repository: hive-git Description --- Problem: Hive doesn't support to create a Avro-based table using HQL create table command. It currently requires to specify Avro schema literal or schema file name. For multiple cases, it is very inconvenient for user. Some of the un-supported use cases: 1. Create table ... Avro-SERDE etc. as SELECT ... from NON-AVRO FILE 2. Create table ... Avro-SERDE etc. as SELECT from AVRO TABLE 3. Create table without specifying Avro schema. Diffs - ql/src/test/queries/clientpositive/avro_create_as_select.q PRE-CREATION ql/src/test/queries/clientpositive/avro_create_as_select2.q PRE-CREATION ql/src/test/queries/clientpositive/avro_no_schema_test.q PRE-CREATION ql/src/test/queries/clientpositive/avro_without_schema.q PRE-CREATION ql/src/test/results/clientpositive/avro_create_as_select.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_create_as_select2.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_no_schema_test.q.out PRE-CREATION ql/src/test/results/clientpositive/avro_without_schema.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 13848b6 serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java 010f614 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java PRE-CREATION Diff: https://reviews.apache.org/r/11925/diff/ Testing --- Wrote a new java Test class for a new Java class. Added a new test case into existing java test class. In addition, there are 4 .q file for testing multiple use-cases. Thanks, Mohammad Islam
Re: Review Request 15151: Better error reporting by async threads in HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15151/ --- (Updated Nov. 12, 2013, 11:06 p.m.) Review request for hive, Carl Steinbach, Prasad Mujumdar, and Thejas Nair. Bugs: HIVE-5230 https://issues.apache.org/jira/browse/HIVE-5230 Repository: hive-git Description --- [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support for async execution in HS2. When a background thread gets an error, currently the client can only poll for the operation state and also the error with its stacktrace is logged. However, it will be useful to provide a richer error response like thrift API does with TStatus (which is constructed while building a Thrift response object). Diffs - itests/hive-unit/src/test/java/org/apache/hive/service/cli/CLIServiceTest.java PRE-CREATION itests/hive-unit/src/test/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java PRE-CREATION jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java fce19bf service/if/TCLIService.thrift 1f49445 service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9dca874 service/src/java/org/apache/hive/service/cli/HiveSQLException.java 74e8b94 service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 service/src/java/org/apache/hive/service/cli/OperationState.java 1ec6bd1 service/src/java/org/apache/hive/service/cli/OperationStatus.java PRE-CREATION service/src/java/org/apache/hive/service/cli/operation/Operation.java 6f4b8dc service/src/java/org/apache/hive/service/cli/operation/OperationManager.java bcdb67f service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 4ee1b74 service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 9df110e service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 9bb2a0f service/src/test/org/apache/hive/service/cli/CLIServiceTest.java cd9d99a service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java ff7166d Diff: https://reviews.apache.org/r/15151/diff/ Testing --- Thanks, Vaibhav Gumashta
[jira] [Updated] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-5717: Attachment: HIVE-5717.2.patch Yep, that sounds better. Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho Attachments: HIVE-5717.2.patch, HIVE-5717.patch We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-5717: Status: Patch Available (was: Open) Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho Attachments: HIVE-5717.2.patch, HIVE-5717.patch We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-3159) Update AvroSerde to determine schema of new tables
[ https://issues.apache.org/jira/browse/HIVE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated HIVE-3159: Attachment: HIVE-3159.4.patch Update AvroSerde to determine schema of new tables -- Key: HIVE-3159 URL: https://issues.apache.org/jira/browse/HIVE-3159 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.10.0, 0.11.0 Reporter: Jakob Homan Assignee: Mohammad Kamrul Islam Attachments: HIVE-3159.4.patch, HIVE-3159v1.patch Currently when writing tables to Avro one must manually provide an Avro schema that matches what is being delivered by Hive. It'd be better to have the serde infer this schema by converting the table's TypeInfo into an appropriate AvroSchema. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-3159) Update AvroSerde to determine schema of new tables
[ https://issues.apache.org/jira/browse/HIVE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated HIVE-3159: Affects Version/s: (was: 0.11.0) (was: 0.10.0) Status: Patch Available (was: Open) Update AvroSerde to determine schema of new tables -- Key: HIVE-3159 URL: https://issues.apache.org/jira/browse/HIVE-3159 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Reporter: Jakob Homan Assignee: Mohammad Kamrul Islam Attachments: HIVE-3159.4.patch, HIVE-3159v1.patch Currently when writing tables to Avro one must manually provide an Avro schema that matches what is being delivered by Hive. It'd be better to have the serde infer this schema by converting the table's TypeInfo into an appropriate AvroSchema. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5745) TestHiveLogging is failing (at least on mac)
[ https://issues.apache.org/jira/browse/HIVE-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5745: --- Fix Version/s: 0.13.0 TestHiveLogging is failing (at least on mac) Key: HIVE-5745 URL: https://issues.apache.org/jira/browse/HIVE-5745 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.13.0 Attachments: HIVE-5745.1.patch The path for the log file on my mac contains two slashes. That causes mvn install fail. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5801) Support for reader/writer of ORC format for R environment
[ https://issues.apache.org/jira/browse/HIVE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820640#comment-13820640 ] Andrew Psaltis commented on HIVE-5801: -- With our use case, being able to read and write ORC files outside of Hive is very desirable. We need to read and or write ORC files from both Hadoop Map Reduce jobs and more often than not outside of Hadoop. Having no Hive installations or Hive Metastore, we really do not have a desire to take on the HCatalog layer integration and the overhead that comes with that. To date we have maintained a patched version of the hive-exec jar that exposes the ORC related classes so that we can use it. Obviously this comes with it's own burden, and just recently we have been evaluating using Parquet over ORC, primarily for this very reason. However, we keep coming back to ORC, as some of the features better suit our use case. I would be more than happy to attach a patch to this JIRA, if that is something is desired. Support for reader/writer of ORC format for R environment - Key: HIVE-5801 URL: https://issues.apache.org/jira/browse/HIVE-5801 Project: Hive Issue Type: Improvement Reporter: Michael Hausenblas Priority: Minor It would be great if the ORC format would directly be accessible from R [1], that is, providing reader/writer for it. [1] http://www.r-project.org/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5804) Support expressions in GBY attributes, JOIN condition etc.
Prasanth J created HIVE-5804: Summary: Support expressions in GBY attributes, JOIN condition etc. Key: HIVE-5804 URL: https://issues.apache.org/jira/browse/HIVE-5804 Project: Hive Issue Type: Sub-task Reporter: Prasanth J Assignee: Prasanth J Support for expression in group by, join conditions are still not complete. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5369) Annotate hive operator tree with statistics from metastore
[ https://issues.apache.org/jira/browse/HIVE-5369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-5369: - Attachment: HIVE-5369.4.patch.txt Refactored patch. Removed dependency from lineage index. Statistics (along with column statistics) are now passed down the operator tree. Still there are few open issues. Will create sub-tasks for open issues. Annotate hive operator tree with statistics from metastore -- Key: HIVE-5369 URL: https://issues.apache.org/jira/browse/HIVE-5369 Project: Hive Issue Type: New Feature Components: Query Processor, Statistics Affects Versions: 0.13.0 Reporter: Prasanth J Assignee: Prasanth J Labels: statistics Fix For: 0.13.0 Attachments: HIVE-5369.1.txt, HIVE-5369.2.WIP.txt, HIVE-5369.2.patch.txt, HIVE-5369.3.patch.txt, HIVE-5369.4.patch.txt, HIVE-5369.WIP.txt, HIVE-5369.refactor.WIP.txt Currently the statistics gathered at table/partition level and column level are not used during query planning stage. Statistics at table/partition and column level can be used for optimizing the query plans. Basic statistics like uncompressed data size can be used for better reducer estimation. Other statistics like number of rows, distinct values of columns, average length of columns etc. can be used by Cost Based Optimizer (CBO) for making better query plan selection. As a first step in improving query planning the statistics that are available in the metastore should be attached to hive operator tree. The operator tree should be walked and annotated with statistics information. The attached statistics will vary for each operator depending on the operation it performs. For example, select operator will change the average row size but doesn't affect the number of rows. Similarly filter operator will change the number of rows but doesn't change the average row size. Similar rules can be applied for other operators as well. Rules for different operators are added as comments in the code. For more detailed information, the reference book that I am using is Database Systems: The Complete Book by Garcia-Molina et.al. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5805) Support for operators like PTF, Script, Extract etc. in statistics annotation.
Prasanth J created HIVE-5805: Summary: Support for operators like PTF, Script, Extract etc. in statistics annotation. Key: HIVE-5805 URL: https://issues.apache.org/jira/browse/HIVE-5805 Project: Hive Issue Type: Sub-task Reporter: Prasanth J Statistics annotation (HIVE-5369) only supports table scan, select, filter, limit, union, groupby, join. This sub task is to add support for remaining operators. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4388) Upgrade HBase to 0.96
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820680#comment-13820680 ] Gunther Hagleitner commented on HIVE-4388: -- +1 It'd be great to get that in. Minor thing: The hamcrest dependency I added is probably no longer needed with your fix of adding junit as a dependency. We can address that in a follow up though. Upgrade HBase to 0.96 - Key: HIVE-4388 URL: https://issues.apache.org/jira/browse/HIVE-4388 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Gunther Hagleitner Assignee: Brock Noland Attachments: HIVE-4388-wip.txt, HIVE-4388.10.patch, HIVE-4388.11.patch, HIVE-4388.12.patch, HIVE-4388.13.patch, HIVE-4388.14.patch, HIVE-4388.15.patch, HIVE-4388.15.patch, HIVE-4388.16.patch, HIVE-4388.17.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch Currently we're building by default against 0.92. When you run against hadoop 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4518) Counter Strike: Operation Operator
[ https://issues.apache.org/jira/browse/HIVE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-4518: - Attachment: HIVE-4518.8.patch doesn't look like pre-commit tests ran from the last patch .. uploading patch again to run to kick off another test run. Counter Strike: Operation Operator -- Key: HIVE-4518 URL: https://issues.apache.org/jira/browse/HIVE-4518 Project: Hive Issue Type: Improvement Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-4518.1.patch, HIVE-4518.2.patch, HIVE-4518.3.patch, HIVE-4518.4.patch, HIVE-4518.5.patch, HIVE-4518.6.patch.txt, HIVE-4518.7.patch, HIVE-4518.8.patch Queries of the form: from foo insert overwrite table bar partition (p) select ... insert overwrite table bar partition (p) select ... insert overwrite table bar partition (p) select ... Generate a huge amount of counters. The reason is that task.progress is turned on for dynamic partitioning queries. The counters not only make queries slower than necessary (up to 50%) you will also eventually run out. That's because we're wrapping them in enum values to comply with hadoop 0.17. The real reason we turn task.progress on is that we need CREATED_FILES and FATAL counters to ensure dynamic partitioning queries don't go haywire. The counters have counter-intuitive names like C1 through C1000 and don't seem really useful by themselves. With hadoop 20+ you don't need to wrap the counters anymore, each operator can simply create and increment counters. That should simplify the code a lot. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4388) Upgrade HBase to 0.96
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820716#comment-13820716 ] Sushanth Sowmyan commented on HIVE-4388: +1 from me as well. Code changes look unchanged and okay. Build changes look okay, and builds okay. Tests are obviously reflected by the automated build. Looks great! :) Upgrade HBase to 0.96 - Key: HIVE-4388 URL: https://issues.apache.org/jira/browse/HIVE-4388 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Gunther Hagleitner Assignee: Brock Noland Attachments: HIVE-4388-wip.txt, HIVE-4388.10.patch, HIVE-4388.11.patch, HIVE-4388.12.patch, HIVE-4388.13.patch, HIVE-4388.14.patch, HIVE-4388.15.patch, HIVE-4388.15.patch, HIVE-4388.16.patch, HIVE-4388.17.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch Currently we're building by default against 0.92. When you run against hadoop 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820734#comment-13820734 ] Lefty Leverenz commented on HIVE-5717: -- Will the javadocs include HCatalog? In the Hive 0.12 javadocs, the only HCat docs I can find are for shims. Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho Attachments: HIVE-5717.2.patch, HIVE-5717.patch We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5731) Use new GenericUDF instead of basic UDF for UDFDate* classes
[ https://issues.apache.org/jira/browse/HIVE-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820736#comment-13820736 ] Mohammad Kamrul Islam commented on HIVE-5731: - RB Updated. Use new GenericUDF instead of basic UDF for UDFDate* classes - Key: HIVE-5731 URL: https://issues.apache.org/jira/browse/HIVE-5731 Project: Hive Issue Type: Improvement Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Attachments: HIVE-5731.1.patch, HIVE-5731.2.patch, HIVE-5731.3.patch, HIVE-5731.4.patch, HIVE-5731.5.patch GenericUDF class is the latest and recommended base class for any UDFs. This JIRA is to change the current UDFDate* classes extended from GenericUDF. The general benefit of GenericUDF is described in comments as * The GenericUDF are superior to normal UDFs in the following ways: 1. It can accept arguments of complex types, and return complex types. 2. It can accept variable length of arguments. 3. It can accept an infinite number of function signature - for example, it's easy to write a GenericUDF that accepts arrayint, arrayarrayint and so on (arbitrary levels of nesting). 4. It can do short-circuit evaluations using DeferedObject. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4388) Upgrade HBase to 0.96
[ https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820744#comment-13820744 ] Brock Noland commented on HIVE-4388: Thanks guys! Upgrade HBase to 0.96 - Key: HIVE-4388 URL: https://issues.apache.org/jira/browse/HIVE-4388 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Gunther Hagleitner Assignee: Brock Noland Attachments: HIVE-4388-wip.txt, HIVE-4388.10.patch, HIVE-4388.11.patch, HIVE-4388.12.patch, HIVE-4388.13.patch, HIVE-4388.14.patch, HIVE-4388.15.patch, HIVE-4388.15.patch, HIVE-4388.16.patch, HIVE-4388.17.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch Currently we're building by default against 0.92. When you run against hadoop 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963. HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5801) Support for reader/writer of ORC format for R environment
[ https://issues.apache.org/jira/browse/HIVE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820750#comment-13820750 ] Owen O'Malley commented on HIVE-5801: - Have you seen HIVE-5728? It is making some changes to the ORC reader/writer so that Pig can add native load/store functions. What else do you require? Support for reader/writer of ORC format for R environment - Key: HIVE-5801 URL: https://issues.apache.org/jira/browse/HIVE-5801 Project: Hive Issue Type: Improvement Reporter: Michael Hausenblas Priority: Minor It would be great if the ORC format would directly be accessible from R [1], that is, providing reader/writer for it. [1] http://www.r-project.org/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5801) Support for reader/writer of ORC format for R environment
[ https://issues.apache.org/jira/browse/HIVE-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820769#comment-13820769 ] Carter Shanklin commented on HIVE-5801: --- Haven't seen that, thanks. Is there a way people can use that before Hive 13? Say on the CDH Hive 12 + a billion patches thing or on older HDP? -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. Support for reader/writer of ORC format for R environment - Key: HIVE-5801 URL: https://issues.apache.org/jira/browse/HIVE-5801 Project: Hive Issue Type: Improvement Reporter: Michael Hausenblas Priority: Minor It would be great if the ORC format would directly be accessible from R [1], that is, providing reader/writer for it. [1] http://www.r-project.org/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5756) Implement vectorization support for IF conditional expression for long and double inputs
[ https://issues.apache.org/jira/browse/HIVE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5756: -- Summary: Implement vectorization support for IF conditional expression for long and double inputs (was: Implement vectorization support for IF conditional expression) Implement vectorization support for IF conditional expression for long and double inputs Key: HIVE-5756 URL: https://issues.apache.org/jira/browse/HIVE-5756 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Implement full, end-to-end support for IF in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because IF is the most popular conditional expression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5799) session/operation timeout for hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5799: Status: Open (was: Patch Available) session/operation timeout for hiveserver2 - Key: HIVE-5799 URL: https://issues.apache.org/jira/browse/HIVE-5799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5799.1.patch.txt Need some timeout facility for preventing resource leakages from instable or bad clients. -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 15449: session/operation timeout for hiveserver2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15449/ --- (Updated Nov. 13, 2013, 1:21 a.m.) Review request for hive. Bugs: HIVE-5799 https://issues.apache.org/jira/browse/HIVE-5799 Repository: hive-git Description --- Need some timeout facility for preventing resource leakages from instable or bad clients. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 32ab3d8 conf/hive-default.xml.template c574ab5 service/src/java/org/apache/hive/service/cli/OperationState.java 1ec6bd1 service/src/java/org/apache/hive/service/cli/operation/Operation.java 6f4b8dc service/src/java/org/apache/hive/service/cli/session/HiveSession.java 00058cc service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java cfda752 service/src/java/org/apache/hive/service/cli/session/SessionManager.java 25c6f38 Diff: https://reviews.apache.org/r/15449/diff/ Testing --- Confirmed in the local environment. Thanks, Navis Ryu
[jira] [Commented] (HIVE-5756) Implement vectorization support for IF conditional expression for long and double inputs
[ https://issues.apache.org/jira/browse/HIVE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820779#comment-13820779 ] Eric Hanson commented on HIVE-5756: --- This was getting big so I am going to do a separate JIRA or JIRAs for string, timestamp, and Boolean types. Implement vectorization support for IF conditional expression for long and double inputs Key: HIVE-5756 URL: https://issues.apache.org/jira/browse/HIVE-5756 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5756.1.patch Implement full, end-to-end support for IF in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because IF is the most popular conditional expression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5756) Implement vectorization support for IF conditional expression for long and double inputs
[ https://issues.apache.org/jira/browse/HIVE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5756: -- Attachment: HIVE-5756.1.patch Includes unit tests for VectorExpressions and VectorizationContext. I also ran end-to-end ad hoc tests to verify IF works in vectorized mode from SQL. Implement vectorization support for IF conditional expression for long and double inputs Key: HIVE-5756 URL: https://issues.apache.org/jira/browse/HIVE-5756 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5756.1.patch Implement full, end-to-end support for IF in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because IF is the most popular conditional expression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5799) session/operation timeout for hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5799: Attachment: HIVE-5799.2.patch.txt session/operation timeout for hiveserver2 - Key: HIVE-5799 URL: https://issues.apache.org/jira/browse/HIVE-5799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5799.1.patch.txt, HIVE-5799.2.patch.txt Need some timeout facility for preventing resource leakages from instable or bad clients. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5756) Implement vectorization support for IF conditional expression for long and double inputs
[ https://issues.apache.org/jira/browse/HIVE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson updated HIVE-5756: -- Status: Patch Available (was: In Progress) Implement vectorization support for IF conditional expression for long and double inputs Key: HIVE-5756 URL: https://issues.apache.org/jira/browse/HIVE-5756 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson Attachments: HIVE-5756.1.patch Implement full, end-to-end support for IF in vectorized mode, including new VectorExpression class(es), VectorizationContext translation to a VectorExpression, and unit tests for these, as well as end-to-end ad hoc testing. An end-to-end .q test is recommended but optional. This is high priority because IF is the most popular conditional expression. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5799) session/operation timeout for hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5799: Status: Patch Available (was: Open) session/operation timeout for hiveserver2 - Key: HIVE-5799 URL: https://issues.apache.org/jira/browse/HIVE-5799 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-5799.1.patch.txt, HIVE-5799.2.patch.txt Need some timeout facility for preventing resource leakages from instable or bad clients. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5807) Implement vectorization support for IF conditional expression for string inputs
Eric Hanson created HIVE-5807: - Summary: Implement vectorization support for IF conditional expression for string inputs Key: HIVE-5807 URL: https://issues.apache.org/jira/browse/HIVE-5807 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5806) Implement vectorization support for IF conditional expression for boolean and timestamp inputs
Eric Hanson created HIVE-5806: - Summary: Implement vectorization support for IF conditional expression for boolean and timestamp inputs Key: HIVE-5806 URL: https://issues.apache.org/jira/browse/HIVE-5806 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5806) Implement vectorization support for IF conditional expression for boolean and timestamp inputs
[ https://issues.apache.org/jira/browse/HIVE-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820785#comment-13820785 ] Eric Hanson commented on HIVE-5806: --- Try to piggy back this on the work for HIVE-5756. Implement vectorization support for IF conditional expression for boolean and timestamp inputs -- Key: HIVE-5806 URL: https://issues.apache.org/jira/browse/HIVE-5806 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HIVE-5806) Implement vectorization support for IF conditional expression for boolean and timestamp inputs
[ https://issues.apache.org/jira/browse/HIVE-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Hanson reassigned HIVE-5806: - Assignee: Eric Hanson Implement vectorization support for IF conditional expression for boolean and timestamp inputs -- Key: HIVE-5806 URL: https://issues.apache.org/jira/browse/HIVE-5806 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Work started] (HIVE-5806) Implement vectorization support for IF conditional expression for boolean and timestamp inputs
[ https://issues.apache.org/jira/browse/HIVE-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-5806 started by Eric Hanson. Implement vectorization support for IF conditional expression for boolean and timestamp inputs -- Key: HIVE-5806 URL: https://issues.apache.org/jira/browse/HIVE-5806 Project: Hive Issue Type: Sub-task Reporter: Eric Hanson Assignee: Eric Hanson -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5581) Implement vectorized year/month/day... etc. for string arguments
[ https://issues.apache.org/jira/browse/HIVE-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820788#comment-13820788 ] Eric Hanson commented on HIVE-5581: --- Teddy, when do you think you can update the patch? Implement vectorized year/month/day... etc. for string arguments Key: HIVE-5581 URL: https://issues.apache.org/jira/browse/HIVE-5581 Project: Hive Issue Type: Sub-task Components: Query Processor Affects Versions: 0.13.0 Reporter: Eric Hanson Assignee: Teddy Choi Attachments: HIVE-5581.1.patch.txt, HIVE-5581.2.patch, HIVE-5581.3.patch Functions year(), month(), day(), weekofyear(), hour(), minute(), second() need to be implemented for string arguments in vectorized mode. They already work for timestamp arguments. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5717) Generate javadoc and source jars
[ https://issues.apache.org/jira/browse/HIVE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820789#comment-13820789 ] Szehon Ho commented on HIVE-5717: - Yes, with this patch we would get that for free, thanks to hcatalog now being part of the same central maven hive build. Generate javadoc and source jars Key: HIVE-5717 URL: https://issues.apache.org/jira/browse/HIVE-5717 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Szehon Ho Attachments: HIVE-5717.2.patch, HIVE-5717.patch We should be generating both javadoc and source jars. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5808) broadcast join in tez discards duplicate records from the broadcasted table
Thejas M Nair created HIVE-5808: --- Summary: broadcast join in tez discards duplicate records from the broadcasted table Key: HIVE-5808 URL: https://issues.apache.org/jira/browse/HIVE-5808 Project: Hive Issue Type: Bug Components: Tez Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: tez-branch When the small(broadcasted table) in a join has duplicate records for a key, they are getting discarded, resulting in incorrect output. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5808) broadcast join in tez discards duplicate records from the broadcasted table
[ https://issues.apache.org/jira/browse/HIVE-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5808: Attachment: HIVE-5808.1.patch broadcast join in tez discards duplicate records from the broadcasted table --- Key: HIVE-5808 URL: https://issues.apache.org/jira/browse/HIVE-5808 Project: Hive Issue Type: Bug Components: Tez Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: tez-branch Attachments: HIVE-5808.1.patch When the small(broadcasted table) in a join has duplicate records for a key, they are getting discarded, resulting in incorrect output. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5808) broadcast join in tez discards duplicate records from the broadcasted table
[ https://issues.apache.org/jira/browse/HIVE-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-5808: Status: Patch Available (was: Open) broadcast join in tez discards duplicate records from the broadcasted table --- Key: HIVE-5808 URL: https://issues.apache.org/jira/browse/HIVE-5808 Project: Hive Issue Type: Bug Components: Tez Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: tez-branch Attachments: HIVE-5808.1.patch When the small(broadcasted table) in a join has duplicate records for a key, they are getting discarded, resulting in incorrect output. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HIVE-5809) incorrect stats in some cases with hive.stats.autogather=true
Ashutosh Chauhan created HIVE-5809: -- Summary: incorrect stats in some cases with hive.stats.autogather=true Key: HIVE-5809 URL: https://issues.apache.org/jira/browse/HIVE-5809 Project: Hive Issue Type: Bug Components: Statistics Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5809) incorrect stats in some cases with hive.stats.autogather=true
[ https://issues.apache.org/jira/browse/HIVE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5809: --- Attachment: HIVE-5809.patch preliminary patch incorrect stats in some cases with hive.stats.autogather=true Key: HIVE-5809 URL: https://issues.apache.org/jira/browse/HIVE-5809 Project: Hive Issue Type: Bug Components: Statistics Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-5809.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-5809) incorrect stats in some cases with hive.stats.autogather=true
[ https://issues.apache.org/jira/browse/HIVE-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5809: --- Status: Patch Available (was: Open) incorrect stats in some cases with hive.stats.autogather=true Key: HIVE-5809 URL: https://issues.apache.org/jira/browse/HIVE-5809 Project: Hive Issue Type: Bug Components: Statistics Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-5809.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HIVE-4621) describe formatted for table shows incorrect numRows data
[ https://issues.apache.org/jira/browse/HIVE-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-4621. Resolution: Invalid Resolving it as invalid, since I can't repro this. Feel free to reopen if you can reproduce it. Also, provide steps to repro in case you are able to repro. describe formatted for table shows incorrect numRows data --- Key: HIVE-4621 URL: https://issues.apache.org/jira/browse/HIVE-4621 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.10.0 Reporter: Gwen Shapira Created table (tab delimited), used load data local to get data into the table. After creating the table, describe formatted showed numRow as 0. select count(*) from... showed 43205 rows. Running describe formatted after select count(*) still shows numRows as 0. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HIVE-4632) Use hadoop counter as a stat publisher
[ https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4632: --- Status: Open (was: Patch Available) This looks useful, [~navis] would you like to rebase this on latest trunk ? Use hadoop counter as a stat publisher -- Key: HIVE-4632 URL: https://issues.apache.org/jira/browse/HIVE-4632 Project: Hive Issue Type: Improvement Components: Statistics Affects Versions: 0.12.0 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4632.D11001.1.patch Currently stats are all long/aggregation type and can be safely acquired by hadoop counter without other db or hbase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4632) Use hadoop counter as a stat publisher
[ https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820852#comment-13820852 ] Ashutosh Chauhan commented on HIVE-4632: Also, I think counter should be default mechanism to gather stats instead of current defaults which never gather stats correctly. Use hadoop counter as a stat publisher -- Key: HIVE-4632 URL: https://issues.apache.org/jira/browse/HIVE-4632 Project: Hive Issue Type: Improvement Components: Statistics Affects Versions: 0.12.0 Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4632.D11001.1.patch Currently stats are all long/aggregation type and can be safely acquired by hadoop counter without other db or hbase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HIVE-3324) analyze command is not gathering num_rows present in the table.
[ https://issues.apache.org/jira/browse/HIVE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-3324. Resolution: Invalid You can't use derby for this, since a remote daemon is required to be running which is accessible to all task nodes. Derby runs in embedded mode, so that won't work. You need mysql or orther RDBMS running as daemon and accessible from all task nodes. analyze command is not gathering num_rows present in the table. - Key: HIVE-3324 URL: https://issues.apache.org/jira/browse/HIVE-3324 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.9.1, 0.10.0 Reporter: Rohith Sharma K S When analyze command is executed, collectableStats i.e num_rows and raw_data_size is always zero even though table contains data. bq. [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 5812, raw_data_size: 0] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3324) analyze command is not gathering num_rows present in the table.
[ https://issues.apache.org/jira/browse/HIVE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820856#comment-13820856 ] Ashutosh Chauhan commented on HIVE-3324: Also, note HIVE-4632 will simplify stats collection mechanism, so that above is no longer a requirement. analyze command is not gathering num_rows present in the table. - Key: HIVE-3324 URL: https://issues.apache.org/jira/browse/HIVE-3324 Project: Hive Issue Type: Bug Components: Statistics Affects Versions: 0.9.1, 0.10.0 Reporter: Rohith Sharma K S When analyze command is executed, collectableStats i.e num_rows and raw_data_size is always zero even though table contains data. bq. [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 5812, raw_data_size: 0] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-4621) describe formatted for table shows incorrect numRows data
[ https://issues.apache.org/jira/browse/HIVE-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820857#comment-13820857 ] Ashutosh Chauhan commented on HIVE-4621: Also, note HIVE-4632 will simplify such that lot of requirements for stats collection will no longer be present. describe formatted for table shows incorrect numRows data --- Key: HIVE-4621 URL: https://issues.apache.org/jira/browse/HIVE-4621 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.10.0 Reporter: Gwen Shapira Created table (tab delimited), used load data local to get data into the table. After creating the table, describe formatted showed numRow as 0. select count(*) from... showed 43205 rows. Running describe formatted after select count(*) still shows numRows as 0. -- This message was sent by Atlassian JIRA (v6.1#6144)
