[jira] [Commented] (HIVE-2250) DESCRIBE EXTENDED table_name shows inconsistent compression information.
[
https://issues.apache.org/jira/browse/HIVE-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507271#comment-13507271
]
Harsh J commented on HIVE-2250:
---
If we don't really make use of the IS_COMPRESSED attribute of a table, should
we just get rid of it (or at least not print it in the {{describe
extended/formatted}} output, which causes great confusion as it is always
certainly {{No}})?
DESCRIBE EXTENDED table_name shows inconsistent compression information.
--
Key: HIVE-2250
URL: https://issues.apache.org/jira/browse/HIVE-2250
Project: Hive
Issue Type: Bug
Components: CLI, Diagnosability
Affects Versions: 0.7.0
Environment: RHEL, Full Cloudera stack
Reporter: Travis Powell
Assignee: subramanian raghunathan
Priority: Critical
Attachments: HIVE-2250.patch
Commands executed in this order:
user@node # hive
hive SET hive.exec.compress.output=true;
hive SET io.seqfile.compression.type=BLOCK;
hive CREATE TABLE table_name ( [...] ) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '\t' STORED AS SEQUENCEFILE;
hive CREATE TABLE staging_table ( [...] ) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '\t';
hive LOAD DATA LOCAL INPATH 'file:///root/input/' OVERWRITE INTO TABLE
staging_table;
hive INSERT OVERWRITE TABLE table_name SELECT * FROM staging_table;
(Map reduce job to change to sequence file...)
hive DESCRIBE EXTENDED table_name;
Detailed Table Information Table(tableName:table_name,
dbName:benchmarking, owner:root, createTime:1309480053, lastAccessTime:0,
retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:session_key,
type:string, comment:null), FieldSchema(name:remote_address, type:string,
comment:null), FieldSchema(name:canister_lssn, type:string, comment:null),
FieldSchema(name:canister_session_id, type:bigint, comment:null),
FieldSchema(name:tltsid, type:string, comment:null), FieldSchema(name:tltuid,
type:string, comment:null), FieldSchema(name:tltvid, type:string,
comment:null), FieldSchema(name:canister_server, type:string, comment:null),
FieldSchema(name:session_timestamp, type:string, comment:null),
FieldSchema(name:session_duration, type:string, comment:null),
FieldSchema(name:hit_count, type:bigint, comment:null),
FieldSchema(name:http_user_agent, type:string, comment:null),
FieldSchema(name:extractid, type:bigint, comment:null),
FieldSchema(name:site_link, type:string, comment:null), FieldSchema(name:dt,
type:string, comment:null), FieldSchema(name:hour, type:int, comment:null)],
location:hdfs://hadoop2/user/hive/warehouse/benchmarking.db/table_name,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
parameters:{serialization.format= , field.delim=
*** SEE ABOVE: Compression is set to FALSE, even though contents of table is
compressed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21 #215
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/215/ -- [...truncated 5810 lines...] [ivy:resolve] (33kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-jobclient;0.23.3!hadoop-mapreduce-client-jobclient.jar (124ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/0.23.3/hadoop-mapreduce-client-jobclient-0.23.3-tests.jar ... [ivy:resolve] . (1264kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/0.23.3/hadoop-annotations-0.23.3.jar ... [ivy:resolve] ... (16kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-annotations;0.23.3!hadoop-annotations.jar (248ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar ... [ivy:resolve] ... (297kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] commons-httpclient#commons-httpclient;3.1!commons-httpclient.jar (41ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/commons-io/commons-io/2.1/commons-io-2.1.jar ... [ivy:resolve] .. (159kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] commons-io#commons-io;2.1!commons-io.jar (83ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar ... [ivy:resolve] .. (102kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] javax.servlet#servlet-api;2.5!servlet-api.jar (34ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/asm/asm/3.2/asm-3.2.jar ... [ivy:resolve] (42kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] asm#asm;3.2!asm.jar (31ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar ... [ivy:resolve] ... (447kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar ... [ivy:resolve] .. (144kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] com.sun.jersey#jersey-json;1.8!jersey-json.jar(bundle) (175ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar ... [ivy:resolve] (678kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar ... [ivy:resolve] . (398kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] tomcat#jasper-compiler;5.5.23!jasper-compiler.jar (97ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar ... [ivy:resolve] .. (75kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar ... [ivy:resolve] . (98kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar ... [ivy:resolve] (43kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] commons-logging#commons-logging-api;1.1!commons-logging-api.jar (86ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/log4j/log4j/1.2.15/log4j-1.2.15.jar ... [ivy:resolve] (382kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar ... [ivy:resolve] . (314kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] net.java.dev.jets3t#jets3t;0.6.1!jets3t.jar (199ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/commons-lang/commons-lang/2.5/commons-lang-2.5.jar ... [ivy:resolve] ... (272kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar ... [ivy:resolve] .. (24kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar ... [ivy:resolve] .. (9kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/aspectj/aspectjrt/1.6.5/aspectjrt-1.6.5.jar ... [ivy:resolve] .. (113kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar ... [ivy:resolve] . (257kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.5.3/avro-ipc-1.5.3.jar ... [ivy:resolve] .. (164kB)
Re: Username/password authentication support in HiveServer
Hi Yu Gao, The submission of HiveServer2 is being tracked by https://issues.apache.org/jira/browse/HIVE-2935 Keep your eyes out for when the status of that JIRA changes:-) Mark On Thu, Nov 29, 2012 at 11:18 PM, 郜瑜 crystalga...@gmail.com wrote: Hi all, I noticed that HiveServer2 will have username/password authentication. When will this HiveServer2 be available? As we know, the existing HiveServer does not have any authentication mechanism, so there's no security control on client connections and operations. It would be good to introduce authentication infrastructure to it as well, so I opened jira HIVE-3719https://issues.apache.org/jira/browse/HIVE-3719 and put the design there to achieve this. Regards, Yu Gao
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #215
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/215/ -- [...truncated 5844 lines...] [ivy:resolve] (42kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] asm#asm;3.2!asm.jar (140ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar ... [ivy:resolve] . (447kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar ... [ivy:resolve] (144kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar ... [ivy:resolve] (678kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] com.sun.jersey#jersey-server;1.8!jersey-server.jar(bundle) (106ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar ... [ivy:resolve] (398kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar ... [ivy:resolve] . (75kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] tomcat#jasper-runtime;5.5.23!jasper-runtime.jar (69ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar ... [ivy:resolve] (98kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] javax.servlet.jsp#jsp-api;2.1!jsp-api.jar (33ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar ... [ivy:resolve] ... (43kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/log4j/log4j/1.2.15/log4j-1.2.15.jar ... [ivy:resolve] .. (382kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] log4j#log4j;1.2.15!log4j.jar (83ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar ... [ivy:resolve] (314kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/commons-lang/commons-lang/2.5/commons-lang-2.5.jar ... [ivy:resolve] . (272kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] commons-lang#commons-lang;2.5!commons-lang.jar (46ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar ... [ivy:resolve] .. (24kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.slf4j#slf4j-api;1.6.1!slf4j-api.jar (30ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar ... [ivy:resolve] .. (9kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.slf4j#slf4j-log4j12;1.6.1!slf4j-log4j12.jar (68ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/aspectj/aspectjrt/1.6.5/aspectjrt-1.6.5.jar ... [ivy:resolve] . (113kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.aspectj#aspectjrt;1.6.5!aspectjrt.jar (173ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar ... [ivy:resolve] . (257kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.avro#avro;1.5.3!avro.jar (98ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/avro/avro-ipc/1.5.3/avro-ipc-1.5.3.jar ... [ivy:resolve] . (164kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.avro#avro-ipc;1.5.3!avro-ipc.jar (70ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar ... [ivy:resolve] . (439kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] com.google.protobuf#protobuf-java;2.4.0a!protobuf-java.jar (183ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/0.23.3/hadoop-auth-0.23.3.jar ... [ivy:resolve] ... (41kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-auth;0.23.3!hadoop-auth.jar (211ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/googlecode/json-simple/json-simple/1.1/json-simple-1.1.jar ... [ivy:resolve] ... (15kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] com.googlecode.json-simple#json-simple;1.1!json-simple.jar (82ms) [ivy:resolve] downloading http://repo1.maven.org/maven2/jdiff/jdiff/1.0.9/jdiff-1.0.9.jar ... [ivy:resolve] ... (126kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar ... [ivy:resolve] ... (32kB) [ivy:resolve] .. (0kB) [ivy:resolve] downloading http://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar ...
Re: Exception with datanucleus while running hive tests in eclipse
I ran into the same problem on the same MAC OS version. This seems to be a JVM command line issue. it exceeds its limits and it's platform independent. I know IntelliJ Idea handles this case. On Wed, May 16, 2012 at 5:40 PM, kulkarni.swar...@gmail.com kulkarni.swar...@gmail.com wrote: I installed datanucleus eclipse plugin as I realized that it is needed to run some of the hive tests in eclipse. While trying to run the enhancer tool, I keep getting this exception: Exception occurred executing command line. Cannot run program /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java ...Argument list is too long Has anyone else encountered this too? My machine is running OS X 10.7. Thanks, Swarnim
[jira] [Updated] (HIVE-3400) Add Retries to Hive MetaStore Connections
[ https://issues.apache.org/jira/browse/HIVE-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhushan Mandhani updated HIVE-3400: --- Labels: metastore (was: ) Status: Patch Available (was: Open) Add Retries to Hive MetaStore Connections - Key: HIVE-3400 URL: https://issues.apache.org/jira/browse/HIVE-3400 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Bhushan Mandhani Assignee: Bhushan Mandhani Priority: Minor Labels: metastore Attachments: HIVE-3400.1.patch.txt, HIVE-3400.2.patch.txt Currently, when using Thrift to access the MetaStore, if the Thrift host dies, there is no mechanism to reconnect to some other host even if the MetaStore URIs variable in the Conf contains multiple hosts. Hive should retry and reconnect rather than throwing a communication link error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Bad performance of Hive meta store for tables with large number of partitions
Does JDO support the features in EclipseLink in this article? http://java-persistence-performance.blogspot.com/2010/08/batch-fetching-optimizing-object-graph.html On Fri, Nov 30, 2012 at 11:36 AM, Benyi Wang bewang.t...@gmail.com wrote: We have some tables with 15K ~ 20K partitions. If I run a query scanning a lot of partitions, Hive could use more than 10 minutes to commit the mapred job. The problem is caused by ObjectStore.getPartitionsByNames when Hive semantic analyzer tries to prune partitions. This method sends a lot of queries to our MySQL database to retrieve ALL information about partitions. Because MPartition and MStroageDescriptor are converted into Partition and StorageDescriptor, every field will be accessed during conversion, in other words, even the fields has nothing to do with partition pruning, such as BucketCols. In our case, 10 queries for each partition will be sent to the database and each query may take 40ms. This is known ORM 1+N problem. But it is really bad user experience. Actually we assembly Partition objects manually, it would only need about 10 queries for a group of partitions (default size is 300). In our environment, it only needs 40 seconds for 30K partitions: 30K / 300 * 10 * 40. I tried to this way: 1. Fetch MPartition with fetch group and fetch_size_greedy, so one query can get MPartition's primary fields and MStorageDescriptor cached. 2. Get all descriptors into a list msds, run another query to get MStorageDescriptor with filter like msds.contains(this), all cached descriptors will be refreshed in one query instead of n queries. This works well for 1-1 relations, but not on 1-N relation like MPartition.values. I didn't find a way to populate those fields in just one query. Because JDO mapping doesn't work well in the conversion (MPartition - Partition), I'm wondering if it is worth doing like this: 1. Query each table in SQL directly PARTITIONS, SDS, etcs. 2. Assembly Partition objects This is a hack and the code will be really bad. But I didn't find JDO support FETCH JOIN or Batch fetch. Any thoughts?
[jira] [Commented] (HIVE-3400) Add Retries to Hive MetaStore Connections
[ https://issues.apache.org/jira/browse/HIVE-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507626#comment-13507626 ] Bhushan Mandhani commented on HIVE-3400: Ashutosh, I've uploaded and submitted the latest patch. Thanks. Add Retries to Hive MetaStore Connections - Key: HIVE-3400 URL: https://issues.apache.org/jira/browse/HIVE-3400 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Bhushan Mandhani Assignee: Bhushan Mandhani Priority: Minor Labels: metastore Attachments: HIVE-3400.1.patch.txt, HIVE-3400.2.patch.txt Currently, when using Thrift to access the MetaStore, if the Thrift host dies, there is no mechanism to reconnect to some other host even if the MetaStore URIs variable in the Conf contains multiple hosts. Hive should retry and reconnect rather than throwing a communication link error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-3384: -- Attachment: D6873.5.patch mbautin updated the revision [jira] [HIVE-3384] HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification. Reviewers: cdrome, weidongbian, JIRA, ashutoshc Rebasing on trunk changes. REVISION DETAIL https://reviews.facebook.net/D6873 AFFECTED FILES build-common.xml build.properties build.xml jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveCallableStatement.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java metastore/build.xml pdk/scripts/build-plugin.xml To: cdrome, weidongbian, JIRA, ashutoshc, mbautin HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, D6873.3.patch, D6873.4.patch, D6873.5.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3400) Add Retries to Hive MetaStore Connections
[ https://issues.apache.org/jira/browse/HIVE-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507722#comment-13507722 ] Ashutosh Chauhan commented on HIVE-3400: Bhushan, You have some changes in RetryingRawStore.java which looks like are unrelated to this patch. It seems like HIVE-3612 is tracking similar enhancements. Will you mind taking those changes out and put on in HIVE-3612? Add Retries to Hive MetaStore Connections - Key: HIVE-3400 URL: https://issues.apache.org/jira/browse/HIVE-3400 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Bhushan Mandhani Assignee: Bhushan Mandhani Priority: Minor Labels: metastore Attachments: HIVE-3400.1.patch.txt, HIVE-3400.2.patch.txt Currently, when using Thrift to access the MetaStore, if the Thrift host dies, there is no mechanism to reconnect to some other host even if the MetaStore URIs variable in the Conf contains multiple hosts. Hive should retry and reconnect rather than throwing a communication link error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3747) Provide hive operation name for hookContext
[
https://issues.apache.org/jira/browse/HIVE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shreepadma Venugopalan reassigned HIVE-3747:
Assignee: Shreepadma Venugopalan
Provide hive operation name for hookContext
---
Key: HIVE-3747
URL: https://issues.apache.org/jira/browse/HIVE-3747
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Sudhanshu Arora
Assignee: Shreepadma Venugopalan
The hookContext exposed through ExecuteWithHookContext, does not provide the
name of the Hive operation.
The following public API should be added in HookContext.
public String getOperationName() {
return SessionState.get().getHiveOperation().name();
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3747) Provide hive operation name for hookContext
[
https://issues.apache.org/jira/browse/HIVE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shreepadma Venugopalan updated HIVE-3747:
-
Status: Patch Available (was: Open)
Provide hive operation name for hookContext
---
Key: HIVE-3747
URL: https://issues.apache.org/jira/browse/HIVE-3747
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Sudhanshu Arora
Assignee: Shreepadma Venugopalan
Attachments: HIVE-3747.1.patch.txt
The hookContext exposed through ExecuteWithHookContext, does not provide the
name of the Hive operation.
The following public API should be added in HookContext.
public String getOperationName() {
return SessionState.get().getHiveOperation().name();
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3747) Provide hive operation name for hookContext
[
https://issues.apache.org/jira/browse/HIVE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shreepadma Venugopalan updated HIVE-3747:
-
Attachment: HIVE-3747.1.patch.txt
Provide hive operation name for hookContext
---
Key: HIVE-3747
URL: https://issues.apache.org/jira/browse/HIVE-3747
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Sudhanshu Arora
Assignee: Shreepadma Venugopalan
Attachments: HIVE-3747.1.patch.txt
The hookContext exposed through ExecuteWithHookContext, does not provide the
name of the Hive operation.
The following public API should be added in HookContext.
public String getOperationName() {
return SessionState.get().getHiveOperation().name();
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2380) Add Binary Datatype in Hive
[ https://issues.apache.org/jira/browse/HIVE-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507761#comment-13507761 ] Chuck Connell commented on HIVE-2380: - I am trying to use this feature (BINARY columns) and I believe I have the perfect use-case for it, but I am missing something. Here is the background... I have some files that each contain just one logical field, which is a binary object. (The files are Google Protobuf format.) I want to put these binary files into a larger file, where each protobuf is a logical record. Then I want to define a Hive table that stores each protobuf as one row, with the entire protobuf object in one BINARY column. Then I will use a custom UDF to select/query the binary object. This is about as simple as can be for putting binary data into Hive. But all of the test cases for this jira seem to draw the binary columns from another existing table and CAST them. I want to load the files from disk. What file format should I use to package the binary rows? What should the Hive table definition be? I cannot use TEXTFILE, since the binary may contain newlines. Many of my attempts have choked on the newlines. Thanks very much, Chuck Connell Nuance Burlington, MA Add Binary Datatype in Hive --- Key: HIVE-2380 URL: https://issues.apache.org/jira/browse/HIVE-2380 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2380_1.patch, hive-2380_2.patch, hive-2380_3.patch, hive-2380_4.patch, hive-2380.patch Add bytearray as a primitive data type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3612) RetryingHMSHandler should retry on MetaExceptions if the cause was a JDOException
[ https://issues.apache.org/jira/browse/HIVE-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean Xu updated HIVE-3612: -- Status: Open (was: Patch Available) This jira is not needed any more . RetryingHMSHandler should retry on MetaExceptions if the cause was a JDOException - Key: HIVE-3612 URL: https://issues.apache.org/jira/browse/HIVE-3612 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Jean Xu The RetryingHMSHandler would be much more effective if it retried on MetaExceptions wrapped around JDOExceptions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3400) Add Retries to Hive MetaStore Connections
[ https://issues.apache.org/jira/browse/HIVE-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507768#comment-13507768 ] Bhushan Mandhani commented on HIVE-3400: Ashutosh, we no longer need HIVE-3612. Jean is about to abandon that diff. I think we should keep these RetryingRawStore changes here since RetryingHMSHandler already catches JDOExceptions. But I can take it out if you prefer that. Add Retries to Hive MetaStore Connections - Key: HIVE-3400 URL: https://issues.apache.org/jira/browse/HIVE-3400 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Bhushan Mandhani Assignee: Bhushan Mandhani Priority: Minor Labels: metastore Attachments: HIVE-3400.1.patch.txt, HIVE-3400.2.patch.txt Currently, when using Thrift to access the MetaStore, if the Thrift host dies, there is no mechanism to reconnect to some other host even if the MetaStore URIs variable in the Conf contains multiple hosts. Hive should retry and reconnect rather than throwing a communication link error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HIVE-3073) Hive List Bucketing - DML support
[ https://issues.apache.org/jira/browse/HIVE-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-3073 started by Gang Tim Liu. Hive List Bucketing - DML support -- Key: HIVE-3073 URL: https://issues.apache.org/jira/browse/HIVE-3073 Project: Hive Issue Type: New Feature Components: SQL Affects Versions: 0.10.0 Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3073.patch.12, HIVE-3073.patch.13, HIVE-3073.patch.15, HIVE-3073.patch.18, HIVE-3073.patch.19, HIVE-3073.patch.21 If a hive table column has skewed keys, query performance on non-skewed key is always impacted. Hive List Bucketing feature will address it: https://cwiki.apache.org/Hive/listbucketing.html This jira issue will track DML change for the feature: 1. single skewed column 2. manual load data -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3073) Hive List Bucketing - DML support
[ https://issues.apache.org/jira/browse/HIVE-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3073: --- Attachment: HIVE-3073.patch.21 Hive List Bucketing - DML support -- Key: HIVE-3073 URL: https://issues.apache.org/jira/browse/HIVE-3073 Project: Hive Issue Type: New Feature Components: SQL Affects Versions: 0.10.0 Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3073.patch.12, HIVE-3073.patch.13, HIVE-3073.patch.15, HIVE-3073.patch.18, HIVE-3073.patch.19, HIVE-3073.patch.21 If a hive table column has skewed keys, query performance on non-skewed key is always impacted. Hive List Bucketing feature will address it: https://cwiki.apache.org/Hive/listbucketing.html This jira issue will track DML change for the feature: 1. single skewed column 2. manual load data -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3073) Hive List Bucketing - DML support
[ https://issues.apache.org/jira/browse/HIVE-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3073: --- Status: Patch Available (was: In Progress) patch is available. Hive List Bucketing - DML support -- Key: HIVE-3073 URL: https://issues.apache.org/jira/browse/HIVE-3073 Project: Hive Issue Type: New Feature Components: SQL Affects Versions: 0.10.0 Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3073.patch.12, HIVE-3073.patch.13, HIVE-3073.patch.15, HIVE-3073.patch.18, HIVE-3073.patch.19, HIVE-3073.patch.21 If a hive table column has skewed keys, query performance on non-skewed key is always impacted. Hive List Bucketing feature will address it: https://cwiki.apache.org/Hive/listbucketing.html This jira issue will track DML change for the feature: 1. single skewed column 2. manual load data -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1827 - Fixed
Changes for Build #1784 [kevinwilfong] HIVE-3675. NaN does not work correctly for round(n). (njain via kevinwilfong) [cws] HIVE-3651. bucketmapjoin?.q tests fail with hadoop 0.23 (Prasad Mujumdar via cws) Changes for Build #1785 [namit] HIVE-3613 Implement grouping_id function (Ian Gorbachev via namit) [namit] HIVE-3692 Update parallel test documentation (Ivan Gorbachev via namit) [namit] HIVE-3649 Hive List Bucketing - enhance DDL to specify list bucketing table (Gang Tim Liu via namit) Changes for Build #1786 [namit] HIVE-3696 Revert HIVE-3483 which causes performance regression (Gang Tim Liu via namit) Changes for Build #1787 [kevinwilfong] HIVE-3621. Make prompt in Hive CLI configurable. (Jingwei Lu via kevinwilfong) [kevinwilfong] HIVE-3695. TestParse breaks due to HIVE-3675. (njain via kevinwilfong) Changes for Build #1788 [kevinwilfong] HIVE-3557. Access to external URLs in hivetest.py. (Ivan Gorbachev via kevinwilfong) Changes for Build #1789 [hashutosh] HIVE-3662 : TestHiveServer: testScratchDirShouldClearWhileStartup is failing on Windows (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3659 : TestHiveHistory::testQueryloglocParentDirNotExist Test fails on Windows because of some resource leaks in ZK (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3663 Unable to display the MR Job file path on Windows in case of MR job failures. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1790 Changes for Build #1791 Changes for Build #1792 Changes for Build #1793 [hashutosh] HIVE-3704 : name of some metastore scripts are not per convention (Ashutosh Chauhan) Changes for Build #1794 [hashutosh] HIVE-3243 : ignore white space between entries of hive/hbase table mapping (Shengsheng Huang via Ashutosh Chauhan) [hashutosh] HIVE-3215 : JobDebugger should use RunningJob.getTrackingURL (Bhushan Mandhani via Ashutosh Chauhan) Changes for Build #1795 [cws] HIVE-3437. 0.23 compatibility: fix unit tests when building against 0.23 (Chris Drome via cws) [hashutosh] HIVE-3626 : RetryingHMSHandler should wrap JDOException inside MetaException (Bhushan Mandhani via Ashutosh Chauhan) [hashutosh] HIVE-3560 : Hive always prints a warning message when using remote metastore (Travis Crawford via Ashutosh Chauhan) Changes for Build #1796 Changes for Build #1797 [hashutosh] HIVE-3664 : Avoid to create a symlink for hive-contrib.jar file in dist\lib folder. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1798 [namit] HIVE-3706 getBoolVar in FileSinkOperator can be optimized (Kevin Wilfong via namit) [namit] HIVE-3707 Round map/reduce progress down when it is in the range [99.5, 100) (Kevin Wilfong via namit) [namit] HIVE-3471 Implement grouping sets in hive (Ivan Gorbachev via namit) Changes for Build #1799 [hashutosh] HIVE-3291 : fix fs resolvers (Ashish Singh via Ashutosh Chauhan) [hashutosh] HIVE-3680 : Include Table information in Hive's AddPartitionEvent. (Mithun Radhakrishnan via Ashutosh Chauhan) Changes for Build #1800 [hashutosh] HIVE-3520 : ivysettings.xml does not let you override .m2/repository (Raja Aluri via Ashutosh Chauhan) [hashutosh] HIVE-3435 : Get pdk pluginTest passed when triggered from both builtin tests and pdk tests on hadoop23 (Zhenxiao Luo via Ashutosh Chauhan) Changes for Build #1801 Changes for Build #1802 Changes for Build #1803 Changes for Build #1804 Changes for Build #1805 Changes for Build #1806 [hashutosh] Preparing for 0.11 development [kevinwilfong] HIVE-3647. map-side groupby wrongly due to HIVE-3432. (njain via kevinwilfong) Changes for Build #1807 [kevinwilfong] HIVE-3679. Unescape partition names returned by show partitions. (Wonho Kim via kevinwilfong) Changes for Build #1808 [kevinwilfong] Remove extra files accidentally committed with HIVE-3679. (kevinwilfong) Changes for Build #1809 Changes for Build #1810 Changes for Build #1811 [namit] HIVE-3735 PTest doesn't work due to hive snapshot version upgrade to 11 (Gang Tim Liu via namit) [namit] HIVE-3722 Create index fails on CLI using remote metastore (Kevin Wilfong via namit) Changes for Build #1812 [namit] HIVE-3703 Hive Query Explain Plan JSON not being created properly (Mayank Garg via namit) [namit] HIVE-3721 ALTER TABLE ADD PARTS should check for valid partition spec and throw a SemanticException if part spec is not valid (Pamela Vagata via namit) Changes for Build #1813 Changes for Build #1814 [hashutosh] HIVE-3698 : enhanceModel.notRequired is incorrectly determined (Chris Drome via Ashutosh Chauhan) Changes for Build #1815 Changes for Build #1816 Changes for Build #1817 [hashutosh] HIVE-3736 : hive unit test case build failure. (Ashish Singh via Ashutosh Chauhan) Changes for Build #1818 Changes for Build #1819 Changes for Build #1820 Changes for Build #1821 [hashutosh] HIVE-3749 : New test cases added by HIVE-3676 in insert1.q is not deterministic (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3648 :
[jira] [Commented] (HIVE-2380) Add Binary Datatype in Hive
[ https://issues.apache.org/jira/browse/HIVE-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507810#comment-13507810 ] Chuck Connell commented on HIVE-2380: - Let me state my earlier comment another way... If the BINARY column type works in narrow sense, but it is impossible to get binary data from a file into Hive, then this feature is essentially useless. So we need some explanation for how to use the existing DDL features to support BINARY columns, or a new file format that allows it. Add Binary Datatype in Hive --- Key: HIVE-2380 URL: https://issues.apache.org/jira/browse/HIVE-2380 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2380_1.patch, hive-2380_2.patch, hive-2380_3.patch, hive-2380_4.patch, hive-2380.patch Add bytearray as a primitive data type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3720) Expand and standardize authorization in Hive
[ https://issues.apache.org/jira/browse/HIVE-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507831#comment-13507831 ] Shreepadma Venugopalan commented on HIVE-3720: -- @Namit: The authorization model in this proposal mirrors that of MySQL as closely as possible. The proposal also documents wherever there is a deviation from MySQL's authorization model. Since Hive's data model is based on that of MySQL, it would make a lot of sense to base the authorization model on MySQL's as well. The proposed functionality is not necessarily a superset of the existing authorization functionality but subsumes some of the existing functionality. While the existing implementation supports authorization on some HiveQL operations, it doesn't secure all of the operations, provide a way to bootstrap the system etc. This proposal expands authorization to all HiveQL operations and direct metadata operations that can be performed by invoking the metastore Thrift API. As discussed earlier, since the proposed model standardizes the authorization model to mirror that of MySQL, it deviates from the existing model where ever the existing implementation deviates from the authorization model of MySQL or other RDBMSs. The proposed model is also more fine grained and supports hierarchical privileges much like an RDBMS. For instance, the proposed model supports CREATE, ALTER, DROP privileges on objects whereas the current model supports an ALTER_METADATA privilege that includes the privileges needed to perform CREATE, ALTER, DROP etc. Note that one of the goals is to propose an authorization model such that finer grained privileges can be added in as necessary later. Since the existing implementation is not complete, it unclear at this point what part of the functionality has been completely implemented. Perhaps we can mark the existing functionality in the wiki once we start implementing the proposed model. Thanks. Expand and standardize authorization in Hive Key: HIVE-3720 URL: https://issues.apache.org/jira/browse/HIVE-3720 Project: Hive Issue Type: Improvement Components: Authorization Affects Versions: 0.9.0 Reporter: Shreepadma Venugopalan Assignee: Shreepadma Venugopalan Attachments: Hive_Authorization_Functionality.pdf The existing implementation of authorization in Hive is not complete. Additionally the existing implementation has security holes. This JIRA is an umbrella JIRA for a) extending authorization to all SQL operations and direct metadata operations, and b) standardizing the authorization model and its semantics to mirror that of MySQL as closely as possible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3612) RetryingHMSHandler should retry on MetaExceptions if the cause was a JDOException
[ https://issues.apache.org/jira/browse/HIVE-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-3612. Resolution: Invalid RetryingHMSHandler should retry on MetaExceptions if the cause was a JDOException - Key: HIVE-3612 URL: https://issues.apache.org/jira/browse/HIVE-3612 Project: Hive Issue Type: Improvement Components: Metastore Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Jean Xu The RetryingHMSHandler would be much more effective if it retried on MetaExceptions wrapped around JDOExceptions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3724) Metastore tests use hardcoded ports
[ https://issues.apache.org/jira/browse/HIVE-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-3724: --- Resolution: Fixed Fix Version/s: 0.11 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Kevin! Metastore tests use hardcoded ports --- Key: HIVE-3724 URL: https://issues.apache.org/jira/browse/HIVE-3724 Project: Hive Issue Type: Bug Components: Tests Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Priority: Minor Fix For: 0.11 Attachments: HIVE-3724.1.patch.txt, HIVE-3724.2.patch.txt Several of the metastore tests use hardcoded ports for remote metastore Thrift servers. This is causing transient failures in Jenkins, e.g. https://builds.apache.org/job/Hive-trunk-h0.21/1804/ A few tests already dynamically determine free ports, and this logic can be shared. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3073) Hive List Bucketing - DML support
[ https://issues.apache.org/jira/browse/HIVE-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-3073: --- Attachment: HIVE-3073.patch.22 Hive List Bucketing - DML support -- Key: HIVE-3073 URL: https://issues.apache.org/jira/browse/HIVE-3073 Project: Hive Issue Type: New Feature Components: SQL Affects Versions: 0.10.0 Reporter: Gang Tim Liu Assignee: Gang Tim Liu Attachments: HIVE-3073.patch.12, HIVE-3073.patch.13, HIVE-3073.patch.15, HIVE-3073.patch.18, HIVE-3073.patch.19, HIVE-3073.patch.21, HIVE-3073.patch.22 If a hive table column has skewed keys, query performance on non-skewed key is always impacted. Hive List Bucketing feature will address it: https://cwiki.apache.org/Hive/listbucketing.html This jira issue will track DML change for the feature: 1. single skewed column 2. manual load data -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
