[jira] [Created] (HIVE-26087) when set hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec , where exec select 1, then nullpointsexception
lkl created HIVE-26087: -- Summary: when set hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec , where exec select 1, then nullpointsexception Key: HIVE-26087 URL: https://issues.apache.org/jira/browse/HIVE-26087 Project: Hive Issue Type: Bug Components: Hooks Affects Versions: 3.1.2 Reporter: lkl Fix For: All Versions hive> set hive.exec.pre.hooks = org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec; hive> select 1; FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.InvalidTableException(Table not found _dummy_table) org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found _dummy_table at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1133) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1103) at org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec.run(UpdateInputAccessTimeHook.java:64) at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296) at org.apache.hadoop.hive.ql.HookRunner.runPreHooks(HookRunner.java:273) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2276) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-26088) a left join b left join c when b is null ，then b join c is null , in this case when set hive.auto.convert.join=true ; map join is not used ,because nullpointexecption
lkl created HIVE-26088: -- Summary: a left join b left join c when b is null ，then b join c is null , in this case when set hive.auto.convert.join=true ; map join is not used ,because nullpointexecption Key: HIVE-26088 URL: https://issues.apache.org/jira/browse/HIVE-26088 Project: Hive Issue Type: Bug Affects Versions: 3.1.2 Reporter: lkl create database pr; use pr; create table t1(id int,name string) PARTITIONED BY (cls string) stored as orc; create table t2(id int,name string) PARTITIONED BY (cls string) stored as orc; create table t3(id int,name string) PARTITIONED BY (cls string) stored as orc; insert into table t1 partition(cls=1) values(1,"zzz"); insert into table t3 partition(cls=1) values(1,"zzz"); sql: select y1.id,y1.name,nvl(y2.id,"其他") as y2id ,nvl(y2.name,"其他") as y2name from (select t1.id,t1.name from t1) y1 left join ( select a.id,a.name ,b.id as bid,b.name as bname from (select t2.id,t2.name from t2) a left join (select t3.id,t3.name from t3) b on a.id = b.id )y2 on y1.id=y2.id; log msg： No Stats for mapjoin@t1, Columns: name, id No Stats for mapjoin@t3, Columns: id Query ID = hadoop_20220330164204_2a822fd5-7630-43f9-929a-b95086df04bf Total jobs = 3 2022-03-30 16:42:11 Starting to launch local task to process map join; maximum memory = 3817865216 2022-03-30 16:42:12 Uploaded 1 File to: file:/data/emr/hive/tmp/23201dda-9c15-4d6e-9904-12ae28d4dccb/hive_2022-03-30_16-42-04_837_944561186926371473-2/-local-10007/HashTable-Stage-6/MapJoin-mapfile30--.hashtable (278 bytes) Execution completed successfully MapredLocal task succeeded Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1646126523394_0063, Tracking URL = http://172.21.64.46:5004/proxy/application_1646126523394_0063/ Kill Command = /usr/local/service/hadoop/bin/mapred job -kill job_1646126523394_0063 Hadoop job information for Stage-6: number of mappers: 0; number of reducers: 0 2022-03-30 16:42:18,781 Stage-6 map = 0%, reduce = 0% Ended Job = job_1646126523394_0063 Stage-7 is filtered out by condition resolver. Stage-1 is selected by condition resolver. Launching Job 2 out of 3 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapreduce.job.reduces= Starting Job = job_1646126523394_0064, Tracking URL = http://172.21.64.46:5004/proxy/application_1646126523394_0064/ Kill Command = /usr/local/service/hadoop/bin/mapred job -kill job_1646126523394_0064 Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1 2022-03-30 16:42:26,673 Stage-1 map = 0%, reduce = 0% 2022-03-30 16:42:30,768 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.39 sec 2022-03-30 16:42:31,792 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.92 sec 2022-03-30 16:42:36,886 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.83 sec MapReduce Total cumulative CPU time: 4 seconds 830 msec Ended Job = job_1646126523394_0064 MapReduce Jobs Launched: Stage-Stage-6: HDFS Read: 0 HDFS Write: 0 SUCCESS Stage-Stage-1: Map: 2 Reduce: 1 Cumulative CPU: 4.83 sec HDFS Read: 15362 HDFS Write: 131 SUCCESS Total MapReduce CPU Time Spent: 4 seconds 830 msec OK 1 zzz 其他 其他 Time taken: 33.102 seconds, Fetched: 1 row(s) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-26090) hive.mv.files.thread and hive.load.dynamic.partitions.thread in Hiveconf.java default value is 15,but in Hive.java is 25
lkl created HIVE-26090:
--
Summary: hive.mv.files.thread and
hive.load.dynamic.partitions.thread in Hiveconf.java default value is 15,but in
Hive.java is 25
Key: HIVE-26090
URL: https://issues.apache.org/jira/browse/HIVE-26090
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 3.1.2, 3.0.0, 2.3.7
Reporter: lkl
hive.mv.files.thread and hive.load.dynamic.partitions.thread in Hiveconf.java
default value is 15,but in Hive.java is 25
Hiveconf.java
HIVE_MOVE_FILES_THREAD_COUNT("hive.mv.files.thread", 15, new SizeValidator(0L,
true, 1024L, true), "Number of threads"
+ " used to move files in move task. Set it to 0 to disable multi-threaded file
moves. This parameter is also used by"
+ " MSCK to check tables."),
HIVE_LOAD_DYNAMIC_PARTITIONS_THREAD_COUNT("hive.load.dynamic.partitions.thread",
15,
new SizeValidator(1L, true, 1024L, true),
Hive.java
final ExecutorService pool =
conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 25) > 0 ?
Executors.newFixedThreadPool(conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname,
25),
new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Move-Thread-%d").build())
: null;
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26091) Support DecimalFilterPredicateLeafBuilder for parquet
Rajesh Balamohan created HIVE-26091:
---
Summary: Support DecimalFilterPredicateLeafBuilder for parquet
Key: HIVE-26091
URL: https://issues.apache.org/jira/browse/HIVE-26091
Project: Hive
Issue Type: Bug
Reporter: Rajesh Balamohan
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/LeafFilterFactory.java#L41
It will nice to have DecimalFilterPredicateLeafBuilder. This will help in
supporting SARG pushdowns.
{noformat}
2022-03-30 08:59:50,040 [ERROR] [TezChild]
|read.ParquetFilterPredicateConverter|: fail to build predicate filter leaf
with errorsorg.apache.hadoop.hive.ql.metadata.HiveException: Conversion to
Parquet FilterPredicate not supported for DECIMAL
org.apache.hadoop.hive.ql.metadata.HiveException: Conversion to Parquet
FilterPredicate not supported for DECIMAL
at
org.apache.hadoop.hive.ql.io.parquet.LeafFilterFactory.getLeafFilterBuilderByType(LeafFilterFactory.java:223)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.buildFilterPredicateFromPredicateLeaf(ParquetFilterPredicateConverter.java:130)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:111)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:97)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:71)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:88)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.toFilterPredicate(ParquetFilterPredicateConverter.java:57)
at
org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.setFilter(ParquetRecordReaderBase.java:184)
at
org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.getSplit(ParquetRecordReaderBase.java:124)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:158)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
at
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:429)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:282)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:265)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(
Re: [VOTE] Apache Hive 4.0.0-alpha-1 Release Candidate 2
Thanks for everyone who checked and voted! The vote was successful with 3 +1s. I will start the release process based on this. > On 2022. Mar 29., at 16:22, Naveen Gangam > wrote: > > I have run some additional testing with ACID tables from the install. No > issues found. > +1 (binding) > > On Mon, Mar 28, 2022 at 5:27 PM Naveen Gangam wrote: > >> >> - Downloaded the candidate bits and confirmed checksums. >> - Built locally and ran Metastore checkin tests and few other qtests. >> - Installed from binaries and ran some DDL/DMLs locally from beeline >> >> +1 (non-binding) >> >> On Mon, Mar 28, 2022 at 9:32 AM Denys Kuzmenko >> wrote: >> >>> +1 >>> >>> * Downloaded the release candidate artifacts and verified the signatures >>> and checksums; >>> * Built from the release sources; >>> * Initialized schema using schematool and launched hiveserver2 locally; >>> * Used beeline to connect: >>>- Created a few tables and loaded data into them. >>>- Executed some simple queries. >>> >>> Best regards, >>> Denys >>> >>> On Sat, Mar 26, 2022 at 11:24 PM Stamatis Zampetakis >>> wrote: >>> Ubuntu 20.04.4 LTS, jdk1.8.0_261, Apache Maven 3.6.3 * Checked signatures and checksums OK * Checked diff between repo and release sources (diff -qr hive apache-hive-4.0.0-alpha-1-src) OK * Built from git tag (mvn clean install -DskipTests -Pitests) OK * Built from release sources (mvn clean install -DskipTests -Pitests) >>> OK * Run smoke tests on pseudo cluster using hive-dev-box OK All of the issues that were found in the previous RC are either >>> resolved or tracked under respective JIRAs to be solved for the next release. Smoke tests included: * Derby metastore initialization * simple CREATE TABLE statements; * basic INSERT INTO VALUES statements; * basic SELECT * FROM WHERE variations; * EXPLAIN statement variations; * ANALYZE TABLE variations; +1 (non-binding) Best, Stamatis On Thu, Mar 24, 2022 at 12:01 PM Peter Vary >>> wrote: > Hi Team, > > Apache Hive 4.0.0-alpha-1 Release Candidate 2 is available here: > https://people.apache.org/~pvary/apache-hive-4.0.0-alpha-1-rc2/ < > https://people.apache.org/~pvary/apache-hive-4.0.0-alpha-1-rc2/> > > The checksums are these: > - 1e450197dbf847696b05042eb68b78b968064f1f1b369a7fb0b77a6329a27809 > apache-hive-4.0.0-alpha-1-bin.tar.gz > - a21a609ec2e30f8cc656242c545bb3a04de21c2a1eee90808648e3aa4bf3d04e > apache-hive-4.0.0-alpha-1-src.tar.gz > > Maven artifacts are available here: > >>> https://repository.apache.org/content/repositories/orgapachehive-1113/ < > >>> https://repository.apache.org/content/repositories/orgapachehive-1113/> > > The tag 4.0.0-alpha-1-rc1 has been applied to the source for this >>> release > in github, you can see it at > https://github.com/apache/hive/tree/release-4.0.0-alpha-1-rc1 < > https://github.com/apache/hive/tree/release-4.0.0-alpha-1-rc1> > > The git commit hash is: > > >>> https://github.com/apache/hive/commit/357d4906f5c806d585fd84db57cf296e12e6049b > < > >>> https://github.com/apache/hive/commit/357d4906f5c806d585fd84db57cf296e12e6049b >> > > Voting will conclude in 72 hours. > > All interested parties: Please test. > Hive PMC Members: Please test and vote. > > Thanks. >>> >>
[jira] [Created] (HIVE-26092) Fix javadoc errors for the 4.0.0 release
Peter Vary created HIVE-26092: - Summary: Fix javadoc errors for the 4.0.0 release Key: HIVE-26092 URL: https://issues.apache.org/jira/browse/HIVE-26092 Project: Hive Issue Type: Task Reporter: Peter Vary Currently there are plenty of errors in the javadoc. We should fix those before a final release -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-26093) Duplicate org.apache.hadoop.hive.metastore.annotation package-info.java
Peter Vary created HIVE-26093:
-
Summary: Duplicate org.apache.hadoop.hive.metastore.annotation
package-info.java
Key: HIVE-26093
URL: https://issues.apache.org/jira/browse/HIVE-26093
Project: Hive
Issue Type: Task
Reporter: Peter Vary
Currently we define
org.apache.hadoop.hive.metastore.annotation.MetastoreVersionAnnotation in 2
places:
-
./standalone-metastore/metastore-common/src/gen/version/org/apache/hadoop/hive/metastore/annotation/package-info.java
-
./standalone-metastore/metastore-server/src/gen/version/org/apache/hadoop/hive/metastore/annotation/package-info.java
This causes javadoc generation to fail with:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:aggregate (default-cli) on
project hive: An error has occurred in Javadoc report generation:
[ERROR] Exit code: 1 -
/Users/pvary/dev/upstream/hive/standalone-metastore/metastore-server/src/gen/version/org/apache/hadoop/hive/metastore/annotation/package-info.java:8:
warning: a package-info.java file has already been seen for package
org.apache.hadoop.hive.metastore.annotation
[ERROR] package org.apache.hadoop.hive.metastore.annotation;
[ERROR] ^
[ERROR] javadoc: warning - Multiple sources of package comments found for
package "org.apache.hive.streaming"
[ERROR]
/Users/pvary/dev/upstream/hive/ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java:556:
error: type MapSerializer does not take parameters
[ERROR] com.esotericsoftware.kryo.serializers.MapSerializer {
[ERROR] ^
[ERROR]
/Users/pvary/dev/upstream/hive/standalone-metastore/metastore-server/src/gen/version/org/apache/hadoop/hive/metastore/annotation/package-info.java:4:
error: package org.apache.hadoop.hive.metastore.annotation has already been
annotated
[ERROR] @MetastoreVersionAnnotation(version="4.0.0-alpha-1",
shortVersion="4.0.0-alpha-1",
[ERROR] ^
[ERROR] java.lang.AssertionError
[ERROR] at com.sun.tools.javac.util.Assert.error(Assert.java:126)
[ERROR] at com.sun.tools.javac.util.Assert.check(Assert.java:45)
[ERROR] at
com.sun.tools.javac.code.SymbolMetadata.setDeclarationAttributesWithCompletion(SymbolMetadata.java:177)
[ERROR] at
com.sun.tools.javac.code.Symbol.setDeclarationAttributesWithCompletion(Symbol.java:215)
[ERROR] at
com.sun.tools.javac.comp.MemberEnter.actualEnterAnnotations(MemberEnter.java:952)
[ERROR] at
com.sun.tools.javac.comp.MemberEnter.access$600(MemberEnter.java:64)
[ERROR] at
com.sun.tools.javac.comp.MemberEnter$5.run(MemberEnter.java:876)
[ERROR] at com.sun.tools.javac.comp.Annotate.flush(Annotate.java:143)
[ERROR] at
com.sun.tools.javac.comp.Annotate.enterDone(Annotate.java:129)
[ERROR] at com.sun.tools.javac.comp.Enter.complete(Enter.java:512)
[ERROR] at com.sun.tools.javac.comp.Enter.main(Enter.java:471)
[ERROR] at com.sun.tools.javadoc.JavadocEnter.main(JavadocEnter.java:78)
[ERROR] at
com.sun.tools.javadoc.JavadocTool.getRootDocImpl(JavadocTool.java:186)
[ERROR] at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:346)
[ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:219)
[ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:205)
[ERROR] at com.sun.tools.javadoc.Main.execute(Main.java:64)
[ERROR] at com.sun.tools.javadoc.Main.main(Main.java:54)
[ERROR] javadoc: error - fatal error
[ERROR]
[ERROR] Command line was:
/usr/local/Cellar/openjdk@8/1.8.0+302/libexec/openjdk.jdk/Contents/Home/jre/../bin/javadoc
@options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in
'/Users/pvary/dev/upstream/hive/target/site/apidocs' dir.
{code}
We should fix this by removing one of the above
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Call for Presentations now open, ApacheCon North America 2022
[You are receiving this because you are subscribed to one or more user or dev mailing list of an Apache Software Foundation project.] ApacheCon draws participants at all levels to explore “Tomorrow’s Technology Today” across 300+ Apache projects and their diverse communities. ApacheCon showcases the latest developments in ubiquitous Apache projects and emerging innovations through hands-on sessions, keynotes, real-world case studies, trainings, hackathons, community events, and more. The Apache Software Foundation will be holding ApacheCon North America 2022 at the New Orleans Sheration, October 3rd through 6th, 2022. The Call for Presentations is now open, and will close at 00:01 UTC on May 23rd, 2022. We are accepting presentation proposals for any topic that is related to the Apache mission of producing free software for the public good. This includes, but is not limited to: Community Big Data Search IoT Cloud Fintech Pulsar Tomcat You can submit your session proposals starting today at https://cfp.apachecon.com/ Rich Bowen, on behalf of the ApacheCon Planners apachecon.com @apachecon
Re: [VOTE] Apache Hive 3.1.3 Release Candidate 2
Still seeking votes. Voting ends tomorrow. Any help would be appreciated. Thank you Naveen On Tue, Mar 29, 2022 at 5:51 AM Peter Vary wrote: > Downloaded the 3.1.3 artifacts, and checked the signatures. They are OK. > Used the binary to run some basic tests, and it seems OK. > > +1 (binding) > > > On 2022. Mar 28., at 23:19, Naveen Gangam > wrote: > > > > Apache Hive 3.1.3 Release Candidate 2 is available here: > > https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-2 > > > > The checksums are these: > > > > > > - 55c58e0111bd32de3d02f5f25d9eb054ba65ab02aaf669637760eaf56ef1fbb1 > > apache-hive-3.1.3-bin.tar.gz > > > > > > - 22862e6bf76a4783a3d8d298634728cc9d6561563af2413a687fe63e35bcc527 > > apache-hive-3.1.3-src.tar.gz > > > > > > Maven artifacts are available here: > > https://repository.apache.org/content/repositories/orgapachehive-1114 > > > > The tag release-3.1.3-rc2 has been applied to the source for this > > release in github, you can see it > > athttps://github.com/apache/hive/tree/release-3.1.3-rc2 > > > > The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975 > > < > https://github.com/apache/hive/commit/4df4d75bf1e16fe0af75aad0b4179c34c07fc975 > > > > Voting will conclude in 72 hours. > > > > Hive PMC Members: Please test and vote. > > > > Thanks. > > > > Naveen > >
[jira] [Created] (HIVE-26094) Compaction Worker thread not freeing the lock when lock status returned as WAITING
Steve Carlin created HIVE-26094:
---
Summary: Compaction Worker thread not freeing the lock when lock
status returned as WAITING
Key: HIVE-26094
URL: https://issues.apache.org/jira/browse/HIVE-26094
Project: Hive
Issue Type: New Feature
Components: Locking
Reporter: Steve Carlin
The following error occurs in the HS2 logs:
{code:java}
(Worker.java:findNextCompactionAndExecute(538)) - Caught an exception in the
main loop of compactor worker
hiveserver2-0.hiveserver2-service.compute-1648522810-vk7c.svc.cluster.local-62
org.apache.thrift.TException: Unable to acquire lock(s) on
{boa_small_feng.stocks_boa_2019.year=2013}, status {WAITING}, reason {null}
{code}
This happens when the compactor thread tries to grab a lock on a table that
already has a lock on it.
On the HMS side, the lock is saved in the database and queues up the lock.
This queued up lock will block other tables from grabbing the lock even after
the original lock is removed.
When the Worker object on the client side sees this status, it should either
unlock immediately if it chooses not to run compaction or keep rechecking the
lock. Currently it does neither.
The effect of this is that the lock stays on the table until it times out. This
prevents other threads from grabbing the lock.
The code should probably change to remove the lock immediately.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26095) Add queryid in QueryLifeTimeHookContext
Stamatis Zampetakis created HIVE-26095:
--
Summary: Add queryid in QueryLifeTimeHookContext
Key: HIVE-26095
URL: https://issues.apache.org/jira/browse/HIVE-26095
Project: Hive
Issue Type: New Feature
Components: Hooks
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
Fix For: 4.0.0-alpha-2
A
[QueryLifeTimeHook|https://github.com/apache/hive/blob/6c0b86ef0cfc67c5acb3468408e1d46fa6ef8024/ql/src/java/org/apache/hadoop/hive/ql/hooks/QueryLifeTimeHook.java]
is executed various times in the life-cycle of a query but it is not always
possible to obtain the id of the query. The query id is inside the
{{HookContext}} but the latter is not always available notably during
compilation.
The query id is useful for many purposes as it is the only way to uniquely
identify the query/command that is currently running. It is also the only way
to match together events appearing in before and after methods.
The goal of this jira is to add the query id in
[QueryLifeTimeHookContext|https://github.com/apache/hive/blob/6c0b86ef0cfc67c5acb3468408e1d46fa6ef8024/ql/src/java/org/apache/hadoop/hive/ql/hooks/QueryLifeTimeHookContext.java]
and make it available during all life-cycle events.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[ANNOUNCE] Apache Hive 4.0.0-alpha-1 Released
The Apache Hive team is proud to announce the release of Apache Hive version 4.0.0-alpha-1 The Apache Hive (TM) data warehouse software facilitates querying and managing large datasets residing in distributed storage. Built on top of Apache Hadoop (TM), it provides, among others: * Tools to enable easy data extract/transform/load (ETL) * A mechanism to impose structure on a variety of data formats * Access to files stored either directly in Apache HDFS (TM) or in other data storage systems such as Apache HBase (TM) * Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark frameworks. For Hive release details and downloads, please visit: https://hive.apache.org/downloads.html Hive 4.0.0-alpha-1 Release Notes are available here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12351399&styleName=Html&projectId=12310843 We would like to thank the many contributors who made this release possible. Regards, The Apache Hive Team
Re: Start releasing the master branch
Thanks to everyone who helped to create the Hive 4.0.0-alpha-1 release! I really hope this helps our users to try out our previously unreleased new features. As a last step of the release process, I will update the versions for the next release. I would like to ask your opinion about the next version. Which version should we use for the development: - 4.0.0-SNAPSHOT - 4.0.0-alpha-2-SNAPSHOT Thanks, Peter On Mon, 21 Mar 2022 at 15:59, Peter Vary wrote: > Hi Team, > > If everyone agrees, tomorrow I would like to start the release process > for 4.0.0-alpha-1. > > Is there any outstanding blocker jira that you know of? > > Thanks, > Peter > > > > On 2022. Mar 9., at 17:01, Stamatis Zampetakis > wrote: > > > > I just logged HIVE-26022 [1] which seems to be another potential blocker > > for 4.0.0-alpha-1. > > > > Best, > > Stamatis > > > > [1] https://issues.apache.org/jira/browse/HIVE-26022 > > > > On Thu, Mar 3, 2022 at 3:54 PM Peter Vary wrote: > > > >> Hi Team, > >> > >> Here is our status: > >> We collected the blocker tickets and marked them with fixVersion > >> 4.0.0-alpha-1: > >> > >> > https://issues.apache.org/jira/issues/?filter=-1&jql=project%20%3D%20HIVE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%204.0.0-alpha-1 > >> < > https://issues.apache.org/jira/issues/?filter=-1&jql=project%20=%20HIVE%20AND%20resolution%20=%20Unresolved%20AND%20fixVersion%20=%204.0.0-alpha-1 > > > >> > >> - HIVE-26002 - Create db scripts for 4.0.0-alpha-1 > >> - HIVE-25994 - Analyze table runs into ClassNotFoundException-s in > >> case binary distribution is used > >> - HIVE-25935 - Cleanup IMetaStoreClient#getPartitionsByNames APIs > >> > >> Please create a jira and mark it with fixVersion 4.0.0-alpha-1, if you > >> happen to know of any other blockers. > >> > >> We plan to fix these jiras, and then release the following artifacts > >> together: > >> > >> - Storage API - 4.0.0-alpha-1 > >> - Standalone Metastore - 4.0.0-alpha-1 > >> - Hive - 4.0.0-alpha-1 > >> > >> > >> Thanks, > >> Peter > >> > >> > >> On 2022. Mar 2., at 11:50, Peter Vary wrote: > >> > >> Will continue this discussion on the #hive ASF slack. If you are > >> interested, please join. > >> We will do updates here time-to-time, so the ones who are not using > slack > >> can participate that way. > >> > >> On 2022. Mar 2., at 11:11, Peter Vary wrote: > >> > >> Good idea Zoltan, joined the channel. > >> I would like to scope reasonably small, so I agree with focusing on > >> 4.0.0-alpha-1 > >> > >> On 2022. Mar 2., at 11:01, Zoltan Haindrich wrote: > >> > >> Hey, > >> > >> regarding 4.0.0 / 4.0.0-alpha-1 target/fix versions in the jira: > >> * I think we should change all already resolved tickets with fix version > >> 4.0.0 to have fix version 4.0.0-alpha-1 > >> ** this could be postponed until we are actually releasing the thing as > I > >> think everyone committing to the master is entering 4.0.0 as fix version > >> without much aftertought...this could probably change after we get the > >> first release out. > >> * regarding the the existing tickets with fix version/target version > 4.0.0 > >> - I think that would be a bit too much (>200 tickets) > >> ** some numbers: > >> *** 239 tickets open now > >> *** 224 was not updated in the last 90 days > >> *** 216 was not changed in the last 180 days > >> *** 178 was not updated in the last 360 days > >> ** as a matter of fact I think many of these tickets shouldn't even > have a > >> target or fix version - and most of them should be unassigned...I don't > >> want to get lost in this right now...I think for now we should keep the > >> scope small and only care with 4.0.0-alpha-1 tickets > >> > >> https://issues.apache.org/jira/issues/? > >> > jql=project%20%3D%20hive%20and%20resolutiondate%20%20is%20empty%20and%20(fixVersion%20%20in%20(%274.0.0%27)%20or%20cf%5B12310320%5D%20%20in%20(%274.0.0%27)) > >> > >> I think for faster communication regarding these things we could also > >> utilize the #hive channel on the ASF slack - what do you guys think? > >> > >> cheers, > >> Zoltan > >> > >> On 3/2/22 9:51 AM, Stamatis Zampetakis wrote: > >> > >> Agree with Peter, creating JIRAs is the way to go. > >> Putting the appropriate priority (e.g., BLOCKER) and version (4.0.0 or > >> 4.0.0-alpha-1) when creating the JIRA should be enough to keep us on > track. > >> I am mentioning both 4.0.0 and 4.0.0-alpha-1 because eventually I think > we > >> are gonna move everything with target 4.0.0 to 4.0.0-alpha-1. > >> On Wed, Mar 2, 2022 at 9:37 AM Peter Vary > >> wrote: > >> > >> Hi Team, > >> > >> Could we create tickets for the issues? > >> I think it would be good to collect the issues/potential blockers in the > >> jira instead of having a complicated mail thread. > >> > >> If we set the target version to 4.0.0-alpha-1, then we can easily use > the > >> following filter to see the status of the tasks: > >> > >> > >> > https://issues.apache.org/jira/issues/?jql=project%3D%22HIVE%22%2
Re: Start releasing the master branch
Thanks for pushing this forward Peter! Being the RM for this huge release was not an easy task. Let's now aim for smaller and much more frequent releases. Personally, I would prefer to keep the alpha-X suffix for a while so I would opt for 4.0.0-alpha-2-SNAPSHOT for the next iteration. Best, Stamatis On Wed, Mar 30, 2022, 8:02 PM Peter Vary wrote: > Thanks to everyone who helped to create the Hive 4.0.0-alpha-1 release! > I really hope this helps our users to try out our previously unreleased new > features. > > As a last step of the release process, I will update the versions for the > next release. > I would like to ask your opinion about the next version. > > Which version should we use for the development: > - 4.0.0-SNAPSHOT > - 4.0.0-alpha-2-SNAPSHOT > > Thanks, > Peter > > On Mon, 21 Mar 2022 at 15:59, Peter Vary wrote: > > > Hi Team, > > > > If everyone agrees, tomorrow I would like to start the release process > > for 4.0.0-alpha-1. > > > > Is there any outstanding blocker jira that you know of? > > > > Thanks, > > Peter > > > > > > > On 2022. Mar 9., at 17:01, Stamatis Zampetakis > > wrote: > > > > > > I just logged HIVE-26022 [1] which seems to be another potential > blocker > > > for 4.0.0-alpha-1. > > > > > > Best, > > > Stamatis > > > > > > [1] https://issues.apache.org/jira/browse/HIVE-26022 > > > > > > On Thu, Mar 3, 2022 at 3:54 PM Peter Vary wrote: > > > > > >> Hi Team, > > >> > > >> Here is our status: > > >> We collected the blocker tickets and marked them with fixVersion > > >> 4.0.0-alpha-1: > > >> > > >> > > > https://issues.apache.org/jira/issues/?filter=-1&jql=project%20%3D%20HIVE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%204.0.0-alpha-1 > > >> < > > > https://issues.apache.org/jira/issues/?filter=-1&jql=project%20=%20HIVE%20AND%20resolution%20=%20Unresolved%20AND%20fixVersion%20=%204.0.0-alpha-1 > > > > > >> > > >> - HIVE-26002 - Create db scripts for 4.0.0-alpha-1 > > >> - HIVE-25994 - Analyze table runs into ClassNotFoundException-s in > > >> case binary distribution is used > > >> - HIVE-25935 - Cleanup IMetaStoreClient#getPartitionsByNames APIs > > >> > > >> Please create a jira and mark it with fixVersion 4.0.0-alpha-1, if you > > >> happen to know of any other blockers. > > >> > > >> We plan to fix these jiras, and then release the following artifacts > > >> together: > > >> > > >> - Storage API - 4.0.0-alpha-1 > > >> - Standalone Metastore - 4.0.0-alpha-1 > > >> - Hive - 4.0.0-alpha-1 > > >> > > >> > > >> Thanks, > > >> Peter > > >> > > >> > > >> On 2022. Mar 2., at 11:50, Peter Vary wrote: > > >> > > >> Will continue this discussion on the #hive ASF slack. If you are > > >> interested, please join. > > >> We will do updates here time-to-time, so the ones who are not using > > slack > > >> can participate that way. > > >> > > >> On 2022. Mar 2., at 11:11, Peter Vary wrote: > > >> > > >> Good idea Zoltan, joined the channel. > > >> I would like to scope reasonably small, so I agree with focusing on > > >> 4.0.0-alpha-1 > > >> > > >> On 2022. Mar 2., at 11:01, Zoltan Haindrich wrote: > > >> > > >> Hey, > > >> > > >> regarding 4.0.0 / 4.0.0-alpha-1 target/fix versions in the jira: > > >> * I think we should change all already resolved tickets with fix > version > > >> 4.0.0 to have fix version 4.0.0-alpha-1 > > >> ** this could be postponed until we are actually releasing the thing > as > > I > > >> think everyone committing to the master is entering 4.0.0 as fix > version > > >> without much aftertought...this could probably change after we get the > > >> first release out. > > >> * regarding the the existing tickets with fix version/target version > > 4.0.0 > > >> - I think that would be a bit too much (>200 tickets) > > >> ** some numbers: > > >> *** 239 tickets open now > > >> *** 224 was not updated in the last 90 days > > >> *** 216 was not changed in the last 180 days > > >> *** 178 was not updated in the last 360 days > > >> ** as a matter of fact I think many of these tickets shouldn't even > > have a > > >> target or fix version - and most of them should be unassigned...I > don't > > >> want to get lost in this right now...I think for now we should keep > the > > >> scope small and only care with 4.0.0-alpha-1 tickets > > >> > > >> https://issues.apache.org/jira/issues/? > > >> > > > jql=project%20%3D%20hive%20and%20resolutiondate%20%20is%20empty%20and%20(fixVersion%20%20in%20(%274.0.0%27)%20or%20cf%5B12310320%5D%20%20in%20(%274.0.0%27)) > > >> > > >> I think for faster communication regarding these things we could also > > >> utilize the #hive channel on the ASF slack - what do you guys think? > > >> > > >> cheers, > > >> Zoltan > > >> > > >> On 3/2/22 9:51 AM, Stamatis Zampetakis wrote: > > >> > > >> Agree with Peter, creating JIRAs is the way to go. > > >> Putting the appropriate priority (e.g., BLOCKER) and version (4.0.0 or > > >> 4.0.0-alpha-1) when creating the JIRA should be enough to keep us on > > track. > > >
[jira] [Created] (HIVE-26096) Select on single column MultiDelimitSerDe table throws AIOBE
Naresh P R created HIVE-26096:
-
Summary: Select on single column MultiDelimitSerDe table throws
AIOBE
Key: HIVE-26096
URL: https://issues.apache.org/jira/browse/HIVE-26096
Project: Hive
Issue Type: Bug
Reporter: Naresh P R
Assignee: Naresh P R
Repro details
{code:java}
create table test_multidelim(col string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe'
with serdeproperties('field.delim'='!^') STORED AS TEXTFILE;
insert into test_multidelim values('aa'),('bb'),('cc'),('dd');
select * from test_multidelim;
{code}
Exception:
{code:java}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit(LazyStruct.java:303)
at
org.apache.hadoop.hive.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:160)
at
org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:74)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:603){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-26097) Hive Table containing unicode data is not writing data to target properly
Hari Prasad G created HIVE-26097: Summary: Hive Table containing unicode data is not writing data to target properly Key: HIVE-26097 URL: https://issues.apache.org/jira/browse/HIVE-26097 Project: Hive Issue Type: Bug Affects Versions: 3.1.2 Reporter: Hari Prasad G *Source Table data which is staged in Amazon S3 location:* ++ |all_char.col| ++ |abcThe Hunger GamesThe Hunger Games 1by Suzanne CollinsPART ITHE TRIBUTES1.When I wake up the other side of the bed is cold. My fingers stretch out seeking Prims warmth but finding only the rough canvas cover of the mattress. She must have had bad dreams| | | |NULL| |abcdefghijklmnopqrstuvwxyz1234567890ABCD| |'Wonderful with quotes'| |russian - удивительный мир chineese 美好的世界 japneese 素晴らしき世界 hindi अनोखी दुनियाँ unicode| |abcdefghijklmnopqrstuvwxyz1234567890A~`!@#$%^&*();| |TrailingwithSpace| *Query which is executed* : INSERT INTO TABLE `all_char_tgt`(`col`) SELECT `col` FROM `all_char`; ++ *Target Table Data:* ++ |all_char_tgt.col| ++ |abcThe Hunger GamesThe Hunger Games 1by Suzanne CollinsPART ITHE TRIBUTES1.When I wake up the other side of the bed is cold. My fingers stretch out seeking Prims warmth but finding only the rough canvas cover of the mattress. She must have had bad dreams| |abcdefghijklmnopqrstuvwxyz1234567890ABCD| |NULL| |russian - �� chineese ��� japneese � hindi ��� � unicode| | | |'Wonderful with quotes'| |abcdefghijklmnopqrstuvwxyz1234567890A~`!@#$%^&*();| |TrailingwithSpace| ++ *Expected* Data Should be written properly in target table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Apache Hive 3.1.3 Release Candidate 2
+1 (non-binding) Tested through: https://github.com/apache/spark/pull/36018 From: Naveen Gangam Date: Wednesday, March 30, 2022 at 21:14 To: dev@hive.apache.org Subject: Re: [VOTE] Apache Hive 3.1.3 Release Candidate 2 External Email Still seeking votes. Voting ends tomorrow. Any help would be appreciated. Thank you Naveen On Tue, Mar 29, 2022 at 5:51 AM Peter Vary wrote: > Downloaded the 3.1.3 artifacts, and checked the signatures. They are OK. > Used the binary to run some basic tests, and it seems OK. > > +1 (binding) > > > On 2022. Mar 28., at 23:19, Naveen Gangam > wrote: > > > > Apache Hive 3.1.3 Release Candidate 2 is available here: > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.apache.org%2F~ngangam%2Fapache-hive-3.1.3-rc-2&data=04%7C01%7Cyumwang%40ebay.com%7Ca9fbec3df79d488725ba08da124f07c7%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637842428468446173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=H9WqpQWLF0KzS1uPi1WXlDYv%2FCOR2KagWLYcotoqBT0%3D&reserved=0 > > > > The checksums are these: > > > > > > - 55c58e0111bd32de3d02f5f25d9eb054ba65ab02aaf669637760eaf56ef1fbb1 > > apache-hive-3.1.3-bin.tar.gz > > > > > > - 22862e6bf76a4783a3d8d298634728cc9d6561563af2413a687fe63e35bcc527 > > apache-hive-3.1.3-src.tar.gz > > > > > > Maven artifacts are available here: > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehive-1114&data=04%7C01%7Cyumwang%40ebay.com%7Ca9fbec3df79d488725ba08da124f07c7%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637842428468446173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=puA%2FrgfwhIA4wCXE162DfjoHtpc4NUGPFkJfY6G1amM%3D&reserved=0 > > > > The tag release-3.1.3-rc2 has been applied to the source for this > > release in github, you can see it > > athttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Ftree%2Frelease-3.1.3-rc2&data=04%7C01%7Cyumwang%40ebay.com%7Ca9fbec3df79d488725ba08da124f07c7%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637842428468446173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=N0Sv3tqLJlaSI%2BCUmB0wzmyObxlfIMvtzHWTySoROVU%3D&reserved=0 > > > > The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975 > > < > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fcommit%2F4df4d75bf1e16fe0af75aad0b4179c34c07fc975&data=04%7C01%7Cyumwang%40ebay.com%7Ca9fbec3df79d488725ba08da124f07c7%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637842428468446173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lZY3bCBmacmI2HbTsFRhgggU242Mba3%2BxVB34z6fJ9s%3D&reserved=0 > > > > Voting will conclude in 72 hours. > > > > Hive PMC Members: Please test and vote. > > > > Thanks. > > > > Naveen > >
