[jira] [Updated] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20441:
--
Fix Version/s: (was: 2.3.3)
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Attachments: HIVE-20441.1.patch, HIVE-20441.2.patch,
> HIVE-20441.3.patch, HIVE-20441.4.patch, HIVE-20441.patch
>
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
> SHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
> T]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
> ~[hive-exec-2.
> 3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.
> 3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20284) In strict mode, if constant propagation is enable, the partition filter may be folded before partition pruner lead to error "No partition predicate for Alias"
[
https://issues.apache.org/jira/browse/HIVE-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20284:
--
Fix Version/s: (was: 2.3.3)
> In strict mode, if constant propagation is enable, the partition filter may
> be folded before partition pruner lead to error "No partition predicate for
> Alias"
>
>
> Key: HIVE-20284
> URL: https://issues.apache.org/jira/browse/HIVE-20284
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20284.1.patch, HIVE-20284.2.patch,
> HIVE-20284.3.patch, HIVE-20284.patch
>
>
> In strict mode and the hive.optimize.constant.propagation is set to true, the
> following sql will failed:
> {code:java}
> hive> desc employee_part;
> OK
> col_name data_type comment
> eid int
> name string
> dept string
> year string
> month string
> # Partition Information
> # col_namedata_type comment
> year string
> month string
> Time taken: 0.564 seconds, Fetched: 11 row(s)
> hive> set hive.mapred.mode=strict;
> hive> select * from employee_part where false and concat(year,month)='201807';
> FAILED: SemanticException Queries against partitioned tables without a
> partition filter are disabled for safety reasons. If you know what you are
> doing, please sethive.strict.checks.large.query to false and that
> hive.mapred.mode is not set to 'strict' to proceed. Note that if you may get
> errors or incorrect results if you make a mistake while using some of the
> unsafe features. No partition predicate for Alias "employee_part" Table
> "employee_part"
> {code}
> The above error message is confusing because the expression
> concat(year,month)='201807' is a partition filter。
> The reason is during logic optimization, the ConstantPropagate optimizer is
> running before partitionPruner optimizer, when found a express like 'false
> and concat(year,month)=', the express is replaced with 'fasle' , and the
> partition filter is dropped. So the PartitionPruner can not get the partition
> filter.
> Users can remove the constant express that always has true/false values to
> work around.
> When views used, if some columns are constant values, users will be
> confusing.
> So we should add some more message in the error msg returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20304) When hive.optimize.skewjoin and hive.auto.convert.join are both set to true, and the execution engine is mr, same stage may launch twice due to the wrong generated plan
[
https://issues.apache.org/jira/browse/HIVE-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20304:
--
Fix Version/s: (was: 2.3.3)
> When hive.optimize.skewjoin and hive.auto.convert.join are both set to true,
> and the execution engine is mr, same stage may launch twice due to the wrong
> generated plan
>
>
> Key: HIVE-20304
> URL: https://issues.apache.org/jira/browse/HIVE-20304
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Fix For: 1.2.1
>
> Attachments: HIVE-20304.1.patch, HIVE-20304.patch
>
>
> `When hive.optimize.skewjoin and hive.auto.convert.join are both set to true,
> and the execution engine is set to mr, same stage of a query may launch twice
> due to the wrong generated plan. If hive.exec.parallel is also true, the same
> stage will launch at the same time and the job will failed due to the first
> completed stage clear the map.xml/reduce.xml file stored in the hdfs.
> use following sql to reproduce the issue:
> {code:java}
> CREATE TABLE `tbl1`(
> `fence` string);
> CREATE TABLE `tbl2`(
> `order_id` string,
> `phone` string,
> `search_id` string
> )
> PARTITIONED BY (
> `dt` string);
> CREATE TABLE `tbl3`(
> `order_id` string,
> `platform` string)
> PARTITIONED BY (
> `dt` string);
> CREATE TABLE `tbl4`(
> `groupname` string,
> `phone` string)
> PARTITIONED BY (
> `dt` string);
> CREATE TABLE `tbl5`(
> `search_id` string,
> `fence` string)
> PARTITIONED BY (
> `dt` string);
> SET hive.exec.parallel = TRUE;
> SET hive.auto.convert.join = TRUE;
> SET hive.optimize.skewjoin = TRUE;
> SELECT dt,
>platform,
>groupname,
>count(1) as cnt
> FROM
> (SELECT dt,
> platform,
> groupname
> FROM
> (SELECT fence
> FROM tbl1)ta
>JOIN
>(SELECT a0.dt,
>a1.platform,
>a2.groupname,
>a3.fence
> FROM
> (SELECT dt,
> order_id,
> phone,
> search_id
> FROM tbl2
> WHERE dt =20180703 )a0
> JOIN
> (SELECT order_id,
> platform,
> dt
>FROM tbl3
>WHERE dt =20180703 )a1 ON a0.order_id = a1.order_id
> INNER JOIN
> (SELECT groupname,
> phone,
> dt
>FROM tbl4
>WHERE dt =20180703 )a2 ON a0.phone = a2.phone
> LEFT JOIN
> (SELECT search_id,
> fence,
> dt
>FROM tbl5
>WHERE dt =20180703)a3 ON a0.search_id = a3.search_id)t0 ON
> ta.fence = t0.fence)t11
> GROUP BY dt,
> platform,
> groupname;
> DROP TABLE tbl1;
> DROP TABLE tbl2;
> DROP TABLE tbl3;
> DROP TABLE tbl4;
> DROP TABLE tbl5;
> {code}
> We will get some error message like this:
> Examining task ID: task_1531284442065_3637_m_00 (and more) from job
> job_1531284442065_3637
> Task with the most failures(4):
>
> Task ID:
> task_1531284442065_3637_m_00
> URL:
>
> [http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1531284442065_3637=task_1531284442065_3637_m_00]
>
> Diagnostic Messages for this Task:
> File does not exist:
> hdfs://test/tmp/hive-hadoop/hadoop/fe5efa94-abb1-420f-b6ba-ec782e7b79ad/hive_2018-08-03_17-00-17_707_592882314975289971-5/-mr-10045/757eb1f7-7a37-4a7e-abc0-4a3b8b06510c/reduce.xml
> java.io.FileNotFoundException: File does not exist:
> hdfs://test/tmp/hive-hadoop/hadoop/fe5efa94-abb1-420f-b6ba-ec782e7b79ad/hive_2018-08-03_17-00-17_707_592882314975289971-5/-mr-10045/757eb1f7-7a37-4a7e-abc0-4a3b8b06510c/reduce.xml
> Looking into the plan by executing explain, I found that the Stage-4 and
> Stage-5 can reached from multi root tasks.
> {code:java}
> Explain
> STAGE DEPENDENCIES:
> Stage-21 is a root stage , consists of Stage-34, Stage-5
> Stage-34 has a backup stage: Stage-5
> Stage-20 depends on stages: Stage-34
> Stage-17 depends on stages: Stage-5, Stage-18, Stage-20 , consists of
> Stage-32, Stage-33, Stage-1
> Stage-32 has a backup stage: Stage-1
> Stage-15 depends on stages: Stage-32
> Stage-10 depends on stages: Stage-1, Stage-15, Stage-16 , consists of
> Stage-31, Stage-2
> Stage-31
> Stage-9 depends on stages: Stage-31
> Stage-2
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654666#comment-16654666 ] Vihang Karajgaonkar commented on HIVE-20697: yeah, nothing to be done with regards to batching of these tests. > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19900) HiveCLI HoS Performs Invalid Impersonation If User Name Truncated
[
https://issues.apache.org/jira/browse/HIVE-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-19900:
--
Fix Version/s: (was: 4.0.0)
(was: 2.3.3)
(was: 1.2.3)
> HiveCLI HoS Performs Invalid Impersonation If User Name Truncated
> -
>
> Key: HIVE-19900
> URL: https://issues.apache.org/jira/browse/HIVE-19900
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Spark
>Affects Versions: 1.2.2, 3.0.0, 2.3.2, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
>
> The HiveCLI HoS code relies on the system property {{user.name}} when
> performing impersonations. The code decides to do an impersonation if the
> {{user.name}} system property does not match the current user who is
> launching the HiveCLI client. However, when confronted with a long user
> name, some shells and linux distros may opt to truncate the user name to a
> certain size to conserve screen space. In these scenarios, the current user
> name does not match the {{user.name}} system property and never will, so
> impersonation will always happen, even though the user is trying to
> impersonate themselves. If YARN is not setup to allow the current user to
> impersonate, YARN will reject the request.
> {code:java}
> if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS)) {
> try {
> String currentUser = Utils.getUGI().getShortUserName();
> // do not do impersonation in CLI mode
> if (!currentUser.equals(System.getProperty("user.name"))) {
> LOG.info("Attempting impersonation of " + currentUser);
> addProxyUser(currentUser);
> }
> } catch (Exception e) {
> String msg = "Cannot obtain username: " + e;
> throw new IllegalStateException(msg, e);
> }
> }
> {code}
>
> [https://github.com/apache/hive/blob/da66386662fbbcbde9501b4a7b27d076bcc790d4/spark-client/src/main/java/org/apache/hive/spark/client/AbstractSparkClient.java#L354-L366]
> Assuming a kerberos enabled environment, the error message in the YARN
> Resource Manager will be:
> {code:java}
> my-really-really-long-user-n...@hadoop.domain.com is not allowed to
> impersonate my-really-really-long-user-name
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20718) Add perf cli driver with constraints
[
https://issues.apache.org/jira/browse/HIVE-20718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654663#comment-16654663
]
Hive QA commented on HIVE-20718:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
35s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} itests/qtest: The patch generated 2 new + 2 unchanged
- 0 fixed = 4 total (was 2) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} itests/util: The patch generated 5 new + 28 unchanged
- 0 fixed = 33 total (was 28) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 13 new + 314 unchanged - 1
fixed = 327 total (was 315) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 405 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
8s{color} | {color:red} The patch 399 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
57s{color} | {color:red} ql generated 11 new + 2306 unchanged - 12 fixed = 2317
total (was 2318) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA232 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46862-46875] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA275 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
47212-47225] |
| | Dead store to LA29_127 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 46600] |
| | Dead store to LA29_129 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 46613] |
| | Dead store to LA29_131 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
[jira] [Commented] (HIVE-20718) Add perf cli driver with constraints
[
https://issues.apache.org/jira/browse/HIVE-20718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654662#comment-16654662
]
Hive QA commented on HIVE-20718:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944383/HIVE-20718.08.patch
{color:green}SUCCESS:{color} +1 due to 100 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15377 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14534/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14534/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14534/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944383 - PreCommit-HIVE-Build
> Add perf cli driver with constraints
>
>
> Key: HIVE-20718
> URL: https://issues.apache.org/jira/browse/HIVE-20718
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20718.01.patch, HIVE-20718.02.patch,
> HIVE-20718.03.patch, HIVE-20718.04.patch, HIVE-20718.05.patch,
> HIVE-20718.06.patch, HIVE-20718.07.patch, HIVE-20718.08.patch,
> HIVE-20718.patch
>
>
> Now that subtasks in HIVE-17039 will be completed, it will be good to have a
> perf cli driver with constraints declaration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18767) Some alterPartitions invocations throw 'NumberFormatException: null'
[
https://issues.apache.org/jira/browse/HIVE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-18767:
--
Fix Version/s: (was: 2.3.3)
2.4.0
2.3.4
> Some alterPartitions invocations throw 'NumberFormatException: null'
>
>
> Key: HIVE-18767
> URL: https://issues.apache.org/jira/browse/HIVE-18767
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3, 3.1.0, 4.0.0, 3.2.0
>Reporter: Yuming Wang
>Assignee: Mass Dosage
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 2.3.4
>
> Attachments: HIVE-18767-branch-2.3.patch, HIVE-18767-branch-2.patch,
> HIVE-18767-branch-3.1.patch, HIVE-18767-branch-3.patch, HIVE-18767.1.patch,
> HIVE-18767.2-branch-2.3.patch, HIVE-18767.2-branch-2.patch,
> HIVE-18767.2-branch-3.1.patch, HIVE-18767.2.patch,
> HIVE-18767.3-branch-3.1.patch, HIVE-18767.3.patch,
> HIVE-18767.4-branch-3.1.patch, HIVE-18767.4.patch, HIVE-18767.5.patch,
> HIVE-18767.6.patch
>
>
> Error messages:
> {noformat}
> [info] Cause: java.lang.NumberFormatException: null
> [info] at java.lang.Long.parseLong(Long.java:552)
> [info] at java.lang.Long.parseLong(Long.java:631)
> [info] at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:315)
> [info] at
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:605)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3837)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:498)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> [info] at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> [info] at
> com.sun.proxy.$Proxy23.alter_partitions_with_environment_context(Unknown
> Source)
> [info] at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1527)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20770) Need improvement in hive for ACID properties and tables
[ https://issues.apache.org/jira/browse/HIVE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pritam reassigned HIVE-20770: - Assignee: Alex Lumpov (was: pritam) > Need improvement in hive for ACID properties and tables > --- > > Key: HIVE-20770 > URL: https://issues.apache.org/jira/browse/HIVE-20770 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: pritam >Assignee: Alex Lumpov >Priority: Major > > Hello Team, > In current version of Apache Hive if we set ACID properties it can not be > revert as well as > Apache Spark does not support Hive ACID table. If it is possible to revert > ACID properties from hive table and read-write Hive table from spark-scala > please let me know. > If there is no provision for above conditions it is important to add features > and improvement in next Apache Hive version. It will be very helpful for all > in distributed batch processing. > I am eager to hear from you. Thank you all in advance for such a great batch > processing tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20770) Need improvement in hive for ACID properties and tables
[ https://issues.apache.org/jira/browse/HIVE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pritam reassigned HIVE-20770: - Assignee: pritam > Need improvement in hive for ACID properties and tables > --- > > Key: HIVE-20770 > URL: https://issues.apache.org/jira/browse/HIVE-20770 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: pritam >Assignee: pritam >Priority: Major > > Hello Team, > In current version of Apache Hive if we set ACID properties it can not be > revert as well as > Apache Spark does not support Hive ACID table. If it is possible to revert > ACID properties from hive table and read-write Hive table from spark-scala > please let me know. > If there is no provision for above conditions it is important to add features > and improvement in next Apache Hive version. It will be very helpful for all > in distributed batch processing. > I am eager to hear from you. Thank you all in advance for such a great batch > processing tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654645#comment-16654645 ] anishek commented on HIVE-20679: [~sankarh] please review > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654641#comment-16654641 ] ASF GitHub Bot commented on HIVE-20679: --- Github user anishek closed the pull request at: https://github.com/apache/hive/pull/446 > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654643#comment-16654643 ] ASF GitHub Bot commented on HIVE-20679: --- GitHub user anishek opened a pull request: https://github.com/apache/hive/pull/449 HIVE-20679: DDL operations on hive might create large messages for DBNotification You can merge this pull request into a Git repository by running: $ git pull https://github.com/anishek/hive HIVE-20679 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/449.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #449 commit 9767cec4e7eb9d05afa6ac6f9279a6358cd50815 Author: Anishek Agarwal Date: 2018-10-03T09:28:40Z HIVE-20679: DDL operations on hive might create large messages for DBNotification > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-20679: -- Labels: pull-request-available (was: ) > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-20679: --- Attachment: HIVE-20679.7.patch > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[ https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654633#comment-16654633 ] anishek commented on HIVE-20697: Thanks [~vihangk1], does this mean skipbatching is available for all commits running / submitted ? There is no patch that needs to be uploaded here for now for this correct ? > Some replication tests are super slow and cause batch timeouts > -- > > Key: HIVE-20697 > URL: https://issues.apache.org/jira/browse/HIVE-20697 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Priority: Major > > Some of these tests are taking a long time and can cause test batch timeouts > given that we only give 40 min for a batch to complete. We should speed these > tests up. > TestReplicationScenarios 20 min > TestReplicationScenariosAcidTables11 min > TestReplicationScenariosAcrossInstances 5 min 14 sec > TestReplicationScenariosIncrementalLoadAcidTables 20 min -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20761) Select for update on notification_sequence table has retry interval and retries count too small.
[
https://issues.apache.org/jira/browse/HIVE-20761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20761:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master!
Thanks [~thejas] for the review!
> Select for update on notification_sequence table has retry interval and
> retries count too small.
>
>
> Key: HIVE-20761
> URL: https://issues.apache.org/jira/browse/HIVE-20761
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: Metastore
> Fix For: 4.0.0
>
> Attachments: HIVE-20761.01.patch
>
>
> Hive DDL's are intermittently failing with Error- Couldn't acquire the DB log
> notification lock because we reached the maximum # of retries: 5 retries
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't
> acquire the DB log notification lock because we reached the maximum # of
> retries: 5 retries. If this happens too often, then is recommended to
> increase the maximum number of retries on the
> hive.notification.sequence.lock.max.retries configuration :: Error executing
> SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".)
> 2018-08-28 01:17:56,808|INFO|MainThread|machine.py:183 -
> run()||GUID=94e6ff4d-5db8-45eb-8654-76f546e7f0b3|java.sql.SQLException: Error
> while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't
> acquire the DB log notification lock because we reached the maximum # of
> retries: 5 retries. If this happens too often, then is recommended to
> increase the maximum number of retries on the
> hive.notification.sequence.lock.max.retries configuration :: Error executing
> SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for
> update".){code}
> It seems, metastore operations are slow in this cluster and hence concurrent
> writes/DDL operations are failing to lock the row for update.
> Currently, the sleep interval between retries is specified via the config
> *hive.notification.sequence.lock.retry.sleep.interval*. The default value is
> 500 ms which seems to be too small. Can we set higher values for sleep
> interval and retries count,
> *hive.notification.sequence.lock.retry.sleep.interval=10s*
> *hive.notification.sequence.lock.max.retries=10*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20761) Select for update on notification_sequence table has retry interval and retries count too small.
[
https://issues.apache.org/jira/browse/HIVE-20761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654620#comment-16654620
]
Hive QA commented on HIVE-20761:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944380/HIVE-20761.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15096 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14533/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14533/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14533/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944380 - PreCommit-HIVE-Build
> Select for update on notification_sequence table has retry interval and
> retries count too small.
>
>
> Key: HIVE-20761
> URL: https://issues.apache.org/jira/browse/HIVE-20761
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: Metastore
> Attachments: HIVE-20761.01.patch
>
>
> Hive DDL's are intermittently failing with Error- Couldn't acquire the DB log
> notification lock because we reached the maximum # of retries: 5 retries
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't
> acquire the DB log notification lock because we reached the maximum # of
> retries: 5 retries. If this happens too often, then is recommended to
> increase the maximum number of retries on the
> hive.notification.sequence.lock.max.retries configuration :: Error executing
> SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".)
> 2018-08-28 01:17:56,808|INFO|MainThread|machine.py:183 -
> run()||GUID=94e6ff4d-5db8-45eb-8654-76f546e7f0b3|java.sql.SQLException: Error
> while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't
> acquire the DB log notification lock because we reached the maximum # of
> retries: 5 retries. If this happens too often, then is recommended to
> increase the maximum number of retries on the
> hive.notification.sequence.lock.max.retries configuration :: Error executing
> SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for
> update".){code}
> It seems, metastore operations are slow in this cluster and hence concurrent
> writes/DDL operations are failing to lock the row for update.
> Currently, the sleep interval between retries is specified via the config
> *hive.notification.sequence.lock.retry.sleep.interval*. The default value is
> 500 ms which seems to be too small. Can we set higher values for sleep
> interval and retries count,
> *hive.notification.sequence.lock.retry.sleep.interval=10s*
> *hive.notification.sequence.lock.max.retries=10*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654615#comment-16654615 ] Jesus Camacho Rodriguez commented on HIVE-20720: [~daijy], thanks. Wrt pattern matching on FROM clause, I believe it is quite safe with your latest change: Calcite will only set the splittable flag to 'true' for Select-Filter-Scan queries (no join, group by, or other statements), and if user is facing issues with hardcoded query, they can always rewrite it. As we move forward and we split more complex computation, we may revisit that logic. +1 (pending tests) > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20720: -- Assignee: Daniel Dai (was: Jesus Camacho Rodriguez) > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20720: -- Assignee: Jesus Camacho Rodriguez (was: Daniel Dai) > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20720: -- Attachment: HIVE-20720.8.patch > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20720) Add partition column option to JDBC handler
[ https://issues.apache.org/jira/browse/HIVE-20720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654591#comment-16654591 ] Daniel Dai commented on HIVE-20720: --- Addressing [~jcamachorodriguez]'s review comments. > Add partition column option to JDBC handler > --- > > Key: HIVE-20720 > URL: https://issues.apache.org/jira/browse/HIVE-20720 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20720.1.patch, HIVE-20720.2.patch, > HIVE-20720.3.patch, HIVE-20720.4.patch, HIVE-20720.5.patch, > HIVE-20720.6.patch, HIVE-20720.7.patch, HIVE-20720.8.patch > > > Currently JdbcStorageHandler does not split input in Tez. The reason is > numSplit of JdbcInputFormat.getSplits can only pass via "mapreduce.job.maps" > in Tez. And "mapreduce.job.maps" is not a valid param if authorizer(eg. > SQLStdAuth) is in use. User ends up always use 1 split. > We need to rely on this new feature if we want to support multi-splits. Here > is my proposal: > 1. Specify partitionColumn/numPartitions, and optional lowerBound/upperBound > in tblproperties if user want to split jdbc data source. In case > lowerBound/upperBound is not specified, JdbcStorageHandler will run max/min > query to get this in planner. We can currently limit partitionColumn to only > numeric/date/timestamp column for simplicity > 2. If partitionColumn/numPartitions are not specified, don't split input > 3. Splits are equal intervals without respect to data distribution > 4. There is also a "hive.sql.query.split" flag vetos the split (can be set > manually or automatically by calcite) > 5. If partitionColumn is not defined, but numPartitions is defined, use > original limit/offset logic (however, don't rely on numSplit). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20761) Select for update on notification_sequence table has retry interval and retries count too small.
[
https://issues.apache.org/jira/browse/HIVE-20761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654586#comment-16654586
]
Hive QA commented on HIVE-20761:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
13s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14533/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common standalone-metastore/metastore-common U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14533/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Select for update on notification_sequence table has retry interval and
> retries count too small.
>
>
> Key: HIVE-20761
> URL: https://issues.apache.org/jira/browse/HIVE-20761
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: Metastore
> Attachments: HIVE-20761.01.patch
>
>
> Hive DDL's are intermittently failing with Error- Couldn't acquire the DB log
> notification lock because we reached the maximum # of retries: 5 retries
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't
> acquire the DB log notification lock because we reached the maximum # of
> retries: 5 retries. If this happens too often, then is recommended to
> increase the maximum number of retries on the
> hive.notification.sequence.lock.max.retries
[jira] [Commented] (HIVE-20477) OptimizedSql is not shown if the expression contains INs
[
https://issues.apache.org/jira/browse/HIVE-20477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654567#comment-16654567
]
Hive QA commented on HIVE-20477:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944340/HIVE-20477.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15096 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=189)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14532/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14532/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14532/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944340 - PreCommit-HIVE-Build
> OptimizedSql is not shown if the expression contains INs
>
>
> Key: HIVE-20477
> URL: https://issues.apache.org/jira/browse/HIVE-20477
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20477.01.patch, HIVE-20477.01wip01.patch
>
>
> This ticket should fix HiveIn to be able to unparse; currently if an IN is
> unparsed - there are some exceptions because HiveIn is a special operator;
> but doesn't have unparse implemented.
> CALCITE-2444 is also needed to fix rel2sql to be able to process INs which
> are not there represent a subquery.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20490) UDAF: Add an `approx_distinct` to Hive
[ https://issues.apache.org/jira/browse/HIVE-20490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654555#comment-16654555 ] Gunther Hagleitner commented on HIVE-20490: --- +1 > UDAF: Add an `approx_distinct` to Hive > -- > > Key: HIVE-20490 > URL: https://issues.apache.org/jira/browse/HIVE-20490 > Project: Hive > Issue Type: Improvement > Components: UDF >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20490.1.patch, HIVE-20490.2.patch > > > This gives a direct pathway to call the equivalent of Hive's default > compute_stats() UDAF which computes a HyperLogLog bitVector + approx distinct > on a column. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20490) UDAF: Add an `approx_distinct` to Hive
[ https://issues.apache.org/jira/browse/HIVE-20490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20490: --- Attachment: HIVE-20490.2.patch > UDAF: Add an `approx_distinct` to Hive > -- > > Key: HIVE-20490 > URL: https://issues.apache.org/jira/browse/HIVE-20490 > Project: Hive > Issue Type: Improvement > Components: UDF >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20490.1.patch, HIVE-20490.2.patch > > > This gives a direct pathway to call the equivalent of Hive's default > compute_stats() UDAF which computes a HyperLogLog bitVector + approx distinct > on a column. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-20763: -- Status: Patch Available (was: Open) > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20768) Adding Tumbling Window UDF
[
https://issues.apache.org/jira/browse/HIVE-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654541#comment-16654541
]
Gopal V commented on HIVE-20768:
LGTM - +1 tests pending
> Adding Tumbling Window UDF
> --
>
> Key: HIVE-20768
> URL: https://issues.apache.org/jira/browse/HIVE-20768
> Project: Hive
> Issue Type: New Feature
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20768.patch
>
>
> Goal is to provide a UDF that truncates a timestamp to a beginning of a
> tumbling window interval.
> {code}
> /**
> * Tumbling windows are a series of fixed-sized, non-overlapping and
> contiguous time intervals.
> * Tumbling windows are inclusive start exclusive end.
> * By default the beginning instant of fist window is Epoch 0 Thu Jan 01
> 00:00:00 1970 UTC.
> * Optionally users may provide a different origin as a timestamp arg3.
> *
> * This an example of series of window with an interval of 5 seconds and
> origin Epoch 0 Thu Jan 01 00:00:00 1970 UTC:
> *
> *
> * interval 1 interval 2interval 3
> * Jan 01 00:00:00 Jan 01 00:00:05 Jan 01 00:00:10
> * 0 -- 4 : 5 --- 9: 10 --- 14
> *
> * This UDF rounds timestamp agr1 to the beginning of window interval where
> it belongs to.
> *
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20477) OptimizedSql is not shown if the expression contains INs
[
https://issues.apache.org/jira/browse/HIVE-20477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654539#comment-16654539
]
Hive QA commented on HIVE-20477:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14532/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14532/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> OptimizedSql is not shown if the expression contains INs
>
>
> Key: HIVE-20477
> URL: https://issues.apache.org/jira/browse/HIVE-20477
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20477.01.patch, HIVE-20477.01wip01.patch
>
>
> This ticket should fix HiveIn to be able to unparse; currently if an IN is
> unparsed - there are some exceptions because HiveIn is a special operator;
> but doesn't have unparse implemented.
> CALCITE-2444 is also needed to fix rel2sql to be able to process INs which
> are not there represent a subquery.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654502#comment-16654502
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color}
| {color:red} common generated 26 new + 0 unchanged - 0 fixed = 26 total (was
0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 51 new + 424 unchanged - 0
fixed = 475 total (was 424) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[lines
164-167] |
| | org.apache.hadoop.hive.common.HiveConfProperties defines equals but not
hashCode At HiveConfProperties.java:hashCode At
HiveConfProperties.java:[lines 201-282] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 44-46] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14531/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/diff-compile-javac-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/new-findbugs-common.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760.patch,
[jira] [Commented] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[
https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654488#comment-16654488
]
Hive QA commented on HIVE-20661:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944329/HIVE-20661.09.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15096 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14530/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14530/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14530/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944329 - PreCommit-HIVE-Build
> Dynamic partitions loading calls add partition for every partition 1-by-1
> -
>
> Key: HIVE-20661
> URL: https://issues.apache.org/jira/browse/HIVE-20661
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20661.01.patch, HIVE-20661.02.patch,
> HIVE-20661.03.patch, HIVE-20661.04.patch, HIVE-20661.05.patch,
> HIVE-20661.06.patch, HIVE-20661.07.patch, HIVE-20661.08.patch,
> HIVE-20661.09.patch
>
>
> Hive.loadDynamicPartitions creates partitions using a threadpool, but the
> update of the Metastore via the MetastoreClient is done 1-by-1. This adds
> unnecessary extra calls. The partitions should be created in one batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20761) Select for update on notification_sequence table has retry interval and retries count too small.
[
https://issues.apache.org/jira/browse/HIVE-20761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1665#comment-1665
]
Thejas M Nair commented on HIVE-20761:
--
+1 pending tests
> Select for update on notification_sequence table has retry interval and
> retries count too small.
>
>
> Key: HIVE-20761
> URL: https://issues.apache.org/jira/browse/HIVE-20761
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Minor
> Labels: Metastore
> Attachments: HIVE-20761.01.patch
>
>
> Hive DDL's are intermittently failing with Error- Couldn't acquire the DB log
> notification lock because we reached the maximum # of retries: 5 retries
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't
> acquire the DB log notification lock because we reached the maximum # of
> retries: 5 retries. If this happens too often, then is recommended to
> increase the maximum number of retries on the
> hive.notification.sequence.lock.max.retries configuration :: Error executing
> SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".)
> 2018-08-28 01:17:56,808|INFO|MainThread|machine.py:183 -
> run()||GUID=94e6ff4d-5db8-45eb-8654-76f546e7f0b3|java.sql.SQLException: Error
> while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't
> acquire the DB log notification lock because we reached the maximum # of
> retries: 5 retries. If this happens too often, then is recommended to
> increase the maximum number of retries on the
> hive.notification.sequence.lock.max.retries configuration :: Error executing
> SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for
> update".){code}
> It seems, metastore operations are slow in this cluster and hence concurrent
> writes/DDL operations are failing to lock the row for update.
> Currently, the sleep interval between retries is specified via the config
> *hive.notification.sequence.lock.retry.sleep.interval*. The default value is
> 500 ms which seems to be too small. Can we set higher values for sleep
> interval and retries count,
> *hive.notification.sequence.lock.retry.sleep.interval=10s*
> *hive.notification.sequence.lock.max.retries=10*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654441#comment-16654441
]
Vihang Karajgaonkar commented on HIVE-20740:
Test failures may be related. Taking a look.
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[
https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654435#comment-16654435
]
Hive QA commented on HIVE-20661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
22s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 18 new + 242 unchanged - 3
fixed = 260 total (was 245) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14530/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14530/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14530/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dynamic partitions loading calls add partition for every partition 1-by-1
> -
>
> Key: HIVE-20661
> URL: https://issues.apache.org/jira/browse/HIVE-20661
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20661.01.patch, HIVE-20661.02.patch,
> HIVE-20661.03.patch, HIVE-20661.04.patch, HIVE-20661.05.patch,
> HIVE-20661.06.patch, HIVE-20661.07.patch, HIVE-20661.08.patch,
> HIVE-20661.09.patch
>
>
> Hive.loadDynamicPartitions creates partitions using a threadpool, but the
> update of the Metastore via the MetastoreClient is done 1-by-1. This adds
> unnecessary extra calls. The partitions should be created in one batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[ https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20752: -- Attachment: HIVE-20752.03.patch > In case of LLAP start failure add info how to find YARN logs > > > Key: HIVE-20752 > URL: https://issues.apache.org/jira/browse/HIVE-20752 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20752.01.patch, HIVE-20752.02.patch, > HIVE-20752.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[ https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20752: -- Status: Open (was: Patch Available) > In case of LLAP start failure add info how to find YARN logs > > > Key: HIVE-20752 > URL: https://issues.apache.org/jira/browse/HIVE-20752 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20752.01.patch, HIVE-20752.02.patch, > HIVE-20752.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20752) In case of LLAP start failure add info how to find YARN logs
[ https://issues.apache.org/jira/browse/HIVE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20752: -- Status: Patch Available (was: Open) > In case of LLAP start failure add info how to find YARN logs > > > Key: HIVE-20752 > URL: https://issues.apache.org/jira/browse/HIVE-20752 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20752.01.patch, HIVE-20752.02.patch, > HIVE-20752.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20768) Adding Tumbling Window UDF
[
https://issues.apache.org/jira/browse/HIVE-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20768:
--
Attachment: HIVE-20768.patch
> Adding Tumbling Window UDF
> --
>
> Key: HIVE-20768
> URL: https://issues.apache.org/jira/browse/HIVE-20768
> Project: Hive
> Issue Type: New Feature
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20768.patch
>
>
> Goal is to provide a UDF that truncates a timestamp to a beginning of a
> tumbling window interval.
> {code}
> /**
> * Tumbling windows are a series of fixed-sized, non-overlapping and
> contiguous time intervals.
> * Tumbling windows are inclusive start exclusive end.
> * By default the beginning instant of fist window is Epoch 0 Thu Jan 01
> 00:00:00 1970 UTC.
> * Optionally users may provide a different origin as a timestamp arg3.
> *
> * This an example of series of window with an interval of 5 seconds and
> origin Epoch 0 Thu Jan 01 00:00:00 1970 UTC:
> *
> *
> * interval 1 interval 2interval 3
> * Jan 01 00:00:00 Jan 01 00:00:05 Jan 01 00:00:10
> * 0 -- 4 : 5 --- 9: 10 --- 14
> *
> * This UDF rounds timestamp agr1 to the beginning of window interval where
> it belongs to.
> *
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20768) Adding Tumbling Window UDF
[
https://issues.apache.org/jira/browse/HIVE-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20768:
--
Status: Patch Available (was: Open)
> Adding Tumbling Window UDF
> --
>
> Key: HIVE-20768
> URL: https://issues.apache.org/jira/browse/HIVE-20768
> Project: Hive
> Issue Type: New Feature
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>
> Goal is to provide a UDF that truncates a timestamp to a beginning of a
> tumbling window interval.
> {code}
> /**
> * Tumbling windows are a series of fixed-sized, non-overlapping and
> contiguous time intervals.
> * Tumbling windows are inclusive start exclusive end.
> * By default the beginning instant of fist window is Epoch 0 Thu Jan 01
> 00:00:00 1970 UTC.
> * Optionally users may provide a different origin as a timestamp arg3.
> *
> * This an example of series of window with an interval of 5 seconds and
> origin Epoch 0 Thu Jan 01 00:00:00 1970 UTC:
> *
> *
> * interval 1 interval 2interval 3
> * Jan 01 00:00:00 Jan 01 00:00:05 Jan 01 00:00:10
> * 0 -- 4 : 5 --- 9: 10 --- 14
> *
> * This UDF rounds timestamp agr1 to the beginning of window interval where
> it belongs to.
> *
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20768) Adding Tumbling Window UDF
[
https://issues.apache.org/jira/browse/HIVE-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654410#comment-16654410
]
slim bouguerra commented on HIVE-20768:
---
[~t3rmin4t0r]/[~ashutoshc]/
can you please take a look.
> Adding Tumbling Window UDF
> --
>
> Key: HIVE-20768
> URL: https://issues.apache.org/jira/browse/HIVE-20768
> Project: Hive
> Issue Type: New Feature
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>
> Goal is to provide a UDF that truncates a timestamp to a beginning of a
> tumbling window interval.
> {code}
> /**
> * Tumbling windows are a series of fixed-sized, non-overlapping and
> contiguous time intervals.
> * Tumbling windows are inclusive start exclusive end.
> * By default the beginning instant of fist window is Epoch 0 Thu Jan 01
> 00:00:00 1970 UTC.
> * Optionally users may provide a different origin as a timestamp arg3.
> *
> * This an example of series of window with an interval of 5 seconds and
> origin Epoch 0 Thu Jan 01 00:00:00 1970 UTC:
> *
> *
> * interval 1 interval 2interval 3
> * Jan 01 00:00:00 Jan 01 00:00:05 Jan 01 00:00:10
> * 0 -- 4 : 5 --- 9: 10 --- 14
> *
> * This UDF rounds timestamp agr1 to the beginning of window interval where
> it belongs to.
> *
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20768) Adding Tumbling Window UDF
[
https://issues.apache.org/jira/browse/HIVE-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra reassigned HIVE-20768:
-
> Adding Tumbling Window UDF
> --
>
> Key: HIVE-20768
> URL: https://issues.apache.org/jira/browse/HIVE-20768
> Project: Hive
> Issue Type: New Feature
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
>
> Goal is to provide a UDF that truncates a timestamp to a beginning of a
> tumbling window interval.
> {code}
> /**
> * Tumbling windows are a series of fixed-sized, non-overlapping and
> contiguous time intervals.
> * Tumbling windows are inclusive start exclusive end.
> * By default the beginning instant of fist window is Epoch 0 Thu Jan 01
> 00:00:00 1970 UTC.
> * Optionally users may provide a different origin as a timestamp arg3.
> *
> * This an example of series of window with an interval of 5 seconds and
> origin Epoch 0 Thu Jan 01 00:00:00 1970 UTC:
> *
> *
> * interval 1 interval 2interval 3
> * Jan 01 00:00:00 Jan 01 00:00:05 Jan 01 00:00:10
> * 0 -- 4 : 5 --- 9: 10 --- 14
> *
> * This UDF rounds timestamp agr1 to the beginning of window interval where
> it belongs to.
> *
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654395#comment-16654395
]
Hive QA commented on HIVE-20617:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944405/HIVE-20617.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15093 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14529/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14529/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14529/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944405 - PreCommit-HIVE-Build
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20767) Multiple project between join operators may affect join reordering using constraints
[ https://issues.apache.org/jira/browse/HIVE-20767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20767: --- Attachment: (was: HIVE-20767.patch) > Multiple project between join operators may affect join reordering using > constraints > > > Key: HIVE-20767 > URL: https://issues.apache.org/jira/browse/HIVE-20767 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20767.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20767) Multiple project between join operators may affect join reordering using constraints
[ https://issues.apache.org/jira/browse/HIVE-20767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20767: --- Attachment: HIVE-20767.patch > Multiple project between join operators may affect join reordering using > constraints > > > Key: HIVE-20767 > URL: https://issues.apache.org/jira/browse/HIVE-20767 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20767.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20748) Disable materialized view rewriting when plan pattern is not allowed
[ https://issues.apache.org/jira/browse/HIVE-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20748: --- Attachment: HIVE-20748.01.patch > Disable materialized view rewriting when plan pattern is not allowed > > > Key: HIVE-20748 > URL: https://issues.apache.org/jira/browse/HIVE-20748 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20748.01.patch, HIVE-20748.01.patch, > HIVE-20748.patch > > > For instance, currently rewriting algorithm does not support some operators. > Or we cannot have non-deterministic function in the MV definition. In those > cases, we should fail either when we try to create the MV with rewriting > enabled, or when when we enable the rewriting for a MV already created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654344#comment-16654344
]
Daniel Dai commented on HIVE-20549:
---
Attached the wrong patch. Thanks [~sershe]!
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20549.1.patch, HIVE-20549.2.patch,
> HIVE-20549.3.patch, HIVE-20549.4.patch, HIVE-20549.5.patch
>
>
> HIVE-19924 add capacity for replication job set a query tag and kill the
> replication distcp job with the tag. Here I make it more general, user can
> set arbitrary "hive.query.tag" in sql script, and kill query with the tag.
> Hive will cancel the corresponding operation in hs2, along with Tez/MR
> application launched for the query. For example:
> {code}
> set hive.query.tag=mytag;
> select . -- long running query
> {code}
> In another session:
> {code}
> kill query 'mytag';
> {code}
> There're limitations in the implementation:
> 1. No tag duplication check. There's nothing to prevent conflicting tag for
> same user, and kill query will kill queries share the same tag. However, kill
> query will not kill queries from different user unless admin. So different
> user might share the same tag
> 2. In multiple hs2 environment, kill statement should be issued to all hs2 to
> make sure the corresponding operation is canceled. When beeline/jdbc connects
> to hs2 using regular way (zookeeper url), the session will connect to random
> hs2, which might be different than the hs2 where query run on. User can use
> HiveConnection.getAllUrls or beeline --getUrlsFromBeelineSite (HIVE-20507) to
> get a list of all hs2 instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20767) Multiple project between join operators may affect join reordering using constraints
[ https://issues.apache.org/jira/browse/HIVE-20767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20767: --- Attachment: HIVE-20767.patch > Multiple project between join operators may affect join reordering using > constraints > > > Key: HIVE-20767 > URL: https://issues.apache.org/jira/browse/HIVE-20767 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20767.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654343#comment-16654343
]
Hive QA commented on HIVE-20617:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 5 new + 85 unchanged - 11
fixed = 90 total (was 96) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
47s{color} | {color:red} ql generated 4 new + 2314 unchanged - 4 fixed = 2318
total (was 2318) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Byte(String) constructor; use
Byte.valueOf(String) instead At TypeCheckProcFactory.java:new Byte(String)
constructor; use Byte.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1391] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Integer(String) constructor; use
Integer.valueOf(String) instead At TypeCheckProcFactory.java:new
Integer(String) constructor; use Integer.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1383] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Long(String) constructor; use
Long.valueOf(String) instead At TypeCheckProcFactory.java:new Long(String)
constructor; use Long.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1385] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
Object, TypeInfo) invokes inefficient new Short(String) constructor; use
Short.valueOf(String) instead At TypeCheckProcFactory.java:new Short(String)
constructor; use Short.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1393] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14529/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14529/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14529/yetus/whitespace-eol.txt
|
|
[jira] [Assigned] (HIVE-20767) Multiple project between join operators may affect join reordering using constraints
[ https://issues.apache.org/jira/browse/HIVE-20767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20767: -- > Multiple project between join operators may affect join reordering using > constraints > > > Key: HIVE-20767 > URL: https://issues.apache.org/jira/browse/HIVE-20767 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20767) Multiple project between join operators may affect join reordering using constraints
[ https://issues.apache.org/jira/browse/HIVE-20767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20767 started by Jesus Camacho Rodriguez. -- > Multiple project between join operators may affect join reordering using > constraints > > > Key: HIVE-20767 > URL: https://issues.apache.org/jira/browse/HIVE-20767 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20767) Multiple project between join operators may affect join reordering using constraints
[ https://issues.apache.org/jira/browse/HIVE-20767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20767: --- Status: Patch Available (was: In Progress) > Multiple project between join operators may affect join reordering using > constraints > > > Key: HIVE-20767 > URL: https://issues.apache.org/jira/browse/HIVE-20767 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20549:
--
Attachment: HIVE-20549.5.patch
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20549.1.patch, HIVE-20549.2.patch,
> HIVE-20549.3.patch, HIVE-20549.4.patch, HIVE-20549.5.patch
>
>
> HIVE-19924 add capacity for replication job set a query tag and kill the
> replication distcp job with the tag. Here I make it more general, user can
> set arbitrary "hive.query.tag" in sql script, and kill query with the tag.
> Hive will cancel the corresponding operation in hs2, along with Tez/MR
> application launched for the query. For example:
> {code}
> set hive.query.tag=mytag;
> select . -- long running query
> {code}
> In another session:
> {code}
> kill query 'mytag';
> {code}
> There're limitations in the implementation:
> 1. No tag duplication check. There's nothing to prevent conflicting tag for
> same user, and kill query will kill queries share the same tag. However, kill
> query will not kill queries from different user unless admin. So different
> user might share the same tag
> 2. In multiple hs2 environment, kill statement should be issued to all hs2 to
> make sure the corresponding operation is canceled. When beeline/jdbc connects
> to hs2 using regular way (zookeeper url), the session will connect to random
> hs2, which might be different than the hs2 where query run on. User can use
> HiveConnection.getAllUrls or beeline --getUrlsFromBeelineSite (HIVE-20507) to
> get a list of all hs2 instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18908: Fix Version/s: 4.0.0 > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0 > > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, > HIVE-18908.097.patch, HIVE-18908.098.patch, HIVE-18908.099.patch, > HIVE-18908.0991.patch, HIVE-18908.0992.patch, HIVE-18908.0993.patch, > HIVE-18908.0994.patch, HIVE-18908.0995.patch, HIVE-18908.0996.patch, > HIVE-18908.0997.patch, HIVE-18908.0998.patch, HIVE-18908.0999.patch, > HIVE-18908.09991.patch, HIVE-18908.09992.patch, HIVE-18908.09993.patch, > HIVE-18908.09994.patch, HIVE-18908.09995.patch, HIVE-18908.09996.patch, > HIVE-18908.09997.patch, HIVE-18908.09998.patch, JOIN to MAPJOIN > Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. > Rough TPC-DS timings run on laptop: > (NOTE: Query 51 has PTF as a bigger serial portion -- Amdahl's law at play) > FULL OUTER MapJoin OFF = MergeJoin > Query 51: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 4:30 minutes > • FULL OUTER MapJoin ON: 4:37 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 2:35 minutes > • FULL OUTER MapJoin ON: 1:47 minutes > Query 97: > o Vectorization OFF > • FULL OUTER MapJoin OFF: 2:37 minutes > • FULL OUTER MapJoin ON: 2:42 minutes > o Vectorization ON > • FULL OUTER MapJoin OFF: 1:17 minutes > • FULL OUTER MapJoin ON: 0:06 minutes > FULL OUTER Join 10,000,000 rows against 323,910 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 14:56 minutes > • FULL OUTER MapJoin ON: 1:45 minutes > FULL OUTER Join 10,000,000 rows against 1,000 small table keys > o Vectorization ON > • FULL OUTER MapJoin OFF: 12:37 minutes > • FULL OUTER MapJoin ON: 1:38 minutes > Hopefully, someone will do large scale cluster testing. > [DynamicPartitionedHashJoin] MapJoin should scale dramatically better than > [Sort] MergeJoin reduce-shuffle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654316#comment-16654316
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944301/HIVE-20679.6.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15198 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14528/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14528/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14528/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944301 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch,
> HIVE-20679.6.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654295#comment-16654295
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
24s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
7s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
40s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 40s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 34 new + 814
unchanged - 36 fixed = 848 total (was 850) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 5 fixed
= 41 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
35s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} |
[jira] [Updated] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[ https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-20737: -- Attachment: HIVE-20737.12.patch > Local SparkContext is shared between user sessions and should be closed only > when there is no active > > > Key: HIVE-20737 > URL: https://issues.apache.org/jira/browse/HIVE-20737 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch, > HIVE-20737.11.patch, HIVE-20737.12.patch, HIVE-20737.2.patch, > HIVE-20737.5.patch, HIVE-20737.6.patch, HIVE-20737.7.patch, > HIVE-20737.8.patch, HIVE-20737.9.patch > > > 1. Local SparkContext is shared between user sessions and should be closed > only when there is no active. > 2. Possible race condition in SparkSession.open() in case when user queries > run in parallel within the same session. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[ https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654283#comment-16654283 ] Denys Kuzmenko commented on HIVE-20737: --- depends on HIVE-20742 > Local SparkContext is shared between user sessions and should be closed only > when there is no active > > > Key: HIVE-20737 > URL: https://issues.apache.org/jira/browse/HIVE-20737 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch, > HIVE-20737.11.patch, HIVE-20737.2.patch, HIVE-20737.5.patch, > HIVE-20737.6.patch, HIVE-20737.7.patch, HIVE-20737.8.patch, HIVE-20737.9.patch > > > 1. Local SparkContext is shared between user sessions and should be closed > only when there is no active. > 2. Possible race condition in SparkSession.open() in case when user queries > run in parallel within the same session. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654282#comment-16654282 ] Sergey Shelukhin commented on HIVE-20763: - +1 > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654280#comment-16654280 ] Prasanth Jayachandran commented on HIVE-20703: -- long executorMem = 40L can be replaced by OperatorDesc.getMaxMemoryAvailable() which give max memory available per container (in case of tez) or executor in case of llap. > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20703) Put dynamic sort partition optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654271#comment-16654271 ] Sergey Shelukhin commented on HIVE-20703: - Can hive.tez.container.size be used for now? Perhaps adding an explicit default in this case if it's not set. > Put dynamic sort partition optimization under cost based decision > - > > Key: HIVE-20703 > URL: https://issues.apache.org/jira/browse/HIVE-20703 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20703.1.patch, HIVE-20703.2.patch, > HIVE-20703.3.patch, HIVE-20703.4.patch, HIVE-20703.5.patch, > HIVE-20703.6.patch, HIVE-20703.7.patch, HIVE-20703.8.patch, HIVE-20703.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18731) Add Documentations about this feature.
[ https://issues.apache.org/jira/browse/HIVE-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-18731: - Assignee: slim bouguerra > Add Documentations about this feature. > --- > > Key: HIVE-18731 > URL: https://issues.apache.org/jira/browse/HIVE-18731 > Project: Hive > Issue Type: Sub-task > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > need to add basic docs about new table properties and what it means in > practice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654257#comment-16654257 ] Sergey Shelukhin commented on HIVE-20765: - We can just get full partitions by name on branch-3 > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[
https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654252#comment-16654252
]
Ashutosh Chauhan commented on HIVE-20765:
-
Patch looks good. +1 I believe {{getPartitionSpecsByFilterAndProjection}} is
not available on branch-3 and backport on branch-3 will be useful.
> fetch partitions for txn stats validation in get_aggr_stats with one call
> -
>
> Key: HIVE-20765
> URL: https://issues.apache.org/jira/browse/HIVE-20765
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20765.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17296) Acid tests with multiple splits
[ https://issues.apache.org/jira/browse/HIVE-17296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654224#comment-16654224 ] Eugene Koifman commented on HIVE-17296: --- see HIVE-20694 and TestVectorizedOrcAcidRowBatchReader > Acid tests with multiple splits > --- > > Key: HIVE-17296 > URL: https://issues.apache.org/jira/browse/HIVE-17296 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > data files in an Acid table are ORC files which may have multiple stripes > for such files in base/ or delta/ (and original files with non acid to acid > conversion) are split by OrcInputFormat into multiple (stripe sized) chunks. > There is additional logic in in OrcRawRecordMerger > (discoverKeyBounds/discoverOriginalKeyBounds) that is not tested by any E2E > tests since none of the have enough data to generate multiple stripes in a > single file. > testRecordReaderOldBaseAndDelta/testRecordReaderNewBaseAndDelta/testOriginalReaderPair > in TestOrcRawRecordMerger has some logic to test this but it really needs e2e > tests. > With ORC-228 it will be possible to write such tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17296) Acid tests with multiple splits
[ https://issues.apache.org/jira/browse/HIVE-17296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17296: -- Priority: Major (was: Blocker) > Acid tests with multiple splits > --- > > Key: HIVE-17296 > URL: https://issues.apache.org/jira/browse/HIVE-17296 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > data files in an Acid table are ORC files which may have multiple stripes > for such files in base/ or delta/ (and original files with non acid to acid > conversion) are split by OrcInputFormat into multiple (stripe sized) chunks. > There is additional logic in in OrcRawRecordMerger > (discoverKeyBounds/discoverOriginalKeyBounds) that is not tested by any E2E > tests since none of the have enough data to generate multiple stripes in a > single file. > testRecordReaderOldBaseAndDelta/testRecordReaderNewBaseAndDelta/testOriginalReaderPair > in TestOrcRawRecordMerger has some logic to test this but it really needs e2e > tests. > With ORC-228 it will be possible to write such tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[
https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654220#comment-16654220
]
Hive QA commented on HIVE-20737:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944286/HIVE-20737.11.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15097 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSessionSparkSessionTimeout
(batchId=249)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14527/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14527/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14527/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944286 - PreCommit-HIVE-Build
> Local SparkContext is shared between user sessions and should be closed only
> when there is no active
>
>
> Key: HIVE-20737
> URL: https://issues.apache.org/jira/browse/HIVE-20737
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch,
> HIVE-20737.11.patch, HIVE-20737.2.patch, HIVE-20737.5.patch,
> HIVE-20737.6.patch, HIVE-20737.7.patch, HIVE-20737.8.patch, HIVE-20737.9.patch
>
>
> 1. Local SparkContext is shared between user sessions and should be closed
> only when there is no active.
> 2. Possible race condition in SparkSession.open() in case when user queries
> run in parallel within the same session.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for txn stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Summary: fetch partitions for txn stats validation in get_aggr_stats with one call (was: fetch partitions for stats validation in get_aggr_stats with one call) > fetch partitions for txn stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: (was: HIVE-20765.patch) > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: HIVE-20765.patch > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654182#comment-16654182 ] Sergey Shelukhin commented on HIVE-20765: - [~ashutoshc] can you take a look? > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Status: Patch Available (was: Open) Small patch... trying to use the new projection mechanism for now. > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Attachment: HIVE-20765.patch > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20765) fetch partitions for stats validation in get_aggr_stats with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20765: Summary: fetch partitions for stats validation in get_aggr_stats with one call (was: validate txn stats for partitions with one call) > fetch partitions for stats validation in get_aggr_stats with one call > - > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20765.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20737) Local SparkContext is shared between user sessions and should be closed only when there is no active
[
https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654166#comment-16654166
]
Hive QA commented on HIVE-20737:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 1
fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14527/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14527/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Local SparkContext is shared between user sessions and should be closed only
> when there is no active
>
>
> Key: HIVE-20737
> URL: https://issues.apache.org/jira/browse/HIVE-20737
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch,
> HIVE-20737.11.patch, HIVE-20737.2.patch, HIVE-20737.5.patch,
> HIVE-20737.6.patch, HIVE-20737.7.patch, HIVE-20737.8.patch, HIVE-20737.9.patch
>
>
> 1. Local SparkContext is shared between user sessions and should be closed
> only when there is no active.
> 2. Possible race condition in SparkSession.open() in case when user queries
> run in parallel within the same session.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20765) validate txn stats for partitions with one call
[ https://issues.apache.org/jira/browse/HIVE-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20765: --- > validate txn stats for partitions with one call > --- > > Key: HIVE-20765 > URL: https://issues.apache.org/jira/browse/HIVE-20765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20751) Upgrade arrow version to 0.10.0
[
https://issues.apache.org/jira/browse/HIVE-20751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654142#comment-16654142
]
Hive QA commented on HIVE-20751:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944280/HIVE-20751.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15096 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14526/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14526/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14526/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944280 - PreCommit-HIVE-Build
> Upgrade arrow version to 0.10.0
> ---
>
> Key: HIVE-20751
> URL: https://issues.apache.org/jira/browse/HIVE-20751
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20751.1.patch
>
>
> Need to upgrade arrow version as spark is moving to arrow version 0.10.0 in
> it's upcoming release 2.4.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20751) Upgrade arrow version to 0.10.0
[
https://issues.apache.org/jira/browse/HIVE-20751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654135#comment-16654135
]
Hive QA commented on HIVE-20751:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 4 new + 318 unchanged - 4
fixed = 322 total (was 322) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14526/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14526/yetus/diff-checkstyle-ql.txt
|
| modules | C: . ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14526/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade arrow version to 0.10.0
> ---
>
> Key: HIVE-20751
> URL: https://issues.apache.org/jira/browse/HIVE-20751
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20751.1.patch
>
>
> Need to upgrade arrow version as spark is moving to arrow version 0.10.0 in
> it's upcoming release 2.4.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.08.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20697) Some replication tests are super slow and cause batch timeouts
[
https://issues.apache.org/jira/browse/HIVE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654106#comment-16654106
]
Vihang Karajgaonkar commented on HIVE-20697:
{noformat}
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=243_TestReplicationScenarios, id=243,
moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenarios]]
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=245_TestReplicationScenariosAcrossInstances, id=245,
moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenariosAcrossInstances]]
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=246_TestReplicationScenariosAcidTables, id=246,
moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenariosAcidTables]]
2018-10-17 19:16:04,963 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch:
UnitTestBatch [name=247_TestReplicationScenariosIncrementalLoadAcidTables,
id=247, moduleName=itests/hive-unit, batchSize=1, isParallel=true,
testList=[TestReplicationScenariosIncrementalLoadAcidTables]]
{noformat}
These tests are not batched anymore. They run in batches of 1 test each.
> Some replication tests are super slow and cause batch timeouts
> --
>
> Key: HIVE-20697
> URL: https://issues.apache.org/jira/browse/HIVE-20697
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Some of these tests are taking a long time and can cause test batch timeouts
> given that we only give 40 min for a batch to complete. We should speed these
> tests up.
> TestReplicationScenarios 20 min
> TestReplicationScenariosAcidTables11 min
> TestReplicationScenariosAcrossInstances 5 min 14 sec
> TestReplicationScenariosIncrementalLoadAcidTables 20 min
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654101#comment-16654101
]
Sergey Shelukhin commented on HIVE-20549:
-
+1 pending tests for 03 patch :) or its correct update
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20549.1.patch, HIVE-20549.2.patch,
> HIVE-20549.3.patch, HIVE-20549.4.patch
>
>
> HIVE-19924 add capacity for replication job set a query tag and kill the
> replication distcp job with the tag. Here I make it more general, user can
> set arbitrary "hive.query.tag" in sql script, and kill query with the tag.
> Hive will cancel the corresponding operation in hs2, along with Tez/MR
> application launched for the query. For example:
> {code}
> set hive.query.tag=mytag;
> select . -- long running query
> {code}
> In another session:
> {code}
> kill query 'mytag';
> {code}
> There're limitations in the implementation:
> 1. No tag duplication check. There's nothing to prevent conflicting tag for
> same user, and kill query will kill queries share the same tag. However, kill
> query will not kill queries from different user unless admin. So different
> user might share the same tag
> 2. In multiple hs2 environment, kill statement should be issued to all hs2 to
> make sure the corresponding operation is canceled. When beeline/jdbc connects
> to hs2 using regular way (zookeeper url), the session will connect to random
> hs2, which might be different than the hs2 where query run on. User can use
> HiveConnection.getAllUrls or beeline --getUrlsFromBeelineSite (HIVE-20507) to
> get a list of all hs2 instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654098#comment-16654098
]
Sergey Shelukhin commented on HIVE-20549:
-
The 04 patch is wrong, it just comments out some stuff in
TestTriggersTezSessionPoolManager
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20549.1.patch, HIVE-20549.2.patch,
> HIVE-20549.3.patch, HIVE-20549.4.patch
>
>
> HIVE-19924 add capacity for replication job set a query tag and kill the
> replication distcp job with the tag. Here I make it more general, user can
> set arbitrary "hive.query.tag" in sql script, and kill query with the tag.
> Hive will cancel the corresponding operation in hs2, along with Tez/MR
> application launched for the query. For example:
> {code}
> set hive.query.tag=mytag;
> select . -- long running query
> {code}
> In another session:
> {code}
> kill query 'mytag';
> {code}
> There're limitations in the implementation:
> 1. No tag duplication check. There's nothing to prevent conflicting tag for
> same user, and kill query will kill queries share the same tag. However, kill
> query will not kill queries from different user unless admin. So different
> user might share the same tag
> 2. In multiple hs2 environment, kill statement should be issued to all hs2 to
> make sure the corresponding operation is canceled. When beeline/jdbc connects
> to hs2 using regular way (zookeeper url), the session will connect to random
> hs2, which might be different than the hs2 where query run on. User can use
> HiveConnection.getAllUrls or beeline --getUrlsFromBeelineSite (HIVE-20507) to
> get a list of all hs2 instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654093#comment-16654093 ] Prasanth Jayachandran commented on HIVE-20763: -- +1 > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20549) Allow user set query tag, and kill query with tag
[
https://issues.apache.org/jira/browse/HIVE-20549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654090#comment-16654090
]
Thejas M Nair commented on HIVE-20549:
--
+1 pending tests
> Allow user set query tag, and kill query with tag
> -
>
> Key: HIVE-20549
> URL: https://issues.apache.org/jira/browse/HIVE-20549
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20549.1.patch, HIVE-20549.2.patch,
> HIVE-20549.3.patch, HIVE-20549.4.patch
>
>
> HIVE-19924 add capacity for replication job set a query tag and kill the
> replication distcp job with the tag. Here I make it more general, user can
> set arbitrary "hive.query.tag" in sql script, and kill query with the tag.
> Hive will cancel the corresponding operation in hs2, along with Tez/MR
> application launched for the query. For example:
> {code}
> set hive.query.tag=mytag;
> select . -- long running query
> {code}
> In another session:
> {code}
> kill query 'mytag';
> {code}
> There're limitations in the implementation:
> 1. No tag duplication check. There's nothing to prevent conflicting tag for
> same user, and kill query will kill queries share the same tag. However, kill
> query will not kill queries from different user unless admin. So different
> user might share the same tag
> 2. In multiple hs2 environment, kill statement should be issued to all hs2 to
> make sure the corresponding operation is canceled. When beeline/jdbc connects
> to hs2 using regular way (zookeeper url), the session will connect to random
> hs2, which might be different than the hs2 where query run on. User can use
> HiveConnection.getAllUrls or beeline --getUrlsFromBeelineSite (HIVE-20507) to
> get a list of all hs2 instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20762) NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small.
[ https://issues.apache.org/jira/browse/HIVE-20762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654088#comment-16654088 ] Thejas M Nair commented on HIVE-20762: -- +1 > NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small. > --- > > Key: HIVE-20762 > URL: https://issues.apache.org/jira/browse/HIVE-20762 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: Metastore > Attachments: HIVE-20762.01.patch > > > NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small. It > should be set to several hours or else the number of metastore calls would be > too high and impact other operations. > Make it configurable item and set it as 2Hrs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-20763: -- Fix Version/s: 4.0.0 > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654083#comment-16654083 ] Siddharth Seth commented on HIVE-20763: --- Uploaded a trivial patch. No explicit tests needed for this. [~sershe], [~prasanth_j] - could you please take a look. > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned HIVE-20763: - > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist
[ https://issues.apache.org/jira/browse/HIVE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-20763: -- Attachment: HIVE-20763.01.patch > Add google cloud storage (gs) to the exim uri schema whitelist > -- > > Key: HIVE-20763 > URL: https://issues.apache.org/jira/browse/HIVE-20763 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Attachments: HIVE-20763.01.patch > > > import/export is enabled for s3a by default. Ideally this list should include > other cloud storage options. This Jira adds Google Storage to the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20707) Automatic partition management
[ https://issues.apache.org/jira/browse/HIVE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654066#comment-16654066 ] Prasanth Jayachandran commented on HIVE-20707: -- Only msck_repair_drop.q seems to be relevant. Added sorting to stabilize the output. > Automatic partition management > -- > > Key: HIVE-20707 > URL: https://issues.apache.org/jira/browse/HIVE-20707 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20702.3.patch, HIVE-20707.1.patch, > HIVE-20707.2.patch, HIVE-20707.4.patch, HIVE-20707.5.patch > > > In current scenario, to add partitions for external tables to metastore, MSCK > REPAIR command has to be executed manually. To avoid this manual step, > external tables can be specified a table property based on which a background > metastore thread can sync partitions periodically. Tables can also be > specified with partition retention period. Any partition whose age exceeds > the retention period will be dropped automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20707) Automatic partition management
[ https://issues.apache.org/jira/browse/HIVE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20707: - Attachment: HIVE-20707.5.patch > Automatic partition management > -- > > Key: HIVE-20707 > URL: https://issues.apache.org/jira/browse/HIVE-20707 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20702.3.patch, HIVE-20707.1.patch, > HIVE-20707.2.patch, HIVE-20707.4.patch, HIVE-20707.5.patch > > > In current scenario, to add partitions for external tables to metastore, MSCK > REPAIR command has to be executed manually. To avoid this manual step, > external tables can be specified a table property based on which a background > metastore thread can sync partitions periodically. Tables can also be > specified with partition retention period. Any partition whose age exceeds > the retention period will be dropped automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654063#comment-16654063
]
Hive QA commented on HIVE-20617:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944307/HIVE-20617.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15084 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14525/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14525/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14525/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944307 - PreCommit-HIVE-Build
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20762) NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small.
[ https://issues.apache.org/jira/browse/HIVE-20762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20762: Status: Patch Available (was: Open) [~thejas] Can you please review the patch? > NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small. > --- > > Key: HIVE-20762 > URL: https://issues.apache.org/jira/browse/HIVE-20762 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: Metastore > Attachments: HIVE-20762.01.patch > > > NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small. It > should be set to several hours or else the number of metastore calls would be > too high and impact other operations. > Make it configurable item and set it as 2Hrs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20762) NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small.
[ https://issues.apache.org/jira/browse/HIVE-20762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20762: Attachment: HIVE-20762.01.patch > NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small. > --- > > Key: HIVE-20762 > URL: https://issues.apache.org/jira/browse/HIVE-20762 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: Metastore > Attachments: HIVE-20762.01.patch > > > NOTIFICATION_LOG cleanup interval is hardcoded as 60s and is too small. It > should be set to several hours or else the number of metastore calls would be > too high and impact other operations. > Make it configurable item and set it as 2Hrs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20756: --- Assignee: Sergey Shelukhin (was: Chiran Ravani) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, > HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20756: --- Assignee: Chiran Ravani (was: Sergey Shelukhin) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Chiran Ravani >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, > HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20756: Attachment: HIVE-20756.01-branch-1.patch > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, > HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20756: Attachment: HIVE-20756.01-branch-1.patch > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, > HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20756) Disable SARG leaf creation for date column until ORC-135
[ https://issues.apache.org/jira/browse/HIVE-20756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20756: Attachment: (was: HIVE-20756.01-branch-1.patch) > Disable SARG leaf creation for date column until ORC-135 > > > Key: HIVE-20756 > URL: https://issues.apache.org/jira/browse/HIVE-20756 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Chiran Ravani >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20756.01-branch-1.patch, > HIVE-20756.01-branch-1.patch, HIVE-20756.1.patch > > > Until ORC-135 is committed and orc version is updated in hive, disable SARG > creation for date columns in hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20718) Add perf cli driver with constraints
[ https://issues.apache.org/jira/browse/HIVE-20718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654035#comment-16654035 ] Jesus Camacho Rodriguez commented on HIVE-20718: Rebased patch. [~ashutoshc], can you take a look? Thanks https://reviews.apache.org/r/69017/ > Add perf cli driver with constraints > > > Key: HIVE-20718 > URL: https://issues.apache.org/jira/browse/HIVE-20718 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20718.01.patch, HIVE-20718.02.patch, > HIVE-20718.03.patch, HIVE-20718.04.patch, HIVE-20718.05.patch, > HIVE-20718.06.patch, HIVE-20718.07.patch, HIVE-20718.08.patch, > HIVE-20718.patch > > > Now that subtasks in HIVE-17039 will be completed, it will be good to have a > perf cli driver with constraints declaration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
