[jira] [Created] (HIVE-26945) Test fixes for query*.q files

[Aman Raj (Jira)]

Aman Raj created HIVE-26945:
---

 Summary: Test fixes for query*.q files
 Key: HIVE-26945
 URL: https://issues.apache.org/jira/browse/HIVE-26945
 Project: Hive
  Issue Type: Sub-task
Reporter: Aman Raj
Assignee: Aman Raj


The tests has outdated q.out files which need to be updated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26944) FileSinkOperator shouldn't check for compactiontable for every row being processed

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26944:
---

 Summary: FileSinkOperator shouldn't check for compactiontable for 
every row being processed
 Key: HIVE-26944
 URL: https://issues.apache.org/jira/browse/HIVE-26944
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2023-01-16 at 10.32.24 AM.png





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline

[Aman Raj]

Hi team,

Need help in a test case failing in branch-3. As you know we have been trying 
to ensure a Green branch-3. When we had started this task the total number of 
failures was 2900+. We have brought it down to less than 20 as of now. JIRA 
link to track the status : https://issues.apache.org/jira/browse/HIVE-26836

I am facing some issues in fixing the following tests :
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver

Issue :
PREHOOK: Input: default@src
PREHOOK: Output: default@src
Failed to monitor Job[-1] with exception 
'java.lang.IllegalStateException(Connection to remote Spark driver was lost)' 
Last known state = SENT
Failed to execute spark task, with exception 
'java.lang.IllegalStateException(RPC channel is closed.)'
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is closed.

History :
Initially the tests had failed with errors which I fixed in the following task 
: https://issues.apache.org/jira/browse/HIVE-26940

Does anyone know what the issue is here ? There are 6-7 failures because of 
this test case. Link to the failed test cases for the stacktrace : 
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3949/2/tests/
Thanks,
Aman.


From: Aman Raj 
Sent: Friday, December 9, 2022 9:33 AM
To: dev@hive.apache.org 
Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline

Thanks Pravin for your support. Can someone please help me merge this PR to 
branch-3 HIVE-26816 : Add Jenkins file for branch-3 by amanraj2520 · Pull 
Request #3841 · apache/hive 
(github.com).
 I do not have access to do that. Then we will start development on it.

Thanks,
Aman.



From: Pravin Sinha 
Sent: Friday, December 9, 2022 1:55 AM
To: dev@hive.apache.org 
Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0 pipeline

[You don't often get email from mailpravi...@gmail.com. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

Hi Aman,
 I also think that we can merge the PR to enable the test pipeline if the
change looks fine and subsequently we can fix the tests to bring it to
green state (hopefully by cherry picking a few commits from branch-3.1
which is already in green state) . Looks like currently the tests are
broken in branch-3.

Thanks,
Pravin

On Thu, Dec 8, 2022 at 3:59 PM Aman Raj 
wrote:

> Hi team,
>
> For the addition of Jenkins file for branch-3, branch-3 has some existing
> tests failing which was because Jenkins was not running on branch-3. We are
> planning to merge this Jenkins file irrespective of this PR having test
> failures, since this does not change the code. We will create separate
> tasks for ensuring that branch-3 has a green build.
>
> Link to the PR : 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F3841&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D7CwaRcRaQ5ubjz3Ki95HkyclN2a%2BBZ7lvTddDQpTLY%3D&reserved=0
>
> Fyi, branch-3.1 has a green build.
>
> Thanks,
> Aman.
> 
> From: Aman Raj 
> Sent: Wednesday, December 7, 2022 3:19 PM
> To: dev@hive.apache.org 
> Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0
> pipeline
>
> Hi Ayush,
>
> Thanks for clarifying. Will wait for it to turn green.
>
> Thanks,
> Aman.
> 
> From: Ayush Saxena 
> Sent: Wednesday, December 7, 2022 3:11 PM
> To: dev@hive.apache.org 
> Subject: Re: [EXTERNAL] Re: Sync of Branch-3 & Branch-3.1 for 3.2.0
> pipeline
>
> Hi Aman,
> The build is already running for your PR:
>
> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fci.hive.apache.org%2Fblue%2Forganizations%2Fjenkins%2Fhive-precommit%2Fdetail%2FPR-3841%2F1%2Fpipeline&data=05%7C01%7Crajaman%40microsoft.com%7C94c1ac2c4ddd40437b5f08dad99a6017%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638061554335365489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=osIglpTld3PtOPyFhGBLTSU9Ku1FWngPMofNQXILpyM%3D&reserved=0
>
> The JenkinsFile is picked from the PR while running rather than the target
> branch.
>
> -Ayush
>
> > On 07-Dec-2022, at 3:03 PM, Aman Raj 
> wrote:
> >
> > Hi Stamatis,
> >
> > How can we ensure that unless the P

Re: [ANNOUNCE] New PMC Member: Stamatis Zampetakis

[Krisztian Kasa]

Congratulations Stamatis :)

On Mon, Jan 16, 2023 at 6:27 AM S T  wrote:

> Congrats Stamatis.
>
> Thanks
>
> On Sat, 14 Jan 2023 at 00:03, Naveen Gangam  wrote:
>
>> Hello Hive Community,
>> Apache Hive PMC is pleased to announce that Stamatis Zampetakis has
>> accepted the Apache Hive PMC's invitation to become PMC Member, and is now
>> our newest PMC member. Please join me in congratulating Stamatis !!!
>>
>> He has been an active member in the hive community across many aspects of
>> the project. Many thanks to Stamatis for all the contributions he has made
>> and looking forward to many more future contributions in the expanded role.
>>
>> Cheers,
>> Naveen (on behalf of Hive PMC)
>>
>

[jira] [Created] (HIVE-26943) Fix NPE during Optimised Bootstrap when db is dropped

[Shreenidhi (Jira)]

Shreenidhi created HIVE-26943:
-

 Summary: Fix NPE during Optimised Bootstrap when db is dropped
 Key: HIVE-26943
 URL: https://issues.apache.org/jira/browse/HIVE-26943
 Project: Hive
  Issue Type: Task
Reporter: Shreenidhi


Consider the steps:

1. Current replication is from A (source) -> B(target)
2. Failover is complete
so now           A (target) <- B(source)
3. Suppose db at A is dropped before reverse replication.
4. Now when reverse replication triggers optimised bootstrap it will throw NPE

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26942) Prevent updation of "repl.target.last.id" when RESUME action initiated

[Shreenidhi (Jira)]

Shreenidhi created HIVE-26942:
-

 Summary: Prevent updation of "repl.target.last.id" when RESUME 
action initiated
 Key: HIVE-26942
 URL: https://issues.apache.org/jira/browse/HIVE-26942
 Project: Hive
  Issue Type: Task
Reporter: Shreenidhi
Assignee: Shreenidhi


*Problem :* 

Currently when "repl.last.id" get's changed/updated this updates the 
corresponding "repl.target.last.id" too. This is problematic incase of resume 
because we are updating both the properties at source.

 

*Solution :*

To prevent the updation of "repl.target.last.id" whenever "repl.last.id" 
changed, we can introduce new db param called "repl.resume.started". This new 
param will be set to "true" by orchestrator when RESUME is initiated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26941) Make SetProcessor configurable to ignore some set variables

[Miklos Szurap (Jira)]

Miklos Szurap created HIVE-26941:


 Summary: Make SetProcessor configurable to ignore some set 
variables
 Key: HIVE-26941
 URL: https://issues.apache.org/jira/browse/HIVE-26941
 Project: Hive
  Issue Type: New Feature
  Components: Configuration, Hive
Reporter: Miklos Szurap
Assignee: Miklos Szurap


In certain environments after upgrades we need to restrict users from changing 
some Hive configurations during runtime (for example "mapreduce.job.queuename" 
or "hive.execution.engine"). 
The "hive.security.authorization.sqlstd.confwhitelist" could  be used for this, 
however:
* it is complex to modify that sometimes (for example to exclude a config which 
is otherwise allowed with a wildcard) 
* when a user script tries to set a parameter not in the 
"hive.security.authorization.sqlstd.confwhitelist" then the whole script just 
fails with "Error: Error while processing statement: Cannot modify  at 
runtime. It is not in list of params that are allowed to be modified at 
runtime". This would require all the user scripts and jobs to be modified (to 
remove that "set" command), that can be a huge effort.

With a new configuration item in hive-site.xml cluster operators can configure 
HiveServer2 to ignore the "set" command requests - essentially making those 
settings "final" on HiveServer2 level. Trying to change these "final" settings 
would not fail the scripts - just ignore their request.

In this jira:
- add a new config "hive.conf.ignored.variable.list"
- accepts strings, comma separated list of variables
- the config is empty by default, it can be set in hive-site.xml only
- adding the "hive.conf.ignored.variable.list" to the restricted list 
("hive.conf.restricted.list") internally - so it cannot be modified during 
runtime
- adding tests for the changes



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26940) Backport of HIVE-19882 : Fix QTestUtil session lifecycle

[Aman Raj (Jira)]

Aman Raj created HIVE-26940:
---

 Summary: Backport of HIVE-19882 : Fix QTestUtil session lifecycle
 Key: HIVE-26940
 URL: https://issues.apache.org/jira/browse/HIVE-26940
 Project: Hive
  Issue Type: Sub-task
Reporter: Aman Raj
Assignee: Aman Raj


A lot of tests on branch-3 are failing with the following errors :
{code:java}
java.lang.AssertionError: Failed during createSources processLine with code=1   
at org.junit.Assert.fail(Assert.java:88)at 
org.apache.hadoop.hive.ql.QTestUtil.initFromScript(QTestUtil.java:1219)  at 
org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:1201)   at 
org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:1188)   at 
org.apache.hadoop.hive.cli.control.CoreCliDriver$3.invokeInternal(CoreCliDriver.java:83)
 at 
org.apache.hadoop.hive.cli.control.CoreCliDriver$3.invokeInternal(CoreCliDriver.java:80)
 at 
org.apache.hadoop.hive.util.ElapsedTimeLoggingWrapper.invoke(ElapsedTimeLoggingWrapper.java:33)
  at 
org.apache.hadoop.hive.cli.control.CoreCliDriver.beforeClass(CoreCliDriver.java:86)
  at 
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:71)  
 at org.junit.rules.RunRules.evaluate(RunRules.java:20)  at 
org.junit.runners.ParentRunner.run(ParentRunner.java:309)at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
  at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) 
 at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
 at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
  at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) {code}
This is because hive.test.shutdown.phase is not set to true before the cleanup 
happens thereby invoking EnforceReadOnlyTables which prevents it from dropping 
tables while running tests. Below is the stack trace :
{code:java}
FAILED: Hive Internal Error: 
org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking PreHook. 
hooks: java.lang.RuntimeException: Cannot overwrite read-only table: srcat 
org.apache.hadoop.hive.ql.hooks.EnforceReadOnlyTables.run(EnforceReadOnlyTables.java:64)
 at 
org.apache.hadoop.hive.ql.hooks.EnforceReadOnlyTables.run(EnforceReadOnlyTables.java:44)
 at 
org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)  at 
org.apache.hadoop.hive.ql.HookRunner.runPreHooks(HookRunner.java:273)at 
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2304)   at 
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2039)   at 
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1737)   at 
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1731)   at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) at 
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)  at 
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) at 
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) at 
org.apache.hadoop.hive.ql.QTestUtil.cleanupFromFile(QTestUtil.java:1155) at 
org.apache.hadoop.hive.ql.QTestUtil.cleanUp(QTestUtil.java:1128) at 
org.apache.hadoop.hive.ql.QTestUtil.cleanUp(QTestUtil.java:1112) at 
org.apache.hadoop.hive.cli.control.CoreCliDriver$2.invokeInternal(CoreCliDriver.java:75)
 at 
org.apache.hadoop.hive.cli.control.CoreCliDriver$2.invokeInternal(CoreCliDriver.java:72)
 at 
org.apache.hadoop.hive.util.ElapsedTimeLoggingWrapper.invoke(ElapsedTimeLoggingWrapper.java:33)
  at 
org.apache.hadoop.hive.cli.control.CoreCliDriver.beforeClass(CoreCliDriver.java:78)
  at 
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:71)  
 at org.junit.rules.RunRules.evaluate(RunRules.java:20)  at 
org.junit.runners.ParentRunner.run(ParentRunner.java:309)at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
  at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) 
 at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
 at 
org.apache.mave