Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's and Bugs on apache hive branch-3

[Aman Raj]

Hi everyone,

I have started categorizing tasks into 4 categories (created 4 subtasks on the 
umbrella JIRA for the same):

  1.  CVE fixes
  2.  Component Upgrades
  3.  Bug fixes and Improvements on top of branch-3 commits.
  4.  Differences between branch-3 and Hive-3.1.3 commits (The is no task 
involved in this as of now. Just a way to track what all commits went in after 
3.1.3 in branch-3)

I have created a new label (release-3.2.0) which can be used to create subtasks 
involving the backports of the JIRAs mentioned in these 4 Subtasks.

I would welcome the community to add more JIRAs which match these categories 
and update the JIRA page as well.

The parent JIRA fyi : https://issues.apache.org/jira/browse/HIVE-26748

Thanks,
Aman.

From: Aman Raj 
Sent: Thursday, November 17, 2022 12:09 PM
To: dev@hive.apache.org 
Subject: Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's and 
Bugs on apache hive branch-3

[You don't often get email from raja...@microsoft.com.invalid. Learn why this 
is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi everyone,

I thank everyone who upvoted for this release and I am sure we will make it a 
success. As a start point, I have created an umbrella JIRA [HIVE-26748] Prepare 
for Hive 3.2.0 Release - ASF JIRA 
(apache.org)
 where I will start adding the JIRAs that will be cherry picked as part of the 
3.2.0 release. I have also included the suggestions given by the community till 
now in the email threads below.

Please feel free to suggest any new and important bug fixes or features that 
can be included as part of this release.

Thanks,
Aman.

From: Naveen Gangam 
Sent: Tuesday, November 8, 2022 7:49 PM
To: dev@hive.apache.org 
Subject: Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's and 
Bugs on apache hive branch-3

[You don't often get email from ngan...@cloudera.com.invalid. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

Thank you Aman for volunteering to drive this. +1 for a release off
branch-3. We can fix all the CVEs we have fixed on master.

IMHO, the hadoop upgrade might be too big a task for this release. Last I
checked, there were some pending items from this upgrade even on master.
They may not be hard dependencies but if we are committing to this, might
take a bit longer to finish the release.

I started to build this Jira Board for the releases. The goal was to use
this to track release items (for all releases) via the use of jira
labels/target versions.
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FRapidBoard.jspa%3FrapidView%3D564data=05%7C01%7Crajaman%40microsoft.com%7C0852325809f74d4baaab08dac8667b44%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638042639758407506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=%2BLGYT%2B4KuHeWjP433lEOKRG87nP%2B%2F5ottrK55DroGwc%3Dreserved=0
At the top of this board, there are some quick filters for release blockers
(jiras with labels "hive-4.0.0-must"). There are currently only 2 jiras
tagged as blockers for 4.0.

If you could tag the jiras for 3.2 release the same way, and add a quick
filter, that would be great.

Thank you again
Naveen

On Fri, Nov 4, 2022 at 7:01 AM Stamatis Zampetakis 
wrote:

> Hey everyone,
>
> It would be nice to have a new release from branch 3 although it might not
> be that trivial to get out.
>
> It will definitely require a bit of investment from multiple people
> including the PMC and the committers of the project. Note that the last
> vote for alpha2 was unsuccessful due to lack of votes, which shows that
> people are pretty busy.
>
> Personally, I support this effort and would like to see it happen but this
> period I don't have sufficient time to invest to help with reviews and
> commits for 3.X line.
>
> Best,
> Stamatis
>
> On Fri, Nov 4, 2022, 5:28 AM Aman Raj 
> wrote:
>
> > Hi Chris,
> >
> > I plan on going through this diff and making a comprehensive list of all
> > the major bug fixes that went into branch-3 and not in hive-313. This
> will
> > be included in the umbrella JIRA that I am creating.
> >
> > In this email thread I have only mentioned CVEs and upgrades that will go
> > on top of these changes in branch-3.
> > Thanks,
> > Aman.
> >
> > 
> > From: Chris Nauroth 
> > Sent: Friday, November 4, 2022 3:44 AM
> > To: dev@hive.apache.org 
> > Subject: Re: [EXTERNAL] Re: 

[jira] [Created] (HIVE-26752) Difference between branch-3 and Hive-3.1.3

[Aman Raj (Jira)]

Aman Raj created HIVE-26752:
---

 Summary: Difference between branch-3 and Hive-3.1.3
 Key: HIVE-26752
 URL: https://issues.apache.org/jira/browse/HIVE-26752
 Project: Hive
  Issue Type: Sub-task
Reporter: Aman Raj
Assignee: Aman Raj
 Fix For: 3.2.0


Created a subtask to track the difference between current branch-3 and 
Hive-3.1.3 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26751) Bug Fixes and Improvements for 3.2.0 release

[Aman Raj (Jira)]

Aman Raj created HIVE-26751:
---

 Summary: Bug Fixes and Improvements for 3.2.0 release
 Key: HIVE-26751
 URL: https://issues.apache.org/jira/browse/HIVE-26751
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Aman Raj
Assignee: Aman Raj


Creating subtask to track the bug fixes that will go as part of 3.2.0 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26750) Component Upgrades for 3.2.0 release

[Aman Raj (Jira)]

Aman Raj created HIVE-26750:
---

 Summary: Component Upgrades for 3.2.0 release
 Key: HIVE-26750
 URL: https://issues.apache.org/jira/browse/HIVE-26750
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Aman Raj
Assignee: Aman Raj


Creating subtask for upgrading major component versions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26749) CVE Upgrades for 3.2.0 release

[Aman Raj (Jira)]

Aman Raj created HIVE-26749:
---

 Summary: CVE Upgrades for 3.2.0 release
 Key: HIVE-26749
 URL: https://issues.apache.org/jira/browse/HIVE-26749
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Aman Raj
Assignee: Aman Raj


Creating subtask for upgrading vulnerabilities in the 3.2.0 release



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's and Bugs on apache hive branch-3

[Aman Raj]

Hi everyone,

I thank everyone who upvoted for this release and I am sure we will make it a 
success. As a start point, I have created an umbrella JIRA [HIVE-26748] Prepare 
for Hive 3.2.0 Release - ASF JIRA 
(apache.org) where I will 
start adding the JIRAs that will be cherry picked as part of the 3.2.0 release. 
I have also included the suggestions given by the community till now in the 
email threads below.

Please feel free to suggest any new and important bug fixes or features that 
can be included as part of this release.

Thanks,
Aman.

From: Naveen Gangam 
Sent: Tuesday, November 8, 2022 7:49 PM
To: dev@hive.apache.org 
Subject: Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's and 
Bugs on apache hive branch-3

[You don't often get email from ngan...@cloudera.com.invalid. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

Thank you Aman for volunteering to drive this. +1 for a release off
branch-3. We can fix all the CVEs we have fixed on master.

IMHO, the hadoop upgrade might be too big a task for this release. Last I
checked, there were some pending items from this upgrade even on master.
They may not be hard dependencies but if we are committing to this, might
take a bit longer to finish the release.

I started to build this Jira Board for the releases. The goal was to use
this to track release items (for all releases) via the use of jira
labels/target versions.
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FRapidBoard.jspa%3FrapidView%3D564data=05%7C01%7Crajaman%40microsoft.com%7Ce3779d61c5654d11a6c608dac1945fdc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638035140295835713%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=UqSB8a8yfaZIdp9TFh4bW3ASSYFxa9ior9rSZz8OhWE%3Dreserved=0
At the top of this board, there are some quick filters for release blockers
(jiras with labels "hive-4.0.0-must"). There are currently only 2 jiras
tagged as blockers for 4.0.

If you could tag the jiras for 3.2 release the same way, and add a quick
filter, that would be great.

Thank you again
Naveen

On Fri, Nov 4, 2022 at 7:01 AM Stamatis Zampetakis 
wrote:

> Hey everyone,
>
> It would be nice to have a new release from branch 3 although it might not
> be that trivial to get out.
>
> It will definitely require a bit of investment from multiple people
> including the PMC and the committers of the project. Note that the last
> vote for alpha2 was unsuccessful due to lack of votes, which shows that
> people are pretty busy.
>
> Personally, I support this effort and would like to see it happen but this
> period I don't have sufficient time to invest to help with reviews and
> commits for 3.X line.
>
> Best,
> Stamatis
>
> On Fri, Nov 4, 2022, 5:28 AM Aman Raj 
> wrote:
>
> > Hi Chris,
> >
> > I plan on going through this diff and making a comprehensive list of all
> > the major bug fixes that went into branch-3 and not in hive-313. This
> will
> > be included in the umbrella JIRA that I am creating.
> >
> > In this email thread I have only mentioned CVEs and upgrades that will go
> > on top of these changes in branch-3.
> > Thanks,
> > Aman.
> >
> > 
> > From: Chris Nauroth 
> > Sent: Friday, November 4, 2022 3:44 AM
> > To: dev@hive.apache.org 
> > Subject: Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's
> > and Bugs on apache hive branch-3
> >
> > I noticed that there is a pretty large delta (256 commits) between
> release
> > 3.1.3 and the current branch-3:
> >
> > > git log --oneline rel/release-3.1.3..upstream-branch-3 | wc
> > 2564208   33558
> >
> > I just wanted to mention that a release from branch-3 would include far
> > more than what we are cataloging on this mail thread.
> >
> > Chris Nauroth
> >
> >
> > On Thu, Nov 3, 2022 at 12:16 PM Pravin Sinha 
> > wrote:
> >
> > > +1,
> > >
> > > Thanks for driving this, Aman. Apart from CVE fixes, do you have a list
> > of
> > > JIRAs to be targeted?
> > >
> > > -Pravin
> > >
> > > On Thu, Nov 3, 2022 at 11:12 PM Chris Nauroth 
> > wrote:
> > >
> > > > Thank you for driving this!
> > > >
> > > > To kick things off, I have filed HIVE-26702 for a backport of
> > HIVE-17315
> > > (a
> > > > total of 5 sub-tasks/patches) to 3.2.0. This adds support for more
> > > flexible
> > > > configuration of the metastore's database connection pooling.
> > Dataproc's
> > > > distribution has been running this in production backported onto
> > release
> > > > 3.1.3, so I can provide the patches.
> > > >
> > > > May I assume that our intent is to keep 3.2.x backward-compatible
> with
> > > > 3.1.x?
> > > >
> > > > Chris Nauroth
> > > >
> > > >
> > > > On Thu, Nov 3, 2022 at 3:53 AM Sankar Hariappan
> > > >  wrote:
> > > >
> > > > > +1, I'm excited to see the scope includes important 

[jira] [Created] (HIVE-26748) Prepare for Hive 3.2.0 Release

[Aman Raj (Jira)]

Aman Raj created HIVE-26748:
---

 Summary: Prepare for Hive 3.2.0 Release
 Key: HIVE-26748
 URL: https://issues.apache.org/jira/browse/HIVE-26748
 Project: Hive
  Issue Type: Task
  Components: Hive
Affects Versions: 3.2.0
Reporter: Aman Raj
Assignee: Aman Raj


This is the Umbrella Jira to track all the commits that would go on top of 
current branch-3 in this new 3.2.0 Hive release. I will add all the JIRAs that 
will be cherry picked as part of this commit by defining subtasks or linking 
the JIRAs.

 

*Please note that this is an Open forum and I welcome all responses for the 
same from the community with regards to any new bug fixes that should be cherry 
picked.*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26747) Remove implementor from HiveRelNode

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-26747:
-

 Summary: Remove implementor from HiveRelNode
 Key: HIVE-26747
 URL: https://issues.apache.org/jira/browse/HIVE-26747
 Project: Hive
  Issue Type: Task
  Components: CBO
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa


Calcite's VolcanoPlanner [1] relies on calling convention [2]. In Hive this is 
represented by the 
[HiveRelNode|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveRelNode.java]
 interface's {{CONVENTION}} field.

This interface has to be implemented by all Hive operators to have the Hive 
calling convention behavior. The interface also defines the
{code:java}
 void implement(Implementor implementor);
{code}
method but none of the operators gives an implementation and the method is 
never called.

[1] 
[https://15721.courses.cs.cmu.edu/spring2017/papers/14-optimizer1/graefe-icde1993.pdf]
[2] [https://arxiv.org/pdf/1802.10233.pdf] (Section 4, traits)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache Hive 4.0.0-alpha-2 Released

[Denys Kuzmenko]

The Apache Hive team is proud to announce the release of Apache Hive
version 4.0.0-alpha-2

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce, Apache Tez and Apache
Spark frameworks.

For Hive release details and downloads, please
visit:https://hive.apache.org/downloads.html

Hive 4.0.0-alpha-2 Release Notes are available
here:https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12351489=Html=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

[jira] [Created] (HIVE-26746) Request tracking: change to X-Request-ID header

[Jira]

László Bodor created HIVE-26746:
---

 Summary: Request tracking: change to X-Request-ID header
 Key: HIVE-26746
 URL: https://issues.apache.org/jira/browse/HIVE-26746
 Project: Hive
  Issue Type: Improvement
Reporter: László Bodor






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26745) HPL unable to handle Decimal or null values in hplsql mode

[Jira]

Ádám Szita created HIVE-26745:
-

 Summary: HPL unable to handle Decimal or null values in hplsql mode
 Key: HIVE-26745
 URL: https://issues.apache.org/jira/browse/HIVE-26745
 Project: Hive
  Issue Type: Bug
Reporter: Ádám Szita
Assignee: Ádám Szita


Decimal or null values coming from Hive service side are not handled properly 
in HPL.

When we're using beeline in HPL mode then the results of queries such as SELECT 
CAST are returned in RowSet objects directly from HS2 code. This is due to 
HplSqlQueryExecutor expecting RowSet type in OperationRowResult. For this 
RowSet interface, a ColumnBasedSet class brings the imlementation which holds a 
ColumnBuffer list to hold the result data.

The internal representation however cannot hold BigDecimal values, as seen 
here: 
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/thrift/ColumnBuffer.java#L396
 so while the SELECT CAST query will produce a HiveBigDecimal instance, it will 
be converted to String at 
https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/ColumnBasedSet.java#L110-L112
 before we add this piece of data to the result set.

Further down the line we will see class java.lang.String cannot be casted to 
class java.math.BigDecimal.

ColumnBuffer seems like a pretty commonly used code and I don't think we would 
want to try and extend this with the ability the carry decimal types. Perhaps 
we could create a Jira to recreate the BigDecimal value from the String 
representation arriving on the HPL side.

Since the two issues are both about value conversion I'm aiming to fix these in 
one go.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26744) Add array_max and array_min UDFs

[Taraka Rama Rao Lethavadla (Jira)]

Taraka Rama Rao Lethavadla created HIVE-26744:
-

 Summary: Add array_max and array_min UDFs 
 Key: HIVE-26744
 URL: https://issues.apache.org/jira/browse/HIVE-26744
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Taraka Rama Rao Lethavadla
Assignee: Taraka Rama Rao Lethavadla






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26743) backport HIVE-24694 to 3.1.x

[shalk (Jira)]

shalk created HIVE-26743:


 Summary: backport HIVE-24694 to 3.1.x
 Key: HIVE-26743
 URL: https://issues.apache.org/jira/browse/HIVE-26743
 Project: Hive
  Issue Type: Bug
Reporter: shalk


we meet  connection leak  when the login fail, these transport is not closed. 
so i try to fix it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)