[jira] [Commented] (HIVE-6756) alter table set fileformat should set serde too

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000967#comment-14000967
 ] 

Hive QA commented on HIVE-6756:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645402/HIVE-6756.1.patch

{color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 5526 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_04_evolved_parts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapreduce6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_pipe
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.common.metrics.TestMetrics.testScopeConcurrency
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hive.hcatalog.cli.TestUseDatabase.testAlterTablePass
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/224/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/224/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 31 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12645402

> alter table set fileformat should set serde too
> ---
>
> Key: HIVE-6756
> URL: https://issues.apache.org/jira/browse/HIVE-6756
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Owen O'Malley
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-6756.1.patch, HIVE-6756.patch
>
>
> Currently doing alter table set fileformat doesn't change the serde. This is 
> unexpected by customers because the serdes are largely file format specific.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Apache Hive 0.13.1 Release Candidate 1

[Lefty Leverenz]

Hive 
bylawssay
the mailing list is used for voting, but as I recall bylaws have some
wiggle room.

Decisions regarding the project are made by votes on the primary project
> development mailing list (u...@hive.apache.org ).
> Where necessary, PMC voting may take place on the private Hive PMC mailing
> list. Votes are clearly indicated by subject line starting with [VOTE].
> Votes may contain multiple items for approval and these should be clearly
> separated. Voting is carried out by replying to the vote mail.


(Hm, the text says "primary project development mailing list" but then
user@hive is shown in parentheses -- is that a typo in the bylaws?)

Would people be willing to vote simultaneously by mail and on a jira?  It's
inconvenient but shouldn't be necessary after this release.

-- Lefty


On Sat, May 17, 2014 at 7:30 PM, Sushanth Sowmyan wrote:

> There is a technical issue as well now, as raised by Prashant. But
> there is also the issue that people aren't reliably able to
> respond/object/approve, and not knowing if/when it'll go through.
>
> I think I like Lefty's jira proposal - we could open out a jira for it
> and address votes there, I think I'll do that for RC2.
>
> On Fri, May 16, 2014 at 2:53 PM, Alan Gates  wrote:
> > So this isn’t a technical issue, just concern about the delays in the
> mailing list?  Why not just extend the voting period then, until say Monday?
> >
> > Alan.
> >
> > On May 15, 2014, at 3:17 PM, Sushanth Sowmyan 
> wrote:
> >
> >> Hi Folks,
> >>
> >> I'm canceling this vote and withdrawing the RC1 candidate for the
> >> following reasons:
> >>
> >> a) I've talked to a couple of other people who haven't seen my mail
> >> updates to this thread, and saw my initial vote mail a bit late too.
> >> b) There's at least one other person that has attempted to reply to
> >> this thread, and I don't see the replies yet.
> >>
> >> Thus, when the mailing list channel isn't reliably working, the
> >> ability for people to +1 or -1 is taken away, and this does not work.
> >> (We don't want a situation where 3 people go ahead and +1, and that
> >> arrives before today evening, thus making the release releasable,
> >> while someone else discovers a breaking issue that should stop it, but
> >> is not able to have their objection or -1 appear in time.)
> >>
> >> I'm open to suggestions on how to proceed with the voting process. We
> >> could wait out this week and hope the ASF mailing list issues are
> >> resolved, but if it takes too much longer than that, we also have the
> >> issue of delaying an important bugfix release.
> >>
> >> Thoughts?
> >>
> >> -Sushanth
> >> (3:15PM PDT, May 15 2014)
> >>
> >>
> >>
> >> On Thu, May 15, 2014 at 11:46 AM, Sushanth Sowmyan 
> wrote:
> >>> The apache dev list seems to still be a little wonky, Prasanth mailed
> >>> me saying he'd replied to this thread with the following content, that
> >>> I don't see in this thread:
> >>>
> >>> "Hi Sushanth
> >>>
> >>> https://issues.apache.org/jira/browse/HIVE-7067
> >>> This bug is critical as it returns wrong results for min(), max(),
> >>> join queries that uses date/timestamp columns from ORC table.
> >>> The reason for this issue is, for these datatypes ORC returns java
> >>> objects whereas for all other types ORC returns writables.
> >>> When get() is performed on their corresponding object inspectors,
> >>> writables return a new object where as java object returns reference.
> >>> This will cause issue when any operator perform comparison on
> >>> date/timestamp values (references will be overwritten with next
> >>> values).
> >>> More information is provided in the description of the jira.
> >>>
> >>> I think the severity of this bug is critical and should be included as
> >>> part of 0.13.1. Can you please include this patch in RC2?”
> >>>
> >>> I think this meets the bar for criticality(actual bug in core feature,
> >>> no workaround) and severity( incorrect results, effectively data
> >>> corruption when used as source for other data), and I'm willing to
> >>> spin an RC2 for this, but I would still like to follow the process I
> >>> set up for jira inclusion though, to make sure I'm not being biased
> >>> about this, so I would request two other +1s to champion this bug's
> >>> inclusion into the release.
> >>>
> >>> Also, another thought here is whether it makes sense for us to try to
> >>> have a VOTE with a 72 hour deadline when the mailing list still seems
> >>> iffy and delaying mails by multiple hours. Any thoughts on how we
> >>> should proceed? (In case this mail goes out much later than I send it
> >>> out, I'm sending it out at 11:45AM PDT, Thu May 15 2014)
> >>>
> >>>
> >>>
> >>> On Thu, May 15, 2014 at 10:06 AM, Sushanth Sowmyan 
> wrote:
>  Eugene, do you know if these two failures happen on 0.13.0 as well?
> 
>  I would assume that TestHive_7 is an issue on 0.13.0 as well, given
>  that the fix for i

Re: [VOTE] Apache Hive 0.13.1 Release Candidate 1

[Sushanth Sowmyan]

There is a technical issue as well now, as raised by Prashant. But
there is also the issue that people aren't reliably able to
respond/object/approve, and not knowing if/when it'll go through.

I think I like Lefty's jira proposal - we could open out a jira for it
and address votes there, I think I'll do that for RC2.

On Fri, May 16, 2014 at 2:53 PM, Alan Gates  wrote:
> So this isn’t a technical issue, just concern about the delays in the mailing 
> list?  Why not just extend the voting period then, until say Monday?
>
> Alan.
>
> On May 15, 2014, at 3:17 PM, Sushanth Sowmyan  wrote:
>
>> Hi Folks,
>>
>> I'm canceling this vote and withdrawing the RC1 candidate for the
>> following reasons:
>>
>> a) I've talked to a couple of other people who haven't seen my mail
>> updates to this thread, and saw my initial vote mail a bit late too.
>> b) There's at least one other person that has attempted to reply to
>> this thread, and I don't see the replies yet.
>>
>> Thus, when the mailing list channel isn't reliably working, the
>> ability for people to +1 or -1 is taken away, and this does not work.
>> (We don't want a situation where 3 people go ahead and +1, and that
>> arrives before today evening, thus making the release releasable,
>> while someone else discovers a breaking issue that should stop it, but
>> is not able to have their objection or -1 appear in time.)
>>
>> I'm open to suggestions on how to proceed with the voting process. We
>> could wait out this week and hope the ASF mailing list issues are
>> resolved, but if it takes too much longer than that, we also have the
>> issue of delaying an important bugfix release.
>>
>> Thoughts?
>>
>> -Sushanth
>> (3:15PM PDT, May 15 2014)
>>
>>
>>
>> On Thu, May 15, 2014 at 11:46 AM, Sushanth Sowmyan  
>> wrote:
>>> The apache dev list seems to still be a little wonky, Prasanth mailed
>>> me saying he'd replied to this thread with the following content, that
>>> I don't see in this thread:
>>>
>>> "Hi Sushanth
>>>
>>> https://issues.apache.org/jira/browse/HIVE-7067
>>> This bug is critical as it returns wrong results for min(), max(),
>>> join queries that uses date/timestamp columns from ORC table.
>>> The reason for this issue is, for these datatypes ORC returns java
>>> objects whereas for all other types ORC returns writables.
>>> When get() is performed on their corresponding object inspectors,
>>> writables return a new object where as java object returns reference.
>>> This will cause issue when any operator perform comparison on
>>> date/timestamp values (references will be overwritten with next
>>> values).
>>> More information is provided in the description of the jira.
>>>
>>> I think the severity of this bug is critical and should be included as
>>> part of 0.13.1. Can you please include this patch in RC2?”
>>>
>>> I think this meets the bar for criticality(actual bug in core feature,
>>> no workaround) and severity( incorrect results, effectively data
>>> corruption when used as source for other data), and I'm willing to
>>> spin an RC2 for this, but I would still like to follow the process I
>>> set up for jira inclusion though, to make sure I'm not being biased
>>> about this, so I would request two other +1s to champion this bug's
>>> inclusion into the release.
>>>
>>> Also, another thought here is whether it makes sense for us to try to
>>> have a VOTE with a 72 hour deadline when the mailing list still seems
>>> iffy and delaying mails by multiple hours. Any thoughts on how we
>>> should proceed? (In case this mail goes out much later than I send it
>>> out, I'm sending it out at 11:45AM PDT, Thu May 15 2014)
>>>
>>>
>>>
>>> On Thu, May 15, 2014 at 10:06 AM, Sushanth Sowmyan  
>>> wrote:
 Eugene, do you know if these two failures happen on 0.13.0 as well?

 I would assume that TestHive_7 is an issue on 0.13.0 as well, given
 that the fix for it went into trunk. What is your sense for how
 important it is that we fix this? i.e., per my understanding, (a) It
 does not cause a crash or adversly affect the ability for webhcat to
 continue operating, and (b) It means that the feature does not work
 (at all, but in isolation), and that there is no work around for it.
 This means I treat it as critical(valid bug without workaround) but
 not severe(breaks product, affects other features from being used).
 Thus, I'm willing to include HIVE-6521 in an RC2 if we have 2 more
 committers +1 an inclusion request for this.

 As for TestHeartbeat_1, that's an interesting failure. Do you have
 logs on what commandline options
 org.apache.hive.hcatalog.templeton.LauncherDelegator sent along that
 caused it to break? Would that affect other job launches?


 On Tue, May 13, 2014 at 8:14 PM, Eugene Koifman
  wrote:
> TestHive_7 is explained by 
> https://issues.apache.org/jira/browse/HIVE-6521,
> which is in trunk but not 13.1
>
>
> On Tue, May 13, 2014 

Re: [VOTE] Apache Hive 0.13.1 Release Candidate 1

[Alan Gates]

On May 16, 2014, at 10:51 PM, Lefty Leverenz  wrote:

>> Any thoughts on how we should proceed?
> 
>  1. Is the mail archive accurate now?  Perhaps it could be used for vote
>  verification.
>  2. What if we voted in comments on a JIRA ticket?  (Lately I'm checking
>  comment order on JIRAs because my inbox receives messages out of order.)
No, it has to use mail as the primary medium I think.  But the archives are 
accurate.

Alan.

> 
> The JIRA is connected to the mailing list, so it might comply with the
> vote-by-email rule.
> 
> -- Lefty
> 
> 
> On Fri, May 16, 2014 at 2:53 PM, Alan Gates  wrote:
> 
>> So this isn’t a technical issue, just concern about the delays in the
>> mailing list?  Why not just extend the voting period then, until say Monday?
>> 
>> Alan.
>> 
>> On May 15, 2014, at 3:17 PM, Sushanth Sowmyan  wrote:
>> 
>>> Hi Folks,
>>> 
>>> I'm canceling this vote and withdrawing the RC1 candidate for the
>>> following reasons:
>>> 
>>> a) I've talked to a couple of other people who haven't seen my mail
>>> updates to this thread, and saw my initial vote mail a bit late too.
>>> b) There's at least one other person that has attempted to reply to
>>> this thread, and I don't see the replies yet.
>>> 
>>> Thus, when the mailing list channel isn't reliably working, the
>>> ability for people to +1 or -1 is taken away, and this does not work.
>>> (We don't want a situation where 3 people go ahead and +1, and that
>>> arrives before today evening, thus making the release releasable,
>>> while someone else discovers a breaking issue that should stop it, but
>>> is not able to have their objection or -1 appear in time.)
>>> 
>>> I'm open to suggestions on how to proceed with the voting process. We
>>> could wait out this week and hope the ASF mailing list issues are
>>> resolved, but if it takes too much longer than that, we also have the
>>> issue of delaying an important bugfix release.
>>> 
>>> Thoughts?
>>> 
>>> -Sushanth
>>> (3:15PM PDT, May 15 2014)
>>> 
>>> 
>>> 
>>> On Thu, May 15, 2014 at 11:46 AM, Sushanth Sowmyan 
>> wrote:
 The apache dev list seems to still be a little wonky, Prasanth mailed
 me saying he'd replied to this thread with the following content, that
 I don't see in this thread:
 
 "Hi Sushanth
 
 https://issues.apache.org/jira/browse/HIVE-7067
 This bug is critical as it returns wrong results for min(), max(),
 join queries that uses date/timestamp columns from ORC table.
 The reason for this issue is, for these datatypes ORC returns java
 objects whereas for all other types ORC returns writables.
 When get() is performed on their corresponding object inspectors,
 writables return a new object where as java object returns reference.
 This will cause issue when any operator perform comparison on
 date/timestamp values (references will be overwritten with next
 values).
 More information is provided in the description of the jira.
 
 I think the severity of this bug is critical and should be included as
 part of 0.13.1. Can you please include this patch in RC2?”
 
 I think this meets the bar for criticality(actual bug in core feature,
 no workaround) and severity( incorrect results, effectively data
 corruption when used as source for other data), and I'm willing to
 spin an RC2 for this, but I would still like to follow the process I
 set up for jira inclusion though, to make sure I'm not being biased
 about this, so I would request two other +1s to champion this bug's
 inclusion into the release.
 
 Also, another thought here is whether it makes sense for us to try to
 have a VOTE with a 72 hour deadline when the mailing list still seems
 iffy and delaying mails by multiple hours. Any thoughts on how we
 should proceed? (In case this mail goes out much later than I send it
 out, I'm sending it out at 11:45AM PDT, Thu May 15 2014)
 
 
 
 On Thu, May 15, 2014 at 10:06 AM, Sushanth Sowmyan 
>> wrote:
> Eugene, do you know if these two failures happen on 0.13.0 as well?
> 
> I would assume that TestHive_7 is an issue on 0.13.0 as well, given
> that the fix for it went into trunk. What is your sense for how
> important it is that we fix this? i.e., per my understanding, (a) It
> does not cause a crash or adversly affect the ability for webhcat to
> continue operating, and (b) It means that the feature does not work
> (at all, but in isolation), and that there is no work around for it.
> This means I treat it as critical(valid bug without workaround) but
> not severe(breaks product, affects other features from being used).
> Thus, I'm willing to include HIVE-6521 in an RC2 if we have 2 more
> committers +1 an inclusion request for this.
> 
> As for TestHeartbeat_1, that's an interesting failure. Do you have
> logs on what commandline options
> org.apache.hive.hcatalog

[jira] [Commented] (HIVE-6636) /user/hive is a bad default for HDFS jars path for Tez

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000944#comment-14000944
 ] 

Lefty Leverenz commented on HIVE-6636:
--

Documented in the wiki:

* [hive.jar.directory | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.jar.directory]

> /user/hive is a bad default for HDFS jars path for Tez
> --
>
> Key: HIVE-6636
> URL: https://issues.apache.org/jira/browse/HIVE-6636
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6636.01.patch, HIVE-6636.02.patch, HIVE-6636.patch
>
>
> If user runs hive under the user name that is not "hive", jobs will fail 
> until everyone is granted write access to /user/hive, which is not nice.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6999) Add streaming mode to PTFs

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000931#comment-14000931
 ] 

Hive QA commented on HIVE-6999:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645431/HIVE-6999.4.patch

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 5526 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_pipe
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/223/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/223/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12645431

> Add streaming mode to PTFs
> --
>
> Key: HIVE-6999
> URL: https://issues.apache.org/jira/browse/HIVE-6999
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6999.1.patch, HIVE-6999.2.patch, HIVE-6999.3.patch, 
> HIVE-6999.4.patch, HIVE-6999.4.patch
>
>
> There are a set of use cases where the Table Function can operate on a 
> Partition row by row or on a subset(window) of rows as it is being streamed 
> to it.
> - Windowing has couple of use cases of this:processing of Rank functions, 
> processing of Window Aggregations.
> - But this is a generic concept: any analysis that operates on an Ordered 
> partition maybe able to operate in Streaming mode.
> This patch introduces streaming mode in PTFs and provides the mechanics to 
> handle PTF chains that contain both modes of PTFs.
> Subsequent patches will introduce Streaming mode for Windowing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6937) Fix test reporting url's after jenkins move from bigtop

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-6937:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Thank you Szehon for the contribution! I have committed this to trunk.

> Fix test reporting url's after jenkins move from bigtop
> ---
>
> Key: HIVE-6937
> URL: https://issues.apache.org/jira/browse/HIVE-6937
> Project: Hive
>  Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Fix For: 0.14.0
>
> Attachments: HIVE-6937.patch
>
>
> This move co-located HivePtest webserver and Jenkins server.  Due to the 
> conflicts, I had to remap some URL's, thus breaking the URL of getting logs 
> and test-reports.
> The Hive Ptest2 framework makes some assumption about the relative location 
> of logs and REST endpoint URL's, that are no longer true, namely that they 
> are located at endpoint:/logs and endpoint:/hive-ptest/api. This needs to be 
> fixed.  Now, the logs are at host/logs, and HivePtest webserver REST 
> endpoints are at: endpoint/hive-ptest/api.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6999) Add streaming mode to PTFs

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-6999:
---

Attachment: HIVE-6999.4.patch

Re-uploading patch to test ptest.

> Add streaming mode to PTFs
> --
>
> Key: HIVE-6999
> URL: https://issues.apache.org/jira/browse/HIVE-6999
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6999.1.patch, HIVE-6999.2.patch, HIVE-6999.3.patch, 
> HIVE-6999.4.patch, HIVE-6999.4.patch
>
>
> There are a set of use cases where the Table Function can operate on a 
> Partition row by row or on a subset(window) of rows as it is being streamed 
> to it.
> - Windowing has couple of use cases of this:processing of Rank functions, 
> processing of Window Aggregations.
> - But this is a generic concept: any analysis that operates on an Ordered 
> partition maybe able to operate in Streaming mode.
> This patch introduces streaming mode in PTFs and provides the mechanics to 
> handle PTF chains that contain both modes of PTFs.
> Subsequent patches will introduce Streaming mode for Windowing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6756) alter table set fileformat should set serde too

[Vasanth kumar RJ (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vasanth kumar RJ updated HIVE-6756:
---

Affects Version/s: 0.13.0
   Status: Patch Available  (was: Open)

> alter table set fileformat should set serde too
> ---
>
> Key: HIVE-6756
> URL: https://issues.apache.org/jira/browse/HIVE-6756
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Owen O'Malley
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-6756.1.patch, HIVE-6756.patch
>
>
> Currently doing alter table set fileformat doesn't change the serde. This is 
> unexpected by customers because the serdes are largely file format specific.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6999) Add streaming mode to PTFs

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000734#comment-14000734
 ] 

Hive QA commented on HIVE-6999:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645377/HIVE-6999.4.patch

{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 5451 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.metastore.TestMetastoreVersion.testDefaults
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/221/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/221/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12645377

> Add streaming mode to PTFs
> --
>
> Key: HIVE-6999
> URL: https://issues.apache.org/jira/browse/HIVE-6999
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6999.1.patch, HIVE-6999.2.patch, HIVE-6999.3.patch, 
> HIVE-6999.4.patch
>
>
> There are a set of use cases where the Table Function can operate on a 
> Partition row by row or on a subset(window) of rows as it is being streamed 
> to it.
> - Windowing has couple of use cases of this:processing of Rank functions, 
> processing of Window Aggregations.
> - But this is a generic concept: any analysis that operates on an Ordered 
> partition maybe able to operate in Streaming mode.
> This patch introduces streaming mode in PTFs and provides the mechanics to 
> handle PTF chains that contain both modes of PTFs.
> Subsequent patches will introduce Streaming mode for Windowing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6756) alter table set fileformat should set serde too

[Vasanth kumar RJ (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000725#comment-14000725
 ] 

Vasanth kumar RJ commented on HIVE-6756:


Hi [~ashutoshc],
Implemented as per your suggestion.
Sorry [~chinnalalam] for taking over this JIRA.

Kindly verify.

Thanks,
Vasanth kumar

> alter table set fileformat should set serde too
> ---
>
> Key: HIVE-6756
> URL: https://issues.apache.org/jira/browse/HIVE-6756
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-6756.1.patch, HIVE-6756.patch
>
>
> Currently doing alter table set fileformat doesn't change the serde. This is 
> unexpected by customers because the serdes are largely file format specific.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6756) alter table set fileformat should set serde too

[Vasanth kumar RJ (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vasanth kumar RJ updated HIVE-6756:
---

Attachment: HIVE-6756.1.patch

> alter table set fileformat should set serde too
> ---
>
> Key: HIVE-6756
> URL: https://issues.apache.org/jira/browse/HIVE-6756
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-6756.1.patch, HIVE-6756.patch
>
>
> Currently doing alter table set fileformat doesn't change the serde. This is 
> unexpected by customers because the serdes are largely file format specific.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6328) Hive script should not overwrite AUX_CLASSPATH with HIVE_AUX_JARS_PATH if the latter is set

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000714#comment-14000714
 ] 

Lefty Leverenz commented on HIVE-6328:
--

Ping.

> Has anyone updated the wiki with information about this jira & related jiras 
> (HIVE-2269, HIVE-3978, HIVE-5363, HIVE-5410)?

> Hive script should not overwrite AUX_CLASSPATH with HIVE_AUX_JARS_PATH if the 
> latter is set
> ---
>
> Key: HIVE-6328
> URL: https://issues.apache.org/jira/browse/HIVE-6328
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-6328.patch
>
>
> Hive script (bin/hive) replaces the value of AUX_CLASSPATH with the value of 
> HIVE_AUX_JARS_PATH if HIVE_AUX_JARS_PATH is defined. This is not desirable 
> because user uses the former to include additional classes when starting 
> hive, while using the latter to specify additional jars that are needed to 
> run MR jobs. The problem can be demonstrated with the script snippet:
> {code}
> elif [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
>   HIVE_AUX_JARS_PATH=`echo $HIVE_AUX_JARS_PATH | sed 's/,/:/g'`
>   if $cygwin; then
>   HIVE_AUX_JARS_PATH=`cygpath -p -w "$HIVE_AUX_JARS_PATH"`
>   HIVE_AUX_JARS_PATH=`echo $HIVE_AUX_JARS_PATH | sed 's/;/,/g'`
>   fi
>   AUX_CLASSPATH=${HIVE_AUX_JARS_PATH}
>   AUX_PARAM="file://$(echo ${HIVE_AUX_JARS_PATH} | sed 's/:/,file:\/\//g')"
> fi
> {code}
> AUX_CLASSPATH should be respected regardless whether HIVE_AUX_JARS_PATH is 
> defined.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7071) Use custom Tez split generator to support schema evolution

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000691#comment-14000691
 ] 

Hive QA commented on HIVE-7071:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12645190/HIVE-7071.1.patch

{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 5450 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/218/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/218/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12645190

> Use custom Tez split generator to support schema evolution
> --
>
> Key: HIVE-7071
> URL: https://issues.apache.org/jira/browse/HIVE-7071
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-7071.1.patch
>
>
> Right now we're falling back to combinehivefileinputformat and switch of am 
> side grouping when there's different schemata in a single vertex. We need to 
> handle this in a custom initializer so we can still group on the AM.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7050) Display table level column stats in DESCRIBE EXTENDED/FORMATTED TABLE

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000671#comment-14000671
 ] 

Xuefu Zhang commented on HIVE-7050:
---

+1, pending on test result.

> Display table level column stats in DESCRIBE EXTENDED/FORMATTED TABLE
> -
>
> Key: HIVE-7050
> URL: https://issues.apache.org/jira/browse/HIVE-7050
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth J
>Assignee: Prasanth J
> Attachments: HIVE-7050.1.patch, HIVE-7050.2.patch, HIVE-7050.3.patch, 
> HIVE-7050.4.patch, HIVE-7050.5.patch
>
>
> There is currently no way to display the column level stats from hive CLI. It 
> will be good to show them in DESCRIBE EXTENDED/FORMATTED TABLE



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Apache Hive 0.13.1 Release Candidate 1

[Lefty Leverenz]

> Any thoughts on how we should proceed?

   1. Is the mail archive accurate now?  Perhaps it could be used for vote
   verification.
   2. What if we voted in comments on a JIRA ticket?  (Lately I'm checking
   comment order on JIRAs because my inbox receives messages out of order.)

The JIRA is connected to the mailing list, so it might comply with the
vote-by-email rule.

-- Lefty


On Fri, May 16, 2014 at 2:53 PM, Alan Gates  wrote:

> So this isn’t a technical issue, just concern about the delays in the
> mailing list?  Why not just extend the voting period then, until say Monday?
>
> Alan.
>
> On May 15, 2014, at 3:17 PM, Sushanth Sowmyan  wrote:
>
> > Hi Folks,
> >
> > I'm canceling this vote and withdrawing the RC1 candidate for the
> > following reasons:
> >
> > a) I've talked to a couple of other people who haven't seen my mail
> > updates to this thread, and saw my initial vote mail a bit late too.
> > b) There's at least one other person that has attempted to reply to
> > this thread, and I don't see the replies yet.
> >
> > Thus, when the mailing list channel isn't reliably working, the
> > ability for people to +1 or -1 is taken away, and this does not work.
> > (We don't want a situation where 3 people go ahead and +1, and that
> > arrives before today evening, thus making the release releasable,
> > while someone else discovers a breaking issue that should stop it, but
> > is not able to have their objection or -1 appear in time.)
> >
> > I'm open to suggestions on how to proceed with the voting process. We
> > could wait out this week and hope the ASF mailing list issues are
> > resolved, but if it takes too much longer than that, we also have the
> > issue of delaying an important bugfix release.
> >
> > Thoughts?
> >
> > -Sushanth
> > (3:15PM PDT, May 15 2014)
> >
> >
> >
> > On Thu, May 15, 2014 at 11:46 AM, Sushanth Sowmyan 
> wrote:
> >> The apache dev list seems to still be a little wonky, Prasanth mailed
> >> me saying he'd replied to this thread with the following content, that
> >> I don't see in this thread:
> >>
> >> "Hi Sushanth
> >>
> >> https://issues.apache.org/jira/browse/HIVE-7067
> >> This bug is critical as it returns wrong results for min(), max(),
> >> join queries that uses date/timestamp columns from ORC table.
> >> The reason for this issue is, for these datatypes ORC returns java
> >> objects whereas for all other types ORC returns writables.
> >> When get() is performed on their corresponding object inspectors,
> >> writables return a new object where as java object returns reference.
> >> This will cause issue when any operator perform comparison on
> >> date/timestamp values (references will be overwritten with next
> >> values).
> >> More information is provided in the description of the jira.
> >>
> >> I think the severity of this bug is critical and should be included as
> >> part of 0.13.1. Can you please include this patch in RC2?”
> >>
> >> I think this meets the bar for criticality(actual bug in core feature,
> >> no workaround) and severity( incorrect results, effectively data
> >> corruption when used as source for other data), and I'm willing to
> >> spin an RC2 for this, but I would still like to follow the process I
> >> set up for jira inclusion though, to make sure I'm not being biased
> >> about this, so I would request two other +1s to champion this bug's
> >> inclusion into the release.
> >>
> >> Also, another thought here is whether it makes sense for us to try to
> >> have a VOTE with a 72 hour deadline when the mailing list still seems
> >> iffy and delaying mails by multiple hours. Any thoughts on how we
> >> should proceed? (In case this mail goes out much later than I send it
> >> out, I'm sending it out at 11:45AM PDT, Thu May 15 2014)
> >>
> >>
> >>
> >> On Thu, May 15, 2014 at 10:06 AM, Sushanth Sowmyan 
> wrote:
> >>> Eugene, do you know if these two failures happen on 0.13.0 as well?
> >>>
> >>> I would assume that TestHive_7 is an issue on 0.13.0 as well, given
> >>> that the fix for it went into trunk. What is your sense for how
> >>> important it is that we fix this? i.e., per my understanding, (a) It
> >>> does not cause a crash or adversly affect the ability for webhcat to
> >>> continue operating, and (b) It means that the feature does not work
> >>> (at all, but in isolation), and that there is no work around for it.
> >>> This means I treat it as critical(valid bug without workaround) but
> >>> not severe(breaks product, affects other features from being used).
> >>> Thus, I'm willing to include HIVE-6521 in an RC2 if we have 2 more
> >>> committers +1 an inclusion request for this.
> >>>
> >>> As for TestHeartbeat_1, that's an interesting failure. Do you have
> >>> logs on what commandline options
> >>> org.apache.hive.hcatalog.templeton.LauncherDelegator sent along that
> >>> caused it to break? Would that affect other job launches?
> >>>
> >>>
> >>> On Tue, May 13, 2014 at 8:14 PM, Eugene Koifman
> >>>  wrote:
>  TestHi