[jira] [Created] (HIVE-10459) Add materialized views to Hive

[Alan Gates (JIRA)]

Alan Gates created HIVE-10459:
-

 Summary: Add materialized views to Hive
 Key: HIVE-10459
 URL: https://issues.apache.org/jira/browse/HIVE-10459
 Project: Hive
  Issue Type: Improvement
Reporter: Alan Gates
Assignee: Alan Gates


Materialized views are useful as ways to store either alternate versions of 
data (e.g. same data, different sort order) or derivatives of data sets (e.g. 
commonly used aggregates).  It is useful to store these as materialized views 
rather than as tables because it can give the optimizer the ability to 
understand how data sets are related.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10458) Enable parallel order by for spark [Spark Branch]

[Rui Li (JIRA)]

Rui Li created HIVE-10458:
-

 Summary: Enable parallel order by for spark [Spark Branch]
 Key: HIVE-10458
 URL: https://issues.apache.org/jira/browse/HIVE-10458
 Project: Hive
  Issue Type: Sub-task
Reporter: Rui Li
Assignee: Rui Li


We don't have to force reducer# to 1 as spark supports parallel sorting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10457) Merge trunk to spark (4/22/15)

[Szehon Ho (JIRA)]

Szehon Ho created HIVE-10457:


 Summary: Merge trunk to spark (4/22/15)
 Key: HIVE-10457
 URL: https://issues.apache.org/jira/browse/HIVE-10457
 Project: Hive
  Issue Type: Bug
Reporter: Szehon Ho






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: hive contributor meetup in bay area

[Lefty Leverenz]

Woohoo!  WebEx is starting, no login required.  (I was trying to join too
early before.)

-- Lefty

On Wed, Apr 22, 2015 at 9:54 PM, Lefty Leverenz 
wrote:

> Well now I've got audio and slides for most of the talks, so I'll get by.
>
> -- Lefty
>
> On Wed, Apr 22, 2015 at 9:45 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
>
>> Not yet. I will let Thejas know about it.
>>
>> Thanks
>> Prasanth
>>
>>
>>
>>
>> On Wed, Apr 22, 2015 at 6:37 PM -0700, "Lefty Leverenz" <
>> leftylever...@gmail.com> wrote:
>>
>> I'm on hold for the audio.  Have introductions started yet?
>>
>> -- Lefty
>>
>> On Wed, Apr 22, 2015 at 9:35 PM, Lefty Leverenz 
>> wrote:
>>
>> > Webex won't let me log in -- the login page seems to expect a
>> Hortonworks
>> > employee, and the forgot-my-password page rejected both of my email
>> > addresses.
>> >
>> > In 2013 we joined with a meeting password, but I don't see anywhere to
>> > enter it this time (and besides, I don't know what it is).
>> >
>> > -- Lefty
>> >
>> > On Tue, Apr 21, 2015 at 11:56 PM, Thejas Nair 
>> > wrote:
>> >
>> >> I don't have the slides yet, I will ask for them.
>> >>
>> >> 
>> >> From: Xu, Cheng A 
>> >> Sent: Tuesday, April 21, 2015 7:49 PM
>> >> To: dev@hive.apache.org
>> >> Subject: RE: hive contributor meetup in bay area
>> >>
>> >> Hi Thejas, could you post the slides in advance on the wiki
>> >> https://cwiki.apache.org/confluence/display/Hive/Presentations if you
>> >> have?
>> >>
>> >> -Original Message-
>> >> From: Thejas Nair [mailto:thejas.n...@gmail.com]
>> >> Sent: Wednesday, April 22, 2015 9:35 AM
>> >> To: dev
>> >> Subject: Re: hive contributor meetup in bay area
>> >>
>> >> I have also created a webex link for those who are unable to attend in
>> >> person -
>> http://www.meetup.com/Hive-Contributors-Group/events/221610423/
>> >>
>> >> Please RSVP yes ONLY if you are planning to attend in person.
>> >>
>> >>
>> >> On Tue, Apr 21, 2015 at 4:49 PM, Thejas Nair 
>> >> wrote:
>> >> > FYI, there is contributor meetup being hosted tomorrow evening at the
>> >> > Hortonworks office in Santa Clara, CA
>> >> >
>> >> > http://www.meetup.com/Hive-Contributors-Group/events/221610423/
>> >> >
>> >> > Please RSVP in the meetup page if you would like to attend.
>> >> >
>> >> > Thanks,
>> >> > Thejas
>> >>
>> >
>> >
>>
>
>

Re: hive contributor meetup in bay area

[Lefty Leverenz]

Well now I've got audio and slides for most of the talks, so I'll get by.

-- Lefty

On Wed, Apr 22, 2015 at 9:45 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> Not yet. I will let Thejas know about it.
>
> Thanks
> Prasanth
>
>
>
>
> On Wed, Apr 22, 2015 at 6:37 PM -0700, "Lefty Leverenz" <
> leftylever...@gmail.com> wrote:
>
> I'm on hold for the audio.  Have introductions started yet?
>
> -- Lefty
>
> On Wed, Apr 22, 2015 at 9:35 PM, Lefty Leverenz 
> wrote:
>
> > Webex won't let me log in -- the login page seems to expect a Hortonworks
> > employee, and the forgot-my-password page rejected both of my email
> > addresses.
> >
> > In 2013 we joined with a meeting password, but I don't see anywhere to
> > enter it this time (and besides, I don't know what it is).
> >
> > -- Lefty
> >
> > On Tue, Apr 21, 2015 at 11:56 PM, Thejas Nair 
> > wrote:
> >
> >> I don't have the slides yet, I will ask for them.
> >>
> >> 
> >> From: Xu, Cheng A 
> >> Sent: Tuesday, April 21, 2015 7:49 PM
> >> To: dev@hive.apache.org
> >> Subject: RE: hive contributor meetup in bay area
> >>
> >> Hi Thejas, could you post the slides in advance on the wiki
> >> https://cwiki.apache.org/confluence/display/Hive/Presentations if you
> >> have?
> >>
> >> -Original Message-
> >> From: Thejas Nair [mailto:thejas.n...@gmail.com]
> >> Sent: Wednesday, April 22, 2015 9:35 AM
> >> To: dev
> >> Subject: Re: hive contributor meetup in bay area
> >>
> >> I have also created a webex link for those who are unable to attend in
> >> person -
> http://www.meetup.com/Hive-Contributors-Group/events/221610423/
> >>
> >> Please RSVP yes ONLY if you are planning to attend in person.
> >>
> >>
> >> On Tue, Apr 21, 2015 at 4:49 PM, Thejas Nair 
> >> wrote:
> >> > FYI, there is contributor meetup being hosted tomorrow evening at the
> >> > Hortonworks office in Santa Clara, CA
> >> >
> >> > http://www.meetup.com/Hive-Contributors-Group/events/221610423/
> >> >
> >> > Please RSVP in the meetup page if you would like to attend.
> >> >
> >> > Thanks,
> >> > Thejas
> >>
> >
> >
>

Re: hive contributor meetup in bay area

[Prasanth Jayachandran]

Not yet. I will let Thejas know about it.

Thanks
Prasanth




On Wed, Apr 22, 2015 at 6:37 PM -0700, "Lefty Leverenz" 
mailto:leftylever...@gmail.com>> wrote:

I'm on hold for the audio.  Have introductions started yet?

-- Lefty

On Wed, Apr 22, 2015 at 9:35 PM, Lefty Leverenz 
wrote:

> Webex won't let me log in -- the login page seems to expect a Hortonworks
> employee, and the forgot-my-password page rejected both of my email
> addresses.
>
> In 2013 we joined with a meeting password, but I don't see anywhere to
> enter it this time (and besides, I don't know what it is).
>
> -- Lefty
>
> On Tue, Apr 21, 2015 at 11:56 PM, Thejas Nair 
> wrote:
>
>> I don't have the slides yet, I will ask for them.
>>
>> 
>> From: Xu, Cheng A 
>> Sent: Tuesday, April 21, 2015 7:49 PM
>> To: dev@hive.apache.org
>> Subject: RE: hive contributor meetup in bay area
>>
>> Hi Thejas, could you post the slides in advance on the wiki
>> https://cwiki.apache.org/confluence/display/Hive/Presentations if you
>> have?
>>
>> -Original Message-
>> From: Thejas Nair [mailto:thejas.n...@gmail.com]
>> Sent: Wednesday, April 22, 2015 9:35 AM
>> To: dev
>> Subject: Re: hive contributor meetup in bay area
>>
>> I have also created a webex link for those who are unable to attend in
>> person - http://www.meetup.com/Hive-Contributors-Group/events/221610423/
>>
>> Please RSVP yes ONLY if you are planning to attend in person.
>>
>>
>> On Tue, Apr 21, 2015 at 4:49 PM, Thejas Nair 
>> wrote:
>> > FYI, there is contributor meetup being hosted tomorrow evening at the
>> > Hortonworks office in Santa Clara, CA
>> >
>> > http://www.meetup.com/Hive-Contributors-Group/events/221610423/
>> >
>> > Please RSVP in the meetup page if you would like to attend.
>> >
>> > Thanks,
>> > Thejas
>>
>
>

Re: hive contributor meetup in bay area

[Lefty Leverenz]

Webex won't let me log in -- the login page seems to expect a Hortonworks
employee, and the forgot-my-password page rejected both of my email
addresses.

In 2013 we joined with a meeting password, but I don't see anywhere to
enter it this time (and besides, I don't know what it is).

-- Lefty

On Tue, Apr 21, 2015 at 11:56 PM, Thejas Nair 
wrote:

> I don't have the slides yet, I will ask for them.
>
> 
> From: Xu, Cheng A 
> Sent: Tuesday, April 21, 2015 7:49 PM
> To: dev@hive.apache.org
> Subject: RE: hive contributor meetup in bay area
>
> Hi Thejas, could you post the slides in advance on the wiki
> https://cwiki.apache.org/confluence/display/Hive/Presentations if you
> have?
>
> -Original Message-
> From: Thejas Nair [mailto:thejas.n...@gmail.com]
> Sent: Wednesday, April 22, 2015 9:35 AM
> To: dev
> Subject: Re: hive contributor meetup in bay area
>
> I have also created a webex link for those who are unable to attend in
> person - http://www.meetup.com/Hive-Contributors-Group/events/221610423/
>
> Please RSVP yes ONLY if you are planning to attend in person.
>
>
> On Tue, Apr 21, 2015 at 4:49 PM, Thejas Nair 
> wrote:
> > FYI, there is contributor meetup being hosted tomorrow evening at the
> > Hortonworks office in Santa Clara, CA
> >
> > http://www.meetup.com/Hive-Contributors-Group/events/221610423/
> >
> > Please RSVP in the meetup page if you would like to attend.
> >
> > Thanks,
> > Thejas
>

Re: hive contributor meetup in bay area

[Lefty Leverenz]

I'm on hold for the audio.  Have introductions started yet?

-- Lefty

On Wed, Apr 22, 2015 at 9:35 PM, Lefty Leverenz 
wrote:

> Webex won't let me log in -- the login page seems to expect a Hortonworks
> employee, and the forgot-my-password page rejected both of my email
> addresses.
>
> In 2013 we joined with a meeting password, but I don't see anywhere to
> enter it this time (and besides, I don't know what it is).
>
> -- Lefty
>
> On Tue, Apr 21, 2015 at 11:56 PM, Thejas Nair 
> wrote:
>
>> I don't have the slides yet, I will ask for them.
>>
>> 
>> From: Xu, Cheng A 
>> Sent: Tuesday, April 21, 2015 7:49 PM
>> To: dev@hive.apache.org
>> Subject: RE: hive contributor meetup in bay area
>>
>> Hi Thejas, could you post the slides in advance on the wiki
>> https://cwiki.apache.org/confluence/display/Hive/Presentations if you
>> have?
>>
>> -Original Message-
>> From: Thejas Nair [mailto:thejas.n...@gmail.com]
>> Sent: Wednesday, April 22, 2015 9:35 AM
>> To: dev
>> Subject: Re: hive contributor meetup in bay area
>>
>> I have also created a webex link for those who are unable to attend in
>> person - http://www.meetup.com/Hive-Contributors-Group/events/221610423/
>>
>> Please RSVP yes ONLY if you are planning to attend in person.
>>
>>
>> On Tue, Apr 21, 2015 at 4:49 PM, Thejas Nair 
>> wrote:
>> > FYI, there is contributor meetup being hosted tomorrow evening at the
>> > Hortonworks office in Santa Clara, CA
>> >
>> > http://www.meetup.com/Hive-Contributors-Group/events/221610423/
>> >
>> > Please RSVP in the meetup page if you would like to attend.
>> >
>> > Thanks,
>> > Thejas
>>
>
>

[jira] [Created] (HIVE-10456) Grace Hash Join should not load spilled partitions on abort

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-10456:


 Summary: Grace Hash Join should not load spilled partitions on 
abort
 Key: HIVE-10456
 URL: https://issues.apache.org/jira/browse/HIVE-10456
 Project: Hive
  Issue Type: Bug
Affects Versions: llap
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Grace Hash Join loads the spilled partitions to complete the join in closeOp(). 
This should not happen when closeOp with abort is invoked. Instead it should 
clean up all the spilled data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10455) CBO (Calcite Return Path): Different data types at Reducer before JoinOp

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-10455:
--

 Summary: CBO (Calcite Return Path): Different data types at 
Reducer before JoinOp
 Key: HIVE-10455
 URL: https://issues.apache.org/jira/browse/HIVE-10455
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


The following error occured for cbo_subq_not_in.q 
{code}
java.lang.Exception: java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to 
deserialize reduce input key from x1x128x0x0x1 with properties 
{columns=reducesinkkey0, 
serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe,
 serialization.sort.order=+, columns.types=double}
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
{code}

A more easier way to reproduce is 
{code}
set hive.cbo.enable=true;
set hive.exec.check.crossproducts=false;

set hive.stats.fetch.column.stats=true;
set hive.auto.convert.join=false;

select p_size, src.key
from 
part join src
on p_size=key;
{code}

As you can see, p_size is integer while src.key is string. Both of them should 
be cast to double when they join. When return path is off, this will happen 
before Join, at RS. However, when return path is on, this will be considered as 
an expression in Join. Thus, when reducer is collecting different types of keys 
from different join branches, it throws exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10454) Query against partitioned table in strict mode failed with "No partition predicate found" even if partition predicate is specified.

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-10454:
---

 Summary: Query against partitioned table in strict mode failed 
with "No partition predicate found" even if partition predicate is specified.
 Key: HIVE-10454
 URL: https://issues.apache.org/jira/browse/HIVE-10454
 Project: Hive
  Issue Type: Bug
Reporter: Aihua Xu


The following queries fail:

{noformat}

create table t1 (c1 int) PARTITIONED BY (c2 string);

set hive.mapred.mode=strict;
 select * from t1 where t1.c2 < to_date(date_add(from_unixtime( 
unix_timestamp() ),1));
{noformat}

The query failed with "No partition predicate found for alias t1".





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10453) HS2 leaking open file descriptors when using UDFs

[Yongzhi Chen (JIRA)]

Yongzhi Chen created HIVE-10453:
---

 Summary: HS2 leaking open file descriptors when using UDFs
 Key: HIVE-10453
 URL: https://issues.apache.org/jira/browse/HIVE-10453
 Project: Hive
  Issue Type: Bug
Reporter: Yongzhi Chen


1. create a custom function by
CREATE FUNCTION myfunc AS 'someudfclass' using jar 'hdfs:///tmp/myudf.jar';
2. Create a simple jdbc client, just do 
connect, 
run simple query which using the function such as:
select myfunc(col1) from sometable
3. Disconnect.
Check open file for HiveServer2 by:
lsof -p HSProcID | grep myudf.jar
You will see the leak as:
{noformat}
java  28718 ychen  txt  REG1,4741 212977666 
/private/var/folders/6p/7_njf13d6h144wldzbbsfpz8gp/T/1bfe3de0-ac63-4eba-a725-6a9840f1f8d5_resources/myudf.jar
java  28718 ychen  330r REG1,4741 212977666 
/private/var/folders/6p/7_njf13d6h144wldzbbsfpz8gp/T/1bfe3de0-ac63-4eba-a725-6a9840f1f8d5_resources/myudf.jar
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10452) Followup fix for HIVE-10202 to restrict it it for script mode.

[Naveen Gangam (JIRA)]

Naveen Gangam created HIVE-10452:


 Summary: Followup fix for HIVE-10202 to restrict it it for script 
mode.
 Key: HIVE-10452
 URL: https://issues.apache.org/jira/browse/HIVE-10452
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 1.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10451) IdentityProjectRemover removed useful project

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-10451:
---

 Summary: IdentityProjectRemover removed useful project
 Key: HIVE-10451
 URL: https://issues.apache.org/jira/browse/HIVE-10451
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 1.0.0, 0.14.0, 1.1.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


In this particular case Select on top of PTF Op is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10450) More than one TableScan in MapWork not supported in Vectorization -- causes query to fail during vectorization

[Matt McCline (JIRA)]

Matt McCline created HIVE-10450:
---

 Summary: More than one TableScan in MapWork not supported in 
Vectorization -- causes  query to fail during vectorization
 Key: HIVE-10450
 URL: https://issues.apache.org/jira/browse/HIVE-10450
 Project: Hive
  Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical



[~gopalv] found a error with this query:

{noformat}
explain select
s_state, count(1)
 from store_sales,
 store,
 date_dim
 where store_sales.ss_sold_date_sk = date_dim.d_date_sk and
   store_sales.ss_store_sk = store.s_store_sk and
   store.s_state in ('KS','AL', 'MN', 'AL', 'SC', 'VT')
 group by s_state
 order by s_state
 limit 100;
{noformat}

Stack trace:
{noformat}
org.apache.hadoop.hive.ql.parse.SemanticException: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationNodeProcessor.doVectorize(Vectorizer.java:676)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$MapWorkVectorizationNodeProcessor.process(Vectorizer.java:735)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
at 
org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:54)
at 
org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:59)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.vectorizeMapWork(Vectorizer.java:422)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:354)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:322)
at 
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
at 
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
at 
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
at 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:877)
at 
org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
at 
org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:270)
at 
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:227)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10084)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:204)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
at 
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:225)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1019)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:993)
at 
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.runTest(TestMiniTezCliDriver.java:244)
at 
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_inner_join5(TestMiniTezCliDriver.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.r

Re: Preparation for Hive-1.2 release

[Thejas Nair]

The Apache Accumulo project has written a nice doc on git use
reccomendations for their project based on experiences in Accumulo and
a similar Kafka document -
http://accumulo.apache.org/git.html. It describes the options for
committing feature branches as well. I propose using that as the
current hive git usage recommendation for now (until we feel a need to
do things differently).

I think we should update
https://cwiki.apache.org/confluence/display/Hive/HowToContribute page
to include this link. Any volunteers ?

Thanks,
Thejas


On Wed, Apr 22, 2015 at 2:37 PM, Szehon Ho  wrote:
> Thanks Sushanth.  Yes, blind applying the patch would work, but I believe
> it is a little more involved to keep the history (git merge, rebase,
> resolve, test, etc).
>
> I am doing the merge process again this time in git and regenerated what is
> hopefully an equivalent patch, hopefully I can get it done soon.
>
> Thanks,
> Szehon
>
> On Wed, Apr 22, 2015 at 1:33 PM, Sushanth Sowmyan 
> wrote:
>
>> Hi Szehon,
>>
>> Sergey tells me that he's verifying the repos, and expects to unblock
>> it in another 20 minutes or so. I think that if HIVE-10347 is ready
>> for commit now to svn, it should be committable to git as well, as-is
>> as soon as that comes back up. In that scenario, there's no extra need
>> to learn how to do merges in git, since I would expect this to be
>> ready before then. If the .patch file uploaded on HIVE-10347 applies
>> appropriately on svn, then there should be no issues applying it to
>> git. I can help there too, if that helps.
>>
>> Also, I think forking is not that drastic a measure in git as it is in
>> svn, and merges are a bit easier to manage under git. I would still
>> like to go ahead with a friday/weekend fork, and allow people to
>> continue submitting patches for inclusion up until we roll out an RC0,
>> and then lock it for bugfixes. (Note : If there are any other jiras
>> you are working on that you want to see in this release, please go add
>> it to that wiki list :
>> https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
>> )
>>
>> I would like to avoid a long-tail process as happened for 0.12.0 &
>> 0.13.0 if possible. This way, I think we'll have more time for
>> testing/solidifying.
>>
>> Thanks,
>> -Sushanth
>>
>> P.S: Also, for additional reference, from the svn->git thread,
>> http://accumulo.apache.org/git.html is a good quick reference.
>>
>> On Wed, Apr 22, 2015 at 11:31 AM, Szehon Ho  wrote:
>> > Yes, I don't think it is fair to fork it this week, the change in
>> question
>> > HIVE-10347 was reviewed and waiting 24 hours, but then svn repo is
>> suddenly
>> > locked, and now the merge process has to be done again in git?  I would
>> > vote to push it back one week to give everyone a chance to figure this
>> out,
>> > like how to do merges now.
>> >
>> > Thanks
>> > Szehon
>> >
>> >
>> >
>> > On Wed, Apr 22, 2015 at 11:10 AM, Xuefu Zhang 
>> wrote:
>> >
>> >> Since the svn repo is locked for git migration and a queue of patches
>> are
>> >> waiting to be committed, I'm not sure if forking on this Friday is
>> >> realistic.
>> >>
>> >> HIVE-9726 is part of the merge (HIVE-10347) which is also in the queue.
>> >>
>> >> Thanks,
>> >> Xuefu
>> >>
>> >> On Wed, Apr 22, 2015 at 9:55 AM, Sushanth Sowmyan 
>> >> wrote:
>> >>
>> >> > @Lefty :
>> >> >
>> >> > Yup, I've been following the other thread and
>> >> > https://issues.apache.org/jira/browse/INFRA-9488 - I think that'll
>> >> > stabilize before we fork. I'll definitely wait for it, but also wanted
>> >> > to do a reminder/call-for-patches before it. :)
>> >> >
>> >> > @Youngwoo:
>> >> >
>> >> > Thanks, I'll add that to the tracker. I do not see a commit on it in
>> >> > trunk for it, so I'll ping Brock on that bug to see if it's included
>> >> > through some other jira on trunk.
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Apr 21, 2015 at 10:58 PM, 김영우  wrote:
>> >> > > Hi,
>> >> > >
>> >> > > I would like to add a request for support Spark 1.3, HIVE-9726.
>> >> Currently
>> >> > > Apache Bigtop includes Spark 1.3 and we ran into a compatibility
>> issue
>> >> > for
>> >> > > Hive 1.1+.
>> >> > > The issue have been resolved on 'spark-branch'  but could not find a
>> >> 'fix
>> >> > > version' for it. I hope that the fix merge into release 1.2.
>> >> > >
>> >> > > Thanks,
>> >> > > Youngwoo
>> >> > >
>> >> > > On Wed, Apr 22, 2015 at 12:33 PM, Sushanth Sowmyan <
>> khorg...@gmail.com
>> >> >
>> >> > > wrote:
>> >> > >
>> >> > >> Hi Folks,
>> >> > >>
>> >> > >> Per my mail 3 weeks back, we should start getting ready to release
>> 1.2
>> >> > >> as a rollup. And as per my proposal to manage this release, I'd
>> like
>> >> > >> to start off the process of forking 1.2, and making trunk 1.3.
>> >> > >>
>> >> > >> I've set up a cwiki page for people to land development patches
>> that
>> >> > >> are almost done, to signal their desire that this be included in
>> 1.2 :
>> >> > >>
>> >> >
>> https://cwiki.apache.org/conflu

Re: svn commits

[Xuefu Zhang]

I think it would be good if we can update the wiki on how to commit as we
switch the tool.

On Wed, Apr 22, 2015 at 3:54 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> I just tried committing a path to git master and it works.
> Try adding this URL to new remote for committing
>
> https://git-wip-us.apache.org/repos/asf/hive.git
>
> > On Apr 22, 2015, at 2:46 PM, Sergey Shelukhin 
> wrote:
> >
> > Git repo should now be writeable. Thanks everyone, sorry for the
> > disturbance.
> >
> > On 15/4/22, 13:39, "Sergey Shelukhin"  wrote:
> >
> >> The git repo should be ready to use soon.
> >>
> >> On 15/4/22, 10:23, "Ashutosh Chauhan"  wrote:
> >>
> >>> are they already turned off because of git transition? I tried a commit
> >>> on
> >>> svn trunk today morning and got access denied exception. Any one else
> >>> getting this exception?
> >>> If it indeed is because of git transition, do any one know when commits
> >>> on
> >>> git repo will be enabled?
> >>
> >
>
>

Re: svn commits

[Prasanth Jayachandran]

I just tried committing a path to git master and it works.
Try adding this URL to new remote for committing

https://git-wip-us.apache.org/repos/asf/hive.git

> On Apr 22, 2015, at 2:46 PM, Sergey Shelukhin  wrote:
> 
> Git repo should now be writeable. Thanks everyone, sorry for the
> disturbance.
> 
> On 15/4/22, 13:39, "Sergey Shelukhin"  wrote:
> 
>> The git repo should be ready to use soon.
>> 
>> On 15/4/22, 10:23, "Ashutosh Chauhan"  wrote:
>> 
>>> are they already turned off because of git transition? I tried a commit
>>> on
>>> svn trunk today morning and got access denied exception. Any one else
>>> getting this exception?
>>> If it indeed is because of git transition, do any one know when commits
>>> on
>>> git repo will be enabled?
>> 
> 

[jira] [Created] (HIVE-10449) LLAP: Make new fast hash table for Native Vector Map Join work with Hybrid Grace

[Matt McCline (JIRA)]

Matt McCline created HIVE-10449:
---

 Summary: LLAP: Make new fast hash table for Native Vector Map Join 
work with Hybrid Grace
 Key: HIVE-10449
 URL: https://issues.apache.org/jira/browse/HIVE-10449
 Project: Hive
  Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline


Today only the ~vector.mapjoin.optimized hash table -- which is a thin shell 
over MapJoinBytesTableContainer / HybridHashTableContainer and 
BytesBytesMutliHashMap -- is Hybrid Grace capable.

Make the ~vector.mapjoin.fast hash table Hybrid Grace capable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10448) LLAP: Consider replacing BytesBytesMultiHashMap with new fast hash table code of Native Vector Map Join

[Matt McCline (JIRA)]

Matt McCline created HIVE-10448:
---

 Summary: LLAP: Consider replacing BytesBytesMultiHashMap with new 
fast hash table code of Native Vector Map Join
 Key: HIVE-10448
 URL: https://issues.apache.org/jira/browse/HIVE-10448
 Project: Hive
  Issue Type: Sub-task
Reporter: Matt McCline
Assignee: Matt McCline


[~sershe] observed that the new fast hash table code developed for Native 
Vector Map Join uses many of the same concepts as BytesBytesMultiHashMap can 
could be considered a v2 replacement for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10447) Beeline JDBC Driver to support 2 way SSL

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-10447:


 Summary: Beeline JDBC Driver to support 2 way SSL
 Key: HIVE-10447
 URL: https://issues.apache.org/jira/browse/HIVE-10447
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


This jira should cover 2-way SSL authentication between the JDBC Client and 
server which requires the driver to support it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: svn commits

[Sergey Shelukhin]

Git repo should now be writeable. Thanks everyone, sorry for the
disturbance.

On 15/4/22, 13:39, "Sergey Shelukhin"  wrote:

>The git repo should be ready to use soon.
>
>On 15/4/22, 10:23, "Ashutosh Chauhan"  wrote:
>
>>are they already turned off because of git transition? I tried a commit
>>on
>>svn trunk today morning and got access denied exception. Any one else
>>getting this exception?
>>If it indeed is because of git transition, do any one know when commits
>>on
>>git repo will be enabled?
>

Re: Preparation for Hive-1.2 release

[Szehon Ho]

Thanks Sushanth.  Yes, blind applying the patch would work, but I believe
it is a little more involved to keep the history (git merge, rebase,
resolve, test, etc).

I am doing the merge process again this time in git and regenerated what is
hopefully an equivalent patch, hopefully I can get it done soon.

Thanks,
Szehon

On Wed, Apr 22, 2015 at 1:33 PM, Sushanth Sowmyan 
wrote:

> Hi Szehon,
>
> Sergey tells me that he's verifying the repos, and expects to unblock
> it in another 20 minutes or so. I think that if HIVE-10347 is ready
> for commit now to svn, it should be committable to git as well, as-is
> as soon as that comes back up. In that scenario, there's no extra need
> to learn how to do merges in git, since I would expect this to be
> ready before then. If the .patch file uploaded on HIVE-10347 applies
> appropriately on svn, then there should be no issues applying it to
> git. I can help there too, if that helps.
>
> Also, I think forking is not that drastic a measure in git as it is in
> svn, and merges are a bit easier to manage under git. I would still
> like to go ahead with a friday/weekend fork, and allow people to
> continue submitting patches for inclusion up until we roll out an RC0,
> and then lock it for bugfixes. (Note : If there are any other jiras
> you are working on that you want to see in this release, please go add
> it to that wiki list :
> https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
> )
>
> I would like to avoid a long-tail process as happened for 0.12.0 &
> 0.13.0 if possible. This way, I think we'll have more time for
> testing/solidifying.
>
> Thanks,
> -Sushanth
>
> P.S: Also, for additional reference, from the svn->git thread,
> http://accumulo.apache.org/git.html is a good quick reference.
>
> On Wed, Apr 22, 2015 at 11:31 AM, Szehon Ho  wrote:
> > Yes, I don't think it is fair to fork it this week, the change in
> question
> > HIVE-10347 was reviewed and waiting 24 hours, but then svn repo is
> suddenly
> > locked, and now the merge process has to be done again in git?  I would
> > vote to push it back one week to give everyone a chance to figure this
> out,
> > like how to do merges now.
> >
> > Thanks
> > Szehon
> >
> >
> >
> > On Wed, Apr 22, 2015 at 11:10 AM, Xuefu Zhang 
> wrote:
> >
> >> Since the svn repo is locked for git migration and a queue of patches
> are
> >> waiting to be committed, I'm not sure if forking on this Friday is
> >> realistic.
> >>
> >> HIVE-9726 is part of the merge (HIVE-10347) which is also in the queue.
> >>
> >> Thanks,
> >> Xuefu
> >>
> >> On Wed, Apr 22, 2015 at 9:55 AM, Sushanth Sowmyan 
> >> wrote:
> >>
> >> > @Lefty :
> >> >
> >> > Yup, I've been following the other thread and
> >> > https://issues.apache.org/jira/browse/INFRA-9488 - I think that'll
> >> > stabilize before we fork. I'll definitely wait for it, but also wanted
> >> > to do a reminder/call-for-patches before it. :)
> >> >
> >> > @Youngwoo:
> >> >
> >> > Thanks, I'll add that to the tracker. I do not see a commit on it in
> >> > trunk for it, so I'll ping Brock on that bug to see if it's included
> >> > through some other jira on trunk.
> >> >
> >> >
> >> >
> >> > On Tue, Apr 21, 2015 at 10:58 PM, 김영우  wrote:
> >> > > Hi,
> >> > >
> >> > > I would like to add a request for support Spark 1.3, HIVE-9726.
> >> Currently
> >> > > Apache Bigtop includes Spark 1.3 and we ran into a compatibility
> issue
> >> > for
> >> > > Hive 1.1+.
> >> > > The issue have been resolved on 'spark-branch'  but could not find a
> >> 'fix
> >> > > version' for it. I hope that the fix merge into release 1.2.
> >> > >
> >> > > Thanks,
> >> > > Youngwoo
> >> > >
> >> > > On Wed, Apr 22, 2015 at 12:33 PM, Sushanth Sowmyan <
> khorg...@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > >> Hi Folks,
> >> > >>
> >> > >> Per my mail 3 weeks back, we should start getting ready to release
> 1.2
> >> > >> as a rollup. And as per my proposal to manage this release, I'd
> like
> >> > >> to start off the process of forking 1.2, and making trunk 1.3.
> >> > >>
> >> > >> I've set up a cwiki page for people to land development patches
> that
> >> > >> are almost done, to signal their desire that this be included in
> 1.2 :
> >> > >>
> >> >
> https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
> >> > >>
> >> > >> A rough timeline I see for this process would be to fork this
> Friday
> >> > >> (24th Apr), and then start rolling out RC0 by, say, Wednesday next
> >> > >> week. This would mean that I would request that if you want your
> jira
> >> > >> included in 1.2, it be close to completion, or have a patch
> available
> >> > >> for review. By mid next week, also, I expect to freeze the wiki
> >> > >> inclusion list for features, and keep it open only for bugfixes
> >> > >> discovered during testing the various RCs.
> >> > >>
> >> > >> Please feel free to edit that jira with your requests, or, if you
> >> > >> don't have edit privileges, if you reply to this mail,

[jira] [Created] (HIVE-10446) Hybrid Hybrid Grace Hash Join : java.lang.IllegalArgumentException in Kryo while spilling big table

[Mostafa Mokhtar (JIRA)]

Mostafa Mokhtar created HIVE-10446:
--

 Summary: Hybrid Hybrid Grace Hash Join : 
java.lang.IllegalArgumentException in Kryo while spilling big table
 Key: HIVE-10446
 URL: https://issues.apache.org/jira/browse/HIVE-10446
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.0
Reporter: Mostafa Mokhtar
Assignee: Wei Zheng
 Fix For: 1.2.0


TPC-DS Q85 fails with Kryo exception when spilling big table data.

Query 
{code}
select  substr(r_reason_desc,1,20) as r
   ,avg(wr_return_ship_cost) wq
   ,avg(wr_refunded_cash) ref
   ,avg(wr_fee) fee
 from web_returns, customer_demographics cd1,
  customer_demographics cd2, customer_address, date_dim, reason 
 where 
   cd1.cd_demo_sk = web_returns.wr_refunded_cdemo_sk 
   and cd2.cd_demo_sk = web_returns.wr_returning_cdemo_sk
   and customer_address.ca_address_sk = web_returns.wr_refunded_addr_sk
   and reason.r_reason_sk = web_returns.wr_reason_sk
   and cd1.cd_marital_status = cd2.cd_marital_status
   and cd1.cd_education_status = cd2.cd_education_status
group by r_reason_desc
order by r, wq, ref, fee
limit 100
{code}

Plan 
{code}
OK
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
  Edges:
Map 1 <- Map 4 (BROADCAST_EDGE), Map 5 (BROADCAST_EDGE), Map 6 
(BROADCAST_EDGE), Map 7 (BROADCAST_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
  DagName: mmokhtar_20150422165209_d8eb5634-c19f-4576-9525-cad248c7ca37:5
  Vertices:
Map 1
Map Operator Tree:
TableScan
  alias: web_returns
  filterExpr: (((wr_refunded_addr_sk is not null and 
wr_reason_sk is not null) and wr_refunded_cdemo_sk is not null) and 
wr_returning_cdemo_sk is not null) (type: boolean)
  Statistics: Num rows: 2062802370 Data size: 185695406284 
Basic stats: COMPLETE Column stats: COMPLETE
  Filter Operator
predicate: (((wr_refunded_addr_sk is not null and 
wr_reason_sk is not null) and wr_refunded_cdemo_sk is not null) and 
wr_returning_cdemo_sk is not null) (type: boolean)
Statistics: Num rows: 1875154723 Data size: 51267313780 
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
  expressions: wr_refunded_cdemo_sk (type: int), 
wr_refunded_addr_sk (type: int), wr_returning_cdemo_sk (type: int), 
wr_reason_sk (type: int), wr_fee (type: float), wr_return_ship_cost (type: 
float), wr_refunded_cash (type: float)
  outputColumnNames: _col0, _col1, _col2, _col3, _col4, 
_col5, _col6
  Statistics: Num rows: 1875154723 Data size: 51267313780 
Basic stats: COMPLETE Column stats: COMPLETE
  Map Join Operator
condition map:
 Inner Join 0 to 1
keys:
  0 _col1 (type: int)
  1 _col0 (type: int)
outputColumnNames: _col0, _col2, _col3, _col4, _col5, 
_col6
input vertices:
  1 Map 4
Statistics: Num rows: 1875154688 Data size: 45003712512 
Basic stats: COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Map Join Operator
  condition map:
   Inner Join 0 to 1
  keys:
0 _col3 (type: int)
1 _col0 (type: int)
  outputColumnNames: _col0, _col2, _col4, _col5, _col6, 
_col9
  input vertices:
1 Map 5
  Statistics: Num rows: 1875154688 Data size: 
219393098496 Basic stats: COMPLETE Column stats: COMPLETE
  HybridGraceHashJoin: true
  Map Join Operator
condition map:
 Inner Join 0 to 1
keys:
  0 _col0 (type: int)
  1 _col0 (type: int)
outputColumnNames: _col2, _col4, _col5, _col6, 
_col9, _col11, _col12
input vertices:
  1 Map 6
Statistics: Num rows: 1875154688 Data size: 
547545168896 Basic stats: COMPLETE Column stats: COMPLETE
HybridGraceHashJoin: true
Map Join Operator
  condition map:
   Inner Join 0 to 1
  keys:

[jira] [Created] (HIVE-10445) Report error when dynamic partition insert is not following the correct syntax

[Chao Sun (JIRA)]

Chao Sun created HIVE-10445:
---

 Summary: Report error when dynamic partition insert is not 
following the correct syntax
 Key: HIVE-10445
 URL: https://issues.apache.org/jira/browse/HIVE-10445
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 1.1.0
Reporter: Chao Sun
Assignee: Chao Sun


With dynamic partition insert, user should follow the syntax as specified in: 
https://cwiki.apache.org/confluence/display/Hive/DynamicPartitions.

However, this is purely enforced on the user side, and there's no checking in 
Hive. As result, this could cause unexpected results for user queries, or 
confusing error messages.

I think we need to explicitly throw error if user query doesn't follow the 
syntax.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: svn commits

[Sergey Shelukhin]

The git repo should be ready to use soon.

On 15/4/22, 10:23, "Ashutosh Chauhan"  wrote:

>are they already turned off because of git transition? I tried a commit on
>svn trunk today morning and got access denied exception. Any one else
>getting this exception?
>If it indeed is because of git transition, do any one know when commits on
>git repo will be enabled?

Re: Preparation for Hive-1.2 release

[Sushanth Sowmyan]

Hi Szehon,

Sergey tells me that he's verifying the repos, and expects to unblock
it in another 20 minutes or so. I think that if HIVE-10347 is ready
for commit now to svn, it should be committable to git as well, as-is
as soon as that comes back up. In that scenario, there's no extra need
to learn how to do merges in git, since I would expect this to be
ready before then. If the .patch file uploaded on HIVE-10347 applies
appropriately on svn, then there should be no issues applying it to
git. I can help there too, if that helps.

Also, I think forking is not that drastic a measure in git as it is in
svn, and merges are a bit easier to manage under git. I would still
like to go ahead with a friday/weekend fork, and allow people to
continue submitting patches for inclusion up until we roll out an RC0,
and then lock it for bugfixes. (Note : If there are any other jiras
you are working on that you want to see in this release, please go add
it to that wiki list :
https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
)

I would like to avoid a long-tail process as happened for 0.12.0 &
0.13.0 if possible. This way, I think we'll have more time for
testing/solidifying.

Thanks,
-Sushanth

P.S: Also, for additional reference, from the svn->git thread,
http://accumulo.apache.org/git.html is a good quick reference.

On Wed, Apr 22, 2015 at 11:31 AM, Szehon Ho  wrote:
> Yes, I don't think it is fair to fork it this week, the change in question
> HIVE-10347 was reviewed and waiting 24 hours, but then svn repo is suddenly
> locked, and now the merge process has to be done again in git?  I would
> vote to push it back one week to give everyone a chance to figure this out,
> like how to do merges now.
>
> Thanks
> Szehon
>
>
>
> On Wed, Apr 22, 2015 at 11:10 AM, Xuefu Zhang  wrote:
>
>> Since the svn repo is locked for git migration and a queue of patches are
>> waiting to be committed, I'm not sure if forking on this Friday is
>> realistic.
>>
>> HIVE-9726 is part of the merge (HIVE-10347) which is also in the queue.
>>
>> Thanks,
>> Xuefu
>>
>> On Wed, Apr 22, 2015 at 9:55 AM, Sushanth Sowmyan 
>> wrote:
>>
>> > @Lefty :
>> >
>> > Yup, I've been following the other thread and
>> > https://issues.apache.org/jira/browse/INFRA-9488 - I think that'll
>> > stabilize before we fork. I'll definitely wait for it, but also wanted
>> > to do a reminder/call-for-patches before it. :)
>> >
>> > @Youngwoo:
>> >
>> > Thanks, I'll add that to the tracker. I do not see a commit on it in
>> > trunk for it, so I'll ping Brock on that bug to see if it's included
>> > through some other jira on trunk.
>> >
>> >
>> >
>> > On Tue, Apr 21, 2015 at 10:58 PM, 김영우  wrote:
>> > > Hi,
>> > >
>> > > I would like to add a request for support Spark 1.3, HIVE-9726.
>> Currently
>> > > Apache Bigtop includes Spark 1.3 and we ran into a compatibility issue
>> > for
>> > > Hive 1.1+.
>> > > The issue have been resolved on 'spark-branch'  but could not find a
>> 'fix
>> > > version' for it. I hope that the fix merge into release 1.2.
>> > >
>> > > Thanks,
>> > > Youngwoo
>> > >
>> > > On Wed, Apr 22, 2015 at 12:33 PM, Sushanth Sowmyan > >
>> > > wrote:
>> > >
>> > >> Hi Folks,
>> > >>
>> > >> Per my mail 3 weeks back, we should start getting ready to release 1.2
>> > >> as a rollup. And as per my proposal to manage this release, I'd like
>> > >> to start off the process of forking 1.2, and making trunk 1.3.
>> > >>
>> > >> I've set up a cwiki page for people to land development patches that
>> > >> are almost done, to signal their desire that this be included in 1.2 :
>> > >>
>> > https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
>> > >>
>> > >> A rough timeline I see for this process would be to fork this Friday
>> > >> (24th Apr), and then start rolling out RC0 by, say, Wednesday next
>> > >> week. This would mean that I would request that if you want your jira
>> > >> included in 1.2, it be close to completion, or have a patch available
>> > >> for review. By mid next week, also, I expect to freeze the wiki
>> > >> inclusion list for features, and keep it open only for bugfixes
>> > >> discovered during testing the various RCs.
>> > >>
>> > >> Please feel free to edit that jira with your requests, or, if you
>> > >> don't have edit privileges, if you reply to this mail, I can add it
>> > >> in. (Also, if you don't have wiki edit privileges, you should probably
>> > >> ask for it. :p)
>> > >>
>> > >> Thanks!
>> > >> -Sushanth
>> > >>
>> >
>>

[jira] [Created] (HIVE-10444) HIVE-10223 breaks hadoop-1 build

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-10444:


 Summary: HIVE-10223 breaks hadoop-1 build
 Key: HIVE-10444
 URL: https://issues.apache.org/jira/browse/HIVE-10444
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Prasanth Jayachandran


FileStatus.isFile() and FileStatus.isDirectory() methods are not present in 
hadoop 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: svn commits

[Jimmy Xiang]

I think it is better to not use the svn any more. Otherwise, we may need to
find those changes in svn but not in git and apply them to git again, most
likely, manually.

On Wed, Apr 22, 2015 at 11:27 AM, Szehon Ho  wrote:

> I spent about few days trying to merge spark branch, including this change,
> and was waiting the 24 hours but now the svn repo is locked.
>
> Is the svn repo read-only for good now?
>
> Szehon
>
>
>
> On Wed, Apr 22, 2015 at 11:06 AM, Jimmy Xiang  wrote:
>
> > The svn tree is locked due to migrating to git. See INFRA-9497 for more
> > info.
> >
> > You can follow INFRA-9488 for the progress.
> >
> >
> > On Wed, Apr 22, 2015 at 10:23 AM, Ashutosh Chauhan  >
> > wrote:
> >
> > > are they already turned off because of git transition? I tried a commit
> > on
> > > svn trunk today morning and got access denied exception. Any one else
> > > getting this exception?
> > > If it indeed is because of git transition, do any one know when commits
> > on
> > > git repo will be enabled?
> > >
> >
>

[jira] [Created] (HIVE-10443) HIVE-9870 broke hadoop-1 build

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-10443:


 Summary: HIVE-9870 broke hadoop-1 build
 Key: HIVE-10443
 URL: https://issues.apache.org/jira/browse/HIVE-10443
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Prasanth Jayachandran


JvmPauseMonitor added in HIVE-9870 is breaking hadoop-1 build. 
HiveServer2.startPauseMonitor() does not use reflection properly to start 
JvmPauseMonitor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10442) HIVE-10098 broke hadoop-1 build

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-10442:


 Summary: HIVE-10098 broke hadoop-1 build
 Key: HIVE-10442
 URL: https://issues.apache.org/jira/browse/HIVE-10442
 Project: Hive
  Issue Type: Bug
Reporter: Prasanth Jayachandran


fs.addDelegationTokens() method does not seem to exist in hadoop 1.2.1. This 
breaks the hadoop-1 builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10441) Fix confusing log statement in SessionState about hive.execution.engine setting

[Jason Dere (JIRA)]

Jason Dere created HIVE-10441:
-

 Summary: Fix confusing log statement in SessionState about 
hive.execution.engine setting
 Key: HIVE-10441
 URL: https://issues.apache.org/jira/browse/HIVE-10441
 Project: Hive
  Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere



{code}
LOG.info("No Tez session required at this point. hive.execution.engine=mr.");
{code}

This statement is misleading. It is true that it is printed in the case that 
Tez session does not need to be created, but it is not necessarily true that 
hive.execution.engine=mr - it could be Spark, or it could even be set to Tez 
but the Session determined that a Tez Session did not need to be created (which 
is the case for HiveServer2).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Preparation for Hive-1.2 release

[Szehon Ho]

Yes, I don't think it is fair to fork it this week, the change in question
HIVE-10347 was reviewed and waiting 24 hours, but then svn repo is suddenly
locked, and now the merge process has to be done again in git?  I would
vote to push it back one week to give everyone a chance to figure this out,
like how to do merges now.

Thanks
Szehon



On Wed, Apr 22, 2015 at 11:10 AM, Xuefu Zhang  wrote:

> Since the svn repo is locked for git migration and a queue of patches are
> waiting to be committed, I'm not sure if forking on this Friday is
> realistic.
>
> HIVE-9726 is part of the merge (HIVE-10347) which is also in the queue.
>
> Thanks,
> Xuefu
>
> On Wed, Apr 22, 2015 at 9:55 AM, Sushanth Sowmyan 
> wrote:
>
> > @Lefty :
> >
> > Yup, I've been following the other thread and
> > https://issues.apache.org/jira/browse/INFRA-9488 - I think that'll
> > stabilize before we fork. I'll definitely wait for it, but also wanted
> > to do a reminder/call-for-patches before it. :)
> >
> > @Youngwoo:
> >
> > Thanks, I'll add that to the tracker. I do not see a commit on it in
> > trunk for it, so I'll ping Brock on that bug to see if it's included
> > through some other jira on trunk.
> >
> >
> >
> > On Tue, Apr 21, 2015 at 10:58 PM, 김영우  wrote:
> > > Hi,
> > >
> > > I would like to add a request for support Spark 1.3, HIVE-9726.
> Currently
> > > Apache Bigtop includes Spark 1.3 and we ran into a compatibility issue
> > for
> > > Hive 1.1+.
> > > The issue have been resolved on 'spark-branch'  but could not find a
> 'fix
> > > version' for it. I hope that the fix merge into release 1.2.
> > >
> > > Thanks,
> > > Youngwoo
> > >
> > > On Wed, Apr 22, 2015 at 12:33 PM, Sushanth Sowmyan  >
> > > wrote:
> > >
> > >> Hi Folks,
> > >>
> > >> Per my mail 3 weeks back, we should start getting ready to release 1.2
> > >> as a rollup. And as per my proposal to manage this release, I'd like
> > >> to start off the process of forking 1.2, and making trunk 1.3.
> > >>
> > >> I've set up a cwiki page for people to land development patches that
> > >> are almost done, to signal their desire that this be included in 1.2 :
> > >>
> > https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
> > >>
> > >> A rough timeline I see for this process would be to fork this Friday
> > >> (24th Apr), and then start rolling out RC0 by, say, Wednesday next
> > >> week. This would mean that I would request that if you want your jira
> > >> included in 1.2, it be close to completion, or have a patch available
> > >> for review. By mid next week, also, I expect to freeze the wiki
> > >> inclusion list for features, and keep it open only for bugfixes
> > >> discovered during testing the various RCs.
> > >>
> > >> Please feel free to edit that jira with your requests, or, if you
> > >> don't have edit privileges, if you reply to this mail, I can add it
> > >> in. (Also, if you don't have wiki edit privileges, you should probably
> > >> ask for it. :p)
> > >>
> > >> Thanks!
> > >> -Sushanth
> > >>
> >
>

Re: svn commits

[Szehon Ho]

I spent about few days trying to merge spark branch, including this change,
and was waiting the 24 hours but now the svn repo is locked.

Is the svn repo read-only for good now?

Szehon



On Wed, Apr 22, 2015 at 11:06 AM, Jimmy Xiang  wrote:

> The svn tree is locked due to migrating to git. See INFRA-9497 for more
> info.
>
> You can follow INFRA-9488 for the progress.
>
>
> On Wed, Apr 22, 2015 at 10:23 AM, Ashutosh Chauhan 
> wrote:
>
> > are they already turned off because of git transition? I tried a commit
> on
> > svn trunk today morning and got access denied exception. Any one else
> > getting this exception?
> > If it indeed is because of git transition, do any one know when commits
> on
> > git repo will be enabled?
> >
>

Re: Preparation for Hive-1.2 release

[Xuefu Zhang]

Since the svn repo is locked for git migration and a queue of patches are
waiting to be committed, I'm not sure if forking on this Friday is
realistic.

HIVE-9726 is part of the merge (HIVE-10347) which is also in the queue.

Thanks,
Xuefu

On Wed, Apr 22, 2015 at 9:55 AM, Sushanth Sowmyan 
wrote:

> @Lefty :
>
> Yup, I've been following the other thread and
> https://issues.apache.org/jira/browse/INFRA-9488 - I think that'll
> stabilize before we fork. I'll definitely wait for it, but also wanted
> to do a reminder/call-for-patches before it. :)
>
> @Youngwoo:
>
> Thanks, I'll add that to the tracker. I do not see a commit on it in
> trunk for it, so I'll ping Brock on that bug to see if it's included
> through some other jira on trunk.
>
>
>
> On Tue, Apr 21, 2015 at 10:58 PM, 김영우  wrote:
> > Hi,
> >
> > I would like to add a request for support Spark 1.3, HIVE-9726. Currently
> > Apache Bigtop includes Spark 1.3 and we ran into a compatibility issue
> for
> > Hive 1.1+.
> > The issue have been resolved on 'spark-branch'  but could not find a 'fix
> > version' for it. I hope that the fix merge into release 1.2.
> >
> > Thanks,
> > Youngwoo
> >
> > On Wed, Apr 22, 2015 at 12:33 PM, Sushanth Sowmyan 
> > wrote:
> >
> >> Hi Folks,
> >>
> >> Per my mail 3 weeks back, we should start getting ready to release 1.2
> >> as a rollup. And as per my proposal to manage this release, I'd like
> >> to start off the process of forking 1.2, and making trunk 1.3.
> >>
> >> I've set up a cwiki page for people to land development patches that
> >> are almost done, to signal their desire that this be included in 1.2 :
> >>
> https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
> >>
> >> A rough timeline I see for this process would be to fork this Friday
> >> (24th Apr), and then start rolling out RC0 by, say, Wednesday next
> >> week. This would mean that I would request that if you want your jira
> >> included in 1.2, it be close to completion, or have a patch available
> >> for review. By mid next week, also, I expect to freeze the wiki
> >> inclusion list for features, and keep it open only for bugfixes
> >> discovered during testing the various RCs.
> >>
> >> Please feel free to edit that jira with your requests, or, if you
> >> don't have edit privileges, if you reply to this mail, I can add it
> >> in. (Also, if you don't have wiki edit privileges, you should probably
> >> ask for it. :p)
> >>
> >> Thanks!
> >> -Sushanth
> >>
>

Re: Review Request 33004: HIVE-10275 GenericUDF getTimestampValue should return Timestamp instead of Date

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33004/#review81192
---

Ship it!


Ship It!

- Alexander Pivovarov


On April 9, 2015, 4:32 a.m., Alexander Pivovarov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33004/
> ---
> 
> (Updated April 9, 2015, 4:32 a.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Bugs: HIVE-10275
> https://issues.apache.org/jira/browse/HIVE-10275
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-10275 GenericUDF getTimestampValue should return Timestamp instead of 
> Date
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 
> 513d030d975fef6dce5e95aac67a87f14b2a3a8d 
> 
> Diff: https://reviews.apache.org/r/33004/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Pivovarov
> 
>

Re: svn commits

[Jimmy Xiang]

The svn tree is locked due to migrating to git. See INFRA-9497 for more
info.

You can follow INFRA-9488 for the progress.


On Wed, Apr 22, 2015 at 10:23 AM, Ashutosh Chauhan 
wrote:

> are they already turned off because of git transition? I tried a commit on
> svn trunk today morning and got access denied exception. Any one else
> getting this exception?
> If it indeed is because of git transition, do any one know when commits on
> git repo will be enabled?
>

Re: svn commits

[Prasanth Jayachandran]

Yes. SVN is made read only yesterday for git migration.

Thanks
Prasanth




On Wed, Apr 22, 2015 at 10:24 AM -0700, "Ashutosh Chauhan" 
mailto:hashut...@apache.org>> wrote:

are they already turned off because of git transition? I tried a commit on
svn trunk today morning and got access denied exception. Any one else
getting this exception?
If it indeed is because of git transition, do any one know when commits on
git repo will be enabled?

Re: svn commits

[Xuefu Zhang]

Yes. I believe so. You can follow the progress in INFRA-9488

On Wed, Apr 22, 2015 at 10:23 AM, Ashutosh Chauhan 
wrote:

> are they already turned off because of git transition? I tried a commit on
> svn trunk today morning and got access denied exception. Any one else
> getting this exception?
> If it indeed is because of git transition, do any one know when commits on
> git repo will be enabled?
>

svn commits

[Ashutosh Chauhan]

are they already turned off because of git transition? I tried a commit on
svn trunk today morning and got access denied exception. Any one else
getting this exception?
If it indeed is because of git transition, do any one know when commits on
git repo will be enabled?

Re: Preparation for Hive-1.2 release

[Sushanth Sowmyan]

@Lefty :

Yup, I've been following the other thread and
https://issues.apache.org/jira/browse/INFRA-9488 - I think that'll
stabilize before we fork. I'll definitely wait for it, but also wanted
to do a reminder/call-for-patches before it. :)

@Youngwoo:

Thanks, I'll add that to the tracker. I do not see a commit on it in
trunk for it, so I'll ping Brock on that bug to see if it's included
through some other jira on trunk.



On Tue, Apr 21, 2015 at 10:58 PM, 김영우  wrote:
> Hi,
>
> I would like to add a request for support Spark 1.3, HIVE-9726. Currently
> Apache Bigtop includes Spark 1.3 and we ran into a compatibility issue for
> Hive 1.1+.
> The issue have been resolved on 'spark-branch'  but could not find a 'fix
> version' for it. I hope that the fix merge into release 1.2.
>
> Thanks,
> Youngwoo
>
> On Wed, Apr 22, 2015 at 12:33 PM, Sushanth Sowmyan 
> wrote:
>
>> Hi Folks,
>>
>> Per my mail 3 weeks back, we should start getting ready to release 1.2
>> as a rollup. And as per my proposal to manage this release, I'd like
>> to start off the process of forking 1.2, and making trunk 1.3.
>>
>> I've set up a cwiki page for people to land development patches that
>> are almost done, to signal their desire that this be included in 1.2 :
>> https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
>>
>> A rough timeline I see for this process would be to fork this Friday
>> (24th Apr), and then start rolling out RC0 by, say, Wednesday next
>> week. This would mean that I would request that if you want your jira
>> included in 1.2, it be close to completion, or have a patch available
>> for review. By mid next week, also, I expect to freeze the wiki
>> inclusion list for features, and keep it open only for bugfixes
>> discovered during testing the various RCs.
>>
>> Please feel free to edit that jira with your requests, or, if you
>> don't have edit privileges, if you reply to this mail, I can add it
>> in. (Also, if you don't have wiki edit privileges, you should probably
>> ask for it. :p)
>>
>> Thanks!
>> -Sushanth
>>

Re: Review Request 33367: Aggregate stats cache for RDBMS based metastore codepath

[Mostafa Mokhtar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33367/#review81170
---



metastore/src/java/org/apache/hadoop/hive/metastore/AggregateStatsCache.java


Can you keeep track of hit ratio?



metastore/src/java/org/apache/hadoop/hive/metastore/AggregateStatsCache.java


Can you add logging on how many nodes where cleaned in how long.
Also print number of nodes before/after


- Mostafa Mokhtar


On April 20, 2015, 6:44 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33367/
> ---
> 
> (Updated April 20, 2015, 6:44 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-10382
> https://issues.apache.org/jira/browse/HIVE-10382
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Similar to the work done on the HBase branch (HIVE-9693), the stats cache can 
> potentially have performance gains.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 65ec1b9 
>   common/src/java/org/apache/hive/common/util/BloomFilter.java PRE-CREATION 
>   common/src/java/org/apache/hive/common/util/Murmur3.java PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestBloomFilter.java 
> PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestMurmur3.java PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/AggregateStatsCache.java 
> PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> bf169c9 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/TestAggregateStatsCache.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/filters/BloomFilter.java 6ab0270 
>   ql/src/java/org/apache/hadoop/hive/ql/io/filters/BloomFilterIO.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/filters/Murmur3.java e733892 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileDump.java 7bfd781 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java 49a8e80 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java bde9fc2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java a319204 
>   ql/src/test/org/apache/hadoop/hive/ql/io/filters/TestBloomFilter.java 
> 32b95ab 
>   ql/src/test/org/apache/hadoop/hive/ql/io/filters/TestMurmur3.java d92a3ce 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java 
> d0f3a5e 
> 
> Diff: https://reviews.apache.org/r/33367/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-10440) Architecture for ResultSet Compression via external plugin

[Rohit Dholakia (JIRA)]

Rohit Dholakia created HIVE-10440:
-

 Summary: Architecture for  ResultSet Compression via external 
plugin
 Key: HIVE-10440
 URL: https://issues.apache.org/jira/browse/HIVE-10440
 Project: Hive
  Issue Type: New Feature
  Components: Hive, Thrift API
Affects Versions: 1.1.0
Reporter: Rohit Dholakia


This JIRA proposes an architecture for enabling ResultSet compression which 
uses an external plugin. 

The patch has three aspects to it: 
0. An architecture for enabling ResultSet compression with external plugins
1. An example plugin to demonstrate end-to-end functionality 
2. A container to allow everyone to write and test ResultSet compressors.

Also attaching a design document explaining the changes, experimental results 
document, and a pdf explaining how to setup the docker container to observe 
end-to-end functionality of ResultSet compression. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10439) Architecture for ResultSet Compression via external plugin

[Rohit Dholakia (JIRA)]

Rohit Dholakia created HIVE-10439:
-

 Summary: Architecture for  ResultSet Compression via external 
plugin
 Key: HIVE-10439
 URL: https://issues.apache.org/jira/browse/HIVE-10439
 Project: Hive
  Issue Type: New Feature
  Components: Hive, Thrift API
Affects Versions: 1.1.0
Reporter: Rohit Dholakia


This JIRA proposes an architecture for enabling ResultSet compression which 
uses an external plugin. 

The patch has three aspects to it: 
0. An architecture for enabling ResultSet compression with external plugins
1. An example plugin to demonstrate end-to-end functionality 
2. A container to allow everyone to write and test ResultSet compressors.

Also attaching a design document explaining the changes, experimental results 
document, and a pdf explaining how to setup the docker container to observe 
end-to-end functionality of ResultSet compression. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33251: HIVE-10302 Cache small tables in memory [Spark Branch]

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33251/
---

(Updated April 22, 2015, 4:36 p.m.)


Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.


Changes
---

Addressed Xuefu's review comments: removed threadlocal variable, added some 
javadoc, fixed some code clarification issue.
In this patch, we still clean up cache based on work id so that we can avoid 
extra memory usage for other works in the same job. Unfortunately, this means, 
if there are other works running in parallel with the mapjoin work, the cache 
may be released when it can still be kept for a while.


Bugs: HIVE-10302
https://issues.apache.org/jira/browse/HIVE-10302


Repository: hive-git


Description
---

Cached the small table containter so that mapjoin tasks can use it if the task 
is executed on the same Spark executor.
The cache is released right before the next job after the mapjoin job is done.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java fe108c4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java 
2f137f9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 
3f240f5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 72ab913 

Diff: https://reviews.apache.org/r/33251/diff/


Testing
---

Ran several queries in live cluster. ptest pending.


Thanks,

Jimmy Xiang

[jira] [Created] (HIVE-10438) Architecture for ResultSet Compression via external plugin

[Rohit Dholakia (JIRA)]

Rohit Dholakia created HIVE-10438:
-

 Summary: Architecture for  ResultSet Compression via external 
plugin
 Key: HIVE-10438
 URL: https://issues.apache.org/jira/browse/HIVE-10438
 Project: Hive
  Issue Type: New Feature
  Components: Hive, Thrift API
Affects Versions: 1.1.0
Reporter: Rohit Dholakia


This JIRA proposes an architecture for enabling ResultSet compression which 
uses an external plugin. 

The patch has three aspects to it: 
0. An architecture for enabling ResultSet compression with external plugins
1. An example plugin to demonstrate end-to-end functionality 
2. A container to allow everyone to write and test ResultSet compressors.

Also attaching a design document explaining the changes, experimental results 
document, and a pdf explaining how to setup the docker container to observe 
end-to-end functionality of ResultSet compression. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33251: HIVE-10302 Cache small tables in memory [Spark Branch]

[Jimmy Xiang]

> On April 21, 2015, 2:17 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java, line 
> > 120
> > 
> >
> > 1. For clarity, it might be good to put this in a separate private 
> > method.
> > 2. Does it work if we just synchronize on mapJoinTables[pos]?

1. Have put it in a separate method. 2. Probabl it does work to synchronize on 
mapJoinTables[pos] since it is not the same object in different tasks.


> On April 21, 2015, 2:17 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java, 
> > line 52
> > 
> >
> > Method naming, see below.

Fixed.  Changed the method a little so that it doesn't do more than clean up 
small table caches.


> On April 21, 2015, 2:17 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java, line 
> > 51
> > 
> >
> > Using thread-local makes me a little nervous, but let's discuss about 
> > this offline.

Per our discussion, we don't use thread-local any more.


> On April 21, 2015, 2:17 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java, line 
> > 60
> > 
> >
> > The method name suggests no indication of a side effect of setting 
> > thread local value. We'd better put this outside of this method.
> > 
> > In addition, the method name seems also a little confusing in that it 
> > suggests cleanup is for sure but in fact it's conditional.

Removed the thread local variable usage.  As to the method name, it means we 
clean up the cache a little, it doesn't mean we remove all the cahced contents. 
I added some javadoc to make it a little clear.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33251/#review80969
---


On April 21, 2015, 1:37 a.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33251/
> ---
> 
> (Updated April 21, 2015, 1:37 a.m.)
> 
> 
> Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-10302
> https://issues.apache.org/jira/browse/HIVE-10302
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Cached the small table containter so that mapjoin tasks can use it if the 
> task is executed on the same Spark executor.
> The cache is released right before the next job after the mapjoin job is done.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java 
> fe108c4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 
> 3f240f5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
> 97b3471 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 
> 72ab913 
> 
> Diff: https://reviews.apache.org/r/33251/diff/
> 
> 
> Testing
> ---
> 
> Ran several queries in live cluster. ptest pending.
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

[jira] [Created] (HIVE-10437) NullPointerException on queries where map/reduce is not involved on tables with partitions

[Demeter Sztanko (JIRA)]

Demeter Sztanko created HIVE-10437:
--

 Summary: NullPointerException on queries where map/reduce is not 
involved on tables with partitions
 Key: HIVE-10437
 URL: https://issues.apache.org/jira/browse/HIVE-10437
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Demeter Sztanko
Priority: Minor


On a table with partitions, whenever I try to do a simple query which tells 
hive not to execute mapreduce but just read data straight from hdfs, it raises 
an exception:
{code}
create external table jsonbug(
a int,
b string
)
PARTITIONED BY (
`c` string)
ROW FORMAT SERDE
  'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
  'ignore.malformed.json'='true')
STORED AS INPUTFORMAT
  'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  '/tmp/jsonbug';

ALTER TABLE jsonbug ADD PARTITION(c='1');
{code}

Runnin simple 
{code}
select * from jsonbug;
{code}

Raises the following exception:
{code}
FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: 
Failed with exception nulljava.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.needConversion(FetchOperator.java:607)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:578)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:455)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{code}

It works fine if I execute a query involving map/reduce job though.

This problem occurs only when using SerDe's created for hive versions pre 
1.1.0, those which do not have @SerDeSpec annotation specified. Most of the 
third party SerDE's, including hcat's JsonSerde have this problem as well. 
It seems like changes made in HIVE-7977 introduce this bug. See 
org.apache.hadoop.hive.ql.exec.FetchOperator.needConversion(FetchOperator.java:607)
{code}
Class tableSerDe = tableDesc.getDeserializerClass();
String[] schemaProps = AnnotationUtils.getAnnotation(tableSerDe, 
SerDeSpec.class).schemaProps();
{code}
And it also seems like a relatively easy fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: HIVE-TRUNK-JAVA8 #62

[hiveqa]

See 


Changes:

[hashutosh] HIVE-10396 : decimal_precision2.q test is failing on trunk (Jason 
Dere via Ashutosh Chauhan)

[prasanthj] HIVE-10303: HIVE-9471 broke forward compatibility of ORC files 
(Prasanth Jayachandran reviewed by Owen O'Malley)

[Szehon Ho] HIVE-9923 : No clear message when from is missing (Yongzhi Chen via 
Szehon)

[vikram] HIVE-10352: Investigate the failure of union_view.q (Vikram Dixit, 
reviewed by Thejas Nair)

[hashutosh] HIVE-10400 : CBO (Calcite Return Path): Exception when column name 
contains dot or colon characters (Jesus Camacho Rodriguez via John Pullokkaran)

[ekoifman] HIVE-7948 - Add an E2E test  to verify fix for HIVE-7155 (Aswathy 
Chellammal Sreekumar via Eugene Koifman)

[hashutosh] HIVE-10268 : Merge cbo branch into trunk (Ashutosh Chauhan)

--
[...truncated 22523 lines...]
2015-04-22 08:50:50,023  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestCliDriver-compute_stats_string.q-load_dyn_part12.q-nullgroup4_multi_distinct.q-and-12-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
2015-04-22 08:50:50,023  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestUDFMath-TEST-org.apache.hadoop.hive.ql.udf.TestUDFMath.xml
2015-04-22 08:50:50,023  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestCliDriver-alter_partition_clusterby_sortby.q-udf_repeat.q-rcfile_merge2.q-and-12-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
2015-04-22 08:50:50,023  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestHBaseCliDriver-hbase_binary_map_queries.q-TEST-org.apache.hadoop.hive.cli.TestHBaseCliDriver.xml
2015-04-22 08:50:50,024  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestPartitionKeySampler-TEST-org.apache.hadoop.hive.ql.exec.TestPartitionKeySampler.xml
2015-04-22 08:50:50,024  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestInstanceCache-TEST-org.apache.hadoop.hive.serde2.avro.TestInstanceCache.xml
2015-04-22 08:50:50,025  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestParquetRowGroupFilter-TEST-org.apache.hadoop.hive.ql.io.parquet.TestParquetRowGroupFilter.xml
2015-04-22 08:50:50,025  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestTypeInfoToSchema-TEST-org.apache.hadoop.hive.serde2.avro.TestTypeInfoToSchema.xml
2015-04-22 08:50:50,025  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestObjectInspectorUtils-TEST-org.apache.hadoop.hive.serde2.objectinspector.TestObjectInspectorUtils.xml
2015-04-22 08:50:50,025  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestHiveSQLException-TEST-org.apache.hive.service.cli.TestHiveSQLException.xml
2015-04-22 08:50:50,026  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestGenericUDFRTrim-TEST-org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFRTrim.xml
2015-04-22 08:50:50,026  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestDynamicSerDe-TEST-org.apache.hadoop.hive.serde2.dynamic_type.TestDynamicSerDe.xml
2015-04-22 08:50:50,026  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestCliDriver-create_genericudf.q-ambiguitycheck.q-join13.q-and-12-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
2015-04-22 08:50:50,026  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestStreamName-TEST-org.apache.hadoop.hive.ql.io.orc.TestStreamName.xml
2015-04-22 08:50:50,027  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestSqlMathUtil-TEST-org.apache.hadoop.hive.common.type.TestSqlMathUtil.xml
2015-04-22 08:50:50,027  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestHCatLoader-TEST-org.apache.hive.hcatalog.pig.TestHCatLoader.xml
2015-04-22 08:50:50,037  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestCliSessionState-TEST-org.apache.hadoop.hive.cli.TestCliSessionState.xml
2015-04-22 08:50:50,037  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestCliDriver-infer_bucket_sort_grouping_operators.q-authorization_set_show_current_role.q-union21.q-and-12-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
2015-04-22 08:50:50,038  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestLazyBinarySerDe-TEST-org.apache.hadoop.hive.serde2.lazybinary.TestLazyBinarySerDe.xml
2015-04-22 08:50:50,039  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestFullMapEqualComparer-TEST-org.apache.hadoop.hive.serde2.objectinspector.TestFullMapEqualComparer.xml
2015-04-22 08:50:50,039  INFO LocalCommand$CollectLogPolicy.handleOutput:69 
test-results/TEST-TestMapredParquetOutputFormat-TEST-org.apache.hadoop.hive.ql.io.parquet.TestMapredParquetOutputFormat.xml
2015-0

[jira] [Created] (HIVE-10436) DEBUG lockmgr.DummyTxnManager: Adding database :mkt_incoming to list of lock outputs

[ankush (JIRA)]

ankush created HIVE-10436:
-

 Summary: DEBUG lockmgr.DummyTxnManager: Adding database 
:mkt_incoming to list of lock outputs
 Key: HIVE-10436
 URL: https://issues.apache.org/jira/browse/HIVE-10436
 Project: Hive
  Issue Type: Bug
  Components: Hive, HiveServer2
Reporter: ankush


Hi,

In Hadoop, while creating Table in hive i am getting stuck in below error

15/04/21 12:35:34 INFO log.PerfLogger: 
15/04/21 12:35:34 INFO log.PerfLogger: 
15/04/21 12:35:34 INFO log.PerfLogger: 
15/04/21 12:35:34 INFO lockmgr.DummyTxnManager: Creating lock manager of type 
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
15/04/21 12:35:34 INFO zookeeper.ZooKeeper: Initiating client connection, 
connectString=dkhc3013.dcsg.com:2181,dkhc3010.dcsg.com:2181,dkhc3011.dcsg.com:2181
 sessionTimeout=60 
watcher=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager$DummyWatcher@5b9e1cd4
15/04/21 12:35:34 DEBUG lockmgr.DummyTxnManager: Adding 
/incoming/mkt/gcdb.etl_master_account_pref to list of lock inputs
15/04/21 12:35:34 DEBUG lockmgr.DummyTxnManager: Adding database:mkt_incoming 
to list of lock outputs

After restart the zookeeper service i am able to successfully run the query,

But after some time again facing the same issue/error, I am stuck on the same 
error.

is there any solution to overcome this issue, or any tuning i can do for 
resolve this issue. ?

Please suggest on this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: hive on tez optimize MRR to MR?

[Gopal Vijayaraghavan]

To prevent bad reducer merging, the reducer merging only kicks in when the
optimizer thinks it gets a perf boost.

MR -> MRR is not a big win when it comes Tez, due to container-reuse -
going wide on the large cardinality in case of missing map-side
aggregation will be safer.

If hive.map.aggr=true and the userid set fits within memory, then smushing
the reducers would be nicer.

To reset the wide-narrow checks, do

set hive.optimize.reducededuplication.min.reducer=1;


But be aware that it will fail (I¹ve seen full disks) as you scale upwards
to the 10+ Tb cases.

Cheers,
Gopal

On 4/22/15, 2:15 PM, "r7raul1...@163.com"  wrote:

>
>
>select userid,count(*) from u_data group by userid order by userid
>will product MRR.
>
>I think when the result of  userid,count(*) is small(one reduce can
>process the result) . This query plan can optimize to MR ?
>
>
>
>
>r7raul1...@163.com

[jira] [Created] (HIVE-10435) Make HiveSession implementation pluggable through configuration

[Amareshwari Sriramadasu (JIRA)]

Amareshwari Sriramadasu created HIVE-10435:
--

 Summary: Make HiveSession implementation pluggable through 
configuration
 Key: HIVE-10435
 URL: https://issues.apache.org/jira/browse/HIVE-10435
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Amareshwari Sriramadasu


SessionManager in CLIService creates and keeps track of HiveSession. 
Right now, it creates HiveSessionImpl which is one implementation of 
HiveSession. This improvement request is to make it pluggable through a 
configuration sothat other implementations can be passed.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

hive on tez optimize MRR to MR?

[r7raul1...@163.com]

select userid,count(*) from u_data group by userid order by useridwill 
product MRR.

I think when the result of  userid,count(*) is small(one reduce can process the 
result) . This query plan can optimize to MR ?




r7raul1...@163.com