Re: Review Request 29898: HIVE-9298: Support reading alternate timestamp formats

[Jason Dere]

> On Jan. 28, 2015, 1:22 a.m., Ashutosh Chauhan wrote:
> > common/pom.xml, lines 59-63
> > 
> >
> > Since joda jar will be shipped to task nodes, this needs to be added in 
> > hive-exec jar. I think we keep that list in one of the pom files. We need 
> > to add this dep there.

Do you mean the artifact set for the shaded JAR goal in ql/pom.xml? I'll take a 
look at doing this.


> On Jan. 28, 2015, 1:22 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hive/common/util/TimestampParser.java, line 76
> > 
> >
> > Name suggests this can be an instance object. If we do that way, than 
> > we can avoid creating this object per invocation, which will be nice if 
> > possible.

The way this is currently set up the LazyTimestampObjectInspector (which I 
believe could be shared by different threads) points to a single 
TimestampParser. The Joda DateTimeFormatter is thread safe, so everything in 
parseTimestamp() should be thread safe except for mdt which is why I was 
creating a new object. I guess mdt could be made thread safe by making it a 
thread-local instance.  I'll make that change.


> On Jan. 28, 2015, 1:22 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hive/common/util/TimestampParser.java, line 127
> > 
> >
> > Can't we do Long.valueOf()? That will be faster than BD parsing, I 
> > presume.

If we don't want to worry about fractional millisecond values, then we can do 
this. We're throwing away the fractional portion anyway since Joda does not 
have precision less than 1 ms. I'll change this.


> On Jan. 28, 2015, 1:22 a.m., Ashutosh Chauhan wrote:
> > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java,
> >  line 114
> > 
> >
> > This is thrift generated file. Instead of hand modifying you need to 
> > put this in thrift file and generate it via thrift compiler.

Whoops missed that, thanks for pointing that out, will fix.


> On Jan. 28, 2015, 1:22 a.m., Ashutosh Chauhan wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java, 
> > lines 135-137
> > 
> >
> > I wonder why these and lastColtakeRest are not included in 
> > LazyOIParams. Seems to me, they should be included too. If you think 
> > otherwise, it will be good to add a comment here about what distinguishes 
> > these two set of params.

So I thought these params had more to do with the SerDe and handling of rows 
than they did with actual values and ObjectInspector-related handling, which is 
why I left those out of the lazy OI params. Admittedly it does look a bit odd 
to bundle some of the params together and leave others out. If you think I 
should just include those in I can do so.


> On Jan. 28, 2015, 1:22 a.m., Ashutosh Chauhan wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java, 
> > line 649
> > 
> >
> > I think there is a helper method in apache commons (or guava) which can 
> > let you do such parsing. Will be good to reuse that, if available.

Not sure if the commons/guava libs have something to escape commas (please 
correct me if I am wrong). I see that Hive uses opencsv which handles CSV-style 
escaping, I will use this to parse the list.


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29898/#review69930
---


On Jan. 20, 2015, 12:34 a.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29898/
> ---
> 
> (Updated Jan. 20, 2015, 12:34 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-9298
> https://issues.apache.org/jira/browse/HIVE-9298
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add new SerDe parameter "timestamp.formats" to specify alternate timestamp 
> patterns
> 
> 
> Diffs
> -
> 
>   common/pom.xml ede8aea 
>   common/src/java/org/apache/hive/common/util/TimestampParser.java 
> PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestTimestampParser.java 
> PRE-CREATION 
>   data/files/ts_formats.txt PRE-CREATION 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/DefaultHBaseKeyFactory.java
>  98bc73f 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java
>  78f23cb 
>   
> hbase-handler/src/java/org/

[jira] [Updated] (HIVE-9486) Use session classloader instead of application loader

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-9486:

Attachment: HIVE-9486.2.patch.txt

> Use session classloader instead of application loader
> -
>
> Key: HIVE-9486
> URL: https://issues.apache.org/jira/browse/HIVE-9486
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-9486.1.patch.txt, HIVE-9486.2.patch.txt
>
>
> From http://www.mail-archive.com/dev@hive.apache.org/msg107615.html
> Looks reasonable



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294911#comment-14294911
 ] 

Lefty Leverenz commented on HIVE-8966:
--

Any documentation needed?

> Delta files created by hive hcatalog streaming cannot be compacted
> --
>
> Key: HIVE-8966
> URL: https://issues.apache.org/jira/browse/HIVE-8966
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
> Environment: hive
>Reporter: Jihong Liu
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: HIVE-8966-branch-1.patch, HIVE-8966.2.patch, 
> HIVE-8966.3.patch, HIVE-8966.4.patch, HIVE-8966.5.patch, HIVE-8966.6.patch, 
> HIVE-8966.patch
>
>
> hive hcatalog streaming will also create a file like bucket_n_flush_length in 
> each delta directory. Where "n" is the bucket number. But the 
> compactor.CompactorMR think this file also needs to compact. However this 
> file of course cannot be compacted, so compactor.CompactorMR will not 
> continue to do the compaction. 
> Did a test, after removed the bucket_n_flush_length file, then the "alter 
> table partition compact" finished successfully. If don't delete that file, 
> nothing will be compacted. 
> This is probably a very severity bug. Both 0.13 and 0.14 have this issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9477) No error thrown when global limit optimization failed to find enough number of rows [Spark Branch]

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-9477:
-
Attachment: HIVE-9477.1-spark.patch

Rerun query when global limit optimization fails.

> No error thrown when global limit optimization failed to find enough number 
> of rows [Spark Branch]
> --
>
> Key: HIVE-9477
> URL: https://issues.apache.org/jira/browse/HIVE-9477
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-9477.1-spark.patch
>
>
> MR will throw an error in such a case and rerun the query with the 
> optimization disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9477) No error thrown when global limit optimization failed to find enough number of rows [Spark Branch]

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-9477:
-
Status: Patch Available  (was: Open)

> No error thrown when global limit optimization failed to find enough number 
> of rows [Spark Branch]
> --
>
> Key: HIVE-9477
> URL: https://issues.apache.org/jira/browse/HIVE-9477
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-9477.1-spark.patch
>
>
> MR will throw an error in such a case and rerun the query with the 
> optimization disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29807: HIVE-9253: MetaStore server should support timeout for long running requests

[Dong Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29807/
---

(Updated Jan. 28, 2015, 8:58 a.m.)


Review request for hive.


Changes
---

Address comments from Lefty and Brock.


Repository: hive-git


Description
---

HIVE-9253: MetaStore server should support timeout for long running requests


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 66f436b 
  metastore/src/java/org/apache/hadoop/hive/metastore/Deadline.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/DeadlineException.java 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
fc6f067 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
574141c 
  metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java 
01ad36a 
  
metastore/src/java/org/apache/hadoop/hive/metastore/SessionPropertiesListener.java
 PRE-CREATION 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestDeadline.java 
PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTimeout.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/29807/diff/


Testing
---

UT passed


Thanks,

Dong Chen

[jira] [Commented] (HIVE-9489) add javadoc for UDFType annotation

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294894#comment-14294894
 ] 

Lefty Leverenz commented on HIVE-9489:
--

Hmph.  Not many typos for me to find.  ;)

{{+   * Certain optimizations should not be applied if UDF is not 
deterministic}}

... needs a period at end of line.

{{+   * don't apply for such UDFS, as they need to be invoked for each record.}}

... UDFs, not UDFS.

{{+   * A UDF is considered distinctLike if the udf can be evaluated on just 
the}}

... udf should be UDF.


> add javadoc for UDFType annotation
> --
>
> Key: HIVE-9489
> URL: https://issues.apache.org/jira/browse/HIVE-9489
> Project: Hive
>  Issue Type: Bug
>  Components: Documentation, UDF
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 1.2.0
>
> Attachments: HIVE-9489.1.patch
>
>
> It is not clearly described, when a UDF should be marked as deterministic, 
> stateful or distinctLike.
> Adding javadoc for now. This information should also be incorporated in the 
> wikidoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9495) Map Side aggregation affecting map performance

[Anand Sridharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anand Sridharan updated HIVE-9495:
--
Attachment: profiler_screenshot.PNG

Profiler screenshot showing GroupByOperator.processHashAggr as hotspot.

> Map Side aggregation affecting map performance
> --
>
> Key: HIVE-9495
> URL: https://issues.apache.org/jira/browse/HIVE-9495
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
> Environment: RHEL 6.4
> Hortonworks Hadoop 2.2
>Reporter: Anand Sridharan
> Attachments: profiler_screenshot.PNG
>
>
> When trying to run a simple aggregation query with hive.map.aggr=true, map 
> tasks take a lot of time in Hive 0.14 as against  with hive.map.aggr=false.
> e.g.
> Consider the query:
> INSERT OVERWRITE TABLE lineitem_tgt_agg SELECT alias.a0 as a0, alias.a2 as 
> a1, alias.a1 as a2, alias.a3 as a3, alias.a4 as a4 FROM (SELECT alias.a0 as 
> a0, SUM(alias.a1) as a1, SUM(alias.a2) as a2, SUM(alias.a3) as a3, 
> SUM(alias.a4) as a4 FROM (SELECT lineitem_sf500.l_orderkey as a0, 
> CAST(lineitem_sf500.l_quantity * lineitem_sf500.l_extendedprice * (1 - 
> lineitem_sf500.l_discount) * (1 + lineitem_sf500.l_tax) AS DOUBLE) as a1, 
> lineitem_sf500.l_quantity as a2, CAST(lineitem_sf500.l_quantity * 
> lineitem_sf500.l_extendedprice * lineitem_sf500.l_discount AS DOUBLE) as a3, 
> CAST(lineitem_sf500.l_quantity * lineitem_sf500.l_extendedprice * 
> lineitem_sf500.l_tax AS DOUBLE) as a4 FROM lineitem_sf500) alias GROUP BY 
> alias.a0) alias;
> The above query was run with ~376GB of data / ~3billion records in the source.
> It takes ~10 minutes with hive.map.aggr=false.
> With map side aggregation set to true, the map tasks don't complete even 
> after an hour.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9495) Map Side aggregation affecting map performance

[Anand Sridharan (JIRA)]

Anand Sridharan created HIVE-9495:
-

 Summary: Map Side aggregation affecting map performance
 Key: HIVE-9495
 URL: https://issues.apache.org/jira/browse/HIVE-9495
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.14.0
 Environment: RHEL 6.4
Hortonworks Hadoop 2.2
Reporter: Anand Sridharan


When trying to run a simple aggregation query with hive.map.aggr=true, map 
tasks take a lot of time in Hive 0.14 as against  with hive.map.aggr=false.

e.g.
Consider the query:
INSERT OVERWRITE TABLE lineitem_tgt_agg SELECT alias.a0 as a0, alias.a2 as a1, 
alias.a1 as a2, alias.a3 as a3, alias.a4 as a4 FROM (SELECT alias.a0 as a0, 
SUM(alias.a1) as a1, SUM(alias.a2) as a2, SUM(alias.a3) as a3, SUM(alias.a4) as 
a4 FROM (SELECT lineitem_sf500.l_orderkey as a0, CAST(lineitem_sf500.l_quantity 
* lineitem_sf500.l_extendedprice * (1 - lineitem_sf500.l_discount) * (1 + 
lineitem_sf500.l_tax) AS DOUBLE) as a1, lineitem_sf500.l_quantity as a2, 
CAST(lineitem_sf500.l_quantity * lineitem_sf500.l_extendedprice * 
lineitem_sf500.l_discount AS DOUBLE) as a3, CAST(lineitem_sf500.l_quantity * 
lineitem_sf500.l_extendedprice * lineitem_sf500.l_tax AS DOUBLE) as a4 FROM 
lineitem_sf500) alias GROUP BY alias.a0) alias;

The above query was run with ~376GB of data / ~3billion records in the source.
It takes ~10 minutes with hive.map.aggr=false.
With map side aggregation set to true, the map tasks don't complete even after 
an hour.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 29807: HIVE-9253: MetaStore server should support timeout for long running requests

[Dong Chen]

> On Jan. 21, 2015, 6:43 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 372-374
> > 
> >
> > Shouldn't "long" & "LONG" be included in the names 
> > "hive.metastore.server.running.method.timeout" & 
> > "METASTORE_SERVER_RUNNING_METHOD_TIMEOUT"?
> > 
> > Also, please specify the JIRA number (HIVE-9253) in this review 
> > request, either under Bugs in the Information section or in the Summary, or 
> > both.

Thanks for your review and suggestion! Lefty. 
I have renamed it in the new patch.


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29807/#review68878
---


On Jan. 22, 2015, 8:22 a.m., Dong Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29807/
> ---
> 
> (Updated Jan. 22, 2015, 8:22 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-9253: MetaStore server should support timeout for long running requests
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5e00575 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> caad948 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> 564ac8b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java 
> 01ad36a 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RuntimeTimeout.java 
> PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/RuntimeTimeoutException.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/SessionPropertiesListener.java
>  PRE-CREATION 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTimeout.java
>  PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/TestRuntimeTimeout.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/29807/diff/
> 
> 
> Testing
> ---
> 
> UT passed
> 
> 
> Thanks,
> 
> Dong Chen
> 
>

[jira] [Commented] (HIVE-9486) Use session classloader instead of application loader

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294883#comment-14294883
 ] 

Hive QA commented on HIVE-9486:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694894/HIVE-9486.1.patch.txt

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2545/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2545/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2545/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2545/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/events/InsertEvent.java'
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java'
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java'
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py'
Reverted 'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py'
Reverted 
'metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote'
Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp'
Reverted 'metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp'
Reverted 'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h'
Reverted 'metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h'
Reverted 
'metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp'
Reverted 'metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb'
Reverted 'metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb'
Reverted 
'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java'
Reverted 
'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java'
Reverted 
'metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java'
Reverted 'metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php'
Reverted 'metastore/src/gen/thrift/gen-php/metastore/Types.php'
Reverted 'metastore/if/hive_metastore.thrift'
Reverted 
'itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java'
Reverted 
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java'
Reverted 
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageDeserializer.java'
Reverted 
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageDeserializer.java'
Reverted 
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONInsertMessage.java'
Reverted 
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageFactory.java'
Reverted 
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/InsertMessage.java'
Reverted 
'hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java'
Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignor

[jira] [Commented] (HIVE-9273) Add option to fire metastore event on insert

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294880#comment-14294880
 ] 

Hive QA commented on HIVE-9273:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694889/HIVE-9273.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7405 tests executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_in
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2544/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2544/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2544/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12694889 - PreCommit-HIVE-TRUNK-Build

> Add option to fire metastore event on insert
> 
>
> Key: HIVE-9273
> URL: https://issues.apache.org/jira/browse/HIVE-9273
> Project: Hive
>  Issue Type: New Feature
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-9273.patch
>
>
> HIVE-9271 adds the ability for the client to request firing metastore events. 
>  This can be used in the MoveTask to fire events when an insert is done that 
> does not add partitions to a table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9493) Failed job may not throw exceptions [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294876#comment-14294876
 ] 

Hive QA commented on HIVE-9493:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694934/HIVE-9493.1-spark.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7357 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/688/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/688/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-688/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12694934 - PreCommit-HIVE-SPARK-Build

> Failed job may not throw exceptions [Spark Branch]
> --
>
> Key: HIVE-9493
> URL: https://issues.apache.org/jira/browse/HIVE-9493
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-9493.1-spark.patch
>
>
> Currently remote driver assumes exception will be thrown when job fails to 
> run. This may not hold since job is submitted asynchronously. And we have to 
> check the futures before we decide the job is successful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9302) Beeline add jar local to client

[Ferdinand Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferdinand Xu updated HIVE-9302:
---
Attachment: HIVE-9302.2.patch

> Beeline add jar local to client
> ---
>
> Key: HIVE-9302
> URL: https://issues.apache.org/jira/browse/HIVE-9302
> Project: Hive
>  Issue Type: New Feature
>Reporter: Brock Noland
>Assignee: Ferdinand Xu
> Attachments: DummyDriver-1.0-SNAPSHOT.jar, HIVE-9302.1.patch, 
> HIVE-9302.2.patch, HIVE-9302.patch, mysql-connector-java-bin.jar, 
> postgresql-9.3.jdbc3.jar
>
>
> At present if a beeline user uses {{add jar}} the path they give is actually 
> on the HS2 server. It'd be great to allow beeline users to add local jars as 
> well.
> It might be useful to do this in the jdbc driver itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9302) Beeline add jar local to client

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294853#comment-14294853
 ] 

Ferdinand Xu commented on HIVE-9302:


Sorry, I meant to. 
There are two kinds of use cases. One is to add an existing known driver like 
mysql driver or postgres driver. Current supported driver are postgres and 
mysql.
{noformat}
# beeline
beeline> !addlocaldriverjar /path/to/mysql-connector-java-5.1.27-bin.jar
beeline> !connect mysql://host:3306/testdb
{noformat}
And another is to add a customized driver.
{noformat}
# beeline
beeline>!addlocaldriverjar /path/to/DummyDriver-1.0-SNAPSHOT.jar
beeline>!!addlocaldrivername org.apache.dummy.DummyDrive
beeline> !connect mysql://host:3306/testdb
{noformat}

> Beeline add jar local to client
> ---
>
> Key: HIVE-9302
> URL: https://issues.apache.org/jira/browse/HIVE-9302
> Project: Hive
>  Issue Type: New Feature
>Reporter: Brock Noland
>Assignee: Ferdinand Xu
> Attachments: DummyDriver-1.0-SNAPSHOT.jar, HIVE-9302.1.patch, 
> HIVE-9302.patch, mysql-connector-java-bin.jar, postgresql-9.3.jdbc3.jar
>
>
> At present if a beeline user uses {{add jar}} the path they give is actually 
> on the HS2 server. It'd be great to allow beeline users to add local jars as 
> well.
> It might be useful to do this in the jdbc driver itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9460) LLAP: Fix some static vars in the operator pipeline

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294854#comment-14294854
 ] 

Lefty Leverenz commented on HIVE-9460:
--

Doc note:  This adds configuration parameter *hive.execution.mode* to 
HiveConf.java, so it will need to be documented in the wiki when the LLAP 
branch gets merged to trunk.

Should we add a TODOC-LLAP label to keep track of these doc issues?

> LLAP: Fix some static vars in the operator pipeline
> ---
>
> Key: HIVE-9460
> URL: https://issues.apache.org/jira/browse/HIVE-9460
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: llap
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-9460.1.patch
>
>
> There are a few static vars left in the operator pipeline. Can't have those 
> with multi-threaded execution...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)