[jira] [Updated] (HIVE-8567) Vectorized queries output extra stuff for Binary columns

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8567:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk and branch. Thanks [~mmccline]!

> Vectorized queries output extra stuff for Binary columns
> 
>
> Key: HIVE-8567
> URL: https://issues.apache.org/jira/browse/HIVE-8567
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8567.01.patch
>
>
> See vector_data_types.q query output.  Non-vectorized output is shorter than 
> vectorized binary column output which seems to include characters from 
> earlier rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8517) When joining on partition column NDV gets overridden by StatsUtils.getColStatisticsFromExpression

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8517:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to branch and trunk. Thanks [~mmokhtar]

> When joining on partition column NDV gets overridden by 
> StatsUtils.getColStatisticsFromExpression
> -
>
> Key: HIVE-8517
> URL: https://issues.apache.org/jira/browse/HIVE-8517
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Mostafa Mokhtar
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8517.1.patch, HIVE-8517.2.patch, HIVE-8517.3.patch
>
>
> When joining on partition column number of partitions is used as NDV which 
> gets overridden by StatsUtils.getColStatisticsFromExpression and the number 
> of partitions used as NDV is replaced by number of rows which results in the 
> same behavior as explained in 
> https://issues.apache.org/jira/browse/HIVE-8196. "Joining on partition 
> columns with fetch column stats enabled results it very small CE which 
> negatively affects query performance "
> This is the call stack.
> {code}
> StatsUtils.getColStatisticsFromExpression(HiveConf, Statistics, ExprNodeDesc) 
> line: 1001  
> StatsRulesProcFactory$ReduceSinkStatsRule.process(Node, Stack, 
> NodeProcessorCtx, Object...) line: 1479  
> DefaultRuleDispatcher.dispatch(Node, Stack, Object...) line: 90 
> PreOrderWalker(DefaultGraphWalker).dispatchAndReturn(Node, Stack) line: 
> 94  
> PreOrderWalker(DefaultGraphWalker).dispatch(Node, Stack) line: 78   
> PreOrderWalker.walk(Node) line: 54
> PreOrderWalker.walk(Node) line: 59
> PreOrderWalker.walk(Node) line: 59
> PreOrderWalker(DefaultGraphWalker).startWalking(Collection, 
> HashMap) line: 109 
> AnnotateWithStatistics.transform(ParseContext) line: 78   
> TezCompiler.runStatsAnnotation(OptimizeTezProcContext) line: 248  
> TezCompiler.optimizeOperatorPlan(ParseContext, Set, 
> Set) line: 120   
> TezCompiler(TaskCompiler).compile(ParseContext, List>, 
> HashSet, HashSet) line: 99 
> SemanticAnalyzer.analyzeInternal(ASTNode) line: 10037 
> SemanticAnalyzer(BaseSemanticAnalyzer).analyze(ASTNode, Context) line: 221
> ExplainSemanticAnalyzer.analyzeInternal(ASTNode) line: 74 
> ExplainSemanticAnalyzer(BaseSemanticAnalyzer).analyze(ASTNode, Context) line: 
> 221 
> Driver.compile(String, boolean) line: 415 
> {code}
> Query
> {code}
> select
>   ss_item_sk item_sk, d_date, sum(ss_sales_price),
>   sum(sum(ss_sales_price))
>   over (partition by ss_item_sk order by d_date rows between unbounded 
> preceding and current row) cume_sales
> from store_sales
> ,date_dim
> where ss_sold_date_sk=d_date_sk
>   and d_month_seq between 1193 and 1193+11
>   and ss_item_sk is not NULL
> group by ss_item_sk, d_date
> {code}
> Plan 
> Notice in the Map join operator the number of rows drop from 82,510,879,939 
> to 36524 after the join.
> {code}
> OK
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-1
> Tez
>   Edges:
> Map 1 <- Map 4 (BROADCAST_EDGE)
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
>   DagName: mmokhtar_20141019131818_086d663a-5621-456c-bf25-8ccb7112ee3b:6
>   Vertices:
> Map 1
> Map Operator Tree:
> TableScan
>   alias: store_sales
>   filterExpr: ss_item_sk is not null (type: boolean)
>   Statistics: Num rows: 82510879939 Data size: 6873789738208 
> Basic stats: COMPLETE Column stats: COMPLETE
>   Filter Operator
> predicate: ss_item_sk is not null (type: boolean)
> Statistics: Num rows: 82510879939 Data size: 652315818272 
> Basic stats: COMPLETE Column stats: COMPLETE
> Map Join Operator
>   condition map:
>Inner Join 0 to 1
>   condition expressions:
> 0 {ss_item_sk} {ss_sales_price} {ss_sold_date_sk}
> 1 {d_date_sk} {d_date} {d_month_seq}
>   keys:
> 0 ss_sold_date_sk (type: int)
> 1 d_date_sk (type: int)
>   outputColumnNames: _col1, _col12, _col22, _col26, 
> _col28, _col29
>   input vertices:
> 1 Map 4
>   Statistics: Num rows: 36524 Data size: 4163736 Basic 
> stats: COMPLETE Column stats: COMPLETE
>  

[jira] [Created] (HIVE-8590) With different parameters or column number dense_rank function gets different count distinct results

[ericni (JIRA)]

ericni created HIVE-8590:


 Summary: With different parameters  or column number dense_rank 
function gets different count distinct results 
 Key: HIVE-8590
 URL: https://issues.apache.org/jira/browse/HIVE-8590
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 0.13.1
 Environment: cdh 4.6.0/hive0.13
Reporter: ericni


We create a table with sql which contains the  dense_rank function,and then run 
count distinct on this table,
we found that with diffrent dense_rank parameters or even defferent columns,we 
will get the defferent count distinct results:
1.Less data will be ok(in our test case,200 million rows will get the same 
results,but 300 million rows will get the different results )
2.Different dense_rank parameters may be get the different results ,e.g  
"dense_rank() over(distribute by a,b sort by c desc)" and "dense_rank() 
over(distribute by a sort by c desc)"
3.All window functions(rank,row_number,dense_rank) have this problem
4.Less column number may be ok
5.Count(1) is ok,but Count distinct gets different results
6.It seems that some rows have been lost and some rows repeated 

test data(File is too large to upload.):
http://pan.baidu.com/s/1hqnCzze

test sql:
http://pan.baidu.com/s/1eQna8q2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8586) Record counters aren't updated correctly for vectorized queries

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8586:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk + branch.

> Record counters aren't updated correctly for vectorized queries
> ---
>
> Key: HIVE-8586
> URL: https://issues.apache.org/jira/browse/HIVE-8586
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.14.0
>
> Attachments: HIVE-8586.1.patch
>
>
> Counts batches not rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8532) return code of "source xxx" clause is missing

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-8532:

   Resolution: Fixed
Fix Version/s: 0.15.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks vitthal (Suhas) Gogate, for the contribution.

> return code of "source xxx" clause is missing
> -
>
> Key: HIVE-8532
> URL: https://issues.apache.org/jira/browse/HIVE-8532
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 0.12.0, 0.13.1
>Reporter: Gordon Wang
> Fix For: 0.15.0
>
> Attachments: HIVE-8532.patch
>
>
> When executing "source "  clause, hive client driver does not catch 
> the return code of this command.
> This behaviour causes an issue when running hive query in Oozie workflow.
> When the "source" clause is put into a Oozie workflow, Oozie can not get the 
> return code of this command. Thus, Oozie consider the "source" clause as 
> successful all the time. 
> So, when the "source" clause fails, the hive query does not abort and the 
> oozie workflow does not abort either.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8583) HIVE-8341 Cleanup & Test for hive.script.operator.env.blacklist

[Lars Francke (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182481#comment-14182481
 ] 

Lars Francke commented on HIVE-8583:


As far as I understand the work is done on the non-replaced original 
configuration properties:

{code}
  void addJobConfToEnvironment(Configuration conf, Map env) {
Iterator> it = conf.iterator();
while (it.hasNext()) {
  Map.Entry en = it.next();
  String name = en.getKey();
  if (!blackListed(name)) {
String value = conf.get(name); // does variable expansion
name = safeEnvVarName(name);
{code}

So the replacing happens later.

BTW. the replaceAll is wrong too. It takes a regex so "." means every 
character. So it'd replace everything with underscores.

> HIVE-8341 Cleanup & Test for hive.script.operator.env.blacklist
> ---
>
> Key: HIVE-8583
> URL: https://issues.apache.org/jira/browse/HIVE-8583
> Project: Hive
>  Issue Type: Improvement
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-8583.1.patch
>
>
> [~alangates] added the following in HIVE-8341:
> {code}
> String bl = 
> hconf.get(HiveConf.ConfVars.HIVESCRIPT_ENV_BLACKLIST.toString());
> if (bl != null && bl.length() > 0) {
>   String[] bls = bl.split(",");
>   for (String b : bls) {
> b.replaceAll(".", "_");
> blackListedConfEntries.add(b);
>   }
> }
> {code}
> The {{replaceAll}} call is confusing as its result is not used at all.
> This patch contains the following:
> * Minor style modification (missorted modifiers)
> * Adds reading of default value for HIVESCRIPT_ENV_BLACKLIST
> * Removes replaceAll
> * Lets blackListed take a Configuration job as parameter which allowed me to 
> add a test for this



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8586) Record counters aren't updated correctly for vectorized queries

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182477#comment-14182477
 ] 

Hive QA commented on HIVE-8586:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676818/HIVE-8586.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6563 tests executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1436/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1436/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1436/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12676818 - PreCommit-HIVE-TRUNK-Build

> Record counters aren't updated correctly for vectorized queries
> ---
>
> Key: HIVE-8586
> URL: https://issues.apache.org/jira/browse/HIVE-8586
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.14.0
>
> Attachments: HIVE-8586.1.patch
>
>
> Counts batches not rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-2573) Create per-session function registry

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-2573:

Attachment: HIVE-2573.10.patch.txt

> Create per-session function registry 
> -
>
> Key: HIVE-2573
> URL: https://issues.apache.org/jira/browse/HIVE-2573
> Project: Hive
>  Issue Type: Improvement
>  Components: Server Infrastructure
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2573.D3231.1.patch, 
> HIVE-2573.1.patch.txt, HIVE-2573.10.patch.txt, HIVE-2573.2.patch.txt, 
> HIVE-2573.3.patch.txt, HIVE-2573.4.patch.txt, HIVE-2573.5.patch, 
> HIVE-2573.6.patch, HIVE-2573.7.patch, HIVE-2573.8.patch.txt, 
> HIVE-2573.9.patch.txt
>
>
> Currently the function registry is shared resource and could be overrided by 
> other users when using HiveServer. If per-session function registry is 
> provided, this situation could be prevented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 26854: HIVE-2573 Create per-session function registry

[Navis Ryu]

> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java, line 42
> > 
> >
> > Can we replace isNative/isPersistent with an enum that has BUILTIN, 
> > PERMANENT, TEMPORARY (or equivalent terms)?

Sure.


> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java, line 128
> > 
> >
> > Why was this check removed? If you are permanent UDFs with Hive CLI, 
> > you would have to make sure the UDF resources are available from the 
> > cluster as opposed to just on the local filesystem of the client that 
> > created the UDF.

Seemed removed by mistake. I'll revert that.


> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 433
> > 
> >
> > I thought builtin functions aren't allowed to be removed?
> > Does this mean that we could create a function using the same class as 
> > a built-in function (create a synonym), and deleting this new function will 
> > cause this class to be removed from the builtin set?

This should be removed. Thanks.


> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
> > 
> >
> > There is no longer a way to query the metastore for UDFs apart from the 
> > static initialization. So if one CLI user creates a permanent UDF, another 
> > user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or 
> > HS2 was initialized before this UDF was created.

Permanent functions (persistent function seemed better name, imho) are 
registered to system registry, which is shared to all clients. So if one user 
creates new permanent function, it's shared to all clients. The time a user 
accesses the function, the class is loaded with required resources and 
registered to session registry as a temporary function.


> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 511
> > 
> >
> > I think I see what you're trying do here, trying to add a mechanism so 
> > that if a function is deleted in one session, the other sessions will also 
> > see it as discarded if they try to look it up. But I don't actually see 
> > discarded being set to true.

My mistake. Fixed.


> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 152
> > 
> >
> > I don't think it's necessary to pre-emptively query the metastore for 
> > permanent UDFs during initialization. If we have user on Hive CLI, we will 
> > automatically lookup metastore/download UDF resources, when they may not 
> > even be using any of these UDFs during their session. How about we keep the 
> > existing behavior that we only look them up when they are used during a 
> > query?
> > 
> > Also, if we are doing this during static initialization, is Hive be in 
> > a state that it can query the metastore? Not sure if there is any other 
> > initialization that may need to take place beforehand.

bq. we only look them up when they are used during a query?
This method just stores meta information (classname, resources, etc.) for the 
function without loading any resources/classes, lessening redundant metastore 
accesses from all clients. When user accesses the function it's registered to 
session registry with the information as described above. 

bq. Not sure if there is any other initialization that may need to take place 
beforehand.
Afaik, metastore has static lock for initialization which is checked before 
client accesses internal of it. If it does not act like that, we should fix 
metastore.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
---


On Oct. 23, 2014, 12:20 a.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> ---
> 
> (Updated Oct. 23, 2014, 12:20 a.m.)
> 
> 
> Review request for hive, Navis Ryu and Thejas Nair.
> 
> 
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Small updates to Navis' changes:
> - session registry doesn't l

[jira] [Commented] (HIVE-6806) CREATE TABLE should support STORED AS AVRO

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182455#comment-14182455
 ] 

Lefty Leverenz commented on HIVE-6806:
--

Shouldn't AVRO be added as a possible value for the *hive.default.fileformat* 
parameter in HiveConf.java?

* [Configuration Properties -- hive.default.fileformat | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.default.fileformat]

> CREATE TABLE should support STORED AS AVRO
> --
>
> Key: HIVE-6806
> URL: https://issues.apache.org/jira/browse/HIVE-6806
> Project: Hive
>  Issue Type: New Feature
>  Components: Serializers/Deserializers
>Affects Versions: 0.12.0
>Reporter: Jeremy Beard
>Assignee: Ashish Kumar Singh
>Priority: Minor
>  Labels: Avro
> Fix For: 0.14.0
>
> Attachments: HIVE-6806.1.patch, HIVE-6806.2.patch, HIVE-6806.3.patch, 
> HIVE-6806.patch
>
>
> Avro is well established and widely used within Hive, however creating 
> Avro-backed tables requires the messy listing of the SerDe, InputFormat and 
> OutputFormat classes.
> Similarly to HIVE-5783 for Parquet, Hive would be easier to use if it had 
> native Avro support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8532) return code of "source xxx" clause is missing

[vitthal (Suhas) Gogate (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182439#comment-14182439
 ] 

vitthal (Suhas) Gogate commented on HIVE-8532:
--

[~ashutoshc] Is it possible for you to commit the changes?  Thx. -Suhas

> return code of "source xxx" clause is missing
> -
>
> Key: HIVE-8532
> URL: https://issues.apache.org/jira/browse/HIVE-8532
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 0.12.0, 0.13.1
>Reporter: Gordon Wang
> Attachments: HIVE-8532.patch
>
>
> When executing "source "  clause, hive client driver does not catch 
> the return code of this command.
> This behaviour causes an issue when running hive query in Oozie workflow.
> When the "source" clause is put into a Oozie workflow, Oozie can not get the 
> return code of this command. Thus, Oozie consider the "source" clause as 
> successful all the time. 
> So, when the "source" clause fails, the hive query does not abort and the 
> oozie workflow does not abort either.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-6940) [WebHCat]Update documentation for Templeton-Sqoop action

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182429#comment-14182429
 ] 

Eugene Koifman commented on HIVE-6940:
--

see TestSqoop group in 
https://github.com/apache/hive/blob/trunk/hcatalog/src/test/e2e/templeton/tests/jobsubmission.conf
 for some examples

> [WebHCat]Update documentation for Templeton-Sqoop action
> 
>
> Key: HIVE-6940
> URL: https://issues.apache.org/jira/browse/HIVE-6940
> Project: Hive
>  Issue Type: Bug
>  Components: Documentation, WebHCat
>Affects Versions: 0.14.0
>Reporter: Shuaishuai Nie
>  Labels: TODOC14
>
> WebHCat documentation need to be updated based on the new feature introduced 
> in HIVE-5072
> Here is some examples using the endpoint "templeton/v1/sqoop"
> example1: (passing Sqoop command directly)
> curl -s -d command="import --connect 
> jdbc:sqlserver://localhost:4033;databaseName=SqoopDB;user=hadoop;password=password
>  --table mytable --target-dir user/hadoop/importtable" -d 
> statusdir="sqoop.output" 
> 'http://localhost:50111/templeton/v1/sqoop?user.name=hadoop'
> example2: (passing source file which contains sqoop command)
> curl -s -d optionsfile="/sqoopcommand/command0.txt"  -d 
> statusdir="sqoop.output" 
> 'http://localhost:50111/templeton/v1/sqoop?user.name=hadoop'
> example3: (using --options-file in the middle of sqoop command to enable 
> reuse part of Sqoop command like connection string)
> curl -s -d files="/sqoopcommand/command1.txt,/sqoopcommand/command2.txt" -d 
> command="import --options-file command1.txt --options-file command2.txt" -d 
> statusdir="sqoop.output" 
> 'http://localhost:50111/templeton/v1/sqoop?user.name=hadoop'
> Also, for user to pass their JDBC driver jar, they can use the "-libjars" 
> generic option in the Sqoop command. This is a functionality provided by 
> Sqoop.
> Set of parameters can be passed to the endpoint:
> command 
> (Sqoop command string to run)
> optionsfile
> (Options file which contain Sqoop command need to run, each section in the 
> Sqoop command separated by space should be a single line in the options file)
> files 
> (Comma seperated files to be copied to the map reduce cluster)
> statusdir 
> (A directory where WebHCat will write the status of the Sqoop job. If 
> provided, it is the caller’s responsibility to remove this directory when 
> done)
> callback 
> (Define a URL to be called upon job completion. You may embed a specific job 
> ID into the URL using $jobId. This tag will be replaced in the callback URL 
> with the job’s job ID. )
> enablelog
> (when set to "true", WebHCat will upload job log to statusdir. Need to define 
> statusdir when enabled)
> All the above parameters are optional, but use have to provide either 
> "command" or "optionsfile" in the command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-6940) [WebHCat]Update documentation for Templeton-Sqoop action

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182428#comment-14182428
 ] 

Eugene Koifman commented on HIVE-6940:
--

The comment about "-libjars" above is wrong.
When WebHCat is configured to auto-ship Sqoop tar file, the user/admin may 
place any necessary JDBC jars into an HDFS directory.  Then use "libdir" param 
when making the REST call to supply this directory path.  WebHCat will then 
make sure that the jars from this dir are are placed in lib/ of the exploded 
Sqoop tar on the remote node.

> [WebHCat]Update documentation for Templeton-Sqoop action
> 
>
> Key: HIVE-6940
> URL: https://issues.apache.org/jira/browse/HIVE-6940
> Project: Hive
>  Issue Type: Bug
>  Components: Documentation, WebHCat
>Affects Versions: 0.14.0
>Reporter: Shuaishuai Nie
>  Labels: TODOC14
>
> WebHCat documentation need to be updated based on the new feature introduced 
> in HIVE-5072
> Here is some examples using the endpoint "templeton/v1/sqoop"
> example1: (passing Sqoop command directly)
> curl -s -d command="import --connect 
> jdbc:sqlserver://localhost:4033;databaseName=SqoopDB;user=hadoop;password=password
>  --table mytable --target-dir user/hadoop/importtable" -d 
> statusdir="sqoop.output" 
> 'http://localhost:50111/templeton/v1/sqoop?user.name=hadoop'
> example2: (passing source file which contains sqoop command)
> curl -s -d optionsfile="/sqoopcommand/command0.txt"  -d 
> statusdir="sqoop.output" 
> 'http://localhost:50111/templeton/v1/sqoop?user.name=hadoop'
> example3: (using --options-file in the middle of sqoop command to enable 
> reuse part of Sqoop command like connection string)
> curl -s -d files="/sqoopcommand/command1.txt,/sqoopcommand/command2.txt" -d 
> command="import --options-file command1.txt --options-file command2.txt" -d 
> statusdir="sqoop.output" 
> 'http://localhost:50111/templeton/v1/sqoop?user.name=hadoop'
> Also, for user to pass their JDBC driver jar, they can use the "-libjars" 
> generic option in the Sqoop command. This is a functionality provided by 
> Sqoop.
> Set of parameters can be passed to the endpoint:
> command 
> (Sqoop command string to run)
> optionsfile
> (Options file which contain Sqoop command need to run, each section in the 
> Sqoop command separated by space should be a single line in the options file)
> files 
> (Comma seperated files to be copied to the map reduce cluster)
> statusdir 
> (A directory where WebHCat will write the status of the Sqoop job. If 
> provided, it is the caller’s responsibility to remove this directory when 
> done)
> callback 
> (Define a URL to be called upon job completion. You may embed a specific job 
> ID into the URL using $jobId. This tag will be replaced in the callback URL 
> with the job’s job ID. )
> enablelog
> (when set to "true", WebHCat will upload job log to statusdir. Need to define 
> statusdir when enabled)
> All the above parameters are optional, but use have to provide either 
> "command" or "optionsfile" in the command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8589) TPCDS query 75 fails with tez

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182421#comment-14182421
 ] 

Gunther Hagleitner commented on HIVE-8589:
--

LGTM +1

> TPCDS query 75 fails with tez
> -
>
> Key: HIVE-8589
> URL: https://issues.apache.org/jira/browse/HIVE-8589
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-8589.1.patch
>
>
> {code}
> WITH all_sales AS (
>  SELECT d_year
>,i_brand_id
>,i_class_id
>,i_category_id
>,i_manufact_id
>,SUM(sales_cnt) AS sales_cnt
>,SUM(sales_amt) AS sales_amt
>  FROM (SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
>  ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
>FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
>   JOIN date_dim ON d_date_sk=cs_sold_date_sk
>   LEFT JOIN catalog_returns ON 
> (cs_order_number=cr_order_number 
> AND cs_item_sk=cr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
>  ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
>FROM store_sales JOIN item ON i_item_sk=ss_item_sk
> JOIN date_dim ON d_date_sk=ss_sold_date_sk
> LEFT JOIN store_returns ON 
> (ss_ticket_number=sr_ticket_number 
> AND ss_item_sk=sr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
>  ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
>FROM web_sales JOIN item ON i_item_sk=ws_item_sk
>   JOIN date_dim ON d_date_sk=ws_sold_date_sk
>   LEFT JOIN web_returns ON 
> (ws_order_number=wr_order_number 
> AND ws_item_sk=wr_item_sk)
>WHERE i_category='Sports') sales_detail
>  GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
>  SELECT  prev_yr.d_year AS prev_year
>   ,curr_yr.d_year AS year
>   ,curr_yr.i_brand_id
>   ,curr_yr.i_class_id
>   ,curr_yr.i_category_id
>   ,curr_yr.i_manufact_id
>   ,prev_yr.sales_cnt AS prev_yr_cnt
>   ,curr_yr.sales_cnt AS curr_yr_cnt
>   ,curr_yr.sales_cnt-prev_yr.sales_cnt AS 
> sales_cnt_diff
>   ,curr_yr.sales_amt-prev_yr.sales_amt AS 
> sales_amt_diff
>  FROM all_sales curr_yr, all_sales prev_yr
>  WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
>AND curr_yr.i_class_id=prev_yr.i_class_id
>AND curr_yr.i_category_id=prev_yr.i_category_id
>AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
>AND curr_yr.d_year=2002
>AND prev_yr.d_year=2002-1
>AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
> DECIMAL(17,2))<0.9
>  ORDER BY sales_cnt_diff
>  limit 100
> {code}
> throws a null pointer exception when executed via tez.
> {code}
> java.lang.IllegalArgumentException: Output vertex null doesn't exist!
> at org.apache.tez.dag.api.DAG.addEdge(DAG.java:236)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:340)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:152)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:161)
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
> at 
> or

[jira] [Updated] (HIVE-8588) sqoop REST endpoint fails to send appropriate JDBC driver to the cluster

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-8588:
-
Description: 
This is originally discovered by [~deepesh]
When running a Sqoop integration test from WebHCat
{noformat}
curl --show-error -d command="export -libjars 
hdfs:///tmp/mysql-connector-java.jar --connect 
jdbc:mysql://deepesh-c6-1.cs1cloud.internal/sqooptest --username sqoop 
--password passwd --export-dir /tmp/templeton_test_data/sqoop --table person" 
-d statusdir=sqoop.output -X POST 
"http://deepesh-c6-1.cs1cloud.internal:50111/templeton/v1/sqoop?user.name=hrt_qa";
{noformat}
the job is failing with the following error:
{noformat}
$ hadoop fs -cat /user/hrt_qa/sqoop.output/stderr
14/10/15 23:52:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.0.0-897
14/10/15 23:52:53 WARN tool.BaseSqoopTool: Setting your password on the 
command-line is insecure. Consider using -P instead.
14/10/15 23:52:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming 
resultset.
14/10/15 23:52:54 INFO tool.CodeGenTool: Beginning code generation
14/10/15 23:52:54 ERROR sqoop.Sqoop: Got exception running Sqoop: 
java.lang.RuntimeException: Could not load db driver class: 
com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: 
com.mysql.jdbc.Driver
at 
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)
at 
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at 
org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at 
org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at 
org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at 
org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at 
org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
{noformat}
Note that the Sqoop tar bundle does not contain the JDBC connector jar. I think 
the problem here maybe that the mysql connector jar added to libjars isn't 
available to the Sqoop tool which first connects to the database through JDBC 
driver to collect some table information before running the MR job. libjars 
will only add the connector jar for the MR job and not the local one.

NO PRECOMMIT TESTS

  was:
This is originally discovered by [~deepesh]
When running a Sqoop integration test from WebHCat
{noformat}
curl --show-error -d command="export -libjars 
hdfs:///tmp/mysql-connector-java.jar --connect 
jdbc:mysql://deepesh-c6-1.cs1cloud.internal/sqooptest --username sqoop 
--password passwd --export-dir /tmp/templeton_test_data/sqoop --table person" 
-d statusdir=sqoop.output -X POST 
"http://deepesh-c6-1.cs1cloud.internal:50111/templeton/v1/sqoop?user.name=hrt_qa";
{noformat}
the job is failing with the following error:
{noformat}
$ hadoop fs -cat /user/hrt_qa/sqoop.output/stderr
14/10/15 23:52:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.0.0-897
14/10/15 23:52:53 WARN tool.BaseSqoopTool: Setting your password on the 
command-line is insecure. Consider using -P instead.
14/10/15 23:52:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming 
resultset.
14/10/15 23:52:54 INFO tool.CodeGenTool: Beginning code generation
14/10/15 23:52:54 ERROR sqoop.Sqoop: Got exception running Sqoop: 
java.lang.RuntimeException: Could not load db driver class: 
com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: 
com.mysql.jdbc.Driver
at 
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)
at 
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at 
org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at 
org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at 
org.apache.sqoop.manager.SqlManager.g

[jira] [Commented] (HIVE-8588) sqoop REST endpoint fails to send appropriate JDBC driver to the cluster

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182419#comment-14182419
 ] 

Eugene Koifman commented on HIVE-8588:
--

[~thejas] https://reviews.apache.org/r/27131/

> sqoop REST endpoint fails to send appropriate JDBC driver to the cluster
> 
>
> Key: HIVE-8588
> URL: https://issues.apache.org/jira/browse/HIVE-8588
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> This is originally discovered by [~deepesh]
> When running a Sqoop integration test from WebHCat
> {noformat}
> curl --show-error -d command="export -libjars 
> hdfs:///tmp/mysql-connector-java.jar --connect 
> jdbc:mysql://deepesh-c6-1.cs1cloud.internal/sqooptest --username sqoop 
> --password passwd --export-dir /tmp/templeton_test_data/sqoop --table person" 
> -d statusdir=sqoop.output -X POST 
> "http://deepesh-c6-1.cs1cloud.internal:50111/templeton/v1/sqoop?user.name=hrt_qa";
> {noformat}
> the job is failing with the following error:
> {noformat}
> $ hadoop fs -cat /user/hrt_qa/sqoop.output/stderr
> 14/10/15 23:52:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.0.0-897
> 14/10/15 23:52:53 WARN tool.BaseSqoopTool: Setting your password on the 
> command-line is insecure. Consider using -P instead.
> 14/10/15 23:52:54 INFO manager.MySQLManager: Preparing to use a MySQL 
> streaming resultset.
> 14/10/15 23:52:54 INFO tool.CodeGenTool: Beginning code generation
> 14/10/15 23:52:54 ERROR sqoop.Sqoop: Got exception running Sqoop: 
> java.lang.RuntimeException: Could not load db driver class: 
> com.mysql.jdbc.Driver
> java.lang.RuntimeException: Could not load db driver class: 
> com.mysql.jdbc.Driver
>   at 
> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)
>   at 
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
>   at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
>   at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
>   at 
> org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
>   at 
> org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
>   at 
> org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
>   at 
> org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
>   at 
> org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
>   at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
>   at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
>   at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
>   at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
>   at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
>   at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
>   at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> {noformat}
> Note that the Sqoop tar bundle does not contain the JDBC connector jar. I 
> think the problem here maybe that the mysql connector jar added to libjars 
> isn't available to the Sqoop tool which first connects to the database 
> through JDBC driver to collect some table information before running the MR 
> job. libjars will only add the connector jar for the MR job and not the local 
> one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8586) Record counters aren't updated correctly for vectorized queries

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182418#comment-14182418
 ] 

Vikram Dixit K commented on HIVE-8586:
--

+1 LGTM. +1 for 0.14 as well.

> Record counters aren't updated correctly for vectorized queries
> ---
>
> Key: HIVE-8586
> URL: https://issues.apache.org/jira/browse/HIVE-8586
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.14.0
>
> Attachments: HIVE-8586.1.patch
>
>
> Counts batches not rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Build failure on trunk

[Gunther Hagleitner]

Thanks Xuefu - I appreciate it!

On Thu, Oct 23, 2014 at 9:15 PM, Xuefu Zhang  wrote:

> You can add CLEAR LIBRARY CACHE in the description for any JIRA, which will
> clear local maven repo. I added it to HIVE-6165.
>
> On Thu, Oct 23, 2014 at 9:09 PM, Gunther Hagleitner <
> ghagleit...@hortonworks.com> wrote:
>
> > Builds are running again (reverted patch). I've re-uploaded the patches
> > that had a failed run because of it.
> >
> > Sorry about that...
> >
> > Thanks,
> > Gunther.
> >
> > On Thu, Oct 23, 2014 at 8:07 PM, Gunther Hagleitner <
> > gunther.hagleit...@gmail.com> wrote:
> >
> > > The builds are failing right now on trunk after I committed a change
> that
> > > requires new/updated calcite libs. (Sorry about that).
> > >
> > > Is it possible for someone to wipe the .m2 cache on the build machine,
> so
> > > it would download a new version with the changes?
> > >
> > > Thank you,
> > > Gunther.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Updated] (HIVE-8589) TPCDS query 75 fails with tez

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8589:
-
Assignee: Vikram Dixit K

> TPCDS query 75 fails with tez
> -
>
> Key: HIVE-8589
> URL: https://issues.apache.org/jira/browse/HIVE-8589
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-8589.1.patch
>
>
> {code}
> WITH all_sales AS (
>  SELECT d_year
>,i_brand_id
>,i_class_id
>,i_category_id
>,i_manufact_id
>,SUM(sales_cnt) AS sales_cnt
>,SUM(sales_amt) AS sales_amt
>  FROM (SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
>  ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
>FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
>   JOIN date_dim ON d_date_sk=cs_sold_date_sk
>   LEFT JOIN catalog_returns ON 
> (cs_order_number=cr_order_number 
> AND cs_item_sk=cr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
>  ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
>FROM store_sales JOIN item ON i_item_sk=ss_item_sk
> JOIN date_dim ON d_date_sk=ss_sold_date_sk
> LEFT JOIN store_returns ON 
> (ss_ticket_number=sr_ticket_number 
> AND ss_item_sk=sr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
>  ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
>FROM web_sales JOIN item ON i_item_sk=ws_item_sk
>   JOIN date_dim ON d_date_sk=ws_sold_date_sk
>   LEFT JOIN web_returns ON 
> (ws_order_number=wr_order_number 
> AND ws_item_sk=wr_item_sk)
>WHERE i_category='Sports') sales_detail
>  GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
>  SELECT  prev_yr.d_year AS prev_year
>   ,curr_yr.d_year AS year
>   ,curr_yr.i_brand_id
>   ,curr_yr.i_class_id
>   ,curr_yr.i_category_id
>   ,curr_yr.i_manufact_id
>   ,prev_yr.sales_cnt AS prev_yr_cnt
>   ,curr_yr.sales_cnt AS curr_yr_cnt
>   ,curr_yr.sales_cnt-prev_yr.sales_cnt AS 
> sales_cnt_diff
>   ,curr_yr.sales_amt-prev_yr.sales_amt AS 
> sales_amt_diff
>  FROM all_sales curr_yr, all_sales prev_yr
>  WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
>AND curr_yr.i_class_id=prev_yr.i_class_id
>AND curr_yr.i_category_id=prev_yr.i_category_id
>AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
>AND curr_yr.d_year=2002
>AND prev_yr.d_year=2002-1
>AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
> DECIMAL(17,2))<0.9
>  ORDER BY sales_cnt_diff
>  limit 100
> {code}
> throws a null pointer exception when executed via tez.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8589) TPCDS query 75 fails with tez

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8589:
-
Description: 
{code}
WITH all_sales AS (
 SELECT d_year
   ,i_brand_id
   ,i_class_id
   ,i_category_id
   ,i_manufact_id
   ,SUM(sales_cnt) AS sales_cnt
   ,SUM(sales_amt) AS sales_amt
 FROM (SELECT d_year
 ,i_brand_id
 ,i_class_id
 ,i_category_id
 ,i_manufact_id
 ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
 ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
  JOIN date_dim ON d_date_sk=cs_sold_date_sk
  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
AND cs_item_sk=cr_item_sk)
   WHERE i_category='Sports'
   UNION ALL
   SELECT d_year
 ,i_brand_id
 ,i_class_id
 ,i_category_id
 ,i_manufact_id
 ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
 ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
JOIN date_dim ON d_date_sk=ss_sold_date_sk
LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_number 
AND ss_item_sk=sr_item_sk)
   WHERE i_category='Sports'
   UNION ALL
   SELECT d_year
 ,i_brand_id
 ,i_class_id
 ,i_category_id
 ,i_manufact_id
 ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
 ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
   FROM web_sales JOIN item ON i_item_sk=ws_item_sk
  JOIN date_dim ON d_date_sk=ws_sold_date_sk
  LEFT JOIN web_returns ON (ws_order_number=wr_order_number 
AND ws_item_sk=wr_item_sk)
   WHERE i_category='Sports') sales_detail
 GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
 SELECT  prev_yr.d_year AS prev_year
  ,curr_yr.d_year AS year
  ,curr_yr.i_brand_id
  ,curr_yr.i_class_id
  ,curr_yr.i_category_id
  ,curr_yr.i_manufact_id
  ,prev_yr.sales_cnt AS prev_yr_cnt
  ,curr_yr.sales_cnt AS curr_yr_cnt
  ,curr_yr.sales_cnt-prev_yr.sales_cnt AS sales_cnt_diff
  ,curr_yr.sales_amt-prev_yr.sales_amt AS sales_amt_diff
 FROM all_sales curr_yr, all_sales prev_yr
 WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
   AND curr_yr.i_class_id=prev_yr.i_class_id
   AND curr_yr.i_category_id=prev_yr.i_category_id
   AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
   AND curr_yr.d_year=2002
   AND prev_yr.d_year=2002-1
   AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
DECIMAL(17,2))<0.9
 ORDER BY sales_cnt_diff
 limit 100
{code}

throws a null pointer exception when executed via tez.

{code}
java.lang.IllegalArgumentException: Output vertex null doesn't exist!
at org.apache.tez.dag.api.DAG.addEdge(DAG.java:236)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:340)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:152)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:161)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.mai

[jira] [Updated] (HIVE-8589) TPCDS query 75 fails with tez

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8589:
-
Attachment: HIVE-8589.1.patch

> TPCDS query 75 fails with tez
> -
>
> Key: HIVE-8589
> URL: https://issues.apache.org/jira/browse/HIVE-8589
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-8589.1.patch
>
>
> {code}
> WITH all_sales AS (
>  SELECT d_year
>,i_brand_id
>,i_class_id
>,i_category_id
>,i_manufact_id
>,SUM(sales_cnt) AS sales_cnt
>,SUM(sales_amt) AS sales_amt
>  FROM (SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
>  ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
>FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
>   JOIN date_dim ON d_date_sk=cs_sold_date_sk
>   LEFT JOIN catalog_returns ON 
> (cs_order_number=cr_order_number 
> AND cs_item_sk=cr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
>  ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
>FROM store_sales JOIN item ON i_item_sk=ss_item_sk
> JOIN date_dim ON d_date_sk=ss_sold_date_sk
> LEFT JOIN store_returns ON 
> (ss_ticket_number=sr_ticket_number 
> AND ss_item_sk=sr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
>  ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
>FROM web_sales JOIN item ON i_item_sk=ws_item_sk
>   JOIN date_dim ON d_date_sk=ws_sold_date_sk
>   LEFT JOIN web_returns ON 
> (ws_order_number=wr_order_number 
> AND ws_item_sk=wr_item_sk)
>WHERE i_category='Sports') sales_detail
>  GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
>  SELECT  prev_yr.d_year AS prev_year
>   ,curr_yr.d_year AS year
>   ,curr_yr.i_brand_id
>   ,curr_yr.i_class_id
>   ,curr_yr.i_category_id
>   ,curr_yr.i_manufact_id
>   ,prev_yr.sales_cnt AS prev_yr_cnt
>   ,curr_yr.sales_cnt AS curr_yr_cnt
>   ,curr_yr.sales_cnt-prev_yr.sales_cnt AS 
> sales_cnt_diff
>   ,curr_yr.sales_amt-prev_yr.sales_amt AS 
> sales_amt_diff
>  FROM all_sales curr_yr, all_sales prev_yr
>  WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
>AND curr_yr.i_class_id=prev_yr.i_class_id
>AND curr_yr.i_category_id=prev_yr.i_category_id
>AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
>AND curr_yr.d_year=2002
>AND prev_yr.d_year=2002-1
>AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
> DECIMAL(17,2))<0.9
>  ORDER BY sales_cnt_diff
>  limit 100
> {code}
> throws a null pointer exception when executed via tez.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8589) TPCDS query 75 fails with tez

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8589:
-
Status: Patch Available  (was: Open)

> TPCDS query 75 fails with tez
> -
>
> Key: HIVE-8589
> URL: https://issues.apache.org/jira/browse/HIVE-8589
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-8589.1.patch
>
>
> {code}
> WITH all_sales AS (
>  SELECT d_year
>,i_brand_id
>,i_class_id
>,i_category_id
>,i_manufact_id
>,SUM(sales_cnt) AS sales_cnt
>,SUM(sales_amt) AS sales_amt
>  FROM (SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
>  ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
>FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
>   JOIN date_dim ON d_date_sk=cs_sold_date_sk
>   LEFT JOIN catalog_returns ON 
> (cs_order_number=cr_order_number 
> AND cs_item_sk=cr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
>  ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
>FROM store_sales JOIN item ON i_item_sk=ss_item_sk
> JOIN date_dim ON d_date_sk=ss_sold_date_sk
> LEFT JOIN store_returns ON 
> (ss_ticket_number=sr_ticket_number 
> AND ss_item_sk=sr_item_sk)
>WHERE i_category='Sports'
>UNION ALL
>SELECT d_year
>  ,i_brand_id
>  ,i_class_id
>  ,i_category_id
>  ,i_manufact_id
>  ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
>  ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
>FROM web_sales JOIN item ON i_item_sk=ws_item_sk
>   JOIN date_dim ON d_date_sk=ws_sold_date_sk
>   LEFT JOIN web_returns ON 
> (ws_order_number=wr_order_number 
> AND ws_item_sk=wr_item_sk)
>WHERE i_category='Sports') sales_detail
>  GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
>  SELECT  prev_yr.d_year AS prev_year
>   ,curr_yr.d_year AS year
>   ,curr_yr.i_brand_id
>   ,curr_yr.i_class_id
>   ,curr_yr.i_category_id
>   ,curr_yr.i_manufact_id
>   ,prev_yr.sales_cnt AS prev_yr_cnt
>   ,curr_yr.sales_cnt AS curr_yr_cnt
>   ,curr_yr.sales_cnt-prev_yr.sales_cnt AS 
> sales_cnt_diff
>   ,curr_yr.sales_amt-prev_yr.sales_amt AS 
> sales_amt_diff
>  FROM all_sales curr_yr, all_sales prev_yr
>  WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
>AND curr_yr.i_class_id=prev_yr.i_class_id
>AND curr_yr.i_category_id=prev_yr.i_category_id
>AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
>AND curr_yr.d_year=2002
>AND prev_yr.d_year=2002-1
>AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
> DECIMAL(17,2))<0.9
>  ORDER BY sales_cnt_diff
>  limit 100
> {code}
> throws a null pointer exception when executed via tez.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8589) TPCDS query 75 fails with tez

[Vikram Dixit K (JIRA)]

Vikram Dixit K created HIVE-8589:


 Summary: TPCDS query 75 fails with tez
 Key: HIVE-8589
 URL: https://issues.apache.org/jira/browse/HIVE-8589
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Vikram Dixit K
Priority: Critical
 Attachments: HIVE-8589.1.patch

{code}
WITH all_sales AS (
 SELECT d_year
   ,i_brand_id
   ,i_class_id
   ,i_category_id
   ,i_manufact_id
   ,SUM(sales_cnt) AS sales_cnt
   ,SUM(sales_amt) AS sales_amt
 FROM (SELECT d_year
 ,i_brand_id
 ,i_class_id
 ,i_category_id
 ,i_manufact_id
 ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
 ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
  JOIN date_dim ON d_date_sk=cs_sold_date_sk
  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
AND cs_item_sk=cr_item_sk)
   WHERE i_category='Sports'
   UNION ALL
   SELECT d_year
 ,i_brand_id
 ,i_class_id
 ,i_category_id
 ,i_manufact_id
 ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
 ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
JOIN date_dim ON d_date_sk=ss_sold_date_sk
LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_number 
AND ss_item_sk=sr_item_sk)
   WHERE i_category='Sports'
   UNION ALL
   SELECT d_year
 ,i_brand_id
 ,i_class_id
 ,i_category_id
 ,i_manufact_id
 ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
 ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
   FROM web_sales JOIN item ON i_item_sk=ws_item_sk
  JOIN date_dim ON d_date_sk=ws_sold_date_sk
  LEFT JOIN web_returns ON (ws_order_number=wr_order_number 
AND ws_item_sk=wr_item_sk)
   WHERE i_category='Sports') sales_detail
 GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
 SELECT  prev_yr.d_year AS prev_year
  ,curr_yr.d_year AS year
  ,curr_yr.i_brand_id
  ,curr_yr.i_class_id
  ,curr_yr.i_category_id
  ,curr_yr.i_manufact_id
  ,prev_yr.sales_cnt AS prev_yr_cnt
  ,curr_yr.sales_cnt AS curr_yr_cnt
  ,curr_yr.sales_cnt-prev_yr.sales_cnt AS sales_cnt_diff
  ,curr_yr.sales_amt-prev_yr.sales_amt AS sales_amt_diff
 FROM all_sales curr_yr, all_sales prev_yr
 WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
   AND curr_yr.i_class_id=prev_yr.i_class_id
   AND curr_yr.i_category_id=prev_yr.i_category_id
   AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
   AND curr_yr.d_year=2002
   AND prev_yr.d_year=2002-1
   AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
DECIMAL(17,2))<0.9
 ORDER BY sales_cnt_diff
 limit 100
{code}

throws a null pointer exception when executed via tez.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8582:
-
Description: CLEAR LIBRARY CACHE

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch, HIVE-8582.patch
>
>
> CLEAR LIBRARY CACHE



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Build failure on trunk

[Xuefu Zhang]

You can add CLEAR LIBRARY CACHE in the description for any JIRA, which will
clear local maven repo. I added it to HIVE-6165.

On Thu, Oct 23, 2014 at 9:09 PM, Gunther Hagleitner <
ghagleit...@hortonworks.com> wrote:

> Builds are running again (reverted patch). I've re-uploaded the patches
> that had a failed run because of it.
>
> Sorry about that...
>
> Thanks,
> Gunther.
>
> On Thu, Oct 23, 2014 at 8:07 PM, Gunther Hagleitner <
> gunther.hagleit...@gmail.com> wrote:
>
> > The builds are failing right now on trunk after I committed a change that
> > requires new/updated calcite libs. (Sorry about that).
> >
> > Is it possible for someone to wipe the .m2 cache on the build machine, so
> > it would download a new version with the changes?
> >
> > Thank you,
> > Gunther.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

[jira] [Updated] (HIVE-6165) Unify HivePreparedStatement from jdbc:hive and jdbc:hive2

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6165:
--
Description: 
org.apache.hadoop.hive.jdbc.HivePreparedStatement.class from the hive jdbc 
driver and org.apache.hive.jdbc.HivePreparedStatement.class from the hive2 jdbc 
drivers contain lots of duplicate code. 

Especially hive-HivePreparedStatement supports "setObject", while the hive2 
version does not.

Share more code between the two to avoid duplicate work and to make sure that 
both support the broadest possible feature set.

CLEAR LIBRARY CACHE

  was:
org.apache.hadoop.hive.jdbc.HivePreparedStatement.class from the hive jdbc 
driver and org.apache.hive.jdbc.HivePreparedStatement.class from the hive2 jdbc 
drivers contain lots of duplicate code. 

Especially hive-HivePreparedStatement supports "setObject", while the hive2 
version does not.

Share more code between the two to avoid duplicate work and to make sure that 
both support the broadest possible feature set.


> Unify HivePreparedStatement from jdbc:hive and jdbc:hive2
> -
>
> Key: HIVE-6165
> URL: https://issues.apache.org/jira/browse/HIVE-6165
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Reporter: Helmut Zechmann
>Priority: Minor
> Attachments: HIVE-6165.1.patch.txt, HIVE-6165.1.patch.txt, 
> HIVE-6165.2.patch, HIVE-6165.2.patch.txt
>
>
> org.apache.hadoop.hive.jdbc.HivePreparedStatement.class from the hive jdbc 
> driver and org.apache.hive.jdbc.HivePreparedStatement.class from the hive2 
> jdbc drivers contain lots of duplicate code. 
> Especially hive-HivePreparedStatement supports "setObject", while the hive2 
> version does not.
> Share more code between the two to avoid duplicate work and to make sure that 
> both support the broadest possible feature set.
> CLEAR LIBRARY CACHE



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Build failure on trunk

[Gunther Hagleitner]

Builds are running again (reverted patch). I've re-uploaded the patches
that had a failed run because of it.

Sorry about that...

Thanks,
Gunther.

On Thu, Oct 23, 2014 at 8:07 PM, Gunther Hagleitner <
gunther.hagleit...@gmail.com> wrote:

> The builds are failing right now on trunk after I committed a change that
> requires new/updated calcite libs. (Sorry about that).
>
> Is it possible for someone to wipe the .m2 cache on the build machine, so
> it would download a new version with the changes?
>
> Thank you,
> Gunther.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-8583) HIVE-8341 Cleanup & Test for hive.script.operator.env.blacklist

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182407#comment-14182407
 ] 

Gunther Hagleitner commented on HIVE-8583:
--

I think that's not only confusing, but it's a bug. The replacement has to 
happen otherwise the comparison will fail (and the entries won't be 
blacklisted). [~alangates] what do you think?

> HIVE-8341 Cleanup & Test for hive.script.operator.env.blacklist
> ---
>
> Key: HIVE-8583
> URL: https://issues.apache.org/jira/browse/HIVE-8583
> Project: Hive
>  Issue Type: Improvement
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-8583.1.patch
>
>
> [~alangates] added the following in HIVE-8341:
> {code}
> String bl = 
> hconf.get(HiveConf.ConfVars.HIVESCRIPT_ENV_BLACKLIST.toString());
> if (bl != null && bl.length() > 0) {
>   String[] bls = bl.split(",");
>   for (String b : bls) {
> b.replaceAll(".", "_");
> blackListedConfEntries.add(b);
>   }
> }
> {code}
> The {{replaceAll}} call is confusing as its result is not used at all.
> This patch contains the following:
> * Minor style modification (missorted modifiers)
> * Adds reading of default value for HIVESCRIPT_ENV_BLACKLIST
> * Removes replaceAll
> * Lets blackListed take a Configuration job as parameter which allowed me to 
> add a test for this



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8588) sqoop REST endpoint fails to send appropriate JDBC driver to the cluster

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-8588:


 Summary: sqoop REST endpoint fails to send appropriate JDBC driver 
to the cluster
 Key: HIVE-8588
 URL: https://issues.apache.org/jira/browse/HIVE-8588
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Affects Versions: 0.14.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


This is originally discovered by [~deepesh]
When running a Sqoop integration test from WebHCat
{noformat}
curl --show-error -d command="export -libjars 
hdfs:///tmp/mysql-connector-java.jar --connect 
jdbc:mysql://deepesh-c6-1.cs1cloud.internal/sqooptest --username sqoop 
--password passwd --export-dir /tmp/templeton_test_data/sqoop --table person" 
-d statusdir=sqoop.output -X POST 
"http://deepesh-c6-1.cs1cloud.internal:50111/templeton/v1/sqoop?user.name=hrt_qa";
{noformat}
the job is failing with the following error:
{noformat}
$ hadoop fs -cat /user/hrt_qa/sqoop.output/stderr
14/10/15 23:52:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.0.0-897
14/10/15 23:52:53 WARN tool.BaseSqoopTool: Setting your password on the 
command-line is insecure. Consider using -P instead.
14/10/15 23:52:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming 
resultset.
14/10/15 23:52:54 INFO tool.CodeGenTool: Beginning code generation
14/10/15 23:52:54 ERROR sqoop.Sqoop: Got exception running Sqoop: 
java.lang.RuntimeException: Could not load db driver class: 
com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: 
com.mysql.jdbc.Driver
at 
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)
at 
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at 
org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at 
org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at 
org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at 
org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at 
org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
{noformat}
Note that the Sqoop tar bundle does not contain the JDBC connector jar. I think 
the problem here maybe that the mysql connector jar added to libjars isn't 
available to the Sqoop tool which first connects to the database through JDBC 
driver to collect some table information before running the MR job. libjars 
will only add the connector jar for the MR job and not the local one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8395) CBO: enable by default

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8395:
-
Attachment: HIVE-8395.12.patch

Re-uploading to re-trigger build.

> CBO: enable by default
> --
>
> Key: HIVE-8395
> URL: https://issues.apache.org/jira/browse/HIVE-8395
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.15.0
>
> Attachments: HIVE-8395.01.patch, HIVE-8395.02.patch, 
> HIVE-8395.03.patch, HIVE-8395.04.patch, HIVE-8395.05.patch, 
> HIVE-8395.06.patch, HIVE-8395.07.patch, HIVE-8395.08.patch, 
> HIVE-8395.09.patch, HIVE-8395.10.patch, HIVE-8395.11.patch, 
> HIVE-8395.12.patch, HIVE-8395.12.patch, HIVE-8395.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8575) CBO: decimal_udf is broken by recent changes (and other tests have type changes)

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8575:
-
Attachment: HIVE-8575.patch

Re-uploading to re-trigger build.

> CBO: decimal_udf is broken by recent changes (and other tests have type 
> changes)
> 
>
> Key: HIVE-8575
> URL: https://issues.apache.org/jira/browse/HIVE-8575
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8575.patch, HIVE-8575.patch
>
>
> Seems like unqualified integral values went from ints to bigints
> {noformat}
> EXPLAIN SELECT histogram_numeric(key, 3) FROM DECIMAL_UDF
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only an integer 
> argument is accepted as parameter 2, but bigint was passed instead.
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFHistogramNumeric.getEvaluator(GenericUDAFHistogramNumeric.java:103)
> at 
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:47)
> at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1143)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:3717)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:4392)
> ... snip
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8435) Add identity project remover optimization

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8435:
-
Attachment: HIVE-8435.03.patch

Re-uploading 03 to retrigger build.

> Add identity project remover optimization
> -
>
> Key: HIVE-8435
> URL: https://issues.apache.org/jira/browse/HIVE-8435
> Project: Hive
>  Issue Type: New Feature
>  Components: Logical Optimizer
>Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8435.02.patch, HIVE-8435.03.patch, 
> HIVE-8435.03.patch, HIVE-8435.1.patch, HIVE-8435.patch
>
>
> In some cases there is an identity project in plan which is useless. Better 
> to optimize it away to avoid evaluating it without any benefit at runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8454) Select Operator does not rename column stats properly in case of select star

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8454:
-
Attachment: HIVE-8454.3.patch

Re-uploading .3 to re-trigger build.

> Select Operator does not rename column stats properly in case of select star
> 
>
> Key: HIVE-8454
> URL: https://issues.apache.org/jira/browse/HIVE-8454
> Project: Hive
>  Issue Type: Sub-task
>  Components: Physical Optimizer
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Prasanth J
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8454.1.patch, HIVE-8454.2.patch, HIVE-8454.3.patch, 
> HIVE-8454.3.patch
>
>
> The estimated data size of some Select Operators is 0. BytesBytesHashMap uses 
> data size to determine the estimated initial number of entries in the 
> hashmap. If this data size is 0 then exception is thrown (refer below)
> Query 
> {code}
> select count(*) from
>  store_sales
> JOIN store_returns ON store_sales.ss_item_sk = 
> store_returns.sr_item_sk and store_sales.ss_ticket_number = 
> store_returns.sr_ticket_number
> JOIN customer ON store_sales.ss_customer_sk = customer.c_customer_sk
> JOIN date_dim d1 ON store_sales.ss_sold_date_sk = d1.d_date_sk
> JOIN date_dim d2 ON customer.c_first_sales_date_sk = d2.d_date_sk 
> JOIN date_dim d3 ON customer.c_first_shipto_date_sk = d3.d_date_sk
> JOIN store ON store_sales.ss_store_sk = store.s_store_sk
>   JOIN item ON store_sales.ss_item_sk = item.i_item_sk
>   JOIN customer_demographics cd1 ON store_sales.ss_cdemo_sk= 
> cd1.cd_demo_sk
> JOIN customer_demographics cd2 ON customer.c_current_cdemo_sk = 
> cd2.cd_demo_sk
> JOIN promotion ON store_sales.ss_promo_sk = promotion.p_promo_sk
> JOIN household_demographics hd1 ON store_sales.ss_hdemo_sk = 
> hd1.hd_demo_sk
> JOIN household_demographics hd2 ON customer.c_current_hdemo_sk = 
> hd2.hd_demo_sk
> JOIN customer_address ad1 ON store_sales.ss_addr_sk = 
> ad1.ca_address_sk
> JOIN customer_address ad2 ON customer.c_current_addr_sk = 
> ad2.ca_address_sk
> JOIN income_band ib1 ON hd1.hd_income_band_sk = ib1.ib_income_band_sk
> JOIN income_band ib2 ON hd2.hd_income_band_sk = ib2.ib_income_band_sk
> JOIN
>  (select cs_item_sk
> ,sum(cs_ext_list_price) as 
> sale,sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit) as refund
>   from catalog_sales JOIN catalog_returns
>   ON catalog_sales.cs_item_sk = catalog_returns.cr_item_sk
> and catalog_sales.cs_order_number = catalog_returns.cr_order_number
>   group by cs_item_sk
>   having 
> sum(cs_ext_list_price)>2*sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit))
>  cs_ui
> ON store_sales.ss_item_sk = cs_ui.cs_item_sk
>   WHERE  
>  cd1.cd_marital_status <> cd2.cd_marital_status and
>  i_color in ('maroon','burnished','dim','steel','navajo','chocolate') 
> and
>  i_current_price between 35 and 35 + 10 and
>  i_current_price between 35 + 1 and 35 + 15
>and d1.d_year = 2001;
> {code}
> {code}
> ], TaskAttempt 3 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.AssertionError: Capacity must be a power of two
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:180)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:172)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:167)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: Capacity 
> must be a power of two
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSou

[jira] [Updated] (HIVE-8585) Constant folding should happen before ppd

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8585:
-
Attachment: HIVE-8585.patch

Re-uploading patch to re-trigger build.

> Constant folding should happen before ppd
> -
>
> Key: HIVE-8585
> URL: https://issues.apache.org/jira/browse/HIVE-8585
> Project: Hive
>  Issue Type: Improvement
>  Components: Logical Optimizer
>Affects Versions: 0.14.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8585.patch, HIVE-8585.patch
>
>
> will help {{NullScanOptimizer}} to kick in more places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8579) Guaranteed NPE in DDLSemanticAnalyzer

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8579:
-
Attachment: HIVE-8579.1.patch

Re-uploading patch to trigger build.

> Guaranteed NPE in DDLSemanticAnalyzer
> -
>
> Key: HIVE-8579
> URL: https://issues.apache.org/jira/browse/HIVE-8579
> Project: Hive
>  Issue Type: Bug
>Reporter: Lars Francke
>Assignee: Jason Dere
> Attachments: HIVE-8579.1.patch, HIVE-8579.1.patch
>
>
> This was added by [~jdere] in HIVE-8411. I don't fully understand the code 
> (i.e. what it means when desc is null) but I'm sure, Jason, you can fix it 
> without much trouble?
> {code}
> if (desc == null || 
> !AlterTableDesc.doesAlterTableTypeSupportPartialPartitionSpec(desc.getOp())) {
>   throw new SemanticException( 
> ErrorMsg.ALTER_TABLE_TYPE_PARTIAL_PARTITION_SPEC_NO_SUPPORTED, 
> desc.getOp().name());
> } else if (!conf.getBoolVar(HiveConf.ConfVars.DYNAMICPARTITIONING)) {
>   throw new SemanticException(ErrorMsg.DYNAMIC_PARTITION_DISABLED);
> }
> {code}
> You check for whether {{desc}} is null but then use it to do {{desc.getOp()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8577) Cannot deserialize Avro schema with a map with null values

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8577:
-
Attachment: HIVE-8577.1.patch

Reuploading same patch to retrigger build

> Cannot deserialize Avro schema with a map with null values
> -
>
> Key: HIVE-8577
> URL: https://issues.apache.org/jira/browse/HIVE-8577
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>  Labels: regression
> Attachments: HIVE-8577.1.patch, HIVE-8577.1.patch, 
> map_null_schema.avro, map_null_val.avro
>
>
> An avro table with a map column that contains null values
> cannot be deserialized when running the select statement.
> Create the following table:
> {noformat}
> CREATE TABLE avro_table (avreau_col_1 map) 
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' 
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' 
> TBLPROPERTIES ('avro.schema.url'='file:///tmp/map_null_schema.avro');
> {noformat}
> Then load the avro data:
> {noformat}
> LOAD DATA LOCAL INPATH '/tmp/map_null_val.avro' OVERWRITE INTO TABLE 
> avro_table;
> {noformat}
> And do the select (it fails):
> {noformat}
> SELECT * FROM avro_table;
> Error: java.io.IOException: org.apache.avro.AvroRuntimeException: Not a map: 
> "null" (state=,code=0)
> {noformat}
> This is a regression bug (it works correctly on hive 0.13.1 version).
> This is the output that hive 0.13.1 displays:
> {noformat}
> {"key3":"val3","key4":null}
> {"key3":"val3","key4":null}
> {"key1":null,"key2":"val2"}
> {"key3":"val3","key4":null}
> {"key3":"val3","key4":null}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6165) Unify HivePreparedStatement from jdbc:hive and jdbc:hive2

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6165:
-
Status: Open  (was: Patch Available)

> Unify HivePreparedStatement from jdbc:hive and jdbc:hive2
> -
>
> Key: HIVE-6165
> URL: https://issues.apache.org/jira/browse/HIVE-6165
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Reporter: Helmut Zechmann
>Priority: Minor
> Attachments: HIVE-6165.1.patch.txt, HIVE-6165.1.patch.txt, 
> HIVE-6165.2.patch, HIVE-6165.2.patch.txt
>
>
> org.apache.hadoop.hive.jdbc.HivePreparedStatement.class from the hive jdbc 
> driver and org.apache.hive.jdbc.HivePreparedStatement.class from the hive2 
> jdbc drivers contain lots of duplicate code. 
> Especially hive-HivePreparedStatement supports "setObject", while the hive2 
> version does not.
> Share more code between the two to avoid duplicate work and to make sure that 
> both support the broadest possible feature set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6165) Unify HivePreparedStatement from jdbc:hive and jdbc:hive2

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6165:
-
Attachment: HIVE-6165.2.patch

Reuploading patch .2 after fixing build breakage.

> Unify HivePreparedStatement from jdbc:hive and jdbc:hive2
> -
>
> Key: HIVE-6165
> URL: https://issues.apache.org/jira/browse/HIVE-6165
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Reporter: Helmut Zechmann
>Priority: Minor
> Attachments: HIVE-6165.1.patch.txt, HIVE-6165.1.patch.txt, 
> HIVE-6165.2.patch, HIVE-6165.2.patch.txt
>
>
> org.apache.hadoop.hive.jdbc.HivePreparedStatement.class from the hive jdbc 
> driver and org.apache.hive.jdbc.HivePreparedStatement.class from the hive2 
> jdbc drivers contain lots of duplicate code. 
> Especially hive-HivePreparedStatement supports "setObject", while the hive2 
> version does not.
> Share more code between the two to avoid duplicate work and to make sure that 
> both support the broadest possible feature set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6165) Unify HivePreparedStatement from jdbc:hive and jdbc:hive2

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6165:
-
Status: Patch Available  (was: Open)

> Unify HivePreparedStatement from jdbc:hive and jdbc:hive2
> -
>
> Key: HIVE-6165
> URL: https://issues.apache.org/jira/browse/HIVE-6165
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Reporter: Helmut Zechmann
>Priority: Minor
> Attachments: HIVE-6165.1.patch.txt, HIVE-6165.1.patch.txt, 
> HIVE-6165.2.patch, HIVE-6165.2.patch.txt
>
>
> org.apache.hadoop.hive.jdbc.HivePreparedStatement.class from the hive jdbc 
> driver and org.apache.hive.jdbc.HivePreparedStatement.class from the hive2 
> jdbc drivers contain lots of duplicate code. 
> Especially hive-HivePreparedStatement supports "setObject", while the hive2 
> version does not.
> Share more code between the two to avoid duplicate work and to make sure that 
> both support the broadest possible feature set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8021) CBO: support CTAS and insert ... select

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8021:
-
Status: Open  (was: Patch Available)

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.01.patch, 
> HIVE-8021.02.patch, HIVE-8021.03.patch, HIVE-8021.04.patch, 
> HIVE-8021.05.patch, HIVE-8021.06.patch, HIVE-8021.06.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8021) CBO: support CTAS and insert ... select

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8021:
-
Attachment: HIVE-8021.06.patch

Reuploading .6 after fixing the build.

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.01.patch, 
> HIVE-8021.02.patch, HIVE-8021.03.patch, HIVE-8021.04.patch, 
> HIVE-8021.05.patch, HIVE-8021.06.patch, HIVE-8021.06.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8021) CBO: support CTAS and insert ... select

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8021:
-
Status: Patch Available  (was: Open)

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.01.patch, 
> HIVE-8021.02.patch, HIVE-8021.03.patch, HIVE-8021.04.patch, 
> HIVE-8021.05.patch, HIVE-8021.06.patch, HIVE-8021.06.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8582:
-
Status: Patch Available  (was: Reopened)

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch, HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8582:
-
Attachment: HIVE-8582.patch

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch, HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8587) Vectorized Extract operator needs to update the Vectorization Context column map

[Matt McCline (JIRA)]

Matt McCline created HIVE-8587:
--

 Summary: Vectorized Extract operator needs to update the 
Vectorization Context column map
 Key: HIVE-8587
 URL: https://issues.apache.org/jira/browse/HIVE-8587
 Project: Hive
  Issue Type: Bug
  Components: Tez, Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
 Fix For: 0.14.0


Fails on Tez:

{noformat}
hive> SET hive.vectorized.execution.enabled=true;
hive> set hive.support.concurrency=true;
hive> set hive.enforce.bucketing=true;
hive> CREATE TABLE non_orc_table(a INT, b STRING) CLUSTERED BY(a) INTO 2 
BUCKETS STORED AS sequencefile; 
insert into table non_orc_table values(1, 'one'),(1, 'one'), (2, 'two'),(3, 
'three'); select a, b from non_orc_table order by a;
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8532) return code of "source xxx" clause is missing

[Gordon Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182375#comment-14182375
 ] 

Gordon Wang commented on HIVE-8532:
---

Looks like the UT failure is not caused by this patch. The failure UT is not in 
the changed code path.

> return code of "source xxx" clause is missing
> -
>
> Key: HIVE-8532
> URL: https://issues.apache.org/jira/browse/HIVE-8532
> Project: Hive
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: 0.12.0, 0.13.1
>Reporter: Gordon Wang
> Attachments: HIVE-8532.patch
>
>
> When executing "source "  clause, hive client driver does not catch 
> the return code of this command.
> This behaviour causes an issue when running hive query in Oozie workflow.
> When the "source" clause is put into a Oozie workflow, Oozie can not get the 
> return code of this command. Thus, Oozie consider the "source" clause as 
> successful all the time. 
> So, when the "source" clause fails, the hive query does not abort and the 
> oozie workflow does not abort either.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8535) Enable compile time skew join optimization for spark [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182374#comment-14182374
 ] 

Hive QA commented on HIVE-8535:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676807/HIVE-8535.2-spark.patch

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 6831 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_timestamp_funcs
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/259/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/259/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-259/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12676807 - PreCommit-HIVE-SPARK-Build

> Enable compile time skew join optimization for spark [Spark Branch]
> ---
>
> Key: HIVE-8535
> URL: https://issues.apache.org/jira/browse/HIVE-8535
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-8535.1-spark.patch, HIVE-8535.2-spark.patch
>
>
> Sub-task of HIVE-8406



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8586) Record counters aren't updated correctly for vectorized queries

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8586:
-
Attachment: HIVE-8586.1.patch

> Record counters aren't updated correctly for vectorized queries
> ---
>
> Key: HIVE-8586
> URL: https://issues.apache.org/jira/browse/HIVE-8586
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.14.0
>
> Attachments: HIVE-8586.1.patch
>
>
> Counts batches not rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8586) Record counters aren't updated correctly for vectorized queries

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8586:
-
Status: Patch Available  (was: Open)

> Record counters aren't updated correctly for vectorized queries
> ---
>
> Key: HIVE-8586
> URL: https://issues.apache.org/jira/browse/HIVE-8586
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.14.0
>
> Attachments: HIVE-8586.1.patch
>
>
> Counts batches not rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8586) Record counters aren't updated correctly for vectorized queries

[Gunther Hagleitner (JIRA)]

Gunther Hagleitner created HIVE-8586:


 Summary: Record counters aren't updated correctly for vectorized 
queries
 Key: HIVE-8586
 URL: https://issues.apache.org/jira/browse/HIVE-8586
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
 Fix For: 0.14.0


Counts batches not rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182372#comment-14182372
 ] 

Gunther Hagleitner commented on HIVE-8582:
--

Reverted - causes build failures

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner reopened HIVE-8582:
--

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8584) Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size delta byte(s) shorter on Windows than Linux

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182369#comment-14182369
 ] 

Gopal V commented on HIVE-8584:
---

Zlib uses the java.util.zip.* libraries (not the libhadoop.so implementation).

If you are suggesting that the JDK internals do not match between Linux & 
Windows - can you file an  OpenJDK bug for this?

> Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size 
> delta byte(s) shorter on Windows than Linux
> -
>
> Key: HIVE-8584
> URL: https://issues.apache.org/jira/browse/HIVE-8584
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
> Environment: Windows
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>Priority: Critical
> Attachments: HIVE-8584.1.patch, orc-win-none-1.dump, 
> orc-win-none-2.dump, orc-win-snappy-1.dump, orc-win-snappy-2.dump, 
> orc-win-zlib-1.dump, orc-win-zlib-2.dump, orc_analyze.q
>
>
> repo steps:
> 1. run query orc_analyze.q
> 2. hive --orcfiledump 
> run 1 and 2 on PST timezone on Linux, and one more time on other timezone 
> e.g. CST on Windows.
> Compare two target orc file dumping. Windows orc file is 1 byte shorter than 
> Linux one.
> That's the case even if running 1 and 2 on Windows for different timezones, 
> however, no problem on Linux.
> The issue only exists by using ZLIB mode, eventually OS native compression 
> lib is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failure on trunk

[Gunther Hagleitner]

The builds are failing right now on trunk after I committed a change that
requires new/updated calcite libs. (Sorry about that).

Is it possible for someone to wipe the .m2 cache on the build machine, so
it would download a new version with the changes?

Thank you,
Gunther.

[jira] [Commented] (HIVE-8395) CBO: enable by default

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182364#comment-14182364
 ] 

Hive QA commented on HIVE-8395:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676793/HIVE-8395.12.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1435/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1435/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1435/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result

[jira] [Updated] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8582:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8575) CBO: decimal_udf is broken by recent changes (and other tests have type changes)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182362#comment-14182362
 ] 

Hive QA commented on HIVE-8575:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676794/HIVE-8575.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1434/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1434/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1434/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result, a

[jira] [Commented] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182361#comment-14182361
 ] 

Gunther Hagleitner commented on HIVE-8582:
--

Committed to trunk and branch.

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8580) Support LateralViewJoinOperator and LateralViewForwardOperator in stats annotation

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182360#comment-14182360
 ] 

Prasanth J commented on HIVE-8580:
--

I meant HIVE-8454 in my previous comment.

> Support LateralViewJoinOperator and LateralViewForwardOperator in stats 
> annotation
> --
>
> Key: HIVE-8580
> URL: https://issues.apache.org/jira/browse/HIVE-8580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Affects Versions: 0.14.0, 0.15.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8580.1.patch
>
>
> HIVE-8454 showed some test failures related to lateral view stats showing 
> data size as 0. The reason for that is column expression map for 
> LateralViewForwardOperator is not populated which results column lists 
> becoming empty. As a result neededColumns and referencedColumns will be 
> empty. Hence no column stats will be fetched making data size as 0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8580) Support LateralViewJoinOperator and LateralViewForwardOperator in stats annotation

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-8580:
-
Attachment: HIVE-8580.1.patch

This patch must go in after HIVE-8584 is committed. I will run the lateral view 
q tests once HIVE-8584 is committed. Will make patch available after that.

> Support LateralViewJoinOperator and LateralViewForwardOperator in stats 
> annotation
> --
>
> Key: HIVE-8580
> URL: https://issues.apache.org/jira/browse/HIVE-8580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Affects Versions: 0.14.0, 0.15.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8580.1.patch
>
>
> HIVE-8454 showed some test failures related to lateral view stats showing 
> data size as 0. The reason for that is column expression map for 
> LateralViewForwardOperator is not populated which results column lists 
> becoming empty. As a result neededColumns and referencedColumns will be 
> empty. Hence no column stats will be fetched making data size as 0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8435) Add identity project remover optimization

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182355#comment-14182355
 ] 

Hive QA commented on HIVE-8435:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676790/HIVE-8435.03.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1433/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1433/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1433/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result

[jira] [Commented] (HIVE-8454) Select Operator does not rename column stats properly in case of select star

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182353#comment-14182353
 ] 

Hive QA commented on HIVE-8454:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676788/HIVE-8454.3.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1432/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1432/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1432/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1432/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/0.20/target shims/0.20S/target 
shims/0.23/target shims/aggregator/target shims/common/target 
shims/common-secure/target metastore/target common/target common/src/gen 
serde/target ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1633991.

At revision 1633991.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12676788 - PreCommit-HIVE-TRUNK-Build

> Select Operator does not rename column stats properly in case of select star
> 
>
> Key: HIVE-8454
> URL: https://issues.apache.org/jira/browse/HIVE-8454
> Project: Hive
>  Issue Type: Sub-task
>  Components: Physical Optimizer
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Prasanth J
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8454.1.patch, HIVE-8454.2.patch, HIVE-8454.3.patch
>
>
> The estimated data size of some Select Operators is 0. BytesBytesHashMap uses 
> data size to determine the estimated initial number of entries in the 
> hashmap. If this data size is 0 then exception is thrown (refer below)
> Query 
> {code}
> select count(*) from
>  store_sales
> JOIN store_returns ON store_sales.ss_item_sk = 
> store_returns.sr_item_sk and store_sales.ss_ticket_number = 
> store_returns.sr_ticket_number
> JOIN customer ON store_sales.ss_customer_sk = customer.c_customer_sk
> JOIN date_dim d1 ON store_sales.ss_sold_date_sk = d1.d_date_sk
> JOIN date_dim d2 ON customer.c_first_sales_date_sk = d2.d_date_sk 
> JOIN date_dim d3 ON customer.c_first_shipto_date_sk = d3.d_date_sk
> JOIN store ON store_sales.ss_store_sk = store.s_store_sk
>   JOIN item ON store_sales.ss_item_sk = item.i_item_sk
>   JOIN customer_demographics cd1 ON store_sales.ss_cdemo_sk= 
> cd1.cd_demo_sk
> JOIN customer_demographics cd2 ON customer.c_current_cdemo_sk = 
> cd2.cd_demo_sk
> JOIN promotion ON store_sales.ss_promo_sk = promo

[jira] [Commented] (HIVE-8585) Constant folding should happen before ppd

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182351#comment-14182351
 ] 

Hive QA commented on HIVE-8585:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676783/HIVE-8585.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1431/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1431/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1431/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result, a

[jira] [Commented] (HIVE-8579) Guaranteed NPE in DDLSemanticAnalyzer

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182348#comment-14182348
 ] 

Hive QA commented on HIVE-8579:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676782/HIVE-8579.1.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1430/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1430/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1430/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result,

[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8409:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to branch and trunk. Thanks [~vikram.dixit]!

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch, HIVE-8409.10.patch, 
> HIVE-8409.11.patch, HIVE-8409.2.patch, HIVE-8409.3.patch, HIVE-8409.7.patch, 
> HIVE-8409.8.patch, HIVE-8409.9.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8573) Fix some non-deterministic vectorization tests

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182344#comment-14182344
 ] 

Hive QA commented on HIVE-8573:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676776/HIVE-8573.2.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1429/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1429/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1429/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result,

[jira] [Commented] (HIVE-8583) HIVE-8341 Cleanup & Test for hive.script.operator.env.blacklist

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182341#comment-14182341
 ] 

Hive QA commented on HIVE-8583:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676749/HIVE-8583.1.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1428/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1428/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1428/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result,

[jira] [Commented] (HIVE-8577) Cannot deserialize Avro schema with a map with null values

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182338#comment-14182338
 ] 

Hive QA commented on HIVE-8577:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676748/HIVE-8577.1.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1427/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1427/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1427/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result,

[jira] [Commented] (HIVE-8582) Outer Join Simplification is broken

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182335#comment-14182335
 ] 

Hive QA commented on HIVE-8582:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676743/HIVE-8582.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1426/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1426/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1426/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1426/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java'
Reverted 
'itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/0.20/target shims/0.20S/target 
shims/0.23/target shims/aggregator/target shims/common/target 
shims/common-secure/target metastore/target common/target common/src/gen 
serde/target ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1633987.

At revision 1633987.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12676743 - PreCommit-HIVE-TRUNK-Build

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-6165) Unify HivePreparedStatement from jdbc:hive and jdbc:hive2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182334#comment-14182334
 ] 

Hive QA commented on HIVE-6165:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676735/HIVE-6165.2.patch.txt

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1425/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1425/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1425/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a res

[jira] [Commented] (HIVE-8584) Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size delta byte(s) shorter on Windows than Linux

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182333#comment-14182333
 ] 

Lefty Leverenz commented on HIVE-8584:
--

The wiki has a few places this could be mentioned:

* [ORC -- Compression | 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-Compression]
* orc.compress table property in [ORC -- HiveQLSyntax | 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax]
* [hive.exec.orc.default.compress in Configuration Properties | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.exec.orc.default.compress]
* no ORC discussion, just Gzip & Bzip2 for TextFile (doc needs updating):  
[Compressed Data Storage | 
https://cwiki.apache.org/confluence/display/Hive/CompressedStorage]

> Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size 
> delta byte(s) shorter on Windows than Linux
> -
>
> Key: HIVE-8584
> URL: https://issues.apache.org/jira/browse/HIVE-8584
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
> Environment: Windows
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>Priority: Critical
> Attachments: HIVE-8584.1.patch, orc-win-none-1.dump, 
> orc-win-none-2.dump, orc-win-snappy-1.dump, orc-win-snappy-2.dump, 
> orc-win-zlib-1.dump, orc-win-zlib-2.dump, orc_analyze.q
>
>
> repo steps:
> 1. run query orc_analyze.q
> 2. hive --orcfiledump 
> run 1 and 2 on PST timezone on Linux, and one more time on other timezone 
> e.g. CST on Windows.
> Compare two target orc file dumping. Windows orc file is 1 byte shorter than 
> Linux one.
> That's the case even if running 1 and 2 on Windows for different timezones, 
> however, no problem on Linux.
> The issue only exists by using ZLIB mode, eventually OS native compression 
> lib is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8021) CBO: support CTAS and insert ... select

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182332#comment-14182332
 ] 

Hive QA commented on HIVE-8021:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676729/HIVE-8021.06.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1424/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1424/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1424/

Messages:
{noformat}
 This message was trimmed, see log for full details 

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 
6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple 
alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER 
KW_BY" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION 
KW_ALL" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" 
using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT 
KW_INTO" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL 
KW_VIEW" using multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple 
alternatives: 8, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE 
KW_BY" using multiple alternatives: 2, 9

As a result

[jira] [Updated] (HIVE-8563) Running annotate_stats_join_pkfk.q in TestMiniTezCliDriver is causing NPE

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8563:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to branch and trunk.

> Running annotate_stats_join_pkfk.q in TestMiniTezCliDriver is causing NPE
> -
>
> Key: HIVE-8563
> URL: https://issues.apache.org/jira/browse/HIVE-8563
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth J
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8563.2.patch, HIVE-8563.WIP.patch
>
>
> I added a test case as part of HIVE-8549 to annotate_stats_join_pkfk.q. This 
> test case fails with NullPointerException when we run using 
> TestMiniTezCliDriver. Here is the stack trace
> {code}
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.plan.PlanUtils.getFieldSchemasFromRowSchema(PlanUtils.java:548)
> at 
> org.apache.hadoop.hive.ql.optimizer.ReduceSinkMapJoinProc.process(ReduceSinkMapJoinProc.java:239)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:87)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.startWalking(GenTezWorkWalker.java:69)
> at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:367)
> at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:202)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10057)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1070)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1132)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1007)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:997)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8409) SMB joins fail intermittently on tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182322#comment-14182322
 ] 

Hive QA commented on HIVE-8409:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676724/HIVE-8409.11.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6563 tests executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1423/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1423/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1423/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12676724 - PreCommit-HIVE-TRUNK-Build

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch, HIVE-8409.10.patch, 
> HIVE-8409.11.patch, HIVE-8409.2.patch, HIVE-8409.3.patch, HIVE-8409.7.patch, 
> HIVE-8409.8.patch, HIVE-8409.9.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8535) Enable compile time skew join optimization for spark [Spark Branch]

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-8535:
-
Attachment: HIVE-8535.2-spark.patch

Enable more tests.

> Enable compile time skew join optimization for spark [Spark Branch]
> ---
>
> Key: HIVE-8535
> URL: https://issues.apache.org/jira/browse/HIVE-8535
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-8535.1-spark.patch, HIVE-8535.2-spark.patch
>
>
> Sub-task of HIVE-8406



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8563) Running annotate_stats_join_pkfk.q in TestMiniTezCliDriver is causing NPE

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182308#comment-14182308
 ] 

Gunther Hagleitner commented on HIVE-8563:
--

+1

> Running annotate_stats_join_pkfk.q in TestMiniTezCliDriver is causing NPE
> -
>
> Key: HIVE-8563
> URL: https://issues.apache.org/jira/browse/HIVE-8563
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth J
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8563.2.patch, HIVE-8563.WIP.patch
>
>
> I added a test case as part of HIVE-8549 to annotate_stats_join_pkfk.q. This 
> test case fails with NullPointerException when we run using 
> TestMiniTezCliDriver. Here is the stack trace
> {code}
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.plan.PlanUtils.getFieldSchemasFromRowSchema(PlanUtils.java:548)
> at 
> org.apache.hadoop.hive.ql.optimizer.ReduceSinkMapJoinProc.process(ReduceSinkMapJoinProc.java:239)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:87)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.startWalking(GenTezWorkWalker.java:69)
> at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:367)
> at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:202)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10057)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1070)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1132)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1007)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:997)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8563) Running annotate_stats_join_pkfk.q in TestMiniTezCliDriver is causing NPE

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8563:
-
Fix Version/s: 0.14.0

> Running annotate_stats_join_pkfk.q in TestMiniTezCliDriver is causing NPE
> -
>
> Key: HIVE-8563
> URL: https://issues.apache.org/jira/browse/HIVE-8563
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth J
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8563.2.patch, HIVE-8563.WIP.patch
>
>
> I added a test case as part of HIVE-8549 to annotate_stats_join_pkfk.q. This 
> test case fails with NullPointerException when we run using 
> TestMiniTezCliDriver. Here is the stack trace
> {code}
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.plan.PlanUtils.getFieldSchemasFromRowSchema(PlanUtils.java:548)
> at 
> org.apache.hadoop.hive.ql.optimizer.ReduceSinkMapJoinProc.process(ReduceSinkMapJoinProc.java:239)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:87)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.walk(GenTezWorkWalker.java:103)
> at 
> org.apache.hadoop.hive.ql.parse.GenTezWorkWalker.startWalking(GenTezWorkWalker.java:69)
> at 
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:367)
> at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:202)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10057)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:417)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1070)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1132)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1007)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:997)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8580) Support LateralViewJoinOperator and LateralViewForwardOperator in stats annotation

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8580:
-
Fix Version/s: (was: 0.13.0)
   0.14.0

> Support LateralViewJoinOperator and LateralViewForwardOperator in stats 
> annotation
> --
>
> Key: HIVE-8580
> URL: https://issues.apache.org/jira/browse/HIVE-8580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Affects Versions: 0.14.0, 0.15.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Critical
> Fix For: 0.14.0
>
>
> HIVE-8454 showed some test failures related to lateral view stats showing 
> data size as 0. The reason for that is column expression map for 
> LateralViewForwardOperator is not populated which results column lists 
> becoming empty. As a result neededColumns and referencedColumns will be 
> empty. Hence no column stats will be fetched making data size as 0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8454) Select Operator does not rename column stats properly in case of select star

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182304#comment-14182304
 ] 

Gunther Hagleitner commented on HIVE-8454:
--

+1

> Select Operator does not rename column stats properly in case of select star
> 
>
> Key: HIVE-8454
> URL: https://issues.apache.org/jira/browse/HIVE-8454
> Project: Hive
>  Issue Type: Sub-task
>  Components: Physical Optimizer
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Prasanth J
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8454.1.patch, HIVE-8454.2.patch, HIVE-8454.3.patch
>
>
> The estimated data size of some Select Operators is 0. BytesBytesHashMap uses 
> data size to determine the estimated initial number of entries in the 
> hashmap. If this data size is 0 then exception is thrown (refer below)
> Query 
> {code}
> select count(*) from
>  store_sales
> JOIN store_returns ON store_sales.ss_item_sk = 
> store_returns.sr_item_sk and store_sales.ss_ticket_number = 
> store_returns.sr_ticket_number
> JOIN customer ON store_sales.ss_customer_sk = customer.c_customer_sk
> JOIN date_dim d1 ON store_sales.ss_sold_date_sk = d1.d_date_sk
> JOIN date_dim d2 ON customer.c_first_sales_date_sk = d2.d_date_sk 
> JOIN date_dim d3 ON customer.c_first_shipto_date_sk = d3.d_date_sk
> JOIN store ON store_sales.ss_store_sk = store.s_store_sk
>   JOIN item ON store_sales.ss_item_sk = item.i_item_sk
>   JOIN customer_demographics cd1 ON store_sales.ss_cdemo_sk= 
> cd1.cd_demo_sk
> JOIN customer_demographics cd2 ON customer.c_current_cdemo_sk = 
> cd2.cd_demo_sk
> JOIN promotion ON store_sales.ss_promo_sk = promotion.p_promo_sk
> JOIN household_demographics hd1 ON store_sales.ss_hdemo_sk = 
> hd1.hd_demo_sk
> JOIN household_demographics hd2 ON customer.c_current_hdemo_sk = 
> hd2.hd_demo_sk
> JOIN customer_address ad1 ON store_sales.ss_addr_sk = 
> ad1.ca_address_sk
> JOIN customer_address ad2 ON customer.c_current_addr_sk = 
> ad2.ca_address_sk
> JOIN income_band ib1 ON hd1.hd_income_band_sk = ib1.ib_income_band_sk
> JOIN income_band ib2 ON hd2.hd_income_band_sk = ib2.ib_income_band_sk
> JOIN
>  (select cs_item_sk
> ,sum(cs_ext_list_price) as 
> sale,sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit) as refund
>   from catalog_sales JOIN catalog_returns
>   ON catalog_sales.cs_item_sk = catalog_returns.cr_item_sk
> and catalog_sales.cs_order_number = catalog_returns.cr_order_number
>   group by cs_item_sk
>   having 
> sum(cs_ext_list_price)>2*sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit))
>  cs_ui
> ON store_sales.ss_item_sk = cs_ui.cs_item_sk
>   WHERE  
>  cd1.cd_marital_status <> cd2.cd_marital_status and
>  i_color in ('maroon','burnished','dim','steel','navajo','chocolate') 
> and
>  i_current_price between 35 and 35 + 10 and
>  i_current_price between 35 + 1 and 35 + 15
>and d1.d_year = 2001;
> {code}
> {code}
> ], TaskAttempt 3 failed, info=[Error: Failure while running 
> task:java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.AssertionError: Capacity must be a power of two
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:187)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:142)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:180)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:172)
>   at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:167)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: Capacity 
> must be a power of two
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.ja

[jira] [Commented] (HIVE-8528) Add remote Spark client to Hive [Spark Branch]

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182297#comment-14182297
 ] 

Lefty Leverenz commented on HIVE-8528:
--

Oh, messages crossing paths -- thanks [~xuefuz] & [~vanzin].

> Add remote Spark client to Hive [Spark Branch]
> --
>
> Key: HIVE-8528
> URL: https://issues.apache.org/jira/browse/HIVE-8528
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
> Fix For: spark-branch
>
> Attachments: HIVE-8528.1-spark-client.patch, HIVE-8528.1-spark.patch, 
> HIVE-8528.2-spark.patch, HIVE-8528.2-spark.patch, HIVE-8528.3-spark.patch
>
>
> For the time being, at least, we've decided to build the Spark client (see 
> SPARK-3215) inside Hive. This task tracks merging the ongoing work into the 
> Spark branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8528) Add remote Spark client to Hive [Spark Branch]

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182296#comment-14182296
 ] 

Lefty Leverenz commented on HIVE-8528:
--

Well, if it needs any setup or configuration you could put that in the "Hive on 
Spark:  Getting Started" doc.  Usage notes too.  If it will need documentation 
later on, you can add the label TODOC-SPARK.  (Oops, I added a TODOC label by 
accident while checking the name of the Spark label.  Gone now.)  A release 
note could hold some doc notes, but it won't get published from the branch.  Of 
course these comments can hold doc notes too.

I just want to make sure docs don't get forgotten, but maybe this doesn't need 
any docs.  Thanks.

> Add remote Spark client to Hive [Spark Branch]
> --
>
> Key: HIVE-8528
> URL: https://issues.apache.org/jira/browse/HIVE-8528
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
> Fix For: spark-branch
>
> Attachments: HIVE-8528.1-spark-client.patch, HIVE-8528.1-spark.patch, 
> HIVE-8528.2-spark.patch, HIVE-8528.2-spark.patch, HIVE-8528.3-spark.patch
>
>
> For the time being, at least, we've decided to build the Spark client (see 
> SPARK-3215) inside Hive. This task tracks merging the ongoing work into the 
> Spark branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8528) Add remote Spark client to Hive [Spark Branch]

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182293#comment-14182293
 ] 

Xuefu Zhang commented on HIVE-8528:
---

Hi [~leftylev], As explained by Marcelo, it's our goal to hide this completely 
from the user, and thus user doc doesn't need to cover it. Thanks for asking 
though.

> Add remote Spark client to Hive [Spark Branch]
> --
>
> Key: HIVE-8528
> URL: https://issues.apache.org/jira/browse/HIVE-8528
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
> Fix For: spark-branch
>
> Attachments: HIVE-8528.1-spark-client.patch, HIVE-8528.1-spark.patch, 
> HIVE-8528.2-spark.patch, HIVE-8528.2-spark.patch, HIVE-8528.3-spark.patch
>
>
> For the time being, at least, we've decided to build the Spark client (see 
> SPARK-3215) inside Hive. This task tracks merging the ongoing work into the 
> Spark branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8528) Add remote Spark client to Hive [Spark Branch]

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-8528:
-
Labels: TODOC  (was: )

> Add remote Spark client to Hive [Spark Branch]
> --
>
> Key: HIVE-8528
> URL: https://issues.apache.org/jira/browse/HIVE-8528
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
>  Labels: TODOC
> Fix For: spark-branch
>
> Attachments: HIVE-8528.1-spark-client.patch, HIVE-8528.1-spark.patch, 
> HIVE-8528.2-spark.patch, HIVE-8528.2-spark.patch, HIVE-8528.3-spark.patch
>
>
> For the time being, at least, we've decided to build the Spark client (see 
> SPARK-3215) inside Hive. This task tracks merging the ongoing work into the 
> Spark branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8528) Add remote Spark client to Hive [Spark Branch]

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-8528:
-
Labels:   (was: TODOC)

> Add remote Spark client to Hive [Spark Branch]
> --
>
> Key: HIVE-8528
> URL: https://issues.apache.org/jira/browse/HIVE-8528
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
> Fix For: spark-branch
>
> Attachments: HIVE-8528.1-spark-client.patch, HIVE-8528.1-spark.patch, 
> HIVE-8528.2-spark.patch, HIVE-8528.2-spark.patch, HIVE-8528.3-spark.patch
>
>
> For the time being, at least, we've decided to build the Spark client (see 
> SPARK-3215) inside Hive. This task tracks merging the ongoing work into the 
> Spark branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8528) Add remote Spark client to Hive [Spark Branch]

[Marcelo Vanzin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182278#comment-14182278
 ] 

Marcelo Vanzin commented on HIVE-8528:
--

Hi Lefty, what kind of documentation are you looking for? This is, at the 
moment, targeted at internal Hive use only, so having nice end-user 
documentation is not currently a goal. (In fact, I should probably go and add 
those annotations to the classes.)

> Add remote Spark client to Hive [Spark Branch]
> --
>
> Key: HIVE-8528
> URL: https://issues.apache.org/jira/browse/HIVE-8528
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
> Fix For: spark-branch
>
> Attachments: HIVE-8528.1-spark-client.patch, HIVE-8528.1-spark.patch, 
> HIVE-8528.2-spark.patch, HIVE-8528.2-spark.patch, HIVE-8528.3-spark.patch
>
>
> For the time being, at least, we've decided to build the Spark client (see 
> SPARK-3215) inside Hive. This task tracks merging the ongoing work into the 
> Spark branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8457) MapOperator initialization fails when multiple Spark threads is enabled [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182275#comment-14182275
 ] 

Hive QA commented on HIVE-8457:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12676787/HIVE-8457.1-spark.patch

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6809 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_dynamic_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_count
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/258/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/258/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-258/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12676787 - PreCommit-HIVE-SPARK-Build

> MapOperator initialization fails when multiple Spark threads is enabled 
> [Spark Branch]
> --
>
> Key: HIVE-8457
> URL: https://issues.apache.org/jira/browse/HIVE-8457
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8457.1-spark.patch
>
>
> Currently, on the Spark branch, each thread it is bound with a thread-local 
> IOContext, which gets initialized when we generates an input {{HadoopRDD}}, 
> and later used in {{MapOperator}}, {{FilterOperator}}, etc.
> And, given the introduction of HIVE-8118, we may have multiple downstream 
> RDDs that share the same input {{HadoopRDD}}, and we would like to have the 
> {{HadoopRDD}} to be cached, to avoid scanning the same table multiple times. 
> A typical case would be like the following:
> {noformat}
>  inputRDD inputRDD
> ||
>MT_11MT_12
> ||
>RT_1 RT_2
> {noformat}
> Here, {{MT_11}} and {{MT_12}} are {{MapTran}} from a splitted {{MapWork}},
> and {{RT_1}} and {{RT_2}} are two {{ReduceTran}}. Note that, this example is 
> simplified, as we may also have {{ShuffleTran}} between {{MapTran}} and 
> {{ReduceTran}}.
> When multiple Spark threads are running, {{MT_11}} may be executed first, and 
> it will ask for an iterator from the {{HadoopRDD}} will trigger the creation 
> of the iterator, which in turn triggers the initialization of the 
> {{IOContext}} associated with that particular thread.
> *Now, the problem is*: before {{MT_12}} starts executing, it will also ask 
> for an iterator from the
> {{HadoopRDD}}, and since the RDD is already cached, instead of creating a new 
> iterator, it will just fetch it from the cached result. However, *this will 
> skip the initialization of the IOContext associated with this particular 
> thread*. And, when {{MT_12}} starts executing, it will try to initialize the 
> {{MapOperator}}, but since the {{IOContext}} is not initialized, this will 
> fail miserably. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8584) Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size delta byte(s) shorter on Windows than Linux

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182274#comment-14182274
 ] 

Prasanth J commented on HIVE-8584:
--

Its not a good idea to change the default encoding just for file size change.

> Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size 
> delta byte(s) shorter on Windows than Linux
> -
>
> Key: HIVE-8584
> URL: https://issues.apache.org/jira/browse/HIVE-8584
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
> Environment: Windows
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>Priority: Critical
> Attachments: HIVE-8584.1.patch, orc-win-none-1.dump, 
> orc-win-none-2.dump, orc-win-snappy-1.dump, orc-win-snappy-2.dump, 
> orc-win-zlib-1.dump, orc-win-zlib-2.dump, orc_analyze.q
>
>
> repo steps:
> 1. run query orc_analyze.q
> 2. hive --orcfiledump 
> run 1 and 2 on PST timezone on Linux, and one more time on other timezone 
> e.g. CST on Windows.
> Compare two target orc file dumping. Windows orc file is 1 byte shorter than 
> Linux one.
> That's the case even if running 1 and 2 on Windows for different timezones, 
> however, no problem on Linux.
> The issue only exists by using ZLIB mode, eventually OS native compression 
> lib is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8584) Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size delta byte(s) shorter on Windows than Linux

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182268#comment-14182268
 ] 

Jason Dere commented on HIVE-8584:
--

When you say not portable, do you mean that an ORC file with zlib compression 
created on Linux cannot be read correctly on Windows? Or is the data readable 
between platforms, just that the size of the files is different when created on 
Linux vs Windows?

> Setting hive.exec.orc.default.compress to ZLIB will lead to orc file size 
> delta byte(s) shorter on Windows than Linux
> -
>
> Key: HIVE-8584
> URL: https://issues.apache.org/jira/browse/HIVE-8584
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
> Environment: Windows
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>Priority: Critical
> Attachments: HIVE-8584.1.patch, orc-win-none-1.dump, 
> orc-win-none-2.dump, orc-win-snappy-1.dump, orc-win-snappy-2.dump, 
> orc-win-zlib-1.dump, orc-win-zlib-2.dump, orc_analyze.q
>
>
> repo steps:
> 1. run query orc_analyze.q
> 2. hive --orcfiledump 
> run 1 and 2 on PST timezone on Linux, and one more time on other timezone 
> e.g. CST on Windows.
> Compare two target orc file dumping. Windows orc file is 1 byte shorter than 
> Linux one.
> That's the case even if running 1 and 2 on Windows for different timezones, 
> however, no problem on Linux.
> The issue only exists by using ZLIB mode, eventually OS native compression 
> lib is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8528) Add remote Spark client to Hive [Spark Branch]

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182269#comment-14182269
 ] 

Lefty Leverenz commented on HIVE-8528:
--

Does this need any documentation, or is it already covered in the PDF attached 
to SPARK-3215?

* [SPARK-3215 -- RemoteSparkContext.pdf | 
https://issues.apache.org/jira/secure/attachment/12664234/RemoteSparkContext.pdf]
* [Hive on Spark:  Getting Started | 
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started]

> Add remote Spark client to Hive [Spark Branch]
> --
>
> Key: HIVE-8528
> URL: https://issues.apache.org/jira/browse/HIVE-8528
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Marcelo Vanzin
>Assignee: Marcelo Vanzin
> Fix For: spark-branch
>
> Attachments: HIVE-8528.1-spark-client.patch, HIVE-8528.1-spark.patch, 
> HIVE-8528.2-spark.patch, HIVE-8528.2-spark.patch, HIVE-8528.3-spark.patch
>
>
> For the time being, at least, we've decided to build the Spark client (see 
> SPARK-3215) inside Hive. This task tracks merging the ongoing work into the 
> Spark branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8582) Outer Join Simplification is broken

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182271#comment-14182271
 ] 

Gunther Hagleitner commented on HIVE-8582:
--

Ran tests locally - passed.

> Outer Join Simplification is broken
> ---
>
> Key: HIVE-8582
> URL: https://issues.apache.org/jira/browse/HIVE-8582
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8582.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27117: HIVE-8457 - MapOperator initialization fails when multiple Spark threads is enabled [Spark Branch]

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27117/#review58183
---



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java


We don't need this, as this class is only used for Spark.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java


Let's give a less conflicting name, such as SPARK_MAP_IO_CONTEXT. Same 
below. Better define a constant in SparkUtils.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java


We may need to copy other fields in IOContext besides input path.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java


Same as above



ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java


We need to copy every field.


- Xuefu Zhang


On Oct. 23, 2014, 11:56 p.m., Chao Sun wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27117/
> ---
> 
> (Updated Oct. 23, 2014, 11:56 p.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8457
> https://issues.apache.org/jira/browse/HIVE-8457
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently, on the Spark branch, each thread it is bound with a thread-local 
> IOContext, which gets initialized when we generates an input HadoopRDD, and 
> later used in MapOperator, FilterOperator, etc.
> And, given the introduction of HIVE-8118, we may have multiple downstream 
> RDDs that share the same input HadoopRDD, and we would like to have the 
> HadoopRDD to be cached, to avoid scanning the same table multiple times. A 
> typical case would be like the following:
>  inputRDD inputRDD
> ||
>MT_11MT_12
> ||
>RT_1 RT_2
> Here, MT_11 and MT_12 are MapTran from a splitted MapWork,
> and RT_1 and RT_2 are two ReduceTran. Note that, this example is simplified, 
> as we may also have ShuffleTran between MapTran and ReduceTran.
> When multiple Spark threads are running, MT_11 may be executed first, and it 
> will ask for an iterator from the HadoopRDD will trigger the creation of the 
> iterator, which in turn triggers the initialization of the IOContext 
> associated with that particular thread.
> Now, the problem is: before MT_12 starts executing, it will also ask for an 
> iterator from the
> HadoopRDD, and since the RDD is already cached, instead of creating a new 
> iterator, it will just fetch it from the cached result. However, this will 
> skip the initialization of the IOContext associated with this particular 
> thread. And, when MT_12 starts executing, it will try to initialize the 
> MapOperator, but since the IOContext is not initialized, this will fail 
> miserably.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
> 20ea977 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 
> 00a6f3d 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java 
> 58e1ceb 
> 
> Diff: https://reviews.apache.org/r/27117/diff/
> 
> 
> Testing
> ---
> 
> All multi-insertion related tests are passing on my local machine.
> 
> 
> Thanks,
> 
> Chao Sun
> 
>

[jira] [Updated] (HIVE-8575) CBO: decimal_udf is broken by recent changes (and other tests have type changes)

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8575:
---
Status: Patch Available  (was: Open)

> CBO: decimal_udf is broken by recent changes (and other tests have type 
> changes)
> 
>
> Key: HIVE-8575
> URL: https://issues.apache.org/jira/browse/HIVE-8575
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8575.patch
>
>
> Seems like unqualified integral values went from ints to bigints
> {noformat}
> EXPLAIN SELECT histogram_numeric(key, 3) FROM DECIMAL_UDF
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only an integer 
> argument is accepted as parameter 2, but bigint was passed instead.
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFHistogramNumeric.getEvaluator(GenericUDAFHistogramNumeric.java:103)
> at 
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:47)
> at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1143)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:3717)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:4392)
> ... snip
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8560) SerDes that do not inherit AbstractSerDe do not get table properties during initialize()

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8560:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk/branch-0.14

> SerDes that do not inherit AbstractSerDe do not get table properties during 
> initialize()
> 
>
> Key: HIVE-8560
> URL: https://issues.apache.org/jira/browse/HIVE-8560
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.14.0
>
> Attachments: HIVE-8560.1.patch
>
>
> Looks like this may have been introduced during HIVE-6835.  During 
> initialize(), 3rd party SerDes which do not inherit AbstractSerDe end up 
> getting a Properties object created by 
> SerDeUtils.createOverlayedProperties().  This properties object receives the 
> table properties as defaults.  So looking up a key by name will yield the 
> default value, but a call like getKeys() will not show any of the table 
> properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8395) CBO: enable by default

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8395:
---
Attachment: HIVE-8395.12.patch

some out file was changed on trunk... rebase and update

> CBO: enable by default
> --
>
> Key: HIVE-8395
> URL: https://issues.apache.org/jira/browse/HIVE-8395
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.15.0
>
> Attachments: HIVE-8395.01.patch, HIVE-8395.02.patch, 
> HIVE-8395.03.patch, HIVE-8395.04.patch, HIVE-8395.05.patch, 
> HIVE-8395.06.patch, HIVE-8395.07.patch, HIVE-8395.08.patch, 
> HIVE-8395.09.patch, HIVE-8395.10.patch, HIVE-8395.11.patch, 
> HIVE-8395.12.patch, HIVE-8395.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8575) CBO: decimal_udf is broken by recent changes (and other tests have type changes)

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8575:
---
Attachment: HIVE-8575.patch

> CBO: decimal_udf is broken by recent changes (and other tests have type 
> changes)
> 
>
> Key: HIVE-8575
> URL: https://issues.apache.org/jira/browse/HIVE-8575
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-8575.patch
>
>
> Seems like unqualified integral values went from ints to bigints
> {noformat}
> EXPLAIN SELECT histogram_numeric(key, 3) FROM DECIMAL_UDF
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only an integer 
> argument is accepted as parameter 2, but bigint was passed instead.
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFHistogramNumeric.getEvaluator(GenericUDAFHistogramNumeric.java:103)
> at 
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:47)
> at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1143)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:3717)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:4392)
> ... snip
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8435) Add identity project remover optimization

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8435:
---
Attachment: HIVE-8435.03.patch

Improve the patch - identity is not checked correctly in the original; instead 
of checking it correctly try to translate column names.

> Add identity project remover optimization
> -
>
> Key: HIVE-8435
> URL: https://issues.apache.org/jira/browse/HIVE-8435
> Project: Hive
>  Issue Type: New Feature
>  Components: Logical Optimizer
>Affects Versions: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8435.02.patch, HIVE-8435.03.patch, 
> HIVE-8435.1.patch, HIVE-8435.patch
>
>
> In some cases there is an identity project in plan which is useless. Better 
> to optimize it away to avoid evaluating it without any benefit at runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8457) MapOperator initialization fails when multiple Spark threads is enabled [Spark Branch]

[Chao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1418#comment-1418
 ] 

Chao commented on HIVE-8457:


This patch also enables caching on input RDD, so once this is resolved, 
HIVE-8437 should also be resolved.

> MapOperator initialization fails when multiple Spark threads is enabled 
> [Spark Branch]
> --
>
> Key: HIVE-8457
> URL: https://issues.apache.org/jira/browse/HIVE-8457
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8457.1-spark.patch
>
>
> Currently, on the Spark branch, each thread it is bound with a thread-local 
> IOContext, which gets initialized when we generates an input {{HadoopRDD}}, 
> and later used in {{MapOperator}}, {{FilterOperator}}, etc.
> And, given the introduction of HIVE-8118, we may have multiple downstream 
> RDDs that share the same input {{HadoopRDD}}, and we would like to have the 
> {{HadoopRDD}} to be cached, to avoid scanning the same table multiple times. 
> A typical case would be like the following:
> {noformat}
>  inputRDD inputRDD
> ||
>MT_11MT_12
> ||
>RT_1 RT_2
> {noformat}
> Here, {{MT_11}} and {{MT_12}} are {{MapTran}} from a splitted {{MapWork}},
> and {{RT_1}} and {{RT_2}} are two {{ReduceTran}}. Note that, this example is 
> simplified, as we may also have {{ShuffleTran}} between {{MapTran}} and 
> {{ReduceTran}}.
> When multiple Spark threads are running, {{MT_11}} may be executed first, and 
> it will ask for an iterator from the {{HadoopRDD}} will trigger the creation 
> of the iterator, which in turn triggers the initialization of the 
> {{IOContext}} associated with that particular thread.
> *Now, the problem is*: before {{MT_12}} starts executing, it will also ask 
> for an iterator from the
> {{HadoopRDD}}, and since the RDD is already cached, instead of creating a new 
> iterator, it will just fetch it from the cached result. However, *this will 
> skip the initialization of the IOContext associated with this particular 
> thread*. And, when {{MT_12}} starts executing, it will try to initialize the 
> {{MapOperator}}, but since the {{IOContext}} is not initialized, this will 
> fail miserably. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8457) MapOperator initialization fails when multiple Spark threads is enabled [Spark Branch]

[Chao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182221#comment-14182221
 ] 

Chao commented on HIVE-8457:


There are two problems I found:
1. one thread may be reused for multiple queries, and therefore, if one 
{{IOContext}} is bound to one thread, and the {{IOContext}} is not initialized, 
we need to clear it when we close the map record handler;
2. synchronization is needed - the thread who doesn't have {{IOContext}} 
initialized may try to retrieve before the thread who have {{IOContext}} 
initialized saving it. Right now, a thread will save {{IOContext}} right after 
initializing it. But, this is not a perfect solution.

> MapOperator initialization fails when multiple Spark threads is enabled 
> [Spark Branch]
> --
>
> Key: HIVE-8457
> URL: https://issues.apache.org/jira/browse/HIVE-8457
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8457.1-spark.patch
>
>
> Currently, on the Spark branch, each thread it is bound with a thread-local 
> IOContext, which gets initialized when we generates an input {{HadoopRDD}}, 
> and later used in {{MapOperator}}, {{FilterOperator}}, etc.
> And, given the introduction of HIVE-8118, we may have multiple downstream 
> RDDs that share the same input {{HadoopRDD}}, and we would like to have the 
> {{HadoopRDD}} to be cached, to avoid scanning the same table multiple times. 
> A typical case would be like the following:
> {noformat}
>  inputRDD inputRDD
> ||
>MT_11MT_12
> ||
>RT_1 RT_2
> {noformat}
> Here, {{MT_11}} and {{MT_12}} are {{MapTran}} from a splitted {{MapWork}},
> and {{RT_1}} and {{RT_2}} are two {{ReduceTran}}. Note that, this example is 
> simplified, as we may also have {{ShuffleTran}} between {{MapTran}} and 
> {{ReduceTran}}.
> When multiple Spark threads are running, {{MT_11}} may be executed first, and 
> it will ask for an iterator from the {{HadoopRDD}} will trigger the creation 
> of the iterator, which in turn triggers the initialization of the 
> {{IOContext}} associated with that particular thread.
> *Now, the problem is*: before {{MT_12}} starts executing, it will also ask 
> for an iterator from the
> {{HadoopRDD}}, and since the RDD is already cached, instead of creating a new 
> iterator, it will just fetch it from the cached result. However, *this will 
> skip the initialization of the IOContext associated with this particular 
> thread*. And, when {{MT_12}} starts executing, it will try to initialize the 
> {{MapOperator}}, but since the {{IOContext}} is not initialized, this will 
> fail miserably. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8560) SerDes that do not inherit AbstractSerDe do not get table properties during initialize()

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182212#comment-14182212
 ] 

Jason Dere commented on HIVE-8560:
--

[~vikram.dixit] would like to add to 0.14

> SerDes that do not inherit AbstractSerDe do not get table properties during 
> initialize()
> 
>
> Key: HIVE-8560
> URL: https://issues.apache.org/jira/browse/HIVE-8560
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.14.0
>
> Attachments: HIVE-8560.1.patch
>
>
> Looks like this may have been introduced during HIVE-6835.  During 
> initialize(), 3rd party SerDes which do not inherit AbstractSerDe end up 
> getting a Properties object created by 
> SerDeUtils.createOverlayedProperties().  This properties object receives the 
> table properties as defaults.  So looking up a key by name will yield the 
> default value, but a call like getKeys() will not show any of the table 
> properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8560) SerDes that do not inherit AbstractSerDe do not get table properties during initialize()

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8560:
-
Fix Version/s: 0.14.0

> SerDes that do not inherit AbstractSerDe do not get table properties during 
> initialize()
> 
>
> Key: HIVE-8560
> URL: https://issues.apache.org/jira/browse/HIVE-8560
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.14.0
>
> Attachments: HIVE-8560.1.patch
>
>
> Looks like this may have been introduced during HIVE-6835.  During 
> initialize(), 3rd party SerDes which do not inherit AbstractSerDe end up 
> getting a Properties object created by 
> SerDeUtils.createOverlayedProperties().  This properties object receives the 
> table properties as defaults.  So looking up a key by name will yield the 
> default value, but a call like getKeys() will not show any of the table 
> properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)