subject:"\[jira\] \[Commented\] \(HIVE\-10083\) SMBJoin fails in case one table is uninitialized"

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

2015-09-25 Thread JIRA


[ 
https://issues.apache.org/jira/browse/HIVE-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907774#comment-14907774
 ] 

Alain Blankenburg-Schröder commented on HIVE-10083:
---

Thanks for your email.
Unfortunately, you will no longer be able to reach me under this mailaccount.
Please note that your email will not be forwarded.
For urgent inquiries, please contact my colleague Philipp Kölmel via email 
p.koel...@bigpoint.net.
Best regards,
Alain Blankenburg-Schröder


> SMBJoin fails in case one table is uninitialized
> 
>
> Key: HIVE-10083
> URL: https://issues.apache.org/jira/browse/HIVE-10083
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 0.13.0
> Environment: MapR Hive 0.13
>Reporter: Alain Schröder
>Assignee: Na Yang
>Priority: Minor
> Fix For: 1.2.0
>
> Attachments: HIVE-10083.patch
>
>
> We experience IndexOutOfBoundsException in a SMBJoin in the case on the 
> tables used for the JOIN is uninitialized. Everything works if both are 
> uninitialized or initialized.
> {code}
> 2015-03-24 09:12:58,967 ERROR [main]: ql.Driver 
> (SessionState.java:printError(545)) - FAILED: IndexOutOfBoundsException 
> Index: 0, Size: 0
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.fillMappingBigTableBucketFileNameToSmallTableBucketFileNames(AbstractBucketJoinProc.java:486)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.convertMapJoinToBucketMapJoin(AbstractBucketJoinProc.java:429)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToBucketMapJoin(AbstractSMBJoinProc.java:540)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToSMBJoin(AbstractSMBJoinProc.java:549)
> at 
> org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:51)
> [...]
> {code}
> Simplest way to reproduce:
> {code}
> SET hive.enforce.sorting=true;
> SET hive.enforce.bucketing=true;
> SET hive.exec.dynamic.partition=true;
> SET mapreduce.reduce.import.limit=-1;
> SET hive.optimize.bucketmapjoin=true;
> SET hive.optimize.bucketmapjoin.sortedmerge=true;
> SET hive.auto.convert.join=true;
> SET hive.auto.convert.sortmerge.join=true;
> SET hive.auto.convert.sortmerge.join.noconditionaltask=true;
> CREATE DATABASE IF NOT EXISTS tmp;
> USE tmp;
> CREATE  TABLE `test1` (
>   `foo` bigint )
> CLUSTERED BY (
>   foo)
> SORTED BY (
>   foo ASC)
> INTO 384 BUCKETS
> stored as orc;
> CREATE  TABLE `test2`(
>   `foo` bigint )
> CLUSTERED BY (
>   foo)
> SORTED BY (
>   foo ASC)
> INTO 384 BUCKETS
> STORED AS ORC;
> -- Initialize ONE table of the two tables with any data.
> INSERT INTO TABLE test1 SELECT foo FROM table_with_some_content LIMIT 100;
> SELECT t1.foo, t2.foo
> FROM test1 t1 INNER JOIN test2 t2 
> ON (t1.foo = t2.foo);
> {code}
> I took a look at the Procedure 
> fillMappingBigTableBucketFileNameToSmallTableBucketFileNames in 
> AbstractBucketJoinProc.java and it does not seem to have changed from our 
> MapR Hive 0.13 to current snapshot, so this should be also an error in the 
> current Version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

2015-09-25 Thread Lefty Leverenz (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907773#comment-14907773
 ] 

Lefty Leverenz commented on HIVE-10083:
---

Version note: This was also committed to branch-1.0 (for release 1.0.2) on 
September 24 with commit a7618dfb9f93eab922f1939680dca4ae5d5a8f6b.

> SMBJoin fails in case one table is uninitialized
> 
>
> Key: HIVE-10083
> URL: https://issues.apache.org/jira/browse/HIVE-10083
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 0.13.0
> Environment: MapR Hive 0.13
>Reporter: Alain Schröder
>Assignee: Na Yang
>Priority: Minor
> Fix For: 1.2.0
>
> Attachments: HIVE-10083.patch
>
>
> We experience IndexOutOfBoundsException in a SMBJoin in the case on the 
> tables used for the JOIN is uninitialized. Everything works if both are 
> uninitialized or initialized.
> {code}
> 2015-03-24 09:12:58,967 ERROR [main]: ql.Driver 
> (SessionState.java:printError(545)) - FAILED: IndexOutOfBoundsException 
> Index: 0, Size: 0
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.fillMappingBigTableBucketFileNameToSmallTableBucketFileNames(AbstractBucketJoinProc.java:486)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.convertMapJoinToBucketMapJoin(AbstractBucketJoinProc.java:429)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToBucketMapJoin(AbstractSMBJoinProc.java:540)
> at 
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToSMBJoin(AbstractSMBJoinProc.java:549)
> at 
> org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:51)
> [...]
> {code}
> Simplest way to reproduce:
> {code}
> SET hive.enforce.sorting=true;
> SET hive.enforce.bucketing=true;
> SET hive.exec.dynamic.partition=true;
> SET mapreduce.reduce.import.limit=-1;
> SET hive.optimize.bucketmapjoin=true;
> SET hive.optimize.bucketmapjoin.sortedmerge=true;
> SET hive.auto.convert.join=true;
> SET hive.auto.convert.sortmerge.join=true;
> SET hive.auto.convert.sortmerge.join.noconditionaltask=true;
> CREATE DATABASE IF NOT EXISTS tmp;
> USE tmp;
> CREATE  TABLE `test1` (
>   `foo` bigint )
> CLUSTERED BY (
>   foo)
> SORTED BY (
>   foo ASC)
> INTO 384 BUCKETS
> stored as orc;
> CREATE  TABLE `test2`(
>   `foo` bigint )
> CLUSTERED BY (
>   foo)
> SORTED BY (
>   foo ASC)
> INTO 384 BUCKETS
> STORED AS ORC;
> -- Initialize ONE table of the two tables with any data.
> INSERT INTO TABLE test1 SELECT foo FROM table_with_some_content LIMIT 100;
> SELECT t1.foo, t2.foo
> FROM test1 t1 INNER JOIN test2 t2 
> ON (t1.foo = t2.foo);
> {code}
> I took a look at the Procedure 
> fillMappingBigTableBucketFileNameToSmallTableBucketFileNames in 
> AbstractBucketJoinProc.java and it does not seem to have changed from our 
> MapR Hive 0.13 to current snapshot, so this should be also an error in the 
> current Version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

2015-03-31 Thread Chao (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388940#comment-14388940
 ] 

Chao commented on HIVE-10083:
-

+1. I think the test failure is unrelated.

 SMBJoin fails in case one table is uninitialized
 

 Key: HIVE-10083
 URL: https://issues.apache.org/jira/browse/HIVE-10083
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 0.13.0
 Environment: MapR Hive 0.13
Reporter: Alain Schröder
Assignee: Na Yang
Priority: Minor
 Attachments: HIVE-10083.patch


 We experience IndexOutOfBoundsException in a SMBJoin in the case on the 
 tables used for the JOIN is uninitialized. Everything works if both are 
 uninitialized or initialized.
 {code}
 2015-03-24 09:12:58,967 ERROR [main]: ql.Driver 
 (SessionState.java:printError(545)) - FAILED: IndexOutOfBoundsException 
 Index: 0, Size: 0
 java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
 at java.util.ArrayList.rangeCheck(ArrayList.java:635)
 at java.util.ArrayList.get(ArrayList.java:411)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.fillMappingBigTableBucketFileNameToSmallTableBucketFileNames(AbstractBucketJoinProc.java:486)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.convertMapJoinToBucketMapJoin(AbstractBucketJoinProc.java:429)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToBucketMapJoin(AbstractSMBJoinProc.java:540)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToSMBJoin(AbstractSMBJoinProc.java:549)
 at 
 org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:51)
 [...]
 {code}
 Simplest way to reproduce:
 {code}
 SET hive.enforce.sorting=true;
 SET hive.enforce.bucketing=true;
 SET hive.exec.dynamic.partition=true;
 SET mapreduce.reduce.import.limit=-1;
 SET hive.optimize.bucketmapjoin=true;
 SET hive.optimize.bucketmapjoin.sortedmerge=true;
 SET hive.auto.convert.join=true;
 SET hive.auto.convert.sortmerge.join=true;
 SET hive.auto.convert.sortmerge.join.noconditionaltask=true;
 CREATE DATABASE IF NOT EXISTS tmp;
 USE tmp;
 CREATE  TABLE `test1` (
   `foo` bigint )
 CLUSTERED BY (
   foo)
 SORTED BY (
   foo ASC)
 INTO 384 BUCKETS
 stored as orc;
 CREATE  TABLE `test2`(
   `foo` bigint )
 CLUSTERED BY (
   foo)
 SORTED BY (
   foo ASC)
 INTO 384 BUCKETS
 STORED AS ORC;
 -- Initialize ONE table of the two tables with any data.
 INSERT INTO TABLE test1 SELECT foo FROM table_with_some_content LIMIT 100;
 SELECT t1.foo, t2.foo
 FROM test1 t1 INNER JOIN test2 t2 
 ON (t1.foo = t2.foo);
 {code}
 I took a look at the Procedure 
 fillMappingBigTableBucketFileNameToSmallTableBucketFileNames in 
 AbstractBucketJoinProc.java and it does not seem to have changed from our 
 MapR Hive 0.13 to current snapshot, so this should be also an error in the 
 current Version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

2015-03-31 Thread Na Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388973#comment-14388973
 ] 

Na Yang commented on HIVE-10083:


Thank you [~csun] for the code review. I ran the q test for smb_mapjoin_8.q on 
my local machine and it was successful.  

 SMBJoin fails in case one table is uninitialized
 

 Key: HIVE-10083
 URL: https://issues.apache.org/jira/browse/HIVE-10083
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 0.13.0
 Environment: MapR Hive 0.13
Reporter: Alain Schröder
Assignee: Na Yang
Priority: Minor
 Attachments: HIVE-10083.patch


 We experience IndexOutOfBoundsException in a SMBJoin in the case on the 
 tables used for the JOIN is uninitialized. Everything works if both are 
 uninitialized or initialized.
 {code}
 2015-03-24 09:12:58,967 ERROR [main]: ql.Driver 
 (SessionState.java:printError(545)) - FAILED: IndexOutOfBoundsException 
 Index: 0, Size: 0
 java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
 at java.util.ArrayList.rangeCheck(ArrayList.java:635)
 at java.util.ArrayList.get(ArrayList.java:411)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.fillMappingBigTableBucketFileNameToSmallTableBucketFileNames(AbstractBucketJoinProc.java:486)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.convertMapJoinToBucketMapJoin(AbstractBucketJoinProc.java:429)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToBucketMapJoin(AbstractSMBJoinProc.java:540)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToSMBJoin(AbstractSMBJoinProc.java:549)
 at 
 org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:51)
 [...]
 {code}
 Simplest way to reproduce:
 {code}
 SET hive.enforce.sorting=true;
 SET hive.enforce.bucketing=true;
 SET hive.exec.dynamic.partition=true;
 SET mapreduce.reduce.import.limit=-1;
 SET hive.optimize.bucketmapjoin=true;
 SET hive.optimize.bucketmapjoin.sortedmerge=true;
 SET hive.auto.convert.join=true;
 SET hive.auto.convert.sortmerge.join=true;
 SET hive.auto.convert.sortmerge.join.noconditionaltask=true;
 CREATE DATABASE IF NOT EXISTS tmp;
 USE tmp;
 CREATE  TABLE `test1` (
   `foo` bigint )
 CLUSTERED BY (
   foo)
 SORTED BY (
   foo ASC)
 INTO 384 BUCKETS
 stored as orc;
 CREATE  TABLE `test2`(
   `foo` bigint )
 CLUSTERED BY (
   foo)
 SORTED BY (
   foo ASC)
 INTO 384 BUCKETS
 STORED AS ORC;
 -- Initialize ONE table of the two tables with any data.
 INSERT INTO TABLE test1 SELECT foo FROM table_with_some_content LIMIT 100;
 SELECT t1.foo, t2.foo
 FROM test1 t1 INNER JOIN test2 t2 
 ON (t1.foo = t2.foo);
 {code}
 I took a look at the Procedure 
 fillMappingBigTableBucketFileNameToSmallTableBucketFileNames in 
 AbstractBucketJoinProc.java and it does not seem to have changed from our 
 MapR Hive 0.13 to current snapshot, so this should be also an error in the 
 current Version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

2015-03-30 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387886#comment-14387886
 ] 

Hive QA commented on HIVE-10083:




{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12708266/HIVE-10083.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8685 tests executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-smb_mapjoin_8.q - did not produce a TEST-*.xml file
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3211/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3211/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3211/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12708266 - PreCommit-HIVE-TRUNK-Build

 SMBJoin fails in case one table is uninitialized
 

 Key: HIVE-10083
 URL: https://issues.apache.org/jira/browse/HIVE-10083
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 0.13.0
 Environment: MapR Hive 0.13
Reporter: Alain Schröder
Assignee: Na Yang
Priority: Minor
 Attachments: HIVE-10083.patch


 We experience IndexOutOfBoundsException in a SMBJoin in the case on the 
 tables used for the JOIN is uninitialized. Everything works if both are 
 uninitialized or initialized.
 {code}
 2015-03-24 09:12:58,967 ERROR [main]: ql.Driver 
 (SessionState.java:printError(545)) - FAILED: IndexOutOfBoundsException 
 Index: 0, Size: 0
 java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
 at java.util.ArrayList.rangeCheck(ArrayList.java:635)
 at java.util.ArrayList.get(ArrayList.java:411)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.fillMappingBigTableBucketFileNameToSmallTableBucketFileNames(AbstractBucketJoinProc.java:486)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractBucketJoinProc.convertMapJoinToBucketMapJoin(AbstractBucketJoinProc.java:429)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToBucketMapJoin(AbstractSMBJoinProc.java:540)
 at 
 org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.convertJoinToSMBJoin(AbstractSMBJoinProc.java:549)
 at 
 org.apache.hadoop.hive.ql.optimizer.SortedMergeJoinProc.process(SortedMergeJoinProc.java:51)
 [...]
 {code}
 Simplest way to reproduce:
 {code}
 SET hive.enforce.sorting=true;
 SET hive.enforce.bucketing=true;
 SET hive.exec.dynamic.partition=true;
 SET mapreduce.reduce.import.limit=-1;
 SET hive.optimize.bucketmapjoin=true;
 SET hive.optimize.bucketmapjoin.sortedmerge=true;
 SET hive.auto.convert.join=true;
 SET hive.auto.convert.sortmerge.join=true;
 SET hive.auto.convert.sortmerge.join.noconditionaltask=true;
 CREATE DATABASE IF NOT EXISTS tmp;
 USE tmp;
 CREATE  TABLE `test1` (
   `foo` bigint )
 CLUSTERED BY (
   foo)
 SORTED BY (
   foo ASC)
 INTO 384 BUCKETS
 stored as orc;
 CREATE  TABLE `test2`(
   `foo` bigint )
 CLUSTERED BY (
   foo)
 SORTED BY (
   foo ASC)
 INTO 384 BUCKETS
 STORED AS ORC;
 -- Initialize ONE table of the two tables with any data.
 INSERT INTO TABLE test1 SELECT foo FROM table_with_some_content LIMIT 100;
 SELECT t1.foo, t2.foo
 FROM test1 t1 INNER JOIN test2 t2 
 ON (t1.foo = t2.foo);
 {code}
 I took a look at the Procedure 
 fillMappingBigTableBucketFileNameToSmallTableBucketFileNames in 
 AbstractBucketJoinProc.java and it does not seem to have changed from our 
 MapR Hive 0.13 to current snapshot, so this should be also an error in the 
 current Version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

[jira] [Commented] (HIVE-10083) SMBJoin fails in case one table is uninitialized

5 matches

Site Navigation

Mail list logo

Footer information