[jira] [Commented] (HIVE-14404) Allow delimiterfordsv to use multiple-character delimiters
[ https://issues.apache.org/jira/browse/HIVE-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407341#comment-15407341 ] Marta Kuczora commented on HIVE-14404: -- The Super CSV library is used to generate the output if the outputformat is dsv. Unfortunately Super CSV doesn’t support multiple characters as delimiter. They have an open issue for this: https://github.com/super-csv/super-csv/issues/10 I can think of two possible ways to solve this issue. 1. Implement the change in Super CSV and get a new version of it. However I don’t know anything about the lifecycle of Super CSV releases, so getting a new version might take some time. 2. Implement a new OutputFormat in Hive for the dsv output format which supports the multi-character delimiters. This could use the same logic as currently if the delimiter is only one character, so only the multiple character case should be implemented. Which one would be the preferred way to go? > Allow delimiterfordsv to use multiple-character delimiters > -- > > Key: HIVE-14404 > URL: https://issues.apache.org/jira/browse/HIVE-14404 > Project: Hive > Issue Type: Improvement >Reporter: Stephen Measmer >Assignee: Marta Kuczora > > HIVE-5871 allows for reading multiple character delimiters. Would like the > ability to use outputformat=dsv and define multiple character delimiters. > Today delimiterfordsv only uses on character even if multiple are passes. > For example: > when I use: > beeline>!set outputformat dsv > beeline>!set delimiterfordsv "^-^" > I get: > 111201081253106275^31-Oct-2011 > 00:00:00^Text^201605232823^2016051968232151^201605232823_2016051968232151_0_0_1 > > Would like it to be: > 111201081253106275^-^31-Oct-2011 > 00:00:00^-^Text^-^201605232823^-^2016051968232151^-^201605232823_2016051968232151_0_0_1 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13985: -- Labels: (was: TODOC1.3 TODOC2.1.1 TODOC2.2) > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch, HIVE-13985.5.patch, HIVE-13985.6.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407370#comment-15407370 ] Lefty Leverenz commented on HIVE-13985: --- Thanks for the doc, [~asears]! Well done. I fixed a couple of typos and did a little editing. Here's the link: * [Configuration Properties -- hive.orc.cache.use.soft.references | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.orc.cache.use.soft.references] Removed the three TODOC labels. > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 1.3.0, 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, > HIVE-13985-branch-2.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, > HIVE-13985.3.patch, HIVE-13985.4.patch, HIVE-13985.5.patch, HIVE-13985.6.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[ https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-14424: Fix Version/s: 2.1.1 2.2.0 1.3.0 Status: Patch Available (was: Open) [~szehon] [~amareshwari] [~sershe] [~jcamachorodriguez] Review please. > CLIRestoreTest failing in branch 2.1 > > > Key: HIVE-14424 > URL: https://issues.apache.org/jira/browse/HIVE-14424 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14424.patch > > > {noformat} > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at org.apache.hive.service.cli.CLIService.init(CLIService.java:113) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195) > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836) > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602) > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385) > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812) > ... 25 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375) > ... 26 more > {noformat} > But is caused by HIVE-14221. Code changes are here: > https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc499507b5f31fecc916 > So the issue is that, in this patch, everywhere the clas
[jira] [Commented] (HIVE-14415) Upgrade qtest execution framework to junit4 - TestPerfCliDriver
[ https://issues.apache.org/jira/browse/HIVE-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407408#comment-15407408 ] Peter Vary commented on HIVE-14415: --- [~kgyrtkirk] You were a tad faster, than me on this problem. In this morning I was about to take one of the jira (HIVE-12033) [~spena] left, which is about the intellij debugging issue too. Good to see more of us working on the testing framework. I will try to look at your patch today, or later tomorrow. Although I am not a committer, but one more eyes always help > Upgrade qtest execution framework to junit4 - TestPerfCliDriver > --- > > Key: HIVE-14415 > URL: https://issues.apache.org/jira/browse/HIVE-14415 > Project: Hive > Issue Type: Improvement > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-14415.1.patch > > > I would like to upgrade the current maven+ant+velocimacro+junit4 qtest > generation&execution framework to use only junit4 - while (trying) to keep > all the existing features it provides. > What I can't really do with the current one: execute easily a single qtests > from an IDE (as a matter of fact I can...but it's way too complicated; after > this it won't be a cake-walk either...but it will be a step closer ;) > I think this change will make it more clear how these tests are configured > and executed. > I will do this in two phases, currently i will only change > {{TestPerfCliDriver}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[ https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-14424: Attachment: HIVE-14424.patch > CLIRestoreTest failing in branch 2.1 > > > Key: HIVE-14424 > URL: https://issues.apache.org/jira/browse/HIVE-14424 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14424.patch > > > {noformat} > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at org.apache.hive.service.cli.CLIService.init(CLIService.java:113) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195) > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836) > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602) > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385) > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812) > ... 25 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375) > ... 26 more > {noformat} > But is caused by HIVE-14221. Code changes are here: > https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc499507b5f31fecc916 > So the issue is that, in this patch, everywhere the class > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory > has been mentioned, except at one place. That one place is using
[jira] [Commented] (HIVE-14384) CBO: Decimal constant folding is failing
[ https://issues.apache.org/jira/browse/HIVE-14384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407421#comment-15407421 ] Jesus Camacho Rodriguez commented on HIVE-14384: Error seems to be in Calcite, in the method {{makeExactLiteral(BigDecimal)}} in RexBuilder. Basically, it will infer the incorrect precision when decimal < 1 e.g. for 0.06 it infers the type to be Decimal(1,2) instead of Decimal(3,2). {code:java} /** * Creates a numeric literal. */ public RexLiteral makeExactLiteral(BigDecimal bd) { RelDataType relType; int scale = bd.scale(); long l = bd.unscaledValue().longValue(); assert scale >= 0; assert scale <= typeFactory.getTypeSystem().getMaxNumericScale() : scale; assert BigDecimal.valueOf(l, scale).equals(bd); if (scale == 0) { if ((l >= Integer.MIN_VALUE) && (l <= Integer.MAX_VALUE)) { relType = typeFactory.createSqlType(SqlTypeName.INTEGER); } else { relType = typeFactory.createSqlType(SqlTypeName.BIGINT); } } else { int precision = bd.unscaledValue().abs().toString().length(); relType = typeFactory.createSqlType(SqlTypeName.DECIMAL, precision, scale); } return makeExactLiteral(bd, relType); } {code} > CBO: Decimal constant folding is failing > > > Key: HIVE-14384 > URL: https://issues.apache.org/jira/browse/HIVE-14384 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.2.0 >Reporter: Gopal V > > {code} > explain select sum(l_extendedprice * l_discount) as revenue from lineitem > where l_shipdate >= '1993-01-01' and l_shipdate < '1994-01-01' and l_discount > between 0.06 - 0.01 and 0.06 + 0.01 and l_quantity < 25; > {code} > Fails CBO because of constant folding errors. > {code} > 2016-07-29T17:15:50,921 ERROR [0b3f62eb-8a80-40cd-8cf2-60bc835191a8 main] > parse.CalcitePlanner: CBO failed, skipping CBO. > java.lang.IllegalArgumentException: Decimal scale must be less than or equal > to precision > at > org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils.validateParameter(HiveDecimalUtils.java:53) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo.(DecimalTypeInfo.java:36) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.createPrimitiveTypeInfo(TypeInfoFactory.java:157) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.getPrimitiveTypeInfo(TypeInfoFactory.java:109) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.getDecimalTypeInfo(TypeInfoFactory.java:175) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitLiteral(ExprNodeConverter.java:259) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitLiteral(ExprNodeConverter.java:82) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at org.apache.calcite.rex.RexLiteral.accept(RexLiteral.java:657) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitCall(ExprNodeConverter.java:144) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitCall(ExprNodeConverter.java:82) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at org.apache.calcite.rex.RexCall.accept(RexCall.java:108) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.hadoop.hive.ql.optimizer.calcite.HiveRexExecutorImpl.reduce(HiveRexExecutorImpl.java:58) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveReduceExpressionsRule.reduceExpressionsInternal(HiveReduceExpressionsRule.java:376) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveReduceExpressionsRule.reduceExpressions(HiveReduceExpressionsRule.java:286) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveReduceExpressionsRule$FilterReduceExpressionsRule.onMatch(HiveReduceExpressionsRule.java:141) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:318) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:514) > ~[calcite-core-1.6.0.jar:1.6.0] > at >
[jira] [Assigned] (HIVE-14384) CBO: Decimal constant folding is failing
[ https://issues.apache.org/jira/browse/HIVE-14384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-14384: -- Assignee: Jesus Camacho Rodriguez > CBO: Decimal constant folding is failing > > > Key: HIVE-14384 > URL: https://issues.apache.org/jira/browse/HIVE-14384 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez > > {code} > explain select sum(l_extendedprice * l_discount) as revenue from lineitem > where l_shipdate >= '1993-01-01' and l_shipdate < '1994-01-01' and l_discount > between 0.06 - 0.01 and 0.06 + 0.01 and l_quantity < 25; > {code} > Fails CBO because of constant folding errors. > {code} > 2016-07-29T17:15:50,921 ERROR [0b3f62eb-8a80-40cd-8cf2-60bc835191a8 main] > parse.CalcitePlanner: CBO failed, skipping CBO. > java.lang.IllegalArgumentException: Decimal scale must be less than or equal > to precision > at > org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils.validateParameter(HiveDecimalUtils.java:53) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo.(DecimalTypeInfo.java:36) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.createPrimitiveTypeInfo(TypeInfoFactory.java:157) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.getPrimitiveTypeInfo(TypeInfoFactory.java:109) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.getDecimalTypeInfo(TypeInfoFactory.java:175) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitLiteral(ExprNodeConverter.java:259) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitLiteral(ExprNodeConverter.java:82) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at org.apache.calcite.rex.RexLiteral.accept(RexLiteral.java:657) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitCall(ExprNodeConverter.java:144) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitCall(ExprNodeConverter.java:82) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at org.apache.calcite.rex.RexCall.accept(RexCall.java:108) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.hadoop.hive.ql.optimizer.calcite.HiveRexExecutorImpl.reduce(HiveRexExecutorImpl.java:58) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveReduceExpressionsRule.reduceExpressionsInternal(HiveReduceExpressionsRule.java:376) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveReduceExpressionsRule.reduceExpressions(HiveReduceExpressionsRule.java:286) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveReduceExpressionsRule$FilterReduceExpressionsRule.onMatch(HiveReduceExpressionsRule.java:141) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:318) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:514) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:392) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:285) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:72) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:207) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:194) > ~[calcite-core-1.6.0.jar:1.6.0] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:1320) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1191) > ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT] > at > org.apache.hadoop.hive.ql.parse.Calci
[jira] [Updated] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[ https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14146: -- Attachment: HIVE-14146.10.patch Several more testcase changes: - create_with_constraints.q.out - printing column names in constrains was not indented correctly before - my patch incidentally solves this - alter_table_invalidate_column_stats.q.out, columnstats_part_coltype.q.out - column headers was not indented appropriately - alter_view_as_select_with_partition.q - forget to add to the patch last time -multiline select is indented The other testcases were about printing null instead of empty string. The patch updated to print it correctly. > Column comments with "\n" character "corrupts" table metadata > - > > Key: HIVE-14146 > URL: https://issues.apache.org/jira/browse/HIVE-14146 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14146.10.patch, HIVE-14146.2.patch, > HIVE-14146.3.patch, HIVE-14146.4.patch, HIVE-14146.5.patch, > HIVE-14146.6.patch, HIVE-14146.7.patch, HIVE-14146.8.patch, > HIVE-14146.9.patch, HIVE-14146.patch > > > Create a table with the following(noting the \n in the COMMENT): > {noformat} > CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an > individual’); > {noformat} > Describe shows that now the metadata is messed up: > {noformat} > beeline> describe commtest; > +---++---+--+ > | col_name | data_type |comment| > +---++---+--+ > | first_nm | string | Indicates First name | > | of an individual | NULL | NULL | > +---++---+--+ > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407437#comment-15407437 ] Ke Jia commented on HIVE-13589: --- Hi [~thejas], current patch works with the case specifying '- p' without an argument to make beeline prompt for password. However it only support the case with "- p" following by the specified option which are added in the Beeline.java [L290-L391]. If not, Apache common CLI will consider the option as the value of "- p". We can limit user to put the "- p" by the end of command line. Do you have better suggestions? > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Attachments: HIVE-13589.1.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14425) java.io.IOException: Could not find status of job:job_1469704050741_109320
[ https://issues.apache.org/jira/browse/HIVE-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua reassigned HIVE-14425: -- Assignee: liuguanghua > java.io.IOException: Could not find status of job:job_1469704050741_109320 > -- > > Key: HIVE-14425 > URL: https://issues.apache.org/jira/browse/HIVE-14425 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: hadoop2.7.2 + hive1.2.1 >Reporter: liuguanghua >Assignee: liuguanghua >Priority: Minor > > java.io.IOException: Could not find status of job:job_1469704050741_109320 > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:295) > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549) > at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437) > at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Ended Job = job_1469704050741_109320 with exception > 'java.io.IOException(Could not find status of job:job_1469704050741_109320)' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14425) java.io.IOException: Could not find status of job:job_*
[ https://issues.apache.org/jira/browse/HIVE-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HIVE-14425: --- Summary: java.io.IOException: Could not find status of job:job_* (was: java.io.IOException: Could not find status of job:job_1469704050741_109320) > java.io.IOException: Could not find status of job:job_* > --- > > Key: HIVE-14425 > URL: https://issues.apache.org/jira/browse/HIVE-14425 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: hadoop2.7.2 + hive1.2.1 >Reporter: liuguanghua >Assignee: liuguanghua >Priority: Minor > > java.io.IOException: Could not find status of job:job_1469704050741_109320 > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:295) > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549) > at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437) > at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > Ended Job = job_1469704050741_109320 with exception > 'java.io.IOException(Could not find status of job:job_1469704050741_109320)' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14374) BeeLine argument, and configuration handling cleanup
[ https://issues.apache.org/jira/browse/HIVE-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407458#comment-15407458 ] Peter Vary commented on HIVE-14374: --- Looked up from the code and jira the remaining attributes: - entirelineascommand - required for SchemaTool, but not intended for public usage - maxheight, maxwidth - was not able to find about the intended usage (do we need a command line argument for it, or not) - maxwidth specifically not saved, but loaded, and possible to set by a commandline argument. Maybe [~cwsteinbach], or [~ashutoshc] could tell more. Otherwise I think it would be a good idea to set it from command line. - timeout - the code does not uses it, jira does not help. Maybe [~cwsteinbach], or [~ashutoshc] could tell more. Otherwise I think it would should remove it altogether. - showelapsedtime - the jira does not help, but I think it would be a good idea to set it from command line. - lastconnectedurl - looking through the jira (HIVE-13670), the author ([~sushanth]) was avare of the "using reflection to set command line arguments" feature, but still not put into the documentation. So I think this is not intended as a command line argument. [~sushanth] please comment, if I missunderstand anything. Otherwise we should remove the command line setting possibility - trimscripts - the jira does not help, but I think it would be a good idea to set it from command line. - historyfile - the jira does not help, but I think it would be a good idea to set it from command line. > BeeLine argument, and configuration handling cleanup > > > Key: HIVE-14374 > URL: https://issues.apache.org/jira/browse/HIVE-14374 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > BeeLine uses reflection, to set the BeeLineOpts attributes when parsing > command line arguments, and when loading the configuration file. > This means, that creating a setXXX, getXXX method in BeeLineOpts is a > potential risk of exposing an attribute for the user unintentionally. There > is a possibility to exclude an attribute from saving the value in the > configuration file with the Ignore annotation. This does not restrict the > loading or command line setting of these parameters which means there are > many undocumented "features" as-is, like setting the lastConnectedUrl, > allowMultilineCommand, maxHeight, trimScripts, etc. from command line. > This part of the code needs a little cleanup. > I think we should make this exposure more explicit, and be able to > differentiate the configurable options depending on the source (command line, > and configuration file), so I propose to create a mechanism to tell > explicitly which BeeLineOpts attributes are settable by command line, and > configuration file, and every other attribute should be inaccessible by the > user of the beeline cli. > One possible solution could be two annotations like these: > - CommandLineOption - there could be a mandatory text parameter here, so the > developer had to provide the help text for it which could be displayed to the > user > - ConfigurationFileOption - no text is required here > Something like this: > - This attribute could be provided by command line, and from a configuration > file too: > {noformat} > @CommandLineOption("automatically save preferences") > @ConfigurationFileOption > public void setAutosave(boolean autosave) { > this.autosave = autosave; > } > public void getAutosave() { > return this.autosave; > } > {noformat} > - This attribute could be set through the configuration only > {noformat} > @ConfigurationFileOption > public void setLastConnectedUrl(String lastConnectedUrl) { > this.lastConnectedUrl = lastConnectedUrl; > } > > public String getLastConnectedUrl() > { > return lastConnectedUrl; > } > {noformat} > - Attribute could be set through command line only - I think this is not too > relevant, but possible > {noformat} > @CommandLineOption("specific command line option") > public void setSpecificCommandLineOption(String specificCommandLineOption) { > this.specificCommandLineOption = specificCommandLineOption; > } > > public String getSpecificCommandLineOption() { > return specificCommandLineOption; > } > {noformat} > - Attribute could not be set > {noformat} > public static Env getEnv() { > return env; > } > public static void setEnv(Env envToUse) { > env = envToUse; > } > {noformat} > Accouring to our previous conversations, I think you might be interested in: > [~spena], [~vihangk1], [~aihuaxu], [~ngangam], [~ychena], [~xuefuz] > but anyone is welcome to discuss this. > What do you think about the proposed solution? > Any better ideas, or extensions? >
[jira] [Resolved] (HIVE-14425) java.io.IOException: Could not find status of job:job_*
[ https://issues.apache.org/jira/browse/HIVE-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua resolved HIVE-14425. Resolution: Fixed I finally resolve this problem. I find the error logs from the AMapplicationMaster: 2016-08-04 14:27:59,012 INFO [Thread-68] org.apache.hadoop.service.AbstractService: Service JobHistoryEventHandler failed in state STOPPED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$PathComponentTooLongException): The maximum path component name limit of job_1470047186803_13-1470292057380-ide-create++table+temp.tem...%28%27%E5%B7%B2%E5%8F%96%E6%B6%88%27%2C%27%E6%8B%92%E6%94%B6%E5%85%A5%E5%BA%93%27%2C%27%E9%A9%B3%E5%9B%9E%27%29%28Stage-1470292073175-1-0-SUCCEEDED-root.data_platform-1470292063756.jhist_tmp in directory /tmp/hadoop-yarn/staging/history/done_intermediate/ide is exceeded: limit=255 length=258 at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxComponentLength(FSDirectory.java:911) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addLastINode(FSDirectory.java:976) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addINode(FSDirectory.java:838) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:426) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2575) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2450) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2334) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:623) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) When the am finished ,it produce the two files,one is *.jhist and the other is *.conf. If hql being with Chinese character,or end with Chinese character ,the part of the filename must be url encode. result to more long filename.So the job cannot find in the historyserver. how to slove this? There is a simple way. set hive.jobname.length=10; or smaller (the default value is 50) > java.io.IOException: Could not find status of job:job_* > --- > > Key: HIVE-14425 > URL: https://issues.apache.org/jira/browse/HIVE-14425 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: hadoop2.7.2 + hive1.2.1 >Reporter: liuguanghua >Assignee: liuguanghua >Priority: Minor > > java.io.IOException: Could not find status of job:job_1469704050741_109320 > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:295) > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549) > at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437) > at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) > at sun.reflec
[jira] [Updated] (HIVE-14425) java.io.IOException: Could not find status of job:job_*
[ https://issues.apache.org/jira/browse/HIVE-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HIVE-14425: --- Description: java.io.IOException: Could not find status of job:job_1470047186803_13 at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:295) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Ended Job = job_1470047186803_13 with exception 'java.io.IOException(Could not find status of job:job_1470047186803_13)' was: java.io.IOException: Could not find status of job:job_1469704050741_109320 at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:295) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Ended Job = job_1469704050741_109320 with exception 'java.io.IOException(Could not find status of job:job_1469704050741_109320)' > java.io.IOException: Could not find status of job:job_* > --- > > Key: HIVE-14425 > URL: https://issues.apache.org/jira/browse/HIVE-14425 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: hadoop2.7.2 + hive1.2.1 >Reporter: liuguanghua >Assignee: liuguanghua >Priority: Minor > > java.io.IOException: Could not find status of job:job_1470047186803_13 > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:295) > at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549) > at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437) > at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) > at org.apache.hado
issues@hive.apache.org
[ https://issues.apache.org/jira/browse/HIVE-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua reassigned HIVE-8339: - Assignee: liuguanghua > Job status not found after 100% succeded map&reduce > --- > > Key: HIVE-8339 > URL: https://issues.apache.org/jira/browse/HIVE-8339 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 > Environment: Hadoop 2.4.0, Hive 0.13.1. > Amazon EMR cluster of 9 i2.4xlarge nodes. > 800+GB of data in HDFS. >Reporter: Valera Chevtaev >Assignee: liuguanghua > > According to the logs it seems that the jobs 100% succeed for both map and > reduce but then wasn't able to get the status of the job from job history > server. > Hive logs: > 2014-10-03 07:57:26,593 INFO [main]: exec.Task > (SessionState.java:printInfo(536)) - 2014-10-03 07:57:26,593 Stage-1 map = > 100%, reduce = 99%, Cumulative CPU 872541.02 sec > 2014-10-03 07:57:47,447 INFO [main]: exec.Task > (SessionState.java:printInfo(536)) - 2014-10-03 07:57:47,446 Stage-1 map = > 100%, reduce = 100%, Cumulative CPU 872566.55 sec > 2014-10-03 07:57:48,710 INFO [main]: mapred.ClientServiceDelegate > (ClientServiceDelegate.java:getProxy(273)) - Application state is completed. > FinalApplicationStatus=SUCCEEDED. Redirecting to job history server > 2014-10-03 07:57:48,716 ERROR [main]: exec.Task > (SessionState.java:printError(545)) - Ended Job = job_1412263771568_0002 with > exception 'java.io.IOException(Could not find status of > job:job_1412263771568_0002)' > java.io.IOException: Could not find status of job:job_1412263771568_0002 >at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:294) >at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:547) >at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426) >at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136) >at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) >at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) >at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503) >at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270) >at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088) >at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) >at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) >at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:275) >at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:227) >at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:430) >at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:366) >at > org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:463) >at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:479) >at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:759) >at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:697) >at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:636) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:606) >at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > 2014-10-03 07:57:48,763 ERROR [main]: ql.Driver > (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 > from org.apache.hadoop.hive.ql.exec.mr.MapRedTask -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14426) Extensive logging on info level in WebHCat
[ https://issues.apache.org/jira/browse/HIVE-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-14426: - Assignee: Peter Vary > Extensive logging on info level in WebHCat > -- > > Key: HIVE-14426 > URL: https://issues.apache.org/jira/browse/HIVE-14426 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > > There is an extensive logging in WebHCat at info level, and even some > sensitive information could be logged -- This message was sent by Atlassian JIRA (v6.3.4#6332)
issues@hive.apache.org
[ https://issues.apache.org/jira/browse/HIVE-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407469#comment-15407469 ] liuguanghua commented on HIVE-8339: --- hello,everyone! This problem maybe very easy to slove if we were met the same problem. please see this https://issues.apache.org/jira/browse/HIVE-14425 > Job status not found after 100% succeded map&reduce > --- > > Key: HIVE-8339 > URL: https://issues.apache.org/jira/browse/HIVE-8339 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 > Environment: Hadoop 2.4.0, Hive 0.13.1. > Amazon EMR cluster of 9 i2.4xlarge nodes. > 800+GB of data in HDFS. >Reporter: Valera Chevtaev >Assignee: liuguanghua > > According to the logs it seems that the jobs 100% succeed for both map and > reduce but then wasn't able to get the status of the job from job history > server. > Hive logs: > 2014-10-03 07:57:26,593 INFO [main]: exec.Task > (SessionState.java:printInfo(536)) - 2014-10-03 07:57:26,593 Stage-1 map = > 100%, reduce = 99%, Cumulative CPU 872541.02 sec > 2014-10-03 07:57:47,447 INFO [main]: exec.Task > (SessionState.java:printInfo(536)) - 2014-10-03 07:57:47,446 Stage-1 map = > 100%, reduce = 100%, Cumulative CPU 872566.55 sec > 2014-10-03 07:57:48,710 INFO [main]: mapred.ClientServiceDelegate > (ClientServiceDelegate.java:getProxy(273)) - Application state is completed. > FinalApplicationStatus=SUCCEEDED. Redirecting to job history server > 2014-10-03 07:57:48,716 ERROR [main]: exec.Task > (SessionState.java:printError(545)) - Ended Job = job_1412263771568_0002 with > exception 'java.io.IOException(Could not find status of > job:job_1412263771568_0002)' > java.io.IOException: Could not find status of job:job_1412263771568_0002 >at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:294) >at > org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:547) >at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426) >at > org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136) >at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) >at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) >at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503) >at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270) >at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088) >at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) >at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) >at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:275) >at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:227) >at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:430) >at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:366) >at > org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:463) >at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:479) >at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:759) >at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:697) >at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:636) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:606) >at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > 2014-10-03 07:57:48,763 ERROR [main]: ql.Driver > (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 > from org.apache.hadoop.hive.ql.exec.mr.MapRedTask -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[ https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-14424: Attachment: HIVE-14424.1.patch > CLIRestoreTest failing in branch 2.1 > > > Key: HIVE-14424 > URL: https://issues.apache.org/jira/browse/HIVE-14424 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14424.1.patch, HIVE-14424.patch > > > {noformat} > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at org.apache.hive.service.cli.CLIService.init(CLIService.java:113) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195) > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836) > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602) > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385) > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812) > ... 25 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375) > ... 26 more > {noformat} > But is caused by HIVE-14221. Code changes are here: > https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc499507b5f31fecc916 > So the issue is that, in this patch, everywhere the class > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory > has been mentioned, except at one place. T
[jira] [Commented] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[ https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407477#comment-15407477 ] Amareshwari Sriramadasu commented on HIVE-14424: [~prongs], Seems you need to overwrite the configuration in CLIRestoreTest, instead of changing in conf.xml > CLIRestoreTest failing in branch 2.1 > > > Key: HIVE-14424 > URL: https://issues.apache.org/jira/browse/HIVE-14424 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14424.1.patch, HIVE-14424.patch > > > {noformat} > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at org.apache.hive.service.cli.CLIService.init(CLIService.java:113) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195) > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836) > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602) > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385) > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812) > ... 25 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375) > ... 26 more > {noformat} > But is caused by HIVE-14221. Code changes are here: > https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc499507b5f31fecc916 > So the issue is that, in this patch, everywhere the
[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.1.patch Upload a PoC patch for review and test. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Status: Patch Available (was: Open) > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14426) Extensive logging on info level in WebHCat
[ https://issues.apache.org/jira/browse/HIVE-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14426: -- Attachment: HIVE-14426.patch Move it to trace level > Extensive logging on info level in WebHCat > -- > > Key: HIVE-14426 > URL: https://issues.apache.org/jira/browse/HIVE-14426 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14426.patch > > > There is an extensive logging in WebHCat at info level, and even some > sensitive information could be logged -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14426) Extensive logging on info level in WebHCat
[ https://issues.apache.org/jira/browse/HIVE-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14426: -- Status: Patch Available (was: Open) > Extensive logging on info level in WebHCat > -- > > Key: HIVE-14426 > URL: https://issues.apache.org/jira/browse/HIVE-14426 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14426.patch > > > There is an extensive logging in WebHCat at info level, and even some > sensitive information could be logged -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407497#comment-15407497 ] Rui Li commented on HIVE-14029: --- I think we can resume the work here since Spark 2.0 has released. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12924) CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_ppr_multi_distinct.q failure
[ https://issues.apache.org/jira/browse/HIVE-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407549#comment-15407549 ] Hive QA commented on HIVE-12924: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12786962/HIVE-12924.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10440 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityMultiplePreemptionsSameHost2 org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/761/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/761/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-761/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12786962 - PreCommit-HIVE-MASTER-Build > CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver > groupby_ppr_multi_distinct.q failure > > > Key: HIVE-12924 > URL: https://issues.apache.org/jira/browse/HIVE-12924 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Vineet Garg > Attachments: HIVE-12924.1.patch, HIVE-12924.2.patch, > HIVE-12924.3.patch > > > {code} > EXPLAIN EXTENDED > FROM srcpart src > INSERT OVERWRITE TABLE dest1 > SELECT substr(src.key,1,1), count(DISTINCT substr(src.value,5)), > concat(substr(src.key,1,1),sum(substr(src.value,5))), sum(DISTINCT > substr(src.value, 5)), count(DISTINCT src.value) > WHERE src.ds = '2008-04-08' > GROUP BY substr(src.key,1,1) > {code} > Ended Job = job_local968043618_0742 with errors > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14423) S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned HIVE-14423: --- Assignee: Rajesh Balamohan > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14423) S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-14423: Attachment: HIVE-14423.1.patch > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14423.1.patch > > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14423) S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-14423: Fix Version/s: (was: 2.2.0) > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14423.1.patch > > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14423) S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-14423: Target Version/s: 2.1.0 > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14423.1.patch > > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14423) S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407573#comment-15407573 ] Rajesh Balamohan commented on HIVE-14423: - https://reviews.apache.org/r/50788/ > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14423.1.patch > > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14423) S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-14423: Fix Version/s: 2.2.0 Status: Patch Available (was: Open) > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14423.1.patch > > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[ https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-14424: Attachment: (was: HIVE-14424.1.patch) > CLIRestoreTest failing in branch 2.1 > > > Key: HIVE-14424 > URL: https://issues.apache.org/jira/browse/HIVE-14424 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14424.1.patch, HIVE-14424.patch > > > {noformat} > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at org.apache.hive.service.cli.CLIService.init(CLIService.java:113) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195) > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836) > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602) > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385) > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812) > ... 25 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375) > ... 26 more > {noformat} > But is caused by HIVE-14221. Code changes are here: > https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc499507b5f31fecc916 > So the issue is that, in this patch, everywhere the class > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory > has been mentioned, except at o
[jira] [Updated] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[ https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-14424: Attachment: HIVE-14424.1.patch > CLIRestoreTest failing in branch 2.1 > > > Key: HIVE-14424 > URL: https://issues.apache.org/jira/browse/HIVE-14424 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14424.1.patch, HIVE-14424.patch > > > {noformat} > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at org.apache.hive.service.cli.CLIService.init(CLIService.java:113) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195) > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836) > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602) > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385) > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812) > ... 25 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375) > ... 26 more > {noformat} > But is caused by HIVE-14221. Code changes are here: > https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc499507b5f31fecc916 > So the issue is that, in this patch, everywhere the class > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory > has been mentioned, except at one place. T
[jira] [Assigned] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[ https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklaus Xiao reassigned HIVE-14411: --- Assignee: Niklaus Xiao > selecting Hive on Hbase table may cause FileNotFoundException > - > > Key: HIVE-14411 > URL: https://issues.apache.org/jira/browse/HIVE-14411 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.3.0 >Reporter: Rudd Chen >Assignee: Niklaus Xiao > > 1. create a Hbase table hbase_table > 2. create a external Hive table test_table mapping to the hbase table > example: > create 'hbase_t' > ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'} > create external table hbase_t_hive(key1 string,cf_train string,cf_flight > string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string) > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > with > serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf") > tblproperties("hbase.table.name"="hbase_t"); > create table test3 as select * from hbase_t_hive where 1=2; > > if hive.optimize.null.scan=true, it will return an FileNotFoundException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[ https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklaus Xiao updated HIVE-14411: Attachment: HIVE-14411.1.patch > selecting Hive on Hbase table may cause FileNotFoundException > - > > Key: HIVE-14411 > URL: https://issues.apache.org/jira/browse/HIVE-14411 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.3.0 >Reporter: Rudd Chen >Assignee: Niklaus Xiao > Attachments: HIVE-14411.1.patch > > > 1. create a Hbase table hbase_table > 2. create a external Hive table test_table mapping to the hbase table > example: > create 'hbase_t' > ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'} > create external table hbase_t_hive(key1 string,cf_train string,cf_flight > string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string) > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > with > serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf") > tblproperties("hbase.table.name"="hbase_t"); > create table test3 as select * from hbase_t_hive where 1=2; > > if hive.optimize.null.scan=true, it will return an FileNotFoundException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[ https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklaus Xiao updated HIVE-14411: Target Version/s: 2.2.0 Status: Patch Available (was: Open) > selecting Hive on Hbase table may cause FileNotFoundException > - > > Key: HIVE-14411 > URL: https://issues.apache.org/jira/browse/HIVE-14411 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.3.0 >Reporter: Rudd Chen >Assignee: Niklaus Xiao > Attachments: HIVE-14411.1.patch > > > 1. create a Hbase table hbase_table > 2. create a external Hive table test_table mapping to the hbase table > example: > create 'hbase_t' > ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'} > create external table hbase_t_hive(key1 string,cf_train string,cf_flight > string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string) > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > with > serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf") > tblproperties("hbase.table.name"="hbase_t"); > create table test3 as select * from hbase_t_hive where 1=2; > > if hive.optimize.null.scan=true, it will return an FileNotFoundException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[ https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407591#comment-15407591 ] Niklaus Xiao commented on HIVE-14411: - We should not apply NullScanOptimizer for non-native table. cc [~ashutoshc] for code review. > selecting Hive on Hbase table may cause FileNotFoundException > - > > Key: HIVE-14411 > URL: https://issues.apache.org/jira/browse/HIVE-14411 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.3.0 >Reporter: Rudd Chen >Assignee: Niklaus Xiao > Attachments: HIVE-14411.1.patch > > > 1. create a Hbase table hbase_table > 2. create a external Hive table test_table mapping to the hbase table > example: > create 'hbase_t' > ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'} > create external table hbase_t_hive(key1 string,cf_train string,cf_flight > string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string) > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > with > serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf") > tblproperties("hbase.table.name"="hbase_t"); > create table test3 as select * from hbase_t_hive where 1=2; > > if hive.optimize.null.scan=true, it will return an FileNotFoundException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[ https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklaus Xiao reassigned HIVE-14411: --- Assignee: Ashutosh Chauhan (was: Niklaus Xiao) > selecting Hive on Hbase table may cause FileNotFoundException > - > > Key: HIVE-14411 > URL: https://issues.apache.org/jira/browse/HIVE-14411 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.3.0 >Reporter: Rudd Chen >Assignee: Ashutosh Chauhan > Attachments: HIVE-14411.1.patch > > > 1. create a Hbase table hbase_table > 2. create a external Hive table test_table mapping to the hbase table > example: > create 'hbase_t' > ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'} > create external table hbase_t_hive(key1 string,cf_train string,cf_flight > string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string) > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > with > serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf") > tblproperties("hbase.table.name"="hbase_t"); > create table test3 as select * from hbase_t_hive where 1=2; > > if hive.optimize.null.scan=true, it will return an FileNotFoundException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14304) Beeline command will fail when entireLineAsCommand set to true
[ https://issues.apache.org/jira/browse/HIVE-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niklaus Xiao reassigned HIVE-14304: --- Assignee: Hari Sankar Sivarama Subramaniyan (was: Niklaus Xiao) > Beeline command will fail when entireLineAsCommand set to true > -- > > Key: HIVE-14304 > URL: https://issues.apache.org/jira/browse/HIVE-14304 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.3.0, 2.2.0 >Reporter: Niklaus Xiao >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.2.0 > > Attachments: HIVE-14304.1.patch > > > Use beeline > {code} > beeline --entireLineAsCommand=true > {code} > show tables fail: > {code} > 0: jdbc:hive2://189.39.151.44:21066/> show tables; > Error: Error while compiling statement: FAILED: ParseException line 1:11 > extraneous input ';' expecting EOF near '' (state=42000,code=4) > {code} > We should remove the trailing semi-colon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14128) Parallelize jobClose phases
[ https://issues.apache.org/jira/browse/HIVE-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407617#comment-15407617 ] Rajesh Balamohan commented on HIVE-14128: - Will revise the patch and upload it. Couple of tests failed locally. > Parallelize jobClose phases > --- > > Key: HIVE-14128 > URL: https://issues.apache.org/jira/browse/HIVE-14128 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Rajesh Balamohan > Attachments: HIVE-14128.1.patch, HIVE-14128.master.2.patch, > HIVE-14128.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14404) Allow delimiterfordsv to use multiple-character delimiters
[ https://issues.apache.org/jira/browse/HIVE-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407680#comment-15407680 ] Stephen Measmer commented on HIVE-14404: My preference would be #2 as Super CSV does not look like its an Apache project. Making the change in the context of hive would contribute to supportability. > Allow delimiterfordsv to use multiple-character delimiters > -- > > Key: HIVE-14404 > URL: https://issues.apache.org/jira/browse/HIVE-14404 > Project: Hive > Issue Type: Improvement >Reporter: Stephen Measmer >Assignee: Marta Kuczora > > HIVE-5871 allows for reading multiple character delimiters. Would like the > ability to use outputformat=dsv and define multiple character delimiters. > Today delimiterfordsv only uses on character even if multiple are passes. > For example: > when I use: > beeline>!set outputformat dsv > beeline>!set delimiterfordsv "^-^" > I get: > 111201081253106275^31-Oct-2011 > 00:00:00^Text^201605232823^2016051968232151^201605232823_2016051968232151_0_0_1 > > Would like it to be: > 111201081253106275^-^31-Oct-2011 > 00:00:00^-^Text^-^201605232823^-^2016051968232151^-^201605232823_2016051968232151_0_0_1 > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14204) Optimize loading dynamic partitions
[ https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407722#comment-15407722 ] Hive QA commented on HIVE-14204: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12821961/HIVE-14204.8.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10440 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityMultiplePreemptionsSameHost2 org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/762/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/762/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-762/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12821961 - PreCommit-HIVE-MASTER-Build > Optimize loading dynamic partitions > > > Key: HIVE-14204 > URL: https://issues.apache.org/jira/browse/HIVE-14204 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch, > HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch, HIVE-14204.8.patch > > > Lots of time is spent in sequential fashion to load dynamic partitioned > dataset in driver side. E.g simple dynamic partitioned load as follows takes > 300+ seconds > {noformat} > INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from > tpcds_bin_partitioned_orc_200.web_sales; > Time taken to load dynamic partitions: 309.22 seconds > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407930#comment-15407930 ] Sergio Peña commented on HIVE-14029: Let's wait until HIVE-14240 is resolved. The current spark assembly used for all itests uses spark 1.6 so this patch won't work. Also, I've heard that spark 2.0 won't use spark assembly anymore, so we need to depend on spark maven dependencies to run the tests. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14270) Write temporary data to HDFS when doing inserts on tables located on S3
[ https://issues.apache.org/jira/browse/HIVE-14270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14270: --- Attachment: HIVE-14270.4.patch Attaching new patch to run another set of tests. [~ashutoshc] I removed the duplication of the rename(), and create HDFS scratch directories instead. It is simpler than the other code, and less prone to errors. Could you help me reviewing it? > Write temporary data to HDFS when doing inserts on tables located on S3 > --- > > Key: HIVE-14270 > URL: https://issues.apache.org/jira/browse/HIVE-14270 > Project: Hive > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14270.1.patch, HIVE-14270.2.patch, > HIVE-14270.3.patch, HIVE-14270.4.patch > > > Currently, when doing INSERT statements on tables located at S3, Hive writes > and reads temporary (or intermediate) files to S3 as well. > If HDFS is still the default filesystem on Hive, then we can keep such > temporary files on HDFS to keep things run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14421) FS.deleteOnExit holds references to _tmp_space.db files
[ https://issues.apache.org/jira/browse/HIVE-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408111#comment-15408111 ] Hive QA commented on HIVE-14421: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12821966/HIVE-14421.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10425 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-insert_values_non_partitioned.q-update_after_multiple_inserts.q-tez_union_dynamic_partition.q-and-12-more - did not produce a TEST-*.xml file TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityMultiplePreemptionsSameHost2 org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/763/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/763/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-763/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12821966 - PreCommit-HIVE-MASTER-Build > FS.deleteOnExit holds references to _tmp_space.db files > --- > > Key: HIVE-14421 > URL: https://issues.apache.org/jira/browse/HIVE-14421 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14421.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[ https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408146#comment-15408146 ] Pengcheng Xiong commented on HIVE-14424: [~prongs], thanks for your attention. Our intention is to make tests exercise the security code path to prevent bugs. Your change looks good. +1. Could you please submit it for a QA run? Thanks. > CLIRestoreTest failing in branch 2.1 > > > Key: HIVE-14424 > URL: https://issues.apache.org/jira/browse/HIVE-14424 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14424.1.patch, HIVE-14424.patch > > > {noformat} > java.lang.RuntimeException: Error applying authorization policy on hive > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at org.apache.hive.service.cli.CLIService.init(CLIService.java:113) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48) > at > org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195) > at > org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runner.JUnitCore.run(JUnitCore.java:160) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836) > at > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602) > at > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126) > at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) > ... 22 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385) > at > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812) > ... 25 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375) > ... 26 more > {noformat} > But is caused by HIVE-14221. Code changes are here: > https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408166#comment-15408166 ] Thejas M Nair commented on HIVE-14129: -- [~ashutoshc] It doesn't look like the 2nd issue is fixed. ie - bq. 2. SessionState is examined by acid code in MoveTask, if it is in a different thread that thread will not have the SessionState object available. TaskRunner would need an additional method - setSessionState, similar to setOperationLog, that sets the thread local sessionstate. Also, we should check if the methods being used in MoveTask on SessionState for acid purposes are thread safe. Looking at the method names, I would expect it to be threadsafe. cc [~ekoifman] Without that we might have issues with ACID and execparallel=true. > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14413: --- Status: Open (was: Patch Available) > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14413.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14413 started by Jesus Camacho Rodriguez. -- > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14413.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14413: --- Status: Patch Available (was: In Progress) > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14413.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14413: --- Attachment: HIVE-14413.02.patch > Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and > extract more deterministic pieces out > > > Key: HIVE-14413 > URL: https://issues.apache.org/jira/browse/HIVE-14413 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14413.01.patch, HIVE-14413.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14429) HadoopMetrics2Reporter leaks memory if the metrics sink is not configured correctly
[ https://issues.apache.org/jira/browse/HIVE-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-14429. --- Resolution: Duplicate HIVE-14228 > HadoopMetrics2Reporter leaks memory if the metrics sink is not configured > correctly > --- > > Key: HIVE-14429 > URL: https://issues.apache.org/jira/browse/HIVE-14429 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Siddharth Seth >Priority: Critical > > About 80MB held after 7 hours of running. Metrics2Collector aggregates only > when it's invoked by the Hadoop sink. > Options - the first one is better IMO. > 1. Fix Metrics2Collector to aggregate more often, and fix the dependency in > Hive accordingly > 2. Don't setup the metrics sub-system if a sink is not configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14429) HadoopMetrics2Reporter leaks memory if the metrics sink is not configured correctly
[ https://issues.apache.org/jira/browse/HIVE-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14429: -- Issue Type: Bug (was: Sub-task) Parent: (was: HIVE-13176) > HadoopMetrics2Reporter leaks memory if the metrics sink is not configured > correctly > --- > > Key: HIVE-14429 > URL: https://issues.apache.org/jira/browse/HIVE-14429 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Siddharth Seth >Priority: Critical > > About 80MB held after 7 hours of running. Metrics2Collector aggregates only > when it's invoked by the Hadoop sink. > Options - the first one is better IMO. > 1. Fix Metrics2Collector to aggregate more often, and fix the dependency in > Hive accordingly > 2. Don't setup the metrics sub-system if a sink is not configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14430) More instances of HiveConf and the associated UDFClassLoader than expected
[ https://issues.apache.org/jira/browse/HIVE-14430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408247#comment-15408247 ] Sergey Shelukhin commented on HIVE-14430: - How many HS2 threadpool threads are there? all the metastore stuff is threadlocal > More instances of HiveConf and the associated UDFClassLoader than expected > -- > > Key: HIVE-14430 > URL: https://issues.apache.org/jira/browse/HIVE-14430 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Siddharth Seth >Priority: Critical > > 841 instances of HiveConf. > 831 instances of UDFClassLoader > This is on a HS2 instance configured to run 10 concurrent queries with LLAP. > 10 SessionState instances. Something is holding on to the additional > HiveConf, UDFClassLoaders - potentially HMSHandler. > This is with an embedded metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14423) S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
[ https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408281#comment-15408281 ] Chris Nauroth commented on HIVE-14423: -- Hello [~rajesh.balamohan]. Thank you for the patch. This looks really valuable for S3A, WASB and other file systems backed by blob stores, but I have a question about whether it will change load patterns and performance characteristics when running on HDFS. For HDFS, {{getContentSummary}} is a single RPC to the NameNode. It's possibly the most expensive NameNode RPC, at least among the read APIs, because the NameNode needs to hold a lock while traversing the entire inode sub-tree. However, it does have the benefit of getting all of the calculation done for a single path/partition in a single network call, so overall, this Hive algorithm is O(N) where N = # partitions. With this patch, it starts using {{FileSystem#listFiles}} with the recursive option, which turns into multiple {{getListing}} NameNode RPCs, one for each sub-directory. The {{getListing}} RPC is less expensive for the NameNode to execute compared to {{getContentSummary}}, but overall this algorithm requires many more network round-trips: O(N * M) where N = # partitions and M = average # directories per partition. At this point in the Hive code, is it possible that the partitions refer to directories in the file system that are multiple levels deep with nested sub-directories? I suspect the answer is yes, because the existing code used {{getContentSummary}}, and your patch used the recursive option for {{listFiles}}. Do you think an alternative approach would be to override {{getContentSummary}} in {{S3AFileSystem}} and optimize it? That might look similar to other optimizations that are making use of S3 bulk listings, such as HADOOP-13208 and HADOOP-13371. Parallelizing the calls for all partitions looks valuable regardless of which approach we take. Cc [~ste...@apache.org] FYI for when he returns. > S3: Fetching partition sizes from FS can be expensive when stats are not > available in metastore > > > Key: HIVE-14423 > URL: https://issues.apache.org/jira/browse/HIVE-14423 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14423.1.patch > > > When partition stats are not available in metastore, it tries to get the file > sizes from FS. > e.g > {noformat} > at > org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144) > at > org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132) > at > org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > {noformat} > This can be quite expensive in some FS like S3. Especially when table is > partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can > spend 1000s of seconds just waiting for these information to be pulled in. > Also, it would be good to remove FS.getContentSummary usage to find out file > sizes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14204) Optimize loading dynamic partitions
[ https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408303#comment-15408303 ] Ashutosh Chauhan commented on HIVE-14204: - Seems like method getPartitionsForPath() is not used anywhere. We can get rid of it. Other than that +1 > Optimize loading dynamic partitions > > > Key: HIVE-14204 > URL: https://issues.apache.org/jira/browse/HIVE-14204 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch, > HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch, HIVE-14204.8.patch > > > Lots of time is spent in sequential fashion to load dynamic partitioned > dataset in driver side. E.g simple dynamic partitioned load as follows takes > 300+ seconds > {noformat} > INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from > tpcds_bin_partitioned_orc_200.web_sales; > Time taken to load dynamic partitions: 309.22 seconds > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14431) Recognize COALESCE as CASE and extend CASE simplification to cover more cases
[ https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14431 started by Jesus Camacho Rodriguez. -- > Recognize COALESCE as CASE and extend CASE simplification to cover more cases > - > > Key: HIVE-14431 > URL: https://issues.apache.org/jira/browse/HIVE-14431 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Transform: > {code} > (COALESCE(a, '') = '') OR >(a = 'A' AND b = c) OR >(a = 'B' AND b = d) OR >(a = 'C' AND b = e) OR >(a = 'D' AND b = f) OR >(a = 'E' AND b = g) OR >(a = 'F' AND b = h) > {code} > into: > {code} > (a='') OR >(a is null) OR >(a = 'A' AND b = c) OR >(a = 'B' AND b = d) OR >(a = 'C' AND b = e) OR >(a = 'D' AND b = f) OR >(a = 'E' AND b = g) OR >(a = 'F' AND b = h) > {code} > With complex queries, this will lead us to factor more predicates that could > be pushed to the TS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14431) Recognize COALESCE as CASE and extend CASE simplification to cover more cases
[ https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14431: --- Status: Patch Available (was: In Progress) > Recognize COALESCE as CASE and extend CASE simplification to cover more cases > - > > Key: HIVE-14431 > URL: https://issues.apache.org/jira/browse/HIVE-14431 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Transform: > {code} > (COALESCE(a, '') = '') OR >(a = 'A' AND b = c) OR >(a = 'B' AND b = d) OR >(a = 'C' AND b = e) OR >(a = 'D' AND b = f) OR >(a = 'E' AND b = g) OR >(a = 'F' AND b = h) > {code} > into: > {code} > (a='') OR >(a is null) OR >(a = 'A' AND b = c) OR >(a = 'B' AND b = d) OR >(a = 'C' AND b = e) OR >(a = 'D' AND b = f) OR >(a = 'E' AND b = g) OR >(a = 'F' AND b = h) > {code} > With complex queries, this will lead us to factor more predicates that could > be pushed to the TS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14431) Recognize COALESCE as CASE and extend CASE simplification to cover more cases
[ https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14431: --- Attachment: HIVE-14431.patch > Recognize COALESCE as CASE and extend CASE simplification to cover more cases > - > > Key: HIVE-14431 > URL: https://issues.apache.org/jira/browse/HIVE-14431 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14431.patch > > > Transform: > {code} > (COALESCE(a, '') = '') OR >(a = 'A' AND b = c) OR >(a = 'B' AND b = d) OR >(a = 'C' AND b = e) OR >(a = 'D' AND b = f) OR >(a = 'E' AND b = g) OR >(a = 'F' AND b = h) > {code} > into: > {code} > (a='') OR >(a is null) OR >(a = 'A' AND b = c) OR >(a = 'B' AND b = d) OR >(a = 'C' AND b = e) OR >(a = 'D' AND b = f) OR >(a = 'E' AND b = g) OR >(a = 'F' AND b = h) > {code} > With complex queries, this will lead us to factor more predicates that could > be pushed to the TS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14432) LLAP signing unit test may be timing-dependent
[ https://issues.apache.org/jira/browse/HIVE-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14432: Attachment: HIVE-14432.patch [~prasanth_j] [~jdere] can you take a look? > LLAP signing unit test may be timing-dependent > -- > > Key: HIVE-14432 > URL: https://issues.apache.org/jira/browse/HIVE-14432 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14432.patch > > > Seems like it's possible for slow background thread to roll the key after we > have signed with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14432) LLAP signing unit test may be timing-dependent
[ https://issues.apache.org/jira/browse/HIVE-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14432: Component/s: Test > LLAP signing unit test may be timing-dependent > -- > > Key: HIVE-14432 > URL: https://issues.apache.org/jira/browse/HIVE-14432 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14432.patch > > > Seems like it's possible for slow background thread to roll the key after we > have signed with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408426#comment-15408426 ] Eugene Koifman commented on HIVE-14129: --- [~thejas], which patch should I look at - here we just toggle 1 line > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Tolpeko updated HIVE-14382: -- Component/s: hpl/sql > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408440#comment-15408440 ] Dmitry Tolpeko commented on HIVE-14382: --- Reviewed and updated docs. > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14411) selecting Hive on Hbase table may cause FileNotFoundException
[ https://issues.apache.org/jira/browse/HIVE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408451#comment-15408451 ] Hive QA commented on HIVE-14411: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822041/HIVE-14411.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10425 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-vectorization_16.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_acid3.q-and-12-more - did not produce a TEST-*.xml file TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/772/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/772/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-772/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12822041 - PreCommit-HIVE-MASTER-Build > selecting Hive on Hbase table may cause FileNotFoundException > - > > Key: HIVE-14411 > URL: https://issues.apache.org/jira/browse/HIVE-14411 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.3.0 >Reporter: Rudd Chen >Assignee: Ashutosh Chauhan > Attachments: HIVE-14411.1.patch > > > 1. create a Hbase table hbase_table > 2. create a external Hive table test_table mapping to the hbase table > example: > create 'hbase_t' > ,{NAME=>'cf',COMPRESSION=>'snappy'},{NUMREGIONS=>15,SPLITALGO=>'HexStringSplit'} > create external table hbase_t_hive(key1 string,cf_train string,cf_flight > string,cf_wbsw string,cf_wbxw string,cf_bgrz string,cf_bgtf string) > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > with > serdeproperties("hbase.columns.mapping"=":key,cf:train,cf:flight,cf:wbsw,cf:wbxw,cf:bgrz,cf:bgtf") > tblproperties("hbase.table.name"="hbase_t"); > create table test3 as select * from hbase_t_hive where 1=2; > > if hive.optimize.null.scan=true, it will return an FileNotFoundException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14421) FS.deleteOnExit holds references to _tmp_space.db files
[ https://issues.apache.org/jira/browse/HIVE-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408459#comment-15408459 ] Gunther Hagleitner commented on HIVE-14421: --- +1 Test run looks ok to me. > FS.deleteOnExit holds references to _tmp_space.db files > --- > > Key: HIVE-14421 > URL: https://issues.apache.org/jira/browse/HIVE-14421 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14421.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Tolpeko updated HIVE-14382: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14382) Improve the Functionality of Reverse FOR Statement
[ https://issues.apache.org/jira/browse/HIVE-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408460#comment-15408460 ] Dmitry Tolpeko commented on HIVE-14382: --- Patch committed. > Improve the Functionality of Reverse FOR Statement > --- > > Key: HIVE-14382 > URL: https://issues.apache.org/jira/browse/HIVE-14382 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Akash Sethi >Assignee: Akash Sethi >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14382.1-branch-2.1.patch, HIVE-14382.1.patch > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > According to SQL Standards, Reverse FOR Statement should be like this:- > FOR index IN Optional[Reverse] Lower_Bound Upper_Bound > but in hive it is like this :- > FOR index IN Optional[Reverse] Upper_Bound Lower_Bound > so i m just trying to improve the functionality for Reverse FOR Statement > REFERNCES :- > https://docs.oracle.com/cloud/latest/db112/LNPLS/for_loop_statement.htm#LNPLS1536 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408468#comment-15408468 ] Naveen Gangam commented on HIVE-14342: -- After quite a bit of testing, I found that the most reliable means to detect if beeline is being run via a remote shell is to check if the for the process is a pipe. So I have added an additional check to the condition before using jline's UnsupportedTerminal. The use of UnsupportedTerminal should be limited to beeline processes running locally as a background process. The output now is as we expected prior to this change. {code} ssh -l root "sudo -u hive beeline -u jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2 -f /tmp/run.sql" root@'s password: 16/08/04 13:46:35 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. key,name,value 1,TRUE,1 $ {code} > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14342: - Attachment: HIVE-14342.patch > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14342: - Status: Patch Available (was: Open) > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[ https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14204: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Rajesh! > Optimize loading dynamic partitions > > > Key: HIVE-14204 > URL: https://issues.apache.org/jira/browse/HIVE-14204 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch, > HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch, HIVE-14204.8.patch > > > Lots of time is spent in sequential fashion to load dynamic partitioned > dataset in driver side. E.g simple dynamic partitioned load as follows takes > 300+ seconds > {noformat} > INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from > tpcds_bin_partitioned_orc_200.web_sales; > Time taken to load dynamic partitions: 309.22 seconds > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
[ https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14204: Component/s: Metastore > Optimize loading dynamic partitions > > > Key: HIVE-14204 > URL: https://issues.apache.org/jira/browse/HIVE-14204 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch, > HIVE-14204.4.patch, HIVE-14204.6.patch, HIVE-14204.7.patch, HIVE-14204.8.patch > > > Lots of time is spent in sequential fashion to load dynamic partitioned > dataset in driver side. E.g simple dynamic partitioned load as follows takes > 300+ seconds > {noformat} > INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from > tpcds_bin_partitioned_orc_200.web_sales; > Time taken to load dynamic partitions: 309.22 seconds > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14433) refactor LLAP plan cache avoidance and fix issue in merge
[ https://issues.apache.org/jira/browse/HIVE-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408494#comment-15408494 ] Sergey Shelukhin commented on HIVE-14433: - [~prasanth_j] fyi > refactor LLAP plan cache avoidance and fix issue in merge > - > > Key: HIVE-14433 > URL: https://issues.apache.org/jira/browse/HIVE-14433 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Map and reduce processors do this: > {noformat} > if (LlapProxy.isDaemon()) { > cache = new org.apache.hadoop.hive.ql.exec.mr.ObjectCache(); // do not > cache plan > ... > {noformat} > but merge processor just gets the plan. If it runs in LLAP, it can get a > cached plan. Need to move this logic into ObjectCache itself, via a isPlan > arg or something. That will also fix this issue for merge processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14433) refactor LLAP plan cache avoidance and fix issue in merge
[ https://issues.apache.org/jira/browse/HIVE-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408493#comment-15408493 ] Sergey Shelukhin commented on HIVE-14433: - [~prasanth_j] fyi > refactor LLAP plan cache avoidance and fix issue in merge > - > > Key: HIVE-14433 > URL: https://issues.apache.org/jira/browse/HIVE-14433 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Map and reduce processors do this: > {noformat} > if (LlapProxy.isDaemon()) { > cache = new org.apache.hadoop.hive.ql.exec.mr.ObjectCache(); // do not > cache plan > ... > {noformat} > but merge processor just gets the plan. If it runs in LLAP, it can get a > cached plan. Need to move this logic into ObjectCache itself, via a isPlan > arg or something. That will also fix this issue for merge processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14415) Upgrade qtest execution framework to junit4 - TestPerfCliDriver
[ https://issues.apache.org/jira/browse/HIVE-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14415: Attachment: HIVE-14415.2.patch > Upgrade qtest execution framework to junit4 - TestPerfCliDriver > --- > > Key: HIVE-14415 > URL: https://issues.apache.org/jira/browse/HIVE-14415 > Project: Hive > Issue Type: Improvement > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-14415.1.patch, HIVE-14415.2.patch > > > I would like to upgrade the current maven+ant+velocimacro+junit4 qtest > generation&execution framework to use only junit4 - while (trying) to keep > all the existing features it provides. > What I can't really do with the current one: execute easily a single qtests > from an IDE (as a matter of fact I can...but it's way too complicated; after > this it won't be a cake-walk either...but it will be a step closer ;) > I think this change will make it more clear how these tests are configured > and executed. > I will do this in two phases, currently i will only change > {{TestPerfCliDriver}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408527#comment-15408527 ] Sergey Shelukhin commented on HIVE-14035: - Some CR feedback on RB, mostly minor. Someone more familiar with ACID should review the core logic, ideally. It makes sense to me. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-14433) refactor LLAP plan cache avoidance and fix issue in merge
[ https://issues.apache.org/jira/browse/HIVE-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14433: Comment: was deleted (was: [~prasanth_j] fyi) > refactor LLAP plan cache avoidance and fix issue in merge > - > > Key: HIVE-14433 > URL: https://issues.apache.org/jira/browse/HIVE-14433 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Map and reduce processors do this: > {noformat} > if (LlapProxy.isDaemon()) { > cache = new org.apache.hadoop.hive.ql.exec.mr.ObjectCache(); // do not > cache plan > ... > {noformat} > but merge processor just gets the plan. If it runs in LLAP, it can get a > cached plan. Need to move this logic into ObjectCache itself, via a isPlan > arg or something. That will also fix this issue for merge processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408536#comment-15408536 ] Sergey Shelukhin commented on HIVE-14035: - Also, I think q file tests are needed... > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14233) Improve vectorization for ACID by eliminating row-by-row stitching
[ https://issues.apache.org/jira/browse/HIVE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408541#comment-15408541 ] Sergey Shelukhin commented on HIVE-14233: - Is it possible to post an RB? An update to RB (or the initial post if done via rb tool, I assume) allows one to have a base patch (HIVE-14035 patch in this case, I assume)... > Improve vectorization for ACID by eliminating row-by-row stitching > -- > > Key: HIVE-14233 > URL: https://issues.apache.org/jira/browse/HIVE-14233 > Project: Hive > Issue Type: New Feature > Components: Transactions, Vectorization >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14233.01.patch, HIVE-14233.02.patch, > HIVE-14233.03.patch, HIVE-14233.04.patch, HIVE-14233.05.patch > > > This JIRA proposes to improve vectorization for ACID by eliminating > row-by-row stitching when reading back ACID files. In the current > implementation, a vectorized row batch is created by populating the batch one > row at a time, before the vectorized batch is passed up along the operator > pipeline. This row-by-row stitching limitation was because of the fact that > the ACID insert/update/delete events from various delta files needed to be > merged together before the actual version of a given row was found out. > HIVE-14035 has enabled us to break away from that limitation by splitting > ACID update events into a combination of delete+insert. In fact, it has now > enabled us to create splits on delta files. > Building on top of HIVE-14035, this JIRA proposes to solve this earlier > bottleneck in the vectorized code path for ACID by now directly reading row > batches from the underlying ORC files and avoiding any stitching altogether. > Once a row batch is read from the split (which may be on a base/delta file), > the deleted rows will be found by cross-referencing them against a data > structure that will just keep track of deleted events (found in the > deleted_delta files). This will lead to a large performance gain when reading > ACID files in vectorized fashion, while enabling further optimizations in > future that can be done on top of that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14433) refactor LLAP plan cache avoidance and fix issue in merge processor
[ https://issues.apache.org/jira/browse/HIVE-14433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14433: Summary: refactor LLAP plan cache avoidance and fix issue in merge processor (was: refactor LLAP plan cache avoidance and fix issue in merge) > refactor LLAP plan cache avoidance and fix issue in merge processor > --- > > Key: HIVE-14433 > URL: https://issues.apache.org/jira/browse/HIVE-14433 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > Map and reduce processors do this: > {noformat} > if (LlapProxy.isDaemon()) { > cache = new org.apache.hadoop.hive.ql.exec.mr.ObjectCache(); // do not > cache plan > ... > {noformat} > but merge processor just gets the plan. If it runs in LLAP, it can get a > cached plan. Need to move this logic into ObjectCache itself, via a isPlan > arg or something. That will also fix this issue for merge processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408584#comment-15408584 ] Sergey Shelukhin commented on HIVE-14035: - Actually, one good thing to do would be to enable it by default in the next patch and see what HiveQA run, with all the existing ACID tests running in new mode, shows. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14402) Vectorization: Fix Mapjoin overflow deserialization
[ https://issues.apache.org/jira/browse/HIVE-14402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14402: --- Status: Patch Available (was: Open) > Vectorization: Fix Mapjoin overflow deserialization > > > Key: HIVE-14402 > URL: https://issues.apache.org/jira/browse/HIVE-14402 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0, 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14402.1.patch, HIVE-14402.2.patch > > > This is in a codepath currently disabled in master, however enabling it > triggers OOB. > {code} > Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024 > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setRef(BytesColumnVector.java:92) > at > org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserializeRowColumn(VectorDeserializeRow.java:415) > at > org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:674) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultLargeMultiValue(VectorMapJoinGenerateResultOperator.java:307) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:226) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultRepeatedAll(VectorMapJoinGenerateResultOperator.java:391) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14402) Vectorization: Fix Mapjoin overflow deserialization
[ https://issues.apache.org/jira/browse/HIVE-14402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14402: --- Attachment: HIVE-14402.2.patch > Vectorization: Fix Mapjoin overflow deserialization > > > Key: HIVE-14402 > URL: https://issues.apache.org/jira/browse/HIVE-14402 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0, 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14402.1.patch, HIVE-14402.2.patch > > > This is in a codepath currently disabled in master, however enabling it > triggers OOB. > {code} > Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024 > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setRef(BytesColumnVector.java:92) > at > org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserializeRowColumn(VectorDeserializeRow.java:415) > at > org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:674) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultLargeMultiValue(VectorMapJoinGenerateResultOperator.java:307) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:226) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultRepeatedAll(VectorMapJoinGenerateResultOperator.java:391) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14402) Vectorization: Fix Mapjoin overflow deserialization
[ https://issues.apache.org/jira/browse/HIVE-14402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14402: --- Status: Open (was: Patch Available) > Vectorization: Fix Mapjoin overflow deserialization > > > Key: HIVE-14402 > URL: https://issues.apache.org/jira/browse/HIVE-14402 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0, 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14402.1.patch, HIVE-14402.2.patch > > > This is in a codepath currently disabled in master, however enabling it > triggers OOB. > {code} > Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024 > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setRef(BytesColumnVector.java:92) > at > org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserializeRowColumn(VectorDeserializeRow.java:415) > at > org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorDeserializeRow.java:674) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultLargeMultiValue(VectorMapJoinGenerateResultOperator.java:307) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:226) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultRepeatedAll(VectorMapJoinGenerateResultOperator.java:391) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14434) Vectorization: BytesBytes lookup capped count can be =0, =1, >=2
[ https://issues.apache.org/jira/browse/HIVE-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14434: --- Attachment: HIVE-14434.1.patch > Vectorization: BytesBytes lookup capped count can be =0, =1, >=2 > > > Key: HIVE-14434 > URL: https://issues.apache.org/jira/browse/HIVE-14434 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14434.1.patch > > > BytesBytesHashmap does not implement deep counters for the depth of a probe - > but however it does distinguish between 0, 1 and > 1 rows. > This information can be used in the vectorized hash join to avoid copying the > probe side keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14434) Vectorization: BytesBytes lookup capped count can be =0, =1, >=2
[ https://issues.apache.org/jira/browse/HIVE-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408596#comment-15408596 ] Sergey Shelukhin commented on HIVE-14434: - "2" value needs to be explained in a comment... otherwise +1 > Vectorization: BytesBytes lookup capped count can be =0, =1, >=2 > > > Key: HIVE-14434 > URL: https://issues.apache.org/jira/browse/HIVE-14434 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14434.1.patch > > > BytesBytesHashmap does not implement deep counters for the depth of a probe - > but however it does distinguish between 0, 1 and > 1 rows. > This information can be used in the vectorized hash join to avoid copying the > probe side keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14270) Write temporary data to HDFS when doing inserts on tables located on S3
[ https://issues.apache.org/jira/browse/HIVE-14270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408615#comment-15408615 ] Hive QA commented on HIVE-14270: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822110/HIVE-14270.4.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 72 failed/errored test(s), 10444 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_windowing_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_multi_insert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pointlookup3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unionDistinct_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_stats org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestSpark
[jira] [Commented] (HIVE-14378) Data size may be estimated as 0 if no columns are being projected after an operator
[ https://issues.apache.org/jira/browse/HIVE-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408616#comment-15408616 ] Hive QA commented on HIVE-14378: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12821934/HIVE-14378.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/774/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/774/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-774/ Messages: {noformat} This message was trimmed, see log for full details [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ spark-client --- [INFO] Compiling 5 source files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes [INFO] [INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client --- [INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar [INFO] Copying guava-14.0.1.jar to /data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-client --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.pom [INFO] [INFO] [INFO] Building Hive Query Language 2.2.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec --- [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-exec --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen Generating vector expression code Generating vector expression test code [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec --- [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java added. [INFO] [INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec --- [INFO] ANTLR: Processing source directory /data/hive-ptest/working/apache-github-source-source/ql/src/java ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/ql/parse/HiveLexer.g org/apache/hadoop/hive/ql/parse/HiveParser.g [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filte
[jira] [Commented] (HIVE-14413) Extend HivePreFilteringRule to traverse inside elements of DNF/CNF and extract more deterministic pieces out
[ https://issues.apache.org/jira/browse/HIVE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408620#comment-15408620 ] Hive QA commented on HIVE-14413: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822120/HIVE-14413.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/775/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/775/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-775/ Messages: {noformat} This message was trimmed, see log for full details [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ spark-client --- [INFO] Compiling 5 source files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes [INFO] [INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client --- [INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar [INFO] Copying guava-14.0.1.jar to /data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-client --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.pom [INFO] [INFO] [INFO] Building Hive Query Language 2.2.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec --- [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-exec --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen Generating vector expression code Generating vector expression test code [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec --- [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java added. [INFO] [INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec --- [INFO] ANTLR: Processing source directory /data/hive-ptest/working/apache-github-source-source/ql/src/java ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/ql/parse/HiveLexer.g org/apache/hadoop/hive/ql/parse/HiveParser.g [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filt
[jira] [Commented] (HIVE-14431) Recognize COALESCE as CASE and extend CASE simplification to cover more cases
[ https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408623#comment-15408623 ] Hive QA commented on HIVE-14431: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822143/HIVE-14431.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/776/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/776/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-776/ Messages: {noformat} This message was trimmed, see log for full details [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ spark-client --- [INFO] Compiling 5 source files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes [INFO] [INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client --- [INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar [INFO] Copying guava-14.0.1.jar to /data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-client --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.pom [INFO] [INFO] [INFO] Building Hive Query Language 2.2.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec --- [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-exec --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen Generating vector expression code Generating vector expression test code [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec --- [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java added. [INFO] [INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec --- [INFO] ANTLR: Processing source directory /data/hive-ptest/working/apache-github-source-source/ql/src/java ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/ql/parse/HiveLexer.g org/apache/hadoop/hive/ql/parse/HiveParser.g [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filtere
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408628#comment-15408628 ] Hive QA commented on HIVE-14342: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822163/HIVE-14342.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/777/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/777/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-777/ Messages: {noformat} This message was trimmed, see log for full details [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ spark-client --- [INFO] Compiling 5 source files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes [INFO] [INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client --- [INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar [INFO] Copying guava-14.0.1.jar to /data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-client --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.pom [INFO] [INFO] [INFO] Building Hive Query Language 2.2.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec --- [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-exec --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen Generating vector expression code Generating vector expression test code [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec --- [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java added. [INFO] [INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec --- [INFO] ANTLR: Processing source directory /data/hive-ptest/working/apache-github-source-source/ql/src/java ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/ql/parse/HiveLexer.g org/apache/hadoop/hive/ql/parse/HiveParser.g [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filtere
[jira] [Updated] (HIVE-14434) Vectorization: BytesBytes lookup capped count can be =0, =1, >=2
[ https://issues.apache.org/jira/browse/HIVE-14434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14434: --- Status: Patch Available (was: Open) > Vectorization: BytesBytes lookup capped count can be =0, =1, >=2 > > > Key: HIVE-14434 > URL: https://issues.apache.org/jira/browse/HIVE-14434 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-14434.1.patch > > > BytesBytesHashmap does not implement deep counters for the depth of a probe - > but however it does distinguish between 0, 1 and > 1 rows. > This information can be used in the vectorized hash join to avoid copying the > probe side keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14342: - Status: Open (was: Patch Available) Pre-commit build failed because of build failure due to missing file from HIVE-14204. Retrying > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14342: - Attachment: HIVE-14342.patch > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14342: - Status: Patch Available (was: Open) > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14415) Upgrade qtest execution framework to junit4 - TestPerfCliDriver
[ https://issues.apache.org/jira/browse/HIVE-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408651#comment-15408651 ] Zoltan Haindrich commented on HIVE-14415: - [~pvary] thank you for the review! i was not aware of HIVE-12033... after these changes - the execution of these tests in ide won't be readily available but will be a step closer...i've seen a few things which will have to be solved prior to getting a working solution...like: * qtests sometime contain some exact pathreference to datafiles...somewhere they are using properties...i'm not sure those will work or not (in case of -Pitests) * there a few maven repo references here and there - not sure how this can be avoided * there is a "poisoning" core-site.xml in the common module...but generally the ide shows all the resources to the test...which may get confused by the union of all hive main&test resources these are just what i know of...if we team up, we can get a fully working ide faster ;) > Upgrade qtest execution framework to junit4 - TestPerfCliDriver > --- > > Key: HIVE-14415 > URL: https://issues.apache.org/jira/browse/HIVE-14415 > Project: Hive > Issue Type: Improvement > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-14415.1.patch, HIVE-14415.2.patch > > > I would like to upgrade the current maven+ant+velocimacro+junit4 qtest > generation&execution framework to use only junit4 - while (trying) to keep > all the existing features it provides. > What I can't really do with the current one: execute easily a single qtests > from an IDE (as a matter of fact I can...but it's way too complicated; after > this it won't be a cake-walk either...but it will be a step closer ;) > I think this change will make it more clear how these tests are configured > and executed. > I will do this in two phases, currently i will only change > {{TestPerfCliDriver}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408659#comment-15408659 ] Saket Saurabh commented on HIVE-14035: -- Thanks [~sershe] for the comments and the feedback. I am working on addressing them. Sure, I will also put up a separate patch with default enabled for HiveQA run. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.12.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-11555) Beeline sends password in clear text if we miss -ssl=true flag in the connect string
[ https://issues.apache.org/jira/browse/HIVE-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junjie Chen updated HIVE-11555: --- Comment: was deleted (was: It should be simple if the ssl option set to true by defualt.) > Beeline sends password in clear text if we miss -ssl=true flag in the connect > string > > > Key: HIVE-11555 > URL: https://issues.apache.org/jira/browse/HIVE-11555 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0 >Reporter: bharath v >Assignee: Junjie Chen > > {code} > I used tcpdump to display the network traffic: > [root@fe01 ~]# beeline > Beeline version 0.13.1-cdh5.3.2 by Apache Hive > beeline> !connect jdbc:hive2://fe01.sectest.poc:1/default > Connecting to jdbc:hive2://fe01.sectest.poc:1/default > Enter username for jdbc:hive2://fe01.sectest.poc:1/default: tdaranyi > Enter password for jdbc:hive2://fe01.sectest.poc:1/default: * > (I entered "cleartext" as the password) > The tcpdump in a different window > tdara...@fe01.sectest.poc:~$ sudo tcpdump -n -X -i lo port 1 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes > (...) > 10:25:16.329974 IP 192.168.32.102.54322 > 192.168.32.102.ndmp: Flags [P.], > seq 11:35, ack 1, win 512, options [nop,nop,TS val 2412851969 ecr > 2412851969], length 24 > 0x: 4500 004c 3dd3 4000 4006 3abc c0a8 2066 E..L=.@.@.:f > 0x0010: c0a8 2066 d432 2710 714c 0edc b45c 9268 ...f.2'.qL...\.h > 0x0020: 8018 0200 c25b 0101 080a 8fd1 3301 .[3. > 0x0030: 8fd1 3301 0500 1300 7464 6172 616e ..3...tdaran > 0x0040: 7969 0063 6c65 6172 7465 7874 yi.cleartext > (...) > {code} > We rely on the user supplied configuration to decide whether to open an SSL > socket or a Plain one. Instead we can negotiate this information from the HS2 > and connect accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)