[jira] [Commented] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440797#comment-15440797 ] Matt McCline commented on HIVE-14515: - 1 day failures are related -- minor -- get rid of EXPLAIN for acid tests and exclude new schema_evol_* tests from TestCliDriver, etc. > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch, HIVE-14515.02.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14440) Fix default value of USE_DEPRECATED_CLI in cli.cmd
[
https://issues.apache.org/jira/browse/HIVE-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440778#comment-15440778
]
Hive QA commented on HIVE-14440:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825680/HIVE-14440.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1015/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1015/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1015/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-1015/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0933416 HIVE-14561. Minor ptest2 improvements. (Siddharth Seth,
reviewed by Prasanth Jayachandran)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 0933416 HIVE-14561. Minor ptest2 improvements. (Siddharth Seth,
reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825680 - PreCommit-HIVE-MASTER-Build
> Fix default value of USE_DEPRECATED_CLI in cli.cmd
> --
>
> Key: HIVE-14440
> URL: https://issues.apache.org/jira/browse/HIVE-14440
> Project: Hive
> Issue Type: Sub-task
> Components: CLI
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14440.01.patch, HIVE-14440.02.patch,
> HIVE-14440.03.patch, HIVE-14440.04.patch, HIVE-14440.05.patch,
> HIVE-14440.06.patch
>
>
> cli.cmd script sets the default value of USE_DEPRECATED_CLI to false when it
> is not set which is different than cli.sh which sets it to true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[
https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440776#comment-15440776
]
Hive QA commented on HIVE-14451:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825631/HIVE-14451.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10463 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1014/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1014/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1014/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825631 - PreCommit-HIVE-MASTER-Build
> Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
> --
>
> Key: HIVE-14451
> URL: https://issues.apache.org/jira/browse/HIVE-14451
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-14451.01.patch
>
>
> In a majority of cases, when using the OptimizedHashMap, the references to
> the byte[] are immutable.
> The hashmap result always allocates on boundary conditions, but never mutates
> a previous buffer.
> Copying Strings out of the hashtable is entirely wasteful and it would be
> easy to know when the currentBytes is a borrowed slice from the original
> input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14648) LLAP: Avoid private pages in the SSD cache
[
https://issues.apache.org/jira/browse/HIVE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440643#comment-15440643
]
Hive QA commented on HIVE-14648:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825602/HIVE-14648.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10463 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1013/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1013/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1013/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825602 - PreCommit-HIVE-MASTER-Build
> LLAP: Avoid private pages in the SSD cache
> --
>
> Key: HIVE-14648
> URL: https://issues.apache.org/jira/browse/HIVE-14648
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-14648.1.patch
>
>
> There's no reason for the SSD cache to have private mappings to the cache
> file, there's only one reader and the memory overheads aren't worth it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13383) RetryingMetaStoreClient retries non retriable embedded metastore client
[
https://issues.apache.org/jira/browse/HIVE-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440615#comment-15440615
]
Thejas M Nair commented on HIVE-13383:
--
Example of the useless error -
{noformat}
2016-08-27 00:47:46,929 WARN [HiveServer2-HttpHandler-Pool: Thread-2813 - /]:
metastore.RetryingMetaStoreClient (RetryingMetaStoreClient.java:invoke(187)) -
MetaStoreClient lost connection. Attempting to reconnect.
MetaException(message:For direct MetaStore DB connections, we don't support
retries at the client level.)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:321)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:151)
at com.sun.proxy.$Proxy14.listPartitionsWithAuthInfo(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:2209)
at
org.apache.hadoop.hive.ql.exec.ArchiveUtils.conflictingArchiveNameOrNull(ArchiveUtils.java:304)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.validate(SemanticAnalyzer.java:10930)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:320)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1219)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1213)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:226)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:276)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:468)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:456)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:298)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
at
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:345)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> RetryingMetaStoreClient retries non retriable embedded metastore client
>
>
> Key: HIVE-13383
> URL: https://issues.apache.org/jira/browse/HIVE-13383
> Project: Hive
>
[jira] [Updated] (HIVE-13383) RetryingMetaStoreClient retries non retriable embedded metastore client
[ https://issues.apache.org/jira/browse/HIVE-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-13383: - Status: Patch Available (was: Open) > RetryingMetaStoreClient retries non retriable embedded metastore client > > > Key: HIVE-13383 > URL: https://issues.apache.org/jira/browse/HIVE-13383 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-13383.1.patch > > > Embedded metastore clients can't be retried, they throw an exception - "For > direct MetaStore DB connections, we don't support retries at the client > level." > This tends to mask the real error that caused the attempts to retry. > RetryingMetaStoreClient shouldn't even attempt to reconnect when > direct/embedded metastore client is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[
https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440516#comment-15440516
]
Hive QA commented on HIVE-14515:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825603/HIVE-14515.02.patch
{color:green}SUCCESS:{color} +1 due to 28 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10444 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_orc_acid_table_update]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[schema_evol_orc_acidvec_table_update]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_acid_table]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_acidvec_table]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_stats]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1012/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1012/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1012/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825603 - PreCommit-HIVE-MASTER-Build
> Schema evolution uses slow INSERT INTO .. VALUES
>
>
> Key: HIVE-14515
> URL: https://issues.apache.org/jira/browse/HIVE-14515
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14515.01.patch, HIVE-14515.02.patch
>
>
> Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440397#comment-15440397
]
Sergey Shelukhin commented on HIVE-14652:
-
No tests fail, surprisingly enough. I wonder if the entire in path could be
removed and just go to regular function handling ;)
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-14652.patch
>
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440390#comment-15440390
]
Hive QA commented on HIVE-14652:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825591/HIVE-14652.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10464 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1011/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1011/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1011/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825591 - PreCommit-HIVE-MASTER-Build
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-14652.patch
>
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14627) Improvements to MiniMr tests
[ https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14627: - Attachment: HIVE-14627.3.patch All tests are just statistics difference caused by missing analyze commands in init scripts. infer_bucket_sort_reducers_power_two.q shows a different column for bucketing. That is because of join reordering not happening as CBO got disabled (no column statistics). But I guess this test is not intended to test CBO, so the diff is expected and safe. > Improvements to MiniMr tests > > > Key: HIVE-14627 > URL: https://issues.apache.org/jira/browse/HIVE-14627 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14627.1.patch, HIVE-14627.2.patch, > HIVE-14627.3.patch > > > Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following > are the execution time breakdown > Total time - 13m59s > Junit reported time for testcase - 50s > Most of the time is spent in creating/loading/analyzing initial tables - ~12m > Cleanup - ~1m > There is huge overhead for running MiniMr tests when compared to the actual > test runtime. > Ran the same test without init script. > Total time - 2m17s > Junit reported time for testcase - 52s > Also I noticed some tests that doesn't have to run on MiniMr (like > udf_using.q that does not require MiniMr. It just reads/write to hdfs which > we can do in MiniTez/MiniLlap which are way faster). Most tests access only > very few initial tables to read few rows from it. We can fix those tests to > load just the table that is required for the table instead of all initial > tables. Also we can remove q_init_script.sql initialization for MiniMr after > rewriting and moving over the unwanted tests which should cut down the > runtime a lot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12806) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure
[
https://issues.apache.org/jira/browse/HIVE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440287#comment-15440287
]
Vineet Garg commented on HIVE-12806:
This alias is actually built on HIVE side not CALCITE side. There are two
aliases on hive side: simple table alias and alias id (which is
:. For HIVE ast path (i.e. without return path)
table alias is used but for some reason with return path we are using alias id
instead.
> CBO: Calcite Operator To Hive Operator (Calcite Return Path):
> MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure
> ---
>
> Key: HIVE-12806
> URL: https://issues.apache.org/jira/browse/HIVE-12806
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Vineet Garg
> Attachments: HIVE-12806.1.patch
>
>
> Step to reproduce:
> mvn test -Dtest=TestMiniTezCliDriver -Dqfile=vector_auto_smb_mapjoin_14.q
> -Dhive.cbo.returnpath.hiveop=true -Dtest.output.overwrite=true
> Query :
> {code}
> select count(*) from (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2
> b on a.key = b.key
> ) subq1
> {code}
> Stack trace :
> {code}
> 2016-01-07T14:08:04,803 ERROR [da534038-d792-4d16-86e9-87b9f971adda main[]]:
> SessionState (SessionState.java:printError(1010)) - Vertex failed,
> vertexName=Map 1, vertexId=vertex_1452204324051_0001_33_00,
> diagnostics=[Vertex vertex_1452204324051_0001_33_00 [Map 1] k\
> illed/failed due to:AM_USERCODE_FAILURE, Exception in VertexManager,
> vertex:vertex_1452204324051_0001_33_00 [Map 1], java.lang.RuntimeException:
> java.lang.RuntimeException: Failed to load plan: null:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: \
> Relative path in absolute URI: subq1:amerge.xml
> at
> org.apache.hadoop.hive.ql.exec.tez.CustomPartitionVertex.onRootVertexInitialized(CustomPartitionVertex.java:314)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventRootInputInitialized.invoke(VertexManager.java:624)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:645)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:640)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:640)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:629)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed to load plan: null:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: subq1:amerge.xml
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMergeWork(Utilities.java:339)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.populateMapWork(SplitGrouper.java:260)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:172)
> at
> org.apache.hadoop.hive.ql.exec.tez.CustomPartitionVertex.onRootVertexInitialized(CustomPartitionVertex.java:277)
> ... 12 more
> Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI: subq1:amerge.xml
> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> at org.apache.hadoop.fs.Path.(Path.java:172)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getPlanPath(Utilities.java:588)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:387)
> ... 16 more
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> subq1:amerge.xml
> at java.net.URI.checkPath(URI.java:1804)
> at java.net.URI.(URI.java:752)
> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> ... 20 more
> ]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14487) Add REBUILD statement for materialized views
[
https://issues.apache.org/jira/browse/HIVE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440237#comment-15440237
]
Eugene Koifman commented on HIVE-14487:
---
I don't know the details of how you are implementing things but I'm assuming
you store the view results in some hidden table and the optimizer replaces the
view ref in a query with this table. Assuming that's the case, one possibility
is to keep several such tables.
Each time you rebuild, you write the data to the next available table. The
system would maintain a pointer somewhere that indicates to the compiler with
is the active data set. Then some kind of ref counting can be used to truncate
one of these tables once all readers of that specific snapshot are finished.
This has the advantage of never having a "materrizled view unavailable"
condition...
This would also fit well with Acid since each txn can add to this ref counter
and make sure it sees a stable world.
This may benefit replication as well.
I think inconsistent data is always bad so i would try to avoid that in the
short term in which ever way is more practical.
> Add REBUILD statement for materialized views
>
>
> Key: HIVE-14487
> URL: https://issues.apache.org/jira/browse/HIVE-14487
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Alan Gates
>
> Support for rebuilding existing materialized views. The statement is the
> following:
> {code:sql}
> ALTER MATERIALIZED VIEW [db_name.]materialized_view_name REBUILD;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[
https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440213#comment-15440213
]
Hive QA commented on HIVE-13589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825617/HIVE-13589.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10396 tests
executed
*Failed tests:*
{noformat}
TestBeeLineWithArgs - did not produce a TEST-*.xml file
TestHiveCli - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1010/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1010/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1010/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825617 - PreCommit-HIVE-MASTER-Build
> beeline - support prompt for password with '-u' option
> --
>
> Key: HIVE-13589
> URL: https://issues.apache.org/jira/browse/HIVE-13589
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Thejas M Nair
>Assignee: Ke Jia
> Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch,
> HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, HIVE-13589.6.patch
>
>
> Specifying connection string using commandline options in beeline is
> convenient, as it gets saved in shell command history, and it is easy to
> retrieve it from there.
> However, specifying the password in command prompt is not secure as it gets
> displayed on screen and saved in the history.
> It should be possible to specify '-p' without an argument to make beeline
> prompt for password.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12806) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure
[
https://issues.apache.org/jira/browse/HIVE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440166#comment-15440166
]
Vineet Garg commented on HIVE-12806:
As Hari earlier mentioned that CALCITE is adding subquery alias with colon to
table alias name. In this case mergejoin work has two inputs, while trying to
initialize inputs TEZ tries to figure out the plan path for inputs based on
input name. Input name for bigtablework seems to be coming out from underlying
table alias which has colon in it in this case. So root cause for this issue is
that table alias have colon in it. Replacing this colon with a dot fixes this
issue. But we would like to understand why CALCITE is appending colon. Also in
a lot of cases CALCITE appends null or wrong alias.
> CBO: Calcite Operator To Hive Operator (Calcite Return Path):
> MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure
> ---
>
> Key: HIVE-12806
> URL: https://issues.apache.org/jira/browse/HIVE-12806
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Vineet Garg
> Attachments: HIVE-12806.1.patch
>
>
> Step to reproduce:
> mvn test -Dtest=TestMiniTezCliDriver -Dqfile=vector_auto_smb_mapjoin_14.q
> -Dhive.cbo.returnpath.hiveop=true -Dtest.output.overwrite=true
> Query :
> {code}
> select count(*) from (
> select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2
> b on a.key = b.key
> ) subq1
> {code}
> Stack trace :
> {code}
> 2016-01-07T14:08:04,803 ERROR [da534038-d792-4d16-86e9-87b9f971adda main[]]:
> SessionState (SessionState.java:printError(1010)) - Vertex failed,
> vertexName=Map 1, vertexId=vertex_1452204324051_0001_33_00,
> diagnostics=[Vertex vertex_1452204324051_0001_33_00 [Map 1] k\
> illed/failed due to:AM_USERCODE_FAILURE, Exception in VertexManager,
> vertex:vertex_1452204324051_0001_33_00 [Map 1], java.lang.RuntimeException:
> java.lang.RuntimeException: Failed to load plan: null:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: \
> Relative path in absolute URI: subq1:amerge.xml
> at
> org.apache.hadoop.hive.ql.exec.tez.CustomPartitionVertex.onRootVertexInitialized(CustomPartitionVertex.java:314)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventRootInputInitialized.invoke(VertexManager.java:624)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:645)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:640)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:640)
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:629)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed to load plan: null:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: subq1:amerge.xml
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMergeWork(Utilities.java:339)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.populateMapWork(SplitGrouper.java:260)
> at
> org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:172)
> at
> org.apache.hadoop.hive.ql.exec.tez.CustomPartitionVertex.onRootVertexInitialized(CustomPartitionVertex.java:277)
> ... 12 more
> Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI: subq1:amerge.xml
> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> at org.apache.hadoop.fs.Path.(Path.java:172)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getPlanPath(Utilities.java:588)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:387)
> ... 16 more
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> subq1:amerge.xml
> at java.net.URI.check
[jira] [Updated] (HIVE-14536) Unit test code cleanup
[ https://issues.apache.org/jira/browse/HIVE-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14536: -- Attachment: HIVE-14536.7.patch Resubmitting patch > Unit test code cleanup > -- > > Key: HIVE-14536 > URL: https://issues.apache.org/jira/browse/HIVE-14536 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14536.5.patch, HIVE-14536.6.patch, > HIVE-14536.7.patch, HIVE-14536.patch > > > Clean up the itest infrastructure, to create a readable, easy to understand > code -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440097#comment-15440097
]
Prasanth Jayachandran commented on HIVE-14651:
--
nit: LOG.error instead?
{code}
try {
fs.mkdirs(userInstallPath);
} catch (IOException e) {
LOG.info("Failed to create path={}. Continuing. Exception message={}",
warehousePath,
e.getMessage());
}
{code}
There are 2 other places as well.
Other that that lgtm, +1
> Add a local cluster for Tez and LLAP
>
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch,
> HIVE-14651.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14651) Add a local cluster for Tez and LLAP
[
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440097#comment-15440097
]
Prasanth Jayachandran edited comment on HIVE-14651 at 8/26/16 10:46 PM:
nit: LOG.error instead?
{code}
try {
fs.mkdirs(userInstallPath);
} catch (IOException e) {
LOG.info("Failed to create path={}. Continuing. Exception message={}",
warehousePath,
e.getMessage());
}
{code}
There are 2 other places as well.
Other than that lgtm, +1
was (Author: prasanth_j):
nit: LOG.error instead?
{code}
try {
fs.mkdirs(userInstallPath);
} catch (IOException e) {
LOG.info("Failed to create path={}. Continuing. Exception message={}",
warehousePath,
e.getMessage());
}
{code}
There are 2 other places as well.
Other that that lgtm, +1
> Add a local cluster for Tez and LLAP
>
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch,
> HIVE-14651.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14651: -- Attachment: HIVE-14651.03.patch That was supposed to be disabling it for non-llap. Updated. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch, > HIVE-14651.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14598) Disallow newlines in query name
[
https://issues.apache.org/jira/browse/HIVE-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440089#comment-15440089
]
Tapan Oza commented on HIVE-14598:
--
I'd like to give this a shot. I think I've got it figured out but I'm still
learning the hive internals.
This occurs when:
String jobname = Utilities.abbreviate(queryStr, maxlen - 6);
is called in execute in org.apache.hadoop.hive.ql.Driver
The solution would be to replace that line with something like:
String jobname = Utilities.abbreviate(queryStr.replace('\n','
').replace('\r',''), maxlen - 6);
If this is right, can I please get it assigned to me so I can write a patch.
Thank you.
> Disallow newlines in query name
> ---
>
> Key: HIVE-14598
> URL: https://issues.apache.org/jira/browse/HIVE-14598
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>
> Hive sets up the query name based on the query text. This is truncated to a
> certain number of characters, but newlines are allowed.
> This makes logs really ugly - spanning across lines for the same log line.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440077#comment-15440077 ] Siddharth Seth commented on HIVE-14651: --- It should not be. Will upload a patch with this removed. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440058#comment-15440058
]
Prasanth Jayachandran commented on HIVE-14651:
--
Mostly looks good. Why is llap.io disabled?
{code}
if (isLlap) {
conf.setBoolean("hive.llap.io.enabled", false);
}
{code}
> Add a local cluster for Tez and LLAP
>
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14561) Minor ptest2 improvements
[ https://issues.apache.org/jira/browse/HIVE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14561: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Minor ptest2 improvements > - > > Key: HIVE-14561 > URL: https://issues.apache.org/jira/browse/HIVE-14561 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14561.01.patch, HIVE-14561.02.patch > > > Re-purposed to track a few more improvements. > - Update spring framework to work with Java8 > - Change elapseTime logging to milliseconds from seconds > - Add thread name to log files. > - Allow an empty logsEndPoint if outputDir is not specified > - Log configuration when starting in a web server > - Allow tests to be run even if no qtests property is set > - Fix an exception on test completion when using FixedExecutionContextProvider -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14561) Minor ptest2 improvements
[ https://issues.apache.org/jira/browse/HIVE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440044#comment-15440044 ] Siddharth Seth commented on HIVE-14561: --- Committing with the fix. > Minor ptest2 improvements > - > > Key: HIVE-14561 > URL: https://issues.apache.org/jira/browse/HIVE-14561 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14561.01.patch, HIVE-14561.02.patch > > > Re-purposed to track a few more improvements. > - Update spring framework to work with Java8 > - Change elapseTime logging to milliseconds from seconds > - Add thread name to log files. > - Allow an empty logsEndPoint if outputDir is not specified > - Log configuration when starting in a web server > - Allow tests to be run even if no qtests property is set > - Fix an exception on test completion when using FixedExecutionContextProvider -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14536) Unit test code cleanup
[
https://issues.apache.org/jira/browse/HIVE-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440005#comment-15440005
]
Hive QA commented on HIVE-14536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825654/HIVE-14536.6.patch
{color:green}SUCCESS:{color} +1 due to 24 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10458 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1009/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1009/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1009/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825654 - PreCommit-HIVE-MASTER-Build
> Unit test code cleanup
> --
>
> Key: HIVE-14536
> URL: https://issues.apache.org/jira/browse/HIVE-14536
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-14536.5.patch, HIVE-14536.6.patch, HIVE-14536.patch
>
>
> Clean up the itest infrastructure, to create a readable, easy to understand
> code
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14532) Enable qtests from IDE
[
https://issues.apache.org/jira/browse/HIVE-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14532:
Attachment: HIVE-14532.2.patch
patch changes: i've added a profile(ide) to disable the maven-shade plugin
during ide project creation...i've created it with this name and expected
further changes...but there were none
* we may possibly rename this profile to something like 'skipShade'...or keep
it as 'ide' if its only usefull for this case
* or i may remove the profile and use `-DshadeGoal`...but in that case when
something other ide related maven issue will arise we have to modify the docs...
* or keep it as-is...
not sure which one to pick ;)
i've written a draft wiki page about these things
http://hastebin.com/paxicutive.vhdl
[~leftylev] may I ask you to read it?
I've tested the draft using eclipse:
3.0 -> 3.1 -> TestCliDriver(combine2)
3.0 -> 3.2 -> TestCliDriver(combine2)
I think this should work with idea too...
[~baluchicken] may I ask you to check this once more? ...it should work "by the
book" ahm "draft" ;)
> Enable qtests from IDE
> --
>
> Key: HIVE-14532
> URL: https://issues.apache.org/jira/browse/HIVE-14532
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14532.1.patch, HIVE-14532.2.patch
>
>
> with HIVE-1 applied; i've played around with executing qtest-s from
> eclipse...after the patch seemed ok; i've checked it with:
> {code}
> git clean -dfx
> mvn package install eclipse:eclipse -Pitests -DskipTests
> mvn -q test -Pitests -Dtest=TestCliDriver -Dqfile=combine2.q
> {code}
> the last step I think is not required...but I bootstrapped and checked my
> project integrity this way.
> After this I was able to execute {{TestCliDriver}} from eclipse using
> {{-Dqfile=combine.q}}, other qfiles may or may not work...but will have at
> least some chances to be usable.
> For my biggest surprise {{alter_concatenate_indexed_table.q}} also
> passed...which contains relative file references - and I suspected that it
> will have issues with that..
> note: I've the datanucleus plugin installed...and i use it when I need to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14657) datediff function produce different results with timestamp and string combination
[
https://issues.apache.org/jira/browse/HIVE-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439954#comment-15439954
]
Anup commented on HIVE-14657:
-
And this occurs with only timestamp values .
select datediff(from_unixtime(unix_timestamp()), '2016-08-03 15:07:29.008')
from dummy_table;
23
select datediff (cast(from_unixtime(unix_timestamp()) AS TIMESTAMP),
cast('2016-08-03 15:07:29.008' AS TIMESTAMP)) from dummy_table;
22
> datediff function produce different results with timestamp and string
> combination
> -
>
> Key: HIVE-14657
> URL: https://issues.apache.org/jira/browse/HIVE-14657
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.13.0
>Reporter: Anup
>Priority: Minor
>
> when we use datediff function with string and timestamp type, it produces
> different results.
> See below queries:
> select datediff("2016-08-18 16:48:12", "2016-07-18 12:54:54") from test2;
> 31
> select datediff("2016-08-18 16:48:12", date) from test2;
> 30
> select datediff("2016-08-18 16:48:12", cast(date as string)) from test2;
> 31
> hive> desc test2;
> OK
> datetimestamp
> hive> select * from test2;
> OK
> 2016-07-18 12:54:54
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14598) Disallow newlines in query name
[ https://issues.apache.org/jira/browse/HIVE-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439902#comment-15439902 ] Siddharth Seth commented on HIVE-14598: --- Replacing with a space should be good enough. (Ensuring there's a space when the newline is removed) > Disallow newlines in query name > --- > > Key: HIVE-14598 > URL: https://issues.apache.org/jira/browse/HIVE-14598 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth > > Hive sets up the query name based on the query text. This is truncated to a > certain number of characters, but newlines are allowed. > This makes logs really ugly - spanning across lines for the same log line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14650) Select fails when ORC file has more columns than table schema
[
https://issues.apache.org/jira/browse/HIVE-14650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439888#comment-15439888
]
Jeff Mink commented on HIVE-14650:
--
I used column removal as an easy example of how to recreate the problem. A user
would also run into in in a scenario where there are multiple tables pointing
to the same location. If, for example, there is one managed table and one
external table pointing to the same location, and columns are added to the
managed table, this error would occur on any SELECT to the external table.
This behavior was introduced in 1.2 (as mentioned in the description)
> Select fails when ORC file has more columns than table schema
> -
>
> Key: HIVE-14650
> URL: https://issues.apache.org/jira/browse/HIVE-14650
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Jeff Mink
>Priority: Minor
>
> When SELECTing from a Hive ORC table, the following IndexOutOfBoundsException
> is thrown if the underlying ORC file has 4 or more columns than the Hive
> schema (where N is the number of columns in the ORC file).
> {noformat}
> Failed with exception
> java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
> 16/08/25 15:22:19 ERROR CliDriver: Failed with exception
> java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
> java.io.IOException: java.lang.IndexOutOfBoundsException: toIndex = N
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1686)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IndexOutOfBoundsException: toIndex = 6
> at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004)
> at java.util.ArrayList.subList(ArrayList.java:996)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:541)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1216)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:673)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:323)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
> ... 15 more
> {noformat}
> This error appears to be related to the patch of HIVE-10591.
> Steps to reproduce (Hive QL):
> {noformat}
> DROP TABLE IF EXISTS orc_drop_column;
> CREATE TABLE orc_drop_column (`id` int, `name` string, `description` string,
> `somevalue` double, `someflag` boolean, `somedate` timestamp) STORED AS ORC;
> INSERT INTO TABLE orc_drop_column select * from (select 1, 'my_name',
> 'my_desc', 5.5, true, '2016-08-25 06:00:00') a;
> ALTER TABLE orc_drop_column SET SERDE
> 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe';
> ALTER TABLE orc_drop_column REPLACE COLUMNS (
> `id` int
[jira] [Updated] (HIVE-14589) add consistent node replacement to LLAP for splits
[ https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14589: Description: See HIVE-14574. (copied from the comment below) This basically creates the nodes in ZK for "slots" in the cluster. The LLAPs try to take the lowest available slot, starting from 0. Unlike worker-... nodes, the slots are reused, which is the intent. The LLAPs are always sorted by the slot number for splits. The idea is that as long as LLAP is running, it will retain the same position in the ordering, regardless of other LLAPs restarting, without knowing about each other, the predecessors location (if restarted in a different place), or the total size of the cluster. The restarting LLAPs may not take the same positions as their predecessors (i.e. if two LLAPs restart they can swap slots) but it shouldn't matter because they have lost their cache anyway. I.e. if you have LLAPs with slots 1-2-3-4 and I nuke and restart 1, 2, and 4, they will take whatever slots, but 3 will stay the 3rd and retain cache locality. This also handles size increase, as new LLAPs will always be added to the end of the sequence, which is what consistent hashing needs. One case it doesn't handle is permanent cluster size reduction. There will be a permanent gap if LLAPs are removed that have the slots in the middle; until some are restarted, it will result in misses was:See HIVE-14574 > add consistent node replacement to LLAP for splits > -- > > Key: HIVE-14589 > URL: https://issues.apache.org/jira/browse/HIVE-14589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14589.01.patch, HIVE-14589.patch > > > See HIVE-14574. (copied from the comment below) This basically creates the > nodes in ZK for "slots" in the cluster. The LLAPs try to take the lowest > available slot, starting from 0. Unlike worker-... nodes, the slots are > reused, which is the intent. The LLAPs are always sorted by the slot number > for splits. > The idea is that as long as LLAP is running, it will retain the same position > in the ordering, regardless of other LLAPs restarting, without knowing about > each other, the predecessors location (if restarted in a different place), or > the total size of the cluster. > The restarting LLAPs may not take the same positions as their predecessors > (i.e. if two LLAPs restart they can swap slots) but it shouldn't matter > because they have lost their cache anyway. > I.e. if you have LLAPs with slots 1-2-3-4 and I nuke and restart 1, 2, and 4, > they will take whatever slots, but 3 will stay the 3rd and retain cache > locality. > This also handles size increase, as new LLAPs will always be added to the end > of the sequence, which is what consistent hashing needs. > One case it doesn't handle is permanent cluster size reduction. There will be > a permanent gap if LLAPs are removed that have the slots in the middle; until > some are restarted, it will result in misses -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14650) Select fails when ORC file has more columns than table schema
[
https://issues.apache.org/jira/browse/HIVE-14650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439805#comment-15439805
]
Sergey Shelukhin commented on HIVE-14650:
-
I don't think column removal is supported by schema evolution yet. cc
[~mmccline]
> Select fails when ORC file has more columns than table schema
> -
>
> Key: HIVE-14650
> URL: https://issues.apache.org/jira/browse/HIVE-14650
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Jeff Mink
>Priority: Minor
>
> When SELECTing from a Hive ORC table, the following IndexOutOfBoundsException
> is thrown if the underlying ORC file has 4 or more columns than the Hive
> schema (where N is the number of columns in the ORC file).
> {noformat}
> Failed with exception
> java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
> 16/08/25 15:22:19 ERROR CliDriver: Failed with exception
> java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
> java.io.IOException: java.lang.IndexOutOfBoundsException: toIndex = N
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1686)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IndexOutOfBoundsException: toIndex = 6
> at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004)
> at java.util.ArrayList.subList(ArrayList.java:996)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:541)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1216)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:673)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:323)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
> ... 15 more
> {noformat}
> This error appears to be related to the patch of HIVE-10591.
> Steps to reproduce (Hive QL):
> {noformat}
> DROP TABLE IF EXISTS orc_drop_column;
> CREATE TABLE orc_drop_column (`id` int, `name` string, `description` string,
> `somevalue` double, `someflag` boolean, `somedate` timestamp) STORED AS ORC;
> INSERT INTO TABLE orc_drop_column select * from (select 1, 'my_name',
> 'my_desc', 5.5, true, '2016-08-25 06:00:00') a;
> ALTER TABLE orc_drop_column SET SERDE
> 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe';
> ALTER TABLE orc_drop_column REPLACE COLUMNS (
> `id` int,
> `name` string
> );
> ALTER TABLE orc_drop_column SET SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde';
> SELECT id, name FROM orc_drop_column;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14654) Enable support for binary and blob datatypes
[ https://issues.apache.org/jira/browse/HIVE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439783#comment-15439783 ] Ashutosh Chauhan commented on HIVE-14654: - Binary datatype exists since 0.8 Details: HIVE-2380 What else are you looking for? > Enable support for binary and blob datatypes > > > Key: HIVE-14654 > URL: https://issues.apache.org/jira/browse/HIVE-14654 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Akshay >Assignee: a > Labels: datatypes, hive > > Currently when you use map reduce across mysql and other data endpoints. Hive > induces some limitations while transferring blob and binary data. It would be > good to add support for binary and other datatypes which are present in RDS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14627) Improvements to MiniMr tests
[
https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439779#comment-15439779
]
Hive QA commented on HIVE-14627:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825583/HIVE-14627.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 10463 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[bucket4]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[bucket5]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[bucket6]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[bucket_many]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[disable_merge_for_bucketing]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[groupby2]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[index_bitmap_auto]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_dyn_part]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_map_operators]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[join1]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[list_bucket_dml_10]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[orc_merge_diff_fs]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[reduce_deduplicate]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1008/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1008/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1008/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825583 - PreCommit-HIVE-MASTER-Build
> Improvements to MiniMr tests
>
>
> Key: HIVE-14627
> URL: https://issues.apache.org/jira/browse/HIVE-14627
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14627.1.patch, HIVE-14627.2.patch
>
>
> Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following
> are the execution time breakdown
> Total time - 13m59s
> Junit reported time for testcase - 50s
> Most of the time is spent in creating/loading/analyzing initial tables - ~12m
> Cleanup - ~1m
> There is huge overhead for running MiniMr tests when compared to the actual
> test runtime.
> Ran the same test without init script.
> Total time - 2m17s
> Junit reported time for testcase - 52s
> Also I noticed some tests that doesn't have to run on MiniMr (like
> udf_using.q that does not require MiniMr. It just reads/write to hdfs which
> we can do in MiniTez/MiniLlap which are way faster). Most tests access only
> very few initial tables to read few rows from it. We can fix those tests to
> load just the table that is required for the table instead of all initial
> tables. Also we can remove q_init_script.sql initialization for MiniMr after
> rewriting and moving over the unwanted tests which should cut down the
> runtime a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14362) Support explain analyze in Hive
[ https://issues.apache.org/jira/browse/HIVE-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14362: --- Status: Patch Available (was: Open) > Support explain analyze in Hive > --- > > Key: HIVE-14362 > URL: https://issues.apache.org/jira/browse/HIVE-14362 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14362.01.patch, HIVE-14362.02.patch, > HIVE-14362.03.patch, HIVE-14362.05.patch, compare_on_cluster.pdf > > > Right now all the explain levels only support stats before query runs. We > would like to have an explain analyze similar to Postgres for real stats > after query runs. This will help to identify the major gap between > estimated/real stats and make not only query optimization better but also > query performance debugging easier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14362) Support explain analyze in Hive
[ https://issues.apache.org/jira/browse/HIVE-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14362: --- Attachment: HIVE-14362.05.patch > Support explain analyze in Hive > --- > > Key: HIVE-14362 > URL: https://issues.apache.org/jira/browse/HIVE-14362 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14362.01.patch, HIVE-14362.02.patch, > HIVE-14362.03.patch, HIVE-14362.05.patch, compare_on_cluster.pdf > > > Right now all the explain levels only support stats before query runs. We > would like to have an explain analyze similar to Postgres for real stats > after query runs. This will help to identify the major gap between > estimated/real stats and make not only query optimization better but also > query performance debugging easier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14362) Support explain analyze in Hive
[ https://issues.apache.org/jira/browse/HIVE-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14362: --- Status: Open (was: Patch Available) > Support explain analyze in Hive > --- > > Key: HIVE-14362 > URL: https://issues.apache.org/jira/browse/HIVE-14362 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14362.01.patch, HIVE-14362.02.patch, > HIVE-14362.03.patch, HIVE-14362.05.patch, compare_on_cluster.pdf > > > Right now all the explain levels only support stats before query runs. We > would like to have an explain analyze similar to Postgres for real stats > after query runs. This will help to identify the major gap between > estimated/real stats and make not only query optimization better but also > query performance debugging easier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[ https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439667#comment-15439667 ] Matt McCline edited comment on HIVE-14451 at 8/26/16 7:52 PM: -- Instead of writing the new bytes data to a private Text object, I think I'll add an option to append that data to a caller provided bytes buffer. If the buffer needs to be expanded... It looks like BytesColumnVector has a performance bug there. It copies the old data to the new twice size buffer but does not fixup the byte[][] vector array. was (Author: mmccline): Instead of writing the new bytes data to a Text object, I think I'll add an option to append that data to a caller provided byte[] buffer and nextFree offset. > Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow > -- > > Key: HIVE-14451 > URL: https://issues.apache.org/jira/browse/HIVE-14451 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-14451.01.patch > > > In a majority of cases, when using the OptimizedHashMap, the references to > the byte[] are immutable. > The hashmap result always allocates on boundary conditions, but never mutates > a previous buffer. > Copying Strings out of the hashtable is entirely wasteful and it would be > easy to know when the currentBytes is a borrowed slice from the original > input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[ https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439667#comment-15439667 ] Matt McCline commented on HIVE-14451: - Instead of writing the new bytes data to a Text object, I think I'll add an option to append that data to a caller provided byte[] buffer and nextFree offset. > Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow > -- > > Key: HIVE-14451 > URL: https://issues.apache.org/jira/browse/HIVE-14451 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-14451.01.patch > > > In a majority of cases, when using the OptimizedHashMap, the references to > the byte[] are immutable. > The hashmap result always allocates on boundary conditions, but never mutates > a previous buffer. > Copying Strings out of the hashtable is entirely wasteful and it would be > easy to know when the currentBytes is a borrowed slice from the original > input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14656) Clean up driver instance in get_splits
[ https://issues.apache.org/jira/browse/HIVE-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14656: -- Attachment: HIVE-14656.1.patch close/destroy driver instance > Clean up driver instance in get_splits > -- > > Key: HIVE-14656 > URL: https://issues.apache.org/jira/browse/HIVE-14656 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14656.1.patch > > > get_splits() creates a Driver instance that needs to be closed/cleaned up > after use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14656) Clean up driver instance in get_splits
[ https://issues.apache.org/jira/browse/HIVE-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14656: -- Status: Patch Available (was: Open) > Clean up driver instance in get_splits > -- > > Key: HIVE-14656 > URL: https://issues.apache.org/jira/browse/HIVE-14656 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14656.1.patch > > > get_splits() creates a Driver instance that needs to be closed/cleaned up > after use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14655) LLAP input format should escape the query string being passed to getSplits()
[ https://issues.apache.org/jira/browse/HIVE-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14655: -- Attachment: HIVE-14655.1.patch > LLAP input format should escape the query string being passed to getSplits() > > > Key: HIVE-14655 > URL: https://issues.apache.org/jira/browse/HIVE-14655 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14655.1.patch > > > Query may not be parsed correctly by get_splits() otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14655) LLAP input format should escape the query string being passed to getSplits()
[ https://issues.apache.org/jira/browse/HIVE-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14655: -- Status: Patch Available (was: Open) > LLAP input format should escape the query string being passed to getSplits() > > > Key: HIVE-14655 > URL: https://issues.apache.org/jira/browse/HIVE-14655 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14655.1.patch > > > Query may not be parsed correctly by get_splits() otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14654) Enable support for binary and blob datatypes
[ https://issues.apache.org/jira/browse/HIVE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay updated HIVE-14654: -- Assignee: a (was: Akshay) > Enable support for binary and blob datatypes > > > Key: HIVE-14654 > URL: https://issues.apache.org/jira/browse/HIVE-14654 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Akshay >Assignee: a > Labels: datatypes, hive > > Currently when you use map reduce across mysql and other data endpoints. Hive > induces some limitations while transferring blob and binary data. It would be > good to add support for binary and other datatypes which are present in RDS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14651) Add a local cluster for Tez and LLAP
[
https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439586#comment-15439586
]
Hive QA commented on HIVE-14651:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825619/HIVE-14651.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10404 tests
executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_join30]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1007/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1007/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1007/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825619 - PreCommit-HIVE-MASTER-Build
> Add a local cluster for Tez and LLAP
>
>
> Key: HIVE-14651
> URL: https://issues.apache.org/jira/browse/HIVE-14651
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14618) beeline fetch logging delays before query completion
[ https://issues.apache.org/jira/browse/HIVE-14618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439580#comment-15439580 ] Tao Li commented on HIVE-14618: --- Test issues seem unrelated. > beeline fetch logging delays before query completion > > > Key: HIVE-14618 > URL: https://issues.apache.org/jira/browse/HIVE-14618 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14618.1.patch, HIVE-14618.2.patch, > HIVE-14618.3.patch > > > Beeline has a thread that fetches logs from HS2. However, it uses the same > HiveStatement object to also wait for query completion using a long-poll > (with default interval of 5 seconds). > The jdbc client has a lock around the thrift api calls, resulting in the > getLogs api blocking on the query completion check. ie the logs would get > shown only every 5 seconds by default. > cc [~vgumashta] [~gopalv] [~thejas] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14648) LLAP: Avoid private pages in the SSD cache
[ https://issues.apache.org/jira/browse/HIVE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439549#comment-15439549 ] Sergey Shelukhin commented on HIVE-14648: - +1 > LLAP: Avoid private pages in the SSD cache > -- > > Key: HIVE-14648 > URL: https://issues.apache.org/jira/browse/HIVE-14648 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Blocker > Fix For: 2.2.0 > > Attachments: HIVE-14648.1.patch > > > There's no reason for the SSD cache to have private mappings to the cache > file, there's only one reader and the memory overheads aren't worth it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14614) Insert overwrite local directory fails with IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-14614: --- Attachment: HIVE-14614.4.patch Attaching the patch with added testcase for local paths for the getTempDirForPath method in TestContext.java as suggested by [~spena] > Insert overwrite local directory fails with IllegalStateException > - > > Key: HIVE-14614 > URL: https://issues.apache.org/jira/browse/HIVE-14614 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > Attachments: HIVE-14614.2.patch, HIVE-14614.3.patch, > HIVE-14614.4.patch > > > insert overwrite local directory select * from table; fails with > "java.lang.IllegalStateException: Cannot create staging directory" when the > path sent to the getTempDirForPath(Path path) is a local fs path. > This is a regression caused by the fix for HIVE-14270 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439494#comment-15439494 ] Sergio Peña commented on HIVE-13930: The patch looks good +1 > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439495#comment-15439495 ] Sergio Peña commented on HIVE-13930: The patch looks good +1 > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439477#comment-15439477 ] Sergey Shelukhin commented on HIVE-13930: - Failures are unrelated. > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14614) Insert overwrite local directory fails with IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439471#comment-15439471
]
Sergio Peña commented on HIVE-14614:
[~vihangk1] One more thing, could you add some unit tests on
{{TestContext.java}} ?
> Insert overwrite local directory fails with IllegalStateException
> -
>
> Key: HIVE-14614
> URL: https://issues.apache.org/jira/browse/HIVE-14614
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14614.2.patch, HIVE-14614.3.patch
>
>
> insert overwrite local directory select * from table; fails with
> "java.lang.IllegalStateException: Cannot create staging directory" when the
> path sent to the getTempDirForPath(Path path) is a local fs path.
> This is a regression caused by the fix for HIVE-14270
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14614) Insert overwrite local directory fails with IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439459#comment-15439459 ] Sergio Peña commented on HIVE-14614: [~vihangk1] I will apply the patch2. There is only one line difference, and It is better than creating another variable that is repeated. > Insert overwrite local directory fails with IllegalStateException > - > > Key: HIVE-14614 > URL: https://issues.apache.org/jira/browse/HIVE-14614 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > Attachments: HIVE-14614.2.patch, HIVE-14614.3.patch > > > insert overwrite local directory select * from table; fails with > "java.lang.IllegalStateException: Cannot create staging directory" when the > path sent to the getTempDirForPath(Path path) is a local fs path. > This is a regression caused by the fix for HIVE-14270 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14614) Insert overwrite local directory fails with IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439455#comment-15439455 ] Sergio Peña commented on HIVE-14614: +1 > Insert overwrite local directory fails with IllegalStateException > - > > Key: HIVE-14614 > URL: https://issues.apache.org/jira/browse/HIVE-14614 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > Attachments: HIVE-14614.2.patch, HIVE-14614.3.patch > > > insert overwrite local directory select * from table; fails with > "java.lang.IllegalStateException: Cannot create staging directory" when the > path sent to the getTempDirForPath(Path path) is a local fs path. > This is a regression caused by the fix for HIVE-14270 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14217) Druid integration
[ https://issues.apache.org/jira/browse/HIVE-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14217: --- Attachment: HIVE-14217.03.patch > Druid integration > - > > Key: HIVE-14217 > URL: https://issues.apache.org/jira/browse/HIVE-14217 > Project: Hive > Issue Type: New Feature > Components: Druid integration >Reporter: Julian Hyde >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14217.01.patch, HIVE-14217.02.patch, > HIVE-14217.03.patch > > > Allow Hive to query data in Druid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14561) Minor ptest2 improvements
[
https://issues.apache.org/jira/browse/HIVE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439430#comment-15439430
]
Hive QA commented on HIVE-14561:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825526/HIVE-14561.02.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10447 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1006/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1006/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1006/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825526 - PreCommit-HIVE-MASTER-Build
> Minor ptest2 improvements
> -
>
> Key: HIVE-14561
> URL: https://issues.apache.org/jira/browse/HIVE-14561
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 2.2.0
>
> Attachments: HIVE-14561.01.patch, HIVE-14561.02.patch
>
>
> Re-purposed to track a few more improvements.
> - Update spring framework to work with Java8
> - Change elapseTime logging to milliseconds from seconds
> - Add thread name to log files.
> - Allow an empty logsEndPoint if outputDir is not specified
> - Log configuration when starting in a web server
> - Allow tests to be run even if no qtests property is set
> - Fix an exception on test completion when using FixedExecutionContextProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14598) Disallow newlines in query name
[ https://issues.apache.org/jira/browse/HIVE-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439411#comment-15439411 ] Tapan Oza commented on HIVE-14598: -- Do we just want to strip out newlines from the query text when determining the query name? Or replace it with a special character? > Disallow newlines in query name > --- > > Key: HIVE-14598 > URL: https://issues.apache.org/jira/browse/HIVE-14598 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth > > Hive sets up the query name based on the query text. This is truncated to a > certain number of characters, but newlines are allowed. > This makes logs really ugly - spanning across lines for the same log line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14654) Enable support for binary and blob datatypes
[ https://issues.apache.org/jira/browse/HIVE-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay reassigned HIVE-14654: - Assignee: Akshay > Enable support for binary and blob datatypes > > > Key: HIVE-14654 > URL: https://issues.apache.org/jira/browse/HIVE-14654 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Reporter: Akshay >Assignee: Akshay > Labels: datatypes, hive > > Currently when you use map reduce across mysql and other data endpoints. Hive > induces some limitations while transferring blob and binary data. It would be > good to add support for binary and other datatypes which are present in RDS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14440) Fix default value of USE_DEPRECATED_CLI in cli.cmd
[ https://issues.apache.org/jira/browse/HIVE-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-14440: --- Attachment: HIVE-14440.06.patch Thanks [~spena]. Attaching the patch again to re-run HiveQA > Fix default value of USE_DEPRECATED_CLI in cli.cmd > -- > > Key: HIVE-14440 > URL: https://issues.apache.org/jira/browse/HIVE-14440 > Project: Hive > Issue Type: Sub-task > Components: CLI >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > Attachments: HIVE-14440.01.patch, HIVE-14440.02.patch, > HIVE-14440.03.patch, HIVE-14440.04.patch, HIVE-14440.05.patch, > HIVE-14440.06.patch > > > cli.cmd script sets the default value of USE_DEPRECATED_CLI to false when it > is not set which is different than cli.sh which sets it to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14554) Download the spark-assembly file on itests only if the MD5 checksum file is different
[
https://issues.apache.org/jira/browse/HIVE-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14554:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks guys. I committed this to master.
> Download the spark-assembly file on itests only if the MD5 checksum file is
> different
> -
>
> Key: HIVE-14554
> URL: https://issues.apache.org/jira/browse/HIVE-14554
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Fix For: 2.2.0
>
> Attachments: HIVE-14554.1.patch
>
>
> The {{itests/thridparty}} directory is created by hive on spark when
> downloading the spark-assembly file. Hive ptest should delete this directory
> everytime it runs a new set of tests to avoid conflicts when a new spark
> tarball is submitted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14554) Download the spark-assembly file on itests only if the MD5 checksum file is different
[
https://issues.apache.org/jira/browse/HIVE-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14554:
---
Summary: Download the spark-assembly file on itests only if the MD5
checksum file is different (was: Hive ptest should delete the
itests/thirdparty directory everytime it builds hive)
> Download the spark-assembly file on itests only if the MD5 checksum file is
> different
> -
>
> Key: HIVE-14554
> URL: https://issues.apache.org/jira/browse/HIVE-14554
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Fix For: 2.2.0
>
> Attachments: HIVE-14554.1.patch
>
>
> The {{itests/thridparty}} directory is created by hive on spark when
> downloading the spark-assembly file. Hive ptest should delete this directory
> everytime it runs a new set of tests to avoid conflicts when a new spark
> tarball is submitted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14440) Fix default value of USE_DEPRECATED_CLI in cli.cmd
[ https://issues.apache.org/jira/browse/HIVE-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439392#comment-15439392 ] Sergio Peña commented on HIVE-14440: [~vihangk1] I replaced the smart-apply-patch.sh with the newest one that contains the git apply. Can you submit the patch again to rerun the tests? > Fix default value of USE_DEPRECATED_CLI in cli.cmd > -- > > Key: HIVE-14440 > URL: https://issues.apache.org/jira/browse/HIVE-14440 > Project: Hive > Issue Type: Sub-task > Components: CLI >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > Attachments: HIVE-14440.01.patch, HIVE-14440.02.patch, > HIVE-14440.03.patch, HIVE-14440.04.patch, HIVE-14440.05.patch > > > cli.cmd script sets the default value of USE_DEPRECATED_CLI to false when it > is not set which is different than cli.sh which sets it to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11957) SHOW TRANSACTIONS should show queryID/agent id of the creator
[
https://issues.apache.org/jira/browse/HIVE-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439298#comment-15439298
]
Hive QA commented on HIVE-11957:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825542/HIVE-11957.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 10460 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testAbortTxn
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testOpenTxn
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testRecoverManyTimeouts
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testValidTxnsSomeOpen
org.apache.hadoop.hive.metastore.txn.TestTxnHandlerNoConnectionPool.testOpenTxn
org.apache.hadoop.hive.ql.TestTxnCommands.testTimeOutReaper
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMetastoreTablesCleanup
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactAfterAbort
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreaming
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.majorCompactWhileStreamingForSplitUpdate
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactAfterAbort
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreaming
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.minorCompactWhileStreamingWithSplitUpdate
org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.cleanEmptyAbortedTxns
org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.majorCompactOnPartitionTooManyAborts
org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.majorCompactOnTableTooManyAborts
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorWithOpenInMiddle
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorWithOpenInMiddle
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorWithOpenInMiddle
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorWithAborted
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.minorWithOpenInMiddle
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.hcatalog.streaming.TestStreaming.testErrorHandling
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1005/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1005/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1005/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825542 - PreCommit-HIVE-MASTER-Build
> SHOW TRANSACTIONS should show queryID/agent id of the creator
> -
>
> Key: HIVE-11957
> URL: https://issues.apache.org/jira/browse/HIVE-11957
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Attachments: HIVE-11957.1.patch
>
>
> this would be very useful for debugging
> should also include heartbeat/create timestamps
> would be nice to support some filtering/sorting options, like sort by create
> time, agent id. filter by table, database, etc
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12720) Partition pruning not happening on left join with CTE
[
https://issues.apache.org/jira/browse/HIVE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-12720:
Description:
In certain cases hive is not pruning partitions as well as it could be, and
instead scanning the whole table.
Minimal use case:
{code:sql}
create table tbl1 (
id1 int,
ext_id2 int
)
PARTITIONED BY (ymd int);
create table tbl2 (
id2 int,
prop string
)
PARTITIONED BY (ymd int);
INSERT INTO tbl1 PARTITION(ymd=20151201) VALUES (1,1);
INSERT INTO tbl1 PARTITION(ymd=20151202) VALUES (1,2);
INSERT INTO tbl1 PARTITION(ymd=20151203) VALUES (1,3);
INSERT INTO tbl2 PARTITION(ymd=20151201) VALUES (1,'testa');
INSERT INTO tbl2 PARTITION(ymd=20151202) VALUES (2,'testb');
INSERT INTO tbl2 PARTITION(ymd=20151203) VALUES (3,'testc');
-- query 1
EXPLAIN
select * from
tbl1
LEFT OUTER JOIN
tbl2
ON
(tbl1.ext_id2 = tbl2.id2
and tbl2.ymd = tbl1.ymd)
WHERE
tbl1.ymd = 20151203
-- query2
EXPLAIN
with cte as (select * from tbl1
WHERE
tbl1.ymd = 20151203)
select * from
cte
LEFT OUTER JOIN
tbl2
ON
(cte.ext_id2 = tbl2.id2
and tbl2.ymd = cte.ymd)
WHERE
tbl2.ymd = 20151203
{code}
Query1 prunes correctly:
alias: tbl2
Statistics: Num rows: 1 Data size: 7 Basic stats: COMPLETE Column stats: NONE
Query2 does a full table scan on tbl2:
Statistics: Num rows: 4 Data size: 28 Basic stats: COMPLETE Column stats: NONE
I know both queries are not equivalent and the second query is not really a
LEFT JOIN in its current state, but I think the optimiser could do a better job
here.
This example was (re)produced with a cdh 5.5 docker image.
This issue can be reproduced when:
set hive.cbo.enable=false;
was:
In certain cases hive is not pruning partitions as well as it could be, and
instead scanning the whole table.
Minimal use case:
{code:sql}
create table tbl1 (
id1 int,
ext_id2 int
)
PARTITIONED BY (ymd int);
create table tbl2 (
id2 int,
prop string
)
PARTITIONED BY (ymd int);
INSERT INTO tbl1 PARTITION(ymd=20151201) VALUES (1,1);
INSERT INTO tbl1 PARTITION(ymd=20151202) VALUES (1,2);
INSERT INTO tbl1 PARTITION(ymd=20151203) VALUES (1,3);
INSERT INTO tbl2 PARTITION(ymd=20151201) VALUES (1,'testa');
INSERT INTO tbl2 PARTITION(ymd=20151202) VALUES (2,'testb');
INSERT INTO tbl2 PARTITION(ymd=20151203) VALUES (3,'testc');
-- query 1
EXPLAIN
select * from
tbl1
LEFT OUTER JOIN
tbl2
ON
(tbl1.ext_id2 = tbl2.id2
and tbl2.ymd = tbl1.ymd)
WHERE
tbl1.ymd = 20151203
-- query2
EXPLAIN
with cte as (select * from tbl1
WHERE
tbl1.ymd = 20151203)
select * from
cte
LEFT OUTER JOIN
tbl2
ON
(cte.ext_id2 = tbl2.id2
and tbl2.ymd = cte.ymd)
WHERE
tbl2.ymd = 20151203
{code}
Query1 prunes correctly:
alias: tbl2
Statistics: Num rows: 1 Data size: 7 Basic stats: COMPLETE Column stats: NONE
Query2 does a full table scan on tbl2:
Statistics: Num rows: 4 Data size: 28 Basic stats: COMPLETE Column stats: NONE
I know both queries are not equivalent and the second query is not really a
LEFT JOIN in its current state, but I think the optimiser could do a better job
here.
This example was (re)produced with a cdh 5.5 docker image.
> Partition pruning not happening on left join with CTE
> -
>
> Key: HIVE-12720
> URL: https://issues.apache.org/jira/browse/HIVE-12720
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Clemens Valiente
>
> In certain cases hive is not pruning partitions as well as it could be, and
> instead scanning the whole table.
> Minimal use case:
> {code:sql}
> create table tbl1 (
> id1 int,
> ext_id2 int
> )
> PARTITIONED BY (ymd int);
> create table tbl2 (
> id2 int,
> prop string
> )
> PARTITIONED BY (ymd int);
> INSERT INTO tbl1 PARTITION(ymd=20151201) VALUES (1,1);
> INSERT INTO tbl1 PARTITION(ymd=20151202) VALUES (1,2);
> INSERT INTO tbl1 PARTITION(ymd=20151203) VALUES (1,3);
> INSERT INTO tbl2 PARTITION(ymd=20151201) VALUES (1,'testa');
> INSERT INTO tbl2 PARTITION(ymd=20151202) VALUES (2,'testb');
> INSERT INTO tbl2 PARTITION(ymd=20151203) VALUES (3,'testc');
> -- query 1
> EXPLAIN
> select * from
> tbl1
> LEFT OUTER JOIN
> tbl2
> ON
> (tbl1.ext_id2 = tbl2.id2
> and tbl2.ymd = tbl1.ymd)
> WHERE
> tbl1.ymd = 20151203
> -- query2
> EXPLAIN
> with cte as (select * from tbl1
> WHERE
> tbl1.ymd = 20151203)
> select * from
> cte
> LEFT OUTER JOIN
> tbl2
> ON
> (cte.ext_id2 = tbl2.id2
> and tbl2.ymd = cte.ymd)
> WHERE
> tbl2.ymd = 20151203
> {code}
> Query1 prunes correctly:
> alias: tbl2
> Statistics: Num rows: 1 Data size: 7 Basic stats: COMPLETE Column stats: NONE
> Query2 does a full table scan on tbl2:
> Statistics: Num rows: 4 Data size: 28 Basic stats: COMPLETE Column stats:
> NONE
> I know both queries are not equivalent and the second query is not really a
> LEFT JOIN in its current state, but I think the optimiser could do a better
> job here.
> This example was (re)prod
[jira] [Updated] (HIVE-14360) Starting BeeLine after using !save, there is an error logged: "Error setting configuration: conf"
[
https://issues.apache.org/jira/browse/HIVE-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14360:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks [~pvary]. I committed this to master.
> Starting BeeLine after using !save, there is an error logged: "Error setting
> configuration: conf"
> -
>
> Key: HIVE-14360
> URL: https://issues.apache.org/jira/browse/HIVE-14360
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14360.patch
>
>
> When saving the configuration in BeeLine the conf attribute is persisted,
> which should not. When loading the configuration this causes an error message
> to be printed:
> {noformat}
> Error setting configuration: conf: java.lang.IllegalArgumentException: No
> method matching "setconf" was found in org.apache.hive.beeline.BeeLineOpts.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14360) Starting BeeLine after using !save, there is an error logged: "Error setting configuration: conf"
[
https://issues.apache.org/jira/browse/HIVE-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439159#comment-15439159
]
Sergio Peña commented on HIVE-14360:
The patch looks good to me
+1
> Starting BeeLine after using !save, there is an error logged: "Error setting
> configuration: conf"
> -
>
> Key: HIVE-14360
> URL: https://issues.apache.org/jira/browse/HIVE-14360
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-14360.patch
>
>
> When saving the configuration in BeeLine the conf attribute is persisted,
> which should not. When loading the configuration this causes an error message
> to be printed:
> {noformat}
> Error setting configuration: conf: java.lang.IllegalArgumentException: No
> method matching "setconf" was found in org.apache.hive.beeline.BeeLineOpts.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14618) beeline fetch logging delays before query completion
[
https://issues.apache.org/jira/browse/HIVE-14618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439115#comment-15439115
]
Hive QA commented on HIVE-14618:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825541/HIVE-14618.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1004/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1004/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1004/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825541 - PreCommit-HIVE-MASTER-Build
> beeline fetch logging delays before query completion
>
>
> Key: HIVE-14618
> URL: https://issues.apache.org/jira/browse/HIVE-14618
> Project: Hive
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-14618.1.patch, HIVE-14618.2.patch,
> HIVE-14618.3.patch
>
>
> Beeline has a thread that fetches logs from HS2. However, it uses the same
> HiveStatement object to also wait for query completion using a long-poll
> (with default interval of 5 seconds).
> The jdbc client has a lock around the thrift api calls, resulting in the
> getLogs api blocking on the query completion check. ie the logs would get
> shown only every 5 seconds by default.
> cc [~vgumashta] [~gopalv] [~thejas]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14418) Hive config validation prevents unsetting the settings
[
https://issues.apache.org/jira/browse/HIVE-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438880#comment-15438880
]
Hive QA commented on HIVE-14418:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825539/HIVE-14418.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1003/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1003/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1003/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825539 - PreCommit-HIVE-MASTER-Build
> Hive config validation prevents unsetting the settings
> --
>
> Key: HIVE-14418
> URL: https://issues.apache.org/jira/browse/HIVE-14418
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14418.01.patch, HIVE-14418.02.patch,
> HIVE-14418.03.patch, HIVE-14418.04.patch, HIVE-14418.patch
>
>
> {noformat}
> hive> set hive.tez.task.scale.memory.reserve.fraction.max=;
> Query returned non-zero code: 1, cause: 'SET
> hive.tez.task.scale.memory.reserve.fraction.max=' FAILED because
> hive.tez.task.scale.memory.reserve.fraction.max expects FLOAT type value.
> hive> set hive.tez.task.scale.memory.reserve.fraction.max=null;
> Query returned non-zero code: 1, cause: 'SET
> hive.tez.task.scale.memory.reserve.fraction.max=null' FAILED because
> hive.tez.task.scale.memory.reserve.fraction.max expects FLOAT type value.
> {noformat}
> unset also doesn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14536) Unit test code cleanup
[ https://issues.apache.org/jira/browse/HIVE-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14536: -- Attachment: HIVE-14536.6.patch Removed independent parts > Unit test code cleanup > -- > > Key: HIVE-14536 > URL: https://issues.apache.org/jira/browse/HIVE-14536 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14536.5.patch, HIVE-14536.6.patch, HIVE-14536.patch > > > Clean up the itest infrastructure, to create a readable, easy to understand > code -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438735#comment-15438735
]
Hive QA commented on HIVE-13930:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825524/HIVE-13930.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1002/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1002/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1002/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825524 - PreCommit-HIVE-MASTER-Build
> upgrade Hive to latest Hadoop version
> -
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch,
> HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[ https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438648#comment-15438648 ] Matt McCline commented on HIVE-14451: - Includes logic described by HIVE-14452. > Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow > -- > > Key: HIVE-14451 > URL: https://issues.apache.org/jira/browse/HIVE-14451 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-14451.01.patch > > > In a majority of cases, when using the OptimizedHashMap, the references to > the byte[] are immutable. > The hashmap result always allocates on boundary conditions, but never mutates > a previous buffer. > Copying Strings out of the hashtable is entirely wasteful and it would be > easy to know when the currentBytes is a borrowed slice from the original > input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[ https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438646#comment-15438646 ] Matt McCline commented on HIVE-14451: - Giving this a shot. Ran: mvn test -Dtest=TestVectorSerDeRow Tests probably need to add escaped strings. And the tests should call new deserializeByRef method. > Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow > -- > > Key: HIVE-14451 > URL: https://issues.apache.org/jira/browse/HIVE-14451 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-14451.01.patch > > > In a majority of cases, when using the OptimizedHashMap, the references to > the byte[] are immutable. > The hashmap result always allocates on boundary conditions, but never mutates > a previous buffer. > Copying Strings out of the hashtable is entirely wasteful and it would be > easy to know when the currentBytes is a borrowed slice from the original > input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[ https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14451: Status: Patch Available (was: Open) > Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow > -- > > Key: HIVE-14451 > URL: https://issues.apache.org/jira/browse/HIVE-14451 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-14451.01.patch > > > In a majority of cases, when using the OptimizedHashMap, the references to > the byte[] are immutable. > The hashmap result always allocates on boundary conditions, but never mutates > a previous buffer. > Copying Strings out of the hashtable is entirely wasteful and it would be > easy to know when the currentBytes is a borrowed slice from the original > input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[ https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14451: Attachment: HIVE-14451.01.patch > Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow > -- > > Key: HIVE-14451 > URL: https://issues.apache.org/jira/browse/HIVE-14451 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-14451.01.patch > > > In a majority of cases, when using the OptimizedHashMap, the references to > the byte[] are immutable. > The hashmap result always allocates on boundary conditions, but never mutates > a previous buffer. > Copying Strings out of the hashtable is entirely wasteful and it would be > easy to know when the currentBytes is a borrowed slice from the original > input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14451) Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow
[ https://issues.apache.org/jira/browse/HIVE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-14451: --- Assignee: Matt McCline > Vectorization: Add byRef mode for borrowed Strings in VectorDeserializeRow > -- > > Key: HIVE-14451 > URL: https://issues.apache.org/jira/browse/HIVE-14451 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Matt McCline > > In a majority of cases, when using the OptimizedHashMap, the references to > the byte[] are immutable. > The hashmap result always allocates on boundary conditions, but never mutates > a previous buffer. > Copying Strings out of the hashtable is entirely wasteful and it would be > easy to know when the currentBytes is a borrowed slice from the original > input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14614) Insert overwrite local directory fails with IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438640#comment-15438640
]
Hive QA commented on HIVE-14614:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825498/HIVE-14614.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1001/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1001/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1001/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825498 - PreCommit-HIVE-MASTER-Build
> Insert overwrite local directory fails with IllegalStateException
> -
>
> Key: HIVE-14614
> URL: https://issues.apache.org/jira/browse/HIVE-14614
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14614.2.patch, HIVE-14614.3.patch
>
>
> insert overwrite local directory select * from table; fails with
> "java.lang.IllegalStateException: Cannot create staging directory" when the
> path sent to the getTempDirForPath(Path path) is a local fs path.
> This is a regression caused by the fix for HIVE-14270
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438579#comment-15438579 ] Ke Jia commented on HIVE-13589: --- Hi [~vihangk1], [~Ferd], I upload the HIVE-13589.6.patch. I test it in beeline and hive cli mode respectively. And it can work well. Pending to the test. > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, HIVE-13589.6.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14619) CASE folding can produce wrong expression
[
https://issues.apache.org/jira/browse/HIVE-14619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14619:
---
Fix Version/s: 2.1.1
> CASE folding can produce wrong expression
> -
>
> Key: HIVE-14619
> URL: https://issues.apache.org/jira/browse/HIVE-14619
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14619.patch
>
>
> This is a case that was not covered in the testsuite. For the following query:
> {code:sql}
> select (CASE WHEN key = value THEN '1' WHEN true THEN '0' ELSE NULL END)
> from src
> {code}
> Currently, we end up folding the select expression to '0', as we fail bail
> out in the second statement and fail to account that there are two different
> possible values for the CASE expression ('1' and '0').
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14624) LLAP: Use FQDN for all communication
[
https://issues.apache.org/jira/browse/HIVE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438572#comment-15438572
]
Siddharth Seth commented on HIVE-14624:
---
Tried this out, along with a change in Tez. Worked much better. Will post a
patch for Hive, and a separate one for Tez soon.
> LLAP: Use FQDN for all communication
> -
>
> Key: HIVE-14624
> URL: https://issues.apache.org/jira/browse/HIVE-14624
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Gopal V
> Fix For: 2.2.0
>
>
> {code}
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java:
> + socketAddress.getHostName());
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java:
> host = socketAddress.getHostName();
> llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java:
> public static String getHostName() {
> llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java:
>return InetAddress.getLocalHost().getHostName();
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java:
> String name = address.getHostName();
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java:
> builder.setAmHost(address.getHostName());
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java:
>nodeId = LlapNodeId.getInstance(localAddress.get().getHostName(),
> localAddress.get().getPort());
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
> localAddress.get().getHostName(), vertex.getDagName(),
> qIdProto.getDagIndex(),
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
> new ExecutionContextImpl(localAddress.get().getHostName()), env,
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java:
>String hostName = MetricsUtils.getHostName();
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java:
> .setBindAddress(addr.getHostName())
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java:
> request.getContainerIdString(), executionContext.getHostName(),
> vertex.getDagName(),
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
>String displayName = "LlapDaemonCacheMetrics-" +
> MetricsUtils.getHostName();
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
>displayName = "LlapDaemonIOMetrics-" + MetricsUtils.getHostName();
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java:
> new LlapProtocolClientImpl(new Configuration(),
> serverAddr.getHostName(),
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java:
> builder.setAmHost(getAddress().getHostName());
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java:
> String displayName = "LlapTaskSchedulerMetrics-" +
> MetricsUtils.getHostName();
> {code}
> In systems where the hostnames do not match FQDN, calling the
> getCanonicalHostName() will allow for resolution of the hostname when
> accessing from a different base domain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14651: -- Attachment: HIVE-14651.02.patch Updated patch with slightly different directory handling. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch, HIVE-14651.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
