[jira] [Updated] (HIVE-12366) Refactor Heartbeater logic for transaction
[ https://issues.apache.org/jira/browse/HIVE-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12366: - Attachment: HIVE-12366.11.patch Upload patch 11 > Refactor Heartbeater logic for transaction > -- > > Key: HIVE-12366 > URL: https://issues.apache.org/jira/browse/HIVE-12366 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-12366.1.patch, HIVE-12366.11.patch, > HIVE-12366.2.patch, HIVE-12366.3.patch, HIVE-12366.4.patch, > HIVE-12366.5.patch, HIVE-12366.6.patch, HIVE-12366.7.patch, > HIVE-12366.8.patch, HIVE-12366.9.patch > > > Currently there is a gap between the time locks acquisition and the first > heartbeat being sent out. Normally the gap is negligible, but when it's big > it will cause query fail since the locks are timed out by the time the > heartbeat is sent. > Need to remove this gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061725#comment-15061725
]
Hive QA commented on HIVE-12685:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778134/HIVE-12685.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9949 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-authorization_create_table_owner_privs.q-create_func1.q-partition_wise_fileformat.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6378/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6378/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6378/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778134 - PreCommit-HIVE-TRUNK-Build
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10498) LLAP: Resolve everything in llap-daemon-site.xml
[ https://issues.apache.org/jira/browse/HIVE-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-10498: Component/s: llap > LLAP: Resolve everything in llap-daemon-site.xml > > > Key: HIVE-10498 > URL: https://issues.apache.org/jira/browse/HIVE-10498 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Gopal V >Assignee: Gopal V > Fix For: llap > > Attachments: HIVE-10498.patch > > > Configuring a sequence of hadoop execution parameters via llap-daemon-site.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10981) LLAP: Accept --hiveconf parameters for the LlapServiceDriver
[
https://issues.apache.org/jira/browse/HIVE-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-10981:
Component/s: llap
> LLAP: Accept --hiveconf parameters for the LlapServiceDriver
>
>
> Key: HIVE-10981
> URL: https://issues.apache.org/jira/browse/HIVE-10981
> Project: Hive
> Issue Type: Sub-task
> Components: CLI, llap
>Affects Versions: llap
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: llap
>
> Attachments: HIVE-10981.1.patch
>
>
> {code}
> Exception in thread "main" org.apache.commons.cli.UnrecognizedOptionException:
> Unrecognized option: --hiveconf
> at
> org.apache.commons.cli.Parser.processOption(Parser.java:363)
> at
> org.apache.commons.cli.Parser.parse(Parser.java:199)
> at
> org.apache.commons.cli.Parser.parse(Parser.java:85)
> at
>
> org.apache.hadoop.hive.llap.cli.LlapOptionsProcessor.processOptions(LlapOptionsProcessor.java:137)
> at
>
> org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:92)
> at
>
> org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:58)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10218) LLAP: Loglevel for daemons as a startup option
[
https://issues.apache.org/jira/browse/HIVE-10218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-10218:
Component/s: llap
> LLAP: Loglevel for daemons as a startup option
> --
>
> Key: HIVE-10218
> URL: https://issues.apache.org/jira/browse/HIVE-10218
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Trivial
> Fix For: llap
>
> Attachments: HIVE-10218.patch
>
>
> Accept {{hive --service llap --loglevel WARN}} as a startup option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10474) LLAP: investigate why TPCH Q1 1k is slow
[ https://issues.apache.org/jira/browse/HIVE-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-10474: Component/s: llap > LLAP: investigate why TPCH Q1 1k is slow > > > Key: HIVE-10474 > URL: https://issues.apache.org/jira/browse/HIVE-10474 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin > Attachments: llap-gc-pauses.png > > > While most queries run faster in LLAP than just Tez with container reuse, > TPCH Q1 is much slower. > On my run, on tez with container reuse (current default LLAP configuration > but mode == container and no daemons running) runs 2-6 (out of 6 consecutive > runs in the same session) finished in 25.5sec average; with 16 LLAP daemons > in default config the average was 35.5sec; same w/o IO elevator (to rule out > its impact) it took 59.7sec w/strange distribution (later runs were slower > than earlier runs, still, fastest run was 49.5sec). > So excluding IO elevator it's more than 2x degradation. > We need to figure out why this is happening. Is it just slot discrepancy? > Regardless, this needs to be addressed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12701) select on table with boolean as partition column shows wrong result
[ https://issues.apache.org/jira/browse/HIVE-12701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-12701: --- Assignee: Chinna Rao Lalam > select on table with boolean as partition column shows wrong result > --- > > Key: HIVE-12701 > URL: https://issues.apache.org/jira/browse/HIVE-12701 > Project: Hive > Issue Type: Bug > Components: Database/Schema, SQL >Affects Versions: 1.1.0 >Reporter: Sudipto Nandan >Assignee: Chinna Rao Lalam > > create table hive_aprm02ht7(a int, b int, c int) partitioned by (p boolean) > row format delimited fields terminated by ',' stored as textfile; > load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition > (p=true); > load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition > (p=false); > describe hive_aprm02ht7; > col_namedata_type comment > a int > b int > c int > p boolean > # Partition Information > # col_name data_type comment > p boolean > show partitions hive_aprm02ht7; > OK > p=false > p=true > Time taken: 0.057 seconds, Fetched: 2 row(s) > -- everything is shown as true. But first three should be true and the last > three rows should be false > hive> select * from hive_aprm02ht7 where p in (true,false); > OK > 1 2 3 true > 4 5 6 true > 7 8 9 true > 1 2 3 true > 4 5 6 true > 7 8 9 true > Time taken: 0.068 seconds, Fetched: 6 row(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12623) Add an option to force allocation of fragments on requested nodes
[ https://issues.apache.org/jira/browse/HIVE-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-12623: Component/s: llap > Add an option to force allocation of fragments on requested nodes > - > > Key: HIVE-12623 > URL: https://issues.apache.org/jira/browse/HIVE-12623 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12623.1.wip.txt > > > Currently, fragments are sent to random nodes if the requested node does not > have capacity. In certain situations there's more to be gained by sending the > fragments to the requested node only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11524) LLAP: tez.runtime.compress doesn't appear to be honored for LLAP
[
https://issues.apache.org/jira/browse/HIVE-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-11524:
Component/s: llap
> LLAP: tez.runtime.compress doesn't appear to be honored for LLAP
>
>
> Key: HIVE-11524
> URL: https://issues.apache.org/jira/browse/HIVE-11524
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
> Fix For: llap
>
>
> When running llap on an openstack cluster without snappy installed, with
> tez.runtime.compress set to false and codec set to snappy, one still gets the
> exceptions due to snappy codec being absent:
> {noformat}
> 2015-08-10 11:14:30,440
> [TezTaskRunner_attempt_1438943112941_0015_2_00_00_0(attempt_1438943112941_0015_2_00_00_0)]
> ERROR org.apache.hadoop.io.compress.snappy.SnappyCompressor: failed to load
> SnappyCompressor
> java.lang.NoSuchFieldError: clazz
> at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native
> Method)
> at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.(SnappyCompressor.java:57)
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:69)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.(IFile.java:153)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.(IFile.java:138)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter$SpillCallable.callInternal(UnorderedPartitionedKVWriter.java:406)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter$SpillCallable.callInternal(UnorderedPartitionedKVWriter.java:367)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.finalSpill(UnorderedPartitionedKVWriter.java:612)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.close(UnorderedPartitionedKVWriter.java:521)
> at
> org.apache.tez.runtime.library.output.UnorderedKVOutput.close(UnorderedKVOutput.java:128)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:376)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:79)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> When it's set to true, the client complains about snappy. When it's set to
> fails, the client doesn't complain but it still tries to use it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12696) LlapServiceDriver can fail if only the packaged logger config is present
[
https://issues.apache.org/jira/browse/HIVE-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-12696:
Component/s: llap
> LlapServiceDriver can fail if only the packaged logger config is present
>
>
> Key: HIVE-12696
> URL: https://issues.apache.org/jira/browse/HIVE-12696
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Sergey Shelukhin
>Priority: Minor
>
> I was incrementally updating my setup on some VM and didn't have the logger
> config file, so the packaged one was picked up apparently, which caused this:
> {noformat}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI:
> jar:file:/home/vagrant/llap/apache-hive-2.0.0-SNAPSHOT-bin/lib/hive-llap-server-2.0.0-SNAPSHOT.jar!/llap-daemon-log4j2.properties
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at
> org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:234)
> at
> org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:58)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> jar:file:/home/vagrant/llap/apache-hive-2.0.0-SNAPSHOT-bin/lib/hive-llap-server-2.0.0-SNAPSHOT.jar!/llap-daemon-log4j2.properties
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12699) LLAP: hive.llap.daemon.work.dirs setting backward compat name doesn't work
[ https://issues.apache.org/jira/browse/HIVE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-12699: Component/s: llap > LLAP: hive.llap.daemon.work.dirs setting backward compat name doesn't work > --- > > Key: HIVE-12699 > URL: https://issues.apache.org/jira/browse/HIVE-12699 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Trivial > Attachments: HIVE-12699.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-12685:
Component/s: Configuration
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12684) NPE in stats annotation when all values in decimal column are NULLs
[
https://issues.apache.org/jira/browse/HIVE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-12684:
Component/s: Statistics
> NPE in stats annotation when all values in decimal column are NULLs
> ---
>
> Key: HIVE-12684
> URL: https://issues.apache.org/jira/browse/HIVE-12684
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12684.1.patch, HIVE-12684.2.patch,
> HIVE-12684.3.patch, HIVE-12684.3.patch
>
>
> When all column values are null for a decimal column and when column stats
> exists. AnnotateWithStatistics optimization can throw NPE. Following is the
> exception trace
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:712)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:764)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:750)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:197)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:143)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:131)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:114)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:228)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10156)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:225)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12701) select on table with boolean as partition column shows wrong result
[
https://issues.apache.org/jira/browse/HIVE-12701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-12701:
Description:
{code:sql}
create table hive_aprm02ht7(a int, b int, c int) partitioned by (p boolean) row
format delimited fields terminated by ',' stored as textfile;
load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition
(p=true);
load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition
(p=false);
describe hive_aprm02ht7;
{code}
{noformat}
col_namedata_type comment
a int
b int
c int
p boolean
# Partition Information
# col_name data_type comment
p boolean
{noformat}
{code:sql}
show partitions hive_aprm02ht7;
{code}
{noformat}
OK
p=false
p=true
Time taken: 0.057 seconds, Fetched: 2 row(s)
{noformat}
-- everything is shown as true. But first three should be true and the last
three rows should be false
{noformat}
hive> select * from hive_aprm02ht7 where p in (true,false);
OK
1 2 3 true
4 5 6 true
7 8 9 true
1 2 3 true
4 5 6 true
7 8 9 true
Time taken: 0.068 seconds, Fetched: 6 row(s)
{noformat}
was:
create table hive_aprm02ht7(a int, b int, c int) partitioned by (p boolean) row
format delimited fields terminated by ',' stored as textfile;
load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition
(p=true);
load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition
(p=false);
describe hive_aprm02ht7;
col_namedata_type comment
a int
b int
c int
p boolean
# Partition Information
# col_name data_type comment
p boolean
show partitions hive_aprm02ht7;
OK
p=false
p=true
Time taken: 0.057 seconds, Fetched: 2 row(s)
-- everything is shown as true. But first three should be true and the last
three rows should be false
hive> select * from hive_aprm02ht7 where p in (true,false);
OK
1 2 3 true
4 5 6 true
7 8 9 true
1 2 3 true
4 5 6 true
7 8 9 true
Time taken: 0.068 seconds, Fetched: 6 row(s)
> select on table with boolean as partition column shows wrong result
> ---
>
> Key: HIVE-12701
> URL: https://issues.apache.org/jira/browse/HIVE-12701
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema, SQL
>Affects Versions: 1.1.0
>Reporter: Sudipto Nandan
>Assignee: Chinna Rao Lalam
>
> {code:sql}
> create table hive_aprm02ht7(a int, b int, c int) partitioned by (p boolean)
> row format delimited fields terminated by ',' stored as textfile;
> load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition
> (p=true);
> load data local inpath 'hive_data8.txt' into table hive_aprm02ht7 partition
> (p=false);
> describe hive_aprm02ht7;
> {code}
> {noformat}
> col_namedata_type comment
> a int
> b int
> c int
> p boolean
> # Partition Information
> # col_name data_type comment
> p boolean
> {noformat}
> {code:sql}
> show partitions hive_aprm02ht7;
> {code}
> {noformat}
> OK
> p=false
> p=true
> Time taken: 0.057 seconds, Fetched: 2 row(s)
> {noformat}
> -- everything is shown as true. But first three should be true and the last
> three rows should be false
> {noformat}
> hive> select * from hive_aprm02ht7 where p in (true,false);
> OK
> 1 2 3 true
> 4 5 6 true
> 7 8 9 true
> 1 2 3 true
> 4 5 6 true
> 7 8 9 true
> Time taken: 0.068 seconds, Fetched: 6 row(s)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12220) LLAP: Usability issues with hive.llap.io.cache.orc.size
[ https://issues.apache.org/jira/browse/HIVE-12220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-12220: Component/s: (was: Hive) llap > LLAP: Usability issues with hive.llap.io.cache.orc.size > --- > > Key: HIVE-12220 > URL: https://issues.apache.org/jira/browse/HIVE-12220 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Carter Shanklin >Assignee: Sergey Shelukhin > Attachments: HIVE-12220.01.patch, HIVE-12220.patch > > > In the llap-daemon site you need to set, among other things, > llap.daemon.memory.per.instance.mb > and > hive.llap.io.cache.orc.size > The use of hive.llap.io.cache.orc.size caused me some unnecessary problems, > initially I entered the value in MB rather than in bytes. Operator error you > could say but I look at this as a fraction of the other value which is in mb. > Second, is this really tied to ORC? E.g. when we have the vectorized text > reader will this data be cached as well? Or might it be in the future? > I would like to propose instead using hive.llap.io.cache.size.mb for this > setting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11245) LLAP: Fix the LLAP to ORC APIs
[ https://issues.apache.org/jira/browse/HIVE-11245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-11245: Component/s: llap > LLAP: Fix the LLAP to ORC APIs > -- > > Key: HIVE-11245 > URL: https://issues.apache.org/jira/browse/HIVE-11245 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Owen O'Malley >Assignee: Sergey Shelukhin >Priority: Blocker > Fix For: llap > > > Currently the LLAP branch has refactored the ORC code to have different code > paths depending on whether the data is coming from the cache or a FileSystem. > We need to introduce a concept of a DataSource that is responsible for > getting the necessary bytes regardless of whether they are coming from a > FileSystem, in memory cache, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9986) LLAP: EOFException in reader
[
https://issues.apache.org/jira/browse/HIVE-9986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-9986:
---
Component/s: llap
> LLAP: EOFException in reader
>
>
> Key: HIVE-9986
> URL: https://issues.apache.org/jira/browse/HIVE-9986
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Fix For: llap
>
>
> From HIVE-9979
> {noformat}
> 2015-03-16 10:20:51,439
> [pool-2-thread-3(container_1_1141_01_000192_gopal_20150316102020_c8c92488-6a61-401e-8298-401dace286dc:1_Map
> 1_191_0)] INFO org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl: Getting
> data for column 9 RG 112 stream DATA at 62278935, 1057137 index position 0:
> compressed [62614934, 63139228)
> 2015-03-16 10:20:51,439
> [pool-2-thread-6(container_1_1141_01_000211_gopal_20150316102020_c8c92488-6a61-401e-8298-401dace286dc:1_Map
> 1_210_0)] INFO org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl: Getting
> stripe-level stream [LENGTH, kind: DICTIONARY_V2
> dictionarySize: 3
> ] for column 9 RG 91 at 64139927, 5
> ...
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderUtils.readDirect(RecordReaderUtils.java:286)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:266)
> at
> org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:234)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:280)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:44)
> at
> org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10113) LLAP: reducers running in LLAP starve out map retries
[ https://issues.apache.org/jira/browse/HIVE-10113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-10113: Component/s: llap > LLAP: reducers running in LLAP starve out map retries > - > > Key: HIVE-10113 > URL: https://issues.apache.org/jira/browse/HIVE-10113 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Siddharth Seth > Fix For: llap > > > When query 17 is run, some mappers from Map 1 currently fail (due to unwrap > issue, and also due to HIVE-10112). > This query has 1000+ reducers; if they are ran in llap, they all queue up, > and the query locks up. > If only mappers run in LLAP, query completes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12695) LLAP: use somebody else's cluster
[ https://issues.apache.org/jira/browse/HIVE-12695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Carol updated HIVE-12695: Component/s: llap > LLAP: use somebody else's cluster > - > > Key: HIVE-12695 > URL: https://issues.apache.org/jira/browse/HIVE-12695 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12695.patch > > > For non-HS2 case cluster sharing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12702) insert overwrite doesn't remove the old file using lzo compresion
[ https://issues.apache.org/jira/browse/HIVE-12702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HIVE-12702: --- Environment: hadoop 2.6.0 + lzo +hive1.2.1 Description: when I insert overwrite derectory or table or partition , the old file doesn't remove. I have tested on hive0.14 and it works well. And my hadoop cluster using lzo comperssion > insert overwrite doesn't remove the old fileusing lzo compresion > -- > > Key: HIVE-12702 > URL: https://issues.apache.org/jira/browse/HIVE-12702 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: hadoop 2.6.0 + lzo +hive1.2.1 >Reporter: liuguanghua > > when I insert overwrite derectory or table or partition , the old file > doesn't remove. I have tested on hive0.14 and it works well. And my hadoop > cluster > using lzo comperssion -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12661) StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly
[
https://issues.apache.org/jira/browse/HIVE-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061833#comment-15061833
]
Hive QA commented on HIVE-12661:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778148/HIVE-12661.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 140 failed/errored test(s), 9965 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_many
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_spark4
[jira] [Updated] (HIVE-11355) Hive on tez: memory manager for sort buffers (input/output) and operators
[ https://issues.apache.org/jira/browse/HIVE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-11355: -- Attachment: HIVE-11355.8.patch > Hive on tez: memory manager for sort buffers (input/output) and operators > - > > Key: HIVE-11355 > URL: https://issues.apache.org/jira/browse/HIVE-11355 > Project: Hive > Issue Type: Improvement > Components: Tez >Affects Versions: 2.0.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-11355.1.patch, HIVE-11355.2.patch, > HIVE-11355.3.patch, HIVE-11355.4.patch, HIVE-11355.5.patch, > HIVE-11355.6.patch, HIVE-11355.7.patch, HIVE-11355.8.patch > > > We need to better manage the sort buffer allocations to ensure better > performance. Also, we need to provide configurations to certain operators to > stay within memory limits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12699) LLAP: hive.llap.daemon.work.dirs setting backward compat name doesn't work
[
https://issues.apache.org/jira/browse/HIVE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062107#comment-15062107
]
Hive QA commented on HIVE-12699:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778162/HIVE-12699.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 9950 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_decimal_round.q-cbo_windowing.q-tez_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6381/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6381/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6381/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778162 - PreCommit-HIVE-TRUNK-Build
> LLAP: hive.llap.daemon.work.dirs setting backward compat name doesn't work
> ---
>
> Key: HIVE-12699
> URL: https://issues.apache.org/jira/browse/HIVE-12699
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-12699.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12542) Create HiveRelFactories
[ https://issues.apache.org/jira/browse/HIVE-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062002#comment-15062002 ] Jesus Camacho Rodriguez commented on HIVE-12542: If you prefer, you can also pick up the code, integrate it there, and I can close this one as duplicate, no problem. > Create HiveRelFactories > --- > > Key: HIVE-12542 > URL: https://issues.apache.org/jira/browse/HIVE-12542 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12542.01.patch, HIVE-12542.02.patch, > HIVE-12542.patch > > > Calcite 1.5.0 introduced the use of RelFactories to create the operators. In > particular, RelFactories contains the factories for all the operators in the > system. Although we can still implement old rules by providing each > individual factory (the constructor is deprecated, but it won't be removed > till Calcite 2.0.0 is out), new rules will only provide constructors based on > RelFactories. Thus, we propose to migrate immediately to the new interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12698) Remove exposure to internal privilege and principal classes in HiveAuthorizer
[
https://issues.apache.org/jira/browse/HIVE-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062253#comment-15062253
]
Hive QA commented on HIVE-12698:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778179/HIVE-12698.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6383/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6383/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6383/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-llap-client ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-llap-client ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/llap-client/target/hive-llap-client-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-llap-client ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-llap-client ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/llap-client/target/hive-llap-client-2.1.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-llap-client/2.1.0-SNAPSHOT/hive-llap-client-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/llap-client/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-llap-client/2.1.0-SNAPSHOT/hive-llap-client-2.1.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Spark Remote Client 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-client ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
spark-client ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-client
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ spark-client ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ spark-client
---
[INFO] Compiling 28 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java
uses or overrides a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ spark-client ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[copy] Copying 16 files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] ---
[jira] [Commented] (HIVE-12684) NPE in stats annotation when all values in decimal column are NULLs
[
https://issues.apache.org/jira/browse/HIVE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062246#comment-15062246
]
Hive QA commented on HIVE-12684:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778173/HIVE-12684.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9965 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6382/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6382/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6382/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778173 - PreCommit-HIVE-TRUNK-Build
> NPE in stats annotation when all values in decimal column are NULLs
> ---
>
> Key: HIVE-12684
> URL: https://issues.apache.org/jira/browse/HIVE-12684
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12684.1.patch, HIVE-12684.2.patch,
> HIVE-12684.3.patch, HIVE-12684.3.patch
>
>
> When all column values are null for a decimal column and when column stats
> exists. AnnotateWithStatistics optimization can throw NPE. Following is the
> exception trace
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:712)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:764)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:750)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:197)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:143)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:131)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:114)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:228)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10156)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:225)
> at
>
[jira] [Commented] (HIVE-12699) LLAP: hive.llap.daemon.work.dirs setting backward compat name doesn't work
[ https://issues.apache.org/jira/browse/HIVE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062485#comment-15062485 ] Sergey Shelukhin commented on HIVE-12699: - Test failures are known or have socket timeouts in metastore > LLAP: hive.llap.daemon.work.dirs setting backward compat name doesn't work > --- > > Key: HIVE-12699 > URL: https://issues.apache.org/jira/browse/HIVE-12699 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Trivial > Attachments: HIVE-12699.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062524#comment-15062524
]
Wei Zheng commented on HIVE-12685:
--
Test failures are not related
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12542) Create HiveRelFactories
[ https://issues.apache.org/jira/browse/HIVE-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062539#comment-15062539 ] Pengcheng Xiong commented on HIVE-12542: patch LGTM +1. pending QA test. > Create HiveRelFactories > --- > > Key: HIVE-12542 > URL: https://issues.apache.org/jira/browse/HIVE-12542 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12542.01.patch, HIVE-12542.02.patch, > HIVE-12542.patch > > > Calcite 1.5.0 introduced the use of RelFactories to create the operators. In > particular, RelFactories contains the factories for all the operators in the > system. Although we can still implement old rules by providing each > individual factory (the constructor is deprecated, but it won't be removed > till Calcite 2.0.0 is out), new rules will only provide constructors based on > RelFactories. Thus, we propose to migrate immediately to the new interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11935) Access HiveMetaStoreClient.currentMetaVars should be synchronized
[
https://issues.apache.org/jira/browse/HIVE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062453#comment-15062453
]
Alan Gates commented on HIVE-11935:
---
+1, looks fine, though I wonder if the TestHCatClient failure is related.
Also, is it an issue that one thread is calling the client after another has
closed it?
> Access HiveMetaStoreClient.currentMetaVars should be synchronized
> -
>
> Key: HIVE-11935
> URL: https://issues.apache.org/jira/browse/HIVE-11935
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-11935.1.patch, HIVE-11935.2.patch
>
>
> We saw intermittent failure of the following stack:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.isCompatibleWith(HiveMetaStoreClient.java:287)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy9.isCompatibleWith(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:206)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.createHiveDB(BaseSemanticAnalyzer.java:205)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.(DDLSemanticAnalyzer.java:223)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:259)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy20.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:171)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
>
[jira] [Updated] (HIVE-12442) Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks
[ https://issues.apache.org/jira/browse/HIVE-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohit Dholakia updated HIVE-12442: -- Attachment: hive-12442.1.patch wip patch. > Refactor/repackage HiveServer2's Thrift code so that it can be used in the > tasks > > > Key: HIVE-12442 > URL: https://issues.apache.org/jira/browse/HIVE-12442 > Project: Hive > Issue Type: Sub-task >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Rohit Dholakia > Labels: hiveserver, thrift > Attachments: hive-12442.1.patch > > > For implementing HIVE-12427, the tasks will need to have knowledge of thrift > types from HS2's thrift API. This jira will look at the least invasive way to > do that. > https://reviews.apache.org/r/41379 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12590) Repeated UDAFs with literals can produce incorrect result
[
https://issues.apache.org/jira/browse/HIVE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062423#comment-15062423
]
Hive QA commented on HIVE-12590:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778186/HIVE-12590.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9961 tests
executed
*Failed tests:*
{noformat}
TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6384/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6384/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6384/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778186 - PreCommit-HIVE-TRUNK-Build
> Repeated UDAFs with literals can produce incorrect result
> -
>
> Key: HIVE-12590
> URL: https://issues.apache.org/jira/browse/HIVE-12590
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.0.1, 1.1.1, 1.2.1, 2.0.0
>Reporter: Laljo John Pullokkaran
>Assignee: Ashutosh Chauhan
>Priority: Critical
> Attachments: HIVE-12590.2.patch, HIVE-12590.3.patch, HIVE-12590.patch
>
>
> Repeated UDAF with literals could produce wrong result.
> This is not a common use case, nevertheless a bug.
> hive> select max('pants'), max('pANTS') from t1 group by key;
> Total MapReduce CPU Time Spent: 0 msec
> OK
> pANTS pANTS
> pANTS pANTS
> pANTS pANTS
> pANTS pANTS
> pANTS pANTS
> Time taken: 296.252 seconds, Fetched: 5 row(s)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12542) Create HiveRelFactories
[ https://issues.apache.org/jira/browse/HIVE-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062495#comment-15062495 ] Pengcheng Xiong commented on HIVE-12542: [~jcamachorodriguez], either way is ok for me. thx > Create HiveRelFactories > --- > > Key: HIVE-12542 > URL: https://issues.apache.org/jira/browse/HIVE-12542 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12542.01.patch, HIVE-12542.02.patch, > HIVE-12542.patch > > > Calcite 1.5.0 introduced the use of RelFactories to create the operators. In > particular, RelFactories contains the factories for all the operators in the > system. Although we can still implement old rules by providing each > individual factory (the constructor is deprecated, but it won't be removed > till Calcite 2.0.0 is out), new rules will only provide constructors based on > RelFactories. Thus, we propose to migrate immediately to the new interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12697) Remove deprecated post option from webhcat test files
[ https://issues.apache.org/jira/browse/HIVE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062591#comment-15062591 ] Eugene Koifman commented on HIVE-12697: --- +1 > Remove deprecated post option from webhcat test files > - > > Key: HIVE-12697 > URL: https://issues.apache.org/jira/browse/HIVE-12697 > Project: Hive > Issue Type: Test > Components: WebHCat >Affects Versions: 2.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Aswathy Chellammal Sreekumar > Labels: test > Attachments: HIVE-12697.1.patch > > > Tests are still having the deprecated post option user.name. Need to remove > them and add the same to query string -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12704) Hive default authorization does not allow cross-database create table
[
https://issues.apache.org/jira/browse/HIVE-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062625#comment-15062625
]
Thejas M Nair commented on HIVE-12704:
--
What is hive.security.authorization.manager set to ?
Seems like you are using the broken old authorization mode, the reccomended
alternative is
https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+based+hive+authorization
(for hiveserver2 use cases).
> Hive default authorization does not allow cross-database create table
> -
>
> Key: HIVE-12704
> URL: https://issues.apache.org/jira/browse/HIVE-12704
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.2.1
>Reporter: Jonathan Hseu
>Assignee: Thejas M Nair
>
> hive> use default;
> OK
> hive> create table jhseu.test_table1 (a int);
> Authorization failed:No privilege 'Create' found for outputs {
> database:default}. Use SHOW GRANT to get more details.
> 15/12/17 16:26:09 ERROR ql.Driver: Authorization failed:No privilege 'Create'
> found for outputs { database:default}. Use SHOW GRANT to get more details.
> hive> use jhseu;
> OK
> hive> create table jhseu.test_table1 (a int);
> OK
> Create table authorization checks against the table you're currently using,
> but it only should check against the destination.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12658) Task rejection by an llap daemon spams the log with RejectedExecutionExceptions
[ https://issues.apache.org/jira/browse/HIVE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12658: - Attachment: HIVE-12658.3.patch Reuploading for precommit test run > Task rejection by an llap daemon spams the log with > RejectedExecutionExceptions > --- > > Key: HIVE-12658 > URL: https://issues.apache.org/jira/browse/HIVE-12658 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Prasanth Jayachandran > Attachments: HIVE-12658.1.patch, HIVE-12658.2.patch, > HIVE-12658.3.patch, HIVE-12658.3.patch > > > The execution queue throws a RejectedExecutionException - which is logged by > the hadoop IPC layer. > Instead of relying on an Exception in the protocol - move to sending back an > explicit response to indicate a rejected fragment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12675) PerfLogger should log performance metrics at debug level
[ https://issues.apache.org/jira/browse/HIVE-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062694#comment-15062694 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-12675: -- The test results are clean and the failures are not related to the patch. > PerfLogger should log performance metrics at debug level > > > Key: HIVE-12675 > URL: https://issues.apache.org/jira/browse/HIVE-12675 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-12675.1.patch > > > As more and more subcomponents of Hive (Tez, Optimizer) etc are using > PerfLogger to track the performance metrics, it will be more meaningful to > set the PerfLogger logging level to DEBUG. Otherwise, we will print the > performance metrics unnecessarily for each and every query if the underlying > subcomponent does not control the PerfLogging via a parameter on its own. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12682) Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
[
https://issues.apache.org/jira/browse/HIVE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12682:
-
Attachment: HIVE-12682.1.patch
> Reducers in dynamic partitioning job spend a lot of time running
> hadoop.conf.Configuration.getOverlay
> -
>
> Key: HIVE-12682
> URL: https://issues.apache.org/jira/browse/HIVE-12682
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Carter Shanklin
>Assignee: Gopal V
> Attachments: HIVE-12682.1.patch, reducer.png
>
>
> I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
> I ran this query:
> {code}
> create table flights (
> …
> )
> PARTITIONED BY (Year int)
> CLUSTERED BY (Month)
> SORTED BY (DayofMonth) into 12 buckets
> STORED AS ORC
> TBLPROPERTIES("orc.bloom.filter.columns"="*")
> ;
> {code}
> (Taken from here:
> https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
> I profiled just the reduce phase and noticed something odd, the attached
> graph shows where time was spent during the reducer phase.
> !reducer.png!
> Problem seems to relate to
> https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
> /cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12685:
-
Attachment: HIVE-12685.4.patch
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12684) NPE in stats annotation when all values in decimal column are NULLs
[
https://issues.apache.org/jira/browse/HIVE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12684:
-
Attachment: (was: HIVE-12684.3.patch)
> NPE in stats annotation when all values in decimal column are NULLs
> ---
>
> Key: HIVE-12684
> URL: https://issues.apache.org/jira/browse/HIVE-12684
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12684.1.patch, HIVE-12684.2.patch
>
>
> When all column values are null for a decimal column and when column stats
> exists. AnnotateWithStatistics optimization can throw NPE. Following is the
> exception trace
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:712)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:764)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:750)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:197)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:143)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:131)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:114)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:228)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10156)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:225)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12684) NPE in stats annotation when all values in decimal column are NULLs
[
https://issues.apache.org/jira/browse/HIVE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12684:
-
Attachment: (was: HIVE-12684.3.patch)
> NPE in stats annotation when all values in decimal column are NULLs
> ---
>
> Key: HIVE-12684
> URL: https://issues.apache.org/jira/browse/HIVE-12684
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12684.1.patch, HIVE-12684.2.patch
>
>
> When all column values are null for a decimal column and when column stats
> exists. AnnotateWithStatistics optimization can throw NPE. Following is the
> exception trace
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:712)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:764)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:750)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:197)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:143)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:131)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:114)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:228)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10156)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:225)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12684) NPE in stats annotation when all values in decimal column are NULLs
[
https://issues.apache.org/jira/browse/HIVE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12684:
-
Attachment: HIVE-12684.3.patch
I uploaded the wrong patch with type fix before.
> NPE in stats annotation when all values in decimal column are NULLs
> ---
>
> Key: HIVE-12684
> URL: https://issues.apache.org/jira/browse/HIVE-12684
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12684.1.patch, HIVE-12684.2.patch,
> HIVE-12684.3.patch
>
>
> When all column values are null for a decimal column and when column stats
> exists. AnnotateWithStatistics optimization can throw NPE. Following is the
> exception trace
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:712)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:764)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:750)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:197)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:143)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:131)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:114)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143)
> at
> org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:228)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10156)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:225)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062703#comment-15062703
]
Mohit Sabharwal commented on HIVE-12685:
Thanks, [~wzheng]
I think "javax.jdo.option.ConnectionDriverName" cannot be used instead of
"test.property1", because that testcase is about using a property only defined
in hive-site.xml. But "javax.jdo.option.ConnectionDriverName" is defined in
ConfVar, no ?
Looks like there are three testcases:
1) Override a property defined in ConfVar in data/conf/hive-site.xml:
We can choose a dummy property defined in ConfVar like:
{code}
// ConfVar overridden in in hive-site.xml
checkHadoopConf(ConfVars.HIVETESTMODEDUMMYSTATAGGR.varname, null);
checkConfVar(ConfVars.HIVETESTMODEDUMMYSTATAGGR, "");
checkHiveConf(ConfVars.HIVETESTMODEDUMMYSTATAGGR.varname, "test");
{code}
And we can override it in data/conf/hive-site.xml:
{code}
+
+ hive.test.dummystats.aggregator
+ test
+
{code}
2) Property only defined in hive-site (but not in ConfVar or hadoop conf):
{code}
// Property defined in hive-site.xml only
checkHadoopConf("test.property1", null);
checkHiveConf("test.property1", "hive-site.xml");
{code}
For this, we can move test.property1 from
common/src/test/resources/hive-site.xml to data/conf/hive-site.xml
3) Test HiveConf property variable substitution in hive-site.xml
For this, we can also move test.var.hiveconf.property from
common/src/test/resources/hive-site.xml to data/conf/hive-site.xml as you did
in your latest patch.
Let me know if this makes sense or I'm missing something.
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-7449) from_utc_timestamp and to_utc_timestamp not returning right values
[
https://issues.apache.org/jira/browse/HIVE-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere resolved HIVE-7449.
--
Resolution: Not A Problem
> from_utc_timestamp and to_utc_timestamp not returning right values
> --
>
> Key: HIVE-7449
> URL: https://issues.apache.org/jira/browse/HIVE-7449
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0, 0.12.0
>Reporter: Mala Chikka Kempanna
>
> Tested it out like below
> 1.Get UTC and local time
> date -u
> Fri Jul 18 18:15:29 UTC 2014
> Local time
> date
> Fri Jul 18 11:15:32 PDT 2014
> 2. Conversion output on HIVE 0.10.0 and HIVE 0.12.0
> From UTC to local
> SELECT from_utc_timestamp('2014-07-18 18:15:29.0','PDT') FROM sample_07 LIMIT
> 1;
> 2014-07-18 18:15:29 - Wrong output , it is supposed to be 2014-07-18 11:15:32
> From local to utc
> SELECT to_utc_timestamp('2014-07-18 11:15:32.0','PDT') FROM sample_07 LIMIT
> 1;
> 2014-07-18 11:15:32 - Wrong output , it is supposed to be 2014-07-18 18:15:29
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11487) Add getNumPartitionsByFilter api in metastore api
[
https://issues.apache.org/jira/browse/HIVE-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062764#comment-15062764
]
Hive QA commented on HIVE-11487:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778188/HIVE-11487.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9949 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_partition_diff_num_cols.q-tez_joins_explain.q-vector_decimal_aggregate.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6385/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6385/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6385/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778188 - PreCommit-HIVE-TRUNK-Build
> Add getNumPartitionsByFilter api in metastore api
> -
>
> Key: HIVE-11487
> URL: https://issues.apache.org/jira/browse/HIVE-11487
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Akshay Goyal
> Attachments: HIVE-11487.01.patch, HIVE-11487.02.patch,
> HIVE-11487.03.patch, HIVE-11487.04.patch
>
>
> Adding api for getting number of partitions for a filter will be more optimal
> when we are only interested in the number. getAllPartitions will construct
> all the partition object which can be time consuming and not required.
> Here is a commit we pushed in a forked repo in our organization -
> https://github.com/inmobi/hive/commit/68b3534d3e6c4d978132043cec668798ed53e444.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062565#comment-15062565
]
Mohit Sabharwal commented on HIVE-12685:
Thanks, [~wzheng]. Couple of comments:
- Better to reference ConfVars.METASTORE_CONNECTION_DRIVER.varname than use
the literal directly.
- Seems like the patch removes some unit test functionality in TestHiveConf.
{code}
// Property defined in hive-site.xml only
-checkHadoopConf("test.property1", null);
-checkHiveConf("test.property1", "hive-site.xml");
{code}
Should we keep the test & move these properties from
common/src/test/resources/hive-site.xml to data/conf/hive-site.xml ?
Also, looks like patch is no longer testing this scenario
{code}
-// Test HiveConf property variable substitution in hive-site.xml
{code}
Perhaps instead of using METASTORE_CONNECTION_DRIVER property for testing, we
can use a dummy property like HIVETESTMODEDUMMYSTATAGGR, so that it doesn't
affect other unit tests?
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12697) Remove deprecated post option from webhcat test files
[ https://issues.apache.org/jira/browse/HIVE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12697: -- Description: Tests are still having the deprecated post option user.name. Need to remove them and add the same to query string Submitting user.name as form parameter in POST method was deprecated in Hive 0.13 was:Tests are still having the deprecated post option user.name. Need to remove them and add the same to query string > Remove deprecated post option from webhcat test files > - > > Key: HIVE-12697 > URL: https://issues.apache.org/jira/browse/HIVE-12697 > Project: Hive > Issue Type: Test > Components: WebHCat >Affects Versions: 2.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Aswathy Chellammal Sreekumar > Labels: test > Attachments: HIVE-12697.1.patch > > > Tests are still having the deprecated post option user.name. Need to remove > them and add the same to query string > Submitting user.name as form parameter in POST method was deprecated in Hive > 0.13 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12682) Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
[
https://issues.apache.org/jira/browse/HIVE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062775#comment-15062775
]
Prasanth Jayachandran commented on HIVE-12682:
--
I see that taskId is already stored in member variable during initialization.
We should use that instead of getting it from conf. There are other places too
that gets taskid from conf object. I will upload a patch shortly to use the
member variable in inner loop.
> Reducers in dynamic partitioning job spend a lot of time running
> hadoop.conf.Configuration.getOverlay
> -
>
> Key: HIVE-12682
> URL: https://issues.apache.org/jira/browse/HIVE-12682
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Carter Shanklin
>Assignee: Gopal V
> Attachments: reducer.png
>
>
> I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
> I ran this query:
> {code}
> create table flights (
> …
> )
> PARTITIONED BY (Year int)
> CLUSTERED BY (Month)
> SORTED BY (DayofMonth) into 12 buckets
> STORED AS ORC
> TBLPROPERTIES("orc.bloom.filter.columns"="*")
> ;
> {code}
> (Taken from here:
> https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
> I profiled just the reduce phase and noticed something odd, the attached
> graph shows where time was spent during the reducer phase.
> !reducer.png!
> Problem seems to relate to
> https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
> /cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12064) prevent transactional=false
[ https://issues.apache.org/jira/browse/HIVE-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062866#comment-15062866 ] Sushanth Sowmyan commented on HIVE-12064: - That's interesting. It would suggest that MetaExceptions are being caught in the case of RemoteMetaStore, and not being translated back to actual errors. That might be indicative of another deeper bug and definitely bears deeper looking into. > prevent transactional=false > --- > > Key: HIVE-12064 > URL: https://issues.apache.org/jira/browse/HIVE-12064 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-12064.2.patch, HIVE-12064.patch > > > currently a tblproperty transactional=true must be set to make a table behave > in ACID compliant way. > This is misleading in that it seems like changing it to transactional=false > makes the table non-acid but on disk layout of acid table is different than > plain tables. So changing this property may cause wrong data to be returned. > Should prevent transactional=false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: HIVE-11775.11.patch > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12366) Refactor Heartbeater logic for transaction
[
https://issues.apache.org/jira/browse/HIVE-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062947#comment-15062947
]
Hive QA commented on HIVE-12366:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778215/HIVE-12366.11.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1543 failed/errored test(s), 9764 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_constprog_dpp
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_bmj_schema_evolution
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_insert_overwrite_local_directory_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_multi_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_schema_evolution
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_self_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_dynamic_partition
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062975#comment-15062975
]
Ashutosh Chauhan commented on HIVE-12685:
-
+1 LGTM
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch, HIVE-12685.5.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12682) Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
[
https://issues.apache.org/jira/browse/HIVE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-12682:
---
Assignee: Prasanth Jayachandran (was: Gopal V)
> Reducers in dynamic partitioning job spend a lot of time running
> hadoop.conf.Configuration.getOverlay
> -
>
> Key: HIVE-12682
> URL: https://issues.apache.org/jira/browse/HIVE-12682
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Carter Shanklin
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12682.1.patch, reducer.png
>
>
> I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
> I ran this query:
> {code}
> create table flights (
> …
> )
> PARTITIONED BY (Year int)
> CLUSTERED BY (Month)
> SORTED BY (DayofMonth) into 12 buckets
> STORED AS ORC
> TBLPROPERTIES("orc.bloom.filter.columns"="*")
> ;
> {code}
> (Taken from here:
> https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
> I profiled just the reduce phase and noticed something odd, the attached
> graph shows where time was spent during the reducer phase.
> !reducer.png!
> Problem seems to relate to
> https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
> /cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062891#comment-15062891
]
Mohit Sabharwal commented on HIVE-12685:
Thanks, [~wzheng]! LGTM. +1(non-binding) pending test.
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch, HIVE-12685.5.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12697) Remove deprecated post option from webhcat test files
[ https://issues.apache.org/jira/browse/HIVE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062941#comment-15062941 ] Aswathy Chellammal Sreekumar commented on HIVE-12697: - One test got missed from the patch, please find attached updated patch. > Remove deprecated post option from webhcat test files > - > > Key: HIVE-12697 > URL: https://issues.apache.org/jira/browse/HIVE-12697 > Project: Hive > Issue Type: Test > Components: WebHCat >Affects Versions: 2.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Aswathy Chellammal Sreekumar > Labels: test > Attachments: HIVE-12697.1.patch > > > Tests are still having the deprecated post option user.name. Need to remove > them and add the same to query string > Submitting user.name as form parameter in POST method was deprecated in Hive > 0.13 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12675) PerfLogger should log performance metrics at debug level
[ https://issues.apache.org/jira/browse/HIVE-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062960#comment-15062960 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-12675: -- https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-HiveLogging has the new documentation. > PerfLogger should log performance metrics at debug level > > > Key: HIVE-12675 > URL: https://issues.apache.org/jira/browse/HIVE-12675 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.1.0 > > Attachments: HIVE-12675.1.patch > > > As more and more subcomponents of Hive (Tez, Optimizer) etc are using > PerfLogger to track the performance metrics, it will be more meaningful to > set the PerfLogger logging level to DEBUG. Otherwise, we will print the > performance metrics unnecessarily for each and every query if the underlying > subcomponent does not control the PerfLogging via a parameter on its own. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12685:
-
Attachment: HIVE-12685.5.patch
OK, those are good points. Please take a look at patch 5.
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch, HIVE-12685.5.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12623) Add an option to force allocation of fragments on requested nodes
[ https://issues.apache.org/jira/browse/HIVE-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12623: -- Attachment: HIVE-12623.1.txt Patch with unit tests, and some fixes identified by the tests. cc [~gopalv], [~sershe], [~rajesh.balamohan] > Add an option to force allocation of fragments on requested nodes > - > > Key: HIVE-12623 > URL: https://issues.apache.org/jira/browse/HIVE-12623 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12623.1.txt, HIVE-12623.1.wip.txt > > > Currently, fragments are sent to random nodes if the requested node does not > have capacity. In certain situations there's more to be gained by sending the > fragments to the requested node only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12623) Add an option to force allocation of fragments on requested nodes
[ https://issues.apache.org/jira/browse/HIVE-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062936#comment-15062936 ] Siddharth Seth commented on HIVE-12623: --- Tried creating a RB link - that isn't working. Please review without - unless someone can successfully upload the patch to rb. > Add an option to force allocation of fragments on requested nodes > - > > Key: HIVE-12623 > URL: https://issues.apache.org/jira/browse/HIVE-12623 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12623.1.txt, HIVE-12623.1.wip.txt > > > Currently, fragments are sent to random nodes if the requested node does not > have capacity. In certain situations there's more to be gained by sending the > fragments to the requested node only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12697) Remove deprecated post option from webhcat test files
[ https://issues.apache.org/jira/browse/HIVE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aswathy Chellammal Sreekumar updated HIVE-12697: Attachment: HIVE-12697.2.patch > Remove deprecated post option from webhcat test files > - > > Key: HIVE-12697 > URL: https://issues.apache.org/jira/browse/HIVE-12697 > Project: Hive > Issue Type: Test > Components: WebHCat >Affects Versions: 2.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Aswathy Chellammal Sreekumar > Labels: test > Attachments: HIVE-12697.1.patch, HIVE-12697.2.patch > > > Tests are still having the deprecated post option user.name. Need to remove > them and add the same to query string > Submitting user.name as form parameter in POST method was deprecated in Hive > 0.13 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12678) BETWEEN relational operator sometimes returns incorrect results against PARQUET tables
[
https://issues.apache.org/jira/browse/HIVE-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063501#comment-15063501
]
Xuefu Zhang commented on HIVE-12678:
cc: [~spena]
> BETWEEN relational operator sometimes returns incorrect results against
> PARQUET tables
> --
>
> Key: HIVE-12678
> URL: https://issues.apache.org/jira/browse/HIVE-12678
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 1.2.1
>Reporter: Nicholas Brenwald
>
> When querying a parquet table, the BETWEEN relational operator returns
> incorrect results when hive.optimize.index.filter and
> hive.optimize.ppd.storage are enabled
> Create a parquet table:
> {code}
> create table t(c string) stored as parquet;
> {code}
> Insert some strings representing dates
> {code}
> insert into t select '2015-12-09' from default.dual limit 1;
> insert into t select '2015-12-10' from default.dual limit 1;
> insert into t select '2015-12-11' from default.dual limit 1;
> {code}
> h3. Example 1
> This query correctly returns 3:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-09' and c <= '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> This query incorrectly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> Disabling hive.optimize.findex.filter resolves the problem. This query now
> correctly returns 3:
> {code}
> set hive.optimize.index.filter=false;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> Disabling hive.optimize.ppd.storage resolves the problem. This query now
> correctly returns 3:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=false;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> h3. Example 2
> This query correctly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> This query incorrectly returns 0:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-10' and '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 0|
> +--+--+
> {code}
> Disabling hive.optimize.findex.filter resolves the problem. This query now
> correctly returns 1:
> {code}
> set hive.optimize.index.filter=false;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> Disabling hive.optimize.ppd.storage resolves the problem. This query now
> correctly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=false;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12698) Remove exposure to internal privilege and principal classes in HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12698: - Attachment: HIVE-12698.4.patch HIVE-12698.4.patch - made a minor change so that the interface does not require the new class being introduced. That way implementations like Ranger don't have to create a shim layer just for this, to be able to compile against older and newer versions of hive. Alternatives to this would have necessiated creation of an extended interface and then checking if the authorizer implementation implements the extended interface as well. That would increase the complexity of the hive side of the code. Also updated in the git pull request. [~Ferd] Can you please take a look at this minor change ? > Remove exposure to internal privilege and principal classes in HiveAuthorizer > - > > Key: HIVE-12698 > URL: https://issues.apache.org/jira/browse/HIVE-12698 > Project: Hive > Issue Type: Bug > Components: Authorization >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12698.1.patch, HIVE-12698.2.patch, > HIVE-12698.3.patch, HIVE-12698.4.patch > > > The changes in HIVE-11179 expose several internal classes to > HiveAuthorization implementations. These include PrivilegeObjectDesc, > PrivilegeDesc, PrincipalDesc and AuthorizationUtils. > We should avoid exposing that to all Authorization implementations, but also > make the ability to customize the mapping of internal classes to the public > api classes possible for Apache Sentry (incubating). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063654#comment-15063654
]
Hive QA commented on HIVE-12685:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778345/HIVE-12685.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 9965 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6392/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6392/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6392/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778345 - PreCommit-HIVE-TRUNK-Build
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch, HIVE-12685.5.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12542) Create HiveRelFactories
[
https://issues.apache.org/jira/browse/HIVE-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063528#comment-15063528
]
Hive QA commented on HIVE-12542:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778244/HIVE-12542.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 9915 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-bucketsortoptimize_insert_7.q-index_bitmap_rc.q-constprog_dpp.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-bucket3.q-vectorization_7.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6389/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6389/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6389/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778244 - PreCommit-HIVE-TRUNK-Build
> Create HiveRelFactories
> ---
>
> Key: HIVE-12542
> URL: https://issues.apache.org/jira/browse/HIVE-12542
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12542.01.patch, HIVE-12542.02.patch,
> HIVE-12542.patch
>
>
> Calcite 1.5.0 introduced the use of RelFactories to create the operators. In
> particular, RelFactories contains the factories for all the operators in the
> system. Although we can still implement old rules by providing each
> individual factory (the constructor is deprecated, but it won't be removed
> till Calcite 2.0.0 is out), new rules will only provide constructors based on
> RelFactories. Thus, we propose to migrate immediately to the new interface.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12698) Remove exposure to internal privilege and principal classes in HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063583#comment-15063583 ] Ferdinand Xu commented on HIVE-12698: - +1 for the .4 patch pending to the test > Remove exposure to internal privilege and principal classes in HiveAuthorizer > - > > Key: HIVE-12698 > URL: https://issues.apache.org/jira/browse/HIVE-12698 > Project: Hive > Issue Type: Bug > Components: Authorization >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12698.1.patch, HIVE-12698.2.patch, > HIVE-12698.3.patch, HIVE-12698.4.patch > > > The changes in HIVE-11179 expose several internal classes to > HiveAuthorization implementations. These include PrivilegeObjectDesc, > PrivilegeDesc, PrincipalDesc and AuthorizationUtils. > We should avoid exposing that to all Authorization implementations, but also > make the ability to customize the mapping of internal classes to the public > api classes possible for Apache Sentry (incubating). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063628#comment-15063628 ] Takanobu Asanuma commented on HIVE-11527: - I updated the patch in Review Board. Please review it when you're free. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12688: - Fix Version/s: 2.1.0 2.0.0 1.3.0 > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Fix For: 1.3.0, 2.0.0, 2.1.0 > > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12632) LLAP: don't use IO elevator for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12632: Priority: Blocker (was: Major) > LLAP: don't use IO elevator for ACID tables > > > Key: HIVE-12632 > URL: https://issues.apache.org/jira/browse/HIVE-12632 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12632.01.patch, HIVE-12632.patch > > > Until HIVE-12631 is fixed, we need to avoid ACID tables in IO elevator. Right > now, a FileNotFound error is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12661) StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly
[
https://issues.apache.org/jira/browse/HIVE-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-12661:
---
Attachment: HIVE-12661.05.patch
> StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly
> ---
>
> Key: HIVE-12661
> URL: https://issues.apache.org/jira/browse/HIVE-12661
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12661.01.patch, HIVE-12661.02.patch,
> HIVE-12661.03.patch, HIVE-12661.04.patch, HIVE-12661.05.patch
>
>
> PROBLEM:
> Hive stats are autogathered properly till an 'analyze table [tablename]
> compute statistics for columns' is run. Then it does not auto-update the
> stats till the command is run again. repo:
> {code}
> set hive.stats.autogather=true;
> set hive.stats.atomic=false ;
> set hive.stats.collect.rawdatasize=true ;
> set hive.stats.collect.scancols=false ;
> set hive.stats.collect.tablekeys=false ;
> set hive.stats.fetch.column.stats=true;
> set hive.stats.fetch.partition.stats=true ;
> set hive.stats.reliable=false ;
> set hive.compute.query.using.stats=true;
> CREATE TABLE `default`.`calendar` (`year` int) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' TBLPROPERTIES (
> 'orc.compress'='NONE') ;
> insert into calendar values (2010), (2011), (2012);
> select * from calendar;
> ++--+
> | calendar.year |
> ++--+
> | 2010 |
> | 2011 |
> | 2012 |
> ++--+
> select max(year) from calendar;
> | 2012 |
> insert into calendar values (2013);
> select * from calendar;
> ++--+
> | calendar.year |
> ++--+
> | 2010 |
> | 2011 |
> | 2012 |
> | 2013 |
> ++--+
> select max(year) from calendar;
> | 2013 |
> insert into calendar values (2014);
> select max(year) from calendar;
> | 2014 |
> analyze table calendar compute statistics for columns;
> insert into calendar values (2015);
> select max(year) from calendar;
> | 2014 |
> insert into calendar values (2016), (2017), (2018);
> select max(year) from calendar;
> | 2014 |
> analyze table calendar compute statistics for columns;
> select max(year) from calendar;
> | 2018 |
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063018#comment-15063018 ] Laljo John Pullokkaran commented on HIVE-11775: --- +1 conditional on clean QA run based on latest patch. > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated HIVE-12619: - Attachment: HIVE-12619.1.patch Uploading for pre-commit build > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.1.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12573) some DPP tests are broken
[ https://issues.apache.org/jira/browse/HIVE-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12573: Assignee: Gunther Hagleitner (was: Sergey Shelukhin) > some DPP tests are broken > - > > Key: HIVE-12573 > URL: https://issues.apache.org/jira/browse/HIVE-12573 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Gunther Hagleitner >Priority: Blocker > Attachments: HIVE-12573.patch > > > -It looks like LLAP out files were not updated in some DPP JIRA because the > test was entirely broken in HiveQA at the time- actually looks like out files > have explain output with a glitch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: HIVE-11775.12.patch > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12366) Refactor Heartbeater logic for transaction
[ https://issues.apache.org/jira/browse/HIVE-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12366: - Attachment: HIVE-12366.12.patch Patch 12, to resolve a DummyTxnManager#releaseLocks NPE issue > Refactor Heartbeater logic for transaction > -- > > Key: HIVE-12366 > URL: https://issues.apache.org/jira/browse/HIVE-12366 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-12366.1.patch, HIVE-12366.11.patch, > HIVE-12366.12.patch, HIVE-12366.2.patch, HIVE-12366.3.patch, > HIVE-12366.4.patch, HIVE-12366.5.patch, HIVE-12366.6.patch, > HIVE-12366.7.patch, HIVE-12366.8.patch, HIVE-12366.9.patch > > > Currently there is a gap between the time locks acquisition and the first > heartbeat being sent out. Normally the gap is negligible, but when it's big > it will cause query fail since the locks are timed out by the time the > heartbeat is sent. > Need to remove this gap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12698) Remove exposure to internal privilege and principal classes in HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12698: - Attachment: HIVE-12698.3.patch HIVE-12698.3.patch - addressing more review comments > Remove exposure to internal privilege and principal classes in HiveAuthorizer > - > > Key: HIVE-12698 > URL: https://issues.apache.org/jira/browse/HIVE-12698 > Project: Hive > Issue Type: Bug > Components: Authorization >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12698.1.patch, HIVE-12698.2.patch, > HIVE-12698.3.patch > > > The changes in HIVE-11179 expose several internal classes to > HiveAuthorization implementations. These include PrivilegeObjectDesc, > PrivilegeDesc, PrincipalDesc and AuthorizationUtils. > We should avoid exposing that to all Authorization implementations, but also > make the ability to customize the mapping of internal classes to the public > api classes possible for Apache Sentry (incubating). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12542) Create HiveRelFactories
[ https://issues.apache.org/jira/browse/HIVE-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061983#comment-15061983 ] Jesus Camacho Rodriguez commented on HIVE-12542: [~pxiong], I saw that HIVE-11927 introduces HiveRelFactories too, but only some of them. We could check this patch in before, as it includes all the factories in HiveRelFactories. What do you think? Thanks > Create HiveRelFactories > --- > > Key: HIVE-12542 > URL: https://issues.apache.org/jira/browse/HIVE-12542 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12542.01.patch, HIVE-12542.patch > > > Calcite 1.5.0 introduced the use of RelFactories to create the operators. In > particular, RelFactories contains the factories for all the operators in the > system. Although we can still implement old rules by providing each > individual factory (the constructor is deprecated, but it won't be removed > till Calcite 2.0.0 is out), new rules will only provide constructors based on > RelFactories. Thus, we propose to migrate immediately to the new interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12702) insert overwrite doesn't remove the old file
[ https://issues.apache.org/jira/browse/HIVE-12702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HIVE-12702: --- Description: when I insert overwrite derectory or table or partition , the old file doesn't remove. I have tested on hive0.14 and it works well. Now I found HIVE-12505.And I think the two are the same problem.You can add the patch for HIVE-12505 to slove this problem. And there is a fast way to reslove using the old hive configuration . Add this property to hive-site.xml hive.exec.stagingdir /tmp/hive was: when I insert overwrite derectory or table or partition , the old file doesn't remove. I have tested on hive0.14 and it works well. And my hadoop cluster using lzo comperssion Summary: insert overwrite doesn't remove the old file (was: insert overwrite doesn't remove the old fileusing lzo compresion) > insert overwrite doesn't remove the old file > -- > > Key: HIVE-12702 > URL: https://issues.apache.org/jira/browse/HIVE-12702 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 > Environment: hadoop 2.6.0 + lzo +hive1.2.1 >Reporter: liuguanghua > > when I insert overwrite derectory or table or partition , the old file > doesn't remove. I have tested on hive0.14 and it works well. > Now I found HIVE-12505.And I think the two are the same problem.You can add > the patch for HIVE-12505 to slove this problem. > And there is a fast way to reslove using the old hive configuration . > Add this property to hive-site.xml > >hive.exec.stagingdir >/tmp/hive > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11487) Add getNumPartitionsByFilter api in metastore api
[ https://issues.apache.org/jira/browse/HIVE-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063403#comment-15063403 ] Amareshwari Sriramadasu commented on HIVE-11487: +1 for the patch. Test failures are not related to the patch and they are being seen in other builds and master branch as well. > Add getNumPartitionsByFilter api in metastore api > - > > Key: HIVE-11487 > URL: https://issues.apache.org/jira/browse/HIVE-11487 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-11487.01.patch, HIVE-11487.02.patch, > HIVE-11487.03.patch, HIVE-11487.04.patch > > > Adding api for getting number of partitions for a filter will be more optimal > when we are only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > Here is a commit we pushed in a forked repo in our organization - > https://github.com/inmobi/hive/commit/68b3534d3e6c4d978132043cec668798ed53e444. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12541) SymbolicTextInputFormat should supports the path with regex
[
https://issues.apache.org/jira/browse/HIVE-12541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061928#comment-15061928
]
Xiaowei Wang commented on HIVE-12541:
-
I checked several times ,all these failed test case cannot reappear in my
local test environment . I do not really understand .
> SymbolicTextInputFormat should supports the path with regex
> ---
>
> Key: HIVE-12541
> URL: https://issues.apache.org/jira/browse/HIVE-12541
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 0.14.0, 1.2.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Xiaowei Wang
> Fix For: 1.2.1
>
> Attachments: HIVE-12541.1.patch, HIVE-12541.2.patch,
> HIVE-12541.3.patch, HIVE-12541.4.patch
>
>
> 1, In fact,SybolicTextInputFormat supports the path with regex .I add some
> test sql .
> 2, But ,when using CombineHiveInputFormat to combine input files , It cannot
> resolve the path with regex ,so it will get a wrong result.I give a example
> ,and fix the problem.
> Table desc :
> {noformat}
> CREATE External TABLE `symlink_text_input_format`(
> `key` string,
> `value` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'viewfs://nsX/user/hive/warehouse/symlink_text_input_format'
> {noformat}
> There is a link file in the dir
> '/user/hive/warehouse/symlink_text_input_format' , the content of the link
> file is
> {noformat}
> viewfs://nsx/tmp/symlink*
> {noformat}
> it contains one path ,and the path contains a regex!
> Execute the sql :
> {noformat}
> set hive.rework.mapredwork = true ;
> set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
> set mapred.min.split.size.per.rack= 0 ;
> set mapred.min.split.size.per.node= 0 ;
> set mapred.max.split.size= 0 ;
> select count(*) from symlink_text_input_format ;
> {noformat}
> It will get a wrong result :0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11927) Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants
[
https://issues.apache.org/jira/browse/HIVE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061942#comment-15061942
]
Hive QA commented on HIVE-11927:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778154/HIVE-11927.14.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 9966 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6380/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6380/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6380/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778154 - PreCommit-HIVE-TRUNK-Build
> Implement/Enable constant related optimization rules in Calcite: enable
> HiveReduceExpressionsRule to fold constants
> ---
>
> Key: HIVE-11927
> URL: https://issues.apache.org/jira/browse/HIVE-11927
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11927.01.patch, HIVE-11927.02.patch,
> HIVE-11927.03.patch, HIVE-11927.04.patch, HIVE-11927.05.patch,
> HIVE-11927.06.patch, HIVE-11927.07.patch, HIVE-11927.08.patch,
> HIVE-11927.09.patch, HIVE-11927.10.patch, HIVE-11927.11.patch,
> HIVE-11927.12.patch, HIVE-11927.13.patch, HIVE-11927.14.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11865: --- Attachment: HIVE-11865.04.patch > Disable Hive PPD optimizer when CBO has optimized the plan > -- > > Key: HIVE-11865 > URL: https://issues.apache.org/jira/browse/HIVE-11865 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch, > HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch, > HIVE-11865.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12697) Remove deprecated post option from webhcat test files
[ https://issues.apache.org/jira/browse/HIVE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12697: -- Target Version/s: 1.3.0, 2.0.0 (was: 2.0.0) > Remove deprecated post option from webhcat test files > - > > Key: HIVE-12697 > URL: https://issues.apache.org/jira/browse/HIVE-12697 > Project: Hive > Issue Type: Test > Components: WebHCat >Affects Versions: 2.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Aswathy Chellammal Sreekumar > Labels: test > Attachments: HIVE-12697.1.patch > > > Tests are still having the deprecated post option user.name. Need to remove > them and add the same to query string -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15062620#comment-15062620
]
Wei Zheng commented on HIVE-12685:
--
[~mohitsabharwal] Thanks for the feedback.
The code below is doing the same thing but serve for different purposes, one to
uses the varname, the other uses the literal.
{code}
// ConfVar overridden in in hive-site.xml
checkHadoopConf(ConfVars.METASTORE_CONNECTION_DRIVER.varname, null);
checkConfVar(ConfVars.METASTORE_CONNECTION_DRIVER,
"org.apache.derby.jdbc.EmbeddedDriver");
// Property defined in hive-site.xml only
checkHadoopConf("javax.jdo.option.ConnectionDriverName", null);
checkHiveConf("javax.jdo.option.ConnectionDriverName",
"org.apache.derby.jdbc.EmbeddedDriver");
{code}
So basically the second part replaces "test.property1".
I agree with you for the last part being removed, I've added that dummy
property to data/conf/hive-site.xml.
Kindly take a look at patch 4.
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated HIVE-12619: - Attachment: (was: HIVE-12619.1.patch) > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.1.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated HIVE-12619: - Attachment: HIVE-12619.1.patch > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.1.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12605) Implement JDBC Connection.isValid
[
https://issues.apache.org/jira/browse/HIVE-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063306#comment-15063306
]
Gabor Liptak commented on HIVE-12605:
-
[~thejas] Would this be a good approach? Thanks
> Implement JDBC Connection.isValid
> -
>
> Key: HIVE-12605
> URL: https://issues.apache.org/jira/browse/HIVE-12605
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Reporter: Thejas M Nair
> Labels: newbie, trivial
> Attachments: HIVE-12605.1.patch
>
>
> http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)
> implementation in Hive JDBC driver throws "SQLException("Method not
> supported")".
> That is a method often used by connection pooling libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12605) Implement JDBC Connection.isValid
[
https://issues.apache.org/jira/browse/HIVE-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Liptak updated HIVE-12605:
Attachment: HIVE-12605.1.patch
> Implement JDBC Connection.isValid
> -
>
> Key: HIVE-12605
> URL: https://issues.apache.org/jira/browse/HIVE-12605
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Reporter: Thejas M Nair
> Labels: newbie, trivial
> Attachments: HIVE-12605.1.patch
>
>
> http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)
> implementation in Hive JDBC driver throws "SQLException("Method not
> supported")".
> That is a method often used by connection pooling libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11531) Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise
[
https://issues.apache.org/jira/browse/HIVE-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063320#comment-15063320
]
Hui Zheng commented on HIVE-11531:
--
Hi [~sershe] and [~prasanth_j]
The union9 is good in my local machine.Could you give more details about how to
reproduce it?
{code}
mvn test -Dtest=TestCliDriver -Dqfile=union9.q,offset_limit.q
..
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-qfile-spark ---
[INFO] Compiling 3 source files to
/Users/huzheng/git/hive/itests/qtest-spark/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-it-qfile-spark
---
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Hive Integration - Parent . SUCCESS [1.191s]
[INFO] Hive Integration - Custom Serde ... SUCCESS [1.955s]
[INFO] Hive Integration - HCatalog Unit Tests SUCCESS [2.974s]
[INFO] Hive Integration - Testing Utilities .. SUCCESS [2.312s]
[INFO] Hive Integration - Unit Tests . SUCCESS [4.672s]
[INFO] Hive Integration - Test Serde . SUCCESS [0.326s]
[INFO] Hive Integration - QFile Tests SUCCESS [1:35.067s]
[INFO] Hive Integration - QFile Accumulo Tests ... SUCCESS [2.523s]
[INFO] JMH benchmark: Hive ... SUCCESS [0.399s]
[INFO] Hive Integration - Unit Tests - Hadoop 2 .. SUCCESS [1.358s]
[INFO] Hive Integration - Unit Tests with miniKdc SUCCESS [1.407s]
[INFO] Hive Integration - QFile Spark Tests .. SUCCESS [3.650s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 1:58.710s
[INFO] Finished at: Fri Dec 18 11:18:28 JST 2015
[INFO] Final Memory: 164M/875M
[INFO]
{code}
> Add mysql-style LIMIT support to Hive, or improve ROW_NUMBER performance-wise
> -
>
> Key: HIVE-11531
> URL: https://issues.apache.org/jira/browse/HIVE-11531
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Hui Zheng
> Fix For: 2.1.0
>
> Attachments: HIVE-11531.02.patch, HIVE-11531.03.patch,
> HIVE-11531.04.patch, HIVE-11531.05.patch, HIVE-11531.06.patch,
> HIVE-11531.07.patch, HIVE-11531.WIP.1.patch, HIVE-11531.WIP.2.patch,
> HIVE-11531.patch
>
>
> For any UIs that involve pagination, it is useful to issue queries in the
> form SELECT ... LIMIT X,Y where X,Y are coordinates inside the result to be
> paginated (which can be extremely large by itself). At present, ROW_NUMBER
> can be used to achieve this effect, but optimizations for LIMIT such as TopN
> in ReduceSink do not apply to ROW_NUMBER. We can add first class support for
> "skip" to existing limit, or improve ROW_NUMBER for better performance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[
https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063329#comment-15063329
]
Hive QA commented on HIVE-11865:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778236/HIVE-11865.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9950 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6388/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6388/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6388/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778236 - PreCommit-HIVE-TRUNK-Build
> Disable Hive PPD optimizer when CBO has optimized the plan
> --
>
> Key: HIVE-11865
> URL: https://issues.apache.org/jira/browse/HIVE-11865
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch,
> HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch,
> HIVE-11865.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12695) LLAP: use somebody else's cluster
[ https://issues.apache.org/jira/browse/HIVE-12695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12695: Reporter: Takahiko Saito (was: Sergey Shelukhin) > LLAP: use somebody else's cluster > - > > Key: HIVE-12695 > URL: https://issues.apache.org/jira/browse/HIVE-12695 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12695.patch > > > For non-HS2 case cluster sharing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12695) LLAP: use somebody else's cluster
[ https://issues.apache.org/jira/browse/HIVE-12695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063182#comment-15063182 ] Sergey Shelukhin commented on HIVE-12695: - Update: the patch works (at least on an unsecure cluster, my secure setup is fubar). [~gopalv] ping > LLAP: use somebody else's cluster > - > > Key: HIVE-12695 > URL: https://issues.apache.org/jira/browse/HIVE-12695 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12695.patch > > > For non-HS2 case cluster sharing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12632) LLAP: don't use IO elevator for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063163#comment-15063163 ] Sergey Shelukhin commented on HIVE-12632: - [~prasanth_j] can you review? posted an RB. This causes incorrect results, so it's a blocker. PS. The first revision on RB is some random unrelated patch. The 2nd revision has the proper patch. > LLAP: don't use IO elevator for ACID tables > > > Key: HIVE-12632 > URL: https://issues.apache.org/jira/browse/HIVE-12632 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12632.01.patch, HIVE-12632.02.patch, > HIVE-12632.patch > > > Until HIVE-12631 is fixed, we need to avoid ACID tables in IO elevator. Right > now, a FileNotFound error is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12632) LLAP: don't use IO elevator for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12632: Attachment: HIVE-12632.02.patch > LLAP: don't use IO elevator for ACID tables > > > Key: HIVE-12632 > URL: https://issues.apache.org/jira/browse/HIVE-12632 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12632.01.patch, HIVE-12632.02.patch, > HIVE-12632.patch > > > Until HIVE-12631 is fixed, we need to avoid ACID tables in IO elevator. Right > now, a FileNotFound error is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12698) Remove exposure to internal privilege and principal classes in HiveAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063169#comment-15063169 ] Ferdinand Xu commented on HIVE-12698: - +1 pending to the test > Remove exposure to internal privilege and principal classes in HiveAuthorizer > - > > Key: HIVE-12698 > URL: https://issues.apache.org/jira/browse/HIVE-12698 > Project: Hive > Issue Type: Bug > Components: Authorization >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12698.1.patch, HIVE-12698.2.patch, > HIVE-12698.3.patch > > > The changes in HIVE-11179 expose several internal classes to > HiveAuthorization implementations. These include PrivilegeObjectDesc, > PrivilegeDesc, PrincipalDesc and AuthorizationUtils. > We should avoid exposing that to all Authorization implementations, but also > make the ability to customize the mapping of internal classes to the public > api classes possible for Apache Sentry (incubating). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12528) don't start HS2 Tez sessions in a single thread
[ https://issues.apache.org/jira/browse/HIVE-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063170#comment-15063170 ] Sergey Shelukhin commented on HIVE-12528: - [~sseth] were the above your only concerns? > don't start HS2 Tez sessions in a single thread > --- > > Key: HIVE-12528 > URL: https://issues.apache.org/jira/browse/HIVE-12528 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12528.01.patch, HIVE-12528.patch > > > Starting sessions in parallel would improve the startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12605) Implement JDBC Connection.isValid
[
https://issues.apache.org/jira/browse/HIVE-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063421#comment-15063421
]
Thejas M Nair commented on HIVE-12605:
--
[~gliptak] I think that's a great approach!
Can you also please add unit tests ?
> Implement JDBC Connection.isValid
> -
>
> Key: HIVE-12605
> URL: https://issues.apache.org/jira/browse/HIVE-12605
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Reporter: Thejas M Nair
> Labels: newbie, trivial
> Attachments: HIVE-12605.1.patch
>
>
> http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)
> implementation in Hive JDBC driver throws "SQLException("Method not
> supported")".
> That is a method often used by connection pooling libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12632) LLAP: don't use IO elevator for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063433#comment-15063433 ] Prasanth Jayachandran commented on HIVE-12632: -- Minor comment about the explain output in RB. Other than that looks good to me. +1 > LLAP: don't use IO elevator for ACID tables > > > Key: HIVE-12632 > URL: https://issues.apache.org/jira/browse/HIVE-12632 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12632.01.patch, HIVE-12632.02.patch, > HIVE-12632.patch > > > Until HIVE-12631 is fixed, we need to avoid ACID tables in IO elevator. Right > now, a FileNotFound error is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12678) BETWEEN relational operator sometimes returns incorrect results against PARQUET tables
[
https://issues.apache.org/jira/browse/HIVE-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063444#comment-15063444
]
Thejas M Nair commented on HIVE-12678:
--
FYI, I tried this with ORC format and it works fine. It seems to be Parquet
specific issue.
> BETWEEN relational operator sometimes returns incorrect results against
> PARQUET tables
> --
>
> Key: HIVE-12678
> URL: https://issues.apache.org/jira/browse/HIVE-12678
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 1.2.1
>Reporter: Nicholas Brenwald
>
> When querying a parquet table, the BETWEEN relational operator returns
> incorrect results when hive.optimize.index.filter and
> hive.optimize.ppd.storage are enabled
> Create a parquet table:
> {code}
> create table t(c string) stored as parquet;
> {code}
> Insert some strings representing dates
> {code}
> insert into t select '2015-12-09' from default.dual limit 1;
> insert into t select '2015-12-10' from default.dual limit 1;
> insert into t select '2015-12-11' from default.dual limit 1;
> {code}
> h3. Example 1
> This query correctly returns 3:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-09' and c <= '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> This query incorrectly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> Disabling hive.optimize.findex.filter resolves the problem. This query now
> correctly returns 3:
> {code}
> set hive.optimize.index.filter=false;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> Disabling hive.optimize.ppd.storage resolves the problem. This query now
> correctly returns 3:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=false;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> h3. Example 2
> This query correctly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> This query incorrectly returns 0:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-10' and '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 0|
> +--+--+
> {code}
> Disabling hive.optimize.findex.filter resolves the problem. This query now
> correctly returns 1:
> {code}
> set hive.optimize.index.filter=false;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> Disabling hive.optimize.ppd.storage resolves the problem. This query now
> correctly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=false;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12678) BETWEEN relational operator sometimes returns incorrect results against PARQUET tables
[
https://issues.apache.org/jira/browse/HIVE-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063454#comment-15063454
]
Thejas M Nair commented on HIVE-12678:
--
Would you like to look into this [~Ferd] [~dongc] [~xuefuz] ?
> BETWEEN relational operator sometimes returns incorrect results against
> PARQUET tables
> --
>
> Key: HIVE-12678
> URL: https://issues.apache.org/jira/browse/HIVE-12678
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 1.2.1
>Reporter: Nicholas Brenwald
>
> When querying a parquet table, the BETWEEN relational operator returns
> incorrect results when hive.optimize.index.filter and
> hive.optimize.ppd.storage are enabled
> Create a parquet table:
> {code}
> create table t(c string) stored as parquet;
> {code}
> Insert some strings representing dates
> {code}
> insert into t select '2015-12-09' from default.dual limit 1;
> insert into t select '2015-12-10' from default.dual limit 1;
> insert into t select '2015-12-11' from default.dual limit 1;
> {code}
> h3. Example 1
> This query correctly returns 3:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-09' and c <= '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> This query incorrectly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> Disabling hive.optimize.findex.filter resolves the problem. This query now
> correctly returns 3:
> {code}
> set hive.optimize.index.filter=false;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> Disabling hive.optimize.ppd.storage resolves the problem. This query now
> correctly returns 3:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=false;
> select count(*) from t where c between '2015-12-09' and '2015-12-11';
> +--+--+
> | _c0 |
> +--+--+
> | 3|
> +--+--+
> {code}
> h3. Example 2
> This query correctly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> This query incorrectly returns 0:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c between '2015-12-10' and '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 0|
> +--+--+
> {code}
> Disabling hive.optimize.findex.filter resolves the problem. This query now
> correctly returns 1:
> {code}
> set hive.optimize.index.filter=false;
> set hive.optimize.ppd.storage=true;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
> Disabling hive.optimize.ppd.storage resolves the problem. This query now
> correctly returns 1:
> {code}
> set hive.optimize.index.filter=true;
> set hive.optimize.ppd.storage=false;
> select count(*) from t where c >= '2015-12-10' and c <= '2015-12-10';
> +--+--+
> | _c0 |
> +--+--+
> | 1|
> +--+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12704) Hive default authorization does not allow cross-database create table
[
https://issues.apache.org/jira/browse/HIVE-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-12704:
-
Assignee: (was: Thejas M Nair)
> Hive default authorization does not allow cross-database create table
> -
>
> Key: HIVE-12704
> URL: https://issues.apache.org/jira/browse/HIVE-12704
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.2.1
>Reporter: Jonathan Hseu
>
> hive> use default;
> OK
> hive> create table jhseu.test_table1 (a int);
> Authorization failed:No privilege 'Create' found for outputs {
> database:default}. Use SHOW GRANT to get more details.
> 15/12/17 16:26:09 ERROR ql.Driver: Authorization failed:No privilege 'Create'
> found for outputs { database:default}. Use SHOW GRANT to get more details.
> hive> use jhseu;
> OK
> hive> create table jhseu.test_table1 (a int);
> OK
> Create table authorization checks against the table you're currently using,
> but it only should check against the destination.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
