[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344170#comment-16344170 ] Sergey Shelukhin commented on HIVE-18575: - Hmm, it took a while to post an RB > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344172#comment-16344172 ] Sergey Shelukhin commented on HIVE-18575: - [~ekoifman] well, to avoid future ambiguity I'm making sure it's named explicitly and that properties must always be set (which they were anyway in all but one place). > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18426) Memory leak in RoutingAppender for every hive operation
[ https://issues.apache.org/jira/browse/HIVE-18426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-18426: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master branch. Thanks [~kkalyan] > Memory leak in RoutingAppender for every hive operation > --- > > Key: HIVE-18426 > URL: https://issues.apache.org/jira/browse/HIVE-18426 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18426.001.patch, HIVE-18426.002.patch, > HIVE-18426.003.patch, HIVE-18426.004.patch, HIVE-18426.005.patch, > HIVE-18426.006.patch, Screen Shot 2018-01-10 at 9.17.19 AM.png > > > Each new operation creates new entry in the ConcurrentMap in RoutingAppender > but when the operation ends, AppenderControl stored in the map is retrieved > and stopped but the entry in ConcurrentMap is never cleaned up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344223#comment-16344223
]
Hive QA commented on HIVE-18573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908211/HIVE-18573.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 223 failed/errored test(s), 12793 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=240)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_7]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ba_table_udfs]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_gby2_map_multi_distinct]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_percentile2]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udf_percentile]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_udf1] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[concat_op] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_3]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantfolding]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_colname]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_udf] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_udf] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic2]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[extract] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[floor_time] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_multi_distinct]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_skew]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew_multi_distinct]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_map_ppr]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_map_ppr_multi_distinct]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr_multi_distinct]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert0] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insertoverwrite_bucket]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_2] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_alt] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_udf]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[localtimezone]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[macro_1] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_rewrite_ssb_2]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch_threshold]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_dictionary_threshold]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ctas]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_join]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_11]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_12]
(batchId=23)
org.apa
[jira] [Commented] (HIVE-18574) LLAP: Ship netty3 as part of LLAP install tarball
[
https://issues.apache.org/jira/browse/HIVE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344303#comment-16344303
]
Hive QA commented on HIVE-18574:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 90f26bb |
| Default Java | 1.8.0_111 |
| modules | C: llap-server . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8914/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Ship netty3 as part of LLAP install tarball
> -
>
> Key: HIVE-18574
> URL: https://issues.apache.org/jira/browse/HIVE-18574
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18574.1.patch
>
>
> Removing netty from Tez libs causes
> {code}
> 2018-01-29T18:28:49,995 ERROR [main ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP
> Daemon with exception
> java.lang.NoClassDefFoundError: org/jboss/netty/channel/group/ChannelGroup
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStart(LlapDaemon.java:410)
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18536: Attachment: HIVE-18536.01.patch > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18574) LLAP: Ship netty3 as part of LLAP install tarball
[
https://issues.apache.org/jira/browse/HIVE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344344#comment-16344344
]
Hive QA commented on HIVE-18574:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908214/HIVE-18574.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12794 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag[0]
(batchId=193)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerSlowQueryExecutionTime
(batchId=235)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalLaunchedTasks
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8914/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8914/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8914/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908214 - PreCommit-HIVE-Build
> LLAP: Ship netty3 as part of LLAP install tarball
> -
>
> Key: HIVE-18574
> URL: https://issues.apache.org/jira/browse/HIVE-18574
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18574.1.patch
>
>
> Removing netty from Tez libs causes
> {code}
> 2018-01-29T18:28:49,995 ERROR [main ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP
> Daemon with exception
> java.lang.NoClassDefFoundError: org/jboss/netty/channel/group/ChannelGroup
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStart(LlapDaemon.java:410)
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344345#comment-16344345 ] Sergey Shelukhin commented on HIVE-18536: - Fixed the comment and whitespace. I think the reason things are parsed in JavaUtils/etc. is that they were/are accessed in places like metastore that don't have access to QL. Is there reason AcidUtils cannot be moved mostly to common, for all the properties, coniguration, etc. stuff? > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Support Struct type with vectorization for Parquet file
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344350#comment-16344350 ] Colin Ma commented on HIVE-18048: - [~Ferd], the patch is updated as your comments and fix the related tests. The failed tests are not patch related. > Support Struct type with vectorization for Parquet file > --- > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344354#comment-16344354 ] Eugene Koifman commented on HIVE-18536: --- I imagine AcidUtils can just as well be in common > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18576) Support to read nested complex type with Parquet in vectorization mode
[ https://issues.apache.org/jira/browse/HIVE-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma reassigned HIVE-18576: --- > Support to read nested complex type with Parquet in vectorization mode > -- > > Key: HIVE-18576 > URL: https://issues.apache.org/jira/browse/HIVE-18576 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > > Nested complex type is common used, eg: Struct, s2 > List>. Currently, nested complex type can't be parsed in vectorization > mode, this ticket is target to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18552) Split hive.strict.checks.large.query into two configs, enable partition filter check by default
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344366#comment-16344366
]
Sahil Takiar commented on HIVE-18552:
-
[~sershe] sounds reasonable. Didn't realize there was an existing config to
control the max # of partitions to scan. Will post a patch that splits
{{hive.strict.checks.large.query}} and keeps both the new ones false by default.
> Split hive.strict.checks.large.query into two configs, enable partition
> filter check by default
> ---
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[
https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344375#comment-16344375
]
Hive QA commented on HIVE-18575:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 7 new + 1205 unchanged - 9
fixed = 1212 total (was 1214) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch llap-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hcatalog/streaming: The patch generated 0 new + 258
unchanged - 2 fixed = 258 total (was 260) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 90f26bb |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8915/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql llap-server hcatalog/core hcatalog/streaming
itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8915/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID properties usage in jobconf is ambiguous for MM tables
> ---
>
> Key: HIVE-18575
> URL: https://issues.apache.org/jira/browse/HIVE-18575
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18575.patch
>
>
> Vectorization checks for ACID table trigger for MM tables where they don't
> apply. Other places seem to set th
[jira] [Commented] (HIVE-18237) missing results for insert_only table after DP insert
[
https://issues.apache.org/jira/browse/HIVE-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344377#comment-16344377
]
Sergey Shelukhin commented on HIVE-18237:
-
+1
This will be modified partially by HIVE-18536, I can rebase later
> missing results for insert_only table after DP insert
> -
>
> Key: HIVE-18237
> URL: https://issues.apache.org/jira/browse/HIVE-18237
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Zoltan Haindrich
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18237.01.patch, HIVE-18237.02.patch,
> HIVE-18237.03.patch, HIVE-18237.04.patch
>
>
> {code}
> set hive.stats.column.autogather=false;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.max.dynamic.partitions.pernode=200;
> set hive.exec.max.dynamic.partitions=200;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table i0 (p int,v int);
> insert into i0 values
> (0,0),
> (2,2),
> (3,3);
> create table p0 (v int) partitioned by (p int) stored as orc
> tblproperties ("transactional"="true",
> "transactional_properties"="insert_only");
> explain insert overwrite table p0 partition (p) select * from i0 where v < 3;
> insert overwrite table p0 partition (p) select * from i0 where v < 3;
> select count(*) from p0 where v!=1;
> {code}
> The table p0 should contain {{2}} rows at this point; but the result is {{0}}.
> * seems to be specific to insert_only tables
> * the existing data appears if an {{insert into}} is executed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Summary: Split hive.strict.checks.large.query into two configs (was: Split
hive.strict.checks.large.query into two configs, enable partition filter check
by default)
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Status: Patch Available (was: Open)
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Attachment: HIVE-18552.1.patch
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17552) Enable bucket map join by default
[ https://issues.apache.org/jira/browse/HIVE-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-17552: -- Attachment: HIVE-17552.7.patch > Enable bucket map join by default > - > > Key: HIVE-17552 > URL: https://issues.apache.org/jira/browse/HIVE-17552 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-17552.1.patch, HIVE-17552.2.patch, > HIVE-17552.3.patch, HIVE-17552.4.patch, HIVE-17552.5.patch, > HIVE-17552.6.patch, HIVE-17552.7.patch > > > Currently bucket map join is disabled by default, however, it is potentially > most optimal join we have. Need to enable it by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[
https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344414#comment-16344414
]
Hive QA commented on HIVE-18575:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908220/HIVE-18575.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12794 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=104)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8915/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8915/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8915/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908220 - PreCommit-HIVE-Build
> ACID properties usage in jobconf is ambiguous for MM tables
> ---
>
> Key: HIVE-18575
> URL: https://issues.apache.org/jira/browse/HIVE-18575
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18575.patch
>
>
> Vectorization checks for ACID table trigger for MM tables where they don't
> apply. Other places seem to set the setting for transactional case while most
> of the code seems to assume it implies full acid.
> Overall, many places in the code use the settings directly or set the ACID
> flag without setting the ACID properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[
https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344418#comment-16344418
]
Hive QA commented on HIVE-18516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908221/HIVE-18516.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8916/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8916/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8916/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-30 02:31:32.455
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8916/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-30 02:31:32.458
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
455b2cc..6dbec04 branch-2 -> origin/branch-2
+ git reset --hard HEAD
HEAD is now at 90f26bb HIVE-18426: Memory leak in RoutingAppender for every
hive operation (kalyan kumar kalvagadda, reviewed by Aihua Xu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 90f26bb HIVE-18426: Memory leak in RoutingAppender for every
hive operation (kalyan kumar kalvagadda, reviewed by Aihua Xu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-30 02:31:35.708
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8916/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/itests/src/test/resources/testconfiguration.properties: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
does not exist in index
error: a/ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java:
does not exist in index
error: a/ql/src/test/queries/clientpositive/smb_mapjoin_7.q: does not exist in
index
error: a/ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/smb_mapjoin_7.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out: does not
exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: protoc version: 250, detected platform: linux/amd64
protoc-jar: executing: [/tmp/protoc7530939041038954533.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logg
[jira] [Commented] (HIVE-18449) Add configurable policy for choosing the HMS URI from hive.metastore.uris
[
https://issues.apache.org/jira/browse/HIVE-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344422#comment-16344422
]
Vihang Karajgaonkar commented on HIVE-18449:
Thanks for the patch [~janulatha] I left a comment on the RB. Can you please
take a look? Thanks
> Add configurable policy for choosing the HMS URI from hive.metastore.uris
> -
>
> Key: HIVE-18449
> URL: https://issues.apache.org/jira/browse/HIVE-18449
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-18449.1.patch
>
>
> HIVE-10815 added logic to randomly choose a HMS URI from
> {{hive.metastore.uris}}. It would be nice if there was a configurable policy
> that determined how a URI is chosen from this list - e.g. one option can be
> to randomly pick a URI, another option can be to choose the first URI in the
> list (which was the behavior prior to HIVE-10815).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[ https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18577: --- > SemanticAnalyzer.validate has some pointless metastore calls > > > Key: HIVE-18577 > URL: https://issues.apache.org/jira/browse/HIVE-18577 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[
https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344429#comment-16344429
]
Hive QA commented on HIVE-18536:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} common: The patch generated 1 new + 6 unchanged - 1
fixed = 7 total (was 7) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 3 new + 359 unchanged - 3
fixed = 362 total (was 362) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 90f26bb |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8917/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8917/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8917/yetus/whitespace-eol.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8917/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> IOW + DP is broken for insert-only ACID
> ---
>
> Key: HIVE-18536
> URL: https://issues.apache.org/jira/browse/HIVE-18536
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18536.01.patch, HIVE-18536.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[ https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18577: Attachment: HIVE-18577.patch > SemanticAnalyzer.validate has some pointless metastore calls > > > Key: HIVE-18577 > URL: https://issues.apache.org/jira/browse/HIVE-18577 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18577.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[ https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18577: Status: Patch Available (was: Open) [~ashutoshc] can you take a look? these metastore calls look completely useless to me cause the code just above seems to access partition objects from the WriteEntity just fine. > SemanticAnalyzer.validate has some pointless metastore calls > > > Key: HIVE-18577 > URL: https://issues.apache.org/jira/browse/HIVE-18577 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18577.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Summary: Vectorization: Support Struct type with vectorization (was: Support Struct type with vectorization for Parquet file) > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Issue Type: Improvement (was: Sub-task) Parent: (was: HIVE-14826) > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang reassigned HIVE-18578: --- > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18578: Attachment: HIVE-18578.1.patch > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-18578.1.patch > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18578: Attachment: (was: HIVE-18578.1.patch) > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18578: Attachment: HIVE-18578.1.patch > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-18578.1.patch > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18578: Status: Patch Available (was: Open) > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-18578.1.patch > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18579) Changes from HIVE-18495 introduced import paths from shaded jars
[
https://issues.apache.org/jira/browse/HIVE-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepesh Khandelwal reassigned HIVE-18579:
-
> Changes from HIVE-18495 introduced import paths from shaded jars
>
>
> Key: HIVE-18579
> URL: https://issues.apache.org/jira/browse/HIVE-18579
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Deepesh Khandelwal
>Assignee: Zoltan Haindrich
>Priority: Critical
>
> When compiling the latest code after HIVE-18495 seeing the following issue:
> {noformat}
> -
> [ERROR] COMPILATION ERROR :
> -
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java:[29,64]
> package org.apache.hadoop.hbase.shaded.com.google.common.collect does not
> exist
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[43,68]
> package org.apache.hadoop.hbase.shaded.com.google.common.annotations does
> not exist
> ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[100,4]
> cannot find symbol
> symbol: class VisibleForTesting
> location: class org.apache.hive.testutils.MiniZooKeeperCluster
> [INFO] 3 errors
> {noformat}
> Seems like org.apache.hadoop.hbase.shaded.com.google.* is being used, I am
> guessing the idea was to use com.google.*. Not sure why we didn't see this
> failing in Apache Hive build system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18571) stats issue for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18571: Status: Patch Available (was: Open) > stats issue for MM tables > - > > Key: HIVE-18571 > URL: https://issues.apache.org/jira/browse/HIVE-18571 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18571.patch > > > There are multiple stats aggregation issues with MM tables. > Some simple stats are double counted and some stats (simple stats) are > invalid for ACID table dirs altogether. > I have a patch almost ready, need to fix some more stuff and clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18571) stats issue for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18571: Attachment: HIVE-18571.patch > stats issue for MM tables > - > > Key: HIVE-18571 > URL: https://issues.apache.org/jira/browse/HIVE-18571 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18571.patch > > > There are multiple stats aggregation issues with MM tables. > Some simple stats are double counted and some stats (simple stats) are > invalid for ACID table dirs altogether. > I have a patch almost ready, need to fix some more stuff and clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18571) stats issues for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18571: Summary: stats issues for MM tables (was: stats issue for MM tables) > stats issues for MM tables > -- > > Key: HIVE-18571 > URL: https://issues.apache.org/jira/browse/HIVE-18571 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18571.patch > > > There are multiple stats aggregation issues with MM tables. > Some simple stats are double counted and some stats (simple stats) are > invalid for ACID table dirs altogether. > I have a patch almost ready, need to fix some more stuff and clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18571) stats issues for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1630#comment-1630 ] Sergey Shelukhin commented on HIVE-18571: - [~ekoifman] [~steveyeom2017] this one is for stats... first, it removes double aggregation of stats in case of insert overwrite. Then, it actually doesn't call populate quick stats for ACID tables anymore after create, because the logic there is completely bogus for ACID tables (# and size of the files). This may affect some existing out file stats for ACID. > stats issues for MM tables > -- > > Key: HIVE-18571 > URL: https://issues.apache.org/jira/browse/HIVE-18571 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18571.patch > > > There are multiple stats aggregation issues with MM tables. > Some simple stats are double counted and some stats (simple stats) are > invalid for ACID table dirs altogether. > I have a patch almost ready, need to fix some more stuff and clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18472) Beeline gives log4j warnings
[ https://issues.apache.org/jira/browse/HIVE-18472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-18472: --- Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) > Beeline gives log4j warnings > > > Key: HIVE-18472 > URL: https://issues.apache.org/jira/browse/HIVE-18472 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-18472.1.patch, HIVE-18472.2.patch, > HIVE-18472.3.patch > > > Starting Beeline gives the following warnings multiple times: > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ERROR StatusLogger No log4j2 configuration file found. Using default > configuration: logging only errors to the console. Set system property > 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show > Log4j2 internal initialization logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18472) Beeline gives log4j warnings
[ https://issues.apache.org/jira/browse/HIVE-18472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344452#comment-16344452 ] Vihang Karajgaonkar commented on HIVE-18472: patch merged in master and branch-2. Thanks [~janulatha] for your contribution. > Beeline gives log4j warnings > > > Key: HIVE-18472 > URL: https://issues.apache.org/jira/browse/HIVE-18472 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-18472.1.patch, HIVE-18472.2.patch, > HIVE-18472.3.patch > > > Starting Beeline gives the following warnings multiple times: > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > ERROR StatusLogger No log4j2 configuration file found. Using default > configuration: logging only errors to the console. Set system property > 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show > Log4j2 internal initialization logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[
https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344459#comment-16344459
]
Hive QA commented on HIVE-18536:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908241/HIVE-18536.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 12794 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[quotedid_basic]
(batchId=61)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8917/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8917/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8917/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908241 - PreCommit-HIVE-Build
> IOW + DP is broken for insert-only ACID
> ---
>
> Key: HIVE-18536
> URL: https://issues.apache.org/jira/browse/HIVE-18536
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18536.01.patch, HIVE-18536.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344475#comment-16344475
]
Hive QA commented on HIVE-18552:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be8a451 |
| Default Java | 1.8.0_111 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8918/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Attachment: HIVE-18516.4.patch > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.2.patch, > HIVE-18516.3.patch, HIVE-18516.4.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344510#comment-16344510
]
Hive QA commented on HIVE-18552:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908249/HIVE-18552.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 12795 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_bmj_schema_evolution]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_view_failure6]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_view_failure6_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[compare_double_bigint]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[compare_double_bigint_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[compare_string_bigint]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[compare_string_bigint_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[input4]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[input4_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[input_part0_neg]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[input_part0_neg_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8918/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8918/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8918/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 30 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908249 - PreCommit-HIVE-Build
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are run
[jira] [Updated] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Description: Struct type is not supported in MapWork with vectorization, it should be supported to improve the performance. New UDF will be added to access the field of Struct. Note: * Filter operator won't be supported. * Nested complex type won't be tested in this ticket. was: Struct type is not supported in MapWork with vectorization, it should be supported to improve the performance. New UDF will be added to access the field of Struct. Note: * Filter operator won't be supported. > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17552) Enable bucket map join by default
[
https://issues.apache.org/jira/browse/HIVE-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344517#comment-16344517
]
Hive QA commented on HIVE-17552:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be8a451 |
| Default Java | 1.8.0_111 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8919/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable bucket map join by default
> -
>
> Key: HIVE-17552
> URL: https://issues.apache.org/jira/browse/HIVE-17552
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-17552.1.patch, HIVE-17552.2.patch,
> HIVE-17552.3.patch, HIVE-17552.4.patch, HIVE-17552.5.patch,
> HIVE-17552.6.patch, HIVE-17552.7.patch
>
>
> Currently bucket map join is disabled by default, however, it is potentially
> most optimal join we have. Need to enable it by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16886) HMS log notifications may have duplicated event IDs if multiple HMS are running concurrently
[
https://issues.apache.org/jira/browse/HIVE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344530#comment-16344530
]
anishek commented on HIVE-16886:
[~akolb] seems to work based on code as you suggest, if you could test this as
well against a mssqlserver that will be great, I am ok with reimplementing this
patch.
[~thejas] anything else we need to take care of ?
> HMS log notifications may have duplicated event IDs if multiple HMS are
> running concurrently
>
>
> Key: HIVE-16886
> URL: https://issues.apache.org/jira/browse/HIVE-16886
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 3.0.0, 2.3.2, 2.3.3
>Reporter: Sergio Peña
>Assignee: anishek
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-16886.1.patch, HIVE-16886.2.patch,
> HIVE-16886.3.patch, HIVE-16886.4.patch, HIVE-16886.5.patch,
> HIVE-16886.6.patch, HIVE-16886.7.patch, HIVE-16886.8.patch,
> datastore-identity-holes.diff
>
>
> When running multiple Hive Metastore servers and DB notifications are
> enabled, I could see that notifications can be persisted with a duplicated
> event ID.
> This does not happen when running multiple threads in a single HMS node due
> to the locking acquired on the DbNotificationsLog class, but multiple HMS
> could cause conflicts.
> The issue is in the ObjectStore#addNotificationEvent() method. The event ID
> fetched from the datastore is used for the new notification, incremented in
> the server itself, then persisted or updated back to the datastore. If 2
> servers read the same ID, then these 2 servers write a new notification with
> the same ID.
> The event ID is not unique nor a primary key.
> Here's a test case using the TestObjectStore class that confirms this issue:
> {noformat}
> @Test
> public void testConcurrentAddNotifications() throws ExecutionException,
> InterruptedException {
> final int NUM_THREADS = 2;
> CountDownLatch countIn = new CountDownLatch(NUM_THREADS);
> CountDownLatch countOut = new CountDownLatch(1);
> HiveConf conf = new HiveConf();
> conf.setVar(HiveConf.ConfVars.METASTORE_EXPRESSION_PROXY_CLASS,
> MockPartitionExpressionProxy.class.getName());
> ExecutorService executorService =
> Executors.newFixedThreadPool(NUM_THREADS);
> FutureTask tasks[] = new FutureTask[NUM_THREADS];
> for (int i=0; i final int n = i;
> tasks[i] = new FutureTask(new Callable() {
> @Override
> public Void call() throws Exception {
> ObjectStore store = new ObjectStore();
> store.setConf(conf);
> NotificationEvent dbEvent =
> new NotificationEvent(0, 0,
> EventMessage.EventType.CREATE_DATABASE.toString(), "CREATE DATABASE DB" + n);
> System.out.println("ADDING NOTIFICATION");
> countIn.countDown();
> countOut.await();
> store.addNotificationEvent(dbEvent);
> System.out.println("FINISH NOTIFICATION");
> return null;
> }
> });
> executorService.execute(tasks[i]);
> }
> countIn.await();
> countOut.countDown();
> for (int i = 0; i < NUM_THREADS; ++i) {
> tasks[i].get();
> }
> NotificationEventResponse eventResponse =
> objectStore.getNextNotification(new NotificationEventRequest());
> Assert.assertEquals(2, eventResponse.getEventsSize());
> Assert.assertEquals(1, eventResponse.getEvents().get(0).getEventId());
> // This fails because the next notification has an event ID = 1
> Assert.assertEquals(2, eventResponse.getEvents().get(1).getEventId());
> }
> {noformat}
> The last assertion fails expecting an event ID 1 instead of 2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[ https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344539#comment-16344539 ] Ashutosh Chauhan commented on HIVE-18577: - +1 > SemanticAnalyzer.validate has some pointless metastore calls > > > Key: HIVE-18577 > URL: https://issues.apache.org/jira/browse/HIVE-18577 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18577.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Attachment: HIVE-18048.006.patch > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch, > HIVE-18048.006.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Attachment: (was: HIVE-18048.006.patch) > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18550) Keep the hbase table name property as hbase.table.name
[ https://issues.apache.org/jira/browse/HIVE-18550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344543#comment-16344543 ] Aihua Xu commented on HIVE-18550: - [~ychena] Thanks for reviewing. This HiveHFileOutputFormat is for hbase bulk loading. To read from the hbase, it can be handled by HiveHBaseTableInputFormat which takes care of the read already. > Keep the hbase table name property as hbase.table.name > -- > > Key: HIVE-18550 > URL: https://issues.apache.org/jira/browse/HIVE-18550 > Project: Hive > Issue Type: Sub-task > Components: HBase Handler >Affects Versions: 3.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-18550.1.patch > > > With hbase 2.0 support, I made some changes to the hbase table name property > change in HIVE-18366 and HIVE-18202. By checking the logic, seems the change > is not necessary since hbase.table.name is internal to hive hbase handler. We > just need to map hbase.table.name to > hbase.mapreduce.hfileoutputformat.table.name for HiveHFileOutputFormat. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16886) HMS log notifications may have duplicated event IDs if multiple HMS are running concurrently
[
https://issues.apache.org/jira/browse/HIVE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344546#comment-16344546
]
Alexander Kolbasov commented on HIVE-16886:
---
[~anishek] I don't have access to MS SQL server - do you have something? The
test requires tables created for HMS (at least two notification tables, the
rest are irrelevant).
> HMS log notifications may have duplicated event IDs if multiple HMS are
> running concurrently
>
>
> Key: HIVE-16886
> URL: https://issues.apache.org/jira/browse/HIVE-16886
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 3.0.0, 2.3.2, 2.3.3
>Reporter: Sergio Peña
>Assignee: anishek
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-16886.1.patch, HIVE-16886.2.patch,
> HIVE-16886.3.patch, HIVE-16886.4.patch, HIVE-16886.5.patch,
> HIVE-16886.6.patch, HIVE-16886.7.patch, HIVE-16886.8.patch,
> datastore-identity-holes.diff
>
>
> When running multiple Hive Metastore servers and DB notifications are
> enabled, I could see that notifications can be persisted with a duplicated
> event ID.
> This does not happen when running multiple threads in a single HMS node due
> to the locking acquired on the DbNotificationsLog class, but multiple HMS
> could cause conflicts.
> The issue is in the ObjectStore#addNotificationEvent() method. The event ID
> fetched from the datastore is used for the new notification, incremented in
> the server itself, then persisted or updated back to the datastore. If 2
> servers read the same ID, then these 2 servers write a new notification with
> the same ID.
> The event ID is not unique nor a primary key.
> Here's a test case using the TestObjectStore class that confirms this issue:
> {noformat}
> @Test
> public void testConcurrentAddNotifications() throws ExecutionException,
> InterruptedException {
> final int NUM_THREADS = 2;
> CountDownLatch countIn = new CountDownLatch(NUM_THREADS);
> CountDownLatch countOut = new CountDownLatch(1);
> HiveConf conf = new HiveConf();
> conf.setVar(HiveConf.ConfVars.METASTORE_EXPRESSION_PROXY_CLASS,
> MockPartitionExpressionProxy.class.getName());
> ExecutorService executorService =
> Executors.newFixedThreadPool(NUM_THREADS);
> FutureTask tasks[] = new FutureTask[NUM_THREADS];
> for (int i=0; i final int n = i;
> tasks[i] = new FutureTask(new Callable() {
> @Override
> public Void call() throws Exception {
> ObjectStore store = new ObjectStore();
> store.setConf(conf);
> NotificationEvent dbEvent =
> new NotificationEvent(0, 0,
> EventMessage.EventType.CREATE_DATABASE.toString(), "CREATE DATABASE DB" + n);
> System.out.println("ADDING NOTIFICATION");
> countIn.countDown();
> countOut.await();
> store.addNotificationEvent(dbEvent);
> System.out.println("FINISH NOTIFICATION");
> return null;
> }
> });
> executorService.execute(tasks[i]);
> }
> countIn.await();
> countOut.countDown();
> for (int i = 0; i < NUM_THREADS; ++i) {
> tasks[i].get();
> }
> NotificationEventResponse eventResponse =
> objectStore.getNextNotification(new NotificationEventRequest());
> Assert.assertEquals(2, eventResponse.getEventsSize());
> Assert.assertEquals(1, eventResponse.getEvents().get(0).getEventId());
> // This fails because the next notification has an event ID = 1
> Assert.assertEquals(2, eventResponse.getEvents().get(1).getEventId());
> }
> {noformat}
> The last assertion fails expecting an event ID 1 instead of 2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17552) Enable bucket map join by default
[
https://issues.apache.org/jira/browse/HIVE-17552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344560#comment-16344560
]
Hive QA commented on HIVE-17552:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908250/HIVE-17552.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12794 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_onesideskew]
(batchId=72)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_5]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_9]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentStatements (batchId=238)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8919/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8919/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8919/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908250 - PreCommit-HIVE-Build
> Enable bucket map join by default
> -
>
> Key: HIVE-17552
> URL: https://issues.apache.org/jira/browse/HIVE-17552
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-17552.1.patch, HIVE-17552.2.patch,
> HIVE-17552.3.patch, HIVE-17552.4.patch, HIVE-17552.5.patch,
> HIVE-17552.6.patch, HIVE-17552.7.patch
>
>
> Currently bucket map join is disabled by default, however, it is potentially
> most optimal join we have. Need to enable it by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[ https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344566#comment-16344566 ] Ashutosh Chauhan commented on HIVE-18577: - HIVE-11145 removed Protect mode because of same reason that this is expensive metadata call, but stopped short of removing corresponding logic which it should had. +1 > SemanticAnalyzer.validate has some pointless metastore calls > > > Key: HIVE-18577 > URL: https://issues.apache.org/jira/browse/HIVE-18577 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18577.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18569) Hive Druid indexing not dealing with decimals in correct way.
[ https://issues.apache.org/jira/browse/HIVE-18569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344572#comment-16344572 ] Ashutosh Chauhan commented on HIVE-18569: - TestMiniDruidCliDriver.testCliDriver[druidmini_mv] may be related. +1 > Hive Druid indexing not dealing with decimals in correct way. > - > > Key: HIVE-18569 > URL: https://issues.apache.org/jira/browse/HIVE-18569 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-18569.patch > > > Currently, a decimal column is indexed as double in druid. > This should not happen and either the user has to add an explicit cast or we > can add a flag to enable approximation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18579) Changes from HIVE-18495 introduced import paths from shaded jars
[
https://issues.apache.org/jira/browse/HIVE-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-18579:
Attachment: HIVE-18579.patch
> Changes from HIVE-18495 introduced import paths from shaded jars
>
>
> Key: HIVE-18579
> URL: https://issues.apache.org/jira/browse/HIVE-18579
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Deepesh Khandelwal
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-18579.patch
>
>
> When compiling the latest code after HIVE-18495 seeing the following issue:
> {noformat}
> -
> [ERROR] COMPILATION ERROR :
> -
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java:[29,64]
> package org.apache.hadoop.hbase.shaded.com.google.common.collect does not
> exist
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[43,68]
> package org.apache.hadoop.hbase.shaded.com.google.common.annotations does
> not exist
> ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[100,4]
> cannot find symbol
> symbol: class VisibleForTesting
> location: class org.apache.hive.testutils.MiniZooKeeperCluster
> [INFO] 3 errors
> {noformat}
> Seems like org.apache.hadoop.hbase.shaded.com.google.* is being used, I am
> guessing the idea was to use com.google.*. Not sure why we didn't see this
> failing in Apache Hive build system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18579) Changes from HIVE-18495 introduced import paths from shaded jars
[
https://issues.apache.org/jira/browse/HIVE-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-18579:
Status: Patch Available (was: Open)
[~kgyrtkirk] Can you please take a look?
> Changes from HIVE-18495 introduced import paths from shaded jars
>
>
> Key: HIVE-18579
> URL: https://issues.apache.org/jira/browse/HIVE-18579
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Deepesh Khandelwal
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-18579.patch
>
>
> When compiling the latest code after HIVE-18495 seeing the following issue:
> {noformat}
> -
> [ERROR] COMPILATION ERROR :
> -
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java:[29,64]
> package org.apache.hadoop.hbase.shaded.com.google.common.collect does not
> exist
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[43,68]
> package org.apache.hadoop.hbase.shaded.com.google.common.annotations does
> not exist
> ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[100,4]
> cannot find symbol
> symbol: class VisibleForTesting
> location: class org.apache.hive.testutils.MiniZooKeeperCluster
> [INFO] 3 errors
> {noformat}
> Seems like org.apache.hadoop.hbase.shaded.com.google.* is being used, I am
> guessing the idea was to use com.google.*. Not sure why we didn't see this
> failing in Apache Hive build system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[
https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344577#comment-16344577
]
Hive QA commented on HIVE-18577:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 780 unchanged - 1
fixed = 781 total (was 781) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / be8a451 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8920/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8920/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SemanticAnalyzer.validate has some pointless metastore calls
>
>
> Key: HIVE-18577
> URL: https://issues.apache.org/jira/browse/HIVE-18577
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18577.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Attachment: HIVE-18048.006.patch > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch, > HIVE-18048.006.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18048) Vectorization: Support Struct type with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344596#comment-16344596 ] Colin Ma commented on HIVE-18048: - [~Ferd], thanks for review. The patch is updated as your comments, new tests for ORC and filter operator are added. > Vectorization: Support Struct type with vectorization > - > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Improvement >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Major > Attachments: HIVE-18048.001.patch, HIVE-18048.002.patch, > HIVE-18048.003.patch, HIVE-18048.004.patch, HIVE-18048.005.patch, > HIVE-18048.006.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > New UDF will be added to access the field of Struct. > Note: > * Filter operator won't be supported. > * Nested complex type won't be tested in this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18546) Remove unnecessary code introduced in HIVE-14498
[ https://issues.apache.org/jira/browse/HIVE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18546: --- Attachment: HIVE-18546.03.patch > Remove unnecessary code introduced in HIVE-14498 > > > Key: HIVE-18546 > URL: https://issues.apache.org/jira/browse/HIVE-18546 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18546.02.patch, HIVE-18546.03.patch > > > HIVE-14498 introduced some code to check the invalidation of materialized > views that can be simplified, relying instead on existing transaction ids. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16886) HMS log notifications may have duplicated event IDs if multiple HMS are running concurrently
[
https://issues.apache.org/jira/browse/HIVE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344606#comment-16344606
]
anishek commented on HIVE-16886:
nope i dont have one either.
> HMS log notifications may have duplicated event IDs if multiple HMS are
> running concurrently
>
>
> Key: HIVE-16886
> URL: https://issues.apache.org/jira/browse/HIVE-16886
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 3.0.0, 2.3.2, 2.3.3
>Reporter: Sergio Peña
>Assignee: anishek
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-16886.1.patch, HIVE-16886.2.patch,
> HIVE-16886.3.patch, HIVE-16886.4.patch, HIVE-16886.5.patch,
> HIVE-16886.6.patch, HIVE-16886.7.patch, HIVE-16886.8.patch,
> datastore-identity-holes.diff
>
>
> When running multiple Hive Metastore servers and DB notifications are
> enabled, I could see that notifications can be persisted with a duplicated
> event ID.
> This does not happen when running multiple threads in a single HMS node due
> to the locking acquired on the DbNotificationsLog class, but multiple HMS
> could cause conflicts.
> The issue is in the ObjectStore#addNotificationEvent() method. The event ID
> fetched from the datastore is used for the new notification, incremented in
> the server itself, then persisted or updated back to the datastore. If 2
> servers read the same ID, then these 2 servers write a new notification with
> the same ID.
> The event ID is not unique nor a primary key.
> Here's a test case using the TestObjectStore class that confirms this issue:
> {noformat}
> @Test
> public void testConcurrentAddNotifications() throws ExecutionException,
> InterruptedException {
> final int NUM_THREADS = 2;
> CountDownLatch countIn = new CountDownLatch(NUM_THREADS);
> CountDownLatch countOut = new CountDownLatch(1);
> HiveConf conf = new HiveConf();
> conf.setVar(HiveConf.ConfVars.METASTORE_EXPRESSION_PROXY_CLASS,
> MockPartitionExpressionProxy.class.getName());
> ExecutorService executorService =
> Executors.newFixedThreadPool(NUM_THREADS);
> FutureTask tasks[] = new FutureTask[NUM_THREADS];
> for (int i=0; i final int n = i;
> tasks[i] = new FutureTask(new Callable() {
> @Override
> public Void call() throws Exception {
> ObjectStore store = new ObjectStore();
> store.setConf(conf);
> NotificationEvent dbEvent =
> new NotificationEvent(0, 0,
> EventMessage.EventType.CREATE_DATABASE.toString(), "CREATE DATABASE DB" + n);
> System.out.println("ADDING NOTIFICATION");
> countIn.countDown();
> countOut.await();
> store.addNotificationEvent(dbEvent);
> System.out.println("FINISH NOTIFICATION");
> return null;
> }
> });
> executorService.execute(tasks[i]);
> }
> countIn.await();
> countOut.countDown();
> for (int i = 0; i < NUM_THREADS; ++i) {
> tasks[i].get();
> }
> NotificationEventResponse eventResponse =
> objectStore.getNextNotification(new NotificationEventRequest());
> Assert.assertEquals(2, eventResponse.getEventsSize());
> Assert.assertEquals(1, eventResponse.getEvents().get(0).getEventId());
> // This fails because the next notification has an event ID = 1
> Assert.assertEquals(2, eventResponse.getEvents().get(1).getEventId());
> }
> {noformat}
> The last assertion fails expecting an event ID 1 instead of 2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18579) Changes from HIVE-18495 introduced import paths from shaded jars
[
https://issues.apache.org/jira/browse/HIVE-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-18579:
---
Assignee: Ashutosh Chauhan (was: Zoltan Haindrich)
+1
I'm not sure how this could have became undetected...it seems that we've ~3
shaded guava libraries loaded..plus the regular :)
this should be checked somehow; to prevent future problems arising from this
source...
> Changes from HIVE-18495 introduced import paths from shaded jars
>
>
> Key: HIVE-18579
> URL: https://issues.apache.org/jira/browse/HIVE-18579
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Deepesh Khandelwal
>Assignee: Ashutosh Chauhan
>Priority: Critical
> Attachments: HIVE-18579.patch
>
>
> When compiling the latest code after HIVE-18495 seeing the following issue:
> {noformat}
> -
> [ERROR] COMPILATION ERROR :
> -
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java:[29,64]
> package org.apache.hadoop.hbase.shaded.com.google.common.collect does not
> exist
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[43,68]
> package org.apache.hadoop.hbase.shaded.com.google.common.annotations does
> not exist
> ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[100,4]
> cannot find symbol
> symbol: class VisibleForTesting
> location: class org.apache.hive.testutils.MiniZooKeeperCluster
> [INFO] 3 errors
> {noformat}
> Seems like org.apache.hadoop.hbase.shaded.com.google.* is being used, I am
> guessing the idea was to use com.google.*. Not sure why we didn't see this
> failing in Apache Hive build system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[
https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344624#comment-16344624
]
Hive QA commented on HIVE-18577:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908252/HIVE-18577.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 12779 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=150)
[intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,table_nonprintable.q,tez_union_dynamic_partition.q,tez_union_dynamic_partition_2.q,temp_table_external.q,global_limit.q,llap_udf.q,schemeAuthority.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,parallel_colstats.q]
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8920/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8920/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8920/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908252 - PreCommit-HIVE-Build
> SemanticAnalyzer.validate has some pointless metastore calls
>
>
> Key: HIVE-18577
> URL: https://issues.apache.org/jira/browse/HIVE-18577
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18577.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18574) LLAP: Ship netty3 as part of LLAP install tarball
[
https://issues.apache.org/jira/browse/HIVE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344623#comment-16344623
]
Ashutosh Chauhan commented on HIVE-18574:
-
On hive repo:
{code}
$ git grep "import io.netty" | wc -l
74
$ git grep "import org.jboss.netty" | wc -l
49
{code}
is what I am talking about. Hive itself uses both netty3 and netty4 which is
unnecessary.
> LLAP: Ship netty3 as part of LLAP install tarball
> -
>
> Key: HIVE-18574
> URL: https://issues.apache.org/jira/browse/HIVE-18574
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18574.1.patch
>
>
> Removing netty from Tez libs causes
> {code}
> 2018-01-29T18:28:49,995 ERROR [main ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP
> Daemon with exception
> java.lang.NoClassDefFoundError: org/jboss/netty/channel/group/ChannelGroup
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStart(LlapDaemon.java:410)
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18574) LLAP: Ship netty3 as part of LLAP install tarball
[
https://issues.apache.org/jira/browse/HIVE-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-18574:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Gopal!
> LLAP: Ship netty3 as part of LLAP install tarball
> -
>
> Key: HIVE-18574
> URL: https://issues.apache.org/jira/browse/HIVE-18574
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18574.1.patch
>
>
> Removing netty from Tez libs causes
> {code}
> 2018-01-29T18:28:49,995 ERROR [main ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP
> Daemon with exception
> java.lang.NoClassDefFoundError: org/jboss/netty/channel/group/ChannelGroup
> at
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.serviceStart(LlapDaemon.java:410)
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18579) Changes from HIVE-18495 introduced import paths from shaded jars
[
https://issues.apache.org/jira/browse/HIVE-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-18579:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
> Changes from HIVE-18495 introduced import paths from shaded jars
>
>
> Key: HIVE-18579
> URL: https://issues.apache.org/jira/browse/HIVE-18579
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Deepesh Khandelwal
>Assignee: Ashutosh Chauhan
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18579.patch
>
>
> When compiling the latest code after HIVE-18495 seeing the following issue:
> {noformat}
> -
> [ERROR] COMPILATION ERROR :
> -
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java:[29,64]
> package org.apache.hadoop.hbase.shaded.com.google.common.collect does not
> exist
> [ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[43,68]
> package org.apache.hadoop.hbase.shaded.com.google.common.annotations does
> not exist
> ERROR]
> /grid/0/jenkins/workspace/Zuul_HDP_Build_Job/build-support/SOURCES/hive/ql/src/test/org/apache/hive/testutils/MiniZooKeeperCluster.java:[100,4]
> cannot find symbol
> symbol: class VisibleForTesting
> location: class org.apache.hive.testutils.MiniZooKeeperCluster
> [INFO] 3 errors
> {noformat}
> Seems like org.apache.hadoop.hbase.shaded.com.google.* is being used, I am
> guessing the idea was to use com.google.*. Not sure why we didn't see this
> failing in Apache Hive build system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18578) Some class has missed the ASF header
[
https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344643#comment-16344643
]
Hive QA commented on HIVE-18578:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} itests/qtest-druid: The patch generated 0 new + 5
unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0edf6b3 |
| Default Java | 1.8.0_111 |
| modules | C: itests/qtest-druid U: itests/qtest-druid |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8921/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Some class has missed the ASF header
>
>
> Key: HIVE-18578
> URL: https://issues.apache.org/jira/browse/HIVE-18578
> Project: Hive
> Issue Type: Bug
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HIVE-18578.1.patch
>
>
> Some class has missed the ASF header
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
