[jira] [Commented] (HIVE-17858) MM - some union cases are broken
[
https://issues.apache.org/jira/browse/HIVE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223256#comment-16223256
]
Hive QA commented on HIVE-17858:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894430/HIVE-17858.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11342 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7523/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7523/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7523/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894430 - PreCommit-HIVE-Build
> MM - some union cases are broken
>
>
> Key: HIVE-17858
> URL: https://issues.apache.org/jira/browse/HIVE-17858
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Labels: mm-gap-1
> Attachments: HIVE-17858.01.patch, HIVE-17858.02.patch,
> HIVE-17858.patch
>
>
> mm_all test no longer runs on LLAP; if it's executed in LLAP, one can see
> that some union cases no longer work.
> Queries on partunion_mm, skew_dp_union_mm produce no results.
> I'm not sure what part of "integration" broke it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17765) expose Hive keywords
[ https://issues.apache.org/jira/browse/HIVE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223252#comment-16223252 ] Lefty Leverenz commented on HIVE-17765: --- bq. Is the original API itself documented? I don't know what you mean by the original API but my guess is that it isn't documented (except in javadocs). This just sounds like something that might need a bit of explanation in the wiki so people can make use of it. > expose Hive keywords > - > > Key: HIVE-17765 > URL: https://issues.apache.org/jira/browse/HIVE-17765 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-17765.01.patch, HIVE-17765.02.patch, > HIVE-17765.03.patch, HIVE-17765.nogen.patch, HIVE-17765.patch > > > This could be useful e.g. for BI tools (via ODBC/JDBC drivers) to decide on > SQL capabilities of Hive -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17832) Allow hive.metastore.disallow.incompatible.col.type.changes to be changed in metastore
[ https://issues.apache.org/jira/browse/HIVE-17832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223246#comment-16223246 ] Lefty Leverenz commented on HIVE-17832: --- Okay, thanks for explaining why no documentation is needed. > Allow hive.metastore.disallow.incompatible.col.type.changes to be changed in > metastore > -- > > Key: HIVE-17832 > URL: https://issues.apache.org/jira/browse/HIVE-17832 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE17832.1.patch, HIVE17832.2.patch > > > hive.metastore.disallow.incompatible.col.type.changes when set to true, will > disallow incompatible column type changes through alter table. But, this > parameter is not modifiable in HMS. If HMS in not embedded into HS2, the > value cannot be changed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223242#comment-16223242 ] Lefty Leverenz commented on HIVE-14731: --- Doc note: This adds *hive.tez.cartesian-product.enabled* to HiveConf.java, so it needs to be documented in the wiki. * [Configuration Properties -- Tez | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Tez] Added a TODOC3.0 label. Also, [~hagleitn] please update the status and fix version. > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Labels: TODOC3.0 > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.12.patch, HIVE-14731.13.patch, > HIVE-14731.14.patch, HIVE-14731.15.patch, HIVE-14731.16.patch, > HIVE-14731.17.patch, HIVE-14731.18.patch, HIVE-14731.19.patch, > HIVE-14731.2.patch, HIVE-14731.20.patch, HIVE-14731.21.patch, > HIVE-14731.22.patch, HIVE-14731.23.patch, HIVE-14731.3.patch, > HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch, > HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch, > HIVE-14731.addendum.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14731: -- Labels: TODOC3.0 (was: ) > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Labels: TODOC3.0 > Attachments: HIVE-14731.1.patch, HIVE-14731.10.patch, > HIVE-14731.11.patch, HIVE-14731.12.patch, HIVE-14731.13.patch, > HIVE-14731.14.patch, HIVE-14731.15.patch, HIVE-14731.16.patch, > HIVE-14731.17.patch, HIVE-14731.18.patch, HIVE-14731.19.patch, > HIVE-14731.2.patch, HIVE-14731.20.patch, HIVE-14731.21.patch, > HIVE-14731.22.patch, HIVE-14731.23.patch, HIVE-14731.3.patch, > HIVE-14731.4.patch, HIVE-14731.5.patch, HIVE-14731.6.patch, > HIVE-14731.7.patch, HIVE-14731.8.patch, HIVE-14731.9.patch, > HIVE-14731.addendum.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15670) column_stats_accurate may not fit in PARTITION_PARAMS.VALUE
[ https://issues.apache.org/jira/browse/HIVE-15670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223239#comment-16223239 ] Alexander Behm commented on HIVE-15670: --- Thanks for the response. From the perspective of a client issuing an RPC to alter column stats, it seems like a rather questionable side-effect to also alter the table metadata with a pretty big payload. Instead of "fixing" this issue by changing the database schema, could we instead remove the JSON string altogether? I'm definitely not familiar with the implementation details, just trying to provide a perspective from a Metastore client that is not Hive. > column_stats_accurate may not fit in PARTITION_PARAMS.VALUE > --- > > Key: HIVE-15670 > URL: https://issues.apache.org/jira/browse/HIVE-15670 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > The JSON can be too big with many columns (see setColumnStatsState method). > We can make JSON more compact by only storing the list of columns with true > values. Or we can even store a bitmask in a dedicated column, and adjust it > when altering table (rare enough). Or we can just change the VALUE column to > text blob (might be a painful change wrt upgrade scripts, and supporting all > the DBs' varied blob implementations, esp. in directsql). > Storing denormalized flags in a separate table will probably be slow, > comparatively. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17748) ReplCopyTask doesn't support multi-file CopyWork
[
https://issues.apache.org/jira/browse/HIVE-17748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223231#comment-16223231
]
Hive QA commented on HIVE-17748:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894435/HIVE-17748.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11341 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7522/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7522/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7522/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894435 - PreCommit-HIVE-Build
> ReplCopyTask doesn't support multi-file CopyWork
>
>
> Key: HIVE-17748
> URL: https://issues.apache.org/jira/browse/HIVE-17748
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17748.01.patch, HIVE-17748.patch
>
>
> has
> {noformat}
> Path fromPath = work.getFromPaths()[0];
> toPath = work.getToPaths()[0];
> {noformat}
> should this throw if from/to paths have > 1 element?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Status: Patch Available (was: In Progress)
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch, HIVE-17433.091.patch,
> HIVE-17433.092.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Status: In Progress (was: Patch Available)
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch, HIVE-17433.091.patch,
> HIVE-17433.092.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Attachment: HIVE-17433.092.patch
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch, HIVE-17433.091.patch,
> HIVE-17433.092.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17927) Pre-Warm Hive on Spark Driver
[ https://issues.apache.org/jira/browse/HIVE-17927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-17927: Summary: Pre-Warm Hive on Spark Driver (was: Pre-Warm Spark Driver) > Pre-Warm Hive on Spark Driver > - > > Key: HIVE-17927 > URL: https://issues.apache.org/jira/browse/HIVE-17927 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar > > Hive-on-Spark has an option to pre-warm executors, it would be good to have > an option to pre-warm the HoS driver. > Basically, submit the HoS application as soon as possible, rather than > waiting until a query is ready to be submitted. This can help startup time > for the first query run in a HoS session. The application can be setup in the > background while queries are being compiled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17750) add a flag to automatically create most tables as MM
[
https://issues.apache.org/jira/browse/HIVE-17750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223186#comment-16223186
]
Hive QA commented on HIVE-17750:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894432/HIVE-17750.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11342 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7521/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7521/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7521/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894432 - PreCommit-HIVE-Build
> add a flag to automatically create most tables as MM
> -
>
> Key: HIVE-17750
> URL: https://issues.apache.org/jira/browse/HIVE-17750
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17750.01.patch, HIVE-17750.patch
>
>
> After merge we are going to do another round of gap identification... similar
> to HIVE-14990.
> However the approach used there is a huge PITA. It'd be much better to make
> tables MM by default at create time, not pretend they are MM at check time,
> from the perspective of spurious error elimination.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-5472) support a simple scalar which returns the current timestamp
[ https://issues.apache.org/jira/browse/HIVE-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223181#comment-16223181 ] Lefty Leverenz commented on HIVE-5472: -- HIVE-11127 requests additional documentation about current_date and current_timestamp. > support a simple scalar which returns the current timestamp > --- > > Key: HIVE-5472 > URL: https://issues.apache.org/jira/browse/HIVE-5472 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.11.0 >Reporter: N Campbell >Assignee: Jason Dere > Fix For: 1.2.0 > > Attachments: HIVE-5472.1.patch, HIVE-5472.2.patch, HIVE-5472.3.patch, > HIVE-5472.4.patch > > > ISO-SQL has two forms of functions > local and current timestamp where the former is a TIMESTAMP WITHOUT TIMEZONE > and the latter with TIME ZONE > select cast ( unix_timestamp() as timestamp ) from T > implement a function which computes LOCAL TIMESTAMP which would be the > current timestamp for the users session time zone. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-11127) Document time zone handling for current_date and current_timestamp
[
https://issues.apache.org/jira/browse/HIVE-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-11127:
--
Labels: TODOC1.2 timestamp (was: timestamp)
> Document time zone handling for current_date and current_timestamp
> --
>
> Key: HIVE-11127
> URL: https://issues.apache.org/jira/browse/HIVE-11127
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.2.0
>Reporter: Punya Biswal
> Labels: TODOC1.2, timestamp
>
> The new {{current_date}} and {{current_timestamp}} functions introduced in
> HIVE-5472 emit dates/timestamps in the user's local timezone. This behavior
> should be documented on [the
> wiki|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-8534) sql std auth : update configuration whitelist for 0.14
[ https://issues.apache.org/jira/browse/HIVE-8534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223163#comment-16223163 ] Lefty Leverenz commented on HIVE-8534: -- Doc update: *hive.security.authorization.sqlstd.confwhitelist.append* still needs to be documented. See HIVE-8937 for new description. > sql std auth : update configuration whitelist for 0.14 > -- > > Key: HIVE-8534 > URL: https://issues.apache.org/jira/browse/HIVE-8534 > Project: Hive > Issue Type: Bug > Components: Authorization, SQLStandardAuthorization >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-8534.1.patch, HIVE-8534.2.patch, HIVE-8534.3.patch, > HIVE-8534.4.patch, HIVE-8534.5.patch > > > New config parameters have been introduced in hive 0.14. SQL standard > authorization needs to be updated to allow some new parameters to be set, > when the authorization mode is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-8937) fix description of hive.security.authorization.sqlstd.confwhitelist.* params
[ https://issues.apache.org/jira/browse/HIVE-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223162#comment-16223162 ] Lefty Leverenz commented on HIVE-8937: -- Doc note: The description of *hive.security.authorization.sqlstd.confwhitelist* needs to be updated in the wiki, and *hive.security.authorization.sqlstd.confwhitelist.append* needs to be documented in the same section of the wiki (it was created by HIVE-8534 in release 0.14.0). * [Configuration Properties -- SQL Standard Based Authorization | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-SQLStandardBasedAuthorization] ** [hive.security.authorization.sqlstd.confwhitelist | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.security.authorization.sqlstd.confwhitelist] ** [hive.security.authorization.sqlstd.confwhitelist.append | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.security.authorization.sqlstd.confwhitelist.append] (this link won't work until the config is documented) Added TODOC3.0 and TODOC14 labels. > fix description of hive.security.authorization.sqlstd.confwhitelist.* params > > > Key: HIVE-8937 > URL: https://issues.apache.org/jira/browse/HIVE-8937 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.14.0 >Reporter: Thejas M Nair >Assignee: Akira Ajisaka > Labels: TODOC14, TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-8937.001.patch, HIVE-8937.002.patch > > > hive.security.authorization.sqlstd.confwhitelist.* param description in > HiveConf is incorrect. The expected value is a regex, not comma separated > regexes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-8937) fix description of hive.security.authorization.sqlstd.confwhitelist.* params
[ https://issues.apache.org/jira/browse/HIVE-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-8937: - Labels: TODOC14 TODOC3.0 (was: ) > fix description of hive.security.authorization.sqlstd.confwhitelist.* params > > > Key: HIVE-8937 > URL: https://issues.apache.org/jira/browse/HIVE-8937 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.14.0 >Reporter: Thejas M Nair >Assignee: Akira Ajisaka > Labels: TODOC14, TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-8937.001.patch, HIVE-8937.002.patch > > > hive.security.authorization.sqlstd.confwhitelist.* param description in > HiveConf is incorrect. The expected value is a regex, not comma separated > regexes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17778) Add support for custom counters in trigger expression
[ https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223142#comment-16223142 ] Lefty Leverenz commented on HIVE-17778: --- Will documentation for this be covered by the umbrella jira HIVE-17481, or does this need a TODOC3.0 label? > Add support for custom counters in trigger expression > - > > Key: HIVE-17778 > URL: https://issues.apache.org/jira/browse/HIVE-17778 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-17778.1.patch, HIVE-17778.2.patch, > HIVE-17778.3.patch, HIVE-17778.4.patch, HIVE-17778.5.patch, > HIVE-17778.6.patch, HIVE-17778.7.patch > > > HIVE-17508 only supports limited counters. This ticket is to extend it to > support custom counters (counters that are not supported by execution engine > will be dropped). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17907) enable and apply resource plan commands in HS2
[ https://issues.apache.org/jira/browse/HIVE-17907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17907: --- Assignee: Sergey Shelukhin > enable and apply resource plan commands in HS2 > -- > > Key: HIVE-17907 > URL: https://issues.apache.org/jira/browse/HIVE-17907 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Enabling and applying the RP should only be runnable in HS2 with active WM. > Both should validate the full resource plan (or at least enable should; users > cannot modify the RP via normal means once enabled, but it might be worth > double checking since we have to fetch it anyway to apply). > Then, apply should propagate the resource plan to the WM instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17903) implement query mapping to WM and non-WM based on policies
[ https://issues.apache.org/jira/browse/HIVE-17903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223118#comment-16223118 ] Sergey Shelukhin commented on HIVE-17903: - A small patch on top of HIVE-17902 cc [~prasanth_j] > implement query mapping to WM and non-WM based on policies > --- > > Key: HIVE-17903 > URL: https://issues.apache.org/jira/browse/HIVE-17903 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17903.patch > > > Depends on HIVE-17902 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223116#comment-16223116 ] Zhiyuan Yang commented on HIVE-15016: - [~aihuaxu] Can you point me to the full log? > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, > HIVE-15016.7.patch, HIVE-15016.8.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17903) implement query mapping to WM and non-WM based on policies
[ https://issues.apache.org/jira/browse/HIVE-17903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17903: Attachment: HIVE-17903.patch > implement query mapping to WM and non-WM based on policies > --- > > Key: HIVE-17903 > URL: https://issues.apache.org/jira/browse/HIVE-17903 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17903.patch > > > Depends on HIVE-17902 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17903) implement query mapping to WM and non-WM based on policies
[ https://issues.apache.org/jira/browse/HIVE-17903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17903: Status: Patch Available (was: Open) > implement query mapping to WM and non-WM based on policies > --- > > Key: HIVE-17903 > URL: https://issues.apache.org/jira/browse/HIVE-17903 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17903.patch > > > Depends on HIVE-17902 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17809) Implement per pool trigger validation
[ https://issues.apache.org/jira/browse/HIVE-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17809: - Attachment: HIVE-17809.2.patch Addressed review comments. Will need some updates after HIVE-17841. For some reason, RB doesn't allow me to update the patch. Will look into it later. > Implement per pool trigger validation > - > > Key: HIVE-17809 > URL: https://issues.apache.org/jira/browse/HIVE-17809 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17809.1.patch, HIVE-17809.2.patch > > > HIVE-17508 trigger validation is applied for all pools at once. This is > follow up to implement trigger validation at per pool level. > This should also implement resolution for multiple applicable actions, as per > the RB discussion -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17778) Add support for custom counters in trigger expression
[ https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17778: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the reviews! > Add support for custom counters in trigger expression > - > > Key: HIVE-17778 > URL: https://issues.apache.org/jira/browse/HIVE-17778 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-17778.1.patch, HIVE-17778.2.patch, > HIVE-17778.3.patch, HIVE-17778.4.patch, HIVE-17778.5.patch, > HIVE-17778.6.patch, HIVE-17778.7.patch > > > HIVE-17508 only supports limited counters. This ticket is to extend it to > support custom counters (counters that are not supported by execution engine > will be dropped). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17778) Add support for custom counters in trigger expression
[ https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223088#comment-16223088 ] Prasanth Jayachandran commented on HIVE-17778: -- testTriggerHighShuffleBytes failure is handled in HIVE-17834. Will commit the patch shortly > Add support for custom counters in trigger expression > - > > Key: HIVE-17778 > URL: https://issues.apache.org/jira/browse/HIVE-17778 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17778.1.patch, HIVE-17778.2.patch, > HIVE-17778.3.patch, HIVE-17778.4.patch, HIVE-17778.5.patch, > HIVE-17778.6.patch, HIVE-17778.7.patch > > > HIVE-17508 only supports limited counters. This ticket is to extend it to > support custom counters (counters that are not supported by execution engine > will be dropped). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17778) Add support for custom counters in trigger expression
[
https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223086#comment-16223086
]
Hive QA commented on HIVE-17778:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894429/HIVE-17778.7.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11341 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7520/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7520/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7520/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894429 - PreCommit-HIVE-Build
> Add support for custom counters in trigger expression
> -
>
> Key: HIVE-17778
> URL: https://issues.apache.org/jira/browse/HIVE-17778
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17778.1.patch, HIVE-17778.2.patch,
> HIVE-17778.3.patch, HIVE-17778.4.patch, HIVE-17778.5.patch,
> HIVE-17778.6.patch, HIVE-17778.7.patch
>
>
> HIVE-17508 only supports limited counters. This ticket is to extend it to
> support custom counters (counters that are not supported by execution engine
> will be dropped).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17766) Support non-equi LEFT SEMI JOIN
[
https://issues.apache.org/jira/browse/HIVE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223084#comment-16223084
]
Jesus Camacho Rodriguez commented on HIVE-17766:
[~ashutoshc], some of these failures were legit. I have uploaded a new patch
and updated RB. Could you take a look? The main problem was in the logic that
generates the right input for the left semijoin. I had to do some work to
propagate the aliases correctly till the join operator, otherwise the execution
would fail when we try to parse the residual predicates in the join as column
references cannot be resolved.
> Support non-equi LEFT SEMI JOIN
> ---
>
> Key: HIVE-17766
> URL: https://issues.apache.org/jira/browse/HIVE-17766
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17766.01.patch, HIVE-17766.02.patch,
> HIVE-17766.patch
>
>
> Currently we get an error like {noformat}Non equality condition not supported
> in Semi-Join{noformat}
> This is required to generate better plan for EXISTS/IN correlated subquery
> where such queries are transformed into LEFT SEMI JOIN.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17766) Support non-equi LEFT SEMI JOIN
[
https://issues.apache.org/jira/browse/HIVE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17766:
---
Attachment: HIVE-17766.02.patch
> Support non-equi LEFT SEMI JOIN
> ---
>
> Key: HIVE-17766
> URL: https://issues.apache.org/jira/browse/HIVE-17766
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17766.01.patch, HIVE-17766.02.patch,
> HIVE-17766.patch
>
>
> Currently we get an error like {noformat}Non equality condition not supported
> in Semi-Join{noformat}
> This is required to generate better plan for EXISTS/IN correlated subquery
> where such queries are transformed into LEFT SEMI JOIN.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223081#comment-16223081 ] Prasanth Jayachandran commented on HIVE-15016: -- [~aplusplus] looks like tez related issue? > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, > HIVE-15016.7.patch, HIVE-15016.8.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-16952) AcidUtils.parseBaseOrDeltaBucketFilename() end clause
[
https://issues.apache.org/jira/browse/HIVE-16952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-16952:
-
Assignee: Eugene Koifman
> AcidUtils.parseBaseOrDeltaBucketFilename() end clause
> -
>
> Key: HIVE-16952
> URL: https://issues.apache.org/jira/browse/HIVE-16952
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
>
> The end of this method
> {noformat}
> } else {
> result.setOldStyle(true).bucket(-1).minimumTransactionId(0)
> .maximumTransactionId(0);
> }
> {noformat}
> should this throw instead? bucket == -1 can't be handled by anything in
> OrcRawRecordMerger or anywhere else
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223037#comment-16223037 ] Ashutosh Chauhan commented on HIVE-15016: - cc: [~prasanth_j] > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, > HIVE-15016.7.patch, HIVE-15016.8.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17926) Support triggers for non-pool sessions
[ https://issues.apache.org/jira/browse/HIVE-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17926: - Attachment: HIVE-17926.1.patch [~sershe] can you please review this patch? > Support triggers for non-pool sessions > -- > > Key: HIVE-17926 > URL: https://issues.apache.org/jira/browse/HIVE-17926 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17926.1.patch > > > Current trigger implementation works only with tez session pools. In case > when tez sessions pools are not used, a new session gets created for every > query in which case trigger validation does not happen. It will be good to > support such one-off session case as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17926) Support triggers for non-pool sessions
[ https://issues.apache.org/jira/browse/HIVE-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17926: - Status: Patch Available (was: Open) > Support triggers for non-pool sessions > -- > > Key: HIVE-17926 > URL: https://issues.apache.org/jira/browse/HIVE-17926 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17926.1.patch > > > Current trigger implementation works only with tez session pools. In case > when tez sessions pools are not used, a new session gets created for every > query in which case trigger validation does not happen. It will be good to > support such one-off session case as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17888) Display the reason for query cancellation
[
https://issues.apache.org/jira/browse/HIVE-17888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223003#comment-16223003
]
Prasanth Jayachandran commented on HIVE-17888:
--
Done
> Display the reason for query cancellation
> -
>
> Key: HIVE-17888
> URL: https://issues.apache.org/jira/browse/HIVE-17888
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17888.1.patch, HIVE-17888.2.patch
>
>
> For user convenience and easy debugging, if a trigger kills a query return
> the reason for the killing the query. Currently the query kill will only
> display the following which is not very useful
> {code}
> Error: Query was cancelled (state=01000,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17903) implement query mapping to WM and non-WM based on policies
[ https://issues.apache.org/jira/browse/HIVE-17903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17903: --- Assignee: Sergey Shelukhin > implement query mapping to WM and non-WM based on policies > --- > > Key: HIVE-17903 > URL: https://issues.apache.org/jira/browse/HIVE-17903 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Depends on HIVE-17902 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Issue Type: New Feature (was: Bug) > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.1.patch, HIVE-17361.2.patch, > HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17833) Publish split generation counters
[ https://issues.apache.org/jira/browse/HIVE-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222984#comment-16222984 ] Sergey Shelukhin commented on HIVE-17833: - Is there a RB? > Publish split generation counters > - > > Key: HIVE-17833 > URL: https://issues.apache.org/jira/browse/HIVE-17833 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17833.1.patch, HIVE-17833.2.patch, > HIVE-17833.3.patch > > > With TEZ-3856, tez counters are exposed via input initializers which can be > used to publish split generation counters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17888) Display the reason for query cancellation
[
https://issues.apache.org/jira/browse/HIVE-17888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222980#comment-16222980
]
Sergey Shelukhin commented on HIVE-17888:
-
Looks like RB was not updated
> Display the reason for query cancellation
> -
>
> Key: HIVE-17888
> URL: https://issues.apache.org/jira/browse/HIVE-17888
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17888.1.patch, HIVE-17888.2.patch
>
>
> For user convenience and easy debugging, if a trigger kills a query return
> the reason for the killing the query. Currently the query kill will only
> display the following which is not very useful
> {code}
> Error: Query was cancelled (state=01000,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17884) Implement create, alter and drop workload management triggers.
[ https://issues.apache.org/jira/browse/HIVE-17884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222974#comment-16222974 ] Sergey Shelukhin commented on HIVE-17884: - TestOperation2Privilege probably needs an update like the ADDENDUM2 commit for HIVE-17771 > Implement create, alter and drop workload management triggers. > -- > > Key: HIVE-17884 > URL: https://issues.apache.org/jira/browse/HIVE-17884 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17884.01.patch, HIVE-17884.02.patch > > > Implement triggers for workload management: > The commands to be implemented: > CREATE TRIGGER `resourceplan_name`.`trigger_name` WHEN condition DO action; > condition is a boolean expression: variable operator value types with 'AND' > and 'OR' support. > action is currently: KILL or MOVE TO pool; > ALTER TRIGGER `plan_name`.`trigger_name` WHEN condition DO action; > DROP TRIGGER `plan_name`.`trigger_name`; > Also add WM_TRIGGERS to information schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17898) Explain plan output enhancement
[ https://issues.apache.org/jira/browse/HIVE-17898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17898: --- Status: Patch Available (was: Open) > Explain plan output enhancement > --- > > Key: HIVE-17898 > URL: https://issues.apache.org/jira/browse/HIVE-17898 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17898.1.patch > > > We would like to enhance the explain plan output to display additional > information e.g.: > TableScan operator should have following additional info > * Actual table name (currently only alias name is displayed) > * Database name > * Column names being scanned -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17898) Explain plan output enhancement
[ https://issues.apache.org/jira/browse/HIVE-17898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17898: --- Status: Open (was: Patch Available) > Explain plan output enhancement > --- > > Key: HIVE-17898 > URL: https://issues.apache.org/jira/browse/HIVE-17898 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17898.1.patch > > > We would like to enhance the explain plan output to display additional > information e.g.: > TableScan operator should have following additional info > * Actual table name (currently only alias name is displayed) > * Database name > * Column names being scanned -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17884) Implement create, alter and drop workload management triggers.
[ https://issues.apache.org/jira/browse/HIVE-17884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222959#comment-16222959 ] Prasanth Jayachandran commented on HIVE-17884: -- Test failures look related. > Implement create, alter and drop workload management triggers. > -- > > Key: HIVE-17884 > URL: https://issues.apache.org/jira/browse/HIVE-17884 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17884.01.patch, HIVE-17884.02.patch > > > Implement triggers for workload management: > The commands to be implemented: > CREATE TRIGGER `resourceplan_name`.`trigger_name` WHEN condition DO action; > condition is a boolean expression: variable operator value types with 'AND' > and 'OR' support. > action is currently: KILL or MOVE TO pool; > ALTER TRIGGER `plan_name`.`trigger_name` WHEN condition DO action; > DROP TRIGGER `plan_name`.`trigger_name`; > Also add WM_TRIGGERS to information schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222957#comment-16222957
]
Aihua Xu commented on HIVE-15016:
-
[~ashutoshc] do you know why we are seeing the following exceptions?
{noformat}
Capacity: -1547
at java.util.ArrayList.(ArrayList.java:156)
at
org.apache.tez.mapreduce.grouper.TezSplitGrouper.getGroupedSplits(TezSplitGrouper.java:189)
at
org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:81)
at org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:91)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:193)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:160)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:207)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch,
> HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch,
> HIVE-15016.7.patch, HIVE-15016.8.patch, HIVE-15016.patch,
> Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0
> GA is released.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17841) implement applying the resource plan
[ https://issues.apache.org/jira/browse/HIVE-17841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222950#comment-16222950 ] Prasanth Jayachandran commented on HIVE-17841: -- +1, pending tests > implement applying the resource plan > > > Key: HIVE-17841 > URL: https://issues.apache.org/jira/browse/HIVE-17841 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17841.01.patch, HIVE-17841.02.patch, > HIVE-17841.03.patch, HIVE-17841.04.patch, HIVE-17841.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17841) implement applying the resource plan
[ https://issues.apache.org/jira/browse/HIVE-17841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17841: Attachment: HIVE-17841.04.patch Addressing the CR feedback and fixing the tests > implement applying the resource plan > > > Key: HIVE-17841 > URL: https://issues.apache.org/jira/browse/HIVE-17841 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17841.01.patch, HIVE-17841.02.patch, > HIVE-17841.03.patch, HIVE-17841.04.patch, HIVE-17841.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17926) Support triggers for non-pool sessions
[ https://issues.apache.org/jira/browse/HIVE-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-17926: > Support triggers for non-pool sessions > -- > > Key: HIVE-17926 > URL: https://issues.apache.org/jira/browse/HIVE-17926 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > Current trigger implementation works only with tez session pools. In case > when tez sessions pools are not used, a new session gets created for every > query in which case trigger validation does not happen. It will be good to > support such one-off session case as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15016: Status: Patch Available (was: In Progress) > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, > HIVE-15016.7.patch, HIVE-15016.8.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15016: Attachment: HIVE-15016.8.patch patch-8: increase maven memory setting. > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, > HIVE-15016.7.patch, HIVE-15016.8.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15016: Status: In Progress (was: Patch Available) > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, > HIVE-15016.7.patch, HIVE-15016.patch, Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222901#comment-16222901
]
Hive QA commented on HIVE-17433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894421/HIVE-17433.091.patch
{color:green}SUCCESS:{color} +1 due to 48 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 11327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid_fast]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_shufflejoin]
(batchId=134)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7519/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7519/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7519/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894421 - PreCommit-HIVE-Build
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch, HIVE-17433.091.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
>
[jira] [Commented] (HIVE-17902) add a notions of default pool and unmanaged mapping part 1
[ https://issues.apache.org/jira/browse/HIVE-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222871#comment-16222871 ] Prasanth Jayachandran commented on HIVE-17902: -- +1, pending tests > add a notions of default pool and unmanaged mapping part 1 > -- > > Key: HIVE-17902 > URL: https://issues.apache.org/jira/browse/HIVE-17902 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17902.01.patch, HIVE-17902.patch > > > This is needed to map queries between WM and non-WM execution -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17918) NPE during semijoin reduction optimization when LLAP caching disabled
[
https://issues.apache.org/jira/browse/HIVE-17918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222870#comment-16222870
]
Sergey Shelukhin commented on HIVE-17918:
-
+1 pending tests
> NPE during semijoin reduction optimization when LLAP caching disabled
> -
>
> Key: HIVE-17918
> URL: https://issues.apache.org/jira/browse/HIVE-17918
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-17918.1.patch, HIVE-17918.2.patch
>
>
> DynamicValue (used by semijoin reduction optimization) relies on the
> ObjectCache. If LLAP cache is disabled then the DynamicValue is broken in
> LLAP:
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:254)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> ... 18 more
> Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value
> for RS_25_household_demographics_hd_demo_sk_min
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterLongColumnBetweenDynamicValue.evaluate(FilterLongColumnBetweenDynamicValue.java:80)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:39)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:41)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:112)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:959)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:907)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:137)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:828)
> ... 19 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:61)
> at
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:50)
> at
> org.apache.hadoop.hive.ql.exec.ObjectCacheWrapper.retrieve(ObjectCacheWrapper.java:40)
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:123)
>
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Status: Patch Available (was: Open) > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17888) Display the reason for query cancellation
[
https://issues.apache.org/jira/browse/HIVE-17888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-17888:
-
Attachment: HIVE-17888.2.patch
Renamed to reason to errorMessage to make it consistent. Also removed an unused
cancel method overload in Operation.
> Display the reason for query cancellation
> -
>
> Key: HIVE-17888
> URL: https://issues.apache.org/jira/browse/HIVE-17888
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17888.1.patch, HIVE-17888.2.patch
>
>
> For user convenience and easy debugging, if a trigger kills a query return
> the reason for the killing the query. Currently the query kill will only
> display the following which is not very useful
> {code}
> Error: Query was cancelled (state=01000,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17902) add a notions of default pool and unmanaged mapping part 1
[ https://issues.apache.org/jira/browse/HIVE-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17902: Attachment: HIVE-17902.01.patch Addressed the RB feedback. Also HiveQA would hopefully run now > add a notions of default pool and unmanaged mapping part 1 > -- > > Key: HIVE-17902 > URL: https://issues.apache.org/jira/browse/HIVE-17902 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17902.01.patch, HIVE-17902.patch > > > This is needed to map queries between WM and non-WM execution -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17900) analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column
[
https://issues.apache.org/jira/browse/HIVE-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-17900:
-
Assignee: (was: Eugene Koifman)
> analyze stats on columns triggered by Compactor generates malformed SQL with
> > 1 partition column
> -
>
> Key: HIVE-17900
> URL: https://issues.apache.org/jira/browse/HIVE-17900
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>
> {noformat}
> 2017-10-16 09:01:51,255 ERROR [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (SessionState.java:printError(993)) - FAILED: ParseException line
> 1:70 mismatched input 'dates' expecting ) near ''201608'' in analyze statement
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:70 mismatched input
> 'dates' expecting ) near ''201608'' in analyze statement
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:205)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1221)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1158)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1148)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:294)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - start=1508162511253 end=1508162511255 duration=2
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (Driver.java:compile(559)) - We are resetting the hadoop caller
> context to
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - method=releaseLocks start=1508162511255 end=1508162511255 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionPoolManager (TezSessionPoolManager.java:close(183)) - Closing
> tez session default? false
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionState (TezSessionState.java:close(294)) - Closing Tez Session
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> client.TezClient (TezClient.java:stop(518)) - Shutting down Tez Session,
> sessionName=HIVE-ae652f03-72c7-4ca8-a2d8-05dcc7392f4f,
> applicationId=application_1507779664083_0159
> 2017-10-16 09:01:51,279 ERROR [haddl0007.mycenterpointenergy.com-51]:
> compactor.Worker (Worker.java:run(191)) - Caught exception while trying to
> compact
> id:3723,dbname:mobiusad,tableName:zces_img_data_small_pt,partName:month=201608/dates=9,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking failed to avoid repeated failures, java.io.IOException: Could not
> update stats for table mobiusad.zces_img_data_small_pt/month=201608/dates=9
> due to: (4,FAILED: ParseException line 1:70 mismatched input 'dates'
> expecting ) near ''201608'' in analyze statement,42000line 1:70 mismatched
> input 'dates' expecting ) near ''201608'' in analyze statement)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:296)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17900) analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column
[
https://issues.apache.org/jira/browse/HIVE-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-17900:
--
Affects Version/s: 1.0.0
> analyze stats on columns triggered by Compactor generates malformed SQL with
> > 1 partition column
> -
>
> Key: HIVE-17900
> URL: https://issues.apache.org/jira/browse/HIVE-17900
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>
> {noformat}
> 2017-10-16 09:01:51,255 ERROR [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (SessionState.java:printError(993)) - FAILED: ParseException line
> 1:70 mismatched input 'dates' expecting ) near ''201608'' in analyze statement
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:70 mismatched input
> 'dates' expecting ) near ''201608'' in analyze statement
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:205)
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:321)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1221)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1158)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1148)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:294)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - start=1508162511253 end=1508162511255 duration=2
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> ql.Driver (Driver.java:compile(559)) - We are resetting the hadoop caller
> context to
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) - method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,255 INFO [haddl0007.mycenterpointenergy.com-51]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) - method=releaseLocks start=1508162511255 end=1508162511255 duration=0
> from=org.apache.hadoop.hive.ql.Driver>
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionPoolManager (TezSessionPoolManager.java:close(183)) - Closing
> tez session default? false
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> tez.TezSessionState (TezSessionState.java:close(294)) - Closing Tez Session
> 2017-10-16 09:01:51,256 INFO [haddl0007.mycenterpointenergy.com-51]:
> client.TezClient (TezClient.java:stop(518)) - Shutting down Tez Session,
> sessionName=HIVE-ae652f03-72c7-4ca8-a2d8-05dcc7392f4f,
> applicationId=application_1507779664083_0159
> 2017-10-16 09:01:51,279 ERROR [haddl0007.mycenterpointenergy.com-51]:
> compactor.Worker (Worker.java:run(191)) - Caught exception while trying to
> compact
> id:3723,dbname:mobiusad,tableName:zces_img_data_small_pt,partName:month=201608/dates=9,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.
> Marking failed to avoid repeated failures, java.io.IOException: Could not
> update stats for table mobiusad.zces_img_data_small_pt/month=201608/dates=9
> due to: (4,FAILED: ParseException line 1:70 mismatched input 'dates'
> expecting ) near ''201608'' in analyze statement,42000line 1:70 mismatched
> input 'dates' expecting ) near ''201608'' in analyze statement)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:296)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:168)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17834) Fix flaky triggers test
[ https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222838#comment-16222838 ] Prasanth Jayachandran commented on HIVE-17834: -- Few more tries. > Fix flaky triggers test > --- > > Key: HIVE-17834 > URL: https://issues.apache.org/jira/browse/HIVE-17834 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17834.1.patch, HIVE-17834.2.patch, > HIVE-17834.3.patch, HIVE-17834.4.patch > > > https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17458) VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
[ https://issues.apache.org/jira/browse/HIVE-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17458: -- Attachment: HIVE-17458.11.patch same as patch 10 > VectorizedOrcAcidRowBatchReader doesn't handle 'original' files > --- > > Key: HIVE-17458 > URL: https://issues.apache.org/jira/browse/HIVE-17458 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17458.01.patch, HIVE-17458.02.patch, > HIVE-17458.03.patch, HIVE-17458.04.patch, HIVE-17458.05.patch, > HIVE-17458.06.patch, HIVE-17458.07.patch, HIVE-17458.07.patch, > HIVE-17458.08.patch, HIVE-17458.09.patch, HIVE-17458.10.patch, > HIVE-17458.11.patch > > > VectorizedOrcAcidRowBatchReader will not be used for original files. This > will likely look like a perf regression when converting a table from non-acid > to acid until it runs through a major compaction. > With Load Data support, if large files are added via Load Data, the read ops > will not vectorize until major compaction. > There is no reason why this should be the case. Just like > OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other > files in the logical tranche/bucket and calculate the offset for the RowBatch > of the split. (Presumably getRecordReader().getRowNumber() works the same in > vector mode). > In this case we don't even need OrcSplit.isOriginal() - the reader can infer > it from file path... which in particular simplifies > OrcInputFormat.determineSplitStrategies() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Attachment: HIVE-17767.1.patch First patch removes earlier commit (which reverted another commit) and adds a rewrite of correlated EXISTS/IN into LEFT SEMI JOIN. This patch isn't yet ready to be committed since HIVE-17766 isn't in yet. > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17834) Fix flaky triggers test
[ https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17834: - Attachment: HIVE-17834.4.patch > Fix flaky triggers test > --- > > Key: HIVE-17834 > URL: https://issues.apache.org/jira/browse/HIVE-17834 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17834.1.patch, HIVE-17834.2.patch, > HIVE-17834.3.patch, HIVE-17834.4.patch > > > https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17925) Fix TestHooks so that it avoids ClassNotFound on teardown
[ https://issues.apache.org/jira/browse/HIVE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley reassigned HIVE-17925: > Fix TestHooks so that it avoids ClassNotFound on teardown > - > > Key: HIVE-17925 > URL: https://issues.apache.org/jira/browse/HIVE-17925 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > > TestHooks gets a ClassNotFound exception during teardown, which messes up > some following tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-9447) Metastore: inefficient Oracle query for removing unused column descriptors when add/drop table/partition
[
https://issues.apache.org/jira/browse/HIVE-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-9447:
-
Status: Patch Available (was: In Progress)
> Metastore: inefficient Oracle query for removing unused column descriptors
> when add/drop table/partition
>
>
> Key: HIVE-9447
> URL: https://issues.apache.org/jira/browse/HIVE-9447
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.1.0, 1.2.0, 1.0.0, 0.14.0
>Reporter: Selina Zhang
>Assignee: Adam Szita
> Attachments: HIVE-9447.1.patch, HIVE-9447.2.patch, HIVE-9447.3.patch,
> HIVE-9447.4.patch, HIVE-9447.5.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Metastore needs removing unused column descriptors when drop/add partitions
> or tables. For query the unused column descriptor, the current implementation
> utilizes datanuleus' range function, which basically equals LIMIT syntax.
> However, Oracle does not support LIMIT, the query is converted as
> {quote}
> SQL> SELECT * FROM (SELECT subq.*,ROWNUM rn FROM (SELECT
> 'org.apache.hadoop.hive.metastore.model.MStorageDescriptor' AS
> NUCLEUS_TYPE,A0.INPUT_FORMAT,A0.IS_COMPRESSED,A0.IS_STOREDASSUBDIRECTORIES,A0.LOCATION,
> A0.NUM_BUCKETS,A0.OUTPUT_FORMAT,A0.SD_ID FROM drhcat.SDS A0
> WHERE A0.CD_ID = ? ) subq ) WHERE rn <= 1;
> {quote}
> Given that CD_ID is not very selective, this query may have to access large
> amount of rows (depends how many partitions the table has, millions of rows
> in our case). Metastore may become unresponsive because of this.
> Since Metastore only needs to know if the specific CD_ID is referenced in SDS
> table and does not need access the whole row. We can use
> {quote}
> select count(1) from SDS where SDS.CD_ID=?
> {quote}
> CD_ID is index column, the above query will do range scan for index, which is
> faster.
> For other DBs support LIMIT syntax such as MySQL, this problem does not
> exist. However, the new query does not hurt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-9447) Metastore: inefficient Oracle query for removing unused column descriptors when add/drop table/partition
[
https://issues.apache.org/jira/browse/HIVE-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-9447:
-
Status: In Progress (was: Patch Available)
> Metastore: inefficient Oracle query for removing unused column descriptors
> when add/drop table/partition
>
>
> Key: HIVE-9447
> URL: https://issues.apache.org/jira/browse/HIVE-9447
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.1.0, 1.2.0, 1.0.0, 0.14.0
>Reporter: Selina Zhang
>Assignee: Adam Szita
> Attachments: HIVE-9447.1.patch, HIVE-9447.2.patch, HIVE-9447.3.patch,
> HIVE-9447.4.patch, HIVE-9447.5.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Metastore needs removing unused column descriptors when drop/add partitions
> or tables. For query the unused column descriptor, the current implementation
> utilizes datanuleus' range function, which basically equals LIMIT syntax.
> However, Oracle does not support LIMIT, the query is converted as
> {quote}
> SQL> SELECT * FROM (SELECT subq.*,ROWNUM rn FROM (SELECT
> 'org.apache.hadoop.hive.metastore.model.MStorageDescriptor' AS
> NUCLEUS_TYPE,A0.INPUT_FORMAT,A0.IS_COMPRESSED,A0.IS_STOREDASSUBDIRECTORIES,A0.LOCATION,
> A0.NUM_BUCKETS,A0.OUTPUT_FORMAT,A0.SD_ID FROM drhcat.SDS A0
> WHERE A0.CD_ID = ? ) subq ) WHERE rn <= 1;
> {quote}
> Given that CD_ID is not very selective, this query may have to access large
> amount of rows (depends how many partitions the table has, millions of rows
> in our case). Metastore may become unresponsive because of this.
> Since Metastore only needs to know if the specific CD_ID is referenced in SDS
> table and does not need access the whole row. We can use
> {quote}
> select count(1) from SDS where SDS.CD_ID=?
> {quote}
> CD_ID is index column, the above query will do range scan for index, which is
> faster.
> For other DBs support LIMIT syntax such as MySQL, this problem does not
> exist. However, the new query does not hurt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-9447) Metastore: inefficient Oracle query for removing unused column descriptors when add/drop table/partition
[
https://issues.apache.org/jira/browse/HIVE-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-9447:
-
Attachment: HIVE-9447.5.patch
> Metastore: inefficient Oracle query for removing unused column descriptors
> when add/drop table/partition
>
>
> Key: HIVE-9447
> URL: https://issues.apache.org/jira/browse/HIVE-9447
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Selina Zhang
>Assignee: Adam Szita
> Attachments: HIVE-9447.1.patch, HIVE-9447.2.patch, HIVE-9447.3.patch,
> HIVE-9447.4.patch, HIVE-9447.5.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Metastore needs removing unused column descriptors when drop/add partitions
> or tables. For query the unused column descriptor, the current implementation
> utilizes datanuleus' range function, which basically equals LIMIT syntax.
> However, Oracle does not support LIMIT, the query is converted as
> {quote}
> SQL> SELECT * FROM (SELECT subq.*,ROWNUM rn FROM (SELECT
> 'org.apache.hadoop.hive.metastore.model.MStorageDescriptor' AS
> NUCLEUS_TYPE,A0.INPUT_FORMAT,A0.IS_COMPRESSED,A0.IS_STOREDASSUBDIRECTORIES,A0.LOCATION,
> A0.NUM_BUCKETS,A0.OUTPUT_FORMAT,A0.SD_ID FROM drhcat.SDS A0
> WHERE A0.CD_ID = ? ) subq ) WHERE rn <= 1;
> {quote}
> Given that CD_ID is not very selective, this query may have to access large
> amount of rows (depends how many partitions the table has, millions of rows
> in our case). Metastore may become unresponsive because of this.
> Since Metastore only needs to know if the specific CD_ID is referenced in SDS
> table and does not need access the whole row. We can use
> {quote}
> select count(1) from SDS where SDS.CD_ID=?
> {quote}
> CD_ID is index column, the above query will do range scan for index, which is
> faster.
> For other DBs support LIMIT syntax such as MySQL, this problem does not
> exist. However, the new query does not hurt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17918) NPE during semijoin reduction optimization when LLAP caching disabled
[
https://issues.apache.org/jira/browse/HIVE-17918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-17918:
--
Status: Patch Available (was: Open)
> NPE during semijoin reduction optimization when LLAP caching disabled
> -
>
> Key: HIVE-17918
> URL: https://issues.apache.org/jira/browse/HIVE-17918
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-17918.1.patch, HIVE-17918.2.patch
>
>
> DynamicValue (used by semijoin reduction optimization) relies on the
> ObjectCache. If LLAP cache is disabled then the DynamicValue is broken in
> LLAP:
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:254)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> ... 18 more
> Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value
> for RS_25_household_demographics_hd_demo_sk_min
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterLongColumnBetweenDynamicValue.evaluate(FilterLongColumnBetweenDynamicValue.java:80)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:39)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:41)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:112)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:959)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:907)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:137)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:828)
> ... 19 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:61)
> at
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:50)
> at
> org.apache.hadoop.hive.ql.exec.ObjectCacheWrapper.retrieve(ObjectCacheWrapper.java:40)
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:123)
> ... 27 more
> Caused by:
[jira] [Updated] (HIVE-17918) NPE during semijoin reduction optimization when LLAP caching disabled
[
https://issues.apache.org/jira/browse/HIVE-17918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-17918:
--
Attachment: HIVE-17918.2.patch
Revising patch per [~sershe]'s feedback
> NPE during semijoin reduction optimization when LLAP caching disabled
> -
>
> Key: HIVE-17918
> URL: https://issues.apache.org/jira/browse/HIVE-17918
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-17918.1.patch, HIVE-17918.2.patch
>
>
> DynamicValue (used by semijoin reduction optimization) relies on the
> ObjectCache. If LLAP cache is disabled then the DynamicValue is broken in
> LLAP:
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:283)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:237)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:254)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> ... 18 more
> Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value
> for RS_25_household_demographics_hd_demo_sk_min
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterLongColumnBetweenDynamicValue.evaluate(FilterLongColumnBetweenDynamicValue.java:80)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:39)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:41)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:112)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:959)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:907)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:137)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:828)
> ... 19 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:61)
> at
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:50)
> at
> org.apache.hadoop.hive.ql.exec.ObjectCacheWrapper.retrieve(ObjectCacheWrapper.java:40)
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:123)
[jira] [Updated] (HIVE-17891) HIVE-13076 uses create table if not exists for the postgres script
[
https://issues.apache.org/jira/browse/HIVE-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17891:
---
Attachment: HIVE-17891.04.patch
> HIVE-13076 uses create table if not exists for the postgres script
> --
>
> Key: HIVE-17891
> URL: https://issues.apache.org/jira/browse/HIVE-17891
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17891.01.patch, HIVE-17891.02.patch,
> HIVE-17891.03.patch, HIVE-17891.04.patch
>
>
> HIVE-13076 addes a new table to the schema but the patch script uses {{CREATE
> TABLE IF NOT EXISTS}} syntax to add the new table. The issue is that the {{IF
> NOT EXISTS}} clause is only available from postgres 9.1 onwards. So the
> script will fail for older versions of postgres.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17748) ReplCopyTask doesn't support multi-file CopyWork
[
https://issues.apache.org/jira/browse/HIVE-17748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17748:
Attachment: HIVE-17748.01.patch
HiveQA
> ReplCopyTask doesn't support multi-file CopyWork
>
>
> Key: HIVE-17748
> URL: https://issues.apache.org/jira/browse/HIVE-17748
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17748.01.patch, HIVE-17748.patch
>
>
> has
> {noformat}
> Path fromPath = work.getFromPaths()[0];
> toPath = work.getToPaths()[0];
> {noformat}
> should this throw if from/to paths have > 1 element?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17673) JavaUtils.extractTxnId() etc
[ https://issues.apache.org/jira/browse/HIVE-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17673: Attachment: (was: HIVE-17673.patch) > JavaUtils.extractTxnId() etc > > > Key: HIVE-17673 > URL: https://issues.apache.org/jira/browse/HIVE-17673 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HIVE-17673.01.patch, HIVE-17673.patch > > > these should be in AcidUtils -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17673) JavaUtils.extractTxnId() etc
[ https://issues.apache.org/jira/browse/HIVE-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17673: Attachment: HIVE-17673.01.patch HiveQA > JavaUtils.extractTxnId() etc > > > Key: HIVE-17673 > URL: https://issues.apache.org/jira/browse/HIVE-17673 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HIVE-17673.01.patch, HIVE-17673.patch > > > these should be in AcidUtils -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17748) ReplCopyTask doesn't support multi-file CopyWork
[
https://issues.apache.org/jira/browse/HIVE-17748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17748:
Attachment: (was: HIVE-17748.patch)
> ReplCopyTask doesn't support multi-file CopyWork
>
>
> Key: HIVE-17748
> URL: https://issues.apache.org/jira/browse/HIVE-17748
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17748.patch
>
>
> has
> {noformat}
> Path fromPath = work.getFromPaths()[0];
> toPath = work.getToPaths()[0];
> {noformat}
> should this throw if from/to paths have > 1 element?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17698) FileSinkDesk.getMergeInputDirName() uses stmtId=0
[ https://issues.apache.org/jira/browse/HIVE-17698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17698: Attachment: HIVE-17698.02.patch Again for hiveqa > FileSinkDesk.getMergeInputDirName() uses stmtId=0 > - > > Key: HIVE-17698 > URL: https://issues.apache.org/jira/browse/HIVE-17698 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin > Attachments: HIVE-17698.01.patch, HIVE-17698.02.patch, > HIVE-17698.patch, HIVE-17698.patch > > > this is certainly wrong for multi statement txn but may also affect writes > from Union All queries if these are made to follow full Acid convention > _return new Path(root, AcidUtils.deltaSubdir(txnId, txnId, 0));_ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17750) add a flag to automatically create most tables as MM
[ https://issues.apache.org/jira/browse/HIVE-17750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17750: Attachment: HIVE-17750.01.patch Again for hiveqa > add a flag to automatically create most tables as MM > - > > Key: HIVE-17750 > URL: https://issues.apache.org/jira/browse/HIVE-17750 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17750.01.patch, HIVE-17750.patch > > > After merge we are going to do another round of gap identification... similar > to HIVE-14990. > However the approach used there is a huge PITA. It'd be much better to make > tables MM by default at create time, not pretend they are MM at check time, > from the perspective of spurious error elimination. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17858) MM - some union cases are broken
[ https://issues.apache.org/jira/browse/HIVE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17858: Attachment: HIVE-17858.02.patch Again for HiveQA > MM - some union cases are broken > > > Key: HIVE-17858 > URL: https://issues.apache.org/jira/browse/HIVE-17858 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: mm-gap-1 > Attachments: HIVE-17858.01.patch, HIVE-17858.02.patch, > HIVE-17858.patch > > > mm_all test no longer runs on LLAP; if it's executed in LLAP, one can see > that some union cases no longer work. > Queries on partunion_mm, skew_dp_union_mm produce no results. > I'm not sure what part of "integration" broke it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17778) Add support for custom counters in trigger expression
[ https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17778: - Attachment: HIVE-17778.7.patch drop table was missing in the test. > Add support for custom counters in trigger expression > - > > Key: HIVE-17778 > URL: https://issues.apache.org/jira/browse/HIVE-17778 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17778.1.patch, HIVE-17778.2.patch, > HIVE-17778.3.patch, HIVE-17778.4.patch, HIVE-17778.5.patch, > HIVE-17778.6.patch, HIVE-17778.7.patch > > > HIVE-17508 only supports limited counters. This ticket is to extend it to > support custom counters (counters that are not supported by execution engine > will be dropped). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17924) Restore SerDe by reverting HIVE-15167 to unbreak API compatibility
[ https://issues.apache.org/jira/browse/HIVE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley reassigned HIVE-17924: > Restore SerDe by reverting HIVE-15167 to unbreak API compatibility > -- > > Key: HIVE-17924 > URL: https://issues.apache.org/jira/browse/HIVE-17924 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 2.3.1 >Reporter: Owen O'Malley >Assignee: Owen O'Malley > > HIVE-15167 broke compatibility badly for very little gain and caused a lot of > pain for our users. We should revert it and restore the SerDe interface. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Status: In Progress (was: Patch Available)
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch, HIVE-17433.091.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Status: Patch Available (was: In Progress)
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch, HIVE-17433.091.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Attachment: HIVE-17433.091.patch
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch, HIVE-17433.091.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17891) HIVE-13076 uses create table if not exists for the postgres script
[
https://issues.apache.org/jira/browse/HIVE-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17891:
---
Attachment: HIVE-17891.03.patch
precommit is having a lot of issues. Attaching one more time.
> HIVE-13076 uses create table if not exists for the postgres script
> --
>
> Key: HIVE-17891
> URL: https://issues.apache.org/jira/browse/HIVE-17891
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17891.01.patch, HIVE-17891.02.patch,
> HIVE-17891.03.patch
>
>
> HIVE-13076 addes a new table to the schema but the patch script uses {{CREATE
> TABLE IF NOT EXISTS}} syntax to add the new table. The issue is that the {{IF
> NOT EXISTS}} clause is only available from postgres 9.1 onwards. So the
> script will fail for older versions of postgres.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17766) Support non-equi LEFT SEMI JOIN
[
https://issues.apache.org/jira/browse/HIVE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222699#comment-16222699
]
Hive QA commented on HIVE-17766:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894257/HIVE-17766.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 11328 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_partitioner]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[leftsemijoin]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reduce_deduplicate_extended2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subq_where_serialization]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists_having]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_in_having]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqual_corr_expr]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqualcolumnrefs]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[leftsemijoin]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_join_ptp]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_exists]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=173)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[annotate_stats_join]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[leftsemijoin]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[semijoin]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_exists]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multiinsert]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=108)
[jira] [Commented] (HIVE-17826) Error writing to RandomAccessFile after operation log is closed
[
https://issues.apache.org/jira/browse/HIVE-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222677#comment-16222677
]
Andrew Sherman commented on HIVE-17826:
---
Thanks [~aihuaxu], do we need more review? If not can you push this when
convenient?
> Error writing to RandomAccessFile after operation log is closed
> ---
>
> Key: HIVE-17826
> URL: https://issues.apache.org/jira/browse/HIVE-17826
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-17826.1.patch
>
>
> We are seeing the error from HS2 process stdout.
> {noformat}
> 2017-09-07 10:17:23,933 AsyncLogger-1 ERROR Attempted to append to
> non-started appender query-file-appender
> 2017-09-07 10:17:23,934 AsyncLogger-1 ERROR Attempted to append to
> non-started appender query-file-appender
> 2017-09-07 10:17:23,935 AsyncLogger-1 ERROR Unable to write to stream
> /var/log/hive/operation_logs/dd38df5b-3c09-48c9-ad64-a2eee093bea6/hive_20170907101723_1a6ad4b9-f662-4e7a-a495-06e3341308f9
> for appender query-file-appender
> 2017-09-07 10:17:23,935 AsyncLogger-1 ERROR An exception occurred processing
> Appender query-file-appender
> org.apache.logging.log4j.core.appender.AppenderLoggingException: Error
> writing to RandomAccessFile
> /var/log/hive/operation_logs/dd38df5b-3c09-48c9-ad64-a2eee093bea6/hive_20170907101723_1a6ad4b9-f662-4e7a-a495-06e3341308f9
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.flush(RandomAccessFileManager.java:114)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.write(RandomAccessFileManager.java:103)
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:136)
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:105)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileAppender.append(RandomAccessFileAppender.java:89)
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:152)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:125)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:116)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:112)
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:152)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:125)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:116)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:390)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:378)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:362)
> at
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
> at
> org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:385)
> at
> org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:103)
> at
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:43)
> at
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:28)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Stream Closed
> at java.io.RandomAccessFile.writeBytes(Native Method)
> at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.flush(RandomAccessFileManager.java:111)
> ... 25 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17826) Error writing to RandomAccessFile after operation log is closed
[
https://issues.apache.org/jira/browse/HIVE-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222661#comment-16222661
]
Aihua Xu commented on HIVE-17826:
-
[~asherman] Make sense. Let's just take your original fix then. +1.
> Error writing to RandomAccessFile after operation log is closed
> ---
>
> Key: HIVE-17826
> URL: https://issues.apache.org/jira/browse/HIVE-17826
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-17826.1.patch
>
>
> We are seeing the error from HS2 process stdout.
> {noformat}
> 2017-09-07 10:17:23,933 AsyncLogger-1 ERROR Attempted to append to
> non-started appender query-file-appender
> 2017-09-07 10:17:23,934 AsyncLogger-1 ERROR Attempted to append to
> non-started appender query-file-appender
> 2017-09-07 10:17:23,935 AsyncLogger-1 ERROR Unable to write to stream
> /var/log/hive/operation_logs/dd38df5b-3c09-48c9-ad64-a2eee093bea6/hive_20170907101723_1a6ad4b9-f662-4e7a-a495-06e3341308f9
> for appender query-file-appender
> 2017-09-07 10:17:23,935 AsyncLogger-1 ERROR An exception occurred processing
> Appender query-file-appender
> org.apache.logging.log4j.core.appender.AppenderLoggingException: Error
> writing to RandomAccessFile
> /var/log/hive/operation_logs/dd38df5b-3c09-48c9-ad64-a2eee093bea6/hive_20170907101723_1a6ad4b9-f662-4e7a-a495-06e3341308f9
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.flush(RandomAccessFileManager.java:114)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.write(RandomAccessFileManager.java:103)
> at
> org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:136)
> at
> org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:105)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileAppender.append(RandomAccessFileAppender.java:89)
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:152)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:125)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:116)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:112)
> at
> org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:152)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:125)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:116)
> at
> org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:390)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:378)
> at
> org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:362)
> at
> org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
> at
> org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:385)
> at
> org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:103)
> at
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:43)
> at
> org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:28)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Stream Closed
> at java.io.RandomAccessFile.writeBytes(Native Method)
> at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
> at
> org.apache.logging.log4j.core.appender.RandomAccessFileManager.flush(RandomAccessFileManager.java:111)
> ... 25 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14305) To/From UTC timestamp may return incorrect result because of DST
[
https://issues.apache.org/jira/browse/HIVE-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222633#comment-16222633
]
Ryan Blue commented on HIVE-14305:
--
[~lirui], the behavior for those functions is already well established and
works fine with the {{timestamp without time zone}} / {{timestamp}} type. I
would be interested to hear your thoughts on implementing {{timestamp with time
zone}}, but I don't think that we need that type to fix bugs.
> To/From UTC timestamp may return incorrect result because of DST
>
>
> Key: HIVE-14305
> URL: https://issues.apache.org/jira/browse/HIVE-14305
> Project: Hive
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Labels: timestamp
>
> If the machine's local timezone involves DST, the UDFs return incorrect
> results.
> For example:
> {code}
> select to_utc_timestamp('2005-04-03 02:01:00','UTC');
> {code}
> returns {{2005-04-03 03:01:00}}. Correct result should be {{2005-04-03
> 02:01:00}}.
> {code}
> select to_utc_timestamp('2005-04-03 10:01:00','Asia/Shanghai');
> {code}
> returns {{2005-04-03 03:01:00}}. Correct result should be {{2005-04-03
> 02:01:00}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-12195) Unknown zones should cause an error instead of silently failing
[
https://issues.apache.org/jira/browse/HIVE-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222616#comment-16222616
]
Hive QA commented on HIVE-12195:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784921/HIVE-12195.4.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11312 tests
executed
*Failed tests:*
{noformat}
TestMetaStoreLimitPartitionRequest - did not produce a TEST-*.xml file (likely
timed out) (batchId=222)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=234)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7506/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7506/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784921 - PreCommit-HIVE-Build
> Unknown zones should cause an error instead of silently failing
> ---
>
> Key: HIVE-12195
> URL: https://issues.apache.org/jira/browse/HIVE-12195
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Ryan Blue
>Assignee: Shinichi Yamashita
> Labels: timestamp
> Attachments: HIVE-12195.1.patch, HIVE-12195.2.patch,
> HIVE-12195.3.patch, HIVE-12195.4.patch
>
>
> Using an unknown time zone with the {{from_utc_timestamp}} or
> {{to_utc_timetamp}} methods returns the time un-adjusted instead of throwing
> an error:
> {code}
> hive> select from_utc_timestamp('2015-04-11 12:24:34.535', 'panda');
> OK
> 2015-04-11 12:24:34.535
> {code}
> This should be an error because users may attempt to adjust to valid but
> unknown zones, like PDT or MDT. This would produce incorrect results with no
> warning or error.
> *Update*: A good work-around is to add a table of known zones that maps to
> offset zone identifiers, like {{GMT-07:00}}. The table is small enough to
> always be a broadcast join and results can be filtered (e.g. {{offset_zone IS
> NOT NULL}}) so that only valid zones are passed to {{from_utc_timestamp}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17458) VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
[ https://issues.apache.org/jira/browse/HIVE-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222604#comment-16222604 ] Eugene Koifman commented on HIVE-17458: --- [~sershe] could you review please > VectorizedOrcAcidRowBatchReader doesn't handle 'original' files > --- > > Key: HIVE-17458 > URL: https://issues.apache.org/jira/browse/HIVE-17458 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17458.01.patch, HIVE-17458.02.patch, > HIVE-17458.03.patch, HIVE-17458.04.patch, HIVE-17458.05.patch, > HIVE-17458.06.patch, HIVE-17458.07.patch, HIVE-17458.07.patch, > HIVE-17458.08.patch, HIVE-17458.09.patch, HIVE-17458.10.patch > > > VectorizedOrcAcidRowBatchReader will not be used for original files. This > will likely look like a perf regression when converting a table from non-acid > to acid until it runs through a major compaction. > With Load Data support, if large files are added via Load Data, the read ops > will not vectorize until major compaction. > There is no reason why this should be the case. Just like > OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other > files in the logical tranche/bucket and calculate the offset for the RowBatch > of the split. (Presumably getRecordReader().getRowNumber() works the same in > vector mode). > In this case we don't even need OrcSplit.isOriginal() - the reader can infer > it from file path... which in particular simplifies > OrcInputFormat.determineSplitStrategies() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17922) Enable runWorker() UDF to launch compactor from .q tests
[
https://issues.apache.org/jira/browse/HIVE-17922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-17922:
-
> Enable runWorker() UDF to launch compactor from .q tests
>
>
> Key: HIVE-17922
> URL: https://issues.apache.org/jira/browse/HIVE-17922
> Project: Hive
> Issue Type: Sub-task
> Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> available since HIVE-17458 (via UDFRunWorker.java)
> The idea is to be able to do
> {noformat}
> alter table over10k_orc_bucketed compact 'major' WITH OVERWRITE TBLPROPERTIES
> ("compactor.mapreduce.map.memory.mb"="500","compactor.hive.tez.container.size"="500");
> select runWorker() from mydual;
> show compactions;
> {noformat}
> but it always fails with
> {noformat}
> Invalid resource request, requested memory < 0, or requested memory > max
> configured, requestedMemory=1536, maxMemory=512
> {noformat}
> ToDo: see if need to fix host name masking in the output from "show
> compactions"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Status: Patch Available (was: In Progress)
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Attachment: HIVE-17433.09.patch
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch, HIVE-17433.09.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-17433:
Status: In Progress (was: Patch Available)
> Vectorization: Support Decimal64 in Hive Query Engine
> -
>
> Key: HIVE-17433
> URL: https://issues.apache.org/jira/browse/HIVE-17433
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-17433.03.patch, HIVE-17433.04.patch,
> HIVE-17433.05.patch, HIVE-17433.06.patch, HIVE-17433.07.patch,
> HIVE-17433.08.patch
>
>
> Provide partial support for Decimal64 within Hive. By partial I mean that
> our current decimal has a large surface area of features (rounding, multiply,
> divide, remainder, power, big precision, and many more) but only a small
> number has been identified as being performance hotspots.
> Those are small precision decimals with precision <= 18 that fit within a
> 64-bit long we are calling Decimal64 . Just as we optimize row-mode
> execution engine hotspots by selectively adding new vectorization code, we
> can treat the current decimal as the full featured one and add additional
> Decimal64 optimization where query benchmarks really show it help.
> This change creates a Decimal64ColumnVector.
> This change currently detects small decimal with Hive for Vectorized text
> input format and uses some new Decimal64 vectorized classes for comparison,
> addition, and later perhaps a few GroupBy aggregations like sum, avg, min,
> max.
> The patch also supports a new annotation that can mark a
> VectorizedInputFormat as supporting Decimal64 (it is called DECIMAL_64). So,
> in separate work those other formats such as ORC, PARQUET, etc can be done in
> later JIRAs so they participate in the Decimal64 performance optimization.
> The idea is when you annotate your input format with:
> @VectorizedInputFormatSupports(supports = {DECIMAL_64})
> the Vectorizer in Hive will plan usage of Decimal64ColumnVector instead of
> DecimalColumnVector. Upon an input format seeing Decimal64ColumnVector being
> used, the input format can fill that column vector with decimal64 longs
> instead of HiveDecimalWritable objects of DecimalColumnVector.
> There will be a Hive environment variable
> hive.vectorized.input.format.supports.enabled that has a string list of
> supported features. The default will start as "decimal_64". It can be
> turned off to allow for performance comparisons and testing.
> The query SELECT * FROM DECIMAL_6_1_txt where key - 100BD < 200BD ORDER BY
> key, value
> Will have a vectorized explain plan looking like:
> ...
> Filter Operator
> Filter Vectorization:
> className: VectorFilterOperator
> native: true
> predicateExpression:
> FilterDecimal64ColLessDecimal64Scalar(col 2, val 2000)(children:
> Decimal64ColSubtractDecimal64Scalar(col 0, val 1000,
> outputDecimal64AbsMax 999) -> 2:decimal(11,5)/DECIMAL_64) -> boolean
> predicate: ((key - 100) < 200) (type: boolean)
> ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17766) Support non-equi LEFT SEMI JOIN
[
https://issues.apache.org/jira/browse/HIVE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222538#comment-16222538
]
Ashutosh Chauhan commented on HIVE-17766:
-
Seems like some of tests need update. Also, for a new test can you have it run
in minillaplocalclidriver only (and not on clidriver).
Looks good otherwise. +1
> Support non-equi LEFT SEMI JOIN
> ---
>
> Key: HIVE-17766
> URL: https://issues.apache.org/jira/browse/HIVE-17766
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17766.01.patch, HIVE-17766.patch
>
>
> Currently we get an error like {noformat}Non equality condition not supported
> in Semi-Join{noformat}
> This is required to generate better plan for EXISTS/IN correlated subquery
> where such queries are transformed into LEFT SEMI JOIN.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-11812) datediff sometimes returns incorrect results when called with dates
[
https://issues.apache.org/jira/browse/HIVE-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222515#comment-16222515
]
Hive QA commented on HIVE-11812:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830654/HIVE-11812.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7505/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7505/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7505/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-10-27 15:03:40.083
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7505/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-10-27 15:03:40.085
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 88bd58e HIVE-17764 : alter view fails when
hive.metastore.disallow.incompatible.col.type.changes set to true (Janaki
Lahorani, reviewed by Andrew Sherman and Vihang Karajgaonkar) (addendum)
+ git clean -f -d
Removing standalone-metastore/src/gen/org/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 88bd58e HIVE-17764 : alter view fails when
hive.metastore.disallow.incompatible.col.type.changes set to true (Janaki
Lahorani, reviewed by Andrew Sherman and Vihang Karajgaonkar) (addendum)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-10-27 15:03:44.874
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateDiff.java: No
such file or directory
error:
a/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFDateDiff.java:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830654 - PreCommit-HIVE-Build
> datediff sometimes returns incorrect results when called with dates
> ---
>
> Key: HIVE-11812
> URL: https://issues.apache.org/jira/browse/HIVE-11812
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.0.0
>Reporter: Nicholas Brenwald
>Assignee: Chetna Chaudhari
>Priority: Minor
> Labels: timestamp
> Attachments: HIVE-11812.1.patch
>
>
> DATEDIFF returns an incorrect result when one of the arguments is a date
> type.
> The Hive Language Manual provides the following signature for datediff:
> {code}
> int datediff(string enddate, string startdate)
> {code}
> I think datediff should either throw an error (if date types are not
> supported), or return the correct result.
> To reproduce, create a table:
> {code}
> create table t (c1 string, c2 date);
> {code}
> Assuming you have a table x containing some data, populate table t with 1 row:
> {code}
> insert into t select '2015-09-15', '2015-09-15' from x limit 1;
> {code}
> Then run the following 12 test queries:
> {code}
> select datediff(c1, '2015-09-14') from t;
> select datediff(c1, '2015-09-15') from t;
> select datediff(c1, '2015-09-16') from t;
> select datediff('2015-09-14', c1) from t;
> select datediff('2015-09-15', c1) from t;
> select datediff('2015-09-16', c1) from t;
> select datediff(c2, '2015-09-14') from t;
> select datediff(c2, '2015-09-15') from t;
> select datediff(c2, '2015-09-16') from t;
> select
[jira] [Commented] (HIVE-17888) Display the reason for query cancellation
[
https://issues.apache.org/jira/browse/HIVE-17888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222513#comment-16222513
]
Hive QA commented on HIVE-17888:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894237/HIVE-17888.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=229)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7504/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7504/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7504/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894237 - PreCommit-HIVE-Build
> Display the reason for query cancellation
> -
>
> Key: HIVE-17888
> URL: https://issues.apache.org/jira/browse/HIVE-17888
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17888.1.patch
>
>
> For user convenience and easy debugging, if a trigger kills a query return
> the reason for the killing the query. Currently the query kill will only
> display the following which is not very useful
> {code}
> Error: Query was cancelled (state=01000,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222406#comment-16222406
]
Hive QA commented on HIVE-17433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894263/HIVE-17433.08.patch
{color:green}SUCCESS:{color} +1 due to 46 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 11327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_6]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid_fast]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_no_row_serde]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_parquet_projection]
(batchId=42)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin1]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin2]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin3]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_parquet_projection]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_shufflejoin]
(batchId=134)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=294)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorDateExpressions.testVectorUDFWeekOfYear
(batchId=276)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorBin
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorHex
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testRegex
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLike
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLikeMultiByte
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLikePatternType
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLikeRandomized
(batchId=277)
org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testAggregateOnUDF
(batchId=273)
org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testValidateNestedExpressions
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7503/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7503/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7503/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894263 - PreCommit-HIVE-Build
> Vectorization:
[jira] [Commented] (HIVE-17766) Support non-equi LEFT SEMI JOIN
[
https://issues.apache.org/jira/browse/HIVE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222310#comment-16222310
]
Hive QA commented on HIVE-17766:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894257/HIVE-17766.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 54 failed/errored test(s), 11328 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_partitioner]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[leftsemijoin]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reduce_deduplicate_extended2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subq_where_serialization]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists_having]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_in_having]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqual_corr_expr]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqualcolumnrefs]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[leftsemijoin]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_join_ptp]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_exists]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mapjoin_reduce]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[annotate_stats_join]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[leftsemijoin]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[semijoin]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_exists]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multiinsert]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=119)
[jira] [Commented] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1668#comment-1668 ] Xu Jin commented on HIVE-17630: --- [~dmtolpeko] Dear dmtolpeko, I have a question. Can we use HPL/SQL in production environment now? what is your advice? thanks. > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 2.2.0, 3.0.0 >Reporter: ZhangBing Lin >Assignee: Dmitry Tolpeko >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16748) Integreate YETUS to Pre-Commit
[
https://issues.apache.org/jira/browse/HIVE-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1605#comment-1605
]
Hive QA commented on HIVE-16748:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894196/HIVE-16748.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7501/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7501/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7501/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894196 - PreCommit-HIVE-Build
> Integreate YETUS to Pre-Commit
> --
>
> Key: HIVE-16748
> URL: https://issues.apache.org/jira/browse/HIVE-16748
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Adam Szita
> Attachments: HIVE-16748.0.patch, HIVE-16748.1.patch, dummytest.patch
>
>
> After HIVE-15051, we should automate the yetus run for the Pre-Commit tests,
> so the results are added in comments like
> https://issues.apache.org/jira/browse/YARN-6363?focusedCommentId=15937570=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937570
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17433) Vectorization: Support Decimal64 in Hive Query Engine
[
https://issues.apache.org/jira/browse/HIVE-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222137#comment-16222137
]
Hive QA commented on HIVE-17433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894263/HIVE-17433.08.patch
{color:green}SUCCESS:{color} +1 due to 46 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 37 failed/errored test(s), 11327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid_fast]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_no_row_serde]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_parquet_projection]
(batchId=42)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin1]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin2]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin3]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=100)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_parquet_projection]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_shufflejoin]
(batchId=134)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=205)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorDateExpressions.testVectorUDFWeekOfYear
(batchId=276)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorBin
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorHex
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testRegex
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLike
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLikeMultiByte
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLikePatternType
(batchId=277)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringLikeRandomized
(batchId=277)
org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testAggregateOnUDF
(batchId=273)
org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testValidateNestedExpressions
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=222)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7500/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7500/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7500/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 37 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12894263 - PreCommit-HIVE-Build
> Vectorization: Support Decimal64 in Hive Query Engine
>
