[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294714#comment-15294714 ] Lefty Leverenz commented on HIVE-13249: --- Doc note: Added TODOC1.3 and TODOC2.1 labels for the new configuration parameters *hive.max.open.txns* and *hive.count.open.txns.interval*. They will need to be documented in the wiki for 1.3.0 and 2.1.0 (but not 2.0.x). * [Hive Transactions -- New Configuration Parameters for Transactions | https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-NewConfigurationParametersforTransactions] * [Configuration Properties -- Transactions and Compactor | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-TransactionsandCompactor] > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC1.3, TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, > HIVE-13249.11.patch, HIVE-13249.12.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, > HIVE-13249.6.patch, HIVE-13249.7.patch, HIVE-13249.8.patch, > HIVE-13249.9.patch, HIVE-13249.branch-1.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13759) LlapTaskUmbilicalExternalClient should be closed by the record reader
[
https://issues.apache.org/jira/browse/HIVE-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294713#comment-15294713
]
Hive QA commented on HIVE-13759:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804985/HIVE-13759.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 9964 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-delete_where_non_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2.q-custom_input_output_format.q-join41.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby6_map.q-join13.q-join_reorder3.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-order.q-auto_join18_multi_distinct.q-union2.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.ql.TestTxnCommands.testSimpleAcidInsert
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/343/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/343/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-343/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12804985 - PreCommit-HIVE-MASTER-Build
> LlapTaskUmbilicalExternalClient should be closed by the record reader
> -
>
> Key: HIVE-13759
> URL: https://issues.apache.org/jira/browse/HIVE-13759
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-13759.1.patch
>
>
> The umbilical external client (and the server socket it creates) doesn't look
> like it's getting closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13249: -- Labels: TODOC1.3 TODOC2.1 (was: ) > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC1.3, TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, > HIVE-13249.11.patch, HIVE-13249.12.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, > HIVE-13249.6.patch, HIVE-13249.7.patch, HIVE-13249.8.patch, > HIVE-13249.9.patch, HIVE-13249.branch-1.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294702#comment-15294702
]
Lefty Leverenz commented on HIVE-13797:
---
This is a great idea!
Capitalization nits: "HiveServer2" in the text for #1, 2, 3, 4 and "Kerberos"
for #3.
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13797.01.patch
>
>
> It would save a bunch of googling if we could provide some examples of
> connection strings directly to beeline help message
> Eg:
> {code}
> ./bin/beeline --help
> Usage: java org.apache.hive.cli.beeline.BeeLine
>-uthe JDBC URL to connect to
>-r reconnect to last saved connect url (in
> conjunction with !save)
>-nthe username to connect as
>-pthe password to connect as
>-dthe driver class to use
>-i script file for initialization
>-e query that should be executed
>-f script file that should be executed
>-w (or) --password-file the password file to read
> password from
>--hiveconf property=value Use value for given property
>--hivevar name=valuehive variable name and value
>This is Hive specific settings in which
> variables
>can be set at session level and referenced
> in Hive
>commands or queries.
>--color=[true/false]control whether color is used for display
>--showHeader=[true/false] show column names in query results
>--headerInterval=ROWS; the interval between which heades are
> displayed
>--fastConnect=[true/false] skip building table/column list for
> tab-completion
>--autoCommit=[true/false] enable/disable automatic transaction commit
>--verbose=[true/false] show verbose error messages and debug info
>--showWarnings=[true/false] display connection warnings
>--showNestedErrs=[true/false] display nested errors
>--numberFormat=[pattern]format numbers using DecimalFormat pattern
>--force=[true/false]continue running script even after errors
>--maxWidth=MAXWIDTH the maximum width of the terminal
>--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
> columns
>--silent=[true/false] be more silent
>--autosave=[true/false] automatically save preferences
>--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
> result display
>Note that csv, and tsv are deprecated -
> use csv2, tsv2 instead
>--incremental=[true/false] Defaults to false. When set to false, the
> entire result set
>is fetched and buffered before being
> displayed, yielding optimal
>display column sizing. When set to true,
> result rows are displayed
>immediately as they are fetched, yielding
> lower latency and
>memory usage at the price of extra display
> column padding.
>Setting --incremental=true is recommended
> if you encounter an OutOfMemory
>on the client side (due to the fetched
> result set size being large).
>--truncateTable=[true/false]truncate table column when it exceeds
> length
>--delimiterForDSV=DELIMITER specify the delimiter for
> delimiter-separated values output format (default: |)
>--isolation=LEVEL set the transaction isolation level
>--nullemptystring=[true/false] set to true to get historic behavior of
> printing null as empty string
>--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline
> client side
>--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in
> the beeline client side
>--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
> beeline connected.
>Only works for HiveServer2 cluster mode.
>--help display this message
>
>Example:
> 1. beeline -u jdbc:hive2://localhost:1 username password
> 2. beeline -n username -p password -u jdbc:hive2://hs2.local:10012
> {code}
--
This message was
[jira] [Commented] (HIVE-13616) Investigate renaming a table without invalidating the column stats
[ https://issues.apache.org/jira/browse/HIVE-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294695#comment-15294695 ] Lefty Leverenz commented on HIVE-13616: --- Does this need to be documented in the wiki? > Investigate renaming a table without invalidating the column stats > -- > > Key: HIVE-13616 > URL: https://issues.apache.org/jira/browse/HIVE-13616 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu > Fix For: 2.1.0 > > Attachments: HIVE-13616.1.patch, HIVE-13616.2.patch > > > Right now when we rename a table, we clear the column stats rather than > updating it (HIVE-9720) since ObjectStore uses DN to talk to DB. Investigate > the possibility that if we can achieve updating the stats without rescanning > the whole table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294685#comment-15294685 ] Lefty Leverenz commented on HIVE-13782: --- [~jxiang], thanks for the explanation. +1 for your new description. > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[
https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294624#comment-15294624
]
Hive QA commented on HIVE-13409:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805277/HIVE-13409-java8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 150 failed/errored test(s), 9854 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-schema_evol_orc_acidvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-bucket_map_join_tez1.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_skewtable
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_json_serde1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_collect_set_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_sort_array
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_udf1
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_sample1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestMiniTezCl
[jira] [Updated] (HIVE-13699) Make JavaDataModel#get thread safe for parallel compilation
[ https://issues.apache.org/jira/browse/HIVE-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13699: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Pushed to master. > Make JavaDataModel#get thread safe for parallel compilation > --- > > Key: HIVE-13699 > URL: https://issues.apache.org/jira/browse/HIVE-13699 > Project: Hive > Issue Type: Bug > Components: HiveServer2, storage-api >Affects Versions: 2.0.0 >Reporter: Peter Slawski >Assignee: Peter Slawski >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-13699.1.patch, HIVE-13699.2.patch > > > The class JavaDataModel has a static method, #get, that is not thread safe. > This may be an issue when parallel query compilation is enabled because two > threads may attempt to call JavaDataModel#get at the same time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11956) SHOW LOCKS should indicate what acquired the lock
[ https://issues.apache.org/jira/browse/HIVE-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11956: -- Target Version/s: 1.3.0, 2.1.0 > SHOW LOCKS should indicate what acquired the lock > - > > Key: HIVE-11956 > URL: https://issues.apache.org/jira/browse/HIVE-11956 > Project: Hive > Issue Type: Improvement > Components: CLI, Transactions >Affects Versions: 0.14.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > This can be a queryId, Flume agent id, Storm bolt id, etc. This would > dramatically help diagnosing issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11956) SHOW LOCKS should indicate what acquired the lock
[ https://issues.apache.org/jira/browse/HIVE-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-11956: - Assignee: Eugene Koifman (was: Wei Zheng) > SHOW LOCKS should indicate what acquired the lock > - > > Key: HIVE-11956 > URL: https://issues.apache.org/jira/browse/HIVE-11956 > Project: Hive > Issue Type: Improvement > Components: CLI, Transactions >Affects Versions: 0.14.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > This can be a queryId, Flume agent id, Storm bolt id, etc. This would > dramatically help diagnosing issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294609#comment-15294609 ] Sergey Shelukhin edited comment on HIVE-13771 at 5/21/16 1:08 AM: -- Tiny patch on top of other patches. The patch that introduces coordinator is the real blocker but I don't remember which one that is. [~jdere] [~sseth] does this method make sense to you? Is it ok to generate bogus cluster timestamp, or does it have to be a real timestamp? was (Author: sershe): Tiny patch on top of other patches. The patch that introduces coordinator is the real blocker but I don't remember which one it is. [~jdere] [~sseth] does this method make sense to you? Is it ok to generate bogus cluster timestamp, or does it have to be a real timestamp? > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Status: Patch Available (was: Open) > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Attachment: HIVE-13771.patch Tiny patch on top of other patches. The patch that introduces coordinator is the real blocker but I don't remember which one it is. [~jdere] [~sseth] does this method make sense to you? Is it ok to generate bogus cluster timestamp, or does it have to be a real timestamp? > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13702) TestHiveSessionImpl fails on master
[ https://issues.apache.org/jira/browse/HIVE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair resolved HIVE-13702. -- Resolution: Duplicate > TestHiveSessionImpl fails on master > --- > > Key: HIVE-13702 > URL: https://issues.apache.org/jira/browse/HIVE-13702 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Vaibhav Gumashta > > Presumably broken by HIVE-4924 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13223) HoS may hang for queries that run on 0 splits
[ https://issues.apache.org/jira/browse/HIVE-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13223: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Fixed via HIVE-13525 > HoS may hang for queries that run on 0 splits > --- > > Key: HIVE-13223 > URL: https://issues.apache.org/jira/browse/HIVE-13223 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13223.1.patch, HIVE-13223.2.patch, HIVE-13223.patch > > > Can be seen on all timed out tests after HIVE-13040 went in -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13197) Add adapted constprog2.q and constprog_partitioner.q tests back
[ https://issues.apache.org/jira/browse/HIVE-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13197: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Pushed to master. > Add adapted constprog2.q and constprog_partitioner.q tests back > --- > > Key: HIVE-13197 > URL: https://issues.apache.org/jira/browse/HIVE-13197 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Jesus Camacho Rodriguez >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13197.patch > > > HIVE-12749 removes constprog2.q and constprog_partitioner.q tests, as they > did not test constant propagation anymore. Ideally, we should create them > again with compatible types to test constant propagation and constant > propagation in the presence of partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13782) Compile async query asynchronously
[ https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294581#comment-15294581 ] Jimmy Xiang commented on HIVE-13782: [~leftylev], any comments on the new parameter description above? > Compile async query asynchronously > -- > > Key: HIVE-13782 > URL: https://issues.apache.org/jira/browse/HIVE-13782 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13782.1.patch > > > Currently, when an async query is submitted to HS2, HS2 does the preparation > synchronously. One of the preparation step is to compile the query, which may > take some time. It will be helpful to provide an option to do the compilation > asynchronously. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[
https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294579#comment-15294579
]
Jimmy Xiang commented on HIVE-13813:
Looks good to me too. +1
> Add Metrics for the number of Hive operations waiting for compile
> -
>
> Key: HIVE-13813
> URL: https://issues.apache.org/jira/browse/HIVE-13813
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-13183.patch
>
>
> Currently, without {{hive.driver.parallel.compilation}} introduced in
> HIVE-4239, only one SQL operation can enter the compilation block per HS2
> instance, and all the rest will be blocked. We should add metrics info for
> the number of operations that are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13778) DROP TABLE PURGE on S3A table with too many files does not delete the files
[ https://issues.apache.org/jira/browse/HIVE-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294575#comment-15294575 ] Aaron Fabbri commented on HIVE-13778: - Thanks for the details [~sailesh]. Namenode should not be involved with s3a paths. Can you re-run with some s3a logging on? i.e. org.apache.hadoop.fs.s3a=DEBUG > DROP TABLE PURGE on S3A table with too many files does not delete the files > --- > > Key: HIVE-13778 > URL: https://issues.apache.org/jira/browse/HIVE-13778 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sailesh Mukil >Priority: Critical > Labels: metastore, s3 > > I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A > that has many files, the files never get deleted. However, the Hive metastore > logs do say that the path was deleted: > "Not moving [path] to trash" > "Deleted the diretory [path]" > I initially thought that this was due to the eventually consistent nature of > S3 for deletes, however, a week later, the files still exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294574#comment-15294574 ] Sergey Shelukhin commented on HIVE-11527: - Looks good to me. [~vgumashta] [~gopalv] any comments? > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13778) DROP TABLE PURGE on S3A table with too many files does not delete the files
[ https://issues.apache.org/jira/browse/HIVE-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294572#comment-15294572 ] Sailesh Mukil commented on HIVE-13778: -- [~rajesh.balamohan] I discovered a few more details as I was trying to reproduce this issue. I didn't realize that this is how it happened before. To reproduce, do the following: - In Hive, "create table purge_test_s3 (x int) location 's3a://[bucket]/purge_test_s3';" - Use the AWS CLI or the AWS Web interface to copy files to the above mentioned location. - In Hive, "drop table purge_test_s3 purge;" The Metastore logs say: 2016-05-20 17:01:41,259 INFO hive.metastore.hivemetastoressimpl: [pool-4-thread-103]: Not moving s3a://[bucket]/purge_test_s3 to trash 2016-05-20 17:01:41,364 INFO hive.metastore.hivemetastoressimpl: [pool-4-thread-103]: Deleted the diretory s3a://[bucket]/purge_test_s3 However, the files are still there. The weird part is that the Hadoop S3A connector reads the files correctly but is not able to delete them. If instead of the AWS CLI or the AWS Web interface, we use the hadoop CLI to copy the files, "drop table ... purge" works just fine. If we insert the files using Hive, it works fine as well. This might be an issue of the HDFS Namenode not getting updated and might be more a problem for the HDFS folks. > DROP TABLE PURGE on S3A table with too many files does not delete the files > --- > > Key: HIVE-13778 > URL: https://issues.apache.org/jira/browse/HIVE-13778 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Sailesh Mukil >Priority: Critical > Labels: metastore, s3 > > I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A > that has many files, the files never get deleted. However, the Hive metastore > logs do say that the path was deleted: > "Not moving [path] to trash" > "Deleted the diretory [path]" > I initially thought that this was due to the eventually consistent nature of > S3 for deletes, however, a week later, the files still exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294568#comment-15294568 ] Gunther Hagleitner commented on HIVE-13628: --- +1 > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch, HIVE-13628.4.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13444) LLAP: add HMAC signatures to LLAP; verify them on LLAP side
[ https://issues.apache.org/jira/browse/HIVE-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13444: Attachment: HIVE-13444.02.patch > LLAP: add HMAC signatures to LLAP; verify them on LLAP side > --- > > Key: HIVE-13444 > URL: https://issues.apache.org/jira/browse/HIVE-13444 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13444.01.patch, HIVE-13444.02.patch, > HIVE-13444.WIP.patch, HIVE-13444.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.4.patch [~hagleitn] review please. > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch, HIVE-13628.4.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-13771: --- Assignee: Sergey Shelukhin > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.3.patch > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294507#comment-15294507
]
Vihang Karajgaonkar commented on HIVE-13797:
Here is sample output with the latest patch.
{noformat}
$ ./beeline --help
Usage: java org.apache.hive.cli.beeline.BeeLine
-uthe JDBC URL to connect to
-nthe username to connect as
-pthe password to connect as
-dthe driver class to use
-i script file for initialization
-e query that should be executed
-f script file that should be executed
-w (or) --password-file the password file to read password
from
--hiveconf property=value Use value for given property
--hivevar name=valuehive variable name and value
This is Hive specific settings in which
variables
can be set at session level and referenced
in Hive
commands or queries.
--color=[true/false]control whether color is used for display
--showHeader=[true/false] show column names in query results
--headerInterval=ROWS; the interval between which heades are
displayed
--fastConnect=[true/false] skip building table/column list for
tab-completion
--autoCommit=[true/false] enable/disable automatic transaction commit
--verbose=[true/false] show verbose error messages and debug info
--showWarnings=[true/false] display connection warnings
--showNestedErrs=[true/false] display nested errors
--numberFormat=[pattern]format numbers using DecimalFormat pattern
--force=[true/false]continue running script even after errors
--maxWidth=MAXWIDTH the maximum width of the terminal
--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
columns
--silent=[true/false] be more silent
--autosave=[true/false] automatically save preferences
--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
result display
Note that csv, and tsv are deprecated - use
csv2, tsv2 instead
--incremental=[true/false] Defaults to false. When set to false, the
entire result set
is fetched and buffered before being
displayed, yielding optimal
display column sizing. When set to true,
result rows are displayed
immediately as they are fetched, yielding
lower latency and
memory usage at the price of extra display
column padding.
Setting --incremental=true is recommended if
you encounter an OutOfMemory
on the client side (due to the fetched
result set size being large).
--truncateTable=[true/false]truncate table column when it exceeds length
--delimiterForDSV=DELIMITER specify the delimiter for
delimiter-separated values output format (default: |)
--isolation=LEVEL set the transaction isolation level
--nullemptystring=[true/false] set to true to get historic behavior of
printing null as empty string
--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline client
side
--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in the
beeline client side
--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
beeline connected.
Only works for HiveServer2 cluster mode.
--help display this message
Example:
1. Connect using simple authentication to Hiveserver2 on localhost:1
$ beeline -u jdbc:hive2://localhost:1 username password
2. Connect using simple authentication to Hiveserver2 on hs.local:1
using -n for username and -p for password
$ beeline -n username -p password -u jdbc:hive2://hs2.local:10012
3. Connect using kerberos authentication with hive/localh...@mydomain.com
as Hiveserver2 principal
$ beeline -u
"jdbc:hive2://hs2.local:10013/default;principal=hive/localh...@mydomain.com
4. Connect using SSL connection to hiveserver2 on localhost at 1
$ beeline
jdbc:hive2://localhost:1/default;ssl=true;sslTrustStore=/usr/local/truststore;trustStorePassword=mytruststorepassword
5. Connect using LDAP authentication
$ beeline -u jdbc:hive2://hs2.local:10013/default
{noformat}
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project:
[jira] [Updated] (HIVE-13441) LLAP: signed fragments
[ https://issues.apache.org/jira/browse/HIVE-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13441: Target Version/s: 2.1.0 > LLAP: signed fragments > -- > > Key: HIVE-13441 > URL: https://issues.apache.org/jira/browse/HIVE-13441 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: llap > > Allows external clients to get securely signed splits from HS2, and submit > them to LLAP without running as a privileged user; LLAP will verify the > splits before running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Target Version/s: 2.1.0 > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-13797:
---
Attachment: HIVE-13797.01.patch
Added a new patch after suggestions from Sergio
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13797.01.patch
>
>
> It would save a bunch of googling if we could provide some examples of
> connection strings directly to beeline help message
> Eg:
> {code}
> ./bin/beeline --help
> Usage: java org.apache.hive.cli.beeline.BeeLine
>-uthe JDBC URL to connect to
>-r reconnect to last saved connect url (in
> conjunction with !save)
>-nthe username to connect as
>-pthe password to connect as
>-dthe driver class to use
>-i script file for initialization
>-e query that should be executed
>-f script file that should be executed
>-w (or) --password-file the password file to read
> password from
>--hiveconf property=value Use value for given property
>--hivevar name=valuehive variable name and value
>This is Hive specific settings in which
> variables
>can be set at session level and referenced
> in Hive
>commands or queries.
>--color=[true/false]control whether color is used for display
>--showHeader=[true/false] show column names in query results
>--headerInterval=ROWS; the interval between which heades are
> displayed
>--fastConnect=[true/false] skip building table/column list for
> tab-completion
>--autoCommit=[true/false] enable/disable automatic transaction commit
>--verbose=[true/false] show verbose error messages and debug info
>--showWarnings=[true/false] display connection warnings
>--showNestedErrs=[true/false] display nested errors
>--numberFormat=[pattern]format numbers using DecimalFormat pattern
>--force=[true/false]continue running script even after errors
>--maxWidth=MAXWIDTH the maximum width of the terminal
>--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
> columns
>--silent=[true/false] be more silent
>--autosave=[true/false] automatically save preferences
>--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
> result display
>Note that csv, and tsv are deprecated -
> use csv2, tsv2 instead
>--incremental=[true/false] Defaults to false. When set to false, the
> entire result set
>is fetched and buffered before being
> displayed, yielding optimal
>display column sizing. When set to true,
> result rows are displayed
>immediately as they are fetched, yielding
> lower latency and
>memory usage at the price of extra display
> column padding.
>Setting --incremental=true is recommended
> if you encounter an OutOfMemory
>on the client side (due to the fetched
> result set size being large).
>--truncateTable=[true/false]truncate table column when it exceeds
> length
>--delimiterForDSV=DELIMITER specify the delimiter for
> delimiter-separated values output format (default: |)
>--isolation=LEVEL set the transaction isolation level
>--nullemptystring=[true/false] set to true to get historic behavior of
> printing null as empty string
>--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline
> client side
>--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in
> the beeline client side
>--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
> beeline connected.
>Only works for HiveServer2 cluster mode.
>--help display this message
>
>Example:
> 1. beeline -u jdbc:hive2://localhost:1 username password
> 2. beeline -n username -p password -u jdbc:hive2://hs2.local:10012
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[
https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294491#comment-15294491
]
Xuefu Zhang commented on HIVE-13813:
Patch looks good to me, but it's good if [~jxiang] can also take a look.
> Add Metrics for the number of Hive operations waiting for compile
> -
>
> Key: HIVE-13813
> URL: https://issues.apache.org/jira/browse/HIVE-13813
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-13183.patch
>
>
> Currently, without {{hive.driver.parallel.compilation}} introduced in
> HIVE-4239, only one SQL operation can enter the compilation block per HS2
> instance, and all the rest will be blocked. We should add metrics info for
> the number of operations that are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-13797:
---
Attachment: (was: HIVE-13797.01.patch)
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> It would save a bunch of googling if we could provide some examples of
> connection strings directly to beeline help message
> Eg:
> {code}
> ./bin/beeline --help
> Usage: java org.apache.hive.cli.beeline.BeeLine
>-uthe JDBC URL to connect to
>-r reconnect to last saved connect url (in
> conjunction with !save)
>-nthe username to connect as
>-pthe password to connect as
>-dthe driver class to use
>-i script file for initialization
>-e query that should be executed
>-f script file that should be executed
>-w (or) --password-file the password file to read
> password from
>--hiveconf property=value Use value for given property
>--hivevar name=valuehive variable name and value
>This is Hive specific settings in which
> variables
>can be set at session level and referenced
> in Hive
>commands or queries.
>--color=[true/false]control whether color is used for display
>--showHeader=[true/false] show column names in query results
>--headerInterval=ROWS; the interval between which heades are
> displayed
>--fastConnect=[true/false] skip building table/column list for
> tab-completion
>--autoCommit=[true/false] enable/disable automatic transaction commit
>--verbose=[true/false] show verbose error messages and debug info
>--showWarnings=[true/false] display connection warnings
>--showNestedErrs=[true/false] display nested errors
>--numberFormat=[pattern]format numbers using DecimalFormat pattern
>--force=[true/false]continue running script even after errors
>--maxWidth=MAXWIDTH the maximum width of the terminal
>--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
> columns
>--silent=[true/false] be more silent
>--autosave=[true/false] automatically save preferences
>--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
> result display
>Note that csv, and tsv are deprecated -
> use csv2, tsv2 instead
>--incremental=[true/false] Defaults to false. When set to false, the
> entire result set
>is fetched and buffered before being
> displayed, yielding optimal
>display column sizing. When set to true,
> result rows are displayed
>immediately as they are fetched, yielding
> lower latency and
>memory usage at the price of extra display
> column padding.
>Setting --incremental=true is recommended
> if you encounter an OutOfMemory
>on the client side (due to the fetched
> result set size being large).
>--truncateTable=[true/false]truncate table column when it exceeds
> length
>--delimiterForDSV=DELIMITER specify the delimiter for
> delimiter-separated values output format (default: |)
>--isolation=LEVEL set the transaction isolation level
>--nullemptystring=[true/false] set to true to get historic behavior of
> printing null as empty string
>--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline
> client side
>--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in
> the beeline client side
>--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
> beeline connected.
>Only works for HiveServer2 cluster mode.
>--help display this message
>
>Example:
> 1. beeline -u jdbc:hive2://localhost:1 username password
> 2. beeline -n username -p password -u jdbc:hive2://hs2.local:10012
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[
https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294467#comment-15294467
]
Chao Sun commented on HIVE-13813:
-
cc [~jxiang] and [~xuefuz].
> Add Metrics for the number of Hive operations waiting for compile
> -
>
> Key: HIVE-13813
> URL: https://issues.apache.org/jira/browse/HIVE-13813
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-13183.patch
>
>
> Currently, without {{hive.driver.parallel.compilation}} introduced in
> HIVE-4239, only one SQL operation can enter the compilation block per HS2
> instance, and all the rest will be blocked. We should add metrics info for
> the number of operations that are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[
https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-13813:
Status: Patch Available (was: Open)
> Add Metrics for the number of Hive operations waiting for compile
> -
>
> Key: HIVE-13813
> URL: https://issues.apache.org/jira/browse/HIVE-13813
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-13183.patch
>
>
> Currently, without {{hive.driver.parallel.compilation}} introduced in
> HIVE-4239, only one SQL operation can enter the compilation block per HS2
> instance, and all the rest will be blocked. We should add metrics info for
> the number of operations that are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13813) Add Metrics for the number of Hive operations waiting for compile
[
https://issues.apache.org/jira/browse/HIVE-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-13813:
Attachment: HIVE-13183.patch
> Add Metrics for the number of Hive operations waiting for compile
> -
>
> Key: HIVE-13813
> URL: https://issues.apache.org/jira/browse/HIVE-13813
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-13183.patch
>
>
> Currently, without {{hive.driver.parallel.compilation}} introduced in
> HIVE-4239, only one SQL operation can enter the compilation block per HS2
> instance, and all the rest will be blocked. We should add metrics info for
> the number of operations that are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13787) LLAP: bug in recent security patches (wrong argument order; using full user name in id)
[ https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294442#comment-15294442 ] Sergey Shelukhin commented on HIVE-13787: - Not as far as I recall. Owner has the short name... iirc realUser si the full name. I'll double check > LLAP: bug in recent security patches (wrong argument order; using full user > name in id) > --- > > Key: HIVE-13787 > URL: https://issues.apache.org/jira/browse/HIVE-13787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13787.01.patch, HIVE-13787.02.patch, > HIVE-13787.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: (was: HIVE-13628.3.patch) > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294342#comment-15294342
]
Ashutosh Chauhan commented on HIVE-13810:
-
+1
> insert overwrite select from some table fails throwing
> org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13810
> URL: https://issues.apache.org/jira/browse/HIVE-13810
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13810.1.patch, HIVE-13810.2.patch
>
>
> When running using beeline (as a non hdfs user)
> {noformat} CREATE EXTERNAL TABLE Batters_txt(
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
> )
> location '/user/tableau/Batters';
> drop table if exists Batters;
> CREATE TABLE Batters (
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
>)
> STORED AS orc tblproperties ("orc.compress"="SNAPPY");
> insert overwrite table Batters select * from Batters_txt;
> {noformat}
> runs into the following error:
> {code}
> 2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
>
[jira] [Updated] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13810:
-
Attachment: (was: HIVE-13810.2.patch)
> insert overwrite select from some table fails throwing
> org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13810
> URL: https://issues.apache.org/jira/browse/HIVE-13810
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13810.1.patch, HIVE-13810.2.patch
>
>
> When running using beeline (as a non hdfs user)
> {noformat} CREATE EXTERNAL TABLE Batters_txt(
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
> )
> location '/user/tableau/Batters';
> drop table if exists Batters;
> CREATE TABLE Batters (
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
>)
> STORED AS orc tblproperties ("orc.compress"="SNAPPY");
> insert overwrite table Batters select * from Batters_txt;
> {noformat}
> runs into the following error:
> {code}
> 2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.security.AccessControlException: User does not b
[jira] [Updated] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13810:
-
Attachment: HIVE-13810.2.patch
> insert overwrite select from some table fails throwing
> org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13810
> URL: https://issues.apache.org/jira/browse/HIVE-13810
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13810.1.patch, HIVE-13810.2.patch
>
>
> When running using beeline (as a non hdfs user)
> {noformat} CREATE EXTERNAL TABLE Batters_txt(
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
> )
> location '/user/tableau/Batters';
> drop table if exists Batters;
> CREATE TABLE Batters (
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
>)
> STORED AS orc tblproperties ("orc.compress"="SNAPPY");
> insert overwrite table Batters select * from Batters_txt;
> {noformat}
> runs into the following error:
> {code}
> 2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.security.AccessControlException: User does not belong to
>
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: HIVE-13068.06.patch > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.06.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Status: Open (was: Patch Available) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Status: Patch Available (was: In Progress) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13068 started by Jesus Camacho Rodriguez. -- > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13810:
-
Attachment: HIVE-13810.2.patch
[~ashutoshc] Makes sense. Realized that I should be rethrowing error soon after
I uploaded patch#1. Please see the change in patch#2. Tested with patch#2 as
well.
> insert overwrite select from some table fails throwing
> org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13810
> URL: https://issues.apache.org/jira/browse/HIVE-13810
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13810.1.patch, HIVE-13810.2.patch
>
>
> When running using beeline (as a non hdfs user)
> {noformat} CREATE EXTERNAL TABLE Batters_txt(
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
> )
> location '/user/tableau/Batters';
> drop table if exists Batters;
> CREATE TABLE Batters (
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
>)
> STORED AS orc tblproperties ("orc.compress"="SNAPPY");
> insert overwrite table Batters select * from Batters_txt;
> {noformat}
> runs into the following error:
> {code}
> 2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Threa
[jira] [Commented] (HIVE-13787) LLAP: bug in recent security patches (wrong argument order; using full user name in id)
[
https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294306#comment-15294306
]
Siddharth Seth commented on HIVE-13787:
---
In SecretManager
{code}
user = ugi.getUserName();
if (ugi.getRealUser() != null) {
realUser = new Text(ugi.getRealUser().getUserName());
}
{code}
Does this need to use getShortUserName as well ?
> LLAP: bug in recent security patches (wrong argument order; using full user
> name in id)
> ---
>
> Key: HIVE-13787
> URL: https://issues.apache.org/jira/browse/HIVE-13787
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13787.01.patch, HIVE-13787.02.patch,
> HIVE-13787.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294298#comment-15294298 ] Jesus Camacho Rodriguez commented on HIVE-13068: I was finishing regenerating the q files :) I will upload the new patch to RB and here in a few minutes. > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13809) Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size
[
https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294295#comment-15294295
]
Wei Zheng commented on HIVE-13809:
--
OK, agree. Btw,
{code}
Estimated small table size: 16
{code}
This is data size (in bytes), not number of keys. Estimated number of keys is
26672 (still a big number).
> Hybrid Grace Hash Join memory usage estimation didn't take into account the
> bloom filter size
> -
>
> Key: HIVE-13809
> URL: https://issues.apache.org/jira/browse/HIVE-13809
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Memory estimation is important during hash table loading, because we need to
> make the decision of whether to load the next hash partition in memory or
> spill it. If the assumption is there's enough memory but it turns out not the
> case, we will run into OOM problem.
> Currently hybrid grace hash join memory usage estimation didn't take into
> account the bloom filter size. In large test cases (TB scale) the bloom
> filter grows as big as hundreds of MB, big enough to cause estimation error.
> The solution is to count in the bloom filter size into memory estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294290#comment-15294290 ] Ashutosh Chauhan commented on HIVE-13068: - Can you also upload patch here for QA run. > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294273#comment-15294273 ] Ashutosh Chauhan commented on HIVE-13566: - +1 pending tests. > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch, HIVE-13566.02.patch > > > This jira adds code and tests for auto-gather column stats. Golden file > update will be done in phase 2 - HIVE-11160 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13809) Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size
[
https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294271#comment-15294271
]
Gopal V commented on HIVE-13809:
{{1,600,000,000}} doesn't look like a good scenario to fix around for a
map-join.
Back of the envelope, 1.6 billion keys in a 2Gb hashtable comes down to <2
bytes per key, which is obvious going to OOM there, even divided by 16 (i.e 32
bytes per key+value).
I suspect the root issue has more to do with the statistics involved here,
which might be completely bogus.
Ideally, we should be capping the bloom filter estimates at 4 million keys,
which is nearly the probe limit of useful hashtables - any real data-set bigger
than 4M, the total number of rehashes at 2Gb will also OOM the hashtables.
So in this scenario, I'm nearly convinced that the 1.6 billion number is bogus
or bad planning.
> Hybrid Grace Hash Join memory usage estimation didn't take into account the
> bloom filter size
> -
>
> Key: HIVE-13809
> URL: https://issues.apache.org/jira/browse/HIVE-13809
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Memory estimation is important during hash table loading, because we need to
> make the decision of whether to load the next hash partition in memory or
> spill it. If the assumption is there's enough memory but it turns out not the
> case, we will run into OOM problem.
> Currently hybrid grace hash join memory usage estimation didn't take into
> account the bloom filter size. In large test cases (TB scale) the bloom
> filter grows as big as hundreds of MB, big enough to cause estimation error.
> The solution is to count in the bloom filter size into memory estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294265#comment-15294265
]
Ashutosh Chauhan commented on HIVE-13810:
-
* If you are setting permission just on dir, then you don't need to do it
recursively. You may pass false flag in there.
* Don't swallow exception. Rethrow after logging.
> insert overwrite select from some table fails throwing
> org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13810
> URL: https://issues.apache.org/jira/browse/HIVE-13810
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13810.1.patch
>
>
> When running using beeline (as a non hdfs user)
> {noformat} CREATE EXTERNAL TABLE Batters_txt(
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
> )
> location '/user/tableau/Batters';
> drop table if exists Batters;
> CREATE TABLE Batters (
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
>)
> STORED AS orc tblproperties ("orc.compress"="SNAPPY");
> insert overwrite table Batters select * from Batters_txt;
> {noformat}
> runs into the following error:
> {code}
> 2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
[jira] [Resolved] (HIVE-10171) Create a storage-api module
[ https://issues.apache.org/jira/browse/HIVE-10171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HIVE-10171. -- Resolution: Fixed Fix Version/s: 2.1.0 The storage-api and orc modules are now standalone. > Create a storage-api module > --- > > Key: HIVE-10171 > URL: https://issues.apache.org/jira/browse/HIVE-10171 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > > To support high performance file formats, I'd like to propose that we move > the minimal set of classes that are required to integrate with Hive into a > new module named "storage-api". This module will include VectorizedRowBatch, > the various ColumnVector classes, and the SARG classes. It will form the > start of an API that high performance storage formats can use to integrate > with Hive. Both ORC and Parquet can use the new API to support vectorization > and SARGs without performance destroying shims. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294260#comment-15294260
]
Vihang Karajgaonkar commented on HIVE-13797:
Sample output after the patch.
{noformat}
$ ./beeline --help
Usage: java org.apache.hive.cli.beeline.BeeLine
-uthe JDBC URL to connect to
-r reconnect to last saved connect url (in
conjunction with !save)
-nthe username to connect as
-pthe password to connect as
-dthe driver class to use
-i script file for initialization
-e query that should be executed
-f script file that should be executed
-w (or) --password-file the password file to read password
from
--hiveconf property=value Use value for given property
--hivevar name=valuehive variable name and value
This is Hive specific settings in which
variables
can be set at session level and referenced
in Hive
commands or queries.
--color=[true/false]control whether color is used for display
--showHeader=[true/false] show column names in query results
--headerInterval=ROWS; the interval between which heades are
displayed
--fastConnect=[true/false] skip building table/column list for
tab-completion
--autoCommit=[true/false] enable/disable automatic transaction commit
--verbose=[true/false] show verbose error messages and debug info
--showWarnings=[true/false] display connection warnings
--showNestedErrs=[true/false] display nested errors
--numberFormat=[pattern]format numbers using DecimalFormat pattern
--force=[true/false]continue running script even after errors
--maxWidth=MAXWIDTH the maximum width of the terminal
--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
columns
--silent=[true/false] be more silent
--autosave=[true/false] automatically save preferences
--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
result display
Note that csv, and tsv are deprecated - use
csv2, tsv2 instead
--incremental=[true/false] Defaults to false. When set to false, the
entire result set
is fetched and buffered before being
displayed, yielding optimal
display column sizing. When set to true,
result rows are displayed
immediately as they are fetched, yielding
lower latency and
memory usage at the price of extra display
column padding.
Setting --incremental=true is recommended if
you encounter an OutOfMemory
on the client side (due to the fetched
result set size being large).
--truncateTable=[true/false]truncate table column when it exceeds length
--delimiterForDSV=DELIMITER specify the delimiter for
delimiter-separated values output format (default: |)
--isolation=LEVEL set the transaction isolation level
--nullemptystring=[true/false] set to true to get historic behavior of
printing null as empty string
--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline client
side
--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in the
beeline client side
--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
beeline connected.
Only works for HiveServer2 cluster mode.
--help display this message
Example:
1. beeline -u jdbc:hive2://localhost:1 username password
2. beeline -n username -p password -u jdbc:hive2://hs2.local:10012
3. beeline -u
"jdbc:hive2://hs2.local:10013/default;principal=hive/hiveserver2h...@your-domain.com"
{noformat}
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13797.01.patch
>
>
> It would save a bunch of googling if we could provide some examples of
> connection strings directly to beeline help message
> Eg:
> {code}
> ./bin/beeline --help
> Usage: java org.apache.hive.cli.beeline.BeeLine
>-uthe JDBC URL to connect to
[jira] [Updated] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11417: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks for the review, Prasanth. > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: > 0001-HIVE-11417.-Move-the-ReaderImpl-and-RowReaderImpl-to.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch.git > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13809) Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size
[
https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294246#comment-15294246
]
Wei Zheng commented on HIVE-13809:
--
[~gopalv] Sure, thanks for your input. Here's the snippet from application log.
It can be seen in this case we got 266 million keys, thus 207 MB for bloom
filter.
{code}
2016-05-20 11:29:56,600 [INFO] [pool-17-thread-2]
|persistence.HybridHashTableContainer|: Total available memory: 2115483632
2016-05-20 11:29:56,601 [INFO] [pool-17-thread-2]
|persistence.HybridHashTableContainer|: Estimated small table size: 16
2016-05-20 11:29:56,601 [INFO] [pool-17-thread-2]
|persistence.HybridHashTableContainer|: Number of hash partitions to be
created: 16
2016-05-20 11:29:56,614 [INFO] [TezChild] |vector.VectorGroupByOperator|:
VectorGroupByOperator is vector output false
2016-05-20 11:29:56,617 [INFO] [TezChild] |exec.ReduceSinkOperator|:
Initializing operator RS[44]
2016-05-20 11:29:56,620 [INFO] [TezChild] |exec.ReduceSinkOperator|: Using tag
= -1
2016-05-20 11:29:56,780 [INFO] [pool-17-thread-2]
|persistence.HybridHashTableContainer|: Using a bloom-1 filter 26672 keys
of size 207840816 bytes
{code}
> Hybrid Grace Hash Join memory usage estimation didn't take into account the
> bloom filter size
> -
>
> Key: HIVE-13809
> URL: https://issues.apache.org/jira/browse/HIVE-13809
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Memory estimation is important during hash table loading, because we need to
> make the decision of whether to load the next hash partition in memory or
> spill it. If the assumption is there's enough memory but it turns out not the
> case, we will run into OOM problem.
> Currently hybrid grace hash join memory usage estimation didn't take into
> account the bloom filter size. In large test cases (TB scale) the bloom
> filter grows as big as hundreds of MB, big enough to cause estimation error.
> The solution is to count in the bloom filter size into memory estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294248#comment-15294248 ] ASF GitHub Bot commented on HIVE-11417: --- Github user asfgit closed the pull request at: https://github.com/apache/hive/pull/72 > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: > 0001-HIVE-11417.-Move-the-ReaderImpl-and-RowReaderImpl-to.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch.git > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13812) Changing column from float to string group type will drop some factional digits
[
https://issues.apache.org/jira/browse/HIVE-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13812:
--
Summary: Changing column from float to string group type will drop some
factional digits (was: Change column from float to string group type will drop
some factional digits)
> Changing column from float to string group type will drop some factional
> digits
> ---
>
> Key: HIVE-13812
> URL: https://issues.apache.org/jira/browse/HIVE-13812
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> Create a textfile table with a float column and insert some values:
> {noformat}
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> create table test(f float);
> No rows affected (0.237 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> insert into table test
> values(-35664.76171875),(29497.349609375);
> INFO : Session is already open
> INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1463771904371_0006)
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://os-r6-pxwhrs-hiveserver2-3re-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-20_21-06-29_377_6487823927119226603-10/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
> rawDataSize=17]
> No rows affected (11.069 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.137 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | f | float | |
> +---++--+--+
> 1 row selected (0.173 seconds)
> {noformat}
> Then change float type to string successfully, but when you select table,
> some fractional digits are lost:
> {noformat}
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> alter table test change column
> f f string;
> No rows affected (0.214 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | f | string | |
> +---++--+--+
> 1 row selected (0.151 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.35 |
> ++--+
> 2 rows selected (0.141 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13812) Change column from float to string group type will drop some factional digits
[
https://issues.apache.org/jira/browse/HIVE-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13812:
--
Description:
Create a textfile table with a float column and insert some values:
{noformat}
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> create table test(f float);
No rows affected (0.237 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> insert into table test
values(-35664.76171875),(29497.349609375);
INFO : Session is already open
INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1463771904371_0006)
INFO : Map 1: 0/1
INFO : Map 1: 0/1
INFO : Map 1: 0/1
INFO : Map 1: 0(+1)/1
INFO : Map 1: 1/1
INFO : Loading data to table default.test from
hdfs://os-r6-pxwhrs-hiveserver2-3re-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-20_21-06-29_377_6487823927119226603-10/-ext-1
INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
rawDataSize=17]
No rows affected (11.069 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
+--+--+
| test.f |
+--+--+
| -35664.76171875 |
| 29497.349609375 |
+--+--+
2 rows selected (0.137 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
+---++--+--+
| col_name | data_type | comment |
+---++--+--+
| f | float | |
+---++--+--+
1 row selected (0.173 seconds)
{noformat}
Then change float type to string successfully, but when you select table, some
fractional digits are lost:
{noformat}
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> alter table test change column f
f string;
No rows affected (0.214 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
+---++--+--+
| col_name | data_type | comment |
+---++--+--+
| f | string | |
+---++--+--+
1 row selected (0.151 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
++--+
| test.f |
++--+
| -35664.76 |
| 29497.35 |
++--+
2 rows selected (0.141 seconds)
{noformat}
was:
Create a table with a float column and insert some values:
{noformat}
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> create table test(f float);
No rows affected (0.237 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> insert into table test
values(-35664.76171875),(29497.349609375);
INFO : Session is already open
INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1463771904371_0006)
INFO : Map 1: 0/1
INFO : Map 1: 0/1
INFO : Map 1: 0/1
INFO : Map 1: 0(+1)/1
INFO : Map 1: 1/1
INFO : Loading data to table default.test from
hdfs://os-r6-pxwhrs-hiveserver2-3re-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-20_21-06-29_377_6487823927119226603-10/-ext-1
INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
rawDataSize=17]
No rows affected (11.069 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
+--+--+
| test.f |
+--+--+
| -35664.76171875 |
| 29497.349609375 |
+--+--+
2 rows selected (0.137 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
+---++--+--+
| col_name | data_type | comment |
+---++--+--+
| f | float | |
+---++--+--+
1 row selected (0.173 seconds)
{noformat}
Then change float type to string successfully, but when you select table, some
fractional digits are lost:
{noformat}
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> alter table test change column f
f string;
No rows affected (0.214 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
+---++--+--+
| col_name | data_type | comment |
+---++--+--+
| f | string | |
+---++--+--+
1 row selected (0.151 seconds)
0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
++--+
| test.f |
++--+
| -35664.76 |
| 29497.35 |
++--+
2 rows selected (0.141 seconds)
{noformat}
> Change column from float to string group type will drop some factional digits
> -
>
> Key: HIVE-13812
> URL: https://issues.apache.org/jira/browse/HIVE-13812
> Project: Hive
> Issue Type: Bug
>Affect
[jira] [Updated] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-13797:
---
Status: Patch Available (was: In Progress)
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13797.01.patch
>
>
> It would save a bunch of googling if we could provide some examples of
> connection strings directly to beeline help message
> Eg:
> {code}
> ./bin/beeline --help
> Usage: java org.apache.hive.cli.beeline.BeeLine
>-uthe JDBC URL to connect to
>-r reconnect to last saved connect url (in
> conjunction with !save)
>-nthe username to connect as
>-pthe password to connect as
>-dthe driver class to use
>-i script file for initialization
>-e query that should be executed
>-f script file that should be executed
>-w (or) --password-file the password file to read
> password from
>--hiveconf property=value Use value for given property
>--hivevar name=valuehive variable name and value
>This is Hive specific settings in which
> variables
>can be set at session level and referenced
> in Hive
>commands or queries.
>--color=[true/false]control whether color is used for display
>--showHeader=[true/false] show column names in query results
>--headerInterval=ROWS; the interval between which heades are
> displayed
>--fastConnect=[true/false] skip building table/column list for
> tab-completion
>--autoCommit=[true/false] enable/disable automatic transaction commit
>--verbose=[true/false] show verbose error messages and debug info
>--showWarnings=[true/false] display connection warnings
>--showNestedErrs=[true/false] display nested errors
>--numberFormat=[pattern]format numbers using DecimalFormat pattern
>--force=[true/false]continue running script even after errors
>--maxWidth=MAXWIDTH the maximum width of the terminal
>--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
> columns
>--silent=[true/false] be more silent
>--autosave=[true/false] automatically save preferences
>--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
> result display
>Note that csv, and tsv are deprecated -
> use csv2, tsv2 instead
>--incremental=[true/false] Defaults to false. When set to false, the
> entire result set
>is fetched and buffered before being
> displayed, yielding optimal
>display column sizing. When set to true,
> result rows are displayed
>immediately as they are fetched, yielding
> lower latency and
>memory usage at the price of extra display
> column padding.
>Setting --incremental=true is recommended
> if you encounter an OutOfMemory
>on the client side (due to the fetched
> result set size being large).
>--truncateTable=[true/false]truncate table column when it exceeds
> length
>--delimiterForDSV=DELIMITER specify the delimiter for
> delimiter-separated values output format (default: |)
>--isolation=LEVEL set the transaction isolation level
>--nullemptystring=[true/false] set to true to get historic behavior of
> printing null as empty string
>--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline
> client side
>--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in
> the beeline client side
>--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
> beeline connected.
>Only works for HiveServer2 cluster mode.
>--help display this message
>
>Example:
> 1. beeline -u jdbc:hive2://localhost:1 username password
> 2. beeline -n username -p password -u jdbc:hive2://hs2.local:10012
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-13797:
---
Attachment: HIVE-13797.01.patch
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13797.01.patch
>
>
> It would save a bunch of googling if we could provide some examples of
> connection strings directly to beeline help message
> Eg:
> {code}
> ./bin/beeline --help
> Usage: java org.apache.hive.cli.beeline.BeeLine
>-uthe JDBC URL to connect to
>-r reconnect to last saved connect url (in
> conjunction with !save)
>-nthe username to connect as
>-pthe password to connect as
>-dthe driver class to use
>-i script file for initialization
>-e query that should be executed
>-f script file that should be executed
>-w (or) --password-file the password file to read
> password from
>--hiveconf property=value Use value for given property
>--hivevar name=valuehive variable name and value
>This is Hive specific settings in which
> variables
>can be set at session level and referenced
> in Hive
>commands or queries.
>--color=[true/false]control whether color is used for display
>--showHeader=[true/false] show column names in query results
>--headerInterval=ROWS; the interval between which heades are
> displayed
>--fastConnect=[true/false] skip building table/column list for
> tab-completion
>--autoCommit=[true/false] enable/disable automatic transaction commit
>--verbose=[true/false] show verbose error messages and debug info
>--showWarnings=[true/false] display connection warnings
>--showNestedErrs=[true/false] display nested errors
>--numberFormat=[pattern]format numbers using DecimalFormat pattern
>--force=[true/false]continue running script even after errors
>--maxWidth=MAXWIDTH the maximum width of the terminal
>--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
> columns
>--silent=[true/false] be more silent
>--autosave=[true/false] automatically save preferences
>--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
> result display
>Note that csv, and tsv are deprecated -
> use csv2, tsv2 instead
>--incremental=[true/false] Defaults to false. When set to false, the
> entire result set
>is fetched and buffered before being
> displayed, yielding optimal
>display column sizing. When set to true,
> result rows are displayed
>immediately as they are fetched, yielding
> lower latency and
>memory usage at the price of extra display
> column padding.
>Setting --incremental=true is recommended
> if you encounter an OutOfMemory
>on the client side (due to the fetched
> result set size being large).
>--truncateTable=[true/false]truncate table column when it exceeds
> length
>--delimiterForDSV=DELIMITER specify the delimiter for
> delimiter-separated values output format (default: |)
>--isolation=LEVEL set the transaction isolation level
>--nullemptystring=[true/false] set to true to get historic behavior of
> printing null as empty string
>--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline
> client side
>--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in
> the beeline client side
>--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
> beeline connected.
>Only works for HiveServer2 cluster mode.
>--help display this message
>
>Example:
> 1. beeline -u jdbc:hive2://localhost:1 username password
> 2. beeline -n username -p password -u jdbc:hive2://hs2.local:10012
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13797 started by Vihang Karajgaonkar.
--
> Provide a connection string example in beeline
> --
>
> Key: HIVE-13797
> URL: https://issues.apache.org/jira/browse/HIVE-13797
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> It would save a bunch of googling if we could provide some examples of
> connection strings directly to beeline help message
> Eg:
> {code}
> ./bin/beeline --help
> Usage: java org.apache.hive.cli.beeline.BeeLine
>-uthe JDBC URL to connect to
>-r reconnect to last saved connect url (in
> conjunction with !save)
>-nthe username to connect as
>-pthe password to connect as
>-dthe driver class to use
>-i script file for initialization
>-e query that should be executed
>-f script file that should be executed
>-w (or) --password-file the password file to read
> password from
>--hiveconf property=value Use value for given property
>--hivevar name=valuehive variable name and value
>This is Hive specific settings in which
> variables
>can be set at session level and referenced
> in Hive
>commands or queries.
>--color=[true/false]control whether color is used for display
>--showHeader=[true/false] show column names in query results
>--headerInterval=ROWS; the interval between which heades are
> displayed
>--fastConnect=[true/false] skip building table/column list for
> tab-completion
>--autoCommit=[true/false] enable/disable automatic transaction commit
>--verbose=[true/false] show verbose error messages and debug info
>--showWarnings=[true/false] display connection warnings
>--showNestedErrs=[true/false] display nested errors
>--numberFormat=[pattern]format numbers using DecimalFormat pattern
>--force=[true/false]continue running script even after errors
>--maxWidth=MAXWIDTH the maximum width of the terminal
>--maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
> columns
>--silent=[true/false] be more silent
>--autosave=[true/false] automatically save preferences
>--outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
> result display
>Note that csv, and tsv are deprecated -
> use csv2, tsv2 instead
>--incremental=[true/false] Defaults to false. When set to false, the
> entire result set
>is fetched and buffered before being
> displayed, yielding optimal
>display column sizing. When set to true,
> result rows are displayed
>immediately as they are fetched, yielding
> lower latency and
>memory usage at the price of extra display
> column padding.
>Setting --incremental=true is recommended
> if you encounter an OutOfMemory
>on the client side (due to the fetched
> result set size being large).
>--truncateTable=[true/false]truncate table column when it exceeds
> length
>--delimiterForDSV=DELIMITER specify the delimiter for
> delimiter-separated values output format (default: |)
>--isolation=LEVEL set the transaction isolation level
>--nullemptystring=[true/false] set to true to get historic behavior of
> printing null as empty string
>--addlocaldriverjar=DRIVERJARNAME Add driver jar file in the beeline
> client side
>--addlocaldrivername=DRIVERNAME Add drvier name needs to be supported in
> the beeline client side
>--showConnectedUrl=[true/false] Prompt HiveServer2s URI to which this
> beeline connected.
>Only works for HiveServer2 cluster mode.
>--help display this message
>
>Example:
> 1. beeline -u jdbc:hive2://localhost:1 username password
> 2. beeline -n username -p password -u jdbc:hive2://hs2.local:10012
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13809) Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size
[ https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294166#comment-15294166 ] Gopal V commented on HIVE-13809: [~wzheng]: before we actually subtract such a large amount of memory from the join algorithms, maybe we should figure out the input parameters to the bloom filter creation. The reason it wasn't accounted for strictly was because of the relatively small size of bloom filters - a 4 million keyset with 0.5% false positive rate should result in a bloom filter which is approx ~6Mb. Would be a good idea to figure out the false positive rate + estimated key count of the bloom filter which grew to hundreds of Mbs and see if there's an implementation issue there. > Hybrid Grace Hash Join memory usage estimation didn't take into account the > bloom filter size > - > > Key: HIVE-13809 > URL: https://issues.apache.org/jira/browse/HIVE-13809 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > > Memory estimation is important during hash table loading, because we need to > make the decision of whether to load the next hash partition in memory or > spill it. If the assumption is there's enough memory but it turns out not the > case, we will run into OOM problem. > Currently hybrid grace hash join memory usage estimation didn't take into > account the bloom filter size. In large test cases (TB scale) the bloom > filter grows as big as hundreds of MB, big enough to cause estimation error. > The solution is to count in the bloom filter size into memory estimation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13810:
-
Attachment: HIVE-13810.1.patch
cc-ing [~ashutoshc] for review.
> insert overwrite select from some table fails throwing
> org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13810
> URL: https://issues.apache.org/jira/browse/HIVE-13810
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13810.1.patch
>
>
> When running using beeline (as a non hdfs user)
> {noformat} CREATE EXTERNAL TABLE Batters_txt(
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
> )
> location '/user/tableau/Batters';
> drop table if exists Batters;
> CREATE TABLE Batters (
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
>)
> STORED AS orc tblproperties ("orc.compress"="SNAPPY");
> insert overwrite table Batters select * from Batters_txt;
> {noformat}
> runs into the following error:
> {code}
> 2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.security.AccessControlException: User does not
[jira] [Updated] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13810:
-
Status: Patch Available (was: Open)
> insert overwrite select from some table fails throwing
> org.apache.hadoop.security.AccessControlException
>
>
> Key: HIVE-13810
> URL: https://issues.apache.org/jira/browse/HIVE-13810
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13810.1.patch
>
>
> When running using beeline (as a non hdfs user)
> {noformat} CREATE EXTERNAL TABLE Batters_txt(
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
> )
> location '/user/tableau/Batters';
> drop table if exists Batters;
> CREATE TABLE Batters (
>Player STRING ,
>Team STRING ,
>League STRING ,
>Year SMALLINT,
>Games DOUBLE,
>AB DOUBLE,
>R DOUBLE,
>H DOUBLE,
>Doubles DOUBLE,
>Triples DOUBLE,
>HR DOUBLE,
>RBI DOUBLE,
>SB DOUBLE,
>CS DOUBLE,
>BB DOUBLE,
>SO DOUBLE,
>IBB DOUBLE,
>HBP DOUBLE,
>SH DOUBLE,
>SF DOUBLE,
>GIDP DOUBLE
>)
> STORED AS orc tblproperties ("orc.compress"="SNAPPY");
> insert overwrite table Batters select * from Batters_txt;
> {noformat}
> runs into the following error:
> {code}
> 2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
> operation.Operation (:()) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask.
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.security.AccessControlException: User does not belong to
> hdfs
>
[jira] [Updated] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13810:
-
Description:
When running using beeline (as a non hdfs user)
{noformat} CREATE EXTERNAL TABLE Batters_txt(
Player STRING ,
Team STRING ,
League STRING ,
Year SMALLINT,
Games DOUBLE,
AB DOUBLE,
R DOUBLE,
H DOUBLE,
Doubles DOUBLE,
Triples DOUBLE,
HR DOUBLE,
RBI DOUBLE,
SB DOUBLE,
CS DOUBLE,
BB DOUBLE,
SO DOUBLE,
IBB DOUBLE,
HBP DOUBLE,
SH DOUBLE,
SF DOUBLE,
GIDP DOUBLE
)
location '/user/tableau/Batters';
drop table if exists Batters;
CREATE TABLE Batters (
Player STRING ,
Team STRING ,
League STRING ,
Year SMALLINT,
Games DOUBLE,
AB DOUBLE,
R DOUBLE,
H DOUBLE,
Doubles DOUBLE,
Triples DOUBLE,
HR DOUBLE,
RBI DOUBLE,
SB DOUBLE,
CS DOUBLE,
BB DOUBLE,
SO DOUBLE,
IBB DOUBLE,
HBP DOUBLE,
SH DOUBLE,
SF DOUBLE,
GIDP DOUBLE
)
STORED AS orc tblproperties ("orc.compress"="SNAPPY");
insert overwrite table Batters select * from Batters_txt;
{noformat}
runs into the following error:
{code}
2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
operation.Operation (:()) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask.
org.apache.hadoop.security.AccessControlException: User does not belong to hdfs
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
at
org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at
org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.security.AccessControlException: User does not belong to hdfs
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
at o
[jira] [Updated] (HIVE-13787) LLAP: bug in recent security patches (wrong argument order; using full user name in id)
[ https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13787: Attachment: HIVE-13787.02.patch Small fix > LLAP: bug in recent security patches (wrong argument order; using full user > name in id) > --- > > Key: HIVE-13787 > URL: https://issues.apache.org/jira/browse/HIVE-13787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13787.01.patch, HIVE-13787.02.patch, > HIVE-13787.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13787) LLAP: bug in recent security patches (wrong argument order; using full user name in id)
[
https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294095#comment-15294095
]
Hive QA commented on HIVE-13787:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804854/HIVE-13787.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 132 failed/errored test(s), 9880 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-delete_where_non_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join_reordering_values.q-ptf_seqfile.q-auto_join18.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2.q-custom_input_output_format.q-join41.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-skewjoinopt8.q-union_remove_1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby6_map.q-join13.q-join_reorder3.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-mapreduce2.q-groupby7_noskew.q-vectorization_5.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-order.q-auto_join18_multi_distinct.q-union2.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_rcfile.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_4.q-groupby8_map.q-groupby4_map.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-stats13.q-stats2.q-ppd_gby_join.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-union_top_level.q-join11.q-auto_join1.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkClient - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_4
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_udf
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_bmj_schema_evolution
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_
[jira] [Updated] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11417: - Attachment: 0001-HIVE-11417.-Move-the-ReaderImpl-and-RowReaderImpl-to.patch Save the shorter version generated with -C. > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: > 0001-HIVE-11417.-Move-the-ReaderImpl-and-RowReaderImpl-to.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch.git > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11417: - Attachment: (was: 0001-HIVE-11417.-Move-the-ReaderImpl-and-RowReaderImpl-to.patch) > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch.git > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11417: - Attachment: 0001-HIVE-11417.-Move-the-ReaderImpl-and-RowReaderImpl-to.patch This just updates another expected qfile test output. > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch.git > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13628) Support for permanent functions - error handling if no restart
[ https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-13628: -- Attachment: HIVE-13628.3.patch > Support for permanent functions - error handling if no restart > -- > > Key: HIVE-13628 > URL: https://issues.apache.org/jira/browse/HIVE-13628 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch, > HIVE-13628.3.patch > > > Support for permanent functions - error handling if no restart -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13787) LLAP: bug in recent security patches (wrong argument order; using full user name in id)
[ https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294015#comment-15294015 ] Sergey Shelukhin commented on HIVE-13787: - Added one more fix > LLAP: bug in recent security patches (wrong argument order; using full user > name in id) > --- > > Key: HIVE-13787 > URL: https://issues.apache.org/jira/browse/HIVE-13787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13787.01.patch, HIVE-13787.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13787) LLAP: bug in recent security patches
[ https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13787: Attachment: HIVE-13787.01.patch > LLAP: bug in recent security patches > > > Key: HIVE-13787 > URL: https://issues.apache.org/jira/browse/HIVE-13787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13787.01.patch, HIVE-13787.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13787) LLAP: bug in recent security patches (wrong argument order; using full user name in id)
[ https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13787: Summary: LLAP: bug in recent security patches (wrong argument order; using full user name in id) (was: LLAP: bug in recent security patches) > LLAP: bug in recent security patches (wrong argument order; using full user > name in id) > --- > > Key: HIVE-13787 > URL: https://issues.apache.org/jira/browse/HIVE-13787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13787.01.patch, HIVE-13787.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[ https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293974#comment-15293974 ] Naveen Gangam commented on HIVE-13502: -- The test failures do not seem related to the fix/patch. The newly added test case passes. +1 for me. > Beeline doesnt support session parameters in JDBC URL as documentation states. > -- > > Key: HIVE-13502 > URL: https://issues.apache.org/jira/browse/HIVE-13502 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, > HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, > HIVE-13502.5.patch, HIVE-13502.patch > > > https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs > documents that sessions variables like credentials etc are accepted as part > of the URL. However, Beeline does not support such URLs today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13566: --- Attachment: HIVE-13566.02.patch > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch, HIVE-13566.02.patch > > > This jira adds code and tests for auto-gather column stats. Golden file > update will be done in phase 2 - HIVE-11160 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13566: --- Status: Open (was: Patch Available) > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch, HIVE-13566.02.patch > > > This jira adds code and tests for auto-gather column stats. Golden file > update will be done in phase 2 - HIVE-11160 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13566) Auto-gather column stats - phase 1
[ https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13566: --- Status: Patch Available (was: Open) > Auto-gather column stats - phase 1 > -- > > Key: HIVE-13566 > URL: https://issues.apache.org/jira/browse/HIVE-13566 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13566.01.patch, HIVE-13566.02.patch > > > This jira adds code and tests for auto-gather column stats. Golden file > update will be done in phase 2 - HIVE-11160 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Status: Open (was: Patch Available) > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13409-java8.patch > > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Status: Patch Available (was: Open) > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13409-java8.patch > > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Attachment: HIVE-13409-java8.patch > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13409-java8.patch > > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Attachment: (was: HIVE-13409.patch) > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Status: Patch Available (was: Open) > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13409.patch > > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Attachment: HIVE-13409.patch > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13409.patch > > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-9660: Attachment: HIVE-9660.patch This patch does: * implements a PositionedOutputStream.Callback to track when compression blocks and RLE are finished. * Adds lengths to the OrcProto.RowIndexEntry. * Uses the lengths when determining the number of bytes to read when doing predicate push down. * Creates a callback for RowIndexEntry in the WriterImpl such that the entry isn't finalized until all of the streams do their callback. To ensure that the entry isn't finalized before all of the streams are added there is an activation after the last stream has been added to the RowIndexEntry. * Removing the positions and lengths from the RowIndexEntry for ispresent stream removal is done softly so that remaining callbacks don't get impacted. * The code dealing with the string columns and the dictionary vs direct encoding has been significantly cleaned up. * TreeWriter.writeStripe has been split into a flush method that will finalize all of the streams. * Lots of test case updates for the changes ORC file sizes. * A new test case that tests the callbacks. > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, > HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, > HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, > HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, > HIVE-9660.10.patch, HIVE-9660.11.patch, HIVE-9660.patch, HIVE-9660.patch, > HIVE-9660.patch, owen-hive-9660.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13783) Display a secondary prompt on beeline for multi-line statements
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13783:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Thanks [~vihangk1] for your contribution.
I committed it to master
> Display a secondary prompt on beeline for multi-line statements
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HIVE-13783.01.patch
>
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13783) Display a secondary prompt on beeline for multi-line statements
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13783:
---
Summary: Display a secondary prompt on beeline for multi-line statements
(was: No secondary prompt)
> Display a secondary prompt on beeline for multi-line statements
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13783.01.patch
>
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13783) No secondary prompt
[
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293855#comment-15293855
]
Sergio Peña commented on HIVE-13783:
Thanks [~vihangk1] for the patch, and [~sircodesalot] for the review.
The patch looks good and simple.
+1
> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13783.01.patch
>
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1>
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and
> pressing Enter, I get back the normal prompt. This easily makes me believe
> I'm about to type a new command from scratch, e.g. a select query as in the
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened
> or anything like this, a differently looking secondary prompt should be
> displayed rather than the normal prompt; as this is done in e.g. hive,
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13800) Disable auth enabled by default on LLAP UI for secure clusters
[ https://issues.apache.org/jira/browse/HIVE-13800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293835#comment-15293835 ] Sergey Shelukhin commented on HIVE-13800: - +1 > Disable auth enabled by default on LLAP UI for secure clusters > -- > > Key: HIVE-13800 > URL: https://issues.apache.org/jira/browse/HIVE-13800 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13800.01.patch > > > There's no sensitive information that I'm aware of. (The logs would be the > most sensitive). > Similar to the HS2 UI, the LLAP UI can be default unprotected even on secure > clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13801) LLAP UI does not seem to accept credentials
[
https://issues.apache.org/jira/browse/HIVE-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293829#comment-15293829
]
Siddharth Seth commented on HIVE-13801:
---
I don't have logs at DEBUG level, will try generating them the next time I have
a secure cluster.
I did a quick scan of the code, and it seems to be identical to HS2 handling.
HS2 does not have this enabled by default - so I wasn't able to test that out.
I suspect that will not work either when secured.
At info level, these are the only relevant logs. The UI shows the same thing.
This was after logging in with the same user as the one running llap.
{code}
2016-05-18T22:11:30,460 WARN [llap-web-40[]]: server.AuthenticationFilter
(:()) - Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
2016-05-18T22:11:32,642 WARN [llap-web-38[]]: server.AuthenticationFilter
(:()) - Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
2016-05-18T22:11:33,648 WARN [llap-web-39[]]: server.AuthenticationFilter
(:()) - Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
{code}
> LLAP UI does not seem to accept credentials
>
>
> Key: HIVE-13801
> URL: https://issues.apache.org/jira/browse/HIVE-13801
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Siddharth Seth
>Priority: Critical
>
> Effectively making it unusable on a secure cluster.
> This could well be a misconfiguration of the cluster - but I tried using the
> same credentials against the YARN Timeline Server - and that worked fine.
> Steps to obtain credentials. kinit - start and configure firefox to use
> SPNEGO, try accessing the UGI.
> cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13682) EOFException with fast hashtable
[
https://issues.apache.org/jira/browse/HIVE-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293827#comment-15293827
]
Sergey Shelukhin commented on HIVE-13682:
-
Makes sense
> EOFException with fast hashtable
>
>
> Key: HIVE-13682
> URL: https://issues.apache.org/jira/browse/HIVE-13682
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Matt McCline
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13682.01.patch
>
>
> While testing something else on recent master, w/Tez 0.8.3, this happened
> (TPCDS q27)
> {noformat}
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:399)
> ... 20 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:106)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:131)
> ... 4 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.EOFException
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:106)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:304)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:185)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:181)
> at
> org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:104)
> ... 5 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readCheckNull(BinarySortableDeserializeRead.java:182)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashTable.putRow(VectorMapJoinFastLongHashTable.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.putRow(VectorMapJoinFastTableContainer.java:181)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:98)
> ... 9 more
> {noformat}
> There's no error if fast hashtable is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.
[
https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293806#comment-15293806
]
Hive QA commented on HIVE-13502:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804965/HIVE-13502.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 9314 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt15.q-join39.q-avro_joins_native.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3_noskew_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_reduce_deduplicate_exclude_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_div0
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/341/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/341/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws
[jira] [Updated] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13249: - Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Committed to master and branch-1. Thanks Eugene for the review! > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, > HIVE-13249.11.patch, HIVE-13249.12.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, > HIVE-13249.6.patch, HIVE-13249.7.patch, HIVE-13249.8.patch, > HIVE-13249.9.patch, HIVE-13249.branch-1.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13249: - Attachment: HIVE-13249.branch-1.patch > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, > HIVE-13249.11.patch, HIVE-13249.12.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, > HIVE-13249.6.patch, HIVE-13249.7.patch, HIVE-13249.8.patch, > HIVE-13249.9.patch, HIVE-13249.branch-1.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Status: Patch Available (was: Open) > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Fix For: 2.0.1 > > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Attachment: HIVE-13584.1.patch > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Fix For: 2.0.1 > > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Status: Open (was: Patch Available) > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Fix For: 2.0.1 > > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293690#comment-15293690 ] Jesus Camacho Rodriguez commented on HIVE-13068: [~ashutoshc], I have done another pass through the new q test files. I have created follow-up JIRA cases for all the issues discussed in RB. I will upload a new patch shortly with the regenerated q files after the new updates to the patch. Let's try to get it in, and then I'll continue with the rest. Thanks > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions
[ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293687#comment-15293687 ] Wei Zheng commented on HIVE-13249: -- Test failures unrelated > Hard upper bound on number of open transactions > --- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, > HIVE-13249.11.patch, HIVE-13249.12.patch, HIVE-13249.2.patch, > HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, > HIVE-13249.6.patch, HIVE-13249.7.patch, HIVE-13249.8.patch, HIVE-13249.9.patch > > > We need to have a safeguard by adding an upper bound for open transactions to > avoid huge number of open-transaction requests, usually due to improper > configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
