[jira] [Commented] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179504#comment-15179504
]
Alejandro Fernandez commented on HIVE-13063:
[~jdere], hopefully this is the last patch after changing char(0) to nul
character
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: (was: HIVE-13063.patch)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Status: Patch Available (was: Open)
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13063) Create UDFs for CHR and REPLACE
[
https://issues.apache.org/jira/browse/HIVE-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated HIVE-13063:
---
Attachment: HIVE-13063.patch
> Create UDFs for CHR and REPLACE
>
>
> Key: HIVE-13063
> URL: https://issues.apache.org/jira/browse/HIVE-13063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Alejandro Fernandez
>Assignee: Alejandro Fernandez
> Fix For: 2.1.0
>
> Attachments: HIVE-13063.patch, Screen Shot 2016-02-17 at 7.20.57
> PM.png, Screen Shot 2016-02-17 at 7.21.07 PM.png
>
>
> Create UDFS for these functions.
> CHR: convert n where n : [0, 256) into the ascii equivalent as a varchar. If
> n is less than 0 or greater than 255, return the empty string. If n is 0,
> return null.
> REPLACE: replace all substrings of 'str' that match 'search' with 'rep'.
> Example. SELECT REPLACE('Hack and Hue', 'H', 'BL');
> Equals 'BLack and BLue'"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179481#comment-15179481 ] Gopal V commented on HIVE-13205: I thought the bottleneck which was there earlier was fixed in HIVE-7344 (hive-1.0?), but the shuffle still moves all rows at least once. So your shuffle operation will end up being the bottleneck if id has a low nDV. Best to print out the summary on a lower run and see if it is running 1 reducer forever or not. > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few records, the query is running successfully. But for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179469#comment-15179469 ] Rahul commented on HIVE-13205: -- Nope, I have tried this with 1 M record. Keep running. I am running this on 226 node hadoop cluster with hive-.14. > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few records, the query is running successfully. But for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179457#comment-15179457 ] Gopal V commented on HIVE-13205: Doesn't first_value with descending sort do that? > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few records, the query is running successfully. But for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179454#comment-15179454 ] Rahul commented on HIVE-13205: -- I have tried many thing, but not able to solve the issue. In the mean while can you please suggest some other method to fill all null with the last known value in the same column. Can we achieve this using lag(). > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few records, the query is running successfully. But for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[
https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179428#comment-15179428
]
Gopal V commented on HIVE-13205:
Actually, that is not a fix - that's something that lets us get closer to the
problem.
I think this query had a streaming rewrite which works similar to
{{Select price,time, id, first_value(price,true) over (partition by id order by
time desc) as LatestPrice from table;}}
> Job with last_value() function keep running forever.
>
>
> Key: HIVE-13205
> URL: https://issues.apache.org/jira/browse/HIVE-13205
> Project: Hive
> Issue Type: Bug
>Reporter: Rahul
>
> Hi,
> I am running following query to fill all null with the last known value in
> the column:
> Select price,time, id,last_value(price,true) over (partition by id order by
> time) as LatestPrice from table;
> For few records, the query is running successfully. But for large number of
> records (2 Bn), the query keep running forever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179427#comment-15179427 ] Rahul commented on HIVE-13205: -- Gopal, I have tried the same with tez as well, but the reducer is still running forever. Also, The query works fine if each partition have number of records less than 30,000. > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few records, the query is running successfully. But for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179416#comment-15179416 ] Szehon Ho commented on HIVE-12270: -- I comment on the review board, lets go with original patch. I studied the code a little bit after our discussion, seems like if we just use raw MSC like I suggest to avoid caching Hive object, then if I understand it right we will never close it if DbTokenStore is invoked by thrift threads.. whereas Hive object does have a static close call later in the thread's life, or it will get closed when Hive.get() is invoked with another conf. So I'll +1 on patch1, it sucks to have to cache and pass the Hive object as its designed to be thread-local, but it's better than risking HMS leaks. I'm not sure a better way, any thoughts are welcome. In this case, we are sure there's no concurrency problem like HIVE-13002 as we are only using the Hive's MSC. Sorry for wasting the time.. I would attach patch1 with thrift to trigger HiveQA for you but I guess its deleted. > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.1.patch, > HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179413#comment-15179413 ] Gopal V commented on HIVE-13205: If you are using Tez, can you run the query with a few million rows and post the summary output (set hive.tez.exec.print.summary=true; to enable). > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few records, the query is running successfully. But for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13112) Expose Lineage information in case of CTAS
[
https://issues.apache.org/jira/browse/HIVE-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179392#comment-15179392
]
Hive QA commented on HIVE-13112:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789075/HIVE-13112.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 226 failed/errored test(s), 9781 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition_authorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_update_status
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_create_temp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_view_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_view_disable_cbo_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autogen_colalias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_SortUnionTransposeRule
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_unionDistinct_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_nested_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_colstats_all_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_default_prop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_char
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_uses_database_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_varchar
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_ddl1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_empty_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_ddl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_global_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input46
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join41
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_uncompressed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_union_src
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nestedvirtual
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hiv
[jira] [Updated] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul updated HIVE-13205: - Description: Hi, I am running following query to fill all null with the last known value in the column: Select price,time, id,last_value(price,true) over (partition by id order by time) as LatestPrice from table; For few records, the query is running successfully. But for large number of records (2 Bn), the query keep running forever. was: Hi, I am running following query to fill all null with the last known value in the column: Select price,time, id,last_value(price,true) over (partition by id order by time) as LatestPrice from table; For few record, the query is running successfully. Buy for large number of records (2 Bn), the query keep running forever. > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few records, the query is running successfully. But for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul updated HIVE-13205: - Description: Hi, I am running following query to fill all null with the last known value in the column: Select price,time, id,last_value(price,true) over (partition by id order by time) as LatestPrice from table; For few record, the query is running successfully. Buy for large number of records (2 Bn), the query keep running forever. > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > > Hi, > I am running following query to fill all null with the last known value in > the column: > Select price,time, id,last_value(price,true) over (partition by id order by > time) as LatestPrice from table; > For few record, the query is running successfully. Buy for large number of > records (2 Bn), the query keep running forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13205) Job with last_value() function keep running forever.
[ https://issues.apache.org/jira/browse/HIVE-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179367#comment-15179367 ] Rahul commented on HIVE-13205: -- Hi, I am running following query to fill all null with the last known value in the column: Select price,time, id,last_value(price,true) over (partition by id order by time) as LatestPrice from table; For few record, the query is running successfully. Buy for large number of records (2 Bn), the query keep running forever. > Job with last_value() function keep running forever. > > > Key: HIVE-13205 > URL: https://issues.apache.org/jira/browse/HIVE-13205 > Project: Hive > Issue Type: Bug >Reporter: Rahul > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal reassigned HIVE-4570: -- Assignee: Rajat Khandelwal > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179283#comment-15179283 ] Amareshwari Sriramadasu commented on HIVE-13188: +1 for https://issues.apache.org/jira/secure/attachment/12791124/HIVE-13188.03.patch > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13188.02.patch, HIVE-13188.03.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179281#comment-15179281 ] Amareshwari Sriramadasu commented on HIVE-13179: +1 for https://issues.apache.org/jira/secure/attachment/12791127/HIVE-13179.05.patch > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13179.05.patch, HIVE-13179.1.patch, > HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179279#comment-15179279 ] Amareshwari Sriramadasu commented on HIVE-11483: +1 > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch, > HIVE-11483.03.patch, HIVE-11483.04.patch > > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.1.patch Revised patch base on review feedback. Thanks Szehon. > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.1.patch, > HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: (was: HIVE-12270.patch) > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12270: --- Attachment: HIVE-12270.1.nothrift.patch > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.nothrift.patch, > HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179202#comment-15179202
]
Hive QA commented on HIVE-13129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790941/HIVE-13129.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9766 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7157/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7157/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7157/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790941 - PreCommit-HIVE-TRUNK-Build
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179182#comment-15179182 ] Aihua Xu commented on HIVE-13171: - [~szehon] It's good to add such unit tests. Can you check if the following are needed? 1. SessionManager and HiveConf properties don't need to be public, can be private, right? 2. Do we need to close the session after each test is done? > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13171.2.patch, HIVE-13171.patch > > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179167#comment-15179167
]
Prasanth Jayachandran commented on HIVE-13160:
--
This breaks compatibility when running CLI with old HMS. Earlier this exception
was ignored by CLI when session is started but now it throws. I am not sure
what the expected behavior here is, should the new clients (> 2.0) be
compatible with old HMS?
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function
[jira] [Updated] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13171: - Attachment: HIVE-13171.2.patch Adding apache license.. > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13171.2.patch, HIVE-13171.patch > > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13171: - Attachment: HIVE-13171.patch Add unit test for both QueryDisplay class, and the jamon template rendering this information in html. Unfortunately the original jsp is not converted into jamon, so not unit-testable as it is. > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13171.patch > > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13171: - Status: Patch Available (was: Open) [~aihuaxu], [~mohitsabharwal] can you guys help do a quick review? > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13171.patch > > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13072) ROW_NUMBER() function creates wrong results
[ https://issues.apache.org/jira/browse/HIVE-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179131#comment-15179131 ] Ashutosh Chauhan commented on HIVE-13072: - [~ychena] Did you try with version 1.1 Reporter has indicated that in Affect Version. [~Zyrix] Can you provide repro query for this? Also, if possible can you try this on master ? > ROW_NUMBER() function creates wrong results > --- > > Key: HIVE-13072 > URL: https://issues.apache.org/jira/browse/HIVE-13072 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Philipp Brandl >Assignee: Yongzhi Chen > > When using ROW_NUMBER() on tables with more than 25000 rows, the function > ROW_NUMBER() duplicates rows with separate row numbers. > Reproduce by using a large table with more than 25000 rows with distinct > values and then using a query involving ROW_NUMBER(). It will then result in > getting the same distinct values twice with separate row numbers apart by > 25000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[
https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179051#comment-15179051
]
Lefty Leverenz commented on HIVE-13156:
---
Nit pick: The description line for *hive.llap.daemon.queue.name* is longer
than 100 chars.
{code}
+LLAP_DAEMON_QUEUE_NAME("hive.llap.daemon.queue.name", null,
+"Queue name within which the llap slider application will run. Used in
LlapServiceDriver and package.py"),
{code}
(Its indentation matches most of the parameter descriptions in HiveConf.java,
even though the adjacent parameters are different.)
> Allow specifying the name of the queue in which llap will run
> -
>
> Key: HIVE-13156
> URL: https://issues.apache.org/jira/browse/HIVE-13156
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-13156.1.patch, HIVE-13156.2.patch
>
>
> llap service driver should accept a parameter for the llap queue name.
> cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179046#comment-15179046 ] Szehon Ho commented on HIVE-12270: -- Thanks Chaoyu, yea silly question :) Left some comment for consideration on the rb. > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch, HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-777) CREATE TABLE with STRUCT type
[
https://issues.apache.org/jira/browse/HIVE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179034#comment-15179034
]
Satyajit varma commented on HIVE-777:
-
Hi ,
Would like to know, if there is any kind of restriction with the number of
fields to be used in the STRUCT type, as i am having issues with having more
than 266 fields inside a STRUCT type.
i was able to create the Table with no issues, but SHOW CREATE TABLE doesnt
show fields are 266 field and queries fail with the same error.
PFB Jira link,_https://issues.apache.org/jira/browse/HIVE-13203.
Please reply in case there is any workaround to accept more fields.
Thank you.
> CREATE TABLE with STRUCT type
> -
>
> Key: HIVE-777
> URL: https://issues.apache.org/jira/browse/HIVE-777
> Project: Hive
> Issue Type: New Feature
> Components: Query Processor
>Reporter: Zheng Shao
>Assignee: He Yongqiang
> Fix For: 0.5.0
>
> Attachments: hive-777-2009-8-21-2.patch, hive-777-2009-8-21.patch,
> hive-777-2009-8-29.patch
>
>
> Currently we only support map/array in type definition.
> We should support STRUCT as well.
> {code}
> CREATE TABLE abc (
> pageid INT,
> ads STRUCT,
> userid INT
> );
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13156: -- Attachment: HIVE-13156.2.patch Updated patch with the option changed to queue. Thanks for the review. > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13156.1.patch, HIVE-13156.2.patch > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13201: - Attachment: HIVE-13201.1.patch > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13156: -- Status: Patch Available (was: Open) > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13156.1.patch, HIVE-13156.2.patch > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13201: - Status: Patch Available (was: Open) > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Dholakia updated HIVE-12049:
--
Attachment: HIVE-12049.11.patch
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.2.patch, HIVE-12049.3.patch, HIVE-12049.4.patch,
> HIVE-12049.5.patch, HIVE-12049.6.patch, HIVE-12049.7.patch, HIVE-12049.9.patch
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178984#comment-15178984 ] Chaoyu Tang commented on HIVE-12270: Directly using HMSHandler in HS2 is actually the HMS local mode. In a cluster with remote HMS but without Database connection information set for HS2, it will end with using embedded HMS and Derby, which is causing the problem in the case with multiple HS2 instances (HA). > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch, HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[
https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178873#comment-15178873
]
Hive QA commented on HIVE-4570:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790911/HIVE-4570.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9767 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7156/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7156/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7156/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790911 - PreCommit-HIVE-TRUNK-Build
> More information to user on GetOperationStatus in Hive Server2 when query is
> still executing
>
>
> Key: HIVE-4570
> URL: https://issues.apache.org/jira/browse/HIVE-4570
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Amareshwari Sriramadasu
> Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch,
> HIVE-4570.03.patch, HIVE-4570.04.patch
>
>
> Currently in Hive Server2, when the query is still executing only the status
> is set as STILL_EXECUTING.
> This issue is to give more information to the user such as progress and
> running job handles, if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12995) LLAP: Synthetic file ids need collision checks
[ https://issues.apache.org/jira/browse/HIVE-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12995: Attachment: HIVE-12995.01.patch Fixed the tests related to LLAP. The tests unrelated to LLAP do not fail for me on Java 7 (or 8). Some test failures are a known issue. > LLAP: Synthetic file ids need collision checks > -- > > Key: HIVE-12995 > URL: https://issues.apache.org/jira/browse/HIVE-12995 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12995.01.patch, HIVE-12995.patch > > > LLAP synthetic file ids do not have any way of checking whether a collision > occurs other than a data-error. > Synthetic file-ids have only been used with unit tests so far - but they will > be needed to add cache mechanisms to non-HDFS filesystems. > In case of Synthetic file-ids, it is recommended that we track the full-tuple > (path, mtime, len) in the cache so that a cache-hit for the synthetic file-id > can be compared against the parameters & only accepted if those match. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: regex-cpu-like.png > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe"; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: (was: regex-like-cpu.png) > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe"; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: regex-like-cpu.png > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe"; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13203) STRUCT type not allowing more than 266 fields. Table get created with more fields but throws error while querying against the table. SHOW CREATE TABLE skips having some f
[ https://issues.apache.org/jira/browse/HIVE-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satyajit varma updated HIVE-13203: -- Description: I am trying to create a table with multiple STRUCT column types, one of which having more than 300 fields. I get the table created with no problem, but when i load i face few errors in loading up the data.when i check SHOW CREATE TABLE TABLENAME, i get fields only till 266th column and rest of them do not appear in the table DDL.(check the issue at StateOrProvinc, search for StateOrProvinc in the second DDL). Would like to know, if there is any restriction on the number of fields to be using in STRUCT datatype. EXAMPLE: BELOW IS THE DDL i run in hive and the second below DLL is the TABLE DDL i get from SHOW CREATE TABLE. CREATE TABLE property_feature.property_feature_new( `_id` STRUCT<`mls_sysid`:String,`type`:String>, all_attrs STRUCT<`1`: String ,`10`: String ,`100`: String ,`105`: String ,`113`: String ,`118`: String ,`126`: String ,`1299`: String ,`1301`: String ,`1305`: String ,`1307`: String ,`1319`: String ,`1325`: String ,`1327`: String ,`1329`: String ,`1339`: String ,`1341`: String ,`1345`: String ,`1351`: String ,`1353`: String ,`1359`: String ,`1363`: String ,`1365`: String ,`1367`: String ,`1373`: String ,`1377`: String ,`1397`: String ,`1401`: String ,`1403`: String ,`1405`: String ,`1409`: String ,`1411`: String ,`1413`: String ,`1417`: String ,`1419`: String ,`1421`: String ,`1423`: String ,`1431`: String ,`1437`: String ,`1439`: String ,`144`: String ,`1441`: String ,`145`: String ,`1451`: String ,`1453`: String ,`146`: String ,`1467`: String ,`1469`: String ,`1483`: String ,`1485`: String ,`1491`: String ,`1493`: String ,`15`: String ,`1505`: String ,`1519`: String ,`1521`: String ,`1523`: String ,`155`: String ,`156`: String ,`158`: String ,`1586`: String ,`1678`: String ,`169`: String ,`17`: String ,`177`: String ,`1783`: String ,`1785`: String ,`1787`: String ,`1789`: String ,`1793`: String ,`1795`: String ,`1797`: String ,`1799`: String ,`1801`: String ,`1803`: String ,`1805`: String ,`1807`: String ,`1809`: String ,`1811`: String ,`1815`: String ,`1818`: String ,`1826`: String ,`1829`: String ,`1874`: String ,`1876`: String ,`1878`: String ,`1880`: String ,`1882`: String ,`207`: String ,`215`: String ,`226`: String ,`2293`: String ,`232`: String ,`239`: String ,`241`: String ,`245`: String ,`2456`: String ,`248`: String ,`2482`: String ,`2490`: String ,`2491`: String ,`2492`: String ,`2493`: String ,`2495`: String ,`25`: String ,`2502`: String ,`2503`: String ,`2517`: String ,`2522`: String ,`255`: String ,`256`: String ,`2573`: String ,`2574`: String ,`260`: String ,`263`: String ,`265`: String ,`276`: String ,`278`: String ,`286`: String ,`293`: String ,`296`: String ,`299`: String ,`3`: String ,`30`: String ,`307`: String ,`312`: String ,`320`: String ,`328`: String ,`340`: String ,`348`: String ,`354`: String ,`363`: String ,`364`: String ,`369`: String ,`375`: String ,`376`: String ,`383`: String ,`401`: String ,`405`: String ,`409`: String ,`412`: String ,`42`: String ,`423`: String ,`444`: String ,`445`: String ,`446`: String ,`447`: String ,`450`: String ,`451`: String ,`452`: String ,`453`: String ,`455`: String ,`458`: String ,`461`: String ,`463`: String ,`464`: String ,`51`: String ,`543472980`: String ,`57`: String ,`75`: String ,`90`: String ,`92`: String ,`AVMYN`: String ,`AccessibilityAmenities`: String ,`AddressPublic`: String ,`Amenities`: String ,`AmountofWaterFront`: String ,`AppliancesIncluded`: String ,`AppraiserInitials`: String ,`AppraiserName`: String ,`Basement`: String ,`BathsFull`: String ,`BathsHalf`: String ,`BathsTotal`: String ,`BedsTotal`: String ,`Block`: String ,`Book`: String ,`BuildingNumber`: String ,`City`: String ,`CloseDate`: String ,`ClosePrice`: String , `CoListAgentDirectWorkPhone`: String ,`CoListAgentEmail`: String ,`CoListAgentFullName`: String ,`CoListAgentMLSID`: String ,`CoListAgent_MUI`: String ,`CoListOfficeMLSID`: String ,`CoListOfficeName`: String ,`CoListOfficePhone`: String ,`CoListOffice_MUI`: String ,`Color`: String ,`CommentsYN`: String ,`Construction`: String ,`Cooling`: String ,`CountyOrParish`: String ,`DOM`: String ,`Deed`: String ,`Directions`: String ,`Driveway`: String ,`Electric`: String ,`Equipment`: String ,`Exterior`: String ,`Fireplace`: String ,`FloorNumber`: String ,`Floors`: String ,`FoundationMaterials`: String ,`FullTaxAmount`: String ,`Gas`: String ,`HeatFuel`: String ,`HeatSystem`: String ,`Internet`: String ,`ListAgentDirectWorkPhone`: String ,`ListAgentEmail`: String ,`ListAgentFullName`: String ,`ListAgentMLSID`: String ,`ListAgent_MUI`: String ,`ListOfficeMLSID`: String ,`ListOfficeName`: String ,`ListOfficePhone`: String ,`ListOffice_MUI`: String ,`ListPrice`: String ,`ListingContractDate`: String ,`Location`: String ,`Lot`: Str
[jira] [Updated] (HIVE-13203) STRUCT type not allowing more than 266 fields. Table get created with more fields but throws error while querying against the table. SHOW CREATE TABLE skips having some f
[ https://issues.apache.org/jira/browse/HIVE-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satyajit varma updated HIVE-13203: -- Affects Version/s: 1.2.0 Description: I am trying to create a table with multiple STRUCT column types, one of which having more than 300 fields. I get the table created with no problem, but when i load i face few errors in loading up the data.when i check SHOW CREATE TABLE TABLENAME, i get fields only till 266th column and rest of them do not appear in the table DDL. Would like to know, if there is any restriction on the number of fields to be using in STRUCT datatype. EXAMPLE: BELOW IS THE DDL i run in hive and the below in red is the TABLE DDL i get from SHOW CREATE TABLE. CREATE TABLE property_feature.property_feature_new( `_id` STRUCT<`mls_sysid`:String,`type`:String>, all_attrs STRUCT<`1`: String ,`10`: String ,`100`: String ,`105`: String ,`113`: String ,`118`: String ,`126`: String ,`1299`: String ,`1301`: String ,`1305`: String ,`1307`: String ,`1319`: String ,`1325`: String ,`1327`: String ,`1329`: String ,`1339`: String ,`1341`: String ,`1345`: String ,`1351`: String ,`1353`: String ,`1359`: String ,`1363`: String ,`1365`: String ,`1367`: String ,`1373`: String ,`1377`: String ,`1397`: String ,`1401`: String ,`1403`: String ,`1405`: String ,`1409`: String ,`1411`: String ,`1413`: String ,`1417`: String ,`1419`: String ,`1421`: String ,`1423`: String ,`1431`: String ,`1437`: String ,`1439`: String ,`144`: String ,`1441`: String ,`145`: String ,`1451`: String ,`1453`: String ,`146`: String ,`1467`: String ,`1469`: String ,`1483`: String ,`1485`: String ,`1491`: String ,`1493`: String ,`15`: String ,`1505`: String ,`1519`: String ,`1521`: String ,`1523`: String ,`155`: String ,`156`: String ,`158`: String ,`1586`: String ,`1678`: String ,`169`: String ,`17`: String ,`177`: String ,`1783`: String ,`1785`: String ,`1787`: String ,`1789`: String ,`1793`: String ,`1795`: String ,`1797`: String ,`1799`: String ,`1801`: String ,`1803`: String ,`1805`: String ,`1807`: String ,`1809`: String ,`1811`: String ,`1815`: String ,`1818`: String ,`1826`: String ,`1829`: String ,`1874`: String ,`1876`: String ,`1878`: String ,`1880`: String ,`1882`: String ,`207`: String ,`215`: String ,`226`: String ,`2293`: String ,`232`: String ,`239`: String ,`241`: String ,`245`: String ,`2456`: String ,`248`: String ,`2482`: String ,`2490`: String ,`2491`: String ,`2492`: String ,`2493`: String ,`2495`: String ,`25`: String ,`2502`: String ,`2503`: String ,`2517`: String ,`2522`: String ,`255`: String ,`256`: String ,`2573`: String ,`2574`: String ,`260`: String ,`263`: String ,`265`: String ,`276`: String ,`278`: String ,`286`: String ,`293`: String ,`296`: String ,`299`: String ,`3`: String ,`30`: String ,`307`: String ,`312`: String ,`320`: String ,`328`: String ,`340`: String ,`348`: String ,`354`: String ,`363`: String ,`364`: String ,`369`: String ,`375`: String ,`376`: String ,`383`: String ,`401`: String ,`405`: String ,`409`: String ,`412`: String ,`42`: String ,`423`: String ,`444`: String ,`445`: String ,`446`: String ,`447`: String ,`450`: String ,`451`: String ,`452`: String ,`453`: String ,`455`: String ,`458`: String ,`461`: String ,`463`: String ,`464`: String ,`51`: String ,`543472980`: String ,`57`: String ,`75`: String ,`90`: String ,`92`: String ,`AVMYN`: String ,`AccessibilityAmenities`: String ,`AddressPublic`: String ,`Amenities`: String ,`AmountofWaterFront`: String ,`AppliancesIncluded`: String ,`AppraiserInitials`: String ,`AppraiserName`: String ,`Basement`: String ,`BathsFull`: String ,`BathsHalf`: String ,`BathsTotal`: String ,`BedsTotal`: String ,`Block`: String ,`Book`: String ,`BuildingNumber`: String ,`City`: String ,`CloseDate`: String ,`ClosePrice`: String , `CoListAgentDirectWorkPhone`: String ,`CoListAgentEmail`: String ,`CoListAgentFullName`: String ,`CoListAgentMLSID`: String ,`CoListAgent_MUI`: String ,`CoListOfficeMLSID`: String ,`CoListOfficeName`: String ,`CoListOfficePhone`: String ,`CoListOffice_MUI`: String ,`Color`: String ,`CommentsYN`: String ,`Construction`: String ,`Cooling`: String ,`CountyOrParish`: String ,`DOM`: String ,`Deed`: String ,`Directions`: String ,`Driveway`: String ,`Electric`: String ,`Equipment`: String ,`Exterior`: String ,`Fireplace`: String ,`FloorNumber`: String ,`Floors`: String ,`FoundationMaterials`: String ,`FullTaxAmount`: String ,`Gas`: String ,`HeatFuel`: String ,`HeatSystem`: String ,`Internet`: String ,`ListAgentDirectWorkPhone`: String ,`ListAgentEmail`: String ,`ListAgentFullName`: String ,`ListAgentMLSID`: String ,`ListAgent_MUI`: String ,`ListOfficeMLSID`: String ,`ListOfficeName`: String ,`ListOfficePhone`: String ,`ListOffice_MUI`: String ,`ListPrice`: String ,`ListingContractDate`: String ,`Location`: String ,`Lot`: String ,`LotSizeArea`: String ,`MLSNumber`: String ,`
[jira] [Commented] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178774#comment-15178774 ] Gopal V commented on HIVE-13156: OPTION_LLAP_QUEUE_NAME="queuename", I think that should also be "--queue" to match Slider. LGTM - +1 > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13156.1.patch > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178762#comment-15178762 ] Szehon Ho commented on HIVE-12270: -- Sorry for the basic question.. but just wanted to clarify what is the issue from using HMSHandler in HS2? Ie, what is the behavior change if we change to HiveMetaStoreClient? > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.nothrift.patch, HIVE-12270.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13156: -- Attachment: HIVE-13156.1.patch The patch allows setting the queue name, as well as the slider AM size. Fixes the heap size for the default AM size of 1GB. [~gopalv] - please review. Tested locally. > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth > Attachments: HIVE-13156.1.patch > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned HIVE-13156: - Assignee: Siddharth Seth > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13156.1.patch > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Status: Patch Available (was: In Progress) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Status: In Progress (was: Patch Available) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.06.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.02.patch > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: (was: HIVE-13178.02.patch) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13202) LLAP: Replace use of ServerSocket with netty in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere resolved HIVE-13202. --- Resolution: Fixed Fix Version/s: llap committed to llap branch > LLAP: Replace use of ServerSocket with netty in LlapOutputFormatService > > > Key: HIVE-13202 > URL: https://issues.apache.org/jira/browse/HIVE-13202 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13186) ALTER TABLE RENAME should lowercase table name and hdfs location
[ https://issues.apache.org/jira/browse/HIVE-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13186: -- Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Committed to master/branch-1 > ALTER TABLE RENAME should lowercase table name and hdfs location > > > Key: HIVE-13186 > URL: https://issues.apache.org/jira/browse/HIVE-13186 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13186.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13196) UDFLike: reduce Regex NFA sizes
[ https://issues.apache.org/jira/browse/HIVE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178598#comment-15178598 ] Gopal V commented on HIVE-13196: update_where_partitioned & TestMiniTezCliDriver are related to Hbase metastore. Unlike that cbo_semijoin is related, but will rerun. > UDFLike: reduce Regex NFA sizes > --- > > Key: HIVE-13196 > URL: https://issues.apache.org/jira/browse/HIVE-13196 > Project: Hive > Issue Type: Improvement > Components: UDF >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-13196.1.patch > > > The NFAs built from complex regexes in UDFLike are extremely complex and > spend a lot of time doing simple expression matching with no backtracking. > Prevent NFA -> DFA explosion by using reluctant regex matches instead of > greedy matches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13196) UDFLike: reduce Regex NFA sizes
[
https://issues.apache.org/jira/browse/HIVE-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178592#comment-15178592
]
Hive QA commented on HIVE-13196:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790892/HIVE-13196.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9781 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7155/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7155/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7155/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790892 - PreCommit-HIVE-TRUNK-Build
> UDFLike: reduce Regex NFA sizes
> ---
>
> Key: HIVE-13196
> URL: https://issues.apache.org/jira/browse/HIVE-13196
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-13196.1.patch
>
>
> The NFAs built from complex regexes in UDFLike are extremely complex and
> spend a lot of time doing simple expression matching with no backtracking.
> Prevent NFA -> DFA explosion by using reluctant regex matches instead of
> greedy matches.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178548#comment-15178548
]
Aihua Xu commented on HIVE-13149:
-
1. Sure. I will change it back.
2. Even if the tasks are on the same thread, if the conf changes, we will get a
new HMS connection since we need to create it based on the new conf.
I'm investigating the test failures which seems to be related. Somehow getting
a MSC every time when we start the session hides the issue. Seems we should
make a copy of conf and pass to Hive object since if pass a reference of conf,
then later if we update the session conf, actually we are also updating the one
within Hive object. Then when we call get(conf), we are getting the old MSC
since conf is determined unchanged.
Probably the right fix is as you said, not to do many things in get() call.
Right now, I will just investigate the test failures and follow up to have
get() call cleaned.
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13108) Operators: SORT BY randomness is not safe with network partitions
[ https://issues.apache.org/jira/browse/HIVE-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13108: Target Version/s: 2.0.1 > Operators: SORT BY randomness is not safe with network partitions > - > > Key: HIVE-13108 > URL: https://issues.apache.org/jira/browse/HIVE-13108 > Project: Hive > Issue Type: Bug > Components: Spark, Tez >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.0.1 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13108.1.patch > > > SORT BY relies on a transient Random object, which is initialized once per > deserialize operation. > This results in complications during a network partition and when Tez/Spark > reuses a cached plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13200) Aggregation functions returning empty rows on partitioned columns
[
https://issues.apache.org/jira/browse/HIVE-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178503#comment-15178503
]
Sergey Shelukhin commented on HIVE-13200:
-
+1 pending tests
> Aggregation functions returning empty rows on partitioned columns
> -
>
> Key: HIVE-13200
> URL: https://issues.apache.org/jira/browse/HIVE-13200
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13200.1.patch
>
>
> Running aggregation functions like MAX, MIN, DISTINCT against partitioned
> columns will return empty rows if table has property:
> 'skip.header.line.count'='1'
> Reproduce:
> {noformat}
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (a int)
> PARTITIONED BY (b int)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
> TBLPROPERTIES('skip.header.line.count'='1');
> INSERT OVERWRITE TABLE test PARTITION (b = 1) VALUES (1), (2), (3), (4);
> INSERT OVERWRITE TABLE test PARTITION (b = 2) VALUES (1), (2), (3), (4);
> SELECT * FROM test;
> SELECT DISTINCT b FROM test;
> SELECT MAX(b) FROM test;
> SELECT DISTINCT a FROM test;
> {noformat}
> The output:
> {noformat}
> 0: jdbc:hive2://localhost:1/default> SELECT * FROM test;
> +-+-+--+
> | test.a | test.b |
> +-+-+--+
> | 2 | 1 |
> | 3 | 1 |
> | 4 | 1 |
> | 2 | 2 |
> | 3 | 2 |
> | 4 | 2 |
> +-+-+--+
> 6 rows selected (0.631 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT DISTINCT b FROM test;
> ++--+
> | b |
> ++--+
> ++--+
> No rows selected (47.229 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT MAX(b) FROM test;
> +---+--+
> | _c0 |
> +---+--+
> | NULL |
> +---+--+
> 1 row selected (49.508 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT DISTINCT a FROM test;
> ++--+
> | a |
> ++--+
> | 2 |
> | 3 |
> | 4 |
> ++--+
> 3 rows selected (46.859 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178499#comment-15178499 ] Sergey Shelukhin commented on HIVE-12558: - Test failures (llap and operation logging) look related. Spark timeouts are a known condition. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.wip.patch, sample-output.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178498#comment-15178498 ] Prasanth Jayachandran commented on HIVE-13199: -- I think it will get picked from hive-jdbc-standalone jar as it shades the world. > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13108) Operators: SORT BY randomness is not safe with network partitions
[ https://issues.apache.org/jira/browse/HIVE-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178497#comment-15178497 ] Gopal V commented on HIVE-13108: Nobody has reported this issue yet, but it is a low-risk change for a problem which is nearly impossible to detect. > Operators: SORT BY randomness is not safe with network partitions > - > > Key: HIVE-13108 > URL: https://issues.apache.org/jira/browse/HIVE-13108 > Project: Hive > Issue Type: Bug > Components: Spark, Tez >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.0.1 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13108.1.patch > > > SORT BY relies on a transient Random object, which is initialized once per > deserialize operation. > This results in complications during a network partition and when Tez/Spark > reuses a cached plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13108) Operators: SORT BY randomness is not safe with network partitions
[ https://issues.apache.org/jira/browse/HIVE-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178489#comment-15178489 ] Sergey Shelukhin commented on HIVE-13108: - Is this important enough for 2.0.1? > Operators: SORT BY randomness is not safe with network partitions > - > > Key: HIVE-13108 > URL: https://issues.apache.org/jira/browse/HIVE-13108 > Project: Hive > Issue Type: Bug > Components: Spark, Tez >Affects Versions: 1.3.0, 1.2.1, 2.0.0, 2.0.1 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13108.1.patch > > > SORT BY relies on a transient Random object, which is initialized once per > deserialize operation. > This results in complications during a network partition and when Tez/Spark > reuses a cached plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13199) NDC stopped working in LLAP logging
[ https://issues.apache.org/jira/browse/HIVE-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178483#comment-15178483 ] Sergey Shelukhin commented on HIVE-13199: - I think we should backport both to 2.0.1. Otherwise, the logging doesn't work there, does it? > NDC stopped working in LLAP logging > --- > > Key: HIVE-13199 > URL: https://issues.apache.org/jira/browse/HIVE-13199 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.1.0 >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-13199.1.patch > > > NDC context were missing from the log lines. Reason for it is NDC class is > part of log4j-1.2-api (bridge jar). This is added as compile time dependency. > Due to the absence of this jar in llap daemons, the NDC context failed to > initialize. Log4j2 replaced NDC with ThreadContext. Hence we need the bridge > jar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4897) Hive should handle AlreadyExists on retries when creating tables/partitions
[
https://issues.apache.org/jira/browse/HIVE-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178474#comment-15178474
]
Sergey Shelukhin commented on HIVE-4897:
I think the simplest path of the approach outlined above will work. I've done
similar work in HBase to make increment operation retries idempotent (so the
requirements were more stringent and tokens actually needed to survive restarts
and failover), and it was pretty manageable. With relaxed requirements like no
persistence it should be simpler still.
> Hive should handle AlreadyExists on retries when creating tables/partitions
> ---
>
> Key: HIVE-4897
> URL: https://issues.apache.org/jira/browse/HIVE-4897
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Aihua Xu
> Attachments: HIVE-4897.patch, hive-snippet.log
>
>
> Creating new tables/partitions may fail with an AlreadyExistsException if
> there is an error part way through the creation and the HMS tries again
> without properly cleaning up or checking if this is a retry.
> While partitioning a new table via a script on distributed hive (MetaStore on
> the same machine) there was a long timeout and then:
> {code}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> AlreadyExistsException(message:Partition already exists:Partition( ...
> {code}
> I am assuming this is due to retry. Perhaps already-exists on retry could be
> handled better.
> A similar error occurred while creating a table through Impala, which issued
> a single createTable call that failed with an AlreadyExistsException. See the
> logs related to table tmp_proc_8_d2b7b0f133be455ca95615818b8a5879_7 in the
> attached hive-snippet.log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178463#comment-15178463 ] Sergey Shelukhin commented on HIVE-11675: - All the failures are known issues. > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch, > HIVE-11675.09.patch, HIVE-11675.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[
https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178435#comment-15178435
]
Wei Zheng commented on HIVE-13201:
--
Create a non-ACID table
{code}
hive> create table nonacid (a int);
OK
Time taken: 0.939 seconds
hive> desc formatted nonacid;
OK
# col_name data_type comment
a int
# Detailed Table Information
Database: default
Owner: hive
CreateTime: Thu Mar 03 19:22:56 UTC 2016
LastAccessTime: UNKNOWN
Protect Mode: None
Retention: 0
Location:
hdfs://jvaria-hive-1-4.novalocal:8020/apps/hive/warehouse/nonacid
Table Type: MANAGED_TABLE
Table Parameters:
transient_lastDdlTime 1457032976
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat:org.apache.hadoop.mapred.TextInputFormat
OutputFormat:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets:-1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format1
Time taken: 0.26 seconds, Fetched: 26 row(s)
hive> alter table nonacid compact 'major';
Compaction enqueued.
OK
Time taken: 0.098 seconds
{code}
On the metastore side, the compaction request is enqueued
{code}
mysql> select * from COMPACTION_QUEUE;
+---+-+--+--+--+-+--+---+---+---+--+--+
| CQ_ID | CQ_DATABASE | CQ_TABLE | CQ_PARTITION | CQ_STATE | CQ_TYPE |
CQ_WORKER_ID | CQ_START | CQ_RUN_AS | CQ_HIGHEST_TXN_ID | CQ_META_INFO |
CQ_HADOOP_JOB_ID |
+---+-+--+--+--+-+--+---+---+---+--+--+
| 5 | default | nonacid | NULL | r| a | NULL
| 1457033021000 | hive | NULL | NULL | NULL
|
+---+-+--+--+--+-+--+---+---+---+--+--+
1 row in set (0.00 sec)
{code}
> Compaction shouldn't be allowed on non-ACID table
> -
>
> Key: HIVE-13201
> URL: https://issues.apache.org/jira/browse/HIVE-13201
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Looks like compaction is allowed on non-ACID table, although that's of no
> sense and does nothing. Moreover the compaction request will be enqueued into
> COMPACTION_QUEUE metastore table, which brings unnecessary overhead.
> We should prevent compaction commands being allowed on non-ACID tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178414#comment-15178414
]
Szehon Ho commented on HIVE-13149:
--
1. Yea I think the right fix would be to not have the method 'get' to do so
many things and actually just return if its already set on thread-local, but
probably that's more change to change all the existing calls.. so I would say
not to add yet another flavor that might confuse even more.
2. Sounds good.. just wanted to check there's no downside to that right? I
just wonder why the original guy tried to initialize it at that time, like
would it be repeated for every task even if on the same thread?
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13106) STARTUP_MSG and SHUTDOWN_MSG are added to HiveMetaStore
[ https://issues.apache.org/jira/browse/HIVE-13106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13106: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Patch committed. Thanks Shinichi for the patch. > STARTUP_MSG and SHUTDOWN_MSG are added to HiveMetaStore > --- > > Key: HIVE-13106 > URL: https://issues.apache.org/jira/browse/HIVE-13106 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Trivial > Fix For: 2.1.0 > > Attachments: HIVE-13106.1.patch > > > When we use Hive remote metastore, STARTUP_MSG and SHUTDOWN_MSG are output in > metastore log like HiveServer2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[
https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178223#comment-15178223
]
Hive QA commented on HIVE-12244:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791208/HIVE-12244.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 4 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/126/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/126/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-METASTORE-Test-126/
Messages:
{noformat}
LXC derby found.
LXC derby is not started. Starting container...
Container started.
Preparing derby container...
Container prepared.
Calling /hive/testutils/metastore/dbs/derby/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/derby/execute.sh ...
Tests executed.
LXC mysql found.
LXC mysql is not started. Starting container...
Container started.
Preparing mysql container...
Container prepared.
Calling /hive/testutils/metastore/dbs/mysql/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/mysql/execute.sh ...
Tests executed.
LXC oracle found.
LXC oracle is not started. Starting container...
Container started.
Preparing oracle container...
Container prepared.
Calling /hive/testutils/metastore/dbs/oracle/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/oracle/execute.sh ...
Tests executed.
LXC postgres found.
LXC postgres is not started. Starting container...
Container started.
Preparing postgres container...
Container prepared.
Calling /hive/testutils/metastore/dbs/postgres/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/postgres/execute.sh ...
Tests executed.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791208 - PreCommit-HIVE-METASTORE-Test
> Refactoring code for avoiding of comparison of Strings and do comparison on
> Path
>
>
> Key: HIVE-12244
> URL: https://issues.apache.org/jira/browse/HIVE-12244
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1
>Reporter: Alina Abramova
>Assignee: Alina Abramova
>Priority: Minor
> Labels: patch
> Fix For: 1.2.1
>
> Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch,
> HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch,
> HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch,
> HIVE-12244.8.patch, HIVE-12244.9.patch
>
>
> In Hive often String is used for representation path and it causes new issues.
> We need to compare it with equals() but comparing Strings often is not right
> in terms comparing paths .
> I think if we use Path from org.apache.hadoop.fs we will avoid new problems
> in future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[
https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178211#comment-15178211
]
Hive QA commented on HIVE-13188:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791124/HIVE-13188.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9735 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-dynpart_sort_optimization2.q-cte_mat_1.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-mapreduce1.q-vectorization_14.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7153/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7153/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7153/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791124 - PreCommit-HIVE-TRUNK-Build
> Allow users of RetryingThriftClient to close transport
> --
>
> Key: HIVE-13188
> URL: https://issues.apache.org/jira/browse/HIVE-13188
> Project: Hive
> Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13188.02.patch, HIVE-13188.03.patch
>
>
> RetryingThriftCLIClient opens a TTransport and leaves it open. there should
> be a way to close that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alina Abramova updated HIVE-12244: -- Attachment: HIVE-12244.9.patch > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, > HIVE-12244.8.patch, HIVE-12244.9.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13200) Aggregation functions returning empty rows on partitioned columns
[
https://issues.apache.org/jira/browse/HIVE-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13200:
Status: Patch Available (was: Open)
Need code review.
> Aggregation functions returning empty rows on partitioned columns
> -
>
> Key: HIVE-13200
> URL: https://issues.apache.org/jira/browse/HIVE-13200
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 2.0.0, 1.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13200.1.patch
>
>
> Running aggregation functions like MAX, MIN, DISTINCT against partitioned
> columns will return empty rows if table has property:
> 'skip.header.line.count'='1'
> Reproduce:
> {noformat}
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (a int)
> PARTITIONED BY (b int)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
> TBLPROPERTIES('skip.header.line.count'='1');
> INSERT OVERWRITE TABLE test PARTITION (b = 1) VALUES (1), (2), (3), (4);
> INSERT OVERWRITE TABLE test PARTITION (b = 2) VALUES (1), (2), (3), (4);
> SELECT * FROM test;
> SELECT DISTINCT b FROM test;
> SELECT MAX(b) FROM test;
> SELECT DISTINCT a FROM test;
> {noformat}
> The output:
> {noformat}
> 0: jdbc:hive2://localhost:1/default> SELECT * FROM test;
> +-+-+--+
> | test.a | test.b |
> +-+-+--+
> | 2 | 1 |
> | 3 | 1 |
> | 4 | 1 |
> | 2 | 2 |
> | 3 | 2 |
> | 4 | 2 |
> +-+-+--+
> 6 rows selected (0.631 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT DISTINCT b FROM test;
> ++--+
> | b |
> ++--+
> ++--+
> No rows selected (47.229 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT MAX(b) FROM test;
> +---+--+
> | _c0 |
> +---+--+
> | NULL |
> +---+--+
> 1 row selected (49.508 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT DISTINCT a FROM test;
> ++--+
> | a |
> ++--+
> | 2 |
> | 3 |
> | 4 |
> ++--+
> 3 rows selected (46.859 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13200) Aggregation functions returning empty rows on partitioned columns
[
https://issues.apache.org/jira/browse/HIVE-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13200:
Attachment: HIVE-13200.1.patch
When select only on partition column with aggregation function, the
MetadataOnlyOptimizer is used for the metadata is good enough to provide the
needed values for the partition column. But it is not true, when need skip
header and footer which should operate on real rows.
The empty return for the select on column b is because the
MetadataOnlyOptimizer creates OneNullRowRecordReader which alway return empty
return after first skip.
The patch fixes the issue by do not use MetadataOnlyOptimizer when need skip
headers/footers.
> Aggregation functions returning empty rows on partitioned columns
> -
>
> Key: HIVE-13200
> URL: https://issues.apache.org/jira/browse/HIVE-13200
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13200.1.patch
>
>
> Running aggregation functions like MAX, MIN, DISTINCT against partitioned
> columns will return empty rows if table has property:
> 'skip.header.line.count'='1'
> Reproduce:
> {noformat}
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (a int)
> PARTITIONED BY (b int)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
> TBLPROPERTIES('skip.header.line.count'='1');
> INSERT OVERWRITE TABLE test PARTITION (b = 1) VALUES (1), (2), (3), (4);
> INSERT OVERWRITE TABLE test PARTITION (b = 2) VALUES (1), (2), (3), (4);
> SELECT * FROM test;
> SELECT DISTINCT b FROM test;
> SELECT MAX(b) FROM test;
> SELECT DISTINCT a FROM test;
> {noformat}
> The output:
> {noformat}
> 0: jdbc:hive2://localhost:1/default> SELECT * FROM test;
> +-+-+--+
> | test.a | test.b |
> +-+-+--+
> | 2 | 1 |
> | 3 | 1 |
> | 4 | 1 |
> | 2 | 2 |
> | 3 | 2 |
> | 4 | 2 |
> +-+-+--+
> 6 rows selected (0.631 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT DISTINCT b FROM test;
> ++--+
> | b |
> ++--+
> ++--+
> No rows selected (47.229 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT MAX(b) FROM test;
> +---+--+
> | _c0 |
> +---+--+
> | NULL |
> +---+--+
> 1 row selected (49.508 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT DISTINCT a FROM test;
> ++--+
> | a |
> ++--+
> | 2 |
> | 3 |
> | 4 |
> ++--+
> 3 rows selected (46.859 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178060#comment-15178060 ] Ashutosh Chauhan commented on HIVE-5370: +1 > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177892#comment-15177892
]
Aihua Xu commented on HIVE-13149:
-
Thanks [~szehon] to review the code.
1. get(Conf) and setConf(Conf) are almost the same, but get(Conf) will give you
an instance of Hive, but setConf(Conf) won't as. Of course get(Conf) can do
what setConf(Conf) does. I feel it's more clear. How do you think? I can remove
it if you don't feel the same.
2. MSC will get initialized when it's actually getting used. In many Task
threads, the tasks actually never need to access databases, but right now we
still open a connection to HMS.
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay Goyal updated HIVE-4570: --- Assignee: (was: Vaibhav Gumashta) > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[
https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177879#comment-15177879
]
Hive QA commented on HIVE-11675:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790842/HIVE-11675.09.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9767 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7152/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7152/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7152/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790842 - PreCommit-HIVE-TRUNK-Build
> make use of file footer PPD API in ETL strategy or separate strategy
>
>
> Key: HIVE-11675
> URL: https://issues.apache.org/jira/browse/HIVE-11675
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch,
> HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch,
> HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch,
> HIVE-11675.09.patch, HIVE-11675.patch
>
>
> Need to take a look at the best flow. It won't be much different if we do
> filtering metastore call for each partition. So perhaps we'd need the custom
> sync point/batching after all.
> Or we can make it opportunistic and not fetch any footers unless it can be
> pushed down to metastore or fetched from local cache, that way the only slow
> threaded op is directory listings
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akshay Goyal updated HIVE-4570: --- Assignee: Vaibhav Gumashta (was: Akshay Goyal) > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Vaibhav Gumashta > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177876#comment-15177876 ] Akshay Goyal commented on HIVE-4570: Unable to continue on this due to other commitments. If anybody else want to pick this up from here please do. HIVE-4570.04.patch is the latest one. And https://reviews.apache.org/r/42134/ the review request for further suggestions. > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.04.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13129) CliService leaks HMS connection
[
https://issues.apache.org/jira/browse/HIVE-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177859#comment-15177859
]
Chaoyu Tang commented on HIVE-13129:
+1
> CliService leaks HMS connection
> ---
>
> Key: HIVE-13129
> URL: https://issues.apache.org/jira/browse/HIVE-13129
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13129.2.patch, HIVE-13129.patch
>
>
> HIVE-12790 fixes the HMS connection leaking. But seems there is one more
> connection from CLIService.
> The init() function in CLIService will get info from DB but we never close
> the HMS connection for this service main thread.
> {noformat}
> // creates connection to HMS and thus *must* occur after kerberos login
> above
> try {
> applyAuthorizationConfigPolicy(hiveConf);
> } catch (Exception e) {
> throw new RuntimeException("Error applying authorization policy on hive
> configuration: "
> + e.getMessage(), e);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4897) Hive should handle AlreadyExists on retries when creating tables/partitions
[
https://issues.apache.org/jira/browse/HIVE-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177849#comment-15177849
]
Aihua Xu commented on HIVE-4897:
Yeah. That scenario definitely will cause the issue, but should be rare? What
we have seen seems to be caused by unsafe concurrent HMS access, which seems to
be fixed.
Let me investigate further how to completely fix this issue including the cases
you mentioned.
> Hive should handle AlreadyExists on retries when creating tables/partitions
> ---
>
> Key: HIVE-4897
> URL: https://issues.apache.org/jira/browse/HIVE-4897
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Aihua Xu
> Attachments: HIVE-4897.patch, hive-snippet.log
>
>
> Creating new tables/partitions may fail with an AlreadyExistsException if
> there is an error part way through the creation and the HMS tries again
> without properly cleaning up or checking if this is a retry.
> While partitioning a new table via a script on distributed hive (MetaStore on
> the same machine) there was a long timeout and then:
> {code}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> AlreadyExistsException(message:Partition already exists:Partition( ...
> {code}
> I am assuming this is due to retry. Perhaps already-exists on retry could be
> handled better.
> A similar error occurred while creating a table through Impala, which issued
> a single createTable call that failed with an AlreadyExistsException. See the
> logs related to table tmp_proc_8_d2b7b0f133be455ca95615818b8a5879_7 in the
> attached hive-snippet.log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.02.patch > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: (was: HIVE-13178.02.patch) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6189) Support top level union all statements
[ https://issues.apache.org/jira/browse/HIVE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Wailliam updated HIVE-6189: Description: I've always wondered why union all has to be in subqueries in hive. After looking at it, problems are: - Hive Parser: - Union happens at the wrong place (insert ... select ... union all select ...) is parsed as (insert select) union select. - There are many rewrite rules in the parser to force any query into the a from - insert -select form. No doubt for historical reasons. - Plan generation/semantic analysis assumes top level "TOK_QUERY" and not top level "TOK_UNION". The rewrite rules don't work when we move the "UNION ALL" into the select statements. However, it's not hard to do that in code. was: I've always wondered why union all has to be in subqueries in hive. After looking at it, problems are: - Hive Parser: - Union happens at the wrong place (insert ... select ... union all select ...) is parsed as (insert select) union select. - There are many rewrite rules in the parser to force any query into the a from - insert -select form. No doubt for historical reasons. - Plan generation/semantic analysis assumes top level "TOK_QUERY" and not top level "TOK_UNION". The rewrite rules don't work when we move the "UNION ALL" recursion into the select statements. However, it's not hard to do that in code. > Support top level union all statements > -- > > Key: HIVE-6189 > URL: https://issues.apache.org/jira/browse/HIVE-6189 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: 0.13.0 > > Attachments: HIVE-6189.1.patch, HIVE-6189.2.patch, HIVE-6189.3.patch > > > I've always wondered why union all has to be in subqueries in hive. > After looking at it, problems are: > - Hive Parser: > - Union happens at the wrong place (insert ... select ... union all select > ...) is parsed as (insert select) union select. > - There are many rewrite rules in the parser to force any query into the a > from - insert -select form. No doubt for historical reasons. > - Plan generation/semantic analysis assumes top level "TOK_QUERY" and not top > level "TOK_UNION". > The rewrite rules don't work when we move the "UNION ALL" into the select > statements. However, it's not hard to do that in code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12995) LLAP: Synthetic file ids need collision checks
[
https://issues.apache.org/jira/browse/HIVE-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177704#comment-15177704
]
Hive QA commented on HIVE-12995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790833/HIVE-12995.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 9736 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-index_compact_2.q-vector_grouping_sets.q-lateral_view_cp.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-llap_acid.q-binarysortable_1.q-orc_merge5.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udf_percentile
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_uncompressed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_null_check
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_folder_constants
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.llap.cache.TestIncrementalObjectSizeEstimator.testMetadata
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7151/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7151/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7151/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790833 - PreCommit-HIVE-TRUNK-Build
> LLAP: Synthetic file ids need collision checks
> --
>
> Key: HIVE-12995
> URL: https://issues.apache.org/jira/browse/HIVE-12995
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12995.patch
>
>
> LLAP synthetic file ids do not have any way of checking whether a collision
> occurs other than a data-error.
> Synthetic file-ids have only been used with unit tests so far - but they will
> be needed to add cache mechanisms to non-HDFS filesystems.
> In case of Synthetic file-ids, it is recommended that we track the full-tuple
> (path, mtime, len) in the cache so that a cache-hit for the synthetic file-id
> can be compared against the parameters & only accepted if those match.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177511#comment-15177511 ] Amareshwari Sriramadasu commented on HIVE-5370: --- Test failures say " java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.cli.TestMiniTezCliDriver". Not related to the patch. > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5370) format_number udf should take user specifed format as argument
[
https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177473#comment-15177473
]
Hive QA commented on HIVE-5370:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12790799/HIVE-5370.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 9765 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_having
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_bucket
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_simple
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_windowing_gby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7150/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7150/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7150/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12790799 - PreCommit-HIVE-TRUNK-Build
> format_number udf should take user specifed format as argument
> --
>
> Key: HIVE-5370
> URL: https://issues.apache.org/jira/browse/HIVE-5370
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch,
> HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch
>
>
> Currently, format_number udf formats the number to #,###,###.##, but it
> should also take a user specified format as optional input.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177468#comment-15177468 ] Rajat Khandelwal commented on HIVE-13179: - Taking patch from reviewboard and attaching > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13179.05.patch, HIVE-13179.1.patch, > HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13179: Attachment: HIVE-13179.05.patch > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13179.05.patch, HIVE-13179.1.patch, > HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177466#comment-15177466 ] Rajat Khandelwal commented on HIVE-13179: - After testing on my local, the tests passed. Updating to master and attaching the patch again so that pre-commit job can run again. > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13179.1.patch, HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13188: Attachment: HIVE-13188.03.patch > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13188.02.patch, HIVE-13188.03.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177456#comment-15177456 ] Rajat Khandelwal commented on HIVE-13188: - Taking patch from reviewboard and attaching > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13188.02.patch, HIVE-13188.03.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-11483: Attachment: HIVE-11483.04.patch > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch, > HIVE-11483.03.patch, HIVE-11483.04.patch > > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177453#comment-15177453 ] Rajat Khandelwal commented on HIVE-11483: - Taking patch from reviewboard and attaching > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch, > HIVE-11483.03.patch, HIVE-11483.04.patch > > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
