[jira] [Created] (HIVE-23552) TestMiniLlapCliDriver.testCliDriver[merge_test_dummy_operator] is unstable

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23552:
---

 Summary: 
TestMiniLlapCliDriver.testCliDriver[merge_test_dummy_operator] is unstable
 Key: HIVE-23552
 URL: https://issues.apache.org/jira/browse/HIVE-23552
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

[Adesh Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72462/
---

(Updated May 27, 2020, 5:02 a.m.)


Review request for hive and Sankar Hariappan.


Repository: hive-git


Description
---

The fix converts partition keys to lowercase present in hdfs directory, but 
store the hdfs directory as is for partition path.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 92ae8c28e8 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
 7821f40a82 
  ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION 
  ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION 
  ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 5287f47e21 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 6f4400a8ef 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 f4e109d1b0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 92d10cd0e1 


Diff: https://reviews.apache.org/r/72462/diff/6/

Changes: https://reviews.apache.org/r/72462/diff/5-6/


Testing
---


Thanks,

Adesh Rao

来自wldd的邮件

[wldd]

--

Best，
wldd

Re: Review Request 72521: HIVE-23487: Optimise PartitionManagementTask

[Rajesh Balamohan]

> On May 26, 2020, 11:48 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
> > Lines 87-92 (original), 87-97 (patched)
> > 
> >
> > I don't follow how this is an improvement. new Configuration() which I 
> > assume is expensive call is still there.
> > If anything, it appears that this change would make perf worse since 
> > earlier new Conf() was guarded by if (msc == null) so would have happened 
> > only once, but now will happen everytime.
> > Can you explain how this change is more performant?
> 
> Rajesh Balamohan wrote:
> This is because, it was creating this for every table.
>  
> With the fix in "PartitionManagementTask::run", patch constructs this 
> conf only once and reuses it across tables. (i.e in Configuration msckConf = 
> Msck.getMsckConf(conf);)
> 
> Ashutosh Chauhan wrote:
> But how? Msck.getMsckConf() constructs new Configuration() every time. 
> Also, it is invoked everytime for execute(). So, I still don't see it.
> 
> Ashutosh Chauhan wrote:
> I meant run() invokes getMsckConf() for every table which in turns does 
> new Configuration()

Issue is w.r.t to construction of "Configuration" in "msck.init" for every 
table.

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L150

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L158

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L223

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L88


With the patch, we set the "EXPRESSION_PROXY_CLASS" just once in 
"PartitionManagementTask::run()" and pass it to MsckThread. This gets rid "new 
Conf" in "Msck::init".


- Rajesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72521/#review220874
---


On May 18, 2020, 12:53 a.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72521/
> ---
> 
> (Updated May 18, 2020, 12:53 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and prasanthj.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Msck.init for every table takes more CPU time than the actual table repair. 
> This was observed on a system which had lots of DB and tables.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckOperation.java 
> c05d699bd8 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  7821f40a82 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  8be31128a1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
>  e4488f4709 
> 
> 
> Diff: https://reviews.apache.org/r/72521/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>

[jira] [Created] (HIVE-23551) Acid: Update queries should purge dir cache entry in AcidUtils

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-23551:
---

 Summary: Acid: Update queries should purge dir cache entry in 
AcidUtils
 Key: HIVE-23551
 URL: https://issues.apache.org/jira/browse/HIVE-23551
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Rajesh Balamohan


Update statements create delta folders at the end of the execution. When 
{{insert overwrite}} followed by {{update}} is executed, it does not get any 
open txns and ends up caching the {{base}} folder. However, the delta folder 
which gets created at the end of the statement never makes it to the cache. 
This creates wrong results.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72521: HIVE-23487: Optimise PartitionManagementTask

[Ashutosh Chauhan]

> On May 26, 2020, 11:48 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
> > Lines 87-92 (original), 87-97 (patched)
> > 
> >
> > I don't follow how this is an improvement. new Configuration() which I 
> > assume is expensive call is still there.
> > If anything, it appears that this change would make perf worse since 
> > earlier new Conf() was guarded by if (msc == null) so would have happened 
> > only once, but now will happen everytime.
> > Can you explain how this change is more performant?
> 
> Rajesh Balamohan wrote:
> This is because, it was creating this for every table.
>  
> With the fix in "PartitionManagementTask::run", patch constructs this 
> conf only once and reuses it across tables. (i.e in Configuration msckConf = 
> Msck.getMsckConf(conf);)
> 
> Ashutosh Chauhan wrote:
> But how? Msck.getMsckConf() constructs new Configuration() every time. 
> Also, it is invoked everytime for execute(). So, I still don't see it.

I meant run() invokes getMsckConf() for every table which in turns does new 
Configuration()


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72521/#review220874
---


On May 18, 2020, 12:53 a.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72521/
> ---
> 
> (Updated May 18, 2020, 12:53 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and prasanthj.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Msck.init for every table takes more CPU time than the actual table repair. 
> This was observed on a system which had lots of DB and tables.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckOperation.java 
> c05d699bd8 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  7821f40a82 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  8be31128a1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
>  e4488f4709 
> 
> 
> Diff: https://reviews.apache.org/r/72521/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>

Re: Review Request 72521: HIVE-23487: Optimise PartitionManagementTask

[Ashutosh Chauhan]

> On May 26, 2020, 11:48 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
> > Lines 87-92 (original), 87-97 (patched)
> > 
> >
> > I don't follow how this is an improvement. new Configuration() which I 
> > assume is expensive call is still there.
> > If anything, it appears that this change would make perf worse since 
> > earlier new Conf() was guarded by if (msc == null) so would have happened 
> > only once, but now will happen everytime.
> > Can you explain how this change is more performant?
> 
> Rajesh Balamohan wrote:
> This is because, it was creating this for every table.
>  
> With the fix in "PartitionManagementTask::run", patch constructs this 
> conf only once and reuses it across tables. (i.e in Configuration msckConf = 
> Msck.getMsckConf(conf);)

But how? Msck.getMsckConf() constructs new Configuration() every time. Also, it 
is invoked everytime for execute(). So, I still don't see it.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72521/#review220874
---


On May 18, 2020, 12:53 a.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72521/
> ---
> 
> (Updated May 18, 2020, 12:53 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and prasanthj.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Msck.init for every table takes more CPU time than the actual table repair. 
> This was observed on a system which had lots of DB and tables.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckOperation.java 
> c05d699bd8 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  7821f40a82 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  8be31128a1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
>  e4488f4709 
> 
> 
> Diff: https://reviews.apache.org/r/72521/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>

Re: Review Request 72521: HIVE-23487: Optimise PartitionManagementTask

[Rajesh Balamohan]

> On May 26, 2020, 11:48 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
> > Lines 87-92 (original), 87-97 (patched)
> > 
> >
> > I don't follow how this is an improvement. new Configuration() which I 
> > assume is expensive call is still there.
> > If anything, it appears that this change would make perf worse since 
> > earlier new Conf() was guarded by if (msc == null) so would have happened 
> > only once, but now will happen everytime.
> > Can you explain how this change is more performant?

This is because, it was creating this for every table.
 
With the fix in "PartitionManagementTask::run", patch constructs this conf only 
once and reuses it across tables. (i.e in Configuration msckConf = 
Msck.getMsckConf(conf);)


- Rajesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72521/#review220874
---


On May 18, 2020, 12:53 a.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72521/
> ---
> 
> (Updated May 18, 2020, 12:53 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and prasanthj.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Msck.init for every table takes more CPU time than the actual table repair. 
> This was observed on a system which had lots of DB and tables.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckOperation.java 
> c05d699bd8 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  7821f40a82 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  8be31128a1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
>  e4488f4709 
> 
> 
> Diff: https://reviews.apache.org/r/72521/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>

Re: Review Request 72521: HIVE-23487: Optimise PartitionManagementTask

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72521/#review220874
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
Lines 87-92 (original), 87-97 (patched)


I don't follow how this is an improvement. new Configuration() which I 
assume is expensive call is still there.
If anything, it appears that this change would make perf worse since 
earlier new Conf() was guarded by if (msc == null) so would have happened only 
once, but now will happen everytime.
Can you explain how this change is more performant?


- Ashutosh Chauhan


On May 18, 2020, 12:53 a.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72521/
> ---
> 
> (Updated May 18, 2020, 12:53 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and prasanthj.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Msck.init for every table takes more CPU time than the actual table repair. 
> This was observed on a system which had lots of DB and tables.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckOperation.java 
> c05d699bd8 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  7821f40a82 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckDropPartitionsInBatches.java
>  8be31128a1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
>  e4488f4709 
> 
> 
> Diff: https://reviews.apache.org/r/72521/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>

Re: Review Request 72526: HIVE-23493

[Krisztian Kasa]

> On May 18, 2020, 11:45 p.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveCardinalityPreservingJoinOptimization.java
> > Lines 325 (patched)
> > 
> >
> > Do you need to override this method?
> 
> Krisztian Kasa wrote:
> I only add collecting the `HiveTableScan` instances in order to copy them 
> when the Join operators created.

Removed the override and create the new HiveTableScan from RelOptHiveTable.


- Krisztian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72526/#review220811
---


On May 26, 2020, 6:29 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72526/
> ---
> 
> (Updated May 26, 2020, 6:29 p.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-23493
> https://issues.apache.org/jira/browse/HIVE-23493
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Rewrite plan to join back tables with many projected columns joined multiple 
> times
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8094d28f21 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveCardinalityPreservingJoinOptimization.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveCardinalityPreservingJoinRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
>  19ce3ea223 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 377e8280e5 
>   ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query11.q.out 
> 0136ee4bb5 
>   ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query4.q.out 
> 987a0f348e 
>   ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query74.q.out 
> 289e5d2569 
> 
> 
> Diff: https://reviews.apache.org/r/72526/diff/4/
> 
> 
> Testing
> ---
> 
> mvn test -Dtest.output.overwrite -DskipSparkTests 
> -Dtest=TestMiniLlapLocalCliDriver 
> -Dqfile=cardinality_preserving_join_opt.q,cardinality_preserving_join_opt_q4.q,cardinality_preserving_join_opt_q11.q,cardinality_preserving_join_opt_q74.q
>  -pl itests/qtest -Pitests
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>

Re: Review Request 72526: HIVE-23493

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72526/
---

(Updated May 26, 2020, 6:29 p.m.)


Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-23493
https://issues.apache.org/jira/browse/HIVE-23493


Repository: hive-git


Description
---

Rewrite plan to join back tables with many projected columns joined multiple 
times


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8094d28f21 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveCardinalityPreservingJoinOptimization.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveCardinalityPreservingJoinRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 19ce3ea223 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 377e8280e5 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query11.q.out 
0136ee4bb5 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query4.q.out 
987a0f348e 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query74.q.out 
289e5d2569 


Diff: https://reviews.apache.org/r/72526/diff/4/

Changes: https://reviews.apache.org/r/72526/diff/3-4/


Testing
---

mvn test -Dtest.output.overwrite -DskipSparkTests 
-Dtest=TestMiniLlapLocalCliDriver 
-Dqfile=cardinality_preserving_join_opt.q,cardinality_preserving_join_opt_q4.q,cardinality_preserving_join_opt_q11.q,cardinality_preserving_join_opt_q74.q
 -pl itests/qtest -Pitests


Thanks,

Krisztian Kasa

[jira] [Created] (HIVE-23550) GetSplits does not retries queries for CacliteSemanticException

[Adesh Kumar Rao (Jira)]

Adesh Kumar Rao created HIVE-23550:
--

 Summary: GetSplits does not retries queries for 
CacliteSemanticException
 Key: HIVE-23550
 URL: https://issues.apache.org/jira/browse/HIVE-23550
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 3.1.0
Reporter: Adesh Kumar Rao
Assignee: Adesh Kumar Rao
 Fix For: 4.0.0


Reproducible case:
{noformat}
create table t1 (c1 int, c2 int, c3 int);

select get_splits("select c2, count(distinct c3) from t1 group by c2 having 
count(distinct c3) > 1",0);{noformat}
 

Error:
{noformat}
Error: java.io.IOException: 
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Distinct 
without an aggregation. (state=,code=0)
{noformat}
This happens because calcite does not understand the query "select c2, 
count(distinct c3) from t1 group by c2 having count(distinct c3) > 1" and 
throws calciteSemanticException

 

If this query is run directly via beeline, hiveserver2 catches this exception 
and re-analyzes the query by turning off the cbo.

 

This retrying mechanism is missing in GetSplits UDF.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

[Adesh Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72462/#review220868
---




ql/src/test/results/clientnegative/msck_repair_6.q.out
Lines 30 (patched)


Most of the other ddl commands throw the same error code when finished. 

Do we need to create a new MetastoreException subclass, and then use it to 
catch and throw new error code?


- Adesh Rao


On May 18, 2020, 10:43 a.m., Adesh Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72462/
> ---
> 
> (Updated May 18, 2020, 10:43 a.m.)
> 
> 
> Review request for hive and Sankar Hariappan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The fix converts partition keys to lowercase present in hdfs directory, but 
> store the hdfs directory as is for partition path.
> 
> 
> Diffs
> -
> 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  7821f40a82 
>   ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION 
>   ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
>  5287f47e21 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
>  6f4400a8ef 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  92d10cd0e1 
> 
> 
> Diff: https://reviews.apache.org/r/72462/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adesh Rao
> 
>

[jira] [Created] (HIVE-23549) TestJdbcWithMiniLlapVectorArrow is unstable

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23549:
---

 Summary: TestJdbcWithMiniLlapVectorArrow is unstable
 Key: HIVE-23549
 URL: https://issues.apache.org/jira/browse/HIVE-23549
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

[Sankar Hariappan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72462/#review220823
---




ql/src/test/results/clientnegative/msck_repair_6.q.out
Lines 30 (patched)


Can we return more specific error code instead of 1?



ql/src/test/results/clientpositive/msck_repair_4.q.out
Lines 34 (patched)


Why the show partitions output is listed in the output file? We need to 
have it to validate this scenario.



standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
Lines 184 (patched)


Do we need this override as it just calls super method?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
Lines 415 (patched)


Shall log the table name as well.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
Line 405 (original), 419 (patched)


Should we also validate the case where old path(lower case) of partition is 
removed but added same path with different case? In this case, partition exist 
in both partitionsNotOnFs and partitionsNotInMs lists. Not sure, if it can 
cause any failures.


- Sankar Hariappan


On May 18, 2020, 10:43 a.m., Adesh Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72462/
> ---
> 
> (Updated May 18, 2020, 10:43 a.m.)
> 
> 
> Review request for hive and Sankar Hariappan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The fix converts partition keys to lowercase present in hdfs directory, but 
> store the hdfs directory as is for partition path.
> 
> 
> Diffs
> -
> 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/TestMsckCreatePartitionsInBatches.java
>  7821f40a82 
>   ql/src/test/queries/clientnegative/msck_repair_5.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/msck_repair_6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION 
>   ql/src/test/results/clientnegative/msck_repair_5.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/msck_repair_6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
>  5287f47e21 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
>  6f4400a8ef 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  92d10cd0e1 
> 
> 
> Diff: https://reviews.apache.org/r/72462/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adesh Rao
> 
>