Review Request 47185: HIVE-13716

2016-05-10 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47185/
---

Review request for hive and Rui Li.


Bugs: HIVE-13716
https://issues.apache.org/jira/browse/HIVE-13716


Repository: hive-git


Description
---

Parallelize setting file permissions for moveFile()


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c4d3bfb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java c2c6c65 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f4a9772 
  shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java e931156 

Diff: https://reviews.apache.org/r/47185/diff/


Testing
---

Existing regression tests.


Thanks,

Ashutosh Chauhan

Re: Review Request 47185: HIVE-13716

2016-05-11 Thread Rui Li 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47185/#review132610
---




ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 4161)


Here we choose non-recursive because location is an empty dir, right?



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2679)


Are we sure destPath is always a file and not a dir?



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2851)


I'm not sure if this is needed here. I set the session state in copyFiles 
because we need to call needToCopy in each thread, which requires the session 
state to be set.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2854)


Is it possible that the rename fails but no exception is thrown? If so, we 
will eventaully return true, which is not correct.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2871)


I guess this is redundant?


- Rui Li


On May 10, 2016, 3:41 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47185/
> ---
> 
> (Updated May 10, 2016, 3:41 p.m.)
> 
> 
> Review request for hive and Rui Li.
> 
> 
> Bugs: HIVE-13716
> https://issues.apache.org/jira/browse/HIVE-13716
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Parallelize setting file permissions for moveFile()
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c4d3bfb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java c2c6c65 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f4a9772 
>   shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java e931156 
> 
> Diff: https://reviews.apache.org/r/47185/diff/
> 
> 
> Testing
> ---
> 
> Existing regression tests.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 47185: HIVE-13716

2016-05-11 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47185/#review132668
---




ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 4161)


yes.. we just deleted location in previous step, so it will be empty.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2679)


yes.. name for destPath is chosen based on the assumption its a file



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2851)


It may not be needed as of now. But I think its a good practice to set 
thread local variable for each thread.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2854)


correct.. i will update the logic.



ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2871)


I think this is useful to force shut current existing threads.


- Ashutosh Chauhan


On May 10, 2016, 3:41 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47185/
> ---
> 
> (Updated May 10, 2016, 3:41 p.m.)
> 
> 
> Review request for hive and Rui Li.
> 
> 
> Bugs: HIVE-13716
> https://issues.apache.org/jira/browse/HIVE-13716
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Parallelize setting file permissions for moveFile()
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c4d3bfb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java c2c6c65 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f4a9772 
>   shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java e931156 
> 
> Diff: https://reviews.apache.org/r/47185/diff/
> 
> 
> Testing
> ---
> 
> Existing regression tests.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 47185: HIVE-13716

2016-05-11 Thread Rui Li 


> On May 11, 2016, 4:14 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 2881
> > 
> >
> > I think this is useful to force shut current existing threads.

Oh I just thought it was shutdown. Makes sense.


- Rui


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47185/#review132668
---


On May 10, 2016, 3:41 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47185/
> ---
> 
> (Updated May 10, 2016, 3:41 p.m.)
> 
> 
> Review request for hive and Rui Li.
> 
> 
> Bugs: HIVE-13716
> https://issues.apache.org/jira/browse/HIVE-13716
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Parallelize setting file permissions for moveFile()
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c4d3bfb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java c2c6c65 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f4a9772 
>   shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java e931156 
> 
> Diff: https://reviews.apache.org/r/47185/diff/
> 
> 
> Testing
> ---
> 
> Existing regression tests.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 47185: HIVE-13716

2016-05-11 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47185/
---

(Updated May 11, 2016, 4:26 p.m.)


Review request for hive and Rui Li.


Changes
---

Addressed review comments.


Bugs: HIVE-13716
https://issues.apache.org/jira/browse/HIVE-13716


Repository: hive-git


Description
---

Parallelize setting file permissions for moveFile()


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 0204fcd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java bdda89a 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 981b961 
  shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java e931156 

Diff: https://reviews.apache.org/r/47185/diff/


Testing
---

Existing regression tests.


Thanks,

Ashutosh Chauhan

Re: Review Request 47185: HIVE-13716

2016-05-11 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47185/
---

(Updated May 11, 2016, 4:29 p.m.)


Review request for hive and Rui Li.


Changes
---

correct patch.


Bugs: HIVE-13716
https://issues.apache.org/jira/browse/HIVE-13716


Repository: hive-git


Description
---

Parallelize setting file permissions for moveFile()


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 0204fcd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java bdda89a 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 981b961 
  shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java e931156 

Diff: https://reviews.apache.org/r/47185/diff/


Testing
---

Existing regression tests.


Thanks,

Ashutosh Chauhan

Re: Review Request 47185: HIVE-13716

2016-05-11 Thread Rui Li 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47185/#review132684
---


Ship it!




Ship It!

- Rui Li


On May 11, 2016, 4:29 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47185/
> ---
> 
> (Updated May 11, 2016, 4:29 p.m.)
> 
> 
> Review request for hive and Rui Li.
> 
> 
> Bugs: HIVE-13716
> https://issues.apache.org/jira/browse/HIVE-13716
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Parallelize setting file permissions for moveFile()
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 0204fcd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java bdda89a 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 981b961 
>   shims/common/src/main/java/org/apache/hadoop/hive/io/HdfsUtils.java e931156 
> 
> Diff: https://reviews.apache.org/r/47185/diff/
> 
> 
> Testing
> ---
> 
> Existing regression tests.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>