Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

2018-10-16 Thread daniel voros 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/#review209630
---


Ship it!




Thanks Fero, for taking care of this. Ship it!

- daniel voros


On Oct. 16, 2018, 9:37 a.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68687/
> ---
> 
> (Updated Oct. 16, 2018, 9:37 a.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3381
> https://issues.apache.org/jira/browse/SQOOP-3381
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This change upgrades our parquet library to the newest version and a whole 
> lot of libraries to newer versions with it.
> 
> As we will need to register a data supplier in the fix for parquet decimal 
> support (SQOOP-3382), we will need a version that contains PARQUET-243. We 
> need to upgrade the Parquet library to a version that contains this fix and 
> is compatible with Hadoop 3.0.
> 
> A few things to note:
> - hadoop's version is still 2.8.0
> - hive is upgraded to 2.1.1
> - the rest of the dependency changes are required for the hive version bump.
> 
> There is are a few changes in the codebase, but of course no new 
> functionality at all:
> - in the TestParquetImport class, the new implementation returns a Utf8 
> object for Strings written out.
> - Added the security policy and related code changes from the patch for 
> SQOOP-3305 (upgrade hadoop) written by Daniel Voros.
> - modified HiveMiniCluster config so it won't try to start a web ui (it's 
> unnecessary during tests anyway)
> 
> 
> Diffs
> -
> 
>   build.gradle fc7fc0c4 
>   gradle.properties 0d30378d 
>   gradle/sqoop-package.gradle 1a8d994d 
>   ivy.xml 670cb32d 
>   ivy/libraries.properties 8f3dab2b 
>   src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
>   src/java/org/apache/sqoop/hive/HiveImport.java 48800366 
>   src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
>  2180cc20 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
>  90b910a3 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
>  66ebc5b8 
>   src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
>   src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
>   src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
>   src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
>   src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 
>   src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 
>   testdata/hcatalog/conf/hive-site.xml 8a84a5d3 
> 
> 
> Diff: https://reviews.apache.org/r/68687/diff/5/
> 
> 
> Testing
> ---
> 
> Ant unit and 3rd party tests were successful.
> gradlew test and thirdpartytest were succesful as well.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

2018-10-16 Thread Fero Szabo via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Oct. 16, 2018, 9:37 a.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version and a whole lot 
of libraries to newer versions with it.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

A few things to note:
- hadoop's version is still 2.8.0
- hive is upgraded to 2.1.1
- the rest of the dependency changes are required for the hive version bump.

There is are a few changes in the codebase, but of course no new functionality 
at all:
- in the TestParquetImport class, the new implementation returns a Utf8 object 
for Strings written out.
- Added the security policy and related code changes from the patch for 
SQOOP-3305 (upgrade hadoop) written by Daniel Voros.
- modified HiveMiniCluster config so it won't try to start a web ui (it's 
unnecessary during tests anyway)


Diffs (updated)
-

  build.gradle fc7fc0c4 
  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  src/java/org/apache/sqoop/hive/HiveImport.java 48800366 
  src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 
  testdata/hcatalog/conf/hive-site.xml 8a84a5d3 


Diff: https://reviews.apache.org/r/68687/diff/5/

Changes: https://reviews.apache.org/r/68687/diff/4-5/


Testing
---

Ant unit and 3rd party tests were successful.
gradlew test and thirdpartytest were succesful as well.


Thanks,

Fero Szabo

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

2018-10-16 Thread Szabolcs Vasas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/#review209602
---


Fix it, then Ship it!




Hi Feró,

Thank you for submitting this improvement!
The unit and third party tests are successful on my side as well, I have found 
only one small thing, please fix that.

Szabolcs


src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java
Lines 30 (patched)


Unused import.


- Szabolcs Vasas


On Oct. 15, 2018, 2:13 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68687/
> ---
> 
> (Updated Oct. 15, 2018, 2:13 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3381
> https://issues.apache.org/jira/browse/SQOOP-3381
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> This change upgrades our parquet library to the newest version and a whole 
> lot of libraries to newer versions with it.
> 
> As we will need to register a data supplier in the fix for parquet decimal 
> support (SQOOP-3382), we will need a version that contains PARQUET-243. We 
> need to upgrade the Parquet library to a version that contains this fix and 
> is compatible with Hadoop 3.0.
> 
> A few things to note:
> - hadoop's version is still 2.8.0
> - hive is upgraded to 2.1.1
> - the rest of the dependency changes are required for the hive version bump.
> 
> There is are a few changes in the codebase, but of course no new 
> functionality at all:
> - in the TestParquetImport class, the new implementation returns a Utf8 
> object for Strings written out.
> - Added the security policy and related code changes from the patch for 
> SQOOP-3305 (upgrade hadoop) written by Daniel Voros.
> - modified HiveMiniCluster config so it won't try to start a web ui (it's 
> unnecessary during tests anyway)
> 
> 
> Diffs
> -
> 
>   build.gradle fc7fc0c4 
>   gradle.properties 0d30378d 
>   gradle/sqoop-package.gradle 1a8d994d 
>   ivy.xml 670cb32d 
>   ivy/libraries.properties 8f3dab2b 
>   src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
>   src/java/org/apache/sqoop/hive/HiveImport.java 48800366 
>   src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
>  2180cc20 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
>  90b910a3 
>   
> src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
>  66ebc5b8 
>   src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
>   src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
>   src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
>   src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
>   src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 
>   src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 
>   testdata/hcatalog/conf/hive-site.xml 8a84a5d3 
> 
> 
> Diff: https://reviews.apache.org/r/68687/diff/4/
> 
> 
> Testing
> ---
> 
> Ant unit and 3rd party tests were successful.
> gradlew test and thirdpartytest were succesful as well.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

2018-10-15 Thread Fero Szabo via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Oct. 15, 2018, 2:13 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description (updated)
---

This change upgrades our parquet library to the newest version and a whole lot 
of libraries to newer versions with it.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

A few things to note:
- hadoop's version is still 2.8.0
- hive is upgraded to 2.1.1
- the rest of the dependency changes are required for the hive version bump.

There is are a few changes in the codebase, but of course no new functionality 
at all:
- in the TestParquetImport class, the new implementation returns a Utf8 object 
for Strings written out.
- Added the security policy and related code changes from the patch for 
SQOOP-3305 (upgrade hadoop) written by Daniel Voros.
- modified HiveMiniCluster config so it won't try to start a web ui (it's 
unnecessary during tests anyway)


Diffs (updated)
-

  build.gradle fc7fc0c4 
  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  src/java/org/apache/sqoop/hive/HiveImport.java 48800366 
  src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 
  testdata/hcatalog/conf/hive-site.xml 8a84a5d3 


Diff: https://reviews.apache.org/r/68687/diff/4/

Changes: https://reviews.apache.org/r/68687/diff/3-4/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test and thirdpartytest were succesful as well.


Thanks,

Fero Szabo

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

2018-09-12 Thread Fero Szabo via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Sept. 12, 2018, 2:53 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/3/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test was successful
gradlew thirdPartyTest was successful as well. (it had 1 failed test at first 
run: HBaseImportTest#testAppendWithTimestampSucceeds)


Thanks,

Fero Szabo

Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library

2018-09-12 Thread Fero Szabo via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

(Updated Sept. 12, 2018, 2:15 p.m.)


Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/3/


Testing (updated)
---

Ant unit and 3rd party tests were successful.
gradlew test was successful
gradlew thirdPartyTest had 1 failed test: 
HBaseImportTest#testAppendWithTimestampSucceeds

This seems to be unrealted.


Thanks,

Fero Szabo

Review Request 68687: SQOOP-3381 Upgrade the Parquet library

2018-09-11 Thread Fero Szabo via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68687/
---

Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas.


Bugs: SQOOP-3381
https://issues.apache.org/jira/browse/SQOOP-3381


Repository: sqoop-trunk


Description
---

This change upgrades our parquet library to the newest version.

As we will need to register a data supplier in the fix for parquet decimal 
support (SQOOP-3382), we will need a version that contains PARQUET-243. We need 
to upgrade the Parquet library to a version that contains this fix and is 
compatible with Hadoop 3.0.

The only difference is found in the TestParquetImport class, the new 
implementation returns a Utf8 object for Strings written out.


Diffs
-

  gradle.properties 0d30378d 
  gradle/sqoop-package.gradle 1a8d994d 
  ivy.xml 670cb32d 
  ivy/libraries.properties 8f3dab2b 
  src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java
 2180cc20 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java
 90b910a3 
  
src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java
 66ebc5b8 
  src/test/org/apache/sqoop/TestParquetExport.java be1d8164 
  src/test/org/apache/sqoop/TestParquetImport.java 2810e318 
  src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 
  src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 
  src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 


Diff: https://reviews.apache.org/r/68687/diff/1/


Testing
---

Ant unit and 3rd party tests were successful.


Thanks,

Fero Szabo