> On July 19, 2014, 10:03 p.m., Jarek Cecho wrote: > > execution/mapreduce/src/main/java/org/apache/sqoop/job/io/SqoopWritable.java, > > line 27 > > <https://reviews.apache.org/r/16812/diff/5/?file=604300#file604300line27> > > > > Didn't we wanted to use the Configurable interface to skip the string > > encoding here? > > Abraham Elmahrek wrote: > Yessir, but we've decided to do this in a follow up Jira when Hadoop 2.5 > is released. > > Jarek Cecho wrote: > Make sense, could you then create the follow up JIRA?
https://issues.apache.org/jira/browse/SQOOP-1349 - Abraham ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16812/#review48182 ----------------------------------------------------------- On July 22, 2014, 7:50 a.m., Abraham Elmahrek wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/16812/ > ----------------------------------------------------------- > > (Updated July 22, 2014, 7:50 a.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-777 > https://issues.apache.org/jira/browse/SQOOP-777 > > > Repository: sqoop-sqoop2 > > > Description > ------- > > Previous review from hari is in first iteration of this review only. > > commit 1c5122611fbd4a46a629421f7c55746cfc14f136 > Author: Hari Shreedharan <[email protected]> > Date: Fri Jan 10 15:07:08 2014 -0800 > > SQOOP-777. Sqoop2: Pluggable Intermediate Data Format > > Implemented a pluggable intermediate data format that decouples the > internal representation of the data from the connector and the output > formats. Connectors can choose to implement and support a format that is more > efficient for them. Also separated the SqoopWritable so th > > I ran a full build - all tests including integration tests pass. I have > not added any new tests, yet. I will add unit tests for the new classes. > Also, I have not tried running this on an actual cluster - so things may be > broken. I'd like some initial feedback based on the c > > I also implemented escaping of characters. There is some work remaining > to support binary format, but it is mostly integration, the basic > implementation is in place. > > :100644 100644 db11b5b... 9bfa07d... M common/pom.xml > :100644 100644 3e1adc7... f971240... M > common/src/main/java/org/apache/sqoop/etl/io/DataReader.java > :100644 100644 d81364e... e547875... M > common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java > :100644 100644 8b630b2... 30c26a3... M > common/src/main/java/org/apache/sqoop/schema/type/Column.java > :100644 100644 e0da80f... 9c70db9... M > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java > :100644 100644 ef39cdc... 075890f... M > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java > :100644 100644 96818ba... 795ffdb... M > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java > :000000 100644 0000000... 6c33423... A > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/util/InitializationUtils.java > :100644 100644 aa1c4ff... 7220018... M > connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java > :100644 100644 a7ed6ba... 28399f2... M > connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java > :100644 100644 4056e14... f54837d... M connector/connector-sdk/pom.xml > :000000 100644 0000000... d4874f2... A > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/CSVIntermediateDataFormat.java > :000000 100644 0000000... 63e14d2... A > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/IntermediateDataFormat.java > :000000 100644 0000000... 6e5479f... A > connector/connector-sdk/src/test/java/org/apache/sqoop/connector/CSVIntermediateDataFormatTest.java > :100644 100644 e052584... 3b2ef94... M > core/src/main/java/org/apache/sqoop/framework/JobManager.java > :100644 100644 53d0039... 9f5c47d... M > core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java > :100644 100644 f9a2a0e... 9754afd... M execution/mapreduce/pom.xml > :100644 100644 5c0a027... 7680e33... M > execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java > :100644 100644 7fd9a01... 3604898... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java > :100644 100644 1978ec6... 099cdd3... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportExtractor.java > :100644 100644 a07c511... ee6bf39... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java > :100644 100644 4621942... 7b799ca... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java > :000000 100644 0000000... 71ae980... A > execution/mapreduce/src/main/java/org/apache/sqoop/job/io/SqoopWritable.java > :100644 100644 356ae8a... b495cc9... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java > :100644 100644 92de37e... 8164ffe... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java > :100644 100644 90de6ef... b3503da... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java > :100644 100644 7dedee9... 16e59d8... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java > :100644 100644 98a2c51... 5aaceb3... M > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java > :100644 100644 e21f15b... 4e9e6ea... M > execution/mapreduce/src/test/java/org/apache/sqoop/job/JobUtils.java > :100644 100644 b7079dd... ddda423... M > execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java > :100644 100644 f849aae... 2968411... M > execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java > :100644 100644 7b264c6... 954990f... M > execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java > :000000 100644 0000000... aea7de3... A > execution/mapreduce/src/test/java/org/apache/sqoop/job/io/SqoopWritableTest.java > :100644 100644 bee8ab7... 663dfb5... M > execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java > :100644 100644 1e2f005... a722c74... M pom.xml > :100644 100644 0b240e8... 43f17d4... M spi/pom.xml > :100644 100644 2becc56... 298e8f5... M > spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java > :100644 100644 6fc485b... 8d1a4e8... M > submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java > > > Diffs > ----- > > common/pom.xml db11b5b > common/src/main/java/org/apache/sqoop/etl/io/DataReader.java 3e1adc7 > common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java d81364e > common/src/main/java/org/apache/sqoop/schema/type/Column.java 8b630b2 > > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java > e0da80f > > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java > 2b1a0ad > > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java > ef39cdc > > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java > 96818ba > > connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java > d4c4565 > > connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java > a7ed6ba > > connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportInitializer.java > a33fa36 > connector/connector-sdk/pom.xml 4056e14 > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java > PRE-CREATION > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormat.java > PRE-CREATION > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormatError.java > PRE-CREATION > > connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormatTest.java > PRE-CREATION > core/src/main/java/org/apache/sqoop/framework/JobManager.java e052584 > core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java > a138db5 > execution/mapreduce/pom.xml f9a2a0e > > execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java > 5c0a027 > execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java > 7fd9a01 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportExtractor.java > 1978ec6 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java > a07c511 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java > 4621942 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/io/SqoopWritable.java > PRE-CREATION > > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java > 356ae8a > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java > 92de37e > > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java > 90de6ef > > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java > 7dedee9 > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java > 98a2c51 > > execution/mapreduce/src/test/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngineTest.java > 39d1b53 > execution/mapreduce/src/test/java/org/apache/sqoop/job/JobUtils.java > e21f15b > execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java > b7079dd > execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java > f849aae > execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java > 7b264c6 > > execution/mapreduce/src/test/java/org/apache/sqoop/job/io/SqoopWritableTest.java > PRE-CREATION > > execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java > bee8ab7 > pom.xml 1e2f005 > spi/pom.xml 0b240e8 > spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java > 2becc56 > > submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java > bfc28ef > > Diff: https://reviews.apache.org/r/16812/diff/ > > > Testing > ------- > > mvn test > > > Thanks, > > Abraham Elmahrek > >
