> On May 26, 2017, 9:54 a.m., Szabolcs Vasas wrote:
> > Hi Eric!
> >
> > Thank you for improving the patch however it turned out that there is one
> > more edge case which needs to be covered.
> > I have realized that there are 2 options for specifying the representation
> > of null: org.apache.sqoop.SqoopOptions#getInNullStringValue (this is used
> > for columns with String data type) and
> > org.apache.sqoop.SqoopOptions#getInNullNonStringValue (this is used for all
> > the other data types). At this point your patch uses the first option only
> > thus the first lines of the __loadFromFields0 method of the generated class
> > look like this:
> >
> > if (__it.hasNext()) {
> > __cur_str = __it.next();
> > } else {
> > __cur_str = "NNUULL";
> > }
> > if (__cur_str.equals("null") || __cur_str.length() == 0) { this.ID =
> > null; } else {
> > this.ID = Integer.valueOf(__cur_str);
> > }
> >
> > Since the ID column is of type Integer Sqoop should use
> > org.apache.sqoop.SqoopOptions#getInNullNonStringValue to generate the first
> > if statement.
> >
> > Sorry for the late notice but I have just realized that there is this other
> > option too.
> >
> > Apart from this I have run all the unit tests and it seems that couple of
> > test cases are failing in org.apache.sqoop.TestExportUsingProcedure.
> > TestExportUsingProcedure is a subclass of TestExport thus inherits the new
> > test cases you added but they fail for some reason in that class. Can you
> > please take a look at it too?
> >
> >
> > Thank you for your efforts!
> >
> > Regards,
> > Szabolcs
Hi Szabolcs,
Thanks for your feedback and suggestion. I have made modifications accordingly
and fixed the failed test case.
Thanks again.
- Eric
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58466/#review176188
-----------------------------------------------------------
On May 30, 2017, 9:36 a.m., Eric Lin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58466/
> -----------------------------------------------------------
>
> (Updated May 30, 2017, 9:36 a.m.)
>
>
> Review request for Sqoop, Attila Szabo and Szabolcs Vasas.
>
>
> Bugs: SQOOP-3158
> https://issues.apache.org/jira/browse/SQOOP-3158
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> I have table in MySQL with 2 columns until yesterday. The columns are id and
> name.
> 1,Raj
> 2,Jack
>
> I have imported this data into HDFS yesterday itself as a file. Today we
> added a new column to the table in MySQL called salary. The table looks like
> below.
>
> 1,Raj
> 2,Jack
> 3,Jill,2000
> 4,Nick,3000
>
> Now I have done Incremental import on this table as a file.
>
> Part-m-00000 file contains
> 1,Raj
> 2,Jack
>
> Part-m-00001 file contains
> 3,Jill,2000
> 4,Nick,3000
>
> Now I created a new table in MySQL with same schema as Original MySQL table
> with columns id name and salary.
>
> Sqoop export will fail with below error:
>
> java.lang.RuntimeException: Can't parse input data: 'Raj'
> at SQOOP_3158.__loadFromFields(SQOOP_3158.java:316)
> at SQOOP_3158.parse(SQOOP_3158.java:254)
> at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
> at
> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.util.NoSuchElementException
> at java.util.ArrayList$Itr.next(ArrayList.java:854)
> at SQOOP_3158.__loadFromFields(SQOOP_3158.java:311)
> ... 12 more
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/orm/ClassWriter.java eaa9123
> src/test/com/cloudera/sqoop/TestExport.java b2edc53
> src/test/org/apache/sqoop/TestExportUsingProcedure.java 519305c
>
>
> Diff: https://reviews.apache.org/r/58466/diff/5/
>
>
> Testing
> -------
>
> There is no existing test class to cover the path and I am not sure the best
> way to add test case for this. If you have any suggestion, please let me know.
>
> I have done manual testing to replicate the issue and confirmed that patch
> fixed the issue. I have also tried different data types, all working.
>
> However, if column in MySQL is defined as NOT NULL, then the export will
> still fail with error, this is expected.
>
>
> Thanks,
>
> Eric Lin
>
>
