Review Request 54575: SQOOP-3075: Simplify Unicode character support in source files (introduced by SQOOP-3074) by defining explicit locales instead of using EscapeUtils
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54575/ --- Review request for Sqoop and Anna Szonyi. Bugs: SQOOP-3075 https://issues.apache.org/jira/browse/SQOOP-3075 Repository: sqoop-trunk Description --- Although SQOOP-3074 achieved that Sqoop won't fail anymore with special characters even if the locale on the caller system is not UTF-8, there's a simpler solution to do the same thing, by tuning the compiler + classwriter itself (by adding explicit encodings) instead of using the StringEscapeUtils#escapeJava mechanism. This solution seems simpler, and won't introduce any confusion around generation of Java identifiers. Diffs - src/java/org/apache/sqoop/avro/AvroUtil.java 8d901302 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 5b1c7457 src/java/org/apache/sqoop/orm/ClassWriter.java 0c8d86d0 src/java/org/apache/sqoop/orm/CompilationManager.java 0a2a87f2 Diff: https://reviews.apache.org/r/54575/diff/ Testing --- ant clean test ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/root -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://mysqlhostname/ -Dsqoop.test.mysql.databasename=sqoop_repo -Dsqoop.test.mysql.password=sqoop -Dsqoop.test.mysql.username=sqoop -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//oraclehostname/orcl -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://postgreshostname/ -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:cubridhotname:33000 -Dsqoop.test.cubrid.connectstring.username=sqoop -Dsqoop.test.cubrid.connectstring.database=sqoop -Dsqoop.test.cubrid.connectstring.password=sqoop -Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" -Dtest.timeout=100 Thanks, Attila Szabo
[jira] [Updated] (SQOOP-3075) Simplify Unicode character support in source files (introduced by SQOOP-3074) by defining explicit locales instead of using EscapeUtils
[ https://issues.apache.org/jira/browse/SQOOP-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Szabo updated SQOOP-3075: Attachment: SQOOP-3075.patch > Simplify Unicode character support in source files (introduced by SQOOP-3074) > by defining explicit locales instead of using EscapeUtils > --- > > Key: SQOOP-3075 > URL: https://issues.apache.org/jira/browse/SQOOP-3075 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Attila Szabo > Attachments: SQOOP-3075.patch > > > Although [SQOOP-3074] achieved that Sqoop won't fail anymore with special > characters even if the locale on the caller system is not UTF-8, there's a > simpler solution to do the same thing, by tuning the compiler + classwriter > itself (by adding explicit encodings) instead of using the > StringEscapeUtils#escapeJava mechanism. > This solution seems simpler, and won't introduce any confusion around > generation of Java identifiers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-3075) Simplify Unicode character support in source files (introduced by SQOOP-3074) by defining explicit locales instead of using EscapeUtils
Attila Szabo created SQOOP-3075: --- Summary: Simplify Unicode character support in source files (introduced by SQOOP-3074) by defining explicit locales instead of using EscapeUtils Key: SQOOP-3075 URL: https://issues.apache.org/jira/browse/SQOOP-3075 Project: Sqoop Issue Type: Improvement Reporter: Attila Szabo Assignee: Attila Szabo Although [SQOOP-3074] achieved that Sqoop won't fail anymore with special characters even if the locale on the caller system is not UTF-8, there's a simpler solution to do the same thing, by tuning the compiler + classwriter itself (by adding explicit encodings) instead of using the StringEscapeUtils#escapeJava mechanism. This solution seems simpler, and won't introduce any confusion around generation of Java identifiers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3074) Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
[ https://issues.apache.org/jira/browse/SQOOP-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733694#comment-15733694 ] Hudson commented on SQOOP-3074: --- SUCCESS: Integrated in Jenkins build Sqoop-hadoop100 #1045 (See [https://builds.apache.org/job/Sqoop-hadoop100/1045/]) SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5771a2da5fc071ca8f80f222e8468a29419e845e]) * (edit) src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java * (edit) src/java/org/apache/sqoop/avro/AvroUtil.java * (edit) src/java/org/apache/sqoop/orm/ClassWriter.java > Fix Avro import not to fail with Javac errors in case of non UTF-8 locale > - > > Key: SQOOP-3074 > URL: https://issues.apache.org/jira/browse/SQOOP-3074 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Attila Szabo > Attachments: SQOOP-3074.patch > > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3074) Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
[ https://issues.apache.org/jira/browse/SQOOP-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733622#comment-15733622 ] Hudson commented on SQOOP-3074: --- SUCCESS: Integrated in Jenkins build Sqoop-hadoop23 #1282 (See [https://builds.apache.org/job/Sqoop-hadoop23/1282/]) SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5771a2da5fc071ca8f80f222e8468a29419e845e]) * (edit) src/java/org/apache/sqoop/avro/AvroUtil.java * (edit) src/java/org/apache/sqoop/orm/ClassWriter.java * (edit) src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java > Fix Avro import not to fail with Javac errors in case of non UTF-8 locale > - > > Key: SQOOP-3074 > URL: https://issues.apache.org/jira/browse/SQOOP-3074 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Attila Szabo > Attachments: SQOOP-3074.patch > > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3074) Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
[ https://issues.apache.org/jira/browse/SQOOP-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733616#comment-15733616 ] Hudson commented on SQOOP-3074: --- SUCCESS: Integrated in Jenkins build Sqoop-hadoop20 #1080 (See [https://builds.apache.org/job/Sqoop-hadoop20/1080/]) SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5771a2da5fc071ca8f80f222e8468a29419e845e]) * (edit) src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java * (edit) src/java/org/apache/sqoop/avro/AvroUtil.java * (edit) src/java/org/apache/sqoop/orm/ClassWriter.java > Fix Avro import not to fail with Javac errors in case of non UTF-8 locale > - > > Key: SQOOP-3074 > URL: https://issues.apache.org/jira/browse/SQOOP-3074 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Attila Szabo > Attachments: SQOOP-3074.patch > > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3074) Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
[ https://issues.apache.org/jira/browse/SQOOP-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733581#comment-15733581 ] Hudson commented on SQOOP-3074: --- SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1086 (See [https://builds.apache.org/job/Sqoop-hadoop200/1086/]) SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5771a2da5fc071ca8f80f222e8468a29419e845e]) * (edit) src/java/org/apache/sqoop/avro/AvroUtil.java * (edit) src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java * (edit) src/java/org/apache/sqoop/orm/ClassWriter.java > Fix Avro import not to fail with Javac errors in case of non UTF-8 locale > - > > Key: SQOOP-3074 > URL: https://issues.apache.org/jira/browse/SQOOP-3074 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Attila Szabo > Attachments: SQOOP-3074.patch > > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3074) Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
[ https://issues.apache.org/jira/browse/SQOOP-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733537#comment-15733537 ] ASF subversion and git services commented on SQOOP-3074: Commit 5771a2da5fc071ca8f80f222e8468a29419e845e in sqoop's branch refs/heads/trunk from [~maugli] [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5771a2d ] SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non UTF-8 locale (Attila Szabo) > Fix Avro import not to fail with Javac errors in case of non UTF-8 locale > - > > Key: SQOOP-3074 > URL: https://issues.apache.org/jira/browse/SQOOP-3074 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Attila Szabo > Attachments: SQOOP-3074.patch > > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 54540: SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54540/#review158582 --- Ship it! Hey Attila, Thanks for the clarification! I've ran ant clean test and thirdparty tests with configured server names and everything seems to be running correctly! Thanks, /Anna - Anna Szonyi On Dec. 8, 2016, 9:53 p.m., Attila Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54540/ > --- > > (Updated Dec. 8, 2016, 9:53 p.m.) > > > Review request for Sqoop and Anna Szonyi. > > > Bugs: SQOOP-3074 > https://issues.apache.org/jira/browse/SQOOP-3074 > > > Repository: sqoop-trunk > > > Description > --- > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. > > > Diffs > - > > src/java/org/apache/sqoop/avro/AvroUtil.java ee29f140 > src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 3c31c43a > src/java/org/apache/sqoop/orm/ClassWriter.java 6f6e66b5 > > Diff: https://reviews.apache.org/r/54540/diff/ > > > Testing > --- > > ant clean test > > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/root > -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://mysqlhostname/ > -Dsqoop.test.mysql.databasename=sqoop_repo -Dsqoop.test.mysql.password=sqoop > -Dsqoop.test.mysql.username=sqoop > -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//oraclehostname/orcl > -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://postgreshostname/ > -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:cubridhotname:33000 > -Dsqoop.test.cubrid.connectstring.username=sqoop > -Dsqoop.test.cubrid.connectstring.database=sqoop > -Dsqoop.test.cubrid.connectstring.password=sqoop > -Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" > -Dtest.timeout=100 > > > Thanks, > > Attila Szabo > >
Re: Review Request 54540: SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
> On Dec. 8, 2016, 9:34 p.m., Anna Szonyi wrote: > > Hi Attila, > > > > Thanks for finding this! Generally I prefer to leave comments out of code, > > however as this is a very particular use-case and the solution is > > non-trivial, in my opinion it would be useful to add a line or two of > > comments on why/in which case the unescape and escape is necessary. > > > > Thanks, > > /Anna Hey Anna, Thanks for the review! I've added some clarification comments. Would you please check if it is clearer now on your side? Thanks, Attila - Attila --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54540/#review158574 --- On Dec. 8, 2016, 9:53 p.m., Attila Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54540/ > --- > > (Updated Dec. 8, 2016, 9:53 p.m.) > > > Review request for Sqoop and Anna Szonyi. > > > Bugs: SQOOP-3074 > https://issues.apache.org/jira/browse/SQOOP-3074 > > > Repository: sqoop-trunk > > > Description > --- > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. > > > Diffs > - > > src/java/org/apache/sqoop/avro/AvroUtil.java ee29f140 > src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 3c31c43a > src/java/org/apache/sqoop/orm/ClassWriter.java 6f6e66b5 > > Diff: https://reviews.apache.org/r/54540/diff/ > > > Testing > --- > > ant clean test > > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/root > -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://mysqlhostname/ > -Dsqoop.test.mysql.databasename=sqoop_repo -Dsqoop.test.mysql.password=sqoop > -Dsqoop.test.mysql.username=sqoop > -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//oraclehostname/orcl > -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://postgreshostname/ > -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:cubridhotname:33000 > -Dsqoop.test.cubrid.connectstring.username=sqoop > -Dsqoop.test.cubrid.connectstring.database=sqoop > -Dsqoop.test.cubrid.connectstring.password=sqoop > -Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" > -Dtest.timeout=100 > > > Thanks, > > Attila Szabo > >
Re: Review Request 54540: SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54540/ --- (Updated Dec. 8, 2016, 9:53 p.m.) Review request for Sqoop and Anna Szonyi. Changes --- Comments had been added. Bugs: SQOOP-3074 https://issues.apache.org/jira/browse/SQOOP-3074 Repository: sqoop-trunk Description --- The current implementation of Sqoop will fail to compile the ClassWriter generated Java class in case of UTF-8 characters are available in the database but the running host don't have UTF-8 or not supporting that. Meanwhile we'd like to keep up the support of UTF-8 characters for example in Avro schemas. A good example for that TestAvroImport#testNonstandardCharactersInColumnName fails if executed form a shell/env with non UTF-8 locale. We have to provide an implementation reflects this. Diffs (updated) - src/java/org/apache/sqoop/avro/AvroUtil.java ee29f140 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 3c31c43a src/java/org/apache/sqoop/orm/ClassWriter.java 6f6e66b5 Diff: https://reviews.apache.org/r/54540/diff/ Testing --- ant clean test ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/root -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://mysqlhostname/ -Dsqoop.test.mysql.databasename=sqoop_repo -Dsqoop.test.mysql.password=sqoop -Dsqoop.test.mysql.username=sqoop -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//oraclehostname/orcl -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://postgreshostname/ -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:cubridhotname:33000 -Dsqoop.test.cubrid.connectstring.username=sqoop -Dsqoop.test.cubrid.connectstring.database=sqoop -Dsqoop.test.cubrid.connectstring.password=sqoop -Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" -Dtest.timeout=100 Thanks, Attila Szabo
Re: Review Request 54540: SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54540/#review158574 --- Hi Attila, Thanks for finding this! Generally I prefer to leave comments out of code, however as this is a very particular use-case and the solution is non-trivial, in my opinion it would be useful to add a line or two of comments on why/in which case the unescape and escape is necessary. Thanks, /Anna - Anna Szonyi On Dec. 8, 2016, 6:25 p.m., Attila Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54540/ > --- > > (Updated Dec. 8, 2016, 6:25 p.m.) > > > Review request for Sqoop and Anna Szonyi. > > > Bugs: SQOOP-3074 > https://issues.apache.org/jira/browse/SQOOP-3074 > > > Repository: sqoop-trunk > > > Description > --- > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. > > > Diffs > - > > src/java/org/apache/sqoop/avro/AvroUtil.java ee29f140 > src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 3c31c43a > src/java/org/apache/sqoop/orm/ClassWriter.java 6f6e66b5 > > Diff: https://reviews.apache.org/r/54540/diff/ > > > Testing > --- > > ant clean test > > ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/root > -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://mysqlhostname/ > -Dsqoop.test.mysql.databasename=sqoop_repo -Dsqoop.test.mysql.password=sqoop > -Dsqoop.test.mysql.username=sqoop > -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//oraclehostname/orcl > -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://postgreshostname/ > -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:cubridhotname:33000 > -Dsqoop.test.cubrid.connectstring.username=sqoop > -Dsqoop.test.cubrid.connectstring.database=sqoop > -Dsqoop.test.cubrid.connectstring.password=sqoop > -Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" > -Dtest.timeout=100 > > > Thanks, > > Attila Szabo > >
Review Request 54540: SQOOP-3074: Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54540/ --- Review request for Sqoop and Anna Szonyi. Bugs: SQOOP-3074 https://issues.apache.org/jira/browse/SQOOP-3074 Repository: sqoop-trunk Description --- The current implementation of Sqoop will fail to compile the ClassWriter generated Java class in case of UTF-8 characters are available in the database but the running host don't have UTF-8 or not supporting that. Meanwhile we'd like to keep up the support of UTF-8 characters for example in Avro schemas. A good example for that TestAvroImport#testNonstandardCharactersInColumnName fails if executed form a shell/env with non UTF-8 locale. We have to provide an implementation reflects this. Diffs - src/java/org/apache/sqoop/avro/AvroUtil.java ee29f140 src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 3c31c43a src/java/org/apache/sqoop/orm/ClassWriter.java 6f6e66b5 Diff: https://reviews.apache.org/r/54540/diff/ Testing --- ant clean test ant clean test -Dthirdparty=true -Dsqoop.thirdparty.lib.dir=/root -Dsqoop.test.mysql.connectstring.host_url=jdbc:mysql://mysqlhostname/ -Dsqoop.test.mysql.databasename=sqoop_repo -Dsqoop.test.mysql.password=sqoop -Dsqoop.test.mysql.username=sqoop -Dsqoop.test.oracle.connectstring=jdbc:oracle:thin:@//oraclehostname/orcl -Dsqoop.test.postgresql.connectstring.host_url=jdbc:postgresql://postgreshostname/ -Dsqoop.test.cubrid.connectstring.host_url=jdbc:cubrid:cubridhotname:33000 -Dsqoop.test.cubrid.connectstring.username=sqoop -Dsqoop.test.cubrid.connectstring.database=sqoop -Dsqoop.test.cubrid.connectstring.password=sqoop -Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" -Dtest.timeout=100 Thanks, Attila Szabo
[jira] [Updated] (SQOOP-3074) Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
[ https://issues.apache.org/jira/browse/SQOOP-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Szabo updated SQOOP-3074: Attachment: SQOOP-3074.patch > Fix Avro import not to fail with Javac errors in case of non UTF-8 locale > - > > Key: SQOOP-3074 > URL: https://issues.apache.org/jira/browse/SQOOP-3074 > Project: Sqoop > Issue Type: Bug >Reporter: Attila Szabo >Assignee: Attila Szabo > Attachments: SQOOP-3074.patch > > > The current implementation of Sqoop will fail to compile the ClassWriter > generated Java class in case of UTF-8 characters are available in the > database but the running host don't have UTF-8 or not supporting that. > Meanwhile we'd like to keep up the support of UTF-8 characters for example in > Avro schemas. > A good example for that TestAvroImport#testNonstandardCharactersInColumnName > fails if executed form a shell/env with non UTF-8 locale. > We have to provide an implementation reflects this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SQOOP-3074) Fix Avro import not to fail with Javac errors in case of non UTF-8 locale
Attila Szabo created SQOOP-3074: --- Summary: Fix Avro import not to fail with Javac errors in case of non UTF-8 locale Key: SQOOP-3074 URL: https://issues.apache.org/jira/browse/SQOOP-3074 Project: Sqoop Issue Type: Bug Reporter: Attila Szabo Assignee: Attila Szabo The current implementation of Sqoop will fail to compile the ClassWriter generated Java class in case of UTF-8 characters are available in the database but the running host don't have UTF-8 or not supporting that. Meanwhile we'd like to keep up the support of UTF-8 characters for example in Avro schemas. A good example for that TestAvroImport#testNonstandardCharactersInColumnName fails if executed form a shell/env with non UTF-8 locale. We have to provide an implementation reflects this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3051) Remove/delete obsolete profiles from build.xml
[ https://issues.apache.org/jira/browse/SQOOP-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732208#comment-15732208 ] Hudson commented on SQOOP-3051: --- FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1085 (See [https://builds.apache.org/job/Sqoop-hadoop200/1085/]) SQOOP-3051:Remove all old profiles, configurations from build.xml and (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=44e05df21520d3f7cf7b21fab0fd1b97fe48a774]) * (edit) ivy/ivysettings.xml * (edit) build.xml * (edit) ivy.xml * (edit) ivy/libraries.properties > Remove/delete obsolete profiles from build.xml > -- > > Key: SQOOP-3051 > URL: https://issues.apache.org/jira/browse/SQOOP-3051 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Anna Szonyi >Priority: Critical > Fix For: 1.4.7 > > Attachments: SQOOP-3051-1.patch, SQOOP-3051.patch > > > As [SQOOP-3050] is implemented all the obsolete profiles/settings should be > eliminated to make the dependency tree and the whole build environment around > Sqoop trunk version simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3051) Remove/delete obsolete profiles from build.xml
[ https://issues.apache.org/jira/browse/SQOOP-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732189#comment-15732189 ] Hudson commented on SQOOP-3051: --- FAILURE: Integrated in Jenkins build Sqoop-hadoop100 #1044 (See [https://builds.apache.org/job/Sqoop-hadoop100/1044/]) SQOOP-3051:Remove all old profiles, configurations from build.xml and (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=44e05df21520d3f7cf7b21fab0fd1b97fe48a774]) * (edit) build.xml * (edit) ivy.xml * (edit) ivy/ivysettings.xml * (edit) ivy/libraries.properties > Remove/delete obsolete profiles from build.xml > -- > > Key: SQOOP-3051 > URL: https://issues.apache.org/jira/browse/SQOOP-3051 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Anna Szonyi >Priority: Critical > Fix For: 1.4.7 > > Attachments: SQOOP-3051-1.patch, SQOOP-3051.patch > > > As [SQOOP-3050] is implemented all the obsolete profiles/settings should be > eliminated to make the dependency tree and the whole build environment around > Sqoop trunk version simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3051) Remove/delete obsolete profiles from build.xml
[ https://issues.apache.org/jira/browse/SQOOP-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732122#comment-15732122 ] Hudson commented on SQOOP-3051: --- FAILURE: Integrated in Jenkins build Sqoop-hadoop23 #1281 (See [https://builds.apache.org/job/Sqoop-hadoop23/1281/]) SQOOP-3051:Remove all old profiles, configurations from build.xml and (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=44e05df21520d3f7cf7b21fab0fd1b97fe48a774]) * (edit) ivy/ivysettings.xml * (edit) ivy.xml * (edit) build.xml * (edit) ivy/libraries.properties > Remove/delete obsolete profiles from build.xml > -- > > Key: SQOOP-3051 > URL: https://issues.apache.org/jira/browse/SQOOP-3051 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Anna Szonyi >Priority: Critical > Fix For: 1.4.7 > > Attachments: SQOOP-3051-1.patch, SQOOP-3051.patch > > > As [SQOOP-3050] is implemented all the obsolete profiles/settings should be > eliminated to make the dependency tree and the whole build environment around > Sqoop trunk version simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3051) Remove/delete obsolete profiles from build.xml
[ https://issues.apache.org/jira/browse/SQOOP-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732114#comment-15732114 ] Hudson commented on SQOOP-3051: --- FAILURE: Integrated in Jenkins build Sqoop-hadoop20 #1079 (See [https://builds.apache.org/job/Sqoop-hadoop20/1079/]) SQOOP-3051:Remove all old profiles, configurations from build.xml and (maugli: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=44e05df21520d3f7cf7b21fab0fd1b97fe48a774]) * (edit) ivy.xml * (edit) build.xml * (edit) ivy/ivysettings.xml * (edit) ivy/libraries.properties > Remove/delete obsolete profiles from build.xml > -- > > Key: SQOOP-3051 > URL: https://issues.apache.org/jira/browse/SQOOP-3051 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Anna Szonyi >Priority: Critical > Fix For: 1.4.7 > > Attachments: SQOOP-3051-1.patch, SQOOP-3051.patch > > > As [SQOOP-3050] is implemented all the obsolete profiles/settings should be > eliminated to make the dependency tree and the whole build environment around > Sqoop trunk version simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3051) Remove/delete obsolete profiles from build.xml
[ https://issues.apache.org/jira/browse/SQOOP-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732073#comment-15732073 ] Attila Szabo commented on SQOOP-3051: - Thank you [~szonyi] for you contribution! This is another great victory on the path to make the build/CI/test system clean and working again! Way to go! Thanks, [~maugli] > Remove/delete obsolete profiles from build.xml > -- > > Key: SQOOP-3051 > URL: https://issues.apache.org/jira/browse/SQOOP-3051 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Anna Szonyi >Priority: Critical > Fix For: 1.4.7 > > Attachments: SQOOP-3051-1.patch, SQOOP-3051.patch > > > As [SQOOP-3050] is implemented all the obsolete profiles/settings should be > eliminated to make the dependency tree and the whole build environment around > Sqoop trunk version simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3051) Remove/delete obsolete profiles from build.xml
[ https://issues.apache.org/jira/browse/SQOOP-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732071#comment-15732071 ] ASF subversion and git services commented on SQOOP-3051: Commit 44e05df21520d3f7cf7b21fab0fd1b97fe48a774 in sqoop's branch refs/heads/trunk from [~maugli] [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=44e05df ] SQOOP-3051:Remove all old profiles, configurations from build.xml and ivy.xml and consolidate them to a single working one (Anna Szonyi via Attila Szabo) > Remove/delete obsolete profiles from build.xml > -- > > Key: SQOOP-3051 > URL: https://issues.apache.org/jira/browse/SQOOP-3051 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Anna Szonyi >Priority: Critical > Fix For: 1.4.7 > > Attachments: SQOOP-3051-1.patch, SQOOP-3051.patch > > > As [SQOOP-3050] is implemented all the obsolete profiles/settings should be > eliminated to make the dependency tree and the whole build environment around > Sqoop trunk version simpler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 54325: SQOOP-3051 Remove all old profiles, configurations from build.xml and ivy.xml and consolidate them to a single working one
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54325/#review158516 --- Ship it! Verification steps: ant clean compile ant clean compile-all ant clean jar ant clean jar-all ant clean test ant clean test (with all 3rd party tests and options) every scenario has been executed successfully. That's another great victory Anna Szonyi! Way to go! - Attila Szabo On Dec. 5, 2016, 1:53 p.m., Anna Szonyi wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54325/ > --- > > (Updated Dec. 5, 2016, 1:53 p.m.) > > > Review request for Sqoop, Abraham Elmahrek, Abraham Fine, Boglarka Egyed, > Jarek Cecho, Attila Szabo, Szabolcs Vasas, Venkat Ranganathan, and Erzsebet > Szilagyi. > > > Bugs: SQOOP-3051 > https://issues.apache.org/jira/browse/SQOOP-3051 > > > Repository: sqoop-trunk > > > Description > --- > > SQOOP-3051 Remove all old profiles, configurations from build.xml and ivy.xml > and consolidate them to a single working one (formerly known as hadoop260). > > > Diffs > - > > build.xml 3484d74 > ivy.xml ee1dafa > ivy/ivysettings.xml ac16b45 > ivy/libraries.properties 8e14624 > > Diff: https://reviews.apache.org/r/54325/diff/ > > > Testing > --- > > ant jar-all, ant package, ant compile, ant eclipse, tried with specifying > -Dhadoop.version, -Davro.version, -Dhbase.version, > -Dhcatalog.version, etc. to verify that it overrides the defaults. > > > Thanks, > > Anna Szonyi > >
[jira] [Comment Edited] (SQOOP-3061) Sqoop --options-file failed with error "Malformed option in options file" even though the query is correct
[
https://issues.apache.org/jira/browse/SQOOP-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728401#comment-15728401
]
Eric Lin edited comment on SQOOP-3061 at 12/8/16 9:50 AM:
--
Provided new patch as suggested in review: https://reviews.apache.org/r/54251/,
SQOOP-3061.2.patch.
Thanks [~vasas]!
was (Author: ericlin):
Provided new patch as suggested in review: https://reviews.apache.org/r/54251/.
Thanks [~vasas]!
> Sqoop --options-file failed with error "Malformed option in options file"
> even though the query is correct
> --
>
> Key: SQOOP-3061
> URL: https://issues.apache.org/jira/browse/SQOOP-3061
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Priority: Minor
> Labels: patch
> Attachments: SQOOP-3061.2.patch, SQOOP-3061.3.patch, SQOOP-3061.patch
>
>
> if you have the following in the options file:
> --query
> SELECT * FROM test WHERE a = 'b'
> and then run
> {code}
> sqoop --options-file
> {code}
> it will fail with the following error:
> {code}
> 16/11/22 16:08:59 ERROR sqoop.Sqoop: Error while expanding arguments
> java.lang.Exception: Malformed option in options
> file(/tmp/sqoop_runner_from_stdin_1112_12354__sqoop_options_file): SELECT *
> FROM test WHERE a = 'b'
> at
> org.apache.sqoop.util.OptionsFileUtil.removeQuoteCharactersIfNecessary(OptionsFileUtil.java:170)
> at
> org.apache.sqoop.util.OptionsFileUtil.removeQuotesEncolosingOption(OptionsFileUtil.java:136)
> at
> org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:90)
> at
> com.cloudera.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:33)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:199)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> Malformed option in options
> file(/tmp/sqoop_runner_from_stdin_1112_12354__sqoop_options_file): SELECT *
> FROM test WHERE a = 'b'
> {code}
> This is caused by function
> org.apache.sqoop.util.OptionsFileUtil.removeQuoteCharactersIfNecessary only
> checks for starting and ending quotes and will fail if the query does not
> start with a quote but ends with a quote, like the example query above.
> {code}
> private static String removeQuoteCharactersIfNecessary(String fileName,
> String option, char quote) throws Exception {
> boolean startingQuote = (option.charAt(0) == quote);
> boolean endingQuote = (option.charAt(option.length() - 1) == quote);
> if (startingQuote && endingQuote) {
> if (option.length() == 1) {
> throw new Exception("Malformed option in options file("
> + fileName + "): " + option);
> }
> return option.substring(1, option.length() - 1);
> }
> if (startingQuote || endingQuote) {
>throw new Exception("Malformed option in options file("
>+ fileName + "): " + option);
> }
> return option;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (SQOOP-3061) Sqoop --options-file failed with error "Malformed option in options file" even though the query is correct
[
https://issues.apache.org/jira/browse/SQOOP-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731695#comment-15731695
]
Eric Lin edited comment on SQOOP-3061 at 12/8/16 9:49 AM:
--
Based on latest review feedback from [~maugli], I have made some changes into
SQOOP-3061.3.patch
was (Author: ericlin):
Based on latest review feedback from [~maugli], I have made some changes.
> Sqoop --options-file failed with error "Malformed option in options file"
> even though the query is correct
> --
>
> Key: SQOOP-3061
> URL: https://issues.apache.org/jira/browse/SQOOP-3061
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Priority: Minor
> Labels: patch
> Attachments: SQOOP-3061.2.patch, SQOOP-3061.3.patch, SQOOP-3061.patch
>
>
> if you have the following in the options file:
> --query
> SELECT * FROM test WHERE a = 'b'
> and then run
> {code}
> sqoop --options-file
> {code}
> it will fail with the following error:
> {code}
> 16/11/22 16:08:59 ERROR sqoop.Sqoop: Error while expanding arguments
> java.lang.Exception: Malformed option in options
> file(/tmp/sqoop_runner_from_stdin_1112_12354__sqoop_options_file): SELECT *
> FROM test WHERE a = 'b'
> at
> org.apache.sqoop.util.OptionsFileUtil.removeQuoteCharactersIfNecessary(OptionsFileUtil.java:170)
> at
> org.apache.sqoop.util.OptionsFileUtil.removeQuotesEncolosingOption(OptionsFileUtil.java:136)
> at
> org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:90)
> at
> com.cloudera.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:33)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:199)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> Malformed option in options
> file(/tmp/sqoop_runner_from_stdin_1112_12354__sqoop_options_file): SELECT *
> FROM test WHERE a = 'b'
> {code}
> This is caused by function
> org.apache.sqoop.util.OptionsFileUtil.removeQuoteCharactersIfNecessary only
> checks for starting and ending quotes and will fail if the query does not
> start with a quote but ends with a quote, like the example query above.
> {code}
> private static String removeQuoteCharactersIfNecessary(String fileName,
> String option, char quote) throws Exception {
> boolean startingQuote = (option.charAt(0) == quote);
> boolean endingQuote = (option.charAt(option.length() - 1) == quote);
> if (startingQuote && endingQuote) {
> if (option.length() == 1) {
> throw new Exception("Malformed option in options file("
> + fileName + "): " + option);
> }
> return option.substring(1, option.length() - 1);
> }
> if (startingQuote || endingQuote) {
>throw new Exception("Malformed option in options file("
>+ fileName + "): " + option);
> }
> return option;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SQOOP-3061) Sqoop --options-file failed with error "Malformed option in options file" even though the query is correct
[
https://issues.apache.org/jira/browse/SQOOP-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Lin updated SQOOP-3061:
Attachment: SQOOP-3061.3.patch
Based on latest review feedback from [~maugli], I have made some changes.
> Sqoop --options-file failed with error "Malformed option in options file"
> even though the query is correct
> --
>
> Key: SQOOP-3061
> URL: https://issues.apache.org/jira/browse/SQOOP-3061
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Priority: Minor
> Labels: patch
> Attachments: SQOOP-3061.2.patch, SQOOP-3061.3.patch, SQOOP-3061.patch
>
>
> if you have the following in the options file:
> --query
> SELECT * FROM test WHERE a = 'b'
> and then run
> {code}
> sqoop --options-file
> {code}
> it will fail with the following error:
> {code}
> 16/11/22 16:08:59 ERROR sqoop.Sqoop: Error while expanding arguments
> java.lang.Exception: Malformed option in options
> file(/tmp/sqoop_runner_from_stdin_1112_12354__sqoop_options_file): SELECT *
> FROM test WHERE a = 'b'
> at
> org.apache.sqoop.util.OptionsFileUtil.removeQuoteCharactersIfNecessary(OptionsFileUtil.java:170)
> at
> org.apache.sqoop.util.OptionsFileUtil.removeQuotesEncolosingOption(OptionsFileUtil.java:136)
> at
> org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:90)
> at
> com.cloudera.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:33)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:199)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> Malformed option in options
> file(/tmp/sqoop_runner_from_stdin_1112_12354__sqoop_options_file): SELECT *
> FROM test WHERE a = 'b'
> {code}
> This is caused by function
> org.apache.sqoop.util.OptionsFileUtil.removeQuoteCharactersIfNecessary only
> checks for starting and ending quotes and will fail if the query does not
> start with a quote but ends with a quote, like the example query above.
> {code}
> private static String removeQuoteCharactersIfNecessary(String fileName,
> String option, char quote) throws Exception {
> boolean startingQuote = (option.charAt(0) == quote);
> boolean endingQuote = (option.charAt(option.length() - 1) == quote);
> if (startingQuote && endingQuote) {
> if (option.length() == 1) {
> throw new Exception("Malformed option in options file("
> + fileName + "): " + option);
> }
> return option.substring(1, option.length() - 1);
> }
> if (startingQuote || endingQuote) {
>throw new Exception("Malformed option in options file("
>+ fileName + "): " + option);
> }
> return option;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
