[jira] [Commented] (SQOOP-2903) Add Kudu connector for Sqoop
[ https://issues.apache.org/jira/browse/SQOOP-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908139#comment-16908139 ] Szabolcs Vasas commented on SQOOP-2903: --- Hi [~tsss686], This feature was never committed so it is not available in Sqoop yet. Since the patch was updated years ago and it conflicts with the latest codebase I suspect it requires a significant amount of work to make it work. Regards, Szabolcs > Add Kudu connector for Sqoop > > > Key: SQOOP-2903 > URL: https://issues.apache.org/jira/browse/SQOOP-2903 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Reporter: Sameer Abhyankar >Assignee: Sameer Abhyankar >Priority: Major > Attachments: SQOOP-2903.1.patch, SQOOP-2903.2.patch, SQOOP-2903.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Sqoop currently does not have a connector for Kudu. We should add the > functionality to allow Sqoop to ingest data directly into Kudu. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (SQOOP-3134) --class-name should override default Avro schema name
[
https://issues.apache.org/jira/browse/SQOOP-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas resolved SQOOP-3134.
---
Resolution: Fixed
Fix Version/s: 3.0.0
1.5.0
Hi [~dvoros],
Your patch is now committed, thank you for your contribution!
Regards,
Szabolcs
> --class-name should override default Avro schema name
> -
>
> Key: SQOOP-3134
> URL: https://issues.apache.org/jira/browse/SQOOP-3134
> Project: Sqoop
> Issue Type: Improvement
>Reporter: Markus Kemper
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3134.1.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Please consider adding an option to configure the Avro schema output file
> name that is created with Sqoop (import + --as-avrodatafile), example cases
> below.
> {noformat}
> #
> # STEP 01 - Create Data
> #
> export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
> export MYUSER=sqoop
> export MYPSWD=cloudera
> sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "drop table t1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "create table t1 (c1 int, c2 date, c3 varchar(10))"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "insert into t1 values (1, current_date, 'some data')"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "select * from t1"
> -
> | c1 | c2 | c3 |
> -
> | 1 | 2017-02-13 | some data |
> -
> #
> # STEP 02 - Import + --table + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
> --as-avrodatafile
> ls -l ./*
> Output:
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Transferred 413 bytes in
> 20.6988 seconds (19.9529 bytes/sec)
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Retrieved 1 records.
>
> -rw-r--r-- 1 root root 492 Feb 13 12:14 ./t1.avsc < want option to
> configure this file name
> -rw-r--r-- 1 root root 12462 Feb 13 12:14 ./t1.java
> #
> # STEP 03 - Import + --query + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "select * from t1 where \$CONDITIONS" --split-by c1 --target-dir
> /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile
> ls -l ./*
> Output:
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Transferred 448 bytes in
> 25.2757 seconds (17.7245 bytes/sec)
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> -rw-r--r-- 1 root root 527 Feb 13 12:16 ./AutoGeneratedSchema.avsc <
> want option to configure this file name
> -rw-r--r-- 1 root root 12590 Feb 13 12:16 ./QueryResult.java
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3134) --class-name should override default Avro schema name
[
https://issues.apache.org/jira/browse/SQOOP-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas updated SQOOP-3134:
--
Summary: --class-name should override default Avro schema name (was: Add
option to configure Avro schema output file name with (import +
--as-avrodatafile) )
> --class-name should override default Avro schema name
> -
>
> Key: SQOOP-3134
> URL: https://issues.apache.org/jira/browse/SQOOP-3134
> Project: Sqoop
> Issue Type: Improvement
>Reporter: Markus Kemper
>Assignee: Daniel Voros
>Priority: Major
> Attachments: SQOOP-3134.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Please consider adding an option to configure the Avro schema output file
> name that is created with Sqoop (import + --as-avrodatafile), example cases
> below.
> {noformat}
> #
> # STEP 01 - Create Data
> #
> export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
> export MYUSER=sqoop
> export MYPSWD=cloudera
> sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "drop table t1"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "create table t1 (c1 int, c2 date, c3 varchar(10))"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "insert into t1 values (1, current_date, 'some data')"
> sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "select * from t1"
> -
> | c1 | c2 | c3 |
> -
> | 1 | 2017-02-13 | some data |
> -
> #
> # STEP 02 - Import + --table + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
> t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
> --as-avrodatafile
> ls -l ./*
> Output:
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Transferred 413 bytes in
> 20.6988 seconds (19.9529 bytes/sec)
> 17/02/13 12:14:52 INFO mapreduce.ImportJobBase: Retrieved 1 records.
>
> -rw-r--r-- 1 root root 492 Feb 13 12:14 ./t1.avsc < want option to
> configure this file name
> -rw-r--r-- 1 root root 12462 Feb 13 12:14 ./t1.java
> #
> # STEP 03 - Import + --query + --as-avrodatafile
> #
> sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --query
> "select * from t1 where \$CONDITIONS" --split-by c1 --target-dir
> /user/root/t1 --delete-target-dir --num-mappers 1 --as-avrodatafile
> ls -l ./*
> Output:
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Transferred 448 bytes in
> 25.2757 seconds (17.7245 bytes/sec)
> 17/02/13 12:16:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.
> ~
> -rw-r--r-- 1 root root 527 Feb 13 12:16 ./AutoGeneratedSchema.avsc <
> want option to configure this file name
> -rw-r--r-- 1 root root 12590 Feb 13 12:16 ./QueryResult.java
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SQOOP-3435) Avoid NullPointerException due to different JSONObject library in classpath
[
https://issues.apache.org/jira/browse/SQOOP-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas reassigned SQOOP-3435:
-
Assignee: Tak Lon (Stephen) Wu
> Avoid NullPointerException due to different JSONObject library in classpath
> ---
>
> Key: SQOOP-3435
> URL: https://issues.apache.org/jira/browse/SQOOP-3435
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7, 1.5.0
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: SQOOP-3435.trunk.001.patch, SQOOP-3435.trunk.002.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In line of
> [SqoopOptions.java#L785|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L785]
> with
> {\{SqoopJsonUtil.getJsonStringforMap((Map) f.get(this))); }}
> Above line should check NULL pointer like in line of
> [SqoopOptions.java#L778|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L778]
> which has
> {{f.get(this) == null ? "null" : f.get(this).toString()}}
> Please see the stacktrace below when running command:
> {{sqoop job --create myjob -- import --connect jdbc:mysql://localhost/db
> --username root --table employee --m 1}}
> {code:java}
> 19/02/02 01:09:21 ERROR sqoop.Sqoop: Got exception running Sqoop:
> java.lang.NullPointerException
> java.lang.NullPointerException
> at org.json.JSONObject.(JSONObject.java:144)
> at
> org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
> at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)
> at
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)
> at
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)
> at org.apache.sqoop.tool.JobTool.createJob(JobTool.java:181)
> at org.apache.sqoop.tool.JobTool.run(JobTool.java:294)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> {code}
> The above {{NullPointerException}} is due to use the of
> [{{com.tdunning}}|https://github.com/tdunning/open-json/blob/rc1.8/src/main/java/org/json/JSONObject.java#L141-L155]
> as part of the HIVE libs (if one is reusing the {{HADOOP_CLASSPATH}}) in the
> classpath. but I think we can better have a checker of {{null}} in
> {{SqoopJsonUtil.getJsonStringforMap(Map map)}} before calling
> {{JSONObject pathPartMap = new JSONObject(map);}}
> Reporting this bug and the right behavior need to be decided by the assignee.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3435) Avoid NullPointerException due to different JSONObject library in classpath
[
https://issues.apache.org/jira/browse/SQOOP-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas updated SQOOP-3435:
--
Summary: Avoid NullPointerException due to different JSONObject library in
classpath (was: SqoopOptions missed a null check)
> Avoid NullPointerException due to different JSONObject library in classpath
> ---
>
> Key: SQOOP-3435
> URL: https://issues.apache.org/jira/browse/SQOOP-3435
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7, 1.5.0
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: SQOOP-3435.trunk.001.patch, SQOOP-3435.trunk.002.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In line of
> [SqoopOptions.java#L785|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L785]
> with
> {\{SqoopJsonUtil.getJsonStringforMap((Map) f.get(this))); }}
> Above line should check NULL pointer like in line of
> [SqoopOptions.java#L778|https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/SqoopOptions.java#L778]
> which has
> {{f.get(this) == null ? "null" : f.get(this).toString()}}
> Please see the stacktrace below when running command:
> {{sqoop job --create myjob -- import --connect jdbc:mysql://localhost/db
> --username root --table employee --m 1}}
> {code:java}
> 19/02/02 01:09:21 ERROR sqoop.Sqoop: Got exception running Sqoop:
> java.lang.NullPointerException
> java.lang.NullPointerException
> at org.json.JSONObject.(JSONObject.java:144)
> at
> org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
> at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)
> at
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)
> at
> org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)
> at org.apache.sqoop.tool.JobTool.createJob(JobTool.java:181)
> at org.apache.sqoop.tool.JobTool.run(JobTool.java:294)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> {code}
> The above {{NullPointerException}} is due to use the of
> [{{com.tdunning}}|https://github.com/tdunning/open-json/blob/rc1.8/src/main/java/org/json/JSONObject.java#L141-L155]
> as part of the HIVE libs (if one is reusing the {{HADOOP_CLASSPATH}}) in the
> classpath. but I think we can better have a checker of {{null}} in
> {{SqoopJsonUtil.getJsonStringforMap(Map map)}} before calling
> {{JSONObject pathPartMap = new JSONObject(map);}}
> Reporting this bug and the right behavior need to be decided by the assignee.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (SQOOP-3427) Add Travis badge to the Readme
[ https://issues.apache.org/jira/browse/SQOOP-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3427. --- Resolution: Fixed HI [~Fokko], Your patch is now committed, thank you for your contribution! > Add Travis badge to the Readme > -- > > Key: SQOOP-3427 > URL: https://issues.apache.org/jira/browse/SQOOP-3427 > Project: Sqoop > Issue Type: Improvement > Components: docs >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We love badges -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3429) Bump Hadoop to 2.9.2
[ https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3429. --- Resolution: Fixed Hi [~Fokko], Your patch is now committed, thank you for your contribution! > Bump Hadoop to 2.9.2 > > > Key: SQOOP-3429 > URL: https://issues.apache.org/jira/browse/SQOOP-3429 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.7 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.5.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I would like to bump Sqoop to Hadoop 2.9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3429) Bump Hadoop to 2.9.2
[ https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3429: -- Summary: Bump Hadoop to 2.9.2 (was: Bump Sqoop to Hadoop 2.9.2) > Bump Hadoop to 2.9.2 > > > Key: SQOOP-3429 > URL: https://issues.apache.org/jira/browse/SQOOP-3429 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.7 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.5.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I would like to bump Sqoop to Hadoop 2.9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3429) Bump Sqoop to Hadoop 2.9.x
[ https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3429: - Assignee: Fokko Driesprong > Bump Sqoop to Hadoop 2.9.x > -- > > Key: SQOOP-3429 > URL: https://issues.apache.org/jira/browse/SQOOP-3429 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.7 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.5.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I would like to bump Sqoop to Hadoop 2.9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3429) Bump Sqoop to Hadoop 2.9.2
[ https://issues.apache.org/jira/browse/SQOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3429: -- Summary: Bump Sqoop to Hadoop 2.9.2 (was: Bump Sqoop to Hadoop 2.9.x) > Bump Sqoop to Hadoop 2.9.2 > -- > > Key: SQOOP-3429 > URL: https://issues.apache.org/jira/browse/SQOOP-3429 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.7 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.5.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I would like to bump Sqoop to Hadoop 2.9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3430) Remove the old maven pom
[ https://issues.apache.org/jira/browse/SQOOP-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3430. --- Resolution: Fixed Fix Version/s: 1.5.0 Hi [~Fokko], Your patch is now committed, thank you for your contribution! > Remove the old maven pom > > > Key: SQOOP-3430 > URL: https://issues.apache.org/jira/browse/SQOOP-3430 > Project: Sqoop > Issue Type: Improvement >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > I think both ant and gradle are enough :) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3420) Invalid ERROR message initiates false alarms
[ https://issues.apache.org/jira/browse/SQOOP-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3420: - Assignee: Denes Bodo > Invalid ERROR message initiates false alarms > > > Key: SQOOP-3420 > URL: https://issues.apache.org/jira/browse/SQOOP-3420 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Labels: usability > Attachments: SQOOP-3420_001.patch > > > In SQOOP-3042, a debug message was refactored to be error instead means false > alarms in customer log analyser. After understanding the functionality it is > recommended to use info level message instead of error in case when > ImportTool is unable to backup generated .java file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3423) Let user pass password to connect Hive when it set to LDAP authentication
[ https://issues.apache.org/jira/browse/SQOOP-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758291#comment-16758291 ] Szabolcs Vasas commented on SQOOP-3423: --- Hi [~dionusos], The direction seems to be correct, feel free to create a PR for it. > Let user pass password to connect Hive when it set to LDAP authentication > - > > Key: SQOOP-3423 > URL: https://issues.apache.org/jira/browse/SQOOP-3423 > Project: Sqoop > Issue Type: Improvement > Components: hive-integration >Affects Versions: 1.4.7 >Reporter: Denes Bodo >Priority: Major > Attachments: SQOOP-3423-001.patch > > > If HiveServer2 is using password based authentication, additional > username/password information has to be provided to be able to connect to it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3420) Invalid ERROR message initiates false alarms
[ https://issues.apache.org/jira/browse/SQOOP-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758118#comment-16758118 ] Szabolcs Vasas commented on SQOOP-3420: --- [~dionusos] Yes, don't worry about the Oracle test failures that happens because of a test infra problem we need to address separately. I have approved the request, if no other reviews will come in I will merge it by the end of today. > Invalid ERROR message initiates false alarms > > > Key: SQOOP-3420 > URL: https://issues.apache.org/jira/browse/SQOOP-3420 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Denes Bodo >Priority: Critical > Labels: usability > Attachments: SQOOP-3420_001.patch > > > In SQOOP-3042, a debug message was refactored to be error instead means false > alarms in customer log analyser. After understanding the functionality it is > recommended to use info level message instead of error in case when > ImportTool is unable to backup generated .java file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3422) Close database connection in Sqoop CLI before starting import/export job
Szabolcs Vasas created SQOOP-3422: - Summary: Close database connection in Sqoop CLI before starting import/export job Key: SQOOP-3422 URL: https://issues.apache.org/jira/browse/SQOOP-3422 Project: Sqoop Issue Type: Improvement Affects Versions: 1.4.7 Reporter: Szabolcs Vasas Sqoop does not close the open database connection before submitting the MR job but in most of the cases (except Hive import) it does not need it after the job finishes. The open connection wastes resources and can cause misleading error messages in the log if it times out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3420) Invalid ERROR message initiates false alarms
[ https://issues.apache.org/jira/browse/SQOOP-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757102#comment-16757102 ] Szabolcs Vasas commented on SQOOP-3420: --- Hi [~dionusos], It looks good, I am also OK with the WARN log level, please create a pull request with the change and we will ask [~ericlin] to review it as well. > Invalid ERROR message initiates false alarms > > > Key: SQOOP-3420 > URL: https://issues.apache.org/jira/browse/SQOOP-3420 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Denes Bodo >Priority: Critical > Labels: usability > Attachments: SQOOP-3420_001.patch > > > In SQOOP-3042, a debug message was refactored to be error instead means false > alarms in customer log analyser. After understanding the functionality it is > recommended to use info level message instead of error in case when > ImportTool is unable to backup generated .java file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3421) Importing data from Oracle to Parquet as incremental dataset name fails
[
https://issues.apache.org/jira/browse/SQOOP-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747986#comment-16747986
]
Szabolcs Vasas commented on SQOOP-3421:
---
Hi [~dmateusp],
You have encountered a Kite limitation here. The problem is that since the
table name is specified in SOME_SCHEMA.SOME_TABLE_NAME form Kite tries to
create a dataset with that name but '.' is not permitted in Kite dataset names.
The reason you get this error with Parquet file format only is that Kite was
only used for Parquet reading/writing.
Kite dependency has been removed from Sqoop a couple of months ago so this
issue is resolved in the latest trunk but unfortunately we do not have any
releases yet which contain the fix.
Btw s3n file system is not deprecated you might want to use s3a in the future.
Regards,
Szabolcs
> Importing data from Oracle to Parquet as incremental dataset name fails
> ---
>
> Key: SQOOP-3421
> URL: https://issues.apache.org/jira/browse/SQOOP-3421
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Daniel Mateus Pires
>Priority: Minor
>
> Hi there, I'm trying to run the following to import an Oracle table into S3
> as Parquet:
> {code:bash}
> sqoop import --connect jdbc:oracle:thin:@//some.host:1521/ORCL
> --where="rownum < 100" --table SOME_SCHEMA.SOME_TABLE_NAME --password
> some_password --username some_username --num-mappers 4 --split-by PRD_ID
> --target-dir s3n://bucket/destination --temporary-rootdir
> s3n://bucket/temp/destination --compress --check-column PRD_MODIFY_DT
> --incremental lastmodified --map-column-java PRD_ATTR_TEXT=String --append
> {code}
> Version of Kite is: kite-data-s3-1.1.0.jar
> Version of Sqoop is: 1.4.7
> And I'm getting the following error:
> {code:text}
> 19/01/21 13:20:33 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM SOME_SCHEMA.SOME_TABLE_NAME t WHERE 1=0
> 19/01/21 13:20:34 INFO conf.HiveConf: Found configuration file
> file:/etc/hive/conf.dist/hive-site.xml
> 19/01/21 13:20:35 ERROR sqoop.Sqoop: Got exception running Sqoop:
> org.kitesdk.data.ValidationException: Dataset name
> 47a2cf963b82475d8eba78c822403204_SOME_SCHEMA.SOME_TABLE_NAME is not
> alphanumeric (plus '_')
> org.kitesdk.data.ValidationException: Dataset name
> 47a2cf963b82475d8eba78c822403204_SOME_SCHEMA.SOME_TABLE_NAME is not
> alphanumeric (plus '_')
> at
> org.kitesdk.data.ValidationException.check(ValidationException.java:55)
> at
> org.kitesdk.data.spi.Compatibility.checkDatasetName(Compatibility.java:105)
> at org.kitesdk.data.spi.Compatibility.check(Compatibility.java:68)
> at
> org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.create(FileSystemMetadataProvider.java:209)
> at
> org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.create(FileSystemDatasetRepository.java:137)
> at org.kitesdk.data.Datasets.create(Datasets.java:239)
> at org.kitesdk.data.Datasets.create(Datasets.java:307)
> at
> org.apache.sqoop.mapreduce.ParquetJob.createDataset(ParquetJob.java:156)
> at
> org.apache.sqoop.mapreduce.ParquetJob.configureImportJob(ParquetJob.java:130)
> at
> org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapper(DataDrivenImportJob.java:132)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:264)
> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
> at
> org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:454)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> {code}
> Importing as text file instead solves the issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically
[
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746220#comment-16746220
]
Szabolcs Vasas commented on SQOOP-3042:
---
Hi [~vasubramanian],
This makes complete sense, I am happy to help committing a patch to restore the
original logging behavior, [~dionusos] do you want to prepare a patch for this?
[~ericlin], feel free to chime in if you have anything to add.
Thanks,
Szabolcs
> Sqoop does not clear compile directory under /tmp/sqoop-/compile
> automatically
>
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
> Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch,
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch,
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left
> behind on disk, so anyone can check those JAVA files to see the schema of
> those tables that Sqoop has been interacting with. By default, the directory
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> }
> if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
> }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SQOOP-3419) Refactor S3 tests for easier extensibility
[ https://issues.apache.org/jira/browse/SQOOP-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3419: - Assignee: Szabolcs Vasas > Refactor S3 tests for easier extensibility > -- > > Key: SQOOP-3419 > URL: https://issues.apache.org/jira/browse/SQOOP-3419 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > It would be great to add Azure cloud storage tests similar to the S3 tests we > have currently but Azure support is implemented in Hadoop 3.x modules only. > However as a first step we could refactor our existing tests to enable better > extensibility in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3419) Refactor S3 tests for easier extensibility
Szabolcs Vasas created SQOOP-3419: - Summary: Refactor S3 tests for easier extensibility Key: SQOOP-3419 URL: https://issues.apache.org/jira/browse/SQOOP-3419 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas It would be great to add Azure cloud storage tests similar to the S3 tests we have currently but Azure support is implemented in Hadoop 3.x modules only. However as a first step we could refactor our existing tests to enable better extensibility in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically
[
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740394#comment-16740394
]
Szabolcs Vasas commented on SQOOP-3042:
---
The error means that the generated source file cannot be moved to the code
output directory, which is the current directory by default but can be
overridden with the --outdir option.
So depending on the use case it could be a real error because I can imagine in
some cases the user needs these generated files but in most of the cases they
care about the import only which is not impacted by this error.
So we could argue for both keeping this behavior and restore the original one
but I think if you explain the impact of the error to the customer they should
be ok.
> Sqoop does not clear compile directory under /tmp/sqoop-/compile
> automatically
>
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
> Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch,
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch,
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left
> behind on disk, so anyone can check those JAVA files to see the schema of
> those tables that Sqoop has been interacting with. By default, the directory
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> }
> if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
> }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3042) Sqoop does not clear compile directory under /tmp/sqoop-/compile automatically
[
https://issues.apache.org/jira/browse/SQOOP-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739556#comment-16739556
]
Szabolcs Vasas commented on SQOOP-3042:
---
Hi [~dionusos],
I am not sure it was intentional, does it cause any problem on your side? If
yes, I think we can submit a patch to restore the original behavior.
> Sqoop does not clear compile directory under /tmp/sqoop-/compile
> automatically
>
>
> Key: SQOOP-3042
> URL: https://issues.apache.org/jira/browse/SQOOP-3042
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Critical
> Labels: patch
> Fix For: 3.0.0
>
> Attachments: SQOOP-3042.1.patch, SQOOP-3042.2.patch,
> SQOOP-3042.4.patch, SQOOP-3042.5.patch, SQOOP-3042.6.patch,
> SQOOP-3042.7.patch, SQOOP-3042.9.patch
>
>
> After running sqoop, all the temp files generated by ClassWriter are left
> behind on disk, so anyone can check those JAVA files to see the schema of
> those tables that Sqoop has been interacting with. By default, the directory
> is under /tmp/sqoop-/compile.
> In class org.apache.sqoop.SqoopOptions, function getNonceJarDir(), I can see
> that we did add "deleteOnExit" on the temp dir:
> {code}
> for (int attempts = 0; attempts < MAX_DIR_CREATE_ATTEMPTS; attempts++) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> while (hashDir.exists()) {
> hashDir = new File(baseDir, RandomHash.generateMD5String());
> }
> if (hashDir.mkdirs()) {
> // We created the directory. Use it.
> // If this directory is not actually filled with files, delete it
> // when the JVM quits.
> hashDir.deleteOnExit();
> break;
> }
> }
> {code}
> However, I believe it failed to delete due to directory is not empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[ https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3396. --- Resolution: Fixed Fix Version/s: 3.0.0 1.5.0 Hi [~fero], Your patch is now committed, thank you for your contribution! > Add parquet numeric support for Parquet in Hive import > -- > > Key: SQOOP-3396 > URL: https://issues.apache.org/jira/browse/SQOOP-3396 > Project: Sqoop > Issue Type: Sub-task >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 1.5.0, 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3149) Sqoop incremental import - NULL column updates are not pulled into HBase table
[ https://issues.apache.org/jira/browse/SQOOP-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709729#comment-16709729 ] Szabolcs Vasas commented on SQOOP-3149: --- Hi [~anjaiahspr], I think you have encountered the issue reported in this JIRA: https://issues.apache.org/jira/browse/SQOOP-3267 It is not yet released upstream but might have included in an HDP release, you could check with your account manager. Regards, Szabolcs > Sqoop incremental import - NULL column updates are not pulled into HBase > table > --- > > Key: SQOOP-3149 > URL: https://issues.apache.org/jira/browse/SQOOP-3149 > Project: Sqoop > Issue Type: Bug > Components: connectors/generic, hbase-integration >Affects Versions: 1.4.6 >Reporter: Jilani Shaik >Priority: Major > Fix For: 1.4.7 > > Attachments: hbase_delete_support_in_incremental_import > > > Sqoop incremental import data from any database to HBase, if source table's > column from a row is updated to NULL, then target HBase table still showing > the previous value for that column. > So if you do a scan on the table for that row, HBase shows the previous > values of the column. > Expected Result: Sqoop incremental import, If NULL columns are there in > source, then HBase need not store that and if it already exists need to > delete that column for a given row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3417) Execute Oracle XE tests on Travis CI
[ https://issues.apache.org/jira/browse/SQOOP-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3417: - Assignee: Szabolcs Vasas > Execute Oracle XE tests on Travis CI > > > Key: SQOOP-3417 > URL: https://issues.apache.org/jira/browse/SQOOP-3417 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > The task is to enable the Travis CI to execute Oracle XE tests too > automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3417) Execute Oracle XE tests on Travis CI
Szabolcs Vasas created SQOOP-3417: - Summary: Execute Oracle XE tests on Travis CI Key: SQOOP-3417 URL: https://issues.apache.org/jira/browse/SQOOP-3417 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas The task is to enable the Travis CI to execute Oracle XE tests too automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3393) TestNetezzaExternalTableExportMapper hangs
[ https://issues.apache.org/jira/browse/SQOOP-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3393. --- Resolution: Fixed Hi Daniel, Your patch is now committed, thank you for your contribution! > TestNetezzaExternalTableExportMapper hangs > -- > > Key: SQOOP-3393 > URL: https://issues.apache.org/jira/browse/SQOOP-3393 > Project: Sqoop > Issue Type: Bug > Components: test >Affects Versions: 1.5.0, 3.0.0 >Reporter: Daniel Voros >Assignee: Daniel Voros >Priority: Major > Fix For: 1.5.0, 3.0.0 > > > Introduced in SQOOP-3378, spotted by [~vasas]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3415) Fix gradle test+build when clean applied as the first command + warning issue fixes
[ https://issues.apache.org/jira/browse/SQOOP-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3415. --- Resolution: Fixed Hi Attila, Your patch is now committed, thank you for your contribution! > Fix gradle test+build when clean applied as the first command + warning issue > fixes > --- > > Key: SQOOP-3415 > URL: https://issues.apache.org/jira/browse/SQOOP-3415 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Attila Szabo >Assignee: Attila Szabo >Priority: Major > Fix For: 1.5.0 > > > If the user wants to build like the following command: > gradlew clean unittest > the gradle process ends up in an exception and the whole process left there > hanging forever. The root cause of this is the following: > tasks.withType runs in the configuration part of the build, where we ensure > the neccessary directories exist. > after that clean is executed and all of the dirs got deleted. > Proposed fix: > Apply directory creation as the first step of test tasks. > on the top: > there are some missing options b/c of Junit annotation processors, and also > Xlint information are swallowed currently. We aim to fix these things as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks
[ https://issues.apache.org/jira/browse/SQOOP-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700691#comment-16700691 ] Szabolcs Vasas commented on SQOOP-3414: --- I attached 2 txt files to demonstrate the behavior of the new flag during a test failure. To produce a test failure I created an always failing test case(SpecialTest) in my working directory which is not present in the patch. > Introduce a Gradle build parameter to set the ignoreTestFailures of the test > tasks > -- > > Key: SQOOP-3414 > URL: https://issues.apache.org/jira/browse/SQOOP-3414 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Attachments: test_with_ignoreTestFailures=true.txt, > test_without_ignoreTestFailures.txt > > > The > [ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures] > parameter of the Gradle test tasks is set to false which means that if a > Gradle test task fails the gradle > process returns with non-zero. In some CI tools (e.g. Jenkins) this will make > the status of the job red and not yellow > which usually means some more serious issue than a test failure. > I would like to introduce a parameter to be able set this parameter of the > test tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks
[ https://issues.apache.org/jira/browse/SQOOP-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3414: -- Attachment: test_without_ignoreTestFailures.txt test_with_ignoreTestFailures=true.txt > Introduce a Gradle build parameter to set the ignoreTestFailures of the test > tasks > -- > > Key: SQOOP-3414 > URL: https://issues.apache.org/jira/browse/SQOOP-3414 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Attachments: test_with_ignoreTestFailures=true.txt, > test_without_ignoreTestFailures.txt > > > The > [ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures] > parameter of the Gradle test tasks is set to false which means that if a > Gradle test task fails the gradle > process returns with non-zero. In some CI tools (e.g. Jenkins) this will make > the status of the job red and not yellow > which usually means some more serious issue than a test failure. > I would like to introduce a parameter to be able set this parameter of the > test tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks
Szabolcs Vasas created SQOOP-3414: - Summary: Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks Key: SQOOP-3414 URL: https://issues.apache.org/jira/browse/SQOOP-3414 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas The [ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures] parameter of the Gradle test tasks is set to false which means that if a Gradle test task fails the gradle process returns with non-zero. In some CI tools (e.g. Jenkins) this will make the status of the job red and not yellow which usually means some more serious issue than a test failure. I would like to introduce a parameter to be able set this parameter of the test tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3414) Introduce a Gradle build parameter to set the ignoreTestFailures of the test tasks
[ https://issues.apache.org/jira/browse/SQOOP-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3414: - Assignee: Szabolcs Vasas > Introduce a Gradle build parameter to set the ignoreTestFailures of the test > tasks > -- > > Key: SQOOP-3414 > URL: https://issues.apache.org/jira/browse/SQOOP-3414 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > The > [ignoreFailures|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:ignoreFailures] > parameter of the Gradle test tasks is set to false which means that if a > Gradle test task fails the gradle > process returns with non-zero. In some CI tools (e.g. Jenkins) this will make > the status of the job red and not yellow > which usually means some more serious issue than a test failure. > I would like to introduce a parameter to be able set this parameter of the > test tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3412) Travis CI should send notifications of failed builds
[ https://issues.apache.org/jira/browse/SQOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3412. --- Resolution: Won't Fix Based on my testing it seems that Travis CI is not able to send emails to dev@sqoop.apache.org email so I am resolving this JIRA with Won't Fix. > Travis CI should send notifications of failed builds > > > Key: SQOOP-3412 > URL: https://issues.apache.org/jira/browse/SQOOP-3412 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3413) TestMainframeManager does not restore the inner state of AccumuloUtil
Szabolcs Vasas created SQOOP-3413: - Summary: TestMainframeManager does not restore the inner state of AccumuloUtil Key: SQOOP-3413 URL: https://issues.apache.org/jira/browse/SQOOP-3413 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas org.apache.sqoop.manager.TestMainframeManager#testImportTableNoAccumuloJarPresent sets the testingMode field of AccumuloUtil to true but it does not restores it so Accumulo tests will fail if they are executed after TestMainframeManager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3413) TestMainframeManager does not restore the inner state of AccumuloUtil
[ https://issues.apache.org/jira/browse/SQOOP-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3413: - Assignee: Szabolcs Vasas > TestMainframeManager does not restore the inner state of AccumuloUtil > - > > Key: SQOOP-3413 > URL: https://issues.apache.org/jira/browse/SQOOP-3413 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > org.apache.sqoop.manager.TestMainframeManager#testImportTableNoAccumuloJarPresent > sets the testingMode field of AccumuloUtil to true but it does not restores > it so Accumulo tests will fail if they are executed after > TestMainframeManager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3412) Travis CI should send notifications of failed builds
[ https://issues.apache.org/jira/browse/SQOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3412: - Assignee: Szabolcs Vasas > Travis CI should send notifications of failed builds > > > Key: SQOOP-3412 > URL: https://issues.apache.org/jira/browse/SQOOP-3412 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3412) Travis CI should send notifications of failed builds
Szabolcs Vasas created SQOOP-3412: - Summary: Travis CI should send notifications of failed builds Key: SQOOP-3412 URL: https://issues.apache.org/jira/browse/SQOOP-3412 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3411) PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent from the database
[ https://issues.apache.org/jira/browse/SQOOP-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3411: -- Summary: PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent from the database (was: PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent in the database) > PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent > from the database > --- > > Key: SQOOP-3411 > URL: https://issues.apache.org/jira/browse/SQOOP-3411 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3411) PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent in the database
Szabolcs Vasas created SQOOP-3411: - Summary: PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent in the database Key: SQOOP-3411 URL: https://issues.apache.org/jira/browse/SQOOP-3411 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3411) PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent in the database
[ https://issues.apache.org/jira/browse/SQOOP-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3411: - Assignee: Szabolcs Vasas > PostgresMetaConnectIncrementalImportTest fails if metastore tables are absent > in the database > - > > Key: SQOOP-3411 > URL: https://issues.apache.org/jira/browse/SQOOP-3411 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3289) Add .travis.yml
[
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas updated SQOOP-3289:
--
Attachment: SQOOP-3289.patch
> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
> Issue Type: Sub-task
> Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Szabolcs Vasas
>Priority: Minor
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3289.patch, SQOOP-3289.patch
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org.
> Currently if you wish to use Travis for testing your changes, you have to
> manually add a .travis.yml to your branch. Having it committed to trunk would
> save us this extra step.
> I currently have an example
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
> on my travis branch running unit tests for every commit and every pull
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see:
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3289) Add .travis.yml
[
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas updated SQOOP-3289:
--
Issue Type: Sub-task (was: Task)
Parent: SQOOP-3228
> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
> Issue Type: Sub-task
> Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: SQOOP-3289.patch
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org.
> Currently if you wish to use Travis for testing your changes, you have to
> manually add a .travis.yml to your branch. Having it committed to trunk would
> save us this extra step.
> I currently have an example
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
> on my travis branch running unit tests for every commit and every pull
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see:
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3289) Add .travis.yml
[
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696056#comment-16696056
]
Szabolcs Vasas commented on SQOOP-3289:
---
Hi All,
This initiative was abandoned for a while, after the Gradle upgrade I have
managed to create a Travis job which can execute almost all of the third party
tests with Travis.
Please see my RB request attached: https://reviews.apache.org/r/69433/
> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
> Issue Type: Task
> Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org.
> Currently if you wish to use Travis for testing your changes, you have to
> manually add a .travis.yml to your branch. Having it committed to trunk would
> save us this extra step.
> I currently have an example
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
> on my travis branch running unit tests for every commit and every pull
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see:
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (SQOOP-3405) Refactor: break up Parameterized tests on a per database basis
[ https://issues.apache.org/jira/browse/SQOOP-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3405. --- Resolution: Fixed Fix Version/s: 3.0.0 Hi [~fero], Your patch is now committed, thank you for your contribution! > Refactor: break up Parameterized tests on a per database basis > -- > > Key: SQOOP-3405 > URL: https://issues.apache.org/jira/browse/SQOOP-3405 > Project: Sqoop > Issue Type: Sub-task >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > Follow the example of the abstract class SavedJobsTestBase and it's > subclasses! > We need this to be able to add test categories (so for Travis integration) as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests
[ https://issues.apache.org/jira/browse/SQOOP-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3408: - Assignee: Szabolcs Vasas > Introduce a Gradle build parameter to set the default forkEvery value for the > tests > --- > > Key: SQOOP-3408 > URL: https://issues.apache.org/jira/browse/SQOOP-3408 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > The [forkEvery > parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery] > of the Gradle test tasks is currently set to 0 which means that all of the > tests run in a single JVM (the only exception is the kerberizedTest task > which requires a new JVM for every test class). > The benefit of this setup is that the test tasks finish much faster since the > JVM creation is a slow operation. However the Sqoop test framework seems to > consume/leak too much memory which can lead to an OutOfMemoryError during the > build if there is not enough memory on the machine running the tests. > The goal of this JIRA is to introduce a new parameter to the Gradle build > which can be used to set the default forkEvery parameter and thus prevent the > JVM running out of memory. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests
Szabolcs Vasas created SQOOP-3408: - Summary: Introduce a Gradle build parameter to set the default forkEvery value for the tests Key: SQOOP-3408 URL: https://issues.apache.org/jira/browse/SQOOP-3408 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas The [forkEvery parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery] of the Gradle test tasks is currently set to 0 which means that all of the tests run in a single JVM (the only exception is the kerberizedTest task which requires a new JVM for every test class). The benefit of this setup is that the test tasks finish much faster since the JVM creation is a slow operation. However the Sqoop test framework seems to consume/leak too much memory which can lead to an OutOfMemoryError during the build if there is not enough memory on the machine running the tests. The goal of this JIRA is to introduce a new parameter to the Gradle build which can be used to set the default forkEvery parameter and thus prevent the JVM running out of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3407) Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields
[ https://issues.apache.org/jira/browse/SQOOP-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3407: - Assignee: Szabolcs Vasas > Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static > fields > > > Key: SQOOP-3407 > URL: https://issues.apache.org/jira/browse/SQOOP-3407 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > BaseSqoopTestCase.TEMP_BASE_DIR and BaseSqoopTestCase.LOCAL_WAREHOUSE_DIR are > public static fields which get initialized once at the JVM startup and store > the paths for the test temp and warehouse directories. > The problem is that HBase test cases change the value of the test.build.data > system property which can cause tests using these static fields to fail. > Since we do not own the code in HBase which changes the system property we > need to turn these static fields into methods which evaluate the > test.build.data system property every time they invoked which will make sure > that the invoking tests will be successful. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3407) Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields
Szabolcs Vasas created SQOOP-3407: - Summary: Introduce methods instead of TEMP_BASE_DIR and LOCAL_WAREHOUSE_DIR static fields Key: SQOOP-3407 URL: https://issues.apache.org/jira/browse/SQOOP-3407 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas BaseSqoopTestCase.TEMP_BASE_DIR and BaseSqoopTestCase.LOCAL_WAREHOUSE_DIR are public static fields which get initialized once at the JVM startup and store the paths for the test temp and warehouse directories. The problem is that HBase test cases change the value of the test.build.data system property which can cause tests using these static fields to fail. Since we do not own the code in HBase which changes the system property we need to turn these static fields into methods which evaluate the test.build.data system property every time they invoked which will make sure that the invoking tests will be successful. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3406) Sqoop should not try to execute test category interfaces as tests with Ant
Szabolcs Vasas created SQOOP-3406: - Summary: Sqoop should not try to execute test category interfaces as tests with Ant Key: SQOOP-3406 URL: https://issues.apache.org/jira/browse/SQOOP-3406 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas When Ant third party test suite is being run Ant tries to execute the test category interfaces too because they end with the 'Test' postfix. These "tests" obviously fail so we need to make sure that Ant does not execute them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3406) Sqoop should not try to execute test category interfaces as tests with Ant
[ https://issues.apache.org/jira/browse/SQOOP-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3406: - Assignee: Szabolcs Vasas > Sqoop should not try to execute test category interfaces as tests with Ant > -- > > Key: SQOOP-3406 > URL: https://issues.apache.org/jira/browse/SQOOP-3406 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > When Ant third party test suite is being run Ant tries to execute the test > category interfaces too because they end with the 'Test' postfix. > These "tests" obviously fail so we need to make sure that Ant does not > execute them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3230) Add a build step which starts RDBMS docker containers before the third party tests are executed
[ https://issues.apache.org/jira/browse/SQOOP-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3230. --- Resolution: Won't Do Resolving this ticked with 'Won't do' as it seems we will choose a different path. The containers will be started up by a script which will be invoked by the CI system. > Add a build step which starts RDBMS docker containers before the third party > tests are executed > --- > > Key: SQOOP-3230 > URL: https://issues.apache.org/jira/browse/SQOOP-3230 > Project: Sqoop > Issue Type: Sub-task >Reporter: Szabolcs Vasas >Priority: Major > > The task is to enable automatic third party test execution using our build > tool. > Ant should start up all the necessary containers before the tests are > executed and shut them down after they are completed. > The solution should address (at least) the below problems: > * Find a way to start docker containers from ant (probably a Java library) > * Random port support: we should make the solution more robust by not relying > on the default ports exposed by the containers since they may be occupied on > the host running the test cases. > * Health check logic should be implemented: the framework should make sure > that the containers are properly started before it executes the test cases. > * The solution should not be too tied to Ant since Gradle build will be > introduced soon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3404) Categorize all tests in the project
[ https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3404: -- Issue Type: Sub-task (was: Test) Parent: SQOOP-3228 > Categorize all tests in the project > --- > > Key: SQOOP-3404 > URL: https://issues.apache.org/jira/browse/SQOOP-3404 > Project: Sqoop > Issue Type: Sub-task >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Fix For: 3.0.0 > > > SQOOP-3104 has introduced test categories but while it was under review many > other patches with new test cases were committed. > The task is to make sure that all of the new tests are properly categorized > and the test tasks are cleaned up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3404) Categorize all tests in the project
[ https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3404: -- Description: SQOOP-3104 has introduced test categories but while it was under review many other patches with new test cases were committed. The task is to make sure that all of the new tests are properly categorized and the test tasks are cleaned up. was: We are currently unsing test naming conventions to differentiate between ManualTests, Unit tests and 3rd party tests. Instead of this, we should use junit4 categories, so it's clear by looking at a test where it fits. This would also remove the reliance on the test class name. > Categorize all tests in the project > --- > > Key: SQOOP-3404 > URL: https://issues.apache.org/jira/browse/SQOOP-3404 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Fix For: 3.0.0 > > > SQOOP-3104 has introduced test categories but while it was under review many > other patches with new test cases were committed. > The task is to make sure that all of the new tests are properly categorized > and the test tasks are cleaned up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3404) Categorize all tests in the project
[ https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3404: - Assignee: Szabolcs Vasas (was: Nguyen Truong) > Categorize all tests in the project > --- > > Key: SQOOP-3404 > URL: https://issues.apache.org/jira/browse/SQOOP-3404 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Fix For: 3.0.0 > > > We are currently unsing test naming conventions to differentiate between > ManualTests, Unit tests and 3rd party tests. > Instead of this, we should use junit4 categories, so it's clear by looking at > a test where it fits. This would also remove the reliance on the test class > name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3404) Categorize all tests in the project
Szabolcs Vasas created SQOOP-3404: - Summary: Categorize all tests in the project Key: SQOOP-3404 URL: https://issues.apache.org/jira/browse/SQOOP-3404 Project: Sqoop Issue Type: Sub-task Reporter: Szabolcs Vasas Assignee: Nguyen Truong Fix For: 3.0.0 We are currently unsing test naming conventions to differentiate between ManualTests, Unit tests and 3rd party tests. Instead of this, we should use junit4 categories, so it's clear by looking at a test where it fits. This would also remove the reliance on the test class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3404) Categorize all tests in the project
[ https://issues.apache.org/jira/browse/SQOOP-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3404: -- Issue Type: Test (was: Sub-task) Parent: (was: SQOOP-3228) > Categorize all tests in the project > --- > > Key: SQOOP-3404 > URL: https://issues.apache.org/jira/browse/SQOOP-3404 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Nguyen Truong >Priority: Major > Fix For: 3.0.0 > > > We are currently unsing test naming conventions to differentiate between > ManualTests, Unit tests and 3rd party tests. > Instead of this, we should use junit4 categories, so it's clear by looking at > a test where it fits. This would also remove the reliance on the test class > name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-2943) Make sqoop able to import to Parquet file format in case of HDFS encryption zones are turned on
[
https://issues.apache.org/jira/browse/SQOOP-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas resolved SQOOP-2943.
---
Resolution: Duplicate
Resolving this as this issue is solved by
https://issues.apache.org/jira/browse/SQOOP-3313
> Make sqoop able to import to Parquet file format in case of HDFS encryption
> zones are turned on
> ---
>
> Key: SQOOP-2943
> URL: https://issues.apache.org/jira/browse/SQOOP-2943
> Project: Sqoop
> Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Attila Szabo
>Priority: Major
> Fix For: 1.5.0
>
>
> If HDFS encryption zones are turned on, and the user tries to import into
> Parquet format, where the target location is in a very different encryption
> zone, than the zone of the /tmp/ location (typical use case for that
> encrypted hive warehouse directory), even the mapper jobs are executed
> successfully, and the partial results stored on the temp storage correctly,
> the MergeOutputMapper class of Kite SDK dies with an HDFS related exception
> ("can't be moved into an encryption zone").
> The problem does not appear in case of clear text output formats.
> Please make Sqoop able to solve this scenario as well!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-2949) SQL Syntax error when split-by column is of character type and min or max value has single quote inside it
[
https://issues.apache.org/jira/browse/SQOOP-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670306#comment-16670306
]
Szabolcs Vasas commented on SQOOP-2949:
---
Hi [~gireeshp], [~fero],
I have committed both the fix and the new tests, thank you for your efforts on
this!
I am resolving the JIRAs, [~fero] please resolve the RB request.
Thanks,
Szabolcs
> SQL Syntax error when split-by column is of character type and min or max
> value has single quote inside it
> --
>
> Key: SQOOP-2949
> URL: https://issues.apache.org/jira/browse/SQOOP-2949
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
> Environment: Sqoop 1.4.6
> Run on Hadoop 2.6.0
> On Ubuntu
>Reporter: Gireesh Puthumana
>Assignee: Gireesh Puthumana
>Priority: Major
> Fix For: 3.0.0
>
>
> Did a sqoop import from mysql table "emp", with split-by column "ename",
> which is a varchar(100) type.
> +Used below command:+
> sqoop import --connect jdbc:mysql://localhost/testdb --username root
> --password * --table emp --m 2 --target-dir /sqoopTest/5 --split-by ename;
> +Ename has following records:+
> | ename |
> | gireesh |
> | aavesh |
> | shiva' |
> | jamir |
> | balu|
> | santosh |
> | sameer |
> Min value is "aavesh" and max value is "shiva'" (please note the single quote
> inside max value).
> When run, it tried to execute below query in mapper 2 and failed:
> SELECT `ename`, `eid`, `deptid` FROM `emp` AS `emp` WHERE ( `ename` >=
> 'jd聯聭聪G耀' ) AND ( `ename` <= 'shiva'' )
> +Stack trace:+
> {quote}
> 2016-06-05 16:54:06,749 ERROR [main]
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
> in your SQL syntax; check the manual that corresponds to your MySQL server
> version for the right syntax to use near ''shiva'' )' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:942)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3966)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3902)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2526)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2549)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
> at
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
> at
> org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
> at
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3232) Remove Sqoop dependency on deprecated HBase APIs
[ https://issues.apache.org/jira/browse/SQOOP-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662406#comment-16662406 ] Szabolcs Vasas commented on SQOOP-3232: --- Hi [~szewinho], Yes, this improvement will be included in the next Sqoop release, I am not sure about the version number yet, it is going to be 1.5.0 or 3.0.0. We are still working on some non-trivial library upgrades (https://issues.apache.org/jira/browse/SQOOP-3305) most probably we will have to wait for an HBase release that supports Hadoop 3.x properly and the community has to find the time for preparing the release so there is no ETA yet. > Remove Sqoop dependency on deprecated HBase APIs > > > Key: SQOOP-3232 > URL: https://issues.apache.org/jira/browse/SQOOP-3232 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.6 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Fix For: 1.5.0 > > Attachments: SQOOP-3232-2.patch, SQOOP-3232.patch > > > Sqoop currently depends on pre HBase 1.0 APIs that have been deprecated and > will be removed in the HBase 2.0 release. > The task is to remove the dependency on these old APIs to make sure that the > upgrade to a newer HBase version will be easier in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided
[ https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3399: -- Attachment: SQOOP-3399.patch > TestS3ImportWithHadoopCredProvider fails if credential generator command is > not provided > > > Key: SQOOP-3399 > URL: https://issues.apache.org/jira/browse/SQOOP-3399 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Attachments: SQOOP-3399.patch > > > BeforeClass method of TestS3ImportWithHadoopCredProvider should not throw > NullPointerException when the credential generator command is not provided > since it fails the test with Gradle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided
[ https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3399: -- Description: BeforeClass method of TestS3ImportWithHadoopCredProvider should not throw NullPointerException when the credential generator command is not provided since it fails the test with Gradle. (was: The task is to fix ) > TestS3ImportWithHadoopCredProvider fails if credential generator command is > not provided > > > Key: SQOOP-3399 > URL: https://issues.apache.org/jira/browse/SQOOP-3399 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > BeforeClass method of TestS3ImportWithHadoopCredProvider should not throw > NullPointerException when the credential generator command is not provided > since it fails the test with Gradle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided
[ https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3399: -- Summary: TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided (was: TestS3ImportWithHadoopCredProvider fails if credential generator coma is not provided) > TestS3ImportWithHadoopCredProvider fails if credential generator command is > not provided > > > Key: SQOOP-3399 > URL: https://issues.apache.org/jira/browse/SQOOP-3399 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided
[ https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3399: -- Description: The task is to fix > TestS3ImportWithHadoopCredProvider fails if credential generator command is > not provided > > > Key: SQOOP-3399 > URL: https://issues.apache.org/jira/browse/SQOOP-3399 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > The task is to fix -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator coma is not provided
Szabolcs Vasas created SQOOP-3399: - Summary: TestS3ImportWithHadoopCredProvider fails if credential generator coma is not provided Key: SQOOP-3399 URL: https://issues.apache.org/jira/browse/SQOOP-3399 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3399) TestS3ImportWithHadoopCredProvider fails if credential generator command is not provided
[ https://issues.apache.org/jira/browse/SQOOP-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3399: - Assignee: Szabolcs Vasas > TestS3ImportWithHadoopCredProvider fails if credential generator command is > not provided > > > Key: SQOOP-3399 > URL: https://issues.apache.org/jira/browse/SQOOP-3399 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3398) Tests using HiveMiniCluster can be unstable on some platforms
[ https://issues.apache.org/jira/browse/SQOOP-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3398: - Assignee: Szabolcs Vasas > Tests using HiveMiniCluster can be unstable on some platforms > - > > Key: SQOOP-3398 > URL: https://issues.apache.org/jira/browse/SQOOP-3398 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > Since the last Hive upgrade TestHiveMiniCluster fails on some platforms > because and older version of the ASM library is picked up. > The task is to exclude the older ASM library in ivy and gradle to make sure > the test passes on all platforms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3398) Tests using HiveMiniCluster can be unstable on some platforms
Szabolcs Vasas created SQOOP-3398: - Summary: Tests using HiveMiniCluster can be unstable on some platforms Key: SQOOP-3398 URL: https://issues.apache.org/jira/browse/SQOOP-3398 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas Since the last Hive upgrade TestHiveMiniCluster fails on some platforms because and older version of the ASM library is picked up. The task is to exclude the older ASM library in ivy and gradle to make sure the test passes on all platforms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3381. --- Resolution: Fixed Hi [~fero], Your patch is now committed, thank you for your contribution! Szabolcs > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3378) Error during direct Netezza import/export can interrupt process in uncontrolled ways
[
https://issues.apache.org/jira/browse/SQOOP-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653101#comment-16653101
]
Szabolcs Vasas commented on SQOOP-3378:
---
Thanks [~dvoros]!
I have also tried to debug and identify the issue, but I did not succeed,
however since then both of the tests consistently fail on my Mac too...
> Error during direct Netezza import/export can interrupt process in
> uncontrolled ways
>
>
> Key: SQOOP-3378
> URL: https://issues.apache.org/jira/browse/SQOOP-3378
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3378.2.patch
>
>
> SQLException during JDBC operation in direct Netezza import/export signals
> parent thread to fail fast by interrupting it (see
> [here|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java#L92]).
> We're [trying to process the interrupt in the
> parent|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java#L232]
> (main) thread, but there's no guarantee that we're not in some blocking
> internal call that will process the interrupted flag and reset it before
> we're able to check.
> It is also possible that the parent thread has passed the "checking part"
> when it gets interrupted. In case of {{NetezzaExternalTableExportMapper}}
> this can interrupt the upload of log files.
> I'd recommend using some other means of communication between the threads
> than interrupts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3378) Error during direct Netezza import/export can interrupt process in uncontrolled ways
[
https://issues.apache.org/jira/browse/SQOOP-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649977#comment-16649977
]
Szabolcs Vasas commented on SQOOP-3378:
---
Hi [~dvoros],
It seems TestNetezzaExternalTableExportMapper is still failing on the Apache
Jenkins server and I have also managed to reproduce the issue locally by
running the unit tests in 1 JVM.
Do you have any idea what can cause this hanging issue?
I have seen "Need to open FIFO for reading, otherwise writing would hang"
comment in the patch so I thought you might have encountered similar thing
during the development.
The branch I used to reproduce the issue is this:
[https://github.com/szvasas/sqoop/commits/SQOOP-3104]
If you execute ./gradlew unitTest then it would hang when executing
TestNetezzaExternalTableExportMapper.
> Error during direct Netezza import/export can interrupt process in
> uncontrolled ways
>
>
> Key: SQOOP-3378
> URL: https://issues.apache.org/jira/browse/SQOOP-3378
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3378.2.patch
>
>
> SQLException during JDBC operation in direct Netezza import/export signals
> parent thread to fail fast by interrupting it (see
> [here|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java#L92]).
> We're [trying to process the interrupt in the
> parent|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java#L232]
> (main) thread, but there's no guarantee that we're not in some blocking
> internal call that will process the interrupted flag and reset it before
> we're able to check.
> It is also possible that the parent thread has passed the "checking part"
> when it gets interrupted. In case of {{NetezzaExternalTableExportMapper}}
> this can interrupt the upload of log files.
> I'd recommend using some other means of communication between the threads
> than interrupts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3378) Error during direct Netezza import/export can interrupt process in uncontrolled ways
[
https://issues.apache.org/jira/browse/SQOOP-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646265#comment-16646265
]
Szabolcs Vasas commented on SQOOP-3378:
---
It seems one of the new tests times out in the Jenkins job which seems really
strange.
I have successfully ran this test with ant on both Mac OS and Ubuntu and worked
fine.
This could an infrastructure issue as well, I will check it again later.
> Error during direct Netezza import/export can interrupt process in
> uncontrolled ways
>
>
> Key: SQOOP-3378
> URL: https://issues.apache.org/jira/browse/SQOOP-3378
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3378.2.patch
>
>
> SQLException during JDBC operation in direct Netezza import/export signals
> parent thread to fail fast by interrupting it (see
> [here|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java#L92]).
> We're [trying to process the interrupt in the
> parent|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java#L232]
> (main) thread, but there's no guarantee that we're not in some blocking
> internal call that will process the interrupted flag and reset it before
> we're able to check.
> It is also possible that the parent thread has passed the "checking part"
> when it gets interrupted. In case of {{NetezzaExternalTableExportMapper}}
> this can interrupt the upload of log files.
> I'd recommend using some other means of communication between the threads
> than interrupts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (SQOOP-3327) Mainframe FTP needs to Include "Migrated" datasets when parsing the FTP list
[ https://issues.apache.org/jira/browse/SQOOP-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3327. --- Resolution: Fixed Fix Version/s: 3.0.0 Hi [~chris.t...@gmail.com], Your patch is now committed, thank you for your contribution! Please resolve your review board request and upload the patch file to this Jira too. Thanks and regards, Szabolcs > Mainframe FTP needs to Include "Migrated" datasets when parsing the FTP list > > > Key: SQOOP-3327 > URL: https://issues.apache.org/jira/browse/SQOOP-3327 > Project: Sqoop > Issue Type: Improvement >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > Fix For: 3.0.0 > > > Need to Include "Migrated" datasets when parsing the FTP list. > > ** This applies to sequential datasets as well as GDG members ** > > Identifying migrated datasets – when performing manual FTP > > ftp> open abc.def.ghi.jkl.mno > Connected to abc.def.ghi.jkl.mno (11.22.33.444). > 220-TCPFTP01 Some FTP Server at abc.def.ghi.jkl.mno, 22:34:11 on 2018-01-22. > 220 Connection will close if idle for more than 10 minutes. > Name (abc.def.ghi.jkl.mno:some_user): some_user > 331 Send password please. > Password: > 230 some_user is logged on. Working directory is "some_user.". > Remote system type is MVS. > ftp> dir > 227 Entering Passive Mode (33,44,555,66,7,8) > 125 List started OK > Volume Unit Referred Ext Used Recfm Lrecl BlkSz Dsorg Dsname > Migrated DEV.DATA > Migrated DUMMY.DATA > OVR343 3390 2018/01/23 1 1 FB 132 27984 PS EMPTY > Migrated JCL.CNTL > OVR346 3390 2018/01/22 1 1 FB 80 27920 PS MIXED.FB80 > Migrated PLAIN.FB80 > OVR341 3390 2018/01/23 1 9 VA 125 129 PS PRDA.SPFLOG1.LIST > G20427 Tape > UNLOAD.ABCDE.ZZ9UYT.FB.TAPE > SEM352 3390 2018/01/23 1 1 FB 150 1500 PS USER.BRODCAST > OVR346 3390 2018/01/23 3 3 FB 80 6160 PO USER.ISPPROF > 250 List completed successfully. > > "Migrated" should be included as one of the regex pattern searches. > Assuming space delimited, first column will be "Migrated", and the second > (and final) column will contain the dataset name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3326) Mainframe FTP listing for GDG should filter out non-GDG datasets in a heterogeneous listing
[
https://issues.apache.org/jira/browse/SQOOP-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas resolved SQOOP-3326.
---
Resolution: Fixed
Fix Version/s: 3.0.0
Hi [~chris.t...@gmail.com],
Your patch is not committed, thank you for your contribution!
Please upload the patch file to this Jira as well and close the review board
request.
Thanks and regards,
Szabolcs
> Mainframe FTP listing for GDG should filter out non-GDG datasets in a
> heterogeneous listing
> ---
>
> Key: SQOOP-3326
> URL: https://issues.apache.org/jira/browse/SQOOP-3326
> Project: Sqoop
> Issue Type: Improvement
>Reporter: Chris Teoh
>Assignee: Chris Teoh
>Priority: Minor
> Fix For: 3.0.0
>
>
> The FTP listing will automatically assume the first file in the listing is
> the most recent GDG file. This is a problem when there are mixed datasets in
> the listing that the FTP listing doesn't filter these out.
>
> GDG base is : HLQ.ABC.DEF.AB15HUP
>
> The sequential dataset in the middle of the GDG member listing is :
> HLQ.ABC.DEF.AB15HUP.DATA
>
> The pattern for listing GDG members should be : < name>>.G\d\{4}V\d\{2}
>
> Sample below:-
> {{ Menu Options View Utilities Compilers Help
>
>
> ss
> DSLIST - Data Sets Matching HLQ.ABC.DEF.GDGBASE Row 1 of 8
> Command ===> Scroll ===>
> PAGE
>
>
> Command - Enter "/" to select action Message
> Volume
>
> ---
> HLQ.ABC.DEF.GDGBASE ??
> HLQ.ABC.DEF.GDGBASE.DUMMYSHT331
> HLQ.ABC.DEF.GDGBASE.G0034V00 H19761
> HLQ.ABC.DEF.GDGBASE.G0035V00 H81751
> HLQ.ABC.DEF.GDGBASE.G0035V00.COPYSHT337
> HLQ.ABC.DEF.GDGBASE.G0036V00 H73545
> HLQ.ABC.DEF.GDGBASE.G0037V00 G10987
> HLQ.ABC.DEF.GDGBASE.HELLOSHT33A
> * End of Data Set list
>
> ftp> open some.machine.network.zxc.au
> Connected to some.machine.network.zxc.au (11.22.33.44).
> 220-TCPFTP01 IBM FTP CS V2R1 at some.machine.network.zxc.au, 00:12:29 on
> 2018-05-29.
> 220 Connection will close if idle for more than 10 minutes.
> Name (some.machine.network.zxc.au:someuser):
> 331 Send password please.
> Password:
> 230 someuser is logged on. Working directory is "someuser.".
> Remote system type is MVS.
> ftp> cd 'HLQ.ABC.DEF.GDGBASE'
> 250 "HLQ.ABC.DEF.GDGBASE." is the working directory name prefix.
> ftp> dir
> 227 Entering Passive Mode (11,22,33,44,55,66)
> 125 List started OK
> Volume UnitReferred Ext Used Recfm Lrecl BlkSz Dsorg Dsname
> H19761 Tape G0034V00
> H81751 Tape G0035V00
> H73545 Tape G0036V00
> G10987 Tape G0037V00
> SHT331 3390 **NONE**1 15 VB 114 27998 PS DUMMY
> SHT337 3390 **NONE**1 15 VB 114 27998 PS G0035V00.COPY
> SHT33A 3390 **NONE**1 15 VB 114 27998 PS HELLO
> 250 List completed successfully.
> ftp>}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SQOOP-3375) HiveMiniCluster does not restore hive-site.xml location
Szabolcs Vasas created SQOOP-3375: - Summary: HiveMiniCluster does not restore hive-site.xml location Key: SQOOP-3375 URL: https://issues.apache.org/jira/browse/SQOOP-3375 Project: Sqoop Issue Type: Sub-task Reporter: Szabolcs Vasas HiveMiniCluster sets the hive-site.xml location using org.apache.hadoop.hive.conf.HiveConf#setHiveSiteLocation static method during startup but it does not restore the original location during shutdown. This makes HCatalogImportTest and HCatalogExportTest fail if they are ran in the same JVM after any test using HiveMiniCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3375) HiveMiniCluster does not restore hive-site.xml location
[ https://issues.apache.org/jira/browse/SQOOP-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3375: - Assignee: Szabolcs Vasas > HiveMiniCluster does not restore hive-site.xml location > --- > > Key: SQOOP-3375 > URL: https://issues.apache.org/jira/browse/SQOOP-3375 > Project: Sqoop > Issue Type: Sub-task >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > HiveMiniCluster sets the hive-site.xml location using > org.apache.hadoop.hive.conf.HiveConf#setHiveSiteLocation static method during > startup but it does not restore the original location during shutdown. > This makes HCatalogImportTest and HCatalogExportTest fail if they are ran in > the same JVM after any test using HiveMiniCluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3365) Improve Sqoop tests to be able to execute them in a single JVM
[ https://issues.apache.org/jira/browse/SQOOP-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3365. --- Resolution: Fixed > Improve Sqoop tests to be able to execute them in a single JVM > -- > > Key: SQOOP-3365 > URL: https://issues.apache.org/jira/browse/SQOOP-3365 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Priority: Major > > Gradle currently creates a new JVM for every test case it executes (the > forkEvery parameter is set to 1, see: > https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery). > This provides better isolation for the tests but it has huge performance > overhead as well since creating a JVM is an expensive operation. It would be > great if we could execute all our tests in a single JVM so we could save that > big cost but unfortunately it does not work out of the box because: > * There are test classes which unnecessarily change the state of the JVM > (for example by setting static fields and system properties) and can cause > other tests to fail if they are run in the same JVM. > * There are test classes which rely on a "clean JVM state" and they fail if > the state is different than implicitly expected by them. > * There are test classes which create a Kerberos KDC which sets a lot of > static fields in the JVM which are really hard if not impossible to restore. > This JIRA addresses the first two of the above issues. > I expect the third category to be addressed by SQOOP-3104. It should > introduce a category for kerberized tests and Gradle should execute this > category with the forkEvery parameter set to 1. > However Gradle should be able to execute the rest of the tests with forkEvery > 0. > Please note that the test execution order with Gradle is not strictly defined > but platform dependent so it is possible that running the tests on a new > platform will produce an execution order which will unveil a test > interdependency which is not resolved by the subtasks of the JIRA. > If you encounter such a situation please feel free to create a new subtasks > here. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3371) Fix tests using HiveMiniCluster
[ https://issues.apache.org/jira/browse/SQOOP-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3371: - Assignee: Szabolcs Vasas > Fix tests using HiveMiniCluster > --- > > Key: SQOOP-3371 > URL: https://issues.apache.org/jira/browse/SQOOP-3371 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > It seems that SQOOP-3360 broke our tests which use HiveMiniCluster because > org.apache.calcite is not present in the hive-exec:core JAR but this > dependency seems to be needed by these tests. > I am not sure why our Jenkins job did not catch the issue earlier but I get > consistent failures when I run these tests with a clean ivy cache and ant so > the dependency issue needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3371) Fix tests using HiveMiniCluster
Szabolcs Vasas created SQOOP-3371: - Summary: Fix tests using HiveMiniCluster Key: SQOOP-3371 URL: https://issues.apache.org/jira/browse/SQOOP-3371 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas It seems that SQOOP-3360 broke our tests which use HiveMiniCluster because org.apache.calcite is not present in the hive-exec:core JAR but this dependency seems to be needed by these tests. I am not sure why our Jenkins job did not catch the issue earlier but I get consistent failures when I run these tests with a clean ivy cache and ant so the dependency issue needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SQOOP-3224) Mainframe FTP transfer should have an option to use binary mode for transfer
[ https://issues.apache.org/jira/browse/SQOOP-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590360#comment-16590360 ] Szabolcs Vasas edited comment on SQOOP-3224 at 8/23/18 3:10 PM: Hi [~chris.t...@gmail.com], Your patch is committed, thank you for your contribution! Please close the review board request and attach the final patch file to this JIRA. Thanks and regards, Szabolcs was (Author: vasas): Hi [~chris.t...@gmail.com], Your patch is committed, thank you for your contribution! Please close the review board request. Regards, Szabolcs > Mainframe FTP transfer should have an option to use binary mode for transfer > > > Key: SQOOP-3224 > URL: https://issues.apache.org/jira/browse/SQOOP-3224 > Project: Sqoop > Issue Type: Improvement >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > Fix For: 3.0.0 > > > Currently the mainframe FTP module is hard coded to use ascii transfer mode. > Propose a mainframe module flag to be able to change modes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3366) Improve unit tests to be able to execute them in a single JVM
[ https://issues.apache.org/jira/browse/SQOOP-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3366: - Assignee: Nguyen Truong > Improve unit tests to be able to execute them in a single JVM > - > > Key: SQOOP-3366 > URL: https://issues.apache.org/jira/browse/SQOOP-3366 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Nguyen Truong >Priority: Major > > The goal of this JIRA is to improve the unit tests to be able to execute them > in a single JVM. See the parent JIRA for the details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3367) Improve third party tests to be able to execute them in a single JVM
[ https://issues.apache.org/jira/browse/SQOOP-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3367: - Assignee: Szabolcs Vasas > Improve third party tests to be able to execute them in a single JVM > > > Key: SQOOP-3367 > URL: https://issues.apache.org/jira/browse/SQOOP-3367 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > The goal of this JIRA is to improve the third party tests to be able to > execute them in a single JVM. See the parent JIRA for the details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3367) Improve third party tests to be able to execute them in a single JVM
[ https://issues.apache.org/jira/browse/SQOOP-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3367: -- Description: The goal of this JIRA is to improve the third party tests to be able to execute them in a single JVM. See the parent JIRA for the details. > Improve third party tests to be able to execute them in a single JVM > > > Key: SQOOP-3367 > URL: https://issues.apache.org/jira/browse/SQOOP-3367 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Priority: Major > > The goal of this JIRA is to improve the third party tests to be able to > execute them in a single JVM. See the parent JIRA for the details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3367) Improve third party tests to be able to execute them in a single JVM
Szabolcs Vasas created SQOOP-3367: - Summary: Improve third party tests to be able to execute them in a single JVM Key: SQOOP-3367 URL: https://issues.apache.org/jira/browse/SQOOP-3367 Project: Sqoop Issue Type: Sub-task Affects Versions: 1.4.7 Reporter: Szabolcs Vasas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3366) Improve unit tests to be able to execute them in a single JVM
Szabolcs Vasas created SQOOP-3366: - Summary: Improve unit tests to be able to execute them in a single JVM Key: SQOOP-3366 URL: https://issues.apache.org/jira/browse/SQOOP-3366 Project: Sqoop Issue Type: Sub-task Affects Versions: 1.4.7 Reporter: Szabolcs Vasas The goal of this JIRA is to improve the unit tests to be able to execute them in a single JVM. See the parent JIRA for the details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3365) Improve Sqoop tests to be able to execute them in a single JVM
Szabolcs Vasas created SQOOP-3365: - Summary: Improve Sqoop tests to be able to execute them in a single JVM Key: SQOOP-3365 URL: https://issues.apache.org/jira/browse/SQOOP-3365 Project: Sqoop Issue Type: Improvement Affects Versions: 1.4.7 Reporter: Szabolcs Vasas Gradle currently creates a new JVM for every test case it executes (the forkEvery parameter is set to 1, see: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery). This provides better isolation for the tests but it has huge performance overhead as well since creating a JVM is an expensive operation. It would be great if we could execute all our tests in a single JVM so we could save that big cost but unfortunately it does not work out of the box because: * There are test classes which unnecessarily change the state of the JVM (for example by setting static fields and system properties) and can cause other tests to fail if they are run in the same JVM. * There are test classes which rely on a "clean JVM state" and they fail if the state is different than implicitly expected by them. * There are test classes which create a Kerberos KDC which sets a lot of static fields in the JVM which are really hard if not impossible to restore. This JIRA addresses the first two of the above issues. I expect the third category to be addressed by SQOOP-3104. It should introduce a category for kerberized tests and Gradle should execute this category with the forkEvery parameter set to 1. However Gradle should be able to execute the rest of the tests with forkEvery 0. Please note that the test execution order with Gradle is not strictly defined but platform dependent so it is possible that running the tests on a new platform will produce an execution order which will unveil a test interdependency which is not resolved by the subtasks of the JIRA. If you encounter such a situation please feel free to create a new subtasks here. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3364) Upgrade Gradle version to 4.9
[ https://issues.apache.org/jira/browse/SQOOP-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3364: -- Attachment: SQOOP-3364.patch > Upgrade Gradle version to 4.9 > - > > Key: SQOOP-3364 > URL: https://issues.apache.org/jira/browse/SQOOP-3364 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Attachments: SQOOP-3364.patch > > > Sqoop uses Gradle 3.5.1 currently which is a pretty old version, let's > upgrade it to the newest 4.9 version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3362) Fix toString() methods of OraOopOracleDataChunk
[ https://issues.apache.org/jira/browse/SQOOP-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas resolved SQOOP-3362. --- Resolution: Fixed Fix Version/s: 3.0.0 Hi [~natalietruong83], Your patch is committed, thank you for your contribution! Please upload the patch file to the Jira as well and link the ReviewBoard. Regards, Szabolcs > Fix toString() methods of OraOopOracleDataChunk > --- > > Key: SQOOP-3362 > URL: https://issues.apache.org/jira/browse/SQOOP-3362 > Project: Sqoop > Issue Type: Bug >Reporter: Nguyen Truong >Assignee: Nguyen Truong >Priority: Major > Fix For: 3.0.0 > > > The method is currently returning the hash of data chunk object. It would be > nice to show the information of the class's variables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3364) Upgrade Gradle version to 4.9
[ https://issues.apache.org/jira/browse/SQOOP-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3364: - Assignee: Szabolcs Vasas > Upgrade Gradle version to 4.9 > - > > Key: SQOOP-3364 > URL: https://issues.apache.org/jira/browse/SQOOP-3364 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > Sqoop uses Gradle 3.5.1 currently which is a pretty old version, let's > upgrade it to the newest 4.9 version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3364) Upgrade Gradle version to 4.9
Szabolcs Vasas created SQOOP-3364: - Summary: Upgrade Gradle version to 4.9 Key: SQOOP-3364 URL: https://issues.apache.org/jira/browse/SQOOP-3364 Project: Sqoop Issue Type: Improvement Affects Versions: 1.4.7 Reporter: Szabolcs Vasas Sqoop uses Gradle 3.5.1 currently which is a pretty old version, let's upgrade it to the newest 4.9 version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-2649) Support for importing data onto Apache Phoenix tables
[ https://issues.apache.org/jira/browse/SQOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579433#comment-16579433 ] Szabolcs Vasas commented on SQOOP-2649: --- Hi [~fwein], The patch was updated a long time ago and it does not apply currently. It should be rebased to the current trunk and most probably another review would be needed. > Support for importing data onto Apache Phoenix tables > - > > Key: SQOOP-2649 > URL: https://issues.apache.org/jira/browse/SQOOP-2649 > Project: Sqoop > Issue Type: New Feature > Components: hbase-integration >Reporter: maghamravikiran >Priority: Major > Fix For: 1.5.0 > > Attachments: phoenix_sqoop.patch, phoenix_sqoop_integration.diff > > > Support importing data to Phoenix backed HBase . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3104) Create test categories instead of test suites and naming conventions
[ https://issues.apache.org/jira/browse/SQOOP-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3104: - Assignee: Nguyen Truong > Create test categories instead of test suites and naming conventions > > > Key: SQOOP-3104 > URL: https://issues.apache.org/jira/browse/SQOOP-3104 > Project: Sqoop > Issue Type: Sub-task >Reporter: Anna Szonyi >Assignee: Nguyen Truong >Priority: Major > > We are currently unsing test naming conventions to differentiate between > ManualTests, Unit tests and 3rd party tests. > Instead of this, we should use junit4 categories, so it's clear by looking at > a test where it fits. This would also remove the reliance on the test class > name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SQOOP-3360) Fix hive-exec dependency issues in Gradle
[ https://issues.apache.org/jira/browse/SQOOP-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3360: - Assignee: Szabolcs Vasas > Fix hive-exec dependency issues in Gradle > - > > Key: SQOOP-3360 > URL: https://issues.apache.org/jira/browse/SQOOP-3360 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Boglarka Egyed >Assignee: Szabolcs Vasas >Priority: Major > > It turned out during making SQOOP-3345 work with Gradle that there are issues > with the dependency handling which cause problems with Gradle. Root cause of > the problem seems to be the hive-exec dependency which also contains its own > dependencies pulling in incorrect libs and versions runtime. This has to be > cleaned up via better dependency management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3349) Remove Kite dependency from the Gradle dependencies
[ https://issues.apache.org/jira/browse/SQOOP-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3349: -- Attachment: SQOOP-3349.patch > Remove Kite dependency from the Gradle dependencies > --- > > Key: SQOOP-3349 > URL: https://issues.apache.org/jira/browse/SQOOP-3349 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > Attachments: SQOOP-3349.patch > > > Since Sqoop can be built using Gradle as well we need to make sure that the > Kite dependency is removed from sqoop-package.gradle too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3350) Fix tests which use warehouse-dir as target-dir
[
https://issues.apache.org/jira/browse/SQOOP-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas updated SQOOP-3350:
--
Attachment: SQOOP-3350.patch
> Fix tests which use warehouse-dir as target-dir
> ---
>
> Key: SQOOP-3350
> URL: https://issues.apache.org/jira/browse/SQOOP-3350
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3350.patch
>
>
> Some Sqoop tests use the value of the
> org.apache.sqoop.testutil.BaseSqoopTestCase#getWarehouseDir as target-dir:
> {code:java}
> args.add("--target-dir");
> args.add(getWarehouseDir());
> {code}
> This leads to an error when the warehouse directory exists which can happen
> if a previous test does not clean up properly.
> The issue was found when executing the tests with Gradle since it probably
> executes the tests in a different order.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3353) Sqoop should not check incremental constraints for HBase imports
[
https://issues.apache.org/jira/browse/SQOOP-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas updated SQOOP-3353:
--
Attachment: SQOOP-3353.patch
> Sqoop should not check incremental constraints for HBase imports
>
>
> Key: SQOOP-3353
> URL: https://issues.apache.org/jira/browse/SQOOP-3353
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3353.patch
>
>
> ImportTool#initIncrementalConstraints method is invoked for every import in
> Sqoop and it can throw the following error even for HBase imports:
> {code:java}
> --merge-key or --append is required when using --incremental lastmodified and
> the output directory exists.{code}
> The task is to fix the validation not to throw an exception when importing
> into an HBase table since in that case it does not matter if the table
> directory exists on HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SQOOP-3353) Sqoop should not check incremental constraints for HBase imports
Szabolcs Vasas created SQOOP-3353:
-
Summary: Sqoop should not check incremental constraints for HBase
imports
Key: SQOOP-3353
URL: https://issues.apache.org/jira/browse/SQOOP-3353
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas
ImportTool#initIncrementalConstraints method is invoked for every import in
Sqoop and it can throw the following error even for HBase imports:
{code:java}
--merge-key or --append is required when using --incremental lastmodified and
the output directory exists.{code}
The task is to fix the validation not to throw an exception when importing into
an HBase table since in that case it does not matter if the table directory
exists on HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
***UNCHECKED*** [jira] [Assigned] (SQOOP-3353) Sqoop should not check incremental constraints for HBase imports
[
https://issues.apache.org/jira/browse/SQOOP-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas reassigned SQOOP-3353:
-
Assignee: Szabolcs Vasas
> Sqoop should not check incremental constraints for HBase imports
>
>
> Key: SQOOP-3353
> URL: https://issues.apache.org/jira/browse/SQOOP-3353
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
>
> ImportTool#initIncrementalConstraints method is invoked for every import in
> Sqoop and it can throw the following error even for HBase imports:
> {code:java}
> --merge-key or --append is required when using --incremental lastmodified and
> the output directory exists.{code}
> The task is to fix the validation not to throw an exception when importing
> into an HBase table since in that case it does not matter if the table
> directory exists on HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SQOOP-3350) Fix tests which use warehouse-dir as target-dir
[
https://issues.apache.org/jira/browse/SQOOP-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szabolcs Vasas reassigned SQOOP-3350:
-
Assignee: Szabolcs Vasas
> Fix tests which use warehouse-dir as target-dir
> ---
>
> Key: SQOOP-3350
> URL: https://issues.apache.org/jira/browse/SQOOP-3350
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
>
> Some Sqoop tests use the value of the
> org.apache.sqoop.testutil.BaseSqoopTestCase#getWarehouseDir as target-dir:
> {code:java}
> args.add("--target-dir");
> args.add(getWarehouseDir());
> {code}
> This leads to an error when the warehouse directory exists which can happen
> if a previous test does not clean up properly.
> The issue was found when executing the tests with Gradle since it probably
> executes the tests in a different order.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SQOOP-3350) Fix tests which use warehouse-dir as target-dir
Szabolcs Vasas created SQOOP-3350:
-
Summary: Fix tests which use warehouse-dir as target-dir
Key: SQOOP-3350
URL: https://issues.apache.org/jira/browse/SQOOP-3350
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas
Some Sqoop tests use the value of the
org.apache.sqoop.testutil.BaseSqoopTestCase#getWarehouseDir as target-dir:
{code:java}
args.add("--target-dir");
args.add(getWarehouseDir());
{code}
This leads to an error when the warehouse directory exists which can happen if
a previous test does not clean up properly.
The issue was found when executing the tests with Gradle since it probably
executes the tests in a different order.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3052) Introduce Gradle based build for Sqoop to make it more developer friendly / open
[ https://issues.apache.org/jira/browse/SQOOP-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3052: -- Attachment: SQOOP-3052.patch > Introduce Gradle based build for Sqoop to make it more developer friendly / > open > > > Key: SQOOP-3052 > URL: https://issues.apache.org/jira/browse/SQOOP-3052 > Project: Sqoop > Issue Type: Improvement >Reporter: Attila Szabo >Assignee: Anna Szonyi >Priority: Major > Fix For: 1.5.0, 3.0.0 > > Attachments: SQOOP-3052.patch, SQOOP-3052.patch > > > The current trunk version can only be build with Ant/Ivy combination, which > has some painful limitations (resolve is slow / needs to be tweaked to use > only caches, the current profile / variable based settings are not working in > IDEs out of the box, the current solution does not download the related > sources, etc.) > It would be nice to provide a solution, which would give the possibility for > the developers to choose between the nowadays well used build infrsturctures > (e.g. Maven, Gradle, etc.). For this solution it would be also essential to > keep the different build files (if there is more then one) synchronized > easily, and the configuration wouldn't diverege by time. Test execution has > to be solved also, and should cover all the available test cases. > In this scenario: > If we can provide one good working solution is much better, then provide > three different ones which become out of sync easily. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
