[jira] [Commented] (SQOOP-2907) Export parquet files to RDBMS: don't require .metadata for parquet files
[ https://issues.apache.org/jira/browse/SQOOP-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153601#comment-16153601 ] Sergey Svinarchuk commented on SQOOP-2907: -- Could somebody say, when this issue will be resolved: Sqoop 1.4.7 or later? > Export parquet files to RDBMS: don't require .metadata for parquet files > > > Key: SQOOP-2907 > URL: https://issues.apache.org/jira/browse/SQOOP-2907 > Project: Sqoop > Issue Type: Improvement > Components: metastore >Affects Versions: 1.4.6 > Environment: sqoop 1.4.6 > export parquet files to Oracle >Reporter: Ruslan Dautkhanov >Assignee: Sandish Kumar HN > Labels: sqoop > Attachments: SQOOP-2907-3.patch, SQOOP-2907.patch, SQOOP-2907.patch1, > SQOOP-2907.patch2 > > > Kite currently requires .metadata. > Parquet files have their own metadata stored along data files. > It would be great for Export operation on parquet files to RDBMS not to > require .metadata. > We have most of the files created by Spark and Hive, and they don't create > .metadata, it only Kite that does. > It makes sqoop export of parquet files usability very limited. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SQOOP-3082) Sqoop import fails after TCP connection reset if split by datetime column
[
https://issues.apache.org/jira/browse/SQOOP-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Svinarchuk updated SQOOP-3082:
-
Attachment: SQOOP-3082.patch
> Sqoop import fails after TCP connection reset if split by datetime column
> -
>
> Key: SQOOP-3082
> URL: https://issues.apache.org/jira/browse/SQOOP-3082
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Sergey Svinarchuk
> Attachments: SQOOP-3082.patch
>
>
> If sqoop-to-mssqlserver connection reset, the whole command fails with
> "Connection reset with com.microsoft.sqlserver.jdbc.SQLServerException:
> Incorrect syntax near '00'" . On reestablishing connection, Sqoop tries to
> resume import from the last record that was successfully read by :
> {code}
> 2016-12-10 15:18:54,523 INFO [main]
> org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select * from
> test.dbo.test1 WITH (nolock) where Date >= '2015-01-10' and Date <=
> '2016-11-24' and ( Date > 2015-09-18 00:00:00.0 ) AND ( Date < '2015-09-23
> 11:48:00.0' )
> {code}
> Not quoted 2015-09-18 00:00:00.0 in SQL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SQOOP-3082) Sqoop import fails after TCP connection reset if split by datetime column
Sergey Svinarchuk created SQOOP-3082:
Summary: Sqoop import fails after TCP connection reset if split by
datetime column
Key: SQOOP-3082
URL: https://issues.apache.org/jira/browse/SQOOP-3082
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.6
Reporter: Sergey Svinarchuk
If sqoop-to-mssqlserver connection reset, the whole command fails with
"Connection reset with com.microsoft.sqlserver.jdbc.SQLServerException:
Incorrect syntax near '00'" . On reestablishing connection, Sqoop tries to
resume import from the last record that was successfully read by :
{code}
2016-12-10 15:18:54,523 INFO [main]
org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select * from
test.dbo.test1 WITH (nolock) where Date >= '2015-01-10' and Date <=
'2016-11-24' and ( Date > 2015-09-18 00:00:00.0 ) AND ( Date < '2015-09-23
11:48:00.0' )
{code}
Not quoted 2015-09-18 00:00:00.0 in SQL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SQOOP-3001) Import from Mysql broken with "Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.math.BigDecimal"
[
https://issues.apache.org/jira/browse/SQOOP-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Svinarchuk updated SQOOP-3001:
-
Description:
After patch from SQOOP-2244, Sqoop incorrect get type for column.
Example for select with automatically escape column names:
{code}
mysql> create table test (id int);
Query OK, 0 rows affected (0.01 sec)
mysql> insert into test value (1);
Query OK, 1 row affected (0.04 sec)
mysql> insert into test value (9);
Query OK, 1 row affected (0.01 sec)
mysql> select min("id"), max("id") from test;
+---+---+
| min("id") | max("id") |
+---+---+
| id | id|
+---+---+
1 row in set (0.00 sec)
{code}
But real results must be:
{code}
mysql> select min(id), max(id) from test;
+-+-+
| min(id) | max(id) |
+-+-+
| 1 |9 |
+-+-+
1 row in set (0.00 sec)
{code}
So when we use auto escape, Sqoop getting String value for int column in Mysql
was:
After patch from SQOOP-2244, Sqoop incorrect get type for column.
Example for select with automatically escape column names:
{code}
mysql> create table test (id int);
Query OK, 0 rows affected (0.01 sec)
mysql> insert into test value (1);
Query OK, 1 row affected (0.04 sec)
mysql> insert into test value (9);
Query OK, 1 row affected (0.01 sec)
mysql> select min("id"), max("id") from test;
+---+---+
| min("id") | max("id") |
+---+---+
| id| id|
+---+---+
1 row in set (0.00 sec)
{code}
But real results must be:
{code}
mysql> select min(id), max(id) from test;
+-+-+
| min(id) | max(id) |
+-+-+
| 1| 9 |
+-+-+
1 row in set (0.00 sec)
{code}
So when we use auto escape, Sqoop getting String value for int column in Mysql
> Import from Mysql broken with "Caused by: java.lang.ClassCastException:
> java.lang.Integer cannot be cast to java.math.BigDecimal"
> -
>
> Key: SQOOP-3001
> URL: https://issues.apache.org/jira/browse/SQOOP-3001
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/generic
>Affects Versions: 1.99.7
>Reporter: Sergey Svinarchuk
>
> After patch from SQOOP-2244, Sqoop incorrect get type for column.
> Example for select with automatically escape column names:
> {code}
> mysql> create table test (id int);
> Query OK, 0 rows affected (0.01 sec)
> mysql> insert into test value (1);
> Query OK, 1 row affected (0.04 sec)
> mysql> insert into test value (9);
> Query OK, 1 row affected (0.01 sec)
> mysql> select min("id"), max("id") from test;
> +---+---+
> | min("id") | max("id") |
> +---+---+
> | id | id|
> +---+---+
> 1 row in set (0.00 sec)
> {code}
> But real results must be:
> {code}
> mysql> select min(id), max(id) from test;
> +-+-+
> | min(id) | max(id) |
> +-+-+
> | 1 |9 |
> +-+-+
> 1 row in set (0.00 sec)
> {code}
> So when we use auto escape, Sqoop getting String value for int column in Mysql
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SQOOP-3001) Import from Mysql broken with "Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.math.BigDecimal"
Sergey Svinarchuk created SQOOP-3001:
Summary: Import from Mysql broken with "Caused by:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.math.BigDecimal"
Key: SQOOP-3001
URL: https://issues.apache.org/jira/browse/SQOOP-3001
Project: Sqoop
Issue Type: Bug
Components: connectors/generic
Affects Versions: 1.99.7
Reporter: Sergey Svinarchuk
After patch from SQOOP-2244, Sqoop incorrect get type for column.
Example for select with automatically escape column names:
{code}
mysql> create table test (id int);
Query OK, 0 rows affected (0.01 sec)
mysql> insert into test value (1);
Query OK, 1 row affected (0.04 sec)
mysql> insert into test value (9);
Query OK, 1 row affected (0.01 sec)
mysql> select min("id"), max("id") from test;
+---+---+
| min("id") | max("id") |
+---+---+
| id| id|
+---+---+
1 row in set (0.00 sec)
{code}
But real results must be:
{code}
mysql> select min(id), max(id) from test;
+-+-+
| min(id) | max(id) |
+-+-+
| 1| 9 |
+-+-+
1 row in set (0.00 sec)
{code}
So when we use auto escape, Sqoop getting String value for int column in Mysql
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SQOOP-2975) Sqoop loses part of data during import
[
https://issues.apache.org/jira/browse/SQOOP-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Svinarchuk updated SQOOP-2975:
-
Attachment: SQOOP-2975.patch
> Sqoop loses part of data during import
> ---
>
> Key: SQOOP-2975
> URL: https://issues.apache.org/jira/browse/SQOOP-2975
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Sergey Svinarchuk
> Attachments: SQOOP-2975.patch
>
>
> Sqoop wrong splits columns with double(float) type.
> For example:
> If we split by column that looks like this:
> {code}
> 22.8
> 29.3
> 29.9
> 39.1
> 56.7
> 66.1
> 88.8
> 88.2
> 88.2
> 95.9
> {code}
> Sqoop splits it on the next ranges:
> First range - from 22.8 to 56.7
> Second range - from 95.9 to 95.9
> And then data will be lose
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SQOOP-2975) Sqoop loses part of data during import
Sergey Svinarchuk created SQOOP-2975:
Summary: Sqoop loses part of data during import
Key: SQOOP-2975
URL: https://issues.apache.org/jira/browse/SQOOP-2975
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.6
Reporter: Sergey Svinarchuk
Sqoop wrong splits columns with double(float) type.
For example:
If we split by column that looks like this:
{code}
22.8
29.3
29.9
39.1
56.7
66.1
88.8
88.2
88.2
95.9
{code}
Sqoop splits it on the next ranges:
First range - from 22.8 to 56.7
Second range - from 95.9 to 95.9
And then data will be lose
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SQOOP-2926) Sqoop build failed with xml.etree.ElementTree.ParseError: syntax error: line 1, column 49
Sergey Svinarchuk created SQOOP-2926:
Summary: Sqoop build failed with xml.etree.ElementTree.ParseError:
syntax error: line 1, column 49
Key: SQOOP-2926
URL: https://issues.apache.org/jira/browse/SQOOP-2926
Project: Sqoop
Issue Type: Bug
Components: build
Affects Versions: 1.4.6
Reporter: Sergey Svinarchuk
Sqoop build sometimes failed during creation relnotes. This bug is randomly
occurring (one build can be successful, but next - failed). I think that it
could be issue with download xml file from https://issues.apache.org
{code}
xml = os.popen("curl -s 'https://issues.apache.org/jira/si/jira.issueviews:"; \
+ "issue-xml/%s/%s.xml?field=key&field=type&field=parent'" % (issue,
issue)).read()
{code}
This is error:
{code}
[exec] rm web/index.xml
[exec] make[1]: Leaving directory `/root/opensource/sqoop-1.4.6/src/docs'
[touch] Creating /root/opensource/sqoop-1.4.6/build/docs.timestamp
relnotes-uptodate:
relnotes:
[exec] Traceback (most recent call last):
[exec] File "/root/opensource/sqoop-1.4.6/src/scripts/relnotes.py", line
251, in
[exec] sys.exit(main(sys.argv))
[exec] File "/root/opensource/sqoop-1.4.6/src/scripts/relnotes.py", line
238, in main
[exec] jira_info = get_jira_issue_types(log)
[exec] File "/root/opensource/sqoop-1.4.6/src/scripts/relnotes.py", line
111, in get_jira_issue_types
[exec] doc = get_jira_doc(jira)
[exec] File "/root/opensource/sqoop-1.4.6/src/scripts/relnotes.py", line
88, in get_jira_doc
[exec] return ElementTree.fromstring(xml)
[exec] File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1301, in
XML
[exec] parser.feed(text)
[exec] File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1643, in
feed
[exec] self._raiseerror(v)
[exec] File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1507, in
_raiseerror
[exec] raise err
[exec] xml.etree.ElementTree.ParseError: syntax error: line 1, column 49
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SQOOP-2408) Sqoop doesnt support --as-parquetfile with -query option.
[ https://issues.apache.org/jira/browse/SQOOP-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Svinarchuk updated SQOOP-2408: - Attachment: SQOOP-2408.patch I attached patch with fix for this issue. The problem was that Avro doesn't support type 'char', but when we run import with "query" parameter, Avro wants create the schema for fields from DelimiterSet class. This bug doesn't reproduce in Unit Test, but reproduced on Hadoop cluster > Sqoop doesnt support --as-parquetfile with -query option. > - > > Key: SQOOP-2408 > URL: https://issues.apache.org/jira/browse/SQOOP-2408 > Project: Sqoop > Issue Type: Improvement > Components: codegen, sqoop2-kite-connector >Reporter: Satyajit varma >Priority: Minor > Attachments: SQOOP-2408.patch > > > Sqoop doesnt support --as-parquetfile option with -query, but works fine with > -table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
