Fwd: Release to support Hadoop 3

[Attila Szabó]

Dear Sqoop community,

Am I the only one who is missing some formal decision making, announcement
and process here?

 - When did the PMC made decision about that we're going for version 3.0
(instead of any other version alternatives)? When and where was it
announced? How is it possible that some of the contributors know this fact
earlier then the rests of the community?

 - When did Bogi become a PMC member, and why was that fact not announced
to the community as it used to be (and still not yet visible on the project
site)? Of course this is just an assumption, but according to some PMC
chair emails back this February: JIRA administrator privileges are only PMC
members, and I guess we still follow this rule, as no other information was
announced, thus I guess the reason why Bogi was able to administrate the
available versions in the JIRA means she has been lifted  (and here I'd
like to send my congrats, if this is true, and my assumption was valid).

 - When did the PMC made decision about dropping Hadoop 2 compatibility?
When and where was it announced?

If as a community which is officially part of the ASF we do have rules, why
do we look like not following them?

Regards,
Attila

ps:
Objections against dropping 1.5, is clearly the fact that having 1.5 was
decided as a community, and yet we're still not sure if those changes would
be only delivered in a next major version, how they would be backported,
cherry-picked, etc. And as the majority of the users are still on 2.x, I
think we cannot force ppl to upgrade, just to being able to use some of the
originally 1.5 planned changes.
So this item is absolutely -1 on my side!

ps2:
Daniel! I've provided some comments on your ORC Jira.

On Thu, May 10, 2018 at 4:00 PM, Boglarka Egyed  wrote:

> Hi All,
>
> Thank you Daniel for the update! I was also writing one when your email
> arrived so I'm just adding a couple of comments to that.
>
> New major version in JIRA:
> Version 3.0.0 has been created in JIRA
> , please feel free
> to use it on the corresponding JIRAs from now. As per my previous email I
> see no point in doing an 1.5.0 release currently so I'm OK with moving all
> the JIRAs having fix/target version of 1.5.0 to 3.0.0. Any objections?
>
> Update on the dependencies of the release:
> * Gradle patch needs some finalization and can be committed soon:
> https://reviews.apache.org/r/66067/
> * Kite removal effort has been started: SQOOP-3313
> 
> * Hive 3.0.0 release is still in an early phase based on this email thread
>  box/%3c2ec60da6-0a2e-4f3a-92f2-e3ce9d497...@hortonworks.com%3E>
> and has no ETA yet
>
> Thanks Daniel for looking into the Hadoop compatibility question, please
> let us know your findings.
>
> Cheers,
> Bogi
>
>
>
> On Thu, May 10, 2018 at 3:27 PM, Dániel Vörös 
> wrote:
>
> > Dear All,
> >
> > After Bogi has created the 3.0.0 version in Jira I've applied it to a
> > couple of tickets that don't make sense on the 1.x line (without
> > Hadoop3/Hive3).
> >
> > However, as Bogi has mentioned in her previous email, it probably doesn't
> > make sense to work on a 1.5 release in parallel with 3.0.0. How would you
> > feel if we were to move all 1.5 issues [1] to 3.0.0?
> >
> > In the meantime I've experimented with running Sqoop 1.4.7 against Hadoop
> > 3.1.0, and I'm planning to do the opposite, running Sqoop 3.0.0-SNAPSHOT
> > against Hadoop 2.x. That way we'd be able to better assess Attila's
> > question about backward compatibility. Please note, that the hard part
> will
> > be Hive integration I'm afraid, and until there's no Hive 3.0 release
> it's
> > hard to test. If anyone's interested in this topic, check out [2].
> >
> > Regards,
> > Daniel
> >
> > [1]
> > https://issues.apache.org/jira/issues?jql=project%20%3D%20SQ
> > OOP%20and%20fixVersion%20%3D%201.5.0%20and%20resolutionDate%
> > 20is%20not%20%20empty%20order%20by%20resolutiondate%20desc
> > [2] https://github.com/dvoros/docker-sqoop
> >
> > On Mon, Apr 16, 2018 at 2:20 PM Szabolcs Vasas  wrote:
> >
> > > Hi All,
> > >
> > > Sqoop NG/Sqoop 3:
> > > As far as I remember Sqoop NG was an alternative name suggested for
> > Sqoop 2
> > > which has a totally different architecture than Sqoop 1. I would not
> use
> > > now since in this release we do not include changes affecting the
> > > architecture but bumping the versions of the dependencies. However
> since
> > > dependencies are bumped to another major releases I think we should
> also
> > > change the major version number of Sqoop.
> > >
> > > Hadoop 2 support:
> > > I agree with Daniel that we should not introduce extra complexity to
> > > support Hadoop 2 as well. However even if we don't support Hadoop 2 in
> > our
> > > next major Sqoop release some features which do not require Hadoop 3
> > 

Re: Review Request 66548: Importing as ORC file to support full ACID Hive tables

[Attila Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66548/#review202879
---



Hi Daniel,

I did not have time for a full review yet, though already found a few smaller 
flaws, and one conceptional one.

Conceptional:
I'd strongly suggest to extract ORC reading/writing only as a subtask as ACID 
Hive tables. The clean advantages would be:
We would be able to test ORC file import/export without any Hive related 
changes, and thus testing the correctness with PrestoDB, or any other tool 
which can read read/write ORC files.
What do you think?

Smaller ones:
See inline.

Regards,
Attila


ivy.xml
Lines 144-149 (patched)


Are we sure this is the total exclude list what we need, and no Hive or 
other library related "debris" are coming with it?



ivy/libraries.properties
Lines 62 (patched)


Shouldn't we aim for the nohive version of 1.4.3?



src/java/org/apache/sqoop/mapreduce/OrcImportMapper.java
Lines 47-51 (patched)


Aren't we missing a super call here?



src/java/org/apache/sqoop/mapreduce/OrcImportMapper.java
Lines 53 (patched)


Occording to the general code guidelines and Joschua Bloch (Effective Java) 
this is a very dangerous practice to initialize an object in a non-final 
initializer. 

Please correct this!


- Attila Szabo


On May 2, 2018, 12:12 p.m., daniel voros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66548/
> ---
> 
> (Updated May 2, 2018, 12:12 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3311
> https://issues.apache.org/jira/browse/SQOOP-3311
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Hive 3 will introduce a switch (HIVE-18294) to create eligible tables as ACID 
> by default. This will probably result in increased usage of ACID tables and 
> the need to support importing into ACID tables with Sqoop.
> 
> Currently the only table format supporting full ACID tables is ORC.
> 
> The easiest and most effective way to support importing into these tables 
> would be to write out files as ORC and keep using LOAD DATA as we do for all 
> other Hive tables (supported since HIVE-17361).
> 
> Workaround could be to create table as textfile (as before) and then CTAS 
> from that. This would push the responsibility of creating ORC format to Hive. 
> However it would result in writing every record twice; in text format and in 
> ORC.
> 
> Note that ORC is only necessary for full ACID tables. Insert-only (aka. 
> micromanaged) ACID tables can use arbitrary file format.
> 
> Supporting full ACID tables would also be the first step in making 
> "lastmodified" incremental imports work with Hive.
> 
> 
> Diffs
> -
> 
>   ivy.xml 6af94d9d 
>   ivy/libraries.properties c44b50bc 
>   src/docs/man/import-common-args.txt 22e3448e 
>   src/docs/man/sqoop-import-all-tables.txt 6db38ad8 
>   src/docs/user/hcatalog.txt 2ae1d54d 
>   src/docs/user/help.txt 8a0d1477 
>   src/docs/user/import-all-tables.txt fbad47b2 
>   src/docs/user/import.txt 2d074f49 
>   src/java/org/apache/sqoop/SqoopOptions.java d9984af3 
>   src/java/org/apache/sqoop/hive/TableDefWriter.java 27d988c5 
>   src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java a5962ba4 
>   src/java/org/apache/sqoop/mapreduce/OrcImportMapper.java PRE-CREATION 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 783651a4 
>   src/java/org/apache/sqoop/tool/ExportTool.java 060f2c07 
>   src/java/org/apache/sqoop/tool/ImportTool.java ee79d8b7 
>   src/java/org/apache/sqoop/util/OrcConversionContext.java PRE-CREATION 
>   src/java/org/apache/sqoop/util/OrcUtil.java PRE-CREATION 
>   src/test/org/apache/sqoop/TestAllTables.java 56d1f577 
>   src/test/org/apache/sqoop/TestOrcImport.java PRE-CREATION 
>   src/test/org/apache/sqoop/hive/TestTableDefWriter.java 3ea61f64 
>   src/test/org/apache/sqoop/util/TestOrcConversionContext.java PRE-CREATION 
>   src/test/org/apache/sqoop/util/TestOrcUtil.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66548/diff/7/
> 
> 
> Testing
> ---
> 
> - added some unit tests
> - tested basic Hive import scenarios on a cluster
> 
> 
> Thanks,
> 
> daniel voros
> 
>

[jira] [Commented] (SQOOP-3322) Version differences between ivy configurations

[Boglarka Egyed (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470842#comment-16470842
 ] 

Boglarka Egyed commented on SQOOP-3322:
---

Thanks [~dvoros] for this clean up! Please close the Review Request too.

> Version differences between ivy configurations
> --
>
> Key: SQOOP-3322
> URL: https://issues.apache.org/jira/browse/SQOOP-3322
> Project: Sqoop
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 3.0.0
>
>
> We have multiple ivy configurations defined in ivy.xml.
>  - The {{redist}} configuration is used to select the artifacts that need to 
> be distributed with Sqoop in its tar.gz.
>  - The {{common}} configuration is used to set the classpath during 
> compilation (also refered to as 'hadoop classpath')
>  -  The {{test}} configuration is used to set the classpath during junit 
> execution. It extends the {{common}} config.
> Some artifacts end up having different versions between these three 
> configurations, which means we're using different versions during 
> compilation/testing/runtime.
> Differences:
> ||Artifact||redist||common (compilation)||test||
> |commons-pool|not in redist|1.5.4|*1.6*|
> |commons-codec|1.4|1.9|*1.9*|
> |commons-io|1.4|2.4|*2.4*|
> |commons-logging|1.1.1|1.2|*1.2*|
> |slf4j-api|1.6.1|1.7.7|*1.7.7*|
> I'd suggest using the version *in bold* in all three configurations to use 
> the latest versions.
> To achieve this we should exclude these artifacts from the transitive 
> dependencies and define them explicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3322) Version differences between ivy configurations

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470839#comment-16470839
 ] 

Hudson commented on SQOOP-3322:
---

SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1165 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1165/])
SQOOP-3322: Version differences between ivy configurations (bogi: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=2ca85527fd9f927add9127f91f3f3ef0c98fed6e])
* (edit) ivy.xml
* (edit) ivy/libraries.properties


> Version differences between ivy configurations
> --
>
> Key: SQOOP-3322
> URL: https://issues.apache.org/jira/browse/SQOOP-3322
> Project: Sqoop
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 3.0.0
>
>
> We have multiple ivy configurations defined in ivy.xml.
>  - The {{redist}} configuration is used to select the artifacts that need to 
> be distributed with Sqoop in its tar.gz.
>  - The {{common}} configuration is used to set the classpath during 
> compilation (also refered to as 'hadoop classpath')
>  -  The {{test}} configuration is used to set the classpath during junit 
> execution. It extends the {{common}} config.
> Some artifacts end up having different versions between these three 
> configurations, which means we're using different versions during 
> compilation/testing/runtime.
> Differences:
> ||Artifact||redist||common (compilation)||test||
> |commons-pool|not in redist|1.5.4|*1.6*|
> |commons-codec|1.4|1.9|*1.9*|
> |commons-io|1.4|2.4|*2.4*|
> |commons-logging|1.1.1|1.2|*1.2*|
> |slf4j-api|1.6.1|1.7.7|*1.7.7*|
> I'd suggest using the version *in bold* in all three configurations to use 
> the latest versions.
> To achieve this we should exclude these artifacts from the transitive 
> dependencies and define them explicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3322) Version differences between ivy configurations

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470713#comment-16470713
 ] 

ASF subversion and git services commented on SQOOP-3322:


Commit 2ca85527fd9f927add9127f91f3f3ef0c98fed6e in sqoop's branch 
refs/heads/trunk from [~BoglarkaEgyed]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=2ca8552 ]

SQOOP-3322: Version differences between ivy configurations

(Daniel Voros via Boglarka Egyed)


> Version differences between ivy configurations
> --
>
> Key: SQOOP-3322
> URL: https://issues.apache.org/jira/browse/SQOOP-3322
> Project: Sqoop
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 3.0.0
>
>
> We have multiple ivy configurations defined in ivy.xml.
>  - The {{redist}} configuration is used to select the artifacts that need to 
> be distributed with Sqoop in its tar.gz.
>  - The {{common}} configuration is used to set the classpath during 
> compilation (also refered to as 'hadoop classpath')
>  -  The {{test}} configuration is used to set the classpath during junit 
> execution. It extends the {{common}} config.
> Some artifacts end up having different versions between these three 
> configurations, which means we're using different versions during 
> compilation/testing/runtime.
> Differences:
> ||Artifact||redist||common (compilation)||test||
> |commons-pool|not in redist|1.5.4|*1.6*|
> |commons-codec|1.4|1.9|*1.9*|
> |commons-io|1.4|2.4|*2.4*|
> |commons-logging|1.1.1|1.2|*1.2*|
> |slf4j-api|1.6.1|1.7.7|*1.7.7*|
> I'd suggest using the version *in bold* in all three configurations to use 
> the latest versions.
> To achieve this we should exclude these artifacts from the transitive 
> dependencies and define them explicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67005: Version differences between ivy configurations

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67005/#review202848
---


Ship it!




Ship It!

- Boglarka Egyed


On May 8, 2018, 1:43 p.m., daniel voros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67005/
> ---
> 
> (Updated May 8, 2018, 1:43 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3322
> https://issues.apache.org/jira/browse/SQOOP-3322
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> We have multiple ivy configurations defined in ivy.xml.
> 
> - The `redist` configuration is used to select the artifacts that need to be 
> distributed with Sqoop in its tar.gz.
> - The `common` configuration is used to set the classpath during compilation 
> (also refered to as 'hadoop classpath')
> - The `test` configuration is used to set the classpath during junit 
> execution. It extends the `common` config.
> 
> Some artifacts end up having different versions between these three 
> configurations, which means we're using different versions during 
> compilation/testing/runtime.
> 
> 
> Diffs
> -
> 
>   ivy.xml 6af94d9d 
>   ivy/libraries.properties c44b50bc 
> 
> 
> Diff: https://reviews.apache.org/r/67005/diff/1/
> 
> 
> Testing
> ---
> 
> - compared the results of ivy-resolve-hadoop, ivy-resolve-test, 
> ivy-resolve-redist tasks to make sure versions are the same
> - checked unit tests just to be on the safe side, test versions weren't 
> changed though (all passed apart from known issues in SQOOP-3321)
> 
> 
> Thanks,
> 
> daniel voros
> 
>

[jira] [Commented] (SQOOP-3321) Fix TestHiveImport failing on Jenkins and Linux

[Boglarka Egyed (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470622#comment-16470622
 ] 

Boglarka Egyed commented on SQOOP-3321:
---

Thanks [~dvoros] for fixing this issue! Please don't forget to close the 
related Review Request too.

> Fix TestHiveImport failing on Jenkins and Linux
> ---
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (SQOOP-3321) Fix TestHiveImport failing on Jenkins and Linux

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed resolved SQOOP-3321.
---
   Resolution: Fixed
Fix Version/s: 3.0.0

> Fix TestHiveImport failing on Jenkins and Linux
> ---
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3321) Fix TestHiveImport failing on Jenkins and Linux

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470615#comment-16470615
 ] 

Hudson commented on SQOOP-3321:
---

SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1164 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1164/])
SQOOP-3321: Fix TestHiveImport failing on Jenkins and Linux (bogi: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=03a895c5881b6dc9f748480b7d0562377d3cc978])
* (edit) src/test/org/apache/sqoop/hive/TestHiveImport.java


> Fix TestHiveImport failing on Jenkins and Linux
> ---
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Daniel Voros
>Priority: Major
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3321) Fix TestHiveImport failing on Jenkins and Linux

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470573#comment-16470573
 ] 

ASF subversion and git services commented on SQOOP-3321:


Commit 03a895c5881b6dc9f748480b7d0562377d3cc978 in sqoop's branch 
refs/heads/trunk from [~BoglarkaEgyed]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=03a895c ]

SQOOP-3321: Fix TestHiveImport failing on Jenkins and Linux

(Daniel Voros via Boglarka Egyed)


> Fix TestHiveImport failing on Jenkins and Linux
> ---
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Daniel Voros
>Priority: Major
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3321) Fix TestHiveImport failing on Jenkins and Linux

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3321:
--
Summary: Fix TestHiveImport failing on Jenkins and Linux  (was: Fix 
TestHiveImport that is failing on Jenkins and Linux)

> Fix TestHiveImport failing on Jenkins and Linux
> ---
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Daniel Voros
>Priority: Major
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3321) Fix TestHiveImport that is failing on Jenkins and Linux

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3321:
--
Summary: Fix TestHiveImport that is failing on Jenkins and Linux  (was: 
TestHiveImport is failing on Jenkins)

> Fix TestHiveImport that is failing on Jenkins and Linux
> ---
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Daniel Voros
>Priority: Major
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3321) TestHiveImport is failing on Jenkins

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed reassigned SQOOP-3321:
-

Assignee: Daniel Voros

> TestHiveImport is failing on Jenkins
> 
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Daniel Voros
>Priority: Major
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67057: TestHiveImport is failing on Jenkins

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67057/#review202845
---


Ship it!




Tested on Ubuntu and Mac OS too, both passed. Thanks for the fix Daniel!

- Boglarka Egyed


On May 10, 2018, 12:53 p.m., daniel voros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67057/
> ---
> 
> (Updated May 10, 2018, 12:53 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3321
> https://issues.apache.org/jira/browse/SQOOP-3321
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> I believe this is due to case sensitivity of file names in Linux (as opposed 
> to MacOS). The table name gets converted to lowercase when importing but 
> we're referring to it with it's original casing when trying to verify its 
> contents in ParquetReader.
> 
> Tests are passing after converting these three table names to all lowercase 
> in TestHiveImport:
> 
> - APPEND_HIVE_IMPORT_AS_PARQUET
> - NORMAL_HIVE_IMPORT_AS_PARQUET
> - CREATE_OVERWRITE_HIVE_IMPORT_AS_PARQUET
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/hive/TestHiveImport.java bc19b697 
> 
> 
> Diff: https://reviews.apache.org/r/67057/diff/1/
> 
> 
> Testing
> ---
> 
> Run TestHiveImport.
> 
> 
> Thanks,
> 
> daniel voros
> 
>

Re: Release to support Hadoop 3

[Boglarka Egyed]

Hi All,

Thank you Daniel for the update! I was also writing one when your email
arrived so I'm just adding a couple of comments to that.

New major version in JIRA:
Version 3.0.0 has been created in JIRA
, please feel free
to use it on the corresponding JIRAs from now. As per my previous email I
see no point in doing an 1.5.0 release currently so I'm OK with moving all
the JIRAs having fix/target version of 1.5.0 to 3.0.0. Any objections?

Update on the dependencies of the release:
* Gradle patch needs some finalization and can be committed soon:
https://reviews.apache.org/r/66067/
* Kite removal effort has been started: SQOOP-3313

* Hive 3.0.0 release is still in an early phase based on this email thread

and has no ETA yet

Thanks Daniel for looking into the Hadoop compatibility question, please
let us know your findings.

Cheers,
Bogi



On Thu, May 10, 2018 at 3:27 PM, Dániel Vörös 
wrote:

> Dear All,
>
> After Bogi has created the 3.0.0 version in Jira I've applied it to a
> couple of tickets that don't make sense on the 1.x line (without
> Hadoop3/Hive3).
>
> However, as Bogi has mentioned in her previous email, it probably doesn't
> make sense to work on a 1.5 release in parallel with 3.0.0. How would you
> feel if we were to move all 1.5 issues [1] to 3.0.0?
>
> In the meantime I've experimented with running Sqoop 1.4.7 against Hadoop
> 3.1.0, and I'm planning to do the opposite, running Sqoop 3.0.0-SNAPSHOT
> against Hadoop 2.x. That way we'd be able to better assess Attila's
> question about backward compatibility. Please note, that the hard part will
> be Hive integration I'm afraid, and until there's no Hive 3.0 release it's
> hard to test. If anyone's interested in this topic, check out [2].
>
> Regards,
> Daniel
>
> [1]
> https://issues.apache.org/jira/issues?jql=project%20%3D%20SQ
> OOP%20and%20fixVersion%20%3D%201.5.0%20and%20resolutionDate%
> 20is%20not%20%20empty%20order%20by%20resolutiondate%20desc
> [2] https://github.com/dvoros/docker-sqoop
>
> On Mon, Apr 16, 2018 at 2:20 PM Szabolcs Vasas  wrote:
>
> > Hi All,
> >
> > Sqoop NG/Sqoop 3:
> > As far as I remember Sqoop NG was an alternative name suggested for
> Sqoop 2
> > which has a totally different architecture than Sqoop 1. I would not use
> > now since in this release we do not include changes affecting the
> > architecture but bumping the versions of the dependencies. However since
> > dependencies are bumped to another major releases I think we should also
> > change the major version number of Sqoop.
> >
> > Hadoop 2 support:
> > I agree with Daniel that we should not introduce extra complexity to
> > support Hadoop 2 as well. However even if we don't support Hadoop 2 in
> our
> > next major Sqoop release some features which do not require Hadoop 3
> could
> > be backported by the vendors to their earlier releases as well. I think
> > introducing a 1.x branch upstream would lead to an increased complexity
> of
> > committing bug fixes and I am not sure the community wants to make a
> > release in Sqoop 1.x branch. Even if at some point somebody wants to do
> > this they could cut the branch and cherry-pick the necessary bug fixes
> > right before the release.
> >
> > Kite removal:
> > I agree that this is quite complex task on its own but we can't bump the
> > Hadoop/Hive/HBase dependencies without deciding what to do with Kite. One
> > option is to bump these dependencies in Kite too, create a new Kite
> release
> > and bump Sqoop's Kite dependency to this new release. Another option is
> to
> > get rid of the Kite dependency before we bump Hadoop/Hive/HBase version.
> In
> > my opinion the latter one makes more sense since we wanted to eliminate
> the
> > Kite dependency anyway and the Kite project seems to be dead so bumping
> the
> > dependencies, making the necessary code changes, fixing tests and
> creating
> > the release might be an overkill.
> >
> > Szabolcs
> >
> > On Mon, Apr 16, 2018 at 11:50 AM, Dániel Vörös 
> > wrote:
> >
> > > Hi All,
> > >
> > > I believe we're all on the same page on removing Kite, so I've opened
> > > SQOOP-3313 to track that. @Attila I'm glad to see you're interest in
> the
> > > ORC part. It would be highly appreciated if you could take a look at
> this
> > > review request[1].
> > >
> > > I'm not that familiar with Flume, but it seems they've added NG after
> > > architectural changes and released FlumeNG 1.0 after Flume 0.9.4 [2].
> > Even
> > > if we go with NG, I'd suggest calling it 3.0, to avoid confusion with
> > > earlier releases.
> > >
> > > I think the biggest part of keeping Hadoop 2 (and previous versions of
> > > downstream projects like Hive) supported would be testing against
> 

Re: Review Request 67005: Version differences between ivy configurations

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67005/#review202841
---


Ship it!




Seems ok to me. I've ran unit and 3rd party tests as well.


ivy.xml
Line 90 (original), 90 (patched)


I'm not an expert on ant, so just asking for an educational purpose: what 
does this exactly mean?


- Fero Szabo


On May 8, 2018, 1:43 p.m., daniel voros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67005/
> ---
> 
> (Updated May 8, 2018, 1:43 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3322
> https://issues.apache.org/jira/browse/SQOOP-3322
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> We have multiple ivy configurations defined in ivy.xml.
> 
> - The `redist` configuration is used to select the artifacts that need to be 
> distributed with Sqoop in its tar.gz.
> - The `common` configuration is used to set the classpath during compilation 
> (also refered to as 'hadoop classpath')
> - The `test` configuration is used to set the classpath during junit 
> execution. It extends the `common` config.
> 
> Some artifacts end up having different versions between these three 
> configurations, which means we're using different versions during 
> compilation/testing/runtime.
> 
> 
> Diffs
> -
> 
>   ivy.xml 6af94d9d 
>   ivy/libraries.properties c44b50bc 
> 
> 
> Diff: https://reviews.apache.org/r/67005/diff/1/
> 
> 
> Testing
> ---
> 
> - compared the results of ivy-resolve-hadoop, ivy-resolve-test, 
> ivy-resolve-redist tasks to make sure versions are the same
> - checked unit tests just to be on the safe side, test versions weren't 
> changed though (all passed apart from known issues in SQOOP-3321)
> 
> 
> Thanks,
> 
> daniel voros
> 
>

Re: Release to support Hadoop 3

[Dániel Vörös]

Dear All,

After Bogi has created the 3.0.0 version in Jira I've applied it to a
couple of tickets that don't make sense on the 1.x line (without
Hadoop3/Hive3).

However, as Bogi has mentioned in her previous email, it probably doesn't
make sense to work on a 1.5 release in parallel with 3.0.0. How would you
feel if we were to move all 1.5 issues [1] to 3.0.0?

In the meantime I've experimented with running Sqoop 1.4.7 against Hadoop
3.1.0, and I'm planning to do the opposite, running Sqoop 3.0.0-SNAPSHOT
against Hadoop 2.x. That way we'd be able to better assess Attila's
question about backward compatibility. Please note, that the hard part will
be Hive integration I'm afraid, and until there's no Hive 3.0 release it's
hard to test. If anyone's interested in this topic, check out [2].

Regards,
Daniel

[1]
https://issues.apache.org/jira/issues?jql=project%20%3D%20SQOOP%20and%20fixVersion%20%3D%201.5.0%20and%20resolutionDate%20is%20not%20%20empty%20order%20by%20resolutiondate%20desc
[2] https://github.com/dvoros/docker-sqoop

On Mon, Apr 16, 2018 at 2:20 PM Szabolcs Vasas  wrote:

> Hi All,
>
> Sqoop NG/Sqoop 3:
> As far as I remember Sqoop NG was an alternative name suggested for Sqoop 2
> which has a totally different architecture than Sqoop 1. I would not use
> now since in this release we do not include changes affecting the
> architecture but bumping the versions of the dependencies. However since
> dependencies are bumped to another major releases I think we should also
> change the major version number of Sqoop.
>
> Hadoop 2 support:
> I agree with Daniel that we should not introduce extra complexity to
> support Hadoop 2 as well. However even if we don't support Hadoop 2 in our
> next major Sqoop release some features which do not require Hadoop 3 could
> be backported by the vendors to their earlier releases as well. I think
> introducing a 1.x branch upstream would lead to an increased complexity of
> committing bug fixes and I am not sure the community wants to make a
> release in Sqoop 1.x branch. Even if at some point somebody wants to do
> this they could cut the branch and cherry-pick the necessary bug fixes
> right before the release.
>
> Kite removal:
> I agree that this is quite complex task on its own but we can't bump the
> Hadoop/Hive/HBase dependencies without deciding what to do with Kite. One
> option is to bump these dependencies in Kite too, create a new Kite release
> and bump Sqoop's Kite dependency to this new release. Another option is to
> get rid of the Kite dependency before we bump Hadoop/Hive/HBase version. In
> my opinion the latter one makes more sense since we wanted to eliminate the
> Kite dependency anyway and the Kite project seems to be dead so bumping the
> dependencies, making the necessary code changes, fixing tests and creating
> the release might be an overkill.
>
> Szabolcs
>
> On Mon, Apr 16, 2018 at 11:50 AM, Dániel Vörös 
> wrote:
>
> > Hi All,
> >
> > I believe we're all on the same page on removing Kite, so I've opened
> > SQOOP-3313 to track that. @Attila I'm glad to see you're interest in the
> > ORC part. It would be highly appreciated if you could take a look at this
> > review request[1].
> >
> > I'm not that familiar with Flume, but it seems they've added NG after
> > architectural changes and released FlumeNG 1.0 after Flume 0.9.4 [2].
> Even
> > if we go with NG, I'd suggest calling it 3.0, to avoid confusion with
> > earlier releases.
> >
> > I think the biggest part of keeping Hadoop 2 (and previous versions of
> > downstream projects like Hive) supported would be testing against those.
> It
> > would also require at least another build profile to build against them,
> > and probably another layer of abstraction in the code (like Hadoop shims
> in
> > Hive).
> > Not sure about vendors, but I think they're usually not adding new
> features
> > to older release lines. In my opinion we should branch off from current
> > trunk to track the 1.x release line (where we keep supporting Hadoop 2)
> and
> > keep adding bugfixes there, but add new features to trunk only and don't
> > worry about Hadoop 2 there.
> >
> > I agree with Attila on the dependencies. We shouldn't release based on
> > non-final releases. We might bump the dependencies to some alpha/beta
> > during development, but don't forget to move to the final version in the
> > end.
> >
> > +1 for Bogi as release manager.
> >
> > Regards,
> > Daniel
> >
> > [1] https://reviews.apache.org/r/66548/
> > [2] https://blogs.apache.org/flume/entry/flume_ng_architecture
> >
> > On Fri, Apr 13, 2018 at 5:24 PM Szabó Attila  wrote:
> >
> > >
> > >
> > > Hello everyone,
> > >
> > >
> > > I'd like to also attach my thoughts:
> > >
> > >
> > > New Sqoop version: Last time when I'd the chance to talk about this
> with
> > > some of the PMC members (e.g. Jarcec, Kate ) we've been on the front to
> > > create Sqoop-NG (NG == Next Generation), 

[jira] [Updated] (SQOOP-3313) Remove Kite dependency

[Daniel Voros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Voros updated SQOOP-3313:

Fix Version/s: 3.0.0

> Remove Kite dependency
> --
>
> Key: SQOOP-3313
> URL: https://issues.apache.org/jira/browse/SQOOP-3313
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 3.0.0
>
>
> Having Kite as a dependency makes it hard to release a version of Sqoop 
> compatible with Hadoop 3.
> For details see discussion on dev list in [this thread|http://example.com] 
> and also SQOOP-3305.
> Let's use this ticket to gather features that need to be 
> changed/reimplemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3305) Upgrade to Hadoop 3, Hive 3, and HBase 2

[Daniel Voros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Voros updated SQOOP-3305:

Fix Version/s: 3.0.0

> Upgrade to Hadoop 3, Hive 3, and HBase 2
> 
>
> Key: SQOOP-3305
> URL: https://issues.apache.org/jira/browse/SQOOP-3305
> Project: Sqoop
>  Issue Type: Task
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 3.0.0
>
>
> To be able to eventually support the latest versions of Hive, HBase and 
> Accumulo, we should start by upgrading our Hadoop dependencies to 3.0.0. See 
> https://hadoop.apache.org/docs/r3.0.0/index.html
> In this ticket I'll collect the necessary changes to do the upgrade. I'm not 
> setting a fix version yet, since this might mean a major release and to be 
> done together with the upgrade of related components.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3322) Version differences between ivy configurations

[Daniel Voros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Voros updated SQOOP-3322:

Fix Version/s: 3.0.0

> Version differences between ivy configurations
> --
>
> Key: SQOOP-3322
> URL: https://issues.apache.org/jira/browse/SQOOP-3322
> Project: Sqoop
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 3.0.0
>
>
> We have multiple ivy configurations defined in ivy.xml.
>  - The {{redist}} configuration is used to select the artifacts that need to 
> be distributed with Sqoop in its tar.gz.
>  - The {{common}} configuration is used to set the classpath during 
> compilation (also refered to as 'hadoop classpath')
>  -  The {{test}} configuration is used to set the classpath during junit 
> execution. It extends the {{common}} config.
> Some artifacts end up having different versions between these three 
> configurations, which means we're using different versions during 
> compilation/testing/runtime.
> Differences:
> ||Artifact||redist||common (compilation)||test||
> |commons-pool|not in redist|1.5.4|*1.6*|
> |commons-codec|1.4|1.9|*1.9*|
> |commons-io|1.4|2.4|*2.4*|
> |commons-logging|1.1.1|1.2|*1.2*|
> |slf4j-api|1.6.1|1.7.7|*1.7.7*|
> I'd suggest using the version *in bold* in all three configurations to use 
> the latest versions.
> To achieve this we should exclude these artifacts from the transitive 
> dependencies and define them explicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3323) Use beeline in (non-JDBC) Hive imports

[Daniel Voros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Voros updated SQOOP-3323:

Affects Version/s: 3.0.0
Fix Version/s: 3.0.0

Thank you!

> Use beeline in (non-JDBC) Hive imports
> --
>
> Key: SQOOP-3323
> URL: https://issues.apache.org/jira/browse/SQOOP-3323
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hive-integration
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 3.0.0
>
>
> When doing Hive imports the old way (not via JDBC that was introduced in 
> SQOOP-3309) we're trying to use the {{CliDriver}} class from Hive and fall 
> back to the {{hive}} executable (a.k.a. [Hive 
> Cli|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli]) if 
> that class is not found.
> Since {{CliDriver}} and the {{hive}} executable that's relying on it are 
> [deprecated|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli]
>  (see also HIVE-10511), we should switch to using {{beeline}} to talk to 
> Hive. With recent additions (e.g. HIVE-18963) this should be easier than 
> before.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3321) TestHiveImport is failing on Jenkins

[Daniel Voros (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470349#comment-16470349
 ] 

Daniel Voros commented on SQOOP-3321:
-

Thank you [~fero]! I've attached a patch on the RB.

> TestHiveImport is failing on Jenkins
> 
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 67057: TestHiveImport is failing on Jenkins

[daniel voros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67057/
---

Review request for Sqoop.


Bugs: SQOOP-3321
https://issues.apache.org/jira/browse/SQOOP-3321


Repository: sqoop-trunk


Description
---

I believe this is due to case sensitivity of file names in Linux (as opposed to 
MacOS). The table name gets converted to lowercase when importing but we're 
referring to it with it's original casing when trying to verify its contents in 
ParquetReader.

Tests are passing after converting these three table names to all lowercase in 
TestHiveImport:

- APPEND_HIVE_IMPORT_AS_PARQUET
- NORMAL_HIVE_IMPORT_AS_PARQUET
- CREATE_OVERWRITE_HIVE_IMPORT_AS_PARQUET


Diffs
-

  src/test/org/apache/sqoop/hive/TestHiveImport.java bc19b697 


Diff: https://reviews.apache.org/r/67057/diff/1/


Testing
---

Run TestHiveImport.


Thanks,

daniel voros

[jira] [Commented] (SQOOP-3323) Use beeline in (non-JDBC) Hive imports

[Boglarka Egyed (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470345#comment-16470345
 ] 

Boglarka Egyed commented on SQOOP-3323:
---

[~dvoros] I have created version 3.0.0, please feel free to use it on the 
corresponding JIRAs.

> Use beeline in (non-JDBC) Hive imports
> --
>
> Key: SQOOP-3323
> URL: https://issues.apache.org/jira/browse/SQOOP-3323
> Project: Sqoop
>  Issue Type: Improvement
>  Components: hive-integration
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
>
> When doing Hive imports the old way (not via JDBC that was introduced in 
> SQOOP-3309) we're trying to use the {{CliDriver}} class from Hive and fall 
> back to the {{hive}} executable (a.k.a. [Hive 
> Cli|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli]) if 
> that class is not found.
> Since {{CliDriver}} and the {{hive}} executable that's relying on it are 
> [deprecated|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli]
>  (see also HIVE-10511), we should switch to using {{beeline}} to talk to 
> Hive. With recent additions (e.g. HIVE-18963) this should be easier than 
> before.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3323) Use beeline in (non-JDBC) Hive imports

[Daniel Voros (JIRA)]

Daniel Voros created SQOOP-3323:
---

 Summary: Use beeline in (non-JDBC) Hive imports
 Key: SQOOP-3323
 URL: https://issues.apache.org/jira/browse/SQOOP-3323
 Project: Sqoop
  Issue Type: Improvement
  Components: hive-integration
Reporter: Daniel Voros
Assignee: Daniel Voros


When doing Hive imports the old way (not via JDBC that was introduced in 
SQOOP-3309) we're trying to use the {{CliDriver}} class from Hive and fall back 
to the {{hive}} executable (a.k.a. [Hive 
Cli|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli]) if 
that class is not found.

Since {{CliDriver}} and the {{hive}} executable that's relying on it are 
[deprecated|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli]
 (see also HIVE-10511), we should switch to using {{beeline}} to talk to Hive. 
With recent additions (e.g. HIVE-18963) this should be easier than before.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3321) TestHiveImport is failing on Jenkins

[Fero Szabo (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470298#comment-16470298
 ] 

Fero Szabo commented on SQOOP-3321:
---

Hi [~dvoros],

Szabolcs is unavailable at the moment, but as I discussed this with 
[~BoglarkaEgyed], we think that we should go ahead with your suggestion. Would 
you like to implement a patch for it?

If you don't have the time, I'm happy to pick this up.

> TestHiveImport is failing on Jenkins
> 
>
> Key: SQOOP-3321
> URL: https://issues.apache.org/jira/browse/SQOOP-3321
> Project: Sqoop
>  Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
> Attachments: TEST-org.apache.sqoop.hive.TestHiveImport.txt
>
>
> org.apache.sqoop.hive.TestHiveImport is failing since 
> [SQOOP-3318|https://reviews.apache.org/r/66761/bugs/SQOOP-3318/] has been 
> committed. This test seem to be failing only in the Jenkins environment as it 
> pass on several local machines. There can be some difference in the 
> filesystem which may cause this issue, it shall be investigated. I am 
> attaching the log from a failed run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)