[jira] [Commented] (SQOOP-2708) Sqoop2: Docs: Update Installation.rst

[Sqoop QA bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032580#comment-15032580
 ] 

Sqoop QA bot commented on SQOOP-2708:
-

Testing file 
[SQOOP-2708.patch|https://issues.apache.org/jira/secure/attachment/12774885/SQOOP-2708.patch]
 against branch sqoop2 took 1:25:20.010083.

{color:red}Overall:{color} -1 due to an error(s), see details below:

{color:green}SUCCESS:{color} Clean was successful
{color:green}SUCCESS:{color} Patch applied correctly
{color:red}ERROR:{color} Patch does not add/modify any test case
{color:green}SUCCESS:{color} License check passed
{color:green}SUCCESS:{color} Patch compiled
{color:green}SUCCESS:{color} All unit tests passed (executed 1640 tests)
{color:green}SUCCESS:{color} Test coverage did not decreased 
([report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1939/artifact/patch-process/cobertura_report.txt])
{color:green}SUCCESS:{color} No new findbugs warnings 
([report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1939/artifact/patch-process/findbugs_report.txt])
{color:red}ERROR:{color} Some of integration tests failed 
([report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1939/artifact/patch-process/test_integration.txt],
 executed 133 tests)
* Test {{new-integration-tests}}
* Test {{org.apache.sqoop.integration.server.ShowJobInOrderTest}}
* Test {{org.apache.sqoop.integration.connector.hdfs.S3Test}}



Console output is available 
[here|https://builds.apache.org/job/PreCommit-SQOOP-Build/1939/console].

This message is automatically generated.

> Sqoop2: Docs: Update Installation.rst
> -
>
> Key: SQOOP-2708
> URL: https://issues.apache.org/jira/browse/SQOOP-2708
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 1.99.7
>
> Attachments: SQOOP-2708.patch, SQOOP-2708.patch, SQOOP-2708.patch
>
>
> I've looked into Installation guide and rewrote it a bit while incorporating 
> SQOOP-2683 and SQOOP-2685.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40576: SQOOP-2708 Sqoop2: Docs: Update Installation.rst

[Abraham Fine]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40576/#review108399
---

Ship it!


Ship It!

- Abraham Fine


On Nov. 30, 2015, 8:31 p.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40576/
> ---
> 
> (Updated Nov. 30, 2015, 8:31 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2708
> https://issues.apache.org/jira/browse/SQOOP-2708
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've updated the installation guide and incoporated changes that were missing.
> 
> 
> Diffs
> -
> 
>   docs/src/site/sphinx/admin/Installation.rst 9d56875 
>   docs/src/site/sphinx/admin/Tools.rst fb0187a 
> 
> Diff: https://reviews.apache.org/r/40576/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40576: SQOOP-2708 Sqoop2: Docs: Update Installation.rst

[Abraham Fine]

> On Nov. 30, 2015, 6:50 p.m., Abraham Fine wrote:
> > docs/src/site/sphinx/admin/Installation.rst, line 97
> > 
> >
> > I would describe the use case here in a little bit more detail. We have 
> > a few different ways of loading jars into sqoop 2. Perhaps we should move 
> > this out of the installation documentation into its own page to keep 
> > initial installation simple.
> 
> Jarek Cecho wrote:
> True, we also have ability to configure jars in jobs and such. Would you 
> mind if I'll keep it this way for now and add a standalone section about 
> classpath later? (especially after Dian will be done with connector classpath 
> isolation project).

ok, let's make a jira for that so we do not forget about it.


- Abraham


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40576/#review108352
---


On Nov. 30, 2015, 8:31 p.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40576/
> ---
> 
> (Updated Nov. 30, 2015, 8:31 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2708
> https://issues.apache.org/jira/browse/SQOOP-2708
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've updated the installation guide and incoporated changes that were missing.
> 
> 
> Diffs
> -
> 
>   docs/src/site/sphinx/admin/Installation.rst 9d56875 
>   docs/src/site/sphinx/admin/Tools.rst fb0187a 
> 
> Diff: https://reviews.apache.org/r/40576/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

[jira] [Commented] (SQOOP-2695) Sqoop2: Doc: Clean up git warnings for doc files

[Abraham Fine (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032168#comment-15032168
 ] 

Abraham Fine commented on SQOOP-2695:
-

+1

> Sqoop2: Doc: Clean up git warnings for doc files
> 
>
> Key: SQOOP-2695
> URL: https://issues.apache.org/jira/browse/SQOOP-2695
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 1.99.7
>
> Attachments: SQOOP-2695.patch
>
>
> We've noticed when moving the files around that they a have bunch of warnings 
> that would be great to fix:
> {code}
> docs/src/site/sphinx/admin/Installation.rst:102: new blank line at EOF.
> docs/src/site/sphinx/admin/Tools.rst:61: leftover conflict marker
> docs/src/site/sphinx/admin/Tools.rst:124: new blank line at EOF.
> docs/src/site/sphinx/admin/Upgrade.rst:17: leftover conflict marker
> docs/src/site/sphinx/admin/Upgrade.rst:19: leftover conflict marker
> docs/src/site/sphinx/dev/ClientAPI.rst:264: trailing whitespace.
> +  //Check job status for a running job
> docs/src/site/sphinx/dev/DevEnv.rst:57: new blank line at EOF.
> docs/src/site/sphinx/dev/Repository.rst:334: new blank line at EOF.
> docs/src/site/sphinx/user/CommandLineClient.rst:361: trailing whitespace.
> +  create job --from 1 --to 2 or create job --f 1 --t 2
> docs/src/site/sphinx/user/Sqoop5MinutesDemo.rst:221: trailing whitespace.
> +  2014-11-04 20:09:16 PST: RUNNING  - 0.00 %
> docs/src/site/sphinx/user/connectors/Connector-Kafka.rst:64: new blank line 
> at EOF.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40558: SQOOP-2697 Sqoop2: Docs: Clean up conf.py file

[Abraham Fine]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40558/#review108348
---

Ship it!


Ship It!

- Abraham Fine


On Nov. 20, 2015, 8 p.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40558/
> ---
> 
> (Updated Nov. 20, 2015, 8 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2697
> https://issues.apache.org/jira/browse/SQOOP-2697
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've removed all properties that we're not explicitly using - e.g. all those 
> that were artifacts after copy-pasting the file when initially creating the 
> docs. I've manually verified that the docs are built correctly and works as 
> expected.
> 
> I've also linked documentation so that it's easier for new people to see 
> where are the configuration options comming from.
> 
> 
> Diffs
> -
> 
>   docs/src/site/sphinx/conf.py 7b620f7 
> 
> Diff: https://reviews.apache.org/r/40558/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Abraham Fine]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108340
---



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
 (line 62)


creating the proxy user and then generating the delegation tokens is a 
pattern that we use twice. once in the hdfstoinitializer and again in the 
hdfsfrominitializer.

perhaps, it could make sense to do something similar to what is being done 
when we load the delegation tokens (`createProxyUserAndLoadDelegationTokens`) 
and have a method `createProxyUserAndGenerateDelegationTokens`). that way we 
can make `generateDelegationTokens` private, and never have to worry about it 
being called outside of a delegation block.



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
 (line 55)


i understand that we used `getLoginUser` previously. is this the correct 
way to go instead of using `getCurrentUser`?



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
 (line 84)


how verbose is this, maybe it should be a debug?



connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
 (line 27)


can we test writing the tokens to and reading them from the context?


- Abraham Fine


On Nov. 24, 2015, 12:58 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Nov. 24, 2015, 12:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40576: SQOOP-2708 Sqoop2: Docs: Update Installation.rst

[Jarek Cecho]

> On Nov. 30, 2015, 6:50 p.m., Abraham Fine wrote:
> > docs/src/site/sphinx/admin/Installation.rst, line 69
> > 
> >
> > "Sqoop server requires further configuration on Hadoop side" is not 
> > needed.
> > 
> > I would split this section into two subsections: "HDFS Proxy Users" and 
> > "YARN Whitelist".

I've nuked the introduction sentence - I already do have two logical sections 
for Proxy/Whitelist already. I didn't want to introduce yet another nested 
structure though so I have it under one "headline".


> On Nov. 30, 2015, 6:50 p.m., Abraham Fine wrote:
> > docs/src/site/sphinx/admin/Installation.rst, line 71
> > 
> >
> > does this always need to be enabled. could there be use cases (such as 
> > moving data for s3 to an rdbms) where the cluster does not need to be 
> > configured to allow impersonation?

The reality is that only if you're using HDFS connector this will be needed. 
But as that is so common use case, I did not wanted to burry that inside the 
HDFS connector docs but rather expose it here in the installation docs.


> On Nov. 30, 2015, 6:50 p.m., Abraham Fine wrote:
> > docs/src/site/sphinx/admin/Installation.rst, line 97
> > 
> >
> > I would describe the use case here in a little bit more detail. We have 
> > a few different ways of loading jars into sqoop 2. Perhaps we should move 
> > this out of the installation documentation into its own page to keep 
> > initial installation simple.

True, we also have ability to configure jars in jobs and such. Would you mind 
if I'll keep it this way for now and add a standalone section about classpath 
later? (especially after Dian will be done with connector classpath isolation 
project).


> On Nov. 30, 2015, 6:50 p.m., Abraham Fine wrote:
> > docs/src/site/sphinx/admin/Tools.rst, line 60
> > 
> >
> > "the same file"

Let's fix the Tools documentation separately - I'm editing the file only to add 
anchors at this point.


> On Nov. 30, 2015, 6:50 p.m., Abraham Fine wrote:
> > docs/src/site/sphinx/admin/Tools.rst, line 67
> > 
> >
> > Perhaps we should make it clear that we are not actually downloading 
> > and installing a new version of sqoop 2 as a side node?
> > 
> > Really this is a "repository upgrade tool".

Let's fix the Tools documentation separately - I'm editing the file only to add 
anchors at this point.


- Jarek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40576/#review108352
---


On Nov. 25, 2015, 7:40 p.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40576/
> ---
> 
> (Updated Nov. 25, 2015, 7:40 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2708
> https://issues.apache.org/jira/browse/SQOOP-2708
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've updated the installation guide and incoporated changes that were missing.
> 
> 
> Diffs
> -
> 
>   docs/src/site/sphinx/admin/Installation.rst 9d56875 
>   docs/src/site/sphinx/admin/Tools.rst fb0187a 
> 
> Diff: https://reviews.apache.org/r/40576/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40576: SQOOP-2708 Sqoop2: Docs: Update Installation.rst

[Jarek Cecho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40576/
---

(Updated Nov. 30, 2015, 8:31 p.m.)


Review request for Sqoop.


Changes
---

Incorporated Abe's feedback.


Bugs: SQOOP-2708
https://issues.apache.org/jira/browse/SQOOP-2708


Repository: sqoop-sqoop2


Description
---

I've updated the installation guide and incoporated changes that were missing.


Diffs (updated)
-

  docs/src/site/sphinx/admin/Installation.rst 9d56875 
  docs/src/site/sphinx/admin/Tools.rst fb0187a 

Diff: https://reviews.apache.org/r/40576/diff/


Testing
---


Thanks,

Jarek Cecho

[jira] [Updated] (SQOOP-2708) Sqoop2: Docs: Update Installation.rst

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated SQOOP-2708:
--
Attachment: SQOOP-2708.patch

> Sqoop2: Docs: Update Installation.rst
> -
>
> Key: SQOOP-2708
> URL: https://issues.apache.org/jira/browse/SQOOP-2708
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 1.99.7
>
> Attachments: SQOOP-2708.patch, SQOOP-2708.patch, SQOOP-2708.patch
>
>
> I've looked into Installation guide and rewrote it a bit while incorporating 
> SQOOP-2683 and SQOOP-2685.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Jarek Cecho]

> On Nov. 25, 2015, 3:33 a.m., Dian Fu wrote:
> >
> 
> Dian Fu wrote:
> For the integration test, I'm currently working on SQOOP-2704 to enable 
> kerberos in the integration tests. I will add tests for this scenario under 
> that JIRA.

You're awesome, thank you Dian!


> On Nov. 25, 2015, 3:33 a.m., Dian Fu wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java,
> >  line 82
> > 
> >
> > The implementation of "SecurityUtils.generateDelegationTokens()" 
> > overrides the field "HdfsConstants.DELEGATION_TOKENS" each time it's 
> > called. I think it should not override the old ones as both 
> > "HdfsFromInitializer" and "HdfsToInitializer" call 
> > "SecurityUtils.generateDelegationTokens()" to generate delegation tokens 
> > for HDFS paths. These paths may be different.

Good thinking.

We're strictly separating Contexts for each From/To portion, so even in case 
when someone will be doing HDFS to HDFS transfer, each Initializator (From/To) 
will be working with it's own Context instance.


> On Nov. 25, 2015, 3:33 a.m., Dian Fu wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java,
> >  line 121
> > 
> >
> > baos.flush() is not needed.

Good point.


- Jarek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review107910
---


On Nov. 24, 2015, 12:58 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Nov. 24, 2015, 12:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Mike Yoder]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108347
---



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
 (line 84)


Please don't put delegation tokens into the log.



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
 (line 104)


Ditto above - delegation token in the log.


- Mike Yoder


On Nov. 24, 2015, 12:58 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Nov. 24, 2015, 12:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40735: SQOOP-2715: Sqoop2: The test case ShowJobInOrderTest always failed

[Colin Ma]

> On Nov. 30, 2015, 5:07 p.m., Jarek Cecho wrote:
> > It seems that we've added the test ShowJobInOrderTest back in SQOOP-2398, 
> > but sadly the description of the JIRA is very sparse.  I'm not clear why we 
> > actually added it as we don't have any guarantees in terms of ordering on 
> > the REST interface.
> > 
> > Nevertheless it seems that the issue is that we're comparing persistence ID 
> > and we are assuming completely new database at the begging (e.g. all 
> > incremental columns to start from 1). That is incorrect assumption and 
> > comparison as a) we're droopping the ID's completely and b) in case that 
> > we're running the test case against real database (MySQL, PostgreSQL) it 
> > will never be true.
> > 
> > So, what about fixing it properly? Rather then keeing the now-not-public 
> > Ids what about setting jobName() properly and compare that instead?

Thanks for the comments. I agree that we should compare the job by jobName, and 
this is implemented in SQOOP-2690.
The root cause for this problem is InformalObjectNameTest, ShowJobInOrderTest, 
SubmissionWithDisabledModelObjectsTest shared the same 
DatabaseInfrastructureProvider. The InformalObjectNameTest is always executed 
first(29 jobs will be created), and this make the ShowJobInOrderTest failed. 
Even compare with the name(already included in SQOOP-2690), the 
ShowJobInOrderTest will be still failed.
Clear DB for after the test can solve the problem, I think maybe afterClass 
will be better than afterMethod.


- Colin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40735/#review108325
---


On Nov. 26, 2015, 8:14 a.m., Colin Ma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40735/
> ---
> 
> (Updated Nov. 26, 2015, 8:14 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> ShowJobInOrderTest always thrown the exception during the test, because the 
> DatabaseInfrastructureProvider is shared with all tests case in one suite.
> 
> 
> Diffs
> -
> 
>   test/src/main/java/org/apache/sqoop/test/infrastructure/SqoopTestCase.java 
> 4c5d3a8 
>   
> test/src/test/java/org/apache/sqoop/integration/server/ShowJobInOrderTest.java
>  cbf1e90 
> 
> Diff: https://reviews.apache.org/r/40735/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Colin Ma
> 
>

[jira] [Updated] (SQOOP-2709) Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Jarek Jarcec Cecho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Jarcec Cecho updated SQOOP-2709:
--
Attachment: SQOOP-2709.patch

> Sqoop2: HDFS: Impersonation on secured cluster doesn't work
> ---
>
> Key: SQOOP-2709
> URL: https://issues.apache.org/jira/browse/SQOOP-2709
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 1.99.7
>
> Attachments: SQOOP-2709.patch, SQOOP-2709.patch
>
>
> Using HDFS connector on secured cluster currently doesn't work with following 
> exception:
> {code}
> 2015-11-19 13:24:30,624 [OutputFormatLoader-consumer] ERROR 
> org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor  - Error while loading 
> data out of MR job.
> org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0005:Error 
> occurs during loader run
>   at org.apache.sqoop.connector.hdfs.HdfsLoader$1.run(HdfsLoader.java:119)
>   at org.apache.sqoop.connector.hdfs.HdfsLoader$1.run(HdfsLoader.java:60)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at org.apache.sqoop.connector.hdfs.HdfsLoader.load(HdfsLoader.java:60)
>   at org.apache.sqoop.connector.hdfs.HdfsLoader.load(HdfsLoader.java:44)
>   at 
> org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:267)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed on local exception: 
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed 
> [Caused by GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)]; Host Details : local host is: 
> "sqoopkrb-4.vpc.cloudera.com/172.28.211.196"; destination host is: 
> "sqoopkrb-1.vpc.cloudera.com":8020; 
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1476)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1403)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>   at com.sun.proxy.$Proxy15.create(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:295)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>   at com.sun.proxy.$Proxy16.create(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1867)
>   at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1737)
>   at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1662)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:404)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:400)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:400)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:343)
>   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:917)
>   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:898)
>   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:795)
>   at 
> org.apache.sqoop.connector.hdfs.hdfsWriter.HdfsTextWriter.initialize(HdfsTextWriter.java:40)
>   at org.apache.sqoop.connector.hdfs.HdfsLoader$1.run(HdfsLoader.java:93)
>   ... 12 more
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS 
> initiate failed [Caused by GSSException: No valid credentials provided 
> (Mechanism level: Failed to find any Kerberos tgt)]
>   at 

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Jarek Cecho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/
---

(Updated Dec. 1, 2015, 12:25 a.m.)


Review request for Sqoop.


Changes
---

Incorporated feedback.


Bugs: SQOOP-2709
https://issues.apache.org/jira/browse/SQOOP-2709


Repository: sqoop-sqoop2


Description
---

I've provided util class that can retrieve delegation token for "current" user 
and store it in our Context that is passed to execution engine.


Diffs (updated)
-

  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
 39ee4a3 
  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
 583acdd 
  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
 be837ca 
  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
 04acd18 
  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
 998b903 
  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
 2bad23a 
  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
 5856371 
  
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
 PRE-CREATION 
  
connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/40625/diff/


Testing
---

I've tested the patch on secured real cluster to make sure that it's working. 
Sadly I did not included any integration test as our suite currently doesn't 
have any support for MiniKDC (this is something that we will add later).


Thanks,

Jarek Cecho

[jira] [Commented] (SQOOP-2692) Sqoop2: Drop Id from shell

[Colin Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032873#comment-15032873
 ] 

Colin Ma commented on SQOOP-2692:
-

[~jarcec], it make sense to use--name in shell. For create link, I prefer using 
--connector, or other proper option name to avoid the confusing.

> Sqoop2: Drop Id from shell
> --
>
> Key: SQOOP-2692
> URL: https://issues.apache.org/jira/browse/SQOOP-2692
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Jarek Jarcec Cecho
>Assignee: Colin Ma
> Fix For: 1.99.7
>
>
> I've noticed that we're displaying the ID in bunch of occasions, for example 
> in tables:
> {code}
> sqoop:000> show link
> +++--++-+
> | Id |Name| Connector Id | Connector Name | Enabled |
> +++--++-+
> | 2  | with space | 6| generic-jdbc-connector | true|
> +++--++-+
> sqoop:000> delete link -l 2
> sqoop:000> show link
> ++--+--++-+
> | Id | Name | Connector Id | Connector Name | Enabled |
> ++--+--++-+
> ++--+--++-+
> {code}
> I think that we should drop it since we're dropping it from public interfaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (SQOOP-2717) Sqoop2: Use connector name in shell

[Colin Ma (JIRA)]

Colin Ma created SQOOP-2717:
---

 Summary: Sqoop2: Use connector name in shell 
 Key: SQOOP-2717
 URL: https://issues.apache.org/jira/browse/SQOOP-2717
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Colin Ma
Assignee: Colin Ma


The connector id is removed from public interface, the shell should use 
connector name instead of connector id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Abraham Fine]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108409
---



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
 (line 82)


if this path is on s3, we end up putting an empty list of tokens in 
credentials. which causes an exception when deserializing the token.


- Abraham Fine


On Nov. 24, 2015, 12:58 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Nov. 24, 2015, 12:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

[jira] [Created] (SQOOP-2716) Sqoop2: Doc: Create special chapter about all the ways how to load jars

[Jarek Jarcec Cecho (JIRA)]

Jarek Jarcec Cecho created SQOOP-2716:
-

 Summary: Sqoop2: Doc: Create special chapter about all the ways 
how to load jars
 Key: SQOOP-2716
 URL: https://issues.apache.org/jira/browse/SQOOP-2716
 Project: Sqoop
  Issue Type: Sub-task
Reporter: Jarek Jarcec Cecho
 Fix For: 1.99.7


>From [review discussion|https://reviews.apache.org/r/40576/] - we should 
>properly document all the various ways we have to add jars to Sqoop classpath. 
>This work is currently blocked on finishing connector classpath isolation in 
>SQOOP-2574.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40576: SQOOP-2708 Sqoop2: Docs: Update Installation.rst

[Jarek Cecho]

> On Nov. 30, 2015, 6:50 p.m., Abraham Fine wrote:
> > docs/src/site/sphinx/admin/Installation.rst, line 97
> > 
> >
> > I would describe the use case here in a little bit more detail. We have 
> > a few different ways of loading jars into sqoop 2. Perhaps we should move 
> > this out of the installation documentation into its own page to keep 
> > initial installation simple.
> 
> Jarek Cecho wrote:
> True, we also have ability to configure jars in jobs and such. Would you 
> mind if I'll keep it this way for now and add a standalone section about 
> classpath later? (especially after Dian will be done with connector classpath 
> isolation project).
> 
> Abraham Fine wrote:
> ok, let's make a jira for that so we do not forget about it.

Agreed: https://issues.apache.org/jira/browse/SQOOP-2716


- Jarek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40576/#review108352
---


On Nov. 30, 2015, 8:31 p.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40576/
> ---
> 
> (Updated Nov. 30, 2015, 8:31 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2708
> https://issues.apache.org/jira/browse/SQOOP-2708
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've updated the installation guide and incoporated changes that were missing.
> 
> 
> Diffs
> -
> 
>   docs/src/site/sphinx/admin/Installation.rst 9d56875 
>   docs/src/site/sphinx/admin/Tools.rst fb0187a 
> 
> Diff: https://reviews.apache.org/r/40576/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Jarek Cecho]

> On Nov. 30, 2015, 8:51 p.m., Mike Yoder wrote:
> >

Thanks for the review Mike!

Good point, we definitely don't want to be leaking sensitive data. I've 
actually got the idea of logging the delegation token from Hadoop code base 
where it seems we're doing that a lot. I had the same concern, but it seems 
because the toString() method generates only high level summary of the DT 
without any secrets. Here is an example log line:

Kind: HDFS_DELEGATION_TOKEN, Service: 172.28.214.13:8020, Ident: 
(HDFS_DELEGATION_TOKEN token 91 for sqoop2)

So I feel that this should be fine.


- Jarek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108347
---


On Nov. 24, 2015, 12:58 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Nov. 24, 2015, 12:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Aaron Myers]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108412
---

Ship it!


Looks good to me, pending addressing my one comment (and Mike's other comment 
re: keeping the DTs out of the logs.)


connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
 (line 84)


+1 to Mike's comment. DTs in a log is no bueno.


- Aaron Myers


On Nov. 24, 2015, 12:58 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Nov. 24, 2015, 12:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Abraham Fine]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108426
---

Ship it!


Ship It!

- Abraham Fine


On Dec. 1, 2015, 12:25 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Dec. 1, 2015, 12:25 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

[jira] [Commented] (SQOOP-2709) Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Sqoop QA bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032795#comment-15032795
 ] 

Sqoop QA bot commented on SQOOP-2709:
-

Testing file 
[SQOOP-2709.patch|https://issues.apache.org/jira/secure/attachment/12774912/SQOOP-2709.patch]
 against branch sqoop2 took 0:00:54.322610.

{color:red}Overall:{color} -1 due to an error(s), see details below:

{color:green}SUCCESS:{color} Clean was successful
{color:green}SUCCESS:{color} Patch applied correctly
{color:green}SUCCESS:{color} Patch add/modify test case
{color:green}SUCCESS:{color} License check passed
{color:red}ERROR:{color} failed to build with patch (exit code 1, 
[report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1940/artifact/patch-process/install.txt])

Console output is available 
[here|https://builds.apache.org/job/PreCommit-SQOOP-Build/1940/console].

This message is automatically generated.

> Sqoop2: HDFS: Impersonation on secured cluster doesn't work
> ---
>
> Key: SQOOP-2709
> URL: https://issues.apache.org/jira/browse/SQOOP-2709
> Project: Sqoop
>  Issue Type: Bug
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
> Fix For: 1.99.7
>
> Attachments: SQOOP-2709.patch, SQOOP-2709.patch
>
>
> Using HDFS connector on secured cluster currently doesn't work with following 
> exception:
> {code}
> 2015-11-19 13:24:30,624 [OutputFormatLoader-consumer] ERROR 
> org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor  - Error while loading 
> data out of MR job.
> org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0005:Error 
> occurs during loader run
>   at org.apache.sqoop.connector.hdfs.HdfsLoader$1.run(HdfsLoader.java:119)
>   at org.apache.sqoop.connector.hdfs.HdfsLoader$1.run(HdfsLoader.java:60)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>   at org.apache.sqoop.connector.hdfs.HdfsLoader.load(HdfsLoader.java:60)
>   at org.apache.sqoop.connector.hdfs.HdfsLoader.load(HdfsLoader.java:44)
>   at 
> org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:267)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed on local exception: 
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed 
> [Caused by GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)]; Host Details : local host is: 
> "sqoopkrb-4.vpc.cloudera.com/172.28.211.196"; destination host is: 
> "sqoopkrb-1.vpc.cloudera.com":8020; 
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1476)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1403)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>   at com.sun.proxy.$Proxy15.create(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:295)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>   at com.sun.proxy.$Proxy16.create(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1867)
>   at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1737)
>   at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1662)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:404)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:400)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> 

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Dian Fu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108429
---

Ship it!


Ship It!

- Dian Fu


On Dec. 1, 2015, 12:25 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Dec. 1, 2015, 12:25 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
>  998b903 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
>  2bad23a 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
>  5856371 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java
>  PRE-CREATION 
>   
> connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40625/diff/
> 
> 
> Testing
> ---
> 
> I've tested the patch on secured real cluster to make sure that it's working. 
> Sadly I did not included any integration test as our suite currently doesn't 
> have any support for MiniKDC (this is something that we will add later).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>

Re: Review Request 40625: SQOOP-2709 Sqoop2: HDFS: Impersonation on secured cluster doesn't work

[Jarek Cecho]

> On Nov. 30, 2015, 5:58 p.m., Abraham Fine wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java,
> >  line 62
> > 
> >
> > creating the proxy user and then generating the delegation tokens is a 
> > pattern that we use twice. once in the hdfstoinitializer and again in the 
> > hdfsfrominitializer.
> > 
> > perhaps, it could make sense to do something similar to what is being 
> > done when we load the delegation tokens 
> > (`createProxyUserAndLoadDelegationTokens`) and have a method 
> > `createProxyUserAndGenerateDelegationTokens`). that way we can make 
> > `generateDelegationTokens` private, and never have to worry about it being 
> > called outside of a delegation block.

I've noticed that as well and I was thinking about creating a 
createProxyUserAndGenerateDelegationTokens. The reasons why I did not do that 
eventually were:

1) The method generateDelegationTokens has to run under to doAs and I didn't 
want to have two doAs blocks (one inside the 
createProxyUserAndGenerateDelegationTokens() and second in the Initializers) - 
this one is kind of subjective, but the objective one is:

2) We're doing bunch of checks inside the initializer that should be done 
before we attemt to create the delegation tokens (e.g. checking if given path 
exists and such). I don't want us to generate error message while generating 
the delegation tokes if the HDFS URL or any other related configs are invalid. 
That should fail as a first class failure and not hidden in delegation token 
creation.


> On Nov. 30, 2015, 5:58 p.m., Abraham Fine wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java,
> >  line 55
> > 
> >
> > i understand that we used `getLoginUser` previously. is this the 
> > correct way to go instead of using `getCurrentUser`?

For our use case it doesn't matter because getLoginUser and getCurrentUser 
should return the same username. I feel that getLoginUser() is a better and 
cleaner because that should always gave us the username of a user who owns the 
kerberos ticket (the user that owns the process) whereas getCurrentUser might 
be differnt if we're running under the doAs() already.


> On Nov. 30, 2015, 5:58 p.m., Abraham Fine wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/security/SecurityUtils.java,
> >  line 84
> > 
> >
> > how verbose is this, maybe it should be a debug?

I find it super helpful, so I would like to keep it at info.


> On Nov. 30, 2015, 5:58 p.m., Abraham Fine wrote:
> > connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/security/TestSecurityUtils.java,
> >  line 27
> > 
> >
> > can we test writing the tokens to and reading them from the context?

I can add such test, but I feel that it won't add much value. We're already 
verifying that we can reconstruct the token given serialized string and adding 
context in/out would be just verifying that we will retrieve stored data which 
we already have tests for.

We should eventually add a real integration tests on kerberos that will test 
this functionality end to end, but that is currently blocked by SQOOP-2704 that 
Dian is working on.


- Jarek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40625/#review108340
---


On Nov. 24, 2015, 12:58 a.m., Jarek Cecho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40625/
> ---
> 
> (Updated Nov. 24, 2015, 12:58 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2709
> https://issues.apache.org/jira/browse/SQOOP-2709
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> ---
> 
> I've provided util class that can retrieve delegation token for "current" 
> user and store it in our Context that is passed to execution engine.
> 
> 
> Diffs
> -
> 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java
>  39ee4a3 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
>  583acdd 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsFromInitializer.java
>  be837ca 
>   
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java
>  04acd18 
>   
> 

FOSDEM 2016 - take action by 4th of December 2015

[Roman Shaposhnik]

As most of you probably know FOSDEM 2016 (the biggest,
100% free open source developer conference) is right 
around the corner:
   https://fosdem.org/2016/

We hope to have an ASF booth and we would love to see as
many ASF projects as possible present at various tracks
(AKA Developer rooms):
   https://fosdem.org/2016/schedule/#devrooms

This year, for the first time, we are running a dedicated
Big Data and HPC Developer Room and given how much of that
open source development is done at ASF it would be great
to have folks submit talks to:
   https://hpc-bigdata-fosdem16.github.io

While the CFPs for different Developer Rooms follow slightly 
different schedules, but if you submit by the end of this week 
you should be fine.

Finally if you don't want to fish for CFP submission URL,
here it is:
   https://fosdem.org/submit

If you have any questions -- please email me *directly* and
hope to see as many of you as possible in two months! 

Thanks,
Roman.

[GitHub] sqoop pull request:

[szvic]

Github user szvic commented on the pull request:


https://github.com/apache/sqoop/commit/c6627c04cdbc048a193d271c6e94023f921db4e3#commitcomment-14700803
  
great!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---