-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66300/#review206151
-----------------------------------------------------------
I've been working on the failure of
`TestHiveMiniCluster#testInsertedRowCanBeReadFromTable[KerberosAuthenticationConfiguration]`
and wanted to give an update.
I think this is the meaningful part of the quite verbose logs:
```
java.lang.Exception:
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle
in localfetcher#1
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
in shuffle in localfetcher#1
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:71)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:57)
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:125)
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
Caused by: java.lang.RuntimeException: Secure IO is not possible without native
code extensions.
at org.apache.hadoop.io.SecureIOUtils.<clinit>(SecureIOUtils.java:71)
... 6 more
```
This is happening in MR's shuffle phase. I was trying to find out how tests in
Hive don't run into this and found out that Secure MR is not supported there.
See here:
https://github.com/apache/hive/blob/dceeefbdf5e4f6fea83cb6ca5c11fbac10e77677/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java#L178-L180
I'm trying to get Tez working with our MiniHiveCluster, but did not succeed
yet. The kerberos ticket is not picked up for some reason:
```
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:755)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:718)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:811)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1383)
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy81.getAMStatus(Unknown Source)
at
org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:772)
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:909)
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:880)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:434)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:360)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.openInternal(TezSessionPoolSession.java:124)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:237)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:364)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:191)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2479)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2150)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1826)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1567)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1561)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:221)
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:313)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:326)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:613)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:409)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:798)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:794)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
... 36 more
```
- daniel voros
On March 27, 2018, 8:50 a.m., daniel voros wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66300/
> -----------------------------------------------------------
>
> (Updated March 27, 2018, 8:50 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-3305
> https://issues.apache.org/jira/browse/SQOOP-3305
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> To be able to eventually support the latest versions of Hive, HBase and
> Accumulo, we should start by upgrading our Hadoop dependencies to 3.0.0. See
> https://hadoop.apache.org/docs/r3.0.0/index.html
>
>
> Diffs
> -----
>
> ivy.xml 1f587f3e
> ivy/libraries.properties 565a8bf5
> src/java/org/apache/sqoop/SqoopOptions.java d9984af3
> src/java/org/apache/sqoop/config/ConfigurationHelper.java fb2ab031
> src/java/org/apache/sqoop/hive/HiveImport.java 5da00a74
> src/java/org/apache/sqoop/mapreduce/JobBase.java 6d1e0499
> src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a
> src/java/org/apache/sqoop/mapreduce/parquet/kite/KiteParquetUtils.java
> e68bba90
> src/java/org/apache/sqoop/util/SqoopJsonUtil.java adf186b7
> src/test/org/apache/sqoop/TestSqoopOptions.java bb7c20dd
> src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 19bb7605
>
> src/test/org/apache/sqoop/hive/minicluster/KerberosAuthenticationConfiguration.java
> 549a8c6c
>
> src/test/org/apache/sqoop/hive/minicluster/PasswordAuthenticationConfiguration.java
> 79881f7b
> src/test/org/apache/sqoop/util/TestSqoopJsonUtil.java fdf972c1
> testdata/hcatalog/conf/hive-site.xml edac7aa9
>
>
> Diff: https://reviews.apache.org/r/66300/diff/7/
>
>
> Testing
> -------
>
> Normal and third-party unit tests.
>
>
> Thanks,
>
> daniel voros
>
>
