Re:Flink 1.10连接hive时kerberos认证异常问题
Hi, 我在配置flink连接hive时，由于集群开启了Kerberos认证，经过一番探索，异常没有了。但是现在连接的时候需要我输入Kerberos用户名和密码。我理解指定了keytab文件路径后，应该不需要用户名和密码了吧？请教各位大神可能的配置问题。 security.kerberos.login.use-ticker-cache: false security.kerberos.login.keytab: /app/flink/flink-1.10.10/kerberos/flink_test.keytab security.kerberos.login.principal: flink_t...@hadoop.htsc.com At 2020-02-21 18:18:57, "sunfulin" wrote: Hi， 我使用Flink 1.10集成hive，在连接metastore的时候由于hive对应CDH集群开启了kerberos认证，抛出了如下异常：请问大家这个该怎么配置或者解决哈？ 999 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://namenode01.htsc.com:9083 1175 [main] ERROR org.apache.thrift.transport.TSaslTransport - SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:181) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.table.catalog.hive.client.HiveShimV200.getHiveMetastoreClient(HiveShimV200.java:43) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:240) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.(HiveMetastoreClientWrapper.java:71) at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35) at org.apache.flink.table.catalog.hive.HiveCatalog.open(HiveCatalog.java:188) at org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:102) at org.apache.flink.table.api.internal.TableEnvironmentImpl.registerCatalog(TableEnvironmentImpl.java:235) at com.htsc.crm_realtime.fatjob.Jobs.hive.HiveMetaJob.doJob(HiveMetaJob.java:44) at com.htsc.crm_realtime.fatjob.Jobs.JobEntryBase.run(JobEntryBase.java:50) at com.htsc.crm_realtime.fatjob.Jobs.hive.HiveMetaJob.main(HiveMetaJob.java:23) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 34 more
????flink-jdbc-driver????mysql??????(flink1.10.0)
??jdbcgatewaymysql??jdbc-drivermysql??SQL-gateway??sqlclient??tablesink??Caused
by: org.apache.flink.table.api.TableException: RetractStreamTableSink and
UpsertStreamTableSink is not supported in Batch environment. at
org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecSink.translateToPlanInternal(BatchExecSink.scala:85)
at
org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecSink.translateToPlanInternal(BatchExecSink.scala:48)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
at
org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecSink.translateToPlan(BatchExecSink.scala:48)
at
org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$translateToPlan$1.apply(BatchPlanner.scala:69)
at
org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$translateToPlan$1.apply(BatchPlanner.scala:68)
Connection connection =
DriverManager.getConnection("jdbc:flink://dataflow1:8083?planner=blink");
Statement statement = connection.createStatement();
sql = "CREATE TABLE t_user_target (\n" +
" id BIGINT,\n" +
" username VARCHAR\n" +
") WITH (\n" +
" 'connector.type' = 'jdbc',\n" +
" 'connector.driver' = 'com.mysql.jdbc.Driver',\n" +
" 'connector.url' = 'jdbc:mysql://172.18.100.85:3306/targetdb',\n" +
" 'connector.table' = 't1target',\n" +
" 'connector.username' = 'root',\n" +
" 'connector.password' = 'root',\n" +
" 'connector.write.flush.max-rows' = '5000'\n" +
");";
statement.executeUpdate(sql);
statement.execute("insert into t_user_target values(1,'fan'),(2,'ss')");
Re: Flink读写kafka数据聚集任务失败问题
看到异常信息 - Closing TaskExecutor connection container_1578492316659_0830_01_06 because: Container [pid=30031,containerID=container_1578492316659_0830_01_06] is running beyond physical memory limits. Current usage: 10.0 GB of 10 GB physical memory used; 11.8 GB of 21 GB virtual memory used. Killing container. 应该是超内存了，容器被 kill 了 chanamper 于 2020年2月24日周一 上午10:33写道： > > 大家好，请教一下，flink任务读取kafka数据进行聚集操作后将结果写回kafka，flink版本为1.8.0。任务运行一段时间后出现如下异常，之后flink任务异常挂掉，请问一下这个问题该如何解决呢？多谢 > > 2020-02-19 10:45:45,314 ERROR > org.apache.flink.runtime.io.network.netty.PartitionRequestQueue > - Encountered error while consuming partitions > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) > at > org.apache.flink.shaded.netty4.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) > at > org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) > at > org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) > at > org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) > at java.lang.Thread.run(Thread.java:748) > 2020-02-19 10:45:45,317 INFO > org.apache.kafka.clients.producer.KafkaProducer - [Producer > clientId=producer-1] Closing the Kafka producer with timeoutMillis = > 9223372036854775807 ms. > 2020-02-19 10:45:45,412 INFO > org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Memory > usage stats: [HEAP: 98/6912/6912 MB, NON HEAP: 81/83/-1 MB > (used/committed/max)] > > 2020-02-19 10:45:45,413 INFO > org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Direct > memory stats: Count: 24596, Total Capacity: 806956211, Used Memory: > 806956212 > > > > > > > 2020-02-19 10:50:31,351 WARN akka.remote.transport.netty.NettyTransport > - Remote connection to [null] failed with > java.net.ConnectException: Connection refused: aj-flinknode01/ > 9.186.36.80:56983 > 2020-02-19 10:50:31,351 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system > [akka.tcp://flink@aj-flinknode01:56983] has failed, address is now gated > for [50] ms. Reason: [Association failed with > [akka.tcp://flink@aj-flinknode01:56983]] Caused by: [Connection refused: > aj-flinknode01/9.186.36.80:56983] > 2020-02-19 10:50:55,419 WARN akka.remote.ReliableDeliverySupervisor > - Association with remote system > [akka.tcp://flink@aj-flinknode01:45703] has failed, address is now gated > for [50] ms. Reason: [Disassociated] > 2020-02-19 10:50:56,370 INFO org.apache.flink.yarn.YarnResourceManager > - Closing TaskExecutor connection > container_1578492316659_0830_01_06 because: Container > [pid=30031,containerID=container_1578492316659_0830_01_06] is running > beyond physical memory limits. Current usage: 10.0 GB of 10 GB physical > memory used; 11.8 GB of 21 GB virtual memory used. Killing container. > Dump of the process-tree for container_1578492316659_0830_01_06 : > |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > |- 30068 30031 30031 30031 (java) 277668 18972 12626370560 2630988 > /data/jdk1.8.0_211/bin/java -Xms6912m -Xmx6912m > -XX:MaxDirectMemorySize=3328m -XX:+UseG1GC > -Dlog.file=/data/hadoop-2.6.0-cdh5.16.1/logs/userlogs/application_1578492316659_0830/container_1578492316659_0830_01_06/taskmanager.log > -Dlogback.configurationFile=file:./logback.xml > -Dlog4j.configuration=file:./log4j.properties > org.apache.flink.yarn.YarnTaskExecutorRunner --configDir . > |- 30031 30027 30031 30031 (bash) 0 0 11001856 329 /bin/bash -c > /data/jdk1.8.0_211/bin/java -Xms6912m -Xmx6912m > -XX:MaxDirectMemorySize=3328m -XX:+UseG1GC >
Re: 1.10 SqlClient启动报错
hi 邮件里的图片看不到，麻烦重新提供一下图片。或者直接把错误消息贴出来 1 于2020年2月24日周一 下午9:39写道： > Hi，all > > 我在linux上启动1.10的sql-client，却遇到了如下错误信息： > > > > 看了一下sql-client.sh脚本需要在FLINK_OPT_DIR路径中找到Sql-client.jar，我就手动export了一下FLINK_OPT_DIR，但还是报上面的错，与此同时，我启动1.9版本的sql-client则可以正常启动，感觉这个问题很诡异，麻烦请大家帮忙看看 > > > >
1.10 SqlClient启动报错
Hi，all 我在linux上启动1.10的sql-client，却遇到了如下错误信息： 看了一下sql-client.sh脚本需要在FLINK_OPT_DIR路径中找到Sql-client.jar，我就手动export了一下FLINK_OPT_DIR，但还是报上面的错，与此同时，我启动1.9版本的sql-client则可以正常启动，感觉这个问题很诡异，麻烦请大家帮忙看看
