Hi Dan, I don't think the SQL Client officially supports running against Kubernetes. What you could try is using an undocumented, untested feature: Put something like jobmanager: kubernetes into the "deployment:" section of the Sql Client configuration <https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sqlClient.html#environment-files> . Proper support for Kubernetes, YARN etc. is (probably stalled) work in progress <https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sqlClient.html#limitations--future> .
On Mon, Sep 14, 2020 at 2:18 PM Arvid Heise <ar...@ververica.com> wrote: > Hi Dan, > > Can you verify from the pod that jobmanager and *10.98.253.58:8081 > <http://10.98.253.58:8081> *is actually accessible (e.g., with curl)? I'd > probably also try out localhost:8081 as you are connecting to the > respective pod directly. > > On Fri, Sep 11, 2020 at 9:59 PM Dan Hill <quietgol...@gmail.com> wrote: > >> Hi Robert! >> >> I have Flink running locally on minikube. I'm running SQL client using >> exec on the jobmanager. >> >> kubectl exec pod/flink-jobmanager-0 -i -t -- /opt/flink/bin/sql-client.sh >> embedded -e /opt/flink/sql-client-defaults.yaml >> >> Here's the sql-client-defaults.yaml. I didn't specify a session. >> execution: >> type: batch >> result-mode: table >> max-table-result-rows: 1000000 >> >> I'm prototyping the Table SQL interface. I got blocked using the Table >> SQL interface and figured I'd try the SQL Client to see if I could get >> unblocked. >> >> >> On Fri, Sep 11, 2020 at 11:18 AM Robert Metzger <rmetz...@apache.org> >> wrote: >> >>> Hi Dan, >>> >>> the notation of "flink-jobmanager/10.98.253.58:8081" is not a problem. >>> It is how java.net.InetAddress stringifies a resolved address (with >>> both hostname and IP). >>> >>> How did you configure the SQL client to work with a Kubernetes Session? >>> Afaik this is not a documented, tested and officially supported feature >>> (this doesn't mean we should not support it -- apparently it is something >>> we should do rather soon ;) ). >>> >>> Best, >>> Robert >>> >>> On Fri, Sep 11, 2020 at 5:25 AM Dan Hill <quietgol...@gmail.com> wrote: >>> >>>> I just tried using the Flink SQL Client. A simple job is not running >>>> because it cannot hit jobmanager. I'm not sure why Flink SQL Client is >>>> hitting "flink-jobmanager/10.98.253.58:8081". I'd expect either >>>> "flink-jobmanager:8081" or "10.98.253.58:8081" (which should work with >>>> my kubernetes setup). >>>> >>>> I'm using riskfocus's Flink helm chart >>>> <https://github.com/riskfocus/helm-charts-public/tree/master/flink>. >>>> >>>> The last SELECT errors out. I have an environment file that indicates >>>> this "execution.type: batch". My setup works when using DataSet and >>>> DataStream. The jobmanager and taskmanager logs look fine. This seems >>>> like a weird configuration with SQL Client that is either broken with that >>>> Flink helm chart or with SQL Client. >>>> >>>> >>>> Flink SQL> DROP TABLE `default_catalog.mydb.user`; >>>> >>>> [INFO] Table has been removed. >>>> >>>> >>>> Flink SQL> CREATE TABLE `default_catalog.mydb.user` (`platformId` >>>> BIGINT, `userId` STRING) WITH ('connector' = 'filesystem', 'path' = >>>> 's3://mys3bucket/users.csv','format' = 'csv'); >>>> >>>> [INFO] Table has been created. >>>> >>>> >>>> Flink SQL> SELECT * FROM `default_catalog.mydb.user` LIMIT 10; >>>> >>>> *[ERROR] Could not execute SQL statement. Reason:* >>>> >>>> *org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: >>>> connection timed out: flink-jobmanager/10.98.253.58:8081 >>>> <http://10.98.253.58:8081>* >>>> >>>> My config logs from jobmanager. >>>> >>>> 2020-09-11 02:33:07,962 INFO >>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>>> [] - Using predefined options: DEFAULT. >>>> >>>> 2020-09-11 02:33:07,962 INFO >>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>>> [] - Using default options factory: >>>> DefaultConfigurableOptionsFactory{configuredOptions={}}. >>>> >>>> 2020-09-11 02:33:08,100 INFO >>>> org.apache.flink.table.client.gateway.local.ProgramDeployer [] - >>>> Submitting job org.apache.flink.streaming.api.graph.StreamGraph@3bd08435 >>>> for query default: SELECT * FROM `default_catalog.mydb.user` LIMIT 10` >>>> >>>> 2020-09-11 02:33:08,112 INFO org.apache.flink.configuration.Configuration >>>> [] - Config uses fallback configuration key >>>> 'jobmanager.rpc.address' instead of key 'rest.address' >>>> >>>> tail: log/flink--sql-client-flink-jobmanager-0.log: file truncated >>>> >>>> 2020-09-11 02:34:35,848 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.numberOfTaskSlots, 2 >>>> >>>> 2020-09-11 02:34:35,884 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: blob.server.port, 6124 >>>> >>>> 2020-09-11 02:34:35,884 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: taskmanager.rpc.port, >>>> 6122 >>>> >>>> 2020-09-11 02:34:35,885 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: jobmanager.heap.size, 1g >>>> >>>> 2020-09-11 02:34:35,886 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.memory.process.size, 1g >>>> >>>> 2020-09-11 02:34:35,886 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.backend, rocksdb >>>> >>>> 2020-09-11 02:34:35,886 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.checkpoints.dir, >>>> file:///flink_state/checkpoints >>>> >>>> 2020-09-11 02:34:35,887 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.savepoints.dir, >>>> file:///flink_state/savepoints >>>> >>>> 2020-09-11 02:34:35,887 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.backend.async, >>>> true >>>> >>>> 2020-09-11 02:34:35,887 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.fs.memory-threshold, 1024 >>>> >>>> 2020-09-11 02:34:35,888 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.fs.write-buffer-size, 4096 >>>> >>>> 2020-09-11 02:34:35,888 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.incremental, true >>>> >>>> 2020-09-11 02:34:35,888 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.local-recovery, true >>>> >>>> 2020-09-11 02:34:35,889 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.checkpoints.num-retained, 1 >>>> >>>> 2020-09-11 02:34:35,889 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.state.local.root-dirs, file:///flink_state/local-recovery >>>> >>>> 2020-09-11 02:34:35,889 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.checkpoint.transfer.thread.num, 1 >>>> >>>> 2020-09-11 02:34:35,890 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.localdir, /flink_state/rocksdb >>>> >>>> 2020-09-11 02:34:35,890 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.options-factory, >>>> org.apache.flink.contrib.streaming.state.DefaultConfigurableOptionsFactory >>>> >>>> 2020-09-11 02:34:35,890 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.predefined-options, DEFAULT >>>> >>>> 2020-09-11 02:34:35,891 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.timer-service.factory, HEAP >>>> >>>> 2020-09-11 02:34:35,891 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.ttl.compaction.filter.enabled, false >>>> >>>> 2020-09-11 02:34:35,892 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: jobmanager.rpc.address, >>>> flink-jobmanager >>>> >>>> 2020-09-11 02:34:35,892 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: jobmanager.rpc.port, >>>> 6123 >>>> >>>> 2020-09-11 02:34:35,892 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.memory.jvm-metaspace.size, 256mb >>>> >>>> 2020-09-11 02:34:35,893 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.endpoint, >>>> http://minio:9000 >>>> >>>> 2020-09-11 02:34:35,893 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.path.style.access, >>>> true >>>> >>>> 2020-09-11 02:34:35,894 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.access-key, >>>> YOURACCESSKEY >>>> >>>> 2020-09-11 02:34:35,894 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.secret-key, ****** >>>> >>>> 2020-09-11 02:34:35,895 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> s3a.aws.credentials.provider, >>>> org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider >>>> >>>> 2020-09-11 02:34:36,492 INFO org.apache.flink.client.cli.CliFrontend >>>> [] - Loading FallbackYarnSessionCli >>>> >>>> 2020-09-11 02:34:36,498 INFO >>>> org.apache.flink.table.client.gateway.local.LocalExecutor [] - >>>> Using default environment file: >>>> file:/opt/flink/conf/sql-client-defaults.yaml >>>> >>>> 2020-09-11 02:34:37,786 INFO org.apache.flink.table.client.SqlClient >>>> [] - Using session environment file: >>>> file:/opt/flink/sql-client-defaults.yaml >>>> >>>> 2020-09-11 02:34:37,931 INFO >>>> org.apache.flink.table.client.config.entries.ExecutionEntry [] - >>>> Property 'execution.restart-strategy.type' not specified. Using default >>>> value: fallback >>>> >>>> 2020-09-11 02:34:43,269 INFO >>>> org.apache.flink.table.client.gateway.local.ExecutionContext >>>> [] - Executor config: {blob.server.port=6124, s3a.secret-key=YOURSECRETKEY, >>>> state.checkpoints.num-retained=1, s3a.access-key=YOURACCESSKEY, >>>> state.backend.rocksdb.options-factory=org.apache.flink.contrib.streaming.state.DefaultConfigurableOptionsFactory, >>>> jobmanager.rpc.address=flink-jobmanager, >>>> taskmanager.memory.jvm-metaspace.size=256mb, >>>> state.backend.rocksdb.predefined-options=DEFAULT, >>>> state.savepoints.dir=file:///flink_state/savepoints, >>>> state.backend.fs.memory-threshold=1024, >>>> state.backend.rocksdb.timer-service.factory=HEAP, >>>> execution.savepoint.ignore-unclaimed-state=false, >>>> taskmanager.numberOfTaskSlots=2, state.backend.async=true, >>>> pipeline.classpaths=[], >>>> state.backend.rocksdb.ttl.compaction.filter.enabled=false, >>>> jobmanager.heap.size=1g, taskmanager.memory.process.size=1g, s3a.endpoint= >>>> http://minio:9000, >>>> state.backend.rocksdb.checkpoint.transfer.thread.num=1, >>>> state.backend.local-recovery=true, >>>> state.backend.rocksdb.localdir=/flink_state/rocksdb, >>>> state.backend.incremental=true, execution.target=remote, >>>> taskmanager.rpc.port=6122, jobmanager.rpc.port=6123, >>>> s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, >>>> state.backend.fs.write-buffer-size=4096, execution.attached=true, >>>> execution.shutdown-on-attached-exit=false, >>>> pipeline.jars=[file:/opt/flink/opt/flink-sql-client_2.12-1.11.1.jar], >>>> state.backend=rocksdb, s3a.path.style.access=true, >>>> taskmanager.state.local.root-dirs=file:///flink_state/local-recovery, >>>> state.checkpoints.dir=file:///flink_state/checkpoints} >>>> >>>> 2020-09-11 02:34:43,286 INFO >>>> org.apache.flink.client.deployment.DefaultClusterClientServiceLoader >>>> [] - Could not load factory due to missing dependencies. >>>> >>>> 2020-09-11 02:34:44,079 INFO >>>> org.apache.flink.table.client.cli.CliClient [] - >>>> Command history file path: /root/.flink-sql-history >>>> >>>> 2020-09-11 02:35:15,396 INFO >>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>>> [] - Using predefined options: DEFAULT. >>>> >>>> 2020-09-11 02:35:15,397 INFO >>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>>> [] - Using default options factory: >>>> DefaultConfigurableOptionsFactory{configuredOptions={}}. >>>> >>>> 2020-09-11 02:35:17,529 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.numberOfTaskSlots, 2 >>>> >>>> 2020-09-11 02:35:17,530 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: blob.server.port, 6124 >>>> >>>> 2020-09-11 02:35:17,530 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: taskmanager.rpc.port, >>>> 6122 >>>> >>>> 2020-09-11 02:35:17,530 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: jobmanager.heap.size, 1g >>>> >>>> 2020-09-11 02:35:17,530 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.memory.process.size, 1g >>>> >>>> 2020-09-11 02:35:17,530 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.backend, rocksdb >>>> >>>> 2020-09-11 02:35:17,530 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.checkpoints.dir, >>>> file:///flink_state/checkpoints >>>> >>>> 2020-09-11 02:35:17,531 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.savepoints.dir, >>>> file:///flink_state/savepoints >>>> >>>> 2020-09-11 02:35:17,531 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: state.backend.async, >>>> true >>>> >>>> 2020-09-11 02:35:17,531 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.fs.memory-threshold, 1024 >>>> >>>> 2020-09-11 02:35:17,531 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.fs.write-buffer-size, 4096 >>>> >>>> 2020-09-11 02:35:17,531 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.incremental, true >>>> >>>> 2020-09-11 02:35:17,531 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.local-recovery, true >>>> >>>> 2020-09-11 02:35:17,532 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.checkpoints.num-retained, 1 >>>> >>>> 2020-09-11 02:35:17,532 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.state.local.root-dirs, file:///flink_state/local-recovery >>>> >>>> 2020-09-11 02:35:17,532 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.checkpoint.transfer.thread.num, 1 >>>> >>>> 2020-09-11 02:35:17,532 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.localdir, /flink_state/rocksdb >>>> >>>> 2020-09-11 02:35:17,532 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.options-factory, >>>> org.apache.flink.contrib.streaming.state.DefaultConfigurableOptionsFactory >>>> >>>> 2020-09-11 02:35:17,532 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.predefined-options, DEFAULT >>>> >>>> 2020-09-11 02:35:17,532 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.timer-service.factory, HEAP >>>> >>>> 2020-09-11 02:35:17,533 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> state.backend.rocksdb.ttl.compaction.filter.enabled, false >>>> >>>> 2020-09-11 02:35:17,533 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: jobmanager.rpc.address, >>>> flink-jobmanager >>>> >>>> 2020-09-11 02:35:17,533 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: jobmanager.rpc.port, >>>> 6123 >>>> >>>> 2020-09-11 02:35:17,533 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> taskmanager.memory.jvm-metaspace.size, 256mb >>>> >>>> 2020-09-11 02:35:17,533 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.endpoint, >>>> http://minio:9000 >>>> >>>> 2020-09-11 02:35:17,534 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.path.style.access, >>>> true >>>> >>>> 2020-09-11 02:35:17,534 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.access-key, >>>> YOURACCESSKEY >>>> >>>> 2020-09-11 02:35:17,534 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: s3a.secret-key, ****** >>>> >>>> 2020-09-11 02:35:17,535 INFO >>>> org.apache.flink.configuration.GlobalConfiguration >>>> [] - Loading configuration property: >>>> s3a.aws.credentials.provider, >>>> org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider >>>> >>>> 2020-09-11 02:35:18,047 INFO >>>> org.apache.flink.table.client.gateway.local.ProgramDeployer [] - >>>> Submitting job org.apache.flink.streaming.api.graph.StreamGraph@2f95653f >>>> for query default: SELECT * FROM `default_catalog.mydb.user` LIMIT 10` >>>> >>>> 2020-09-11 02:35:18,382 INFO org.apache.flink.configuration.Configuration >>>> [] - Config uses fallback configuration key >>>> 'jobmanager.rpc.address' instead of key 'rest.address' >>>> >>>> 2020-09-11 02:41:39,539 WARN org.apache.flink.runtime.rest.RestClient >>>> [] - Rest endpoint shutdown failed. >>>> >>>> java.util.concurrent.TimeoutException: null >>>> >>> > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
