[
https://issues.apache.org/jira/browse/FLINK-30326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644312#comment-17644312
]
Alex Sorokoumov edited comment on FLINK-30326 at 12/7/22 12:57 PM:
-------------------------------------------------------------------
I used the following example to conclude that the SQL client creates the table
structure.
h2. Setup
I used a docker-compose with 1 JM and 1 TM that do not have shared volumes as
well as an SQL client started on the JM.
_NOTE_ I used the local file system for the warehouse to find out what
component was responsible for creation of what directories. I understand that
it is not the production-ready setup.
h3. Step 1: Create FTS catalog and schema via a SQL client on JM and see where
the schema files appear
{noformat}
Flink SQL> CREATE CATALOG my_catalog WITH (
> 'type'='table-store',
> 'warehouse'='file:/tmp/table_store'
> );
[INFO] Execute statement succeed.Flink SQL> USE CATALOG my_catalog;
>
Flink SQL> CREATE TABLE word_count (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> );
[INFO] Execute statement succeed.{noformat}
h3. Step 2: Check where the directory was created
JM:
{noformat}
root@flink:/opt/flink# ls /tmp/table_store/default.db/word_count/schema/schema-0
/tmp/table_store/default.db/word_count/schema/schema-0
{noformat}
TM:
{noformat}
ls /tmp/table_store/default.db/
ls: cannot access '/tmp/table_store/default.db/': No such file or directory
{noformat}
h3. Step 3: Do a single insert and see where the new files appear
{noformat}
Flink SQL> INSERT INTO word_count (word, cnt) VALUES ('foo', 1);
[INFO] Submitting SQL update statement to the cluster...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner
(file:/opt/flink/lib/flink-dist-1.17-SNAPSHOT.jar) to field
java.lang.Class.ANNOTATION
WARNING: Please consider reporting this to the maintainers of
org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: c9c175d0137280aeeba83cc7b2ce8454
{noformat}
On JM: nothing changed
On TM:
{noformat}
$ ls /tmp/table_store/default.db/word_count/
bucket-0 manifest snapshot
{noformat}
was (Author: gerrrr):
I used the following example to conclude that the SQL client creates the table
structure.
h2. Setup
I used a docker-compose with 1 JM and 1 TM that do not have shared volumes as
well as an SQL client started on the JM.
h3. Step 1: Create FTS catalog and schema via a SQL client on JM and see where
the schema files appear
{noformat}
Flink SQL> CREATE CATALOG my_catalog WITH (
> 'type'='table-store',
> 'warehouse'='file:/tmp/table_store'
> );
[INFO] Execute statement succeed.Flink SQL> USE CATALOG my_catalog;
>
Flink SQL> CREATE TABLE word_count (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> );
[INFO] Execute statement succeed.{noformat}
h3. Step 2: Check where the directory was created
JM:
{noformat}
root@flink:/opt/flink# ls /tmp/table_store/default.db/word_count/schema/schema-0
/tmp/table_store/default.db/word_count/schema/schema-0
{noformat}
TM:
{noformat}
ls /tmp/table_store/default.db/
ls: cannot access '/tmp/table_store/default.db/': No such file or directory
{noformat}
h3. Step 3: Do a single insert and see where the new files appear
{noformat}
Flink SQL> INSERT INTO word_count (word, cnt) VALUES ('foo', 1);
[INFO] Submitting SQL update statement to the cluster...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner
(file:/opt/flink/lib/flink-dist-1.17-SNAPSHOT.jar) to field
java.lang.Class.ANNOTATION
WARNING: Please consider reporting this to the maintainers of
org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: c9c175d0137280aeeba83cc7b2ce8454
{noformat}
On JM: nothing changed
On TM:
{noformat}
$ ls /tmp/table_store/default.db/word_count/
bucket-0 manifest snapshot
{noformat}
> SQL Client should not create table directory structure
> ------------------------------------------------------
>
> Key: FLINK-30326
> URL: https://issues.apache.org/jira/browse/FLINK-30326
> Project: Flink
> Issue Type: Bug
> Components: Table Store
> Reporter: Alex Sorokoumov
> Priority: Major
>
> The SQL client creates the schema part of the directory structure for new
> tables. This behavior is incorrect because the SQL client could be attached
> to a SQL gateway that does not necessarily have access to DFSes for all
> catalogs/tables.
> I propose to change the behavior such that FTS sink creates the schema,
> either on the fly if `auto-create=true` or just as a separate job if
> `auto-create=false`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
