Re: Re: INSERT INTO will work faster in Flink than in regular database?

I was thinking if I can use Flink to process large files and save result to another file or database (jdbc). So loading file into Flink table and then loop through rows inserting result to some temp Flink table. Then save result to jdbc table or file.   Sent: Thursday, September 22, 2022 at

Re: INSERT INTO will work faster in Flink than in regular database?

Thank you - I'll try. There is no 'UPDATE' clause in Flink SQL?     Sent: Monday, September 19, 2022 at 4:09 AM From: "Shengkai Fang" To: pod...@gmx.com Cc: user@flink.apache.org Subject: Re: INSERT INTO will work faster in Flink than in regular database? Hi. I think you can write a udf[1]

INSERT INTO will work faster in Flink than in regular database?

What's the most effective way (performance) to update big no of rows? Sure this will be probably "INSERT INTO table (column1) SELECT column1 FROM ...". Anyway, I do not see any "UPDATE" in Flink? But sometimes SQL is not enough. Suppose I have code:   TableResult tableResult1 =

Re: Insert into JDBC table

I see I can only insert into JDBC table with select from another table, something like:   tEnv.executeSql("INSERT INTO Customers SELECT customer_number, pid_no, name FROM another_table"); But what if I want to insert row that I created within Flink? For instance I made some calculation and I

Insert into JDBC table

Why this INSERT does not insert row in table (jdbc connection works, I can create 'Customers' table from MySQL table)?   tEnv.executeSql("CREATE TABLE Customers (" + " customer_number INT, " + " pid_no INT, " + "

Re: Why this example does not save anything to file?

Thank you. So I understand 'this is normal Flink property - live with it' :-)   M.   Sent: Wednesday, August 31, 2022 at 9:37 PM From: "David Anderson" To: pod...@gmx.com Cc: "user" Subject: Re: Why this example does not save anything to file? With the parallelism set to 2, you will get 2

Re: Why this example does not save anything to file?

Doesn't it depends on 'sink.parallelism'? If I set 'sink.parallelism' = '2' I get two files, 'sink.parallelism' = '1' just one file...   But I think doing like that I reduce the number of tasks so it will have negative impact on performance :-(     Sent: Tuesday, August 30, 2022 at 3:22 PM

How Flink knows that CREATE TABLE is sometimes about creating table, sometimes about creating file?

  To create table from file:   "CREATE TABLE Table1 (column_name1 STRING, column_name2 DOUBLE) WITH ('connector' = 'filesystem', 'path' = 'file:///C:/temp/test4.txt', 'format' = 'csv', 'csv.field-delimiter' = ';')");   To create file:   "CREATE TABLE Table1 (column_name1 STRING,

Re: Why this example does not save anything to file?

Thank you very much Martijn you dedicated your productive time to help me! I'm new noob in this subject - I took that example somewhere from Internet. I see problem for guys like me is that Flink syntax changes from version to version quite significantly. So here not 'connector.type' but

Re: Why this example does not save anything to file?

No, I do not have it     Sent: Monday, August 01, 2022 at 4:43 PM From: "Martijn Visser" To: pod...@gmx.com Cc: user@flink.apache.org Subject: Re: Why this example does not save anything to file? That's Flink fault-tolerance mechanism, see 

Re: Why this example does not save anything to file?

What's that?     Sent: Monday, August 01, 2022 at 2:49 PM From: "Martijn Visser" To: pod...@gmx.com Cc: user@flink.apache.org Subject: Re: Why this example does not save anything to file? Do you have checkpointing enabled?    Op za 30 jul. 2022 om 17:31 schreef :

Re: Why this example does not save anything to file?

Thanks David but there's no problem with that (probably ";" is default separator). I can read the file and insert into "Table1" (I said that in my mail). Problem is to save to CSV.     Sent: Saturday, July 30, 2022 at 3:33 PM From: "David Anderson" To: pod...@gmx.com Cc: "user" Subject: Re:

Re: Why this example does not save anything to file?

Hi,   you mean adding:   " 'csv.field-delimiter'=';', "   like:      tEnv.executeSql("CREATE TABLE fs_table ("                 + "    column_nameA STRING, "                 + "    column_nameB DOUBLE "                 + "    ) WITH ( "                 + "    'connector'='filesystem',

Re: Why this example does not save anything to file?

In Eclipse only:   WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/C:/Flink/flink-1.14.4/lib/flink-dist_2.11-1.14.4.jar) to field java.lang.String.value WARNING: Please consider reporting this to

Why this example does not save anything to file?

If I get it correctly this is the way how I can save to CSV: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#full-example   So my code is (read from file, save to file):     package flinkCSV; import org.apache.flink.table.api.EnvironmentSettings;

Re: Does Table API connector, csv, has some option to ignore some columns

Could this not be as it was with readCsvFile and the "includeFields" option? That would be nice CSV is just a text file and headers are not required (but can be for human).     Sent: Tuesday, July 12, 2022 at 2:48 PM From: "yuxia" To: "podunk" Cc: "User"

Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released

lov" <alexan...@ververica.com> To: "user" <user@flink.apache.org> Cc: pod...@gmx.com Subject: Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released Hi podunk, please share exceptions that you find in the log/ folder of your Flink distribution. The Taskmanger startup issues

Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released

WARNING: All illegal access operations will be denied in a future release     Sent: Monday, July 11, 2022 at 11:36 PM From: "Alexander Fedulov" To: "user" Cc: pod...@gmx.com Subject: Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released Hi podunk, please share exceptions that

Re: Re: Does Table API connector, csv, has some option to ignore some columns

Does Table API connector, csv, has some option to ignore some columns Hi podunk,   no, this is currently not possible:  > Currently, the CSV schema is derived from table schema. [1]   So the Table schema is used to define how Jackson CSV parses the lines and hence needs to be complete.  

Re: Re: Is it possible to save Table to CSV?

No. But I do not want any 'parquet'. I need CSV.   Which code should created file?      EnvironmentSettings settings = EnvironmentSettings            .newInstance()            //.inStreamingMode()            .inBatchMode()            .build();         final

Re: Re: Does Table API connector, csv, has some option to ignore some columns

No, I did not mean. I said 'Does Table API connector, CSV, has some option to ignore some columns in source file?'     Sent: Monday, July 11, 2022 at 5:28 PM From: "Xuyang" To: pod...@gmx.com Cc: user@flink.apache.org Subject: Re:Re: Does Table API connector, csv, has some option to ignore

Re: Is it possible to save Table to CSV?

This example?:   CREATE TABLE kafka_table ( user_id STRING, order_amount DOUBLE, log_ts TIMESTAMP(3), WATERMARK FOR log_ts AS log_ts - INTERVAL '5' SECOND ) WITH (...); CREATE TABLE fs_table ( user_id STRING, order_amount DOUBLE, dt STRING, `hour` STRING ) PARTITIONED BY (dt,

Is it possible to save Table to CSV?

  If I create dynamic table with:   CREATE TABLE some_table (name STRING, score INT) WITH ( 'format' = 'csv', '...' ); //do some other stuff here   Then how to save table result to CSV file?   Best,   Mike    

Re: Does Table API connector, csv, has some option to ignore some columns

I want to control what I insert in table not what I get from table.     Sent: Monday, July 11, 2022 at 3:37 AM From: "Shengkai Fang" To: pod...@gmx.com Cc: "user" Subject: Re: Does Table API connector, csv, has some option to ignore some columns Hi.    In Flink SQL, you can select the

Re: [ANNOUNCE] Apache Flink 1.15.1 released

I run Flink in Windows and in version 1.15.1 Task Managers fails to start. Works without problems in 1.14.5     Sent: Friday, July 08, 2022 at 12:18 AM From: "David Anderson" To: "dev" , "user" , "user-zh" , annou...@apache.org Subject: [ANNOUNCE] Apache Flink 1.15.1 released   The

Does Table API connector, csv, has some option to ignore some columns

Does Table API connector, CSV, has some option to ignore some columns in source file? For instance read only first, second, nine... but not the others?   Or any other trick? CREATE TABLE some_table ( some_id BIGINT, ... ) WITH ( 'format' = 'csv', ... )      

Re: Is Flink able to read a CSV file or just like in Blink this function does not work?

  Fink will not run natively in windows - that is why I use Github CLI   I made test with Flink version 1.14.4 - Taskmanager is running. But no chance with 1.15.0 or 1.15.1 - taskmanager does not start. Something's wrong with Flink?     Sent: Friday, July 08, 2022 at 5:15 PM From: 

Re: Is Flink able to read a CSV file or just like in Blink this function does not work?

Yep, sorry, my fault.   I see now that this is not enough to just start cluster; taskmanager have to be started as well. How I start it (Github CLI):   export FLINK_HOME=/C/Flink/flink-1.15.0 $FLINK_HOME/bin/start-cluster.sh   Flink is running (I can reach it going to

Is Flink able to read a CSV file or just like in Blink this function does not work?

If I'm reading Flink manul correctly (and this is not simple - no examples), this code should read CSV file:     package flinkTest2; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.TableEnvironment;   public class flinkTest2 {     public static

How can I convert a DataSet into a Table?

  My code is:   package flinkTest2;   import org.apache.flink.api.java.DataSet; import org.apache.flink.api.java.ExecutionEnvironment; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.Table; import

Python and first code to execute Beam pipeline on Flink

  Hello there!   I'm learning Beam and Flink. I made Flink is running on my PC, I installed Beam module (pip install apache-beam). My first step is to execute simple script:   def run():  import apache_beam as beam  from apache_beam.options.pipeline_options import PipelineOptions    options