I was thinking if I can use Flink to process large files and save result to
another file or database (jdbc).
So loading file into Flink table and then loop through rows inserting result to
some temp Flink table.
Then save result to jdbc table or file.
Sent: Thursday, September 22, 2022 at
Thank you - I'll try.
There is no 'UPDATE' clause in Flink SQL?
Sent: Monday, September 19, 2022 at 4:09 AM
From: "Shengkai Fang"
To: pod...@gmx.com
Cc: user@flink.apache.org
Subject: Re: INSERT INTO will work faster in Flink than in regular database?
Hi. I think you can write a udf[1]
What's the most effective way (performance) to update big no of rows?
Sure this will be probably "INSERT INTO table (column1) SELECT column1 FROM
...". Anyway, I do not see any "UPDATE" in Flink?
But sometimes SQL is not enough.
Suppose I have code:
TableResult tableResult1 =
I see I can only insert into JDBC table with select from another table,
something like:
tEnv.executeSql("INSERT INTO Customers SELECT customer_number, pid_no, name
FROM another_table");
But what if I want to insert row that I created within Flink? For instance I
made some calculation and I
Why this INSERT does not insert row in table (jdbc connection works, I can
create 'Customers' table from MySQL table)?
tEnv.executeSql("CREATE TABLE Customers ("
+ " customer_number INT, "
+ " pid_no INT, "
+ "
Thank you.
So I understand 'this is normal Flink property - live with it' :-)
M.
Sent: Wednesday, August 31, 2022 at 9:37 PM
From: "David Anderson"
To: pod...@gmx.com
Cc: "user"
Subject: Re: Why this example does not save anything to file?
With the parallelism set to 2, you will get 2
Doesn't it depends on 'sink.parallelism'?
If I set 'sink.parallelism' = '2' I get two files, 'sink.parallelism' = '1'
just one file...
But I think doing like that I reduce the number of tasks so it will have
negative impact on performance :-(
Sent: Tuesday, August 30, 2022 at 3:22 PM
To create table from file:
"CREATE TABLE Table1 (column_name1 STRING, column_name2 DOUBLE) WITH ('connector' = 'filesystem', 'path' = 'file:///C:/temp/test4.txt', 'format' = 'csv', 'csv.field-delimiter' = ';')");
To create file:
"CREATE TABLE Table1 (column_name1 STRING,
Thank you very much Martijn you dedicated your productive time to help me!
I'm new noob in this subject - I took that example somewhere from Internet. I
see problem for guys like me is that Flink syntax changes from version to
version quite significantly. So here not 'connector.type' but
No, I do not have it
Sent: Monday, August 01, 2022 at 4:43 PM
From: "Martijn Visser"
To: pod...@gmx.com
Cc: user@flink.apache.org
Subject: Re: Why this example does not save anything to file?
That's Flink fault-tolerance mechanism, see
What's that?
Sent: Monday, August 01, 2022 at 2:49 PM
From: "Martijn Visser"
To: pod...@gmx.com
Cc: user@flink.apache.org
Subject: Re: Why this example does not save anything to file?
Do you have checkpointing enabled?
Op za 30 jul. 2022 om 17:31 schreef :
Thanks David but there's no problem with that (probably ";" is default separator).
I can read the file and insert into "Table1" (I said that in my mail).
Problem is to save to CSV.
Sent: Saturday, July 30, 2022 at 3:33 PM
From: "David Anderson"
To: pod...@gmx.com
Cc: "user"
Subject: Re:
Hi,
you mean adding:
" 'csv.field-delimiter'=';', "
like:
tEnv.executeSql("CREATE TABLE fs_table ("
+ " column_nameA STRING, "
+ " column_nameB DOUBLE "
+ " ) WITH ( "
+ " 'connector'='filesystem',
In Eclipse only:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/C:/Flink/flink-1.14.4/lib/flink-dist_2.11-1.14.4.jar) to field java.lang.String.value
WARNING: Please consider reporting this to
If I get it correctly this is the way how I can save to CSV:
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#full-example
So my code is (read from file, save to file):
package flinkCSV;
import org.apache.flink.table.api.EnvironmentSettings;
Could this not be as it was with readCsvFile and the "includeFields" option? That would be nice
CSV is just a text file and headers are not required (but can be for human).
Sent: Tuesday, July 12, 2022 at 2:48 PM
From: "yuxia"
To: "podunk"
Cc: "User"
lov" <alexan...@ververica.com>
To: "user" <user@flink.apache.org>
Cc: pod...@gmx.com
Subject: Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released
Hi podunk,
please share exceptions that you find in the log/ folder of your Flink distribution.
The Taskmanger startup issues
WARNING: All illegal access operations will be denied in a future release
Sent: Monday, July 11, 2022 at 11:36 PM
From: "Alexander Fedulov"
To: "user"
Cc: pod...@gmx.com
Subject: Re: Re: [ANNOUNCE] Apache Flink 1.15.1 released
Hi podunk,
please share exceptions that
Does Table API connector, csv, has some option to ignore some columns
Hi podunk,
no, this is currently not possible:
> Currently, the CSV schema is derived from table schema. [1]
So the Table schema is used to define how Jackson CSV parses the lines and hence needs to be complete.
No. But I do not want any 'parquet'. I need CSV.
Which code should created file?
EnvironmentSettings settings = EnvironmentSettings
.newInstance()
//.inStreamingMode()
.inBatchMode()
.build();
final
No, I did not mean.
I said 'Does Table API connector, CSV, has some option to ignore some columns in source file?'
Sent: Monday, July 11, 2022 at 5:28 PM
From: "Xuyang"
To: pod...@gmx.com
Cc: user@flink.apache.org
Subject: Re:Re: Does Table API connector, csv, has some option to ignore
This example?:
CREATE TABLE kafka_table (
user_id STRING,
order_amount DOUBLE,
log_ts TIMESTAMP(3),
WATERMARK FOR log_ts AS log_ts - INTERVAL '5' SECOND
) WITH (...);
CREATE TABLE fs_table (
user_id STRING,
order_amount DOUBLE,
dt STRING,
`hour` STRING
) PARTITIONED BY (dt,
If I create dynamic table with:
CREATE TABLE some_table (name STRING, score INT)
WITH (
'format' = 'csv',
'...'
);
//do some other stuff here
Then how to save table result to CSV file?
Best,
Mike
I want to control what I insert in table not what I get from table.
Sent: Monday, July 11, 2022 at 3:37 AM
From: "Shengkai Fang"
To: pod...@gmx.com
Cc: "user"
Subject: Re: Does Table API connector, csv, has some option to ignore some columns
Hi.
In Flink SQL, you can select the
I run Flink in Windows and in version 1.15.1 Task Managers fails to start.
Works without problems in 1.14.5
Sent: Friday, July 08, 2022 at 12:18 AM
From: "David Anderson"
To: "dev" , "user" , "user-zh" , annou...@apache.org
Subject: [ANNOUNCE] Apache Flink 1.15.1 released
The
Does Table API connector, CSV, has some option to ignore some columns in source file?
For instance read only first, second, nine... but not the others?
Or any other trick?
CREATE TABLE some_table (
some_id BIGINT,
...
) WITH (
'format' = 'csv',
...
)
Fink will not run natively in windows - that is why I use Github CLI
I made test with Flink version 1.14.4 - Taskmanager is running. But no chance with 1.15.0 or 1.15.1 - taskmanager does not start.
Something's wrong with Flink?
Sent: Friday, July 08, 2022 at 5:15 PM
From:
Yep, sorry, my fault.
I see now that this is not enough to just start cluster; taskmanager have to be started as well.
How I start it (Github CLI):
export FLINK_HOME=/C/Flink/flink-1.15.0
$FLINK_HOME/bin/start-cluster.sh
Flink is running (I can reach it going to
If I'm reading Flink manul correctly (and this is not simple - no examples), this code should read CSV file:
package flinkTest2;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.TableEnvironment;
public class flinkTest2 {
public static
My code is:
package flinkTest2;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.Table;
import
Hello there!
I'm learning Beam and Flink. I made Flink is running on my PC, I installed Beam module (pip install apache-beam).
My first step is to execute simple script:
def run():
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
options
31 matches
Mail list logo