Re: JDBC Streaming Connector

2016-02-26 Thread Timothy Farkas
Thanks everyone. I'll take a look at JDBCOutputFormat, and FLINK-3332 <https://issues.apache.org/jira/browse/FLINK-3332> along with other related tickets. Tim On Fri, Feb 26, 2016 at 1:55 AM, Chesnay Schepler wrote: > I'm actually working on a JDBC streaming connecto

Re: JDBC Streaming Connector

2016-02-26 Thread Chesnay Schepler
I'm actually working on a JDBC streaming connector with transactions. On 26.02.2016 10:33, Stephan Ewen wrote: Hi Timothy! I would first just write a stream data sink based on the JDBCOutputFormat, and then let's look into making it exactly-once. For JDBC with support for transac

Re: JDBC Streaming Connector

2016-02-26 Thread Stephan Ewen
Hi Timothy! I would first just write a stream data sink based on the JDBCOutputFormat, and then let's look into making it exactly-once. For JDBC with support for transactions, the basic idea would be to start a transaction, insert elements, commit when a checkpoint is confirmed, and start a new t

Re: JDBC Streaming Connector

2016-02-26 Thread Till Rohrmann
Hi Timothy, you can use the `JDBCOutputFormat` in combination with the `DataStream.writeUsingOutputFormat` method. However, this won't give you exactly once guarantees since the output formats don't take part in the checkpointing mechanism. Currently, Chesnay is working on a generic framework for

JDBC Streaming Connector

2016-02-25 Thread Timothy Farkas
Hi All, I'm new to Flink and was looking for a JDBC stream sink connector, and didn't see one in flink-streaming-connectors. Is there one somewhere else, or is there one currently in development. If not could I pick up a ticket to add it? Thanks, Tim