[jira] [Updated] (FLUME-2256) Generic JDBC Sink
[ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-2256: Fix Version/s: (was: 1.8.0) 1.9.0 > Generic JDBC Sink > - > > Key: FLUME-2256 > URL: https://issues.apache.org/jira/browse/FLUME-2256 > Project: Flume > Issue Type: New Feature >Reporter: Jeremy Karlson >Assignee: Jeremy Karlson > Fix For: 1.9.0 > > Attachments: FLUME-2256.diff > > > I've been working on a generic JDBC sink. It needs a bit more testing, but I > think it's ready for review and feedback. I have not yet updated the Flume > documentation, but I can / will if people are happy with this. > Since the config file is how you’d interact with it, here’s a working example > from my source tree: > {code} > a.sinks.k.type=jdbc > a.sinks.k.channel=c > a.sinks.k.driver=com.mysql.jdbc.Driver > a.sinks.k.url=jdbc:mysql://localhost:8889/flume > a.sinks.k.user=username > a.sinks.k.password=password > a.sinks.k.batchSize=100 > a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, > ${header.timestamp:long}) > {code} > The interesting part is the SQL statement. You can put anything you want in > there - it will get converted to a prepared statement on execution. The > Ant-ish tokens get parsed and replaced with parameters at startup. > The tokens are three part. For example, in: > {code} > ${body:string(UTF-8)} > {code} > The first is a place in the event to get the value from (“body”, > “header.foo”, or “custom”). The second part ("string") is a type identifier > that converts into an appropriate JDBC parameter. The third part (“UTF-8") > is a configuration string for that type, if needed. As for types, so far > I’ve defined: > body: string (with optional charset encoding), bytearray > header: string, long, int, float, double, date (with mandatory date format > and optional timezone) > Additionally, if none of those make you happy you can define you own > parameter converters: > {code} > ${custom:com.company.foo.MyConverter(optionaltextconfig)} > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2256) Generic JDBC Sink
[ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2256: - Fix Version/s: v1.8.0 > Generic JDBC Sink > - > > Key: FLUME-2256 > URL: https://issues.apache.org/jira/browse/FLUME-2256 > Project: Flume > Issue Type: New Feature >Reporter: Jeremy Karlson >Assignee: Jeremy Karlson > Fix For: v1.8.0 > > Attachments: FLUME-2256.diff > > > I've been working on a generic JDBC sink. It needs a bit more testing, but I > think it's ready for review and feedback. I have not yet updated the Flume > documentation, but I can / will if people are happy with this. > Since the config file is how you’d interact with it, here’s a working example > from my source tree: > {code} > a.sinks.k.type=jdbc > a.sinks.k.channel=c > a.sinks.k.driver=com.mysql.jdbc.Driver > a.sinks.k.url=jdbc:mysql://localhost:8889/flume > a.sinks.k.user=username > a.sinks.k.password=password > a.sinks.k.batchSize=100 > a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, > ${header.timestamp:long}) > {code} > The interesting part is the SQL statement. You can put anything you want in > there - it will get converted to a prepared statement on execution. The > Ant-ish tokens get parsed and replaced with parameters at startup. > The tokens are three part. For example, in: > {code} > ${body:string(UTF-8)} > {code} > The first is a place in the event to get the value from (“body”, > “header.foo”, or “custom”). The second part ("string") is a type identifier > that converts into an appropriate JDBC parameter. The third part (“UTF-8") > is a configuration string for that type, if needed. As for types, so far > I’ve defined: > body: string (with optional charset encoding), bytearray > header: string, long, int, float, double, date (with mandatory date format > and optional timezone) > Additionally, if none of those make you happy you can define you own > parameter converters: > {code} > ${custom:com.company.foo.MyConverter(optionaltextconfig)} > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2256) Generic JDBC Sink
[ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Karlson updated FLUME-2256: -- Attachment: FLUME-2256.diff > Generic JDBC Sink > - > > Key: FLUME-2256 > URL: https://issues.apache.org/jira/browse/FLUME-2256 > Project: Flume > Issue Type: New Feature >Reporter: Jeremy Karlson >Assignee: Jeremy Karlson > Attachments: FLUME-2256.diff > > > I've been working on a generic JDBC sink. It needs a bit more testing, but I > think it's ready for review and feedback. I have not yet updated the Flume > documentation, but I can / will if people are happy with this. > Since the config file is how you’d interact with it, here’s a working example > from my source tree: > {code} > a.sinks.k.type=jdbc > a.sinks.k.channel=c > a.sinks.k.driver=com.mysql.jdbc.Driver > a.sinks.k.url=jdbc:mysql://localhost:8889/flume > a.sinks.k.user=username > a.sinks.k.password=password > a.sinks.k.batchSize=100 > a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, > ${header.timestamp:long}) > {code} > The interesting part is the SQL statement. You can put anything you want in > there - it will get converted to a prepared statement on execution. The > Ant-ish tokens get parsed and replaced with parameters at startup. > The tokens are three part. For example, in: > {code} > ${body:string(UTF-8)} > {code} > The first is a place in the event to get the value from (“body”, > “header.foo”, or “custom”). The second part ("string") is a type identifier > that converts into an appropriate JDBC parameter. The third part (“UTF-8") > is a configuration string for that type, if needed. As for types, so far > I’ve defined: > body: string (with optional charset encoding), bytearray > header: string, long, int, float, double, date (with mandatory date format > and optional timezone) > Additionally, if none of those make you happy you can define you own > parameter converters: > {code} > ${custom:com.company.foo.MyConverter(optionaltextconfig)} > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (FLUME-2256) Generic JDBC Sink
[ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Karlson updated FLUME-2256: -- Attachment: (was: FLUME-2256.diff) > Generic JDBC Sink > - > > Key: FLUME-2256 > URL: https://issues.apache.org/jira/browse/FLUME-2256 > Project: Flume > Issue Type: New Feature >Reporter: Jeremy Karlson >Assignee: Jeremy Karlson > Attachments: FLUME-2256.diff > > > I've been working on a generic JDBC sink. It needs a bit more testing, but I > think it's ready for review and feedback. I have not yet updated the Flume > documentation, but I can / will if people are happy with this. > Since the config file is how you’d interact with it, here’s a working example > from my source tree: > {code} > a.sinks.k.type=jdbc > a.sinks.k.channel=c > a.sinks.k.driver=com.mysql.jdbc.Driver > a.sinks.k.url=jdbc:mysql://localhost:8889/flume > a.sinks.k.user=username > a.sinks.k.password=password > a.sinks.k.batchSize=100 > a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, > ${header.timestamp:long}) > {code} > The interesting part is the SQL statement. You can put anything you want in > there - it will get converted to a prepared statement on execution. The > Ant-ish tokens get parsed and replaced with parameters at startup. > The tokens are three part. For example, in: > {code} > ${body:string(UTF-8)} > {code} > The first is a place in the event to get the value from (“body”, > “header.foo”, or “custom”). The second part ("string") is a type identifier > that converts into an appropriate JDBC parameter. The third part (“UTF-8") > is a configuration string for that type, if needed. As for types, so far > I’ve defined: > body: string (with optional charset encoding), bytearray > header: string, long, int, float, double, date (with mandatory date format > and optional timezone) > Additionally, if none of those make you happy you can define you own > parameter converters: > {code} > ${custom:com.company.foo.MyConverter(optionaltextconfig)} > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (FLUME-2256) Generic JDBC Sink
[ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Karlson updated FLUME-2256: -- Attachment: FLUME-2256.diff > Generic JDBC Sink > - > > Key: FLUME-2256 > URL: https://issues.apache.org/jira/browse/FLUME-2256 > Project: Flume > Issue Type: New Feature >Reporter: Jeremy Karlson >Assignee: Jeremy Karlson > Attachments: FLUME-2256.diff > > > I've been working on a generic JDBC sink. It needs a bit more testing, but I > think it's ready for review and feedback. I have not yet updated the Flume > documentation, but I can / will if people are happy with this. > Since the config file is how you’d interact with it, here’s a working example > from my source tree: > {code} > a.sinks.k.type=jdbc > a.sinks.k.channel=c > a.sinks.k.driver=com.mysql.jdbc.Driver > a.sinks.k.url=jdbc:mysql://localhost:8889/flume > a.sinks.k.user=username > a.sinks.k.password=password > a.sinks.k.batchSize=100 > a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, > ${header.timestamp:long}) > {code} > The interesting part is the SQL statement. You can put anything you want in > there - it will get converted to a prepared statement on execution. The > Ant-ish tokens get parsed and replaced with parameters at startup. > The tokens are three part. For example, in: > {code} > ${body:string(UTF-8)} > {code} > The first is a place in the event to get the value from (“body”, > “header.foo”, or “custom”). The second part ("string") is a type identifier > that converts into an appropriate JDBC parameter. The third part (“UTF-8") > is a configuration string for that type, if needed. As for types, so far > I’ve defined: > body: string (with optional charset encoding), bytearray > header: string, long, int, float, double, date (with mandatory date format > and optional timezone) > Additionally, if none of those make you happy you can define you own > parameter converters: > {code} > ${custom:com.company.foo.MyConverter(optionaltextconfig)} > {code} -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (FLUME-2256) Generic JDBC Sink
[ https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Karlson updated FLUME-2256: -- Description: I've been working on a generic JDBC sink. It needs a bit more testing, but I think it's ready for review and feedback. I have not yet updated the Flume documentation, but I can / will if people are happy with this. Since the config file is how you’d interact with it, here’s a working example from my source tree: {code} a.sinks.k.type=jdbc a.sinks.k.channel=c a.sinks.k.driver=com.mysql.jdbc.Driver a.sinks.k.url=jdbc:mysql://localhost:8889/flume a.sinks.k.user=username a.sinks.k.password=password a.sinks.k.batchSize=100 a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long}) {code} The interesting part is the SQL statement. You can put anything you want in there - it will get converted to a prepared statement on execution. The Ant-ish tokens get parsed and replaced with parameters at startup. The tokens are three part. For example, in: {code} ${body:string(UTF-8)} {code} The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”). The second part ("string") is a type identifier that converts into an appropriate JDBC parameter. The third part (“UTF-8") is a configuration string for that type, if needed. As for types, so far I’ve defined: body: string (with optional charset encoding), bytearray header: string, long, int, float, double, date (with mandatory date format and optional timezone) Additionally, if none of those make you happy you can define you own parameter converters: {code} ${custom:com.company.foo.MyConverter(optionaltextconfig)} {code} was: I've been working on a generic JDBC sink. It needs a bit more testing, but I think it's ready for review and feedback. I have not yet updated the Flume documentation, but I can / will if people are happy with this. Since the config file is how you’d interact with it, here’s a working example from my source tree: a.sinks.k.type=jdbc a.sinks.k.channel=c a.sinks.k.driver=com.mysql.jdbc.Driver a.sinks.k.url=jdbc:mysql://localhost:8889/flume a.sinks.k.user=username a.sinks.k.password=password a.sinks.k.batchSize=100 a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long}) The interesting part is the SQL statement. You can put anything you want in there - it will get converted to a prepared statement on execution. The Ant-ish tokens get parsed and replaced with parameters at startup. The tokens are three part. For example, in: ${body:string(UTF-8)} The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”). The second part ("string") is a type identifier that converts into an appropriate JDBC parameter. The third part (“UTF-8") is a configuration string for that type, if needed. As for types, so far I’ve defined: body: string (with optional charset encoding), bytearray header: string, long, int, float, double, date (with mandatory date format and optional timezone) Additionally, if none of those make you happy you can define you own parameter converters: ${custom:com.company.foo.MyConverter(optionaltextconfig)} > Generic JDBC Sink > - > > Key: FLUME-2256 > URL: https://issues.apache.org/jira/browse/FLUME-2256 > Project: Flume > Issue Type: New Feature >Reporter: Jeremy Karlson >Assignee: Jeremy Karlson > > I've been working on a generic JDBC sink. It needs a bit more testing, but I > think it's ready for review and feedback. I have not yet updated the Flume > documentation, but I can / will if people are happy with this. > Since the config file is how you’d interact with it, here’s a working example > from my source tree: > {code} > a.sinks.k.type=jdbc > a.sinks.k.channel=c > a.sinks.k.driver=com.mysql.jdbc.Driver > a.sinks.k.url=jdbc:mysql://localhost:8889/flume > a.sinks.k.user=username > a.sinks.k.password=password > a.sinks.k.batchSize=100 > a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, > ${header.timestamp:long}) > {code} > The interesting part is the SQL statement. You can put anything you want in > there - it will get converted to a prepared statement on execution. The > Ant-ish tokens get parsed and replaced with parameters at startup. > The tokens are three part. For example, in: > {code} > ${body:string(UTF-8)} > {code} > The first is a place in the event to get the value from (“body”, > “header.foo”, or “custom”). The second part ("string") is a type identifier > that converts into an appropriate JDBC parameter. The third part (“UTF-8") > is a configuration string for that type, if needed. As for types, so far > I’ve defined: > body: string (with optional charset encoding), bytearray > he