Re: Flink Operator in Golang?

Golang doesn't seem to have anything similar to Flink or Spark. On Thu, Nov 17, 2022 at 8:11 PM Mark Lee wrote: > I got it, Thanks Zhanghao! > > > > *发件人:* user-return-51640-lifuqiong00=126@flink.apache.org > *代表 * > zhanghao.c...@outlook.com > *发送时间:* 2022年11月17日 23:36 > *收件人:* Mark Lee ;

Re: what is the difference between map vs process on a datastream?

map, but with a Collector that can be used to > emit zero, one, or many events in response to each event, just like a > process function. > > David > > > On Tue, Mar 17, 2020 at 11:50 AM kant kodali wrote: > >> what is the difference between map vs process on a datastream? they look >> very similar. >> >> Thanks! >> >>

Re: Apache Airflow - Question about checkpointing and re-run a job

Does Airflow has a Flink Operator? I am not seeing it? Can you please point me? On Mon, Nov 18, 2019 at 3:10 AM M Singh wrote: > Thanks Congxian for your answer and reference. Mans > > On Sunday, November 17, 2019, 08:59:16 PM EST, Congxian Qiu < > qcx978132...@gmail.com> wrote: > > > Hi >

what is the difference between map vs process on a datastream?

what is the difference between map vs process on a datastream? they look very similar. Thanks!

a question on window trigger and delta output

Hi All, I set a transformation like this and my events in the stream have a sequential timestamp like 1,2,3, and I set the watermark to event time. myStream .keyBy(0) .timeWindow(Time.of(1000, TimeUnit.MILLISECONDS)) .aggregate(new myAggregateFunction())

Re: Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime?

on. >>> >> >> So it's neither possible to assign timestamps nor watermark, but it seems >> as if the default behavior is exactly as you want it to be. If that doesn't >> work for you, could you please rephrase your last question or describe your >> use case? I di

Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime?

Hi All, Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime? say I set my time characteristic of stream execution environment to Ingestion time as follows streamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime); do I

How does custom object/data structures get mapped into rocksdb underneath?

Hi All, I want to do stateful streaming and I was wondering how Custom objects get mapped into rocksdb? say I have the following class that represents my state public class MyState { private HashMap map1 ; // T can be any type private HashMap map2; // S can be any type } I wonder how

Re: How to print the aggregated state everytime it is updated?

vent time. then > you should call `DataStream.assignTimestampsAndWatermarks` to set the > timestamp and watermark. > Window is triggered when the watermark exceed the window end time > > Best, > Congxian > > > kant kodali 于2020年3月4日周三 上午5:11写道： > >> Hi All, >> &g

How to print the aggregated state everytime it is updated?

Hi All, I have a custom aggregated state that is represent by Set and I have a stream of values coming in from Kafka where I inspect, compute the custom aggregation and store it in Set. Now, I am trying to figureout how do I print the updated value everytime this state is updated? Imagine I have

Re: java.util.concurrent.ExecutionException

Hi Gary, This has to do with my Kafka. After restarting Kafka it seems to work fine! Thanks! On Tue, Mar 3, 2020 at 8:18 AM kant kodali wrote: > The program finished with the following exception: > > > org.apache.flink.client.program.ProgramInvocationException: The main >

Re: java.util.concurrent.ExecutionException

Hi, > > Can you post the complete stacktrace? > > Best, > Gary > > On Tue, Mar 3, 2020 at 1:08 PM kant kodali wrote: > >> Hi All, >> >> I am just trying to read edges which has the following format in Kafka >> >> 1,2 >> 1,3 >> 1,

zookeeper.connect is not needed but Flink requires it

Hi All, The zookeeper.connect is not needed for KafkaConsumer or KafkaAdminClient however Flink requires it. You can also see in the Flink TaskManager logs the KafkaConsumer is not recognizing this property anyways. bsTableEnv.connect( new Kafka() .property("bootstrap.servers",

How is state stored in rocksdb?

Hi All, I am wondering how Flink serializes and deserializes state from rockdb? What is the format used? For example, say I am doing some stateful streaming and say an object for my class below represents a state. how does Flink serializes and deserializes the object of MyClass below? is it just

Re: Do I need to use Table API or DataStream API to construct sources?

ql-connector-kafka_2.11:${flinkVersion}" > > > On Sun, Mar 1, 2020 at 7:31 PM kant kodali wrote: > >> * What went wrong: >> Could not determine the dependencies of task ':shadowJar'. >> > Could not resolve all dependencies for configuration ':flinkShadowJar'. >>

Re: Do I need to use Table API or DataStream API to construct sources?

lement such a sink for *both *batch and streaming > here: > https://issues.apache.org/jira/browse/FLINK-14807?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=17046455#comment-17046455 > > Best, > > Dawid > On 01/03/2020 12:00, kant kodali wrote: > > Hi Benc

Re: Do I need to use Table API or DataStream API to construct sources?

t in Maven, packaging dependencies into > one fat jar needs to specify how SPI property files should be dealt with, > like > > > implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> > > > Could you check that your fina

Re: Do I need to use Table API or DataStream API to construct sources?

soon. > > > The program finished with the following exception: > please make sure that the kafka version in Test class and the kafka > version in pom dependency are same. I tested your code successfully. > > Bests, > Godfrey > &g

Re: Do I need to use Table API or DataStream API to construct sources?

t; Piotrek > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/org/apache/flink/table/api/TableEnvironment.html#registerTableSink-java.lang.String-org.apache.flink.table.sinks.TableSink- > > On 1 Mar 2020, at 07:50, kant kodali wrote: > > Here is my u

Re: Do I need to use Table API or DataStream API to construct sources?

mat(new Csv()) .withSchema(new Schema().field("f0", DataTypes.STRING())) .inAppendMode() .createTemporaryTable("kafka_target"); tableEnvironment.insertInto("kafka_target", resultTable); tableEnvironment.execute("

Re: Do I need to use Table API or DataStream API to construct sources?

ery useful, and I've created an > issue[2] to track this. > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sourceSinks.html#define-a-tablesink > [2] https://issues.apache.org/jira/browse/FLINK-16354 > > kant kodali 于2020年3月1日周日 上午2:30写道：

Is CSV format supported for Kafka in Flink 1.10?

Hi, Is CSV format supported for Kafka in Flink 1.10? It says I need to specify connector.type as Filesystem but documentation says it is supported for Kafka? import org.apache.flink.contrib.streaming.state.RocksDBStateBackend; import org.apache.flink.runtime.state.StateBackend; import

Re: Do I need to use Table API or DataStream API to construct sources?

.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#overview > [3] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/connect.html#kafka-connector > > On

Re: Do I need to use Table API or DataStream API to construct sources?

Also why do I need to convert to DataStream to print the rows of a table? Why not have a print method in the Table itself? On Sat, Feb 29, 2020 at 3:40 AM kant kodali wrote: > Hi All, > > Do I need to use DataStream API or Table API to construct sources? I am > just trying to rea

Do I need to use Table API or DataStream API to construct sources?

Hi All, Do I need to use DataStream API or Table API to construct sources? I am just trying to read from Kafka and print it to console. And yes I tried it with datastreams and it works fine but I want to do it using Table related APIs. I don't see any documentation or a sample on how to create

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

m.getProperty('user.name'), 'Build-Jdk': System.getProperty('java.version') } } shadowJar { configurations = [project.configurations.flinkShadowJar] } On Fri, Feb 28, 2020 at 1:09 AM Jark Wu wrote: > Hi Kant, > > You shouldn't compile `flink-table-planner` or `f

Re: Flink 1.10 exception : Unable to instantiate java compiler

Same problem! On Thu, Feb 27, 2020 at 11:10 PM LakeShen wrote: > Hi community, > now I am using the flink 1.10 to run the flink task > ,cluster type is yarn . I use commandline to submit my flink job , the > commandline just like this : > > flink run -m yarn-cluster

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

that default > policy is child first [1]. More details can find in [2]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/release-notes/flink-1.10.html#flink-client-respects-classloading-policy-flink-13749 > [2] https://issues.apache.org/jira/browse/FLINK-13749 > &g

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

It works within IDE but not when I submit using command using flink run myApp.jar On Thu, Feb 27, 2020 at 3:32 PM kant kodali wrote: > Below is the sample code using Flink 1.10 > > public class Test { > > public static void main(String... args) t

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Query("SELECT * FROM sample1 INNER JOIN sample2 on sample1.f0=sample2.f0"); result.printSchema(); bsTableEnv.toRetractStream(result, Row.class).print(); bsTableEnv.execute("sample job"); } } On Thu, Feb 27, 2020 at 3:22 PM kant kodali wrote: > Fixed the typo.

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Fixed the typo. Hi All, My sample program works in Flink 1.9 but in 1.10 I get the following error when I am submitting the job. otherwords it fails to submit a job. any idea? Thanks! On Thu, Feb 27, 2020 at 2:19 PM kant kodali wrote: > Hi All, > > > My sample program works

The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Hi All, My sample program works in Flink 1.9 but in 1.0 I get the following error when I am submitting the job. otherwords it fails to submit a job. any idea? Thanks! org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to instantiate java

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html#hivecatalog Can I use the hive catalog to store view definitions in HDFS? I am assuming the metastore can be anything or does it have to be have MySQL? On Thu, Feb 27, 2020 at 4:46 AM kant kodali wrote

How can I programmatically set RocksDBStateBackend?

Hi All, How can I programmatically set RocksDBStateBackend? I did the following [image: Screen Shot 2020-02-27 at 4.53.38 AM.png] env.setStateBackend always shows deprecated. so what is the right way to do this in flink 1.10? Thanks!

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

godfrey he wrote: > Hi Kant, if you want the store the catalog data in Local Filesystem/HDFS, > you can implement a user defined catalog (just need to implement Catalog > interface) > > Bests, > Godfrey > > kant kodali 于2020年2月26日周三 下午12:28写道： > >> Hi Jingsong, >>

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

n close. >> I think the persisted views will be supported in 1.11. >> >> Best, >> Jark >> >> 2020年1月20日 18:46，kant kodali 写道： >> >> Hi Jingsong, >> >> Thanks a lot, I think I can live with >> TableEnvironment.createTemporaryView in

Re: Can Connected Components run on a streaming dataset using iterate delta?

Hi, Thanks for that but Looks like it is already available https://github.com/vasia/gelly-streaming in streaming but I wonder why this is not part of Flink? there are no releases either. Thanks! On Tue, Feb 18, 2020 at 9:13 AM Yun Gao wrote: >Hi Kant, > > As far as I

Can Connected Components run on a streaming dataset using iterate delta?

Hi All, I am wondering if connected components can run on a streaming data? or say incremental batch? I see that with delta iteration not all vertices need to participate at every iteration

Re: is streaming outer join sending unnecessary traffic?

. > > Cheers, > Till > > On Tue, Jan 28, 2020 at 10:46 PM kant kodali wrote: > >> Sorry. fixed some typos. >> >> I am doing a streaming outer join from four topics in Kafka lets call >> them sample1, sample2, sample3, sample4. Each of these test topics has

Cypher support for flink graphs?

Hi All, Can we expect open cypher support for Flink graphs? Thanks!

Re: is streaming outer join sending unnecessary traffic?

Sorry. fixed some typos. I am doing a streaming outer join from four topics in Kafka lets call them sample1, sample2, sample3, sample4. Each of these test topics has just one column which is of tuple string. my query is this SELECT * FROM sample1 FULL OUTER JOIN sample2 on sample1.f0=sample2.f0

is streaming outer join sending unnecessary traffic?

Hi All, I am doing a streaming outer join from four topics in Kafka lets call them sample1, sample2, sample3, sample4. Each of these test topics has just one column which is of tuple string. my query is this SELECT * FROM sample1 FULL OUTER JOIN sample2 on sample1.f0=sample2.f0 FULL OUTER JOIN

Re: where does flink store the intermediate results of a join and what is the key?

QueryableStateStream/QueryableStateClient [1], or around "FOR SYSTEM_TIME >>>> AS OF" [2], or on other APIs/concepts (is there a FLIP?)? >>>> >>>> Cheers >>>> Ben >>>> >>>> [1] >>>> https://ci.apache.or

Re: where does flink store the intermediate results of a join and what is the key?

. However, uid of operators is not set in Table API & SQL. > So I’m not sure whether it works or not. > > 3)You can have a custom statebackend by > implement org.apache.flink.runtime.state.StateBackend interface, and use it > via `env.setStateBackend(…)`. > > Best, > Ja

Re: where does flink store the intermediate results of a join and what is the key?

/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/join/stream/state/JoinRecordStateViews.java#L45 > > > 2020年1月21日 18:01，kant kodali 写道： > > Hi All, > > If I run a query like this > > StreamTableEnvironment.sqlQuery(&qu

where does flink store the intermediate results of a join and what is the key?

Hi All, If I run a query like this StreamTableEnvironment.sqlQuery("select * from table1 join table2 on table1.col1 = table2.col1") 1) Where will flink store the intermediate result? Imagine flink-conf.yaml says state.backend = 'rocksdb' 2) If the intermediate results are stored in rockdb then

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

lient. > - Or using hive catalog, in 1.10, we support query catalog views. > > FLIP-71 will be finished in 1.11 soon. > > Best, > Jingsong Lee > > On Sun, Jan 19, 2020 at 4:10 PM kant kodali wrote: > >> I tried the following. >> >> bsTableEnv.sqlUpdate("

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

the moment? Thanks! On Sat, Jan 18, 2020 at 6:24 PM kant kodali wrote: > Hi All, > > Does Flink 1.9 support create or replace views syntax in raw SQL? like > spark streaming does? > > Thanks! >

Re: some basic questions

to work fine although I am not sure which one is the correct usage. Thanks! On Sat, Jan 18, 2020 at 6:52 PM kant kodali wrote: > Hi Godfrey, > > Thanks a lot for your response. I just tried it with env.execute("simple > job") but I still get the same error message. > > K

Re: some basic questions

0, blink > planner is more statable, we are switching the blink planner to the default > step by step [0]. > > [0] > http://mail-archives.apache.org/mod_mbox/flink-dev/202001.mbox/%3CCAELO930%2B3RJ5m4hGQ7fbS-CS%3DcfJe5ENcRmZ%3DT_hey-uL6c27g%40mail.gmail.com%3E > > kant kodali 于20

Does Flink 1.9 support create or replace views syntax in raw sql?

Hi All, Does Flink 1.9 support create or replace views syntax in raw SQL? like spark streaming does? Thanks!

some basic questions

Hi All, 1) The Documentation says full outer join is supported however the below code just exits with value 1. No error message. import org.apache.flink.api.common.serialization.SimpleStringSchema; import org.apache.flink.streaming.api.datastream.DataStream; import

Why would indefinitely growing state an issue for Flink while doing stream to stream joins?

Hi All, The doc https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/joins.html#regular-joins says the following. "However, this operation has an important implication: it requires to keep both sides of the join input in Flink’s state forever. Thus, the resource usage will

Is there a way to clean up the state based on custom/business logic?

Hi All, I read through the doc below and I am wondering if I can clean up the state based on custom logic rather min and max retention time? For example, I want to say clean up all the state where the key = foo or say the value = bar. so until the keys reach a particular value just keep

Re: are blink changes merged into flink 1.9?

ommon.html#main-differences-between-the-two-planners > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/common.html#create-a-tableenvironment > > > kant kodali 于2020年1月12日周日 上午7:48写道： > >> Hi All, >> >> Are blink changes merged into fli

are blink changes merged into flink 1.9?

Hi All, Are blink changes merged into flink 1.9? It looks like there are a lot of features and optimizations in Blink and if they aren't merged into flink 1.9 I am not sure on which one to use? is there any plan towards merging it? Thanks!

Re: Are there pipeline API's for ETL?

> > Here are the generic APIs list.[1] > > Best, > Vino > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html > > kant kodali 于2020年1月11日周六 上午9:06写道： >> Hi All, >> >> I am wondering if there are pipeline API's for ETL? >> >> Thanks! >> >>

Re: Are there pipeline API's for ETL?

pends on your business logic. > > Here are the generic APIs list.[1] > > Best, > Vino > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html > > kant kodali 于2020年1月11日周六 上午9:06写道： >> Hi All, >> >> I am wondering if there are pipeline API's for ETL? >> >> Thanks! >> >>

Are there pipeline API's for ETL?

Hi All, I am wondering if there are pipeline API's for ETL? Thanks!

Re: How to emit changed data only w/ Flink trigger?

I am new to Flink so I am not sure if I am giving you the correct answer so you might want to wait for others to respond. But I think you should do .inUpsertMode() On Fri, Nov 1, 2019 at 2:38 AM Qi Kang wrote: > Hi all, > > > We have a Flink job which aggregates sales volume and GMV data of

is Streaming Ledger open source?

Hi All, Is https://github.com/dataArtisans/da-streamingledger an open-source project? Looks to me that this project is not actively maintained. is that correct? since the last commit is one year ago and it shows there are 0 contributors? Thanks!

Re: How to stream intermediate data that is stored in external storage?

le C. That upsert > action itself serves as a join function, there's no need to join in Flink > at all. > > There are many tools out there can be used for that ingestion. Flink, of > course, can be used for that purpose. But for me, it's an overkill. > > Regards, > Averell > &

Re: How to stream intermediate data that is stored in external storage?

Hi Averell, yes, I want to run ad-hoc SQL queries on the joined data as well as data that may join in the future. For example, let's say if you take datasets A and B in streaming mode a row in A can join with a row B in some time in future let's say but meanwhile if I query the intermediate

Are Dynamic tables backed by rocksdb?

Hi All, Are Dynamic tables backed by Rocksdb or in memory? if they are backed by RocksDB can I use SQL to query the state? Thanks!

Re: How to stream intermediate data that is stored in external storage?

Hi Averell, I want to write intermediate results (A join B) incrementally and in real-time to some external storage so I can query it using SQL. I am new to Flink so I am trying to find out if 1) such mechanism exists? 2) If not, what are the alternatives? Thanks On Thu, Oct 31, 2019 at 1:42 AM

Re: How to stream intermediate data that is stored in external storage?

never used it. > > I think you would have to implement your own custom operator that would > output changes to it’s internal state as a side output. > > Piotrek > > On 30 Oct 2019, at 16:14, kant kodali wrote: > > Hi Piotr, > > I am talking about the internal sta

Re: How to stream intermediate data that is stored in external storage?

/libs/state_processor_api.html > [2] https://flink.apache.org/feature/2019/09/13/state-processor-api.html > [3] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html > > On 29 Oct 2019, at 16:42, kant kodali wrote: > > Hi All, > >

How to stream intermediate data that is stored in external storage?

Hi All, I want to do a full outer join on two streaming data sources and store the state of full outer join in some external storage like rocksdb or something else. And then want to use this intermediate state as a streaming source again, do some transformation and write it to some external

Re: can we do Flink CEP on event stream or batch or both?

processing semantics. > > Best, > Fabian > > > Am Di., 30. Apr. 2019 um 06:49 Uhr schrieb kant kodali >: > >> Hi All, >> >> I have the following questions. >> >> 1) can we do Flink CEP on event stream or batch? >> 2) If we can do streami

https://github.com/google/zetasql

https://github.com/google/zetasql

can we do Flink CEP on event stream or batch or both?

Hi All, I have the following questions. 1) can we do Flink CEP on event stream or batch? 2) If we can do streaming I wonder how long can we keep the stream stateful? I also wonder if anyone successfully had done any stateful streaming for days or months(with or without CEP)? or is stateful

Re: status on FLINK-7129

ote: > >> +1 >> >> On Tue, Apr 23, 2019, 4:57 AM kant kodali wrote: >> >>> Thanks all for the reply. I believe this is one of the most important >>> feature that differentiates flink from other stream processing engines as >>> others don't even have CEP

Re: status on FLINK-7129

te: > > Hi Kant, > > as far as I know, no one is currently working on this. Dawid (cc) maybe > knows more. > > Cheers, > > Konstantin > > On Sat, Apr 20, 2019 at 12:12 PM kant kodali wrote: > >> Hi All, >> >> There seems to be a lot of inte

status on FLINK-7129

Hi All, There seems to be a lot of interest for https://issues.apache.org/jira/browse/FLINK-7129 Any rough idea on the status of this issue? Thanks!

Re: Does Flink support stream-stream outer joins in the latest version?

t; >> SELECT * >> FROM Orders o, Shipments s >> WHERE o.id = s.orderId AND >> o.ordertime BETWEEN s.shiptime - INTERVAL '4' HOUR AND s.shiptime > > > Non-windowed Join: > >> SELECT * >> FROM Orders o, Shipments s >> WHERE o.id =

Re: Does Flink support stream-stream outer joins in the latest version?

on 1.5 and the non-windowed outer >> join is still work in progress. >> >> Hope that helps. >> >> Best, >> Xingcan >> >> [1] https://ci.apache.org/projects/flink/flink-docs-master/ >> dev/table/tableApi.html#joins >> [2] https://ci.apache.org/p

Does Flink support stream-stream outer joins in the latest version?

Hi All, Does Flink support stream-stream outer joins in the latest version? Thanks!

Re: Does Queryable State only support K/V queries not SQL?

> Queryable state only supports key point queries, i.e., you can query a > keyed state for the value of a key. > Support for SQL is not on the roadmap. > > Best, Fabian > > 2018-02-25 14:26 GMT+01:00 kant kodali <kanth...@gmail.com>: > >> Hi All, >> >> 1

Re: Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

Thanks a lot! On Mon, Feb 26, 2018 at 9:19 AM, Nico Kruber <n...@data-artisans.com> wrote: > Judging from the code, you should separate different jars with a colon > ":", i.e. "—addclasspath jar1:jar2" > > > Nico > > On 26/02/18 10:36, k

Re: Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

h allows you to specify paths to the dependency > jars. > > Cheers, > Gordon > > > On 25 February 2018 at 12:22:28 PM, kant kodali (kanth...@gmail.com) > wrote: > > Exception went away after downloading > flink-connector-kafka-base_2.11-1.4.1.jar > to lib folder

Does Queryable State only support K/V queries not SQL?

Hi All, 1) Does Queryable State support SQL? By which I mean I can do issue a full-fledged sql query like say ("select * from table where foo='hello' group by name") 2) Does Queryable state support offset and limit? Because if I have a million rows I don't want to get all at once. Sorry if

Re: Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

Exception went away after downloading flink-connector-kafka-base_2.11-1.4.1.jar to lib folder On Sat, Feb 24, 2018 at 6:36 PM, kant kodali <kanth...@gmail.com> wrote: > Hi, > > I couldn't get flink and kafka working together. It looks like all > examples I tried f

Re: How to create TableEnvrionment using scala-shell

Please ignore this. I fixed it by moving opt/flink-table_2.11-1.4.1.jar to lib/flink-table_2.11-1.4.1.jar On Sat, Feb 24, 2018 at 4:06 AM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > I am new to Flink and I am wondering how to create a TableEnvironment in > s

Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

Hi, I couldn't get flink and kafka working together. It looks like all examples I tried from web site fails with the following Exception. Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase *or when I do something like this like it is

How to create TableEnvrionment using scala-shell

Hi All, I am new to Flink and I am wondering how to create a TableEnvironment in scala-shell? I get an import error below. I am using Flink 1.4.1 63:error: object table is not a member of package org.apache.flink` I tried to do the following ./start-scala-shell.sh local import

Re: Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

t; client that way. > The submission would work with the regular Flink job client, i.e., it > would pickup the regular Flink config. > > Best, Fabian > > 2017-09-08 10:05 GMT+02:00 kant kodali <kanth...@gmail.com>: > >> Hi Fabian, >> >> Thanks for the response. I

Re: Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

inks.html > [2] https://issues.apache.org/jira/browse/FLINK-7594 > > > 2017-09-07 21:43 GMT+02:00 kant kodali <kanth...@gmail.com>: > >> Hi All, >> >> Does Flink has a JDBC server where I can submit Calcite Streaming >> Queries? such that I get Stream of responses back from Flink forever via >> JDBC ? What is the standard way to do this? >> >> Thanks, >> Kant >> > >

Does Flink has a JDBC server where I can submit Calcite Streaming Queries?

Hi All, Does Flink has a JDBC server where I can submit Calcite Streaming Queries? such that I get Stream of responses back from Flink forever via JDBC ? What is the standard way to do this? Thanks, Kant

Re: can flink do streaming from data sources other than Kafka?

(Gordon) Tai <tzuli...@apache.org> wrote: > Ah, I see. I’m not aware of any existing work / JIRAs on streaming sources > for Cassandra or HBase, only sinks. > If you are interested in one, could you open JIRAs for them? > > > On 7 September 2017 at 4:11:05 PM, kant kodali (k

Re: can flink do streaming from data sources other than Kafka?

https://ci.apache.org/projects/flink/flink-docs- > release-1.3/dev/connectors/index.html > [2] http://bahir.apache.org/ > [3] https://issues.apache.org/jira/browse/FLINK-4266 > > On 7 September 2017 at 2:58:38 PM, kant kodali (kanth...@gmail.com) wrote: > > Hi All, > > I am wonde

can flink do streaming from data sources other than Kafka?

Hi All, I am wondering if Flink can do streaming from data sources other than Kafka. For example can Flink do streaming from a database like Cassandra, HBase, MongoDb to sinks like says Elastic search or Kafka. Also for out of core stateful streaming. Is RocksDB the only option? Can I use some

Comparsion between Flink vs Kafka Stream Processing

Hi All, I have simple question. Here is a article that addresses the differences between Flink vs Kafka Streaming (in fact there is a table if you scroll down). While I understand those are the

Hi

Hi All, I read the docs however I still have the following question For Stateful stream processing is HDFS mandatory? because In some places I see it is required and other places I see that rocksDB can be used. I just want to know if HDFS is mandatory for Stateful stream processing? Thanks!

Re: Can we do batch writes on cassandra using flink while leveraging the locality?

Cassandra… if you are maintaining state in Flink and the > partitioning changes, your job might produce inaccurate output. If, on the > other hand, you are only using the partitioner just before the output, > dynamic partitioning changes might be ok. > > > From: kant kodali <kanth...@gmail.

Can we do batch writes on cassandra using flink while leveraging the locality?

Can we do batch writes on Cassandra using Flink while leveraging the locality? For example the batch writes in Cassandra will put pressure on the coordinator but since the connectors are built by leveraging the locality I was wondering if we could do batch of writes on a node where the batch