Hi, Thanks for reaching out!
First of all, I would like to point out that an interesting alternative to the per-job cluster could be running your jobs in application mode [1]. Given that you want to run arbitrary SQL queries, I do not think you can "share" across queries the part of the job graph that reads a topic. In general, Flink (not only in SQL) creates the graph of a job before the job is executed. And especially in SQL you do not even have control over the graph, as the translation logic from query to physical operators is opaque and not exposed to the user. That said, you may want to have a look at [2]. It is pretty old but it describes a potentially similar usecase. Unfortunately, it does not support SQL. Cheers, Kostas [1] https://flink.apache.org/news/2020/07/14/application-mode.html [2] https://www.ververica.com/blog/rbea-scalable-real-time-analytics-at-king On Sun, Nov 15, 2020 at 10:11 AM lalala <lal...@activist.com> wrote: > > Hi all, > > I would like to consult with you regarding deployment strategies. > > We have +250 Kafka topics that we want users of the platform to submit SQL > queries that will run indefinitely. We have a query parsers to extract topic > names from user queries, and the application locally creates Kafka tables > and execute the query. The result can be collected to multiple sinks such as > databases, files, cloud services. > > We want to have the best isolation between queries, so in case of failures, > the other jobs will not get affected. We have a huge YARN cluster to handle > 1PB a day scale from Kafka. I believe cluster per job type deployment makes > sense for the sake of isolation. However, that creates some scalability > problems. There might be SQL queries running on the same Kafka topic that we > do not want to read them again for each query in different sessions. The > ideal case is that we read the topic once and executes multiple queries on > this data to avoid rereading the same topic. That breaks the desire of a > fully isolated system, but it improves network and Kafka performance and > still provides isolation on the topic level as we just read the topic once > and execute multiple SQL queries on it. > > We are quite new to Flink, but we have experience with Spark. In Spark, we > can submit an application, and in master, that can listen a query queue and > submit jobs to the cluster dynamically from different threads. However, In > Flink, it looks like the main() has to produce the job the graph in advance. > > We do use an EMR cluster; what would you recommend for my use case? > > Thank you. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
