Re: Can Apache Drill perform streaming queries?

Hi Saurabh, Yes those concept do exist in Spark SQL and Spark in general is awesome but what Spark SQL lacks is the REST interface where user can submit normal or streaming queries via REST and get the results out . Right now, a user have to write imperative code to achieve whatever they want and

Re: Can Apache Drill perform streaming queries?

You are correct Kant. It will be great, If you can raise a JIRA for discussing *feasibility* of incremental query support for Drill. Because, I can also see this is a very good requirement for plugins like Kafka, HBase and Cassandra and thanks for asking this question. Thanks & Regards, B Anil

Re: Can Apache Drill perform streaming queries?

HI Anil, Thanks a lot for your response and look like I am indeed looking for incremental queries. so if I have a thread that polls every second to get the latest updates I just have to change partition values to minimize the scans right? Also I guess I can build some notification mechanism in

Re: Can Apache Drill perform streaming queries?

Hi Kant, If I understand your questions properly, you are looking for incremental queries. Drill supports predicates pushed down with most of the Data sources. In your case, suppose you are generating hourly partitions in HDFS using Spark aplication. Then Drill is optmized to scan specific

Re: Can Apache Drill perform streaming queries?

Can someone elaborate on what happens underneath if I poll every second (Specifically related to my questions in my previous email)? Thanks! On Thu, Nov 9, 2017 at 7:56 AM, Ted Dunning wrote: > Confluent has a non-Apache product, I think, for streaming SQL. > > > On Thu,

Re: Can Apache Drill perform streaming queries?

Confluent has a non-Apache product, I think, for streaming SQL. On Thu, Nov 9, 2017 at 4:50 PM, Saurabh Mahapatra wrote: > Isn't there the new Kafka plugin? What does that exactly do? > > Best, > Saurabh > > Sent from my iPhone > > > > > On Nov 9, 2017, at 5:15 AM, kant

Re: Can Apache Drill perform streaming queries?

Isn't there the new Kafka plugin? What does that exactly do? Best, Saurabh Sent from my iPhone > On Nov 9, 2017, at 5:15 AM, kant kodali wrote: > > Hi Tug, > > It's Parquet data on HDFS and the data to HDFS is constantly written by > spark while consuming from Kafka. >

Re: Can Apache Drill perform streaming queries?

Hi Tug, It's Parquet data on HDFS and the data to HDFS is constantly written by spark while consuming from Kafka. Is polling a common technique for say real time analytics dashboard ? More importantly if I poll does Drill due the scan every time? if the answer is no, how does it know which is

Re: Can Apache Drill perform streaming queries?

Hello, Today Drill cannot do continuous/streaming query, so as you mentioned you will have to use a polling technique. Just out of curiosity, Which data source are you planning to use ? Regards Tug On Thu 9 Nov 2017 at 04:31, kant kodali wrote: > Hi All, > > I am new

Can Apache Drill perform streaming queries?

Hi All, I am new to Apache Drill. I am wondering if Apache Drill can perform Streaming Queries? For example, I have a constant stream of data in 24 hour period and I would like to get updates as soon as I receive them. Do I need to have a polling thread that issues a Drill query every second?