Trigger on GroupStateTimeout with no new data in group

2021-02-11 Thread Abhishek Gupta
Hi All, I had a question about modeling a user session kind of analytics use-case in Spark Structured Streaming. Is there a way to model something like this using Arbitrary stateful Spark streaming User session -> reads a few FAQS on a website and then decides to create a ticket or not FAQ Deflec

Ingesting data in parallel across workers in Data Frame

2017-01-20 Thread Abhishek Gupta
I am trying to load data from the database into DataFrame using JDBC driver.I want to get data into partitions the following document has the nice explanation how to achieve so. https://docs.databricks.com/spark/latest/data-sources/sql-databases.html