Hello folks, I am new to Apache Livy and currently trying to understand how feasible will Livy be for interactive query for a 300 user app.
Latency to first results is critical for customer experience. The biggest concern I have is the 1000 limit associated with the take/collect. Most of the ad-how queries will return more than 10k rows easily. In my view there are two options: 1- Livy batch submission with S3 as the destination. Then read the results from the app from S3. This will not be the best experience as customer can’t see results right away. 2- interactive query submission via Livy. Then add a mechanism to perform pagination or write results to S3 if more than 1000 rows returned. The app would know this query has more than 1000 rows and automatically start paginating from S3 after 1000. My question: how have other Livy users with a requirement of low latency to first result solve this? Thank you, Gos.
