Ah, you're right. I misread the original question. In that case, also try checking out: https://github.com/implydata/druid-hadoop-inputformat, an unofficial Druid InputFormat. Spark can use that to read Druid data into an RDD - check the example in the README. It's also unofficial and, currently, unmaintained, so you'd be taking on some maintenance effort if you want to use it.
On Wed, Feb 6, 2019 at 3:01 PM Julian Jaffe <[email protected]> wrote: > I think this question is going the other way (e.g. how to read data into > Spark, as opposed to into Druid). For that, the quickest and dirtiest > approach is probably to use Spark's json support to parse a Druid response. > You may also be able to repurpose some code from > https://github.com/SparklineData/spark-druid-olap, but I don't think > there's any official guidance on this. > > On Wed, Feb 6, 2019 at 2:21 PM Gian Merlino <[email protected]> wrote: > > > Hey Rajiv, > > > > There's an unofficial Druid/Spark adapter at: > > https://github.com/metamx/druid-spark-batch. If you want to stick to > > official things, then the best approach would be to use Spark to write > data > > to HDFS or S3 and then ingest it into Druid using Druid's Hadoop-based or > > native batch ingestion. (Or even write it to Kafka using Spark Streaming > > and ingest from Kafka into Druid using Druid's Kafka indexing service.) > > > > On Wed, Feb 6, 2019 at 12:04 PM Rajiv Mordani > <[email protected] > > > > > wrote: > > > > > Is there a best practice for how to load data from druid to use in a > > spark > > > batch job? I asked this question on the user alias but got no response > > > hence reposting here. > > > > > > > > > * Rajiv > > > > > >
