imilar to your filter/aggregate previously computed spark results.
>
> Regards,
> Yohann
>
>
> --
> *De :* Rick Moritz <rah...@gmail.com>
> *Envoyé :* jeudi 16 mars 2017 10:37
> *À :* user
> *Objet :* Re: RE: Fast write datastore...
>
&
lar to
> your filter/aggregate previously computed spark results.
>
> Regards,
> Yohann
>
> De : Rick Moritz <rah...@gmail.com>
> Envoyé : jeudi 16 mars 2017 10:37
> À : user
> Objet : Re: RE: Fast write datastore...
>
> If you have enough RAM/SSDs avail
017 10:37
À : user
Objet : Re: RE: Fast write datastore...
If you have enough RAM/SSDs available, maybe tiered HDFS storage and Parquet
might also be an option. Of course, management-wise it has much more overhead
than using ES, since you need to manually define partitions and buckets,
>>
>>
>>
>>
>>
>>
>> *From:* Vova Shelgunov [mailto:vvs...@gmail.com]
>> *Sent:* Wednesday, March 15, 2017 11:51 PM
>> *To:* Muthu Jayakumar <bablo...@gmail.com>
>> *Cc:* vincent gromakowski <vincent.gromakow...@gmail.com>; Rich
makowski <vincent.gromakow...@gmail.com>; Richard
> Siebeling <rsiebel...@gmail.com>; user <user@spark.apache.org>; Shiva
> Ramagopal <tr.s...@gmail.com>
> *Subject:* Re: Fast write datastore...
>
>
>
> Hi Muthu,.
>
>
>
> I did not catch from yo
ow...@gmail.com>; Richard Siebeling <rsiebel...@gmail.com>; user <user@spark.apache.org>; Shiva Ramagopal <tr.s...@gmail.com>
Subject: Re: Fast write datastore...
Hi Muthu,.
I did not catch from your message, what performance do you expect from subsequent queries?
om>; user <user@spark.apache.org>; Shiva Ramagopal
<tr.s...@gmail.com>
Subject: Re: Fast write datastore...
Hi Muthu,.
I did not catch from your message, what performance do you expect from
subsequent queries?
Regards,
Uladzimir
On Mar 15, 2017 9:03 PM, "Muthu Jayakumar"
<b
perform some aggregate
>>>>>> queries
>>>>>> using Spark Dataframe. I would like to find a reasonable fast
>>>>>> datastore that
>>>>>> allows me to write the results for subsequent (simpler queries).
>>>>>> I did attempt to use ElasticSearch to write the query results using
>>>>>> ElasticSearch Hadoop connector. But I am running into connector write
>>>>>> issues
>>>>>> if the number of Spark executors are too many for ElasticSearch to
>>>>>> handle.
>>>>>> But in the schema sense, this seems a great fit as ElasticSearch has
>>>>>> smartz
>>>>>> in place to discover the schema. Also in the query sense, I can
>>>>>> perform
>>>>>> simple filters and sort using ElasticSearch and for more complex
>>>>>> aggregate,
>>>>>> Spark Dataframe can come back to the rescue :).
>>>>>> Please advice on other possible data-stores I could use?
>>>>>>
>>>>>> Thanks,
>>>>>> Muthu
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context: http://apache-spark-user-list.
>>>>>> 1001560.n3.nabble.com/Fast-write-datastore-tp28497.html
>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>> Nabble.com.
>>>>>>
>>>>>> -
>>>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>>> queries
>>>>> using Spark Dataframe. I would like to find a reasonable fast
>>>>> datastore that
>>>>> allows me to write the results for subsequent (simpler queries).
>>>>> I did attempt to use ElasticSearch to write the quer
; simple filters and sort using ElasticSearch and for more complex aggregate,
> Spark Dataframe can come back to the rescue :).
> Please advice on other possible data-stores I could use?
>
> Thanks,
> Muthu
>
>
>
> --
> View this message in context: http://apache-spark-
me to write the results for subsequent (simpler queries).
>>>>> I did attempt to use ElasticSearch to write the query results using
>>>>> ElasticSearch Hadoop connector. But I am running into connector write
>>>>> issues
>>>>> if the number
>> if the number of Spark executors are too many for ElasticSearch to
>>>> handle.
>>>> But in the schema sense, this seems a great fit as ElasticSearch has
>>>> smartz
>>>> in place to discover the schema. Also in the query sense, I can p
to
>>> handle.
>>> But in the schema sense, this seems a great fit as ElasticSearch has
>>> smartz
>>> in place to discover the schema. Also in the query sense, I can perform
>>> simple filters and sort using ElasticSearch and for more complex
>>&
ch has
>> smartz
>> in place to discover the schema. Also in the query sense, I can perform
>> simple filters and sort using ElasticSearch and for more complex
>> aggregate,
>> Spark Dataframe can come back to the rescue :).
>> Please advice on other possible data-s
e?
>
> Thanks,
> Muthu
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Fast-write-datastore-tp28497.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
>
).
Please advice on other possible data-stores I could use?
Thanks,
Muthu
--
View this message in context: http://apache-spark-user-list.
1001560.n3.nabble.com/Fast-write-datastore-tp28497.html
Sent from the Apache Spark User List mailing list archi
perform
simple filters and sort using ElasticSearch and for more complex aggregate,
Spark Dataframe can come back to the rescue :).
Please advice on other possible data-stores I could use?
Thanks,
Muthu
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Fast-write
17 matches
Mail list logo
