perform
simple filters and sort using ElasticSearch and for more complex aggregate,
Spark Dataframe can come back to the rescue :).
Please advice on other possible data-stores I could use?
Thanks,
Muthu
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Fast-write
sible data-stores I could use?
Thanks,
Muthu
--
View this message in context: http://apache-spark-user-list.
1001560.n3.nabble.com/Fast-write-datastore-tp28497.html
Sent from the Apache Spark User List mailing list archive at
uthu
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Fast-write-datastore-tp28497.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
>
the schema. Also in the query sense, I can perform
>> simple filters and sort using ElasticSearch and for more complex
>> aggregate,
>> Spark Dataframe can come back to the rescue :).
>> Please advice on other possible data-stores I could use?
>>
>> Thanks,
>>
a great fit as ElasticSearch has
>>> smartz
>>> in place to discover the schema. Also in the query sense, I can perform
>>> simple filters and sort using ElasticSearch and for more complex
>>> aggregate,
>>> Spark Datafr
mpt to use ElasticSearch to write the query results using
>>>> ElasticSearch Hadoop connector. But I am running into connector write
>>>> issues
>>>> if the number of Spark executors are too many for ElasticSearch to
>>>> handle.
>>>> But in the
>>>>> using Spark Dataframe. I would like to find a reasonable fast
>>>>> datastore that
>>>>> allows me to write the results for subsequent (simpler queries).
>>>>> I did attempt to use ElasticSearch to write the query results using
>&
sing ElasticSearch and for more complex aggregate,
> Spark Dataframe can come back to the rescue :).
> Please advice on other possible data-stores I could use?
>
> Thanks,
> Muthu
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.n
>>>>> allows me to write the results for subsequent (simpler queries).
>>>>> I did attempt to use ElasticSearch to write the query results using
>>>>> ElasticSearch Hadoop connector. But I am running into con
7:04 AM, "muthu" a écrit :
>>>>>>
>>>>>> Hello there,
>>>>>>
>>>>>> I have one or more parquet files to read and perform some aggregate
>>>>>> queries
>>>>>> using S
Hi,
Will MongoDB not fit this solution?
From: Vova Shelgunov [mailto:vvs...@gmail.com]
Sent: Wednesday, March 15, 2017 11:51 PM
To: Muthu Jayakumar
Cc: vincent gromakowski ; Richard Siebeling
; user ; Shiva Ramagopal
Subject: Re: Fast write datastore...
Hi Muthu,.
I did not catch from
agopal
Subject: Re: Fast write datastore...
Hi Muthu,.
I did not catch from your message, what performance do you expect from subsequent queries?
Regards,
Uladzimir
On Mar 15, 2017 9:03 PM, "Muthu Jayakumar" <bablo...@gmail.com> wrote:
H
wrote:
>
> Hi,
>
>
>
> Will MongoDB not fit this solution?
>
>
>
>
>
>
>
> *From:* Vova Shelgunov [mailto:vvs...@gmail.com]
> *Sent:* Wednesday, March 15, 2017 11:51 PM
> *To:* Muthu Jayakumar
> *Cc:* vincent gromakowski ; Richard
> Siebeling ; user ; Shiva
&g
;
>>
>> *From:* Vova Shelgunov [mailto:vvs...@gmail.com]
>> *Sent:* Wednesday, March 15, 2017 11:51 PM
>> *To:* Muthu Jayakumar
>> *Cc:* vincent gromakowski ; Richard
>> Siebeling ; user ; Shiva
>> Ramagopal
>> *Subject:* Re: Fast write datastore...
>&
Objet : Re: RE: Fast write datastore...
If you have enough RAM/SSDs available, maybe tiered HDFS storage and Parquet
might also be an option. Of course, management-wise it has much more overhead
than using ES, since you need to manually define partitions and buckets, which
is suboptimal. On the
/aggregate previously computed spark results.
>
> Regards,
> Yohann
>
> De : Rick Moritz
> Envoyé : jeudi 16 mars 2017 10:37
> À : user
> Objet : Re: RE: Fast write datastore...
>
> If you have enough RAM/SSDs available, maybe tiered HDFS storage and Parquet
> might al
ate previously computed spark results.
>
> Regards,
> Yohann
>
>
> --
> *De :* Rick Moritz
> *Envoyé :* jeudi 16 mars 2017 10:37
> *À :* user
> *Objet :* Re: RE: Fast write datastore...
>
> If you have enough RAM/SSDs available, maybe ti
17 matches
Mail list logo
