Any updates on this please...
On Mon, 3 Jul 2023 at 18:01, Arunima Barik <[email protected]> wrote:
> I am reading a parquet file using Spark dataframe as df
>
> I want to write some part of this data to ignite cache
>
> Assume I want to write df2 to the cache
>
> I used df2.write.format('ignite')
> Is there a better way to do this or this is the only way??
>
> Regards
> Arunima
>
> On Mon, 3 Jul, 2023, 1:19 pm Stephen Darlington, <
> [email protected]> wrote:
>
>> Commercial options are available, but otherwise help would generally be
>> limited to email lists and Stack Overflow.
>>
>> On 1 Jul 2023, at 06:59, Arunima Barik <[email protected]> wrote:
>>
>> Are there any provisions wherein I can discuss about my project
>> implementation with someone from the Ignite team to clarify some doubts?
>>
>> Preferably through a small online meet?
>>
>> Regards
>> Arunima
>>
>> On Sat, 1 Jul, 2023, 12:03 am Jeremy McMillan, <
>> [email protected]> wrote:
>>
>>> Python doesn't at this time go anywhere near Ignite CacheStore. You
>>> would need to implement the CacheStore in Java or some other language which
>>> compiles to JVM runtime/jar. There's a talk from the most recent summit on
>>> using Groovy, if you want a higher level language than Java, but
>>> theoretically you could use Jython (if you are willing to experiment and
>>> can find a compatible JVM that runs both Ignite and Jython).
>>>
>>> Ignite can operate like a federated query proxy if different caches are
>>> implemented with different external persistence for each cache. CacheStore
>>> is the interface Ignite would use to send a cache miss to a backend
>>> database. In your original question you intended to use Parquet files as a
>>> backend database, but Ignite does not (yet) provide one for Parquet. If
>>> someone were to donate a supportable Java implementation, I suspect the
>>> community would adopt and support it. Since Parquet is columnar, I also
>>> suspect it would need to target Ignite 3 to adopt conventions around
>>> columnar data, and then might be backported to Ignite 2.
>>>
>>>
>>> On Fri, Jun 30, 2023 at 12:13 PM Arunima Barik <[email protected]>
>>> wrote:
>>>
>>>> Which do you think would be a better option?
>>>>
>>>> Federated queries or CacheStore
>>>>
>>>> And is CacheStore supported in Python?
>>>>
>>>> On Fri, 30 Jun, 2023, 1:50 pm Stephen Darlington, <
>>>> [email protected]> wrote:
>>>>
>>>>> You’d need to implement your own Cache Store.
>>>>> https://ignite.apache.org/docs/latest/persistence/custom-cache-store
>>>>>
>>>>> On 30 Jun 2023, at 06:46, Arunima Barik <[email protected]>
>>>>> wrote:
>>>>>
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: Arunima Barik <[email protected]>
>>>>> Date: Fri, 30 Jun, 2023, 10:52 am
>>>>> Subject: Ignite for Parquet files
>>>>> To: <[email protected]>
>>>>>
>>>>>
>>>>> Hello Team
>>>>>
>>>>> I have my data stored as parquet files. I want a caching layer on top
>>>>> of this existing file system. I am going to use Ignite for that but I do
>>>>> not need native persistence for that.
>>>>>
>>>>> I want that any changes to database should be reflected in both cache
>>>>> and file.
>>>>> And same for read queries. It should automatically read from disk if
>>>>> data is not present in cache.
>>>>>
>>>>> I want to do all this is python. Please let me know how the same can
>>>>> be done.
>>>>> Resources if any as well.
>>>>>
>>>>> Thank you and looking forward to hearing from you.
>>>>>
>>>>> Regard,
>>>>> Arunima Barik
>>>>>
>>>>>
>>>>>
>>
