Ignite can use RDDs or DataFrames. If you’re using a DataFrame, that’s the way to do it. Depending on how you configure it, it will create or use an existing table/cache in Ignite.
> On 3 Jul 2023, at 13:31, Arunima Barik <[email protected]> wrote: > > I am reading a parquet file using Spark dataframe as df > > I want to write some part of this data to ignite cache > > Assume I want to write df2 to the cache > > I used df2.write.format('ignite') > Is there a better way to do this or this is the only way?? > > Regards > Arunima > > On Mon, 3 Jul, 2023, 1:19 pm Stephen Darlington, > <[email protected] <mailto:[email protected]>> > wrote: >> Commercial options are available, but otherwise help would generally be >> limited to email lists and Stack Overflow. >> >>> On 1 Jul 2023, at 06:59, Arunima Barik <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Are there any provisions wherein I can discuss about my project >>> implementation with someone from the Ignite team to clarify some doubts? >>> >>> Preferably through a small online meet? >>> >>> Regards >>> Arunima >>> >>> On Sat, 1 Jul, 2023, 12:03 am Jeremy McMillan, >>> <[email protected] <mailto:[email protected]>> wrote: >>>> Python doesn't at this time go anywhere near Ignite CacheStore. You would >>>> need to implement the CacheStore in Java or some other language which >>>> compiles to JVM runtime/jar. There's a talk from the most recent summit on >>>> using Groovy, if you want a higher level language than Java, but >>>> theoretically you could use Jython (if you are willing to experiment and >>>> can find a compatible JVM that runs both Ignite and Jython). >>>> >>>> Ignite can operate like a federated query proxy if different caches are >>>> implemented with different external persistence for each cache. CacheStore >>>> is the interface Ignite would use to send a cache miss to a backend >>>> database. In your original question you intended to use Parquet files as a >>>> backend database, but Ignite does not (yet) provide one for Parquet. If >>>> someone were to donate a supportable Java implementation, I suspect the >>>> community would adopt and support it. Since Parquet is columnar, I also >>>> suspect it would need to target Ignite 3 to adopt conventions around >>>> columnar data, and then might be backported to Ignite 2. >>>> >>>> >>>> On Fri, Jun 30, 2023 at 12:13 PM Arunima Barik <[email protected] >>>> <mailto:[email protected]>> wrote: >>>>> Which do you think would be a better option? >>>>> >>>>> Federated queries or CacheStore >>>>> >>>>> And is CacheStore supported in Python? >>>>> >>>>> On Fri, 30 Jun, 2023, 1:50 pm Stephen Darlington, >>>>> <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>>> You’d need to implement your own Cache Store. >>>>>> https://ignite.apache.org/docs/latest/persistence/custom-cache-store >>>>>> >>>>>>> On 30 Jun 2023, at 06:46, Arunima Barik <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> >>>>>>> ---------- Forwarded message --------- >>>>>>> From: Arunima Barik <[email protected] >>>>>>> <mailto:[email protected]>> >>>>>>> Date: Fri, 30 Jun, 2023, 10:52 am >>>>>>> Subject: Ignite for Parquet files >>>>>>> To: <[email protected] <mailto:[email protected]>> >>>>>>> >>>>>>> >>>>>>> Hello Team >>>>>>> >>>>>>> I have my data stored as parquet files. I want a caching layer on top >>>>>>> of this existing file system. I am going to use Ignite for that but I >>>>>>> do not need native persistence for that. >>>>>>> >>>>>>> I want that any changes to database should be reflected in both cache >>>>>>> and file. >>>>>>> And same for read queries. It should automatically read from disk if >>>>>>> data is not present in cache. >>>>>>> >>>>>>> I want to do all this is python. Please let me know how the same can be >>>>>>> done. >>>>>>> Resources if any as well. >>>>>>> >>>>>>> Thank you and looking forward to hearing from you. >>>>>>> >>>>>>> Regard, >>>>>>> Arunima Barik >>>>>> >>
