Agreed
But the servers do not have so much of capacity.
Hence only in memory data grid works for us.
On Wed, 19 Jul, 2023, 6:35 pm Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:
> Ignite is horizontally scalable. It can use as much memory as you have.
> 1Tb isn’t *that* much; I
Ignite is horizontally scalable. It can use as much memory as you have. 1Tb
isn’t that much; I know of people with over 20Tb of data in memory. If you
don’t want to keep everything in memory, you can use native persistence and
keep less used data only on disk.
> On 19 Jul 2023, at 11:52,
Just an example. I mean to say a part of my data resides in Ignite, not the
complete data.
Rest data is present in Spark.
On Wed, 19 Jul, 2023, 2:07 pm Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:
> Why you would have Ignite, a horizontally scalable, in-memory database, to
>
My data is around 1TB huge
I don't think so much can be loaded into memory
Plus my data is in parquet files. So loading it to spark and writing to
Ignite is very time consuming.
Any thoughts on this please.
Regards
On Wed, 19 Jul, 2023, 2:14 pm Stephen Darlington, <
As a follow up to this:
We tried removing both those in the walstore and walarchive. Problem is
that somewhere there is a checkpoint that says its up to wal index
2414...yet we only have 2413...2412...etc
We need to find where it stores this checkpoint index and change it, it
seems.
On Wed, 19
Just FYI, we have held off any memory pressure changes in the meantime
while we continue to investigate the memory issues we have.
On Tue, 18 Jul 2023 at 9:07 AM, Raymond Wilson
wrote:
> Hi Pavel,
>
> This area is confusing. There is no indication that the memory pressure
> applies to any
A more common approach would be that Ignite has all your data and Spark has a
subset. Ignite SQL is generally faster than Spark SQL, since it can use
indexes, etc. But it’s not magic; it can’t query data it doesn’t know about.
> On 19 Jul 2023, at 04:40, Arunima Barik wrote:
>
> How does
Why you would have Ignite, a horizontally scalable, in-memory database, to
store 100 records?
> On 19 Jul 2023, at 04:37, Arunima Barik wrote:
>
> I have a huge dataset and I am keeping few (say 100) rows in Ignite and the
> entire dataset remains in Spark
>
> When I query Ignite I want to
