Hello,
In short, Ignite persistence primary advantages are:
- Enables multi-tiered storage across RAM and disk - 100% of data
persisted to disk ("warm" and "cold" data sets) while "hot" data always
stays in RAM. You lay out the data the way you need. Applications just run
queries and Ignite internally goes either to RAM or disk, transparently.
This property is crucial for real-time analytics and data warehousing
offloading.
- Instantaneous restarts and advanced high-availability - in case of
full cluster restarts, the cluster becomes fully operational as soon as the
nodes are interconnected. No need to preload anything from disk to RAM
(based on the advantage above).
Now, talking about Ignite and data warehousing. Ignite is used for
real-time analytics and Hadoop offloading. But don't treat it as a Hadoop
replacement or a solution for data warehousing. Ignite is used together
with Hadoop but deployed separately. Here are things to consider:
- Use Ignite for business operations/computations that require
low-latency response time (seconds or milliseconds) and high-throughput.
Preload data to Ignite cluster needed for this computations. Enabled Ignite
persistence for the sake of durability.
- Keep using Hadoop for high-latency workloads (minutes and hours) and
batch processing.
- APIs: modify your applications to ensure that Ignite APIs are used for
Ignite cluster access (SQL, compute grid, ML). Spark can be used as a
generic API that can connect to both Hadoop and Ignite and run joins across
2 storages (use DataFrames).
- Tooling: data preloading from Hadoop, bi-directional synchronization,
advanced Spark integration, etc. - reach out GridGain, they have been
working on a special Hadoop pack.
Hope it helps.
-
Denis
On Tue, May 14, 2019 at 3:20 PM m4mmr <rmam...@gmail.com> wrote:
> Hi,
>
> I am in a project where we are building a new database with strong
> normalisation requirements - very much like a relational data warehouse.
> We
> source the data from HDFS. And the maintenance team requires the data
> movement to be implemented through SQL APIs.
>
> Main Question: Would it be a viable use case to use Ignite with native
> persistence store as storage when building a relational data warehouse? So
> that we do not have to manage both a RDBMS and Ignite.
>
> I don’t fully see a clear picture of what the limitations would be of
> using
> native persistence instead of RDBMS for persistence - but I have not seen a
> single use case where someone use native for relational data warehouses
> either.
>
> Again - I think I am missing some basic understanding of the native
> persistence here - even after reading through the docs I could find. So
> would be happy if someone could shed some light on it.
>
> Would be very thankful for all type of help/assistance!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
