Re: How do I support schema change in Ignite as well as addition of new Cache?

[Sumit Sethia]

Hi,
Thanks for the suggestion. I used BinaryObjectBuilder and then create/drop
cache at runtime in case schema changes. I would like to understand more
about the underlying implementations and also want to contribute. Can you
point me to some documentation which talks about the design and the kinds
of problems which are faced and how is Ignite community trying to solve it.

Thanks,
Sumit.



On 2 October 2017 at 23:55, Denis Magda  wrote:

> Hi Sumit,
>
> In addition to the approach suggested by Nick you can go for DDL way:
> https://apacheignite.readme.io/docs#section-create-table
>
> You can create tables (and underlying caches), indexes in runtime. To tie
> such a table up with a MySQL layer you need to register a template with
> your CacheStore implementation and pass the template as a parameter to
> CREATE TABLE command:
> https://apacheignite.readme.io/docs#section-extended-parameters
>
> Ignite doesn’t automatically update its scheme in response to MySQL
> related changes. So this is what you need to do manually for both storages.
>
> —
> Denis
>
>
> On Sep 30, 2017, at 2:18 AM, Sumit Sethia  wrote:
>
> Hi,
> I want to dynamically create cache for new mysql table in Ignite. I don't
> have table's class definition at runtime. I regularly get new data in HDFS
> for those tables in avro format. I want to ingest that incremental data
> into Ignite Cache by creating cache at runtime if it doesn't exist for that
> table and then put data into cache. Also I want to query on that data. Plus
> what happens if schema changes in MySql ? Will BinaryObject in Ignite help
> in solving my use case ? I tried to read Binary Object documentation but
> couldn't get it. Please help.
>
> Thanks,
> Sumit.
>
>
>

Re: How do I support schema change in Ignite as well as addition of new Cache?

[Denis Magda]

Hi Sumit,

In addition to the approach suggested by Nick you can go for DDL way:
https://apacheignite.readme.io/docs#section-create-table

You can create tables (and underlying caches), indexes in runtime. To tie such 
a table up with a MySQL layer you need to register a template with your 
CacheStore implementation and pass the template as a parameter to CREATE TABLE 
command:
https://apacheignite.readme.io/docs#section-extended-parameters

Ignite doesn’t automatically update its scheme in response to MySQL related 
changes. So this is what you need to do manually for both storages.

—
Denis


> On Sep 30, 2017, at 2:18 AM, Sumit Sethia  wrote:
> 
> Hi,
> I want to dynamically create cache for new mysql table in Ignite. I don't 
> have table's class definition at runtime. I regularly get new data in HDFS 
> for those tables in avro format. I want to ingest that incremental data into 
> Ignite Cache by creating cache at runtime if it doesn't exist for that table 
> and then put data into cache. Also I want to query on that data. Plus what 
> happens if schema changes in MySql ? Will BinaryObject in Ignite help in 
> solving my use case ? I tried to read Binary Object documentation but 
> couldn't get it. Please help.
> 
> Thanks,
> Sumit.

Re: How do I support schema change in Ignite as well as addition of new Cache?

[Nikolai Tikhonov]

Hi,

You can create/drop caches (hence tables) in runtime. If we don't have
classes in runtime then you can configure via QueryEntitry. Refere the
following page https://apacheignite.readme.io/docs/indexes section *QueryEntity
Based Configuration*. In the next 2.3 release (which community plan to
release in nearest future) Apache Ignite will support alter table add
column command. You can track status this features by the following links:

https://issues.apache.org/jira/browse/IGNITE-5572
https://issues.apache.org/jira/browse/IGNITE-6283


On Sat, Sep 30, 2017 at 12:18 PM, Sumit Sethia 
wrote:

> Hi,
> I want to dynamically create cache for new mysql table in Ignite. I don't
> have table's class definition at runtime. I regularly get new data in HDFS
> for those tables in avro format. I want to ingest that incremental data
> into Ignite Cache by creating cache at runtime if it doesn't exist for that
> table and then put data into cache. Also I want to query on that data. Plus
> what happens if schema changes in MySql ? Will BinaryObject in Ignite help
> in solving my use case ? I tried to read Binary Object documentation but
> couldn't get it. Please help.
>
> Thanks,
> Sumit.
>

How do I support schema change in Ignite as well as addition of new Cache?

[Sumit Sethia]

Hi,
I want to dynamically create cache for new mysql table in Ignite. I don't
have table's class definition at runtime. I regularly get new data in HDFS
for those tables in avro format. I want to ingest that incremental data
into Ignite Cache by creating cache at runtime if it doesn't exist for that
table and then put data into cache. Also I want to query on that data. Plus
what happens if schema changes in MySql ? Will BinaryObject in Ignite help
in solving my use case ? I tried to read Binary Object documentation but
couldn't get it. Please help.

Thanks,
Sumit.