Re: Optimistic Serializable SQL Transactions

[Ivan Pavlukhin]

Hi Justin,

Thank you for sharing a details about your use case. Quite interesting.

It seems that in Ignite something like that is possible with
optimistic cache transactions (key-value API). Technically it seems to
be achievable when transaction protocol accumulates transaction
read-write set in memory area allocated for each transaction. Ignite
key-value transactions behaves so. But MVCC transactions was designed
to support SQL update operations as well which possible be larger than
available memory. So, new row versions are written in the same place
where a regular (already committed) data resides. Supporting multiple
not committed versions will complexify existing implementation a lot
(do not have good ideas from scratch how to implement it).

Really interesting if cockroachdb supports something like this.

сб, 19 окт. 2019 г. в 03:36, Justin Moore :
>
> Thanks Ivan,
>
> First, apologies, I don't have a simple example to share...
>
> I'm trying to explore the possibility answering "what-if" hypothetical 
> queries using (SQL) transactions that don't commit (i.e. transactions that 
> always rollback/abort).  Each what-if transaction would be like a 
> self-contained alternate reality that can surface information from an 
> alternate future -- but to do so effectively, the transaction ought to 
> process all statements, even though it will never (succeed to) commit, so it 
> must not abort early/unintentionally.
>
> I think a couple of other "distributed SQL" offerings (cockroachDB/yugabyte) 
> are architected in a way that makes this possible, where traditionally 
> "unclustered" databases (Postgres) generally seemed to rely on locking that 
> would prevent the capability.  I was -- and still am -- looking for other 
> (probably distributed) options that might make this feasible, which lead me 
> to the Ignite docs.
>
> My specific use case is to do something kind of like a Log Sorted Merge 
> representation with a twist, using a shared database and a shared Write Ahead 
> Log (the twist is that the WAL is not strictly append-only).  So concurrent 
> clients would assemble the current state on demand by applying the log to the 
> database -- which is a "snapshot" practically speaking -- in a transaction 
> that would not be committed (a separate "compaction" process would apply a 
> prefix of the log to be persisted permanently).  As such, concurrent clients 
> are going to be trying to do the exact same writes to the database in their 
> transactions -- they need not commit but all other statements should be 
> executed.
>
> Sorry if it's a bit confusing...
>
> Cheers,
> Justin
>
> On Fri, Oct 18, 2019 at 6:31 AM Ivan Pavlukhin  wrote:
>>
>> Hi,
>>
>> Currently there are no activity on optimistic transaction support for SQL.
>>
>> A transaction will be aborted on a first write conflict. May be I got
>> it wrong, but what is the benefit of aborting later (on commit)
>> instead of earlier (on write conflict)? Perhaps a scenario written in
>> terms of Ignite operations (cache.get, cache.put, cache.query) can
>> illustrate your problem better for my understanding.
>>
>> пт, 18 окт. 2019 г. в 06:58, Justin Moore :
>> >
>> > Hi,
>> >
>> > Very new to Ignite and just trying to assess its capabilities.
>> >
>> > tl;dr: are optimistic serializable sql transactions on the roadmap?
>> >
>> > It seems that currently only pessimistic repeatable_read sql transactions 
>> > are supported (in beta as of the current version -- 2.7).  I'm trying to 
>> > confirm that, in Ignite, if I started two concurrent transactions (from 
>> > the same snapshot) where one intends to execute statements #1 
>> > (compare-and-set), #2 (read-only), and #3 (whatever else), while the other 
>> > intends to execute the exact same update statements #1, #2, and #3, but 
>> > also a subsequent #4, understanding that (all but) one of those 
>> > transactions would probably fail to commit, I'm looking to clarify whether 
>> > or not the failing one would throw/abort even before reaching statement #2 
>> > (which might be a read returning values)?
>> >
>> > If I'm reading the docs correctly it seems that in pessimistic 
>> > repeatable_read mode the transaction would fail one of the transactions at 
>> > statement #1 (due to write conflict), but if it could have been optimistic 
>> > serializable, the transaction would simply rollback at the point a commit 
>> > was attempted.  Please correct me if I'm wrong.
>> >
>> > Lastly, are optimistic serializable sql transactions on the roadmap?
>> >
>> > Thanks
>>
>>
>>
>> --
>> Best regards,
>> Ivan Pavlukhin



-- 
Best regards,
Ivan Pavlukhin

[DISCUSS] Proposal for Ignite Extensions as a separate Bahir module or Incubator project

[Saikat Maitra]

Hello,

We wanted to discuss on a proposal to move and support the Apache Ignite
integrations as separate Ignite Extensions as discussed here
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Pub-Sub-Streamer-Implementation-td43944.html
.

The reason we wanted to move our Apache Ignite integration as  separate
Extensions is this will help us to manage and maintain separate lifecycle
for Apache Ignite integrations.

All the integrations will continue to be part of ASF and we will  keep
supporting and developing in accordance with ASF vision and practices.

We are considering following two choices for moving to Apache Ignite
Extensions:

1. Reach out to Apache Bahir community and propose to make Ignite
Extensions a separate module as part of Apache Bahir project.

https://bahir.apache.org/

https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces96


2. Reach out to Apache Incubator community and request for a new project
for Ignite Extensions.

Please review and share feedback on our proposal.

Warm Regards,
Saikat

Re: Ingite node in SpringBoot App

[niamin]

Looks like it is something with the using ARInvoiceId as key. All the other
combinations of key/value(String/String, String/ARInovice) can be retrieved
from thin client. I also serialized the ARInvoiceId object to a file from
client project and was able to to deserialize successfully from the server
project. So still not sure why using ARInvoiceId as key is not working



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ingite node in SpringBoot App

[niamin]

Looks like it is something with the using ARInvoiceId as key. All the other
combinations of key/value(String/String, String/ARInovice) can be retrieved
from thin client. I also serialized the ARInvoiceId object to a file from
client project and was able to to deserialize successfully from the server
project. So still not sure why using ARInvoiceId as key is not working



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Striim support for Ignite

[niamin]

So my use case is to be able use SQL/Query API on my existing system to
retrieve from Ignite in-memory Grid. I don't expect a lot of write-through
from Ignite. I DO expect most of my writes to happen on my RDBMS. I don't
see any option rather than writing custom code on RDBMS(may be triggers) to
capture the changes and then publish them as Events to a topic which Ignite
can subscribe to. Striim provides this set of functionalities as out of the
box solution for Hazelcast behind an RDBMS. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ingite node in SpringBoot App

[niamin]

Thanks! That worked. Now when I try to fetch object using thin client from a
seperate process I get no object. However, when I fetch object using the
same server process I get object.  Difference on fetching object is the
usage of Ignite or IgniteClient. Attached is the client driver. 

IgniteClientStarter.java

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Unresponsive cluster after "Checkpoint read lock acquisition has been timed out" Error

[ihalilaltun]

Hi Ilya,

Sorry for the late response. We don't use lock mechanism in our environment.
We have a lot of put, get operaitons, as far as i remember these operations
does not hold the locks. In addition to these operations, in many update/put
operations we use CacheEntryProcessor which also does not hold the locks.

My guess would be this; sometimes we put big objects to cache. I know using
big objects causes this kind of problems but can you confirm this also?

Regards.



-
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: nodes in the baseline topology is going to OFFLINE state

[Ilya Kasnacheev]

Hello!

There is no supported way to gracefully restart cluster currently. You will
have to stop all nodes, start them again and then activate (or
auto-activate).

Regards,
-- 
Ilya Kasnacheev


пт, 18 окт. 2019 г. в 16:42, Shiva Kumar :

> Hi Ilya,
> My goal is to deactivate the cluster and not restart !! There is an issue
> in deactivating the cluster in my deployment so I am going with restart.
>
> I have the ignite deployment on kubernetes and during deactivation entire
> cluster and even request to deactivate (rest or control.sh) hangs because I
> have few applications which connected to this ignite  cluster over JDBC and
> try to run some queries and also inserts records to many tables parallelly.
> At this time if I issue a deactivate request it hangs for more than 25
> minutes. I am in a impression that since there are many clients established
> TCP connections and running queries, this is causing the cluster to hang
> and thinking of restarting the cluster so that I can proceed with
> deactivation easily once restart is done.
> Any suggestions is appreciated.
>
> Regards,
> Shiva
>
>
> On Fri, 18 Oct, 2019, 6:37 PM Ilya Kasnacheev, 
> wrote:
>
>> Hello!
>>
>> If cluster is persistent, you can deactivate it and then restart.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пт, 18 окт. 2019 г. в 09:51, shivakumar :
>>
>>> Hi Ilya Kasnacheev,
>>> Is there any other way of gracefully shutting down/restart the entire
>>> cluster?
>>>
>>> regards,
>>> shiva
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: Striim support for Ignite

[Ilya Kasnacheev]

Hello!

Ignite can fetch data from RDBMS on the fly and even write it back when it
changes, it is called Cache Store.

https://apacheignite.readme.io/docs/3rd-party-store

Regards,
-- 
Ilya Kasnacheev


пт, 18 окт. 2019 г. в 17:02, niamin :

> There isn't anything that I can find. For synching data i.e. CDC between
> RDBMS and Ignite what would you recommend?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Striim support for Ignite

[Evgeniy Rudenko]

 Ignite has support for Kafka. You can add it to the middle between your
RDBMS and Ignite.

On Fri, Oct 18, 2019 at 5:02 PM niamin  wrote:

> There isn't anything that I can find. For synching data i.e. CDC between
> RDBMS and Ignite what would you recommend?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
Best regards,
Evgeniy

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

so spark + jdbc thin client is the only way to go forward?

 Reading:-

val df = spark.read
  .format("jdbc")
  .option("url", "jdbc:ignite:thin://18.206.247.40")
  .option("fetchsize",100)
  //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
  .option("dbtable", "Person2").load()

df.printSchema()

df.createOrReplaceTempView("test")

spark.sql("select * from test where id=1").show(10)

spark.sql("select 4,'blah',124232").show(10)


Writing:-

import java.sql.DriverManager
val connection = DriverManager.getConnection("jdbc:ignite:thin://18.206.247.40")

import java.util.Properties
val connectionProperties = new Properties()

connectionProperties.put("url", "jdbc:ignite:thin://18.206.247.40")

spark.sql("select 4 as ID,'blah' as STREET,124232 as
ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://18.206.247.40",
  "Person2",connectionProperties)


On Fri, Oct 18, 2019 at 12:00 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> unfortunately, tables that are created by spark don't exist in ignite
> when I try to query using sqlline jdbc thin client, so the spark is still
> running locally the tables which are created by spark exists only for that
> session.
>
> did anyone come across this issue? how to resolve it?
>
>
>
>
>
> On Fri, Oct 18, 2019 at 11:32 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Stephen/All,
>>
>> got it working somewhat using below, but have an issue table and data
>> which is created using thin client is failing to read using spark but table
>> created by spark can be read using a thin client, that means table created
>> in Ignite using spark are the only ones read using spark in Ignite?
>>
>> example-default.xml
>>
>> 
>> 
>> 
>> 18.206.247.40:10800
>> 
>> 
>>
>>
>> NotWorking Scala Function:-
>>
>> def readThinClientTableUsingSpark(implicit spark: SparkSession) = {
>>   val personDataFrame = spark.read
>> .format(FORMAT_IGNITE)
>> .option(OPTION_CONFIG_FILE, CONFIG)
>> .option(OPTION_TABLE, "person")
>> .load()
>>
>>   println()
>>   println("Data frame thin connection content:person")
>>   println()
>>
>>   //Printing content of data frame to console.
>>   personDataFrame.show()
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Exception in thread "main" class org.apache.ignite.IgniteException:
>> Unknown table person at
>> org.apache.ignite.spark.impl.IgniteSQLRelation$$anonfun$schema$2.apply(IgniteSQLRelation.scala:46)
>> at
>> org.apache.ignite.spark.impl.IgniteSQLRelation$$anonfun$schema$2.apply(IgniteSQLRelation.scala:46)
>> at scala.Option.getOrElse(Option.scala:121) at
>> org.apache.ignite.spark.impl.IgniteSQLRelation.schema(IgniteSQLRelation.scala:46)
>> at
>> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:431)
>> at
>> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227) at
>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164) at
>> com.ignite.examples.spark.SparkIgniteCleanCode$.readThinClientTableUsingSpark(SparkIgniteCleanCode.scala:154)
>> at
>> com.ignite.examples.spark.SparkIgniteCleanCode$.main(SparkIgniteCleanCode.scala:216)
>> at
>> com.ignite.examples.spark.SparkIgniteCleanCode.main(SparkIgniteCleanCode.scala)*
>>
>> *Full Code:- (step 7 Fails)*
>>
>> package com.ignite.examples.spark
>>
>> import com.ignite.examples.model.Address
>> import org.apache.ignite.{Ignite, Ignition}
>> import org.apache.ignite.cache.query.SqlFieldsQuery
>> import org.apache.ignite.client.{ClientCache, IgniteClient}
>> import org.apache.ignite.configuration.{CacheConfiguration, 
>> ClientConfiguration}
>> import java.lang.{Long => JLong, String => JString}
>>
>> import com.ignite.examples.spark.SparkClientConnectionTest.{CACHE_NAME, 
>> CONFIG}
>> import org.apache.ignite.internal.util.IgniteUtils.resolveIgnitePath
>> import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE, 
>> OPTION_CONFIG_FILE, OPTION_CREATE_TABLE_PARAMETERS, 
>> OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS, OPTION_TABLE}
>> import org.apache.log4j.{Level, Logger}
>> import org.apache.spark.sql.{SaveMode, SparkSession}
>>
>> object SparkIgniteCleanCode {
>>
>>   private val CACHE_NAME = "SparkCache"
>>
>>   private val CONFIG = 
>> "/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheIgnitePoc/src/main/scala/com/ignite/examples/config/example-ignite.xml"
>>
>>   def setupExampleData = {
>>
>> val cfg2 = new ClientConfiguration().setAddresses("18.206.247.40:10800")
>> val igniteClient:IgniteClient = Ignition.startClient(cfg2)
>>
>> System.out.format(">>> Created cache [%s].\n", CACHE_NAME)
>>
>> val cache:ClientCache[Integer, Address] = 
>> igniteClient.getOrCreateCache(CACHE_NAME)
>>
>> cache.query(new SqlFieldsQuery(String.format("DROP TABLE IF EXISTS 
>> Person"))
>>   

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

Hi Stephen/All,

got it working somewhat using below, but have an issue table and data which
is created using thin client is failing to read using spark but table
created by spark can be read using a thin client, that means table created
in Ignite using spark are the only ones read using spark in Ignite?

example-default.xml




18.206.247.40:10800




NotWorking Scala Function:-

def readThinClientTableUsingSpark(implicit spark: SparkSession) = {
  val personDataFrame = spark.read
.format(FORMAT_IGNITE)
.option(OPTION_CONFIG_FILE, CONFIG)
.option(OPTION_TABLE, "person")
.load()

  println()
  println("Data frame thin connection content:person")
  println()

  //Printing content of data frame to console.
  personDataFrame.show()
}













*Exception in thread "main" class org.apache.ignite.IgniteException:
Unknown table person at
org.apache.ignite.spark.impl.IgniteSQLRelation$$anonfun$schema$2.apply(IgniteSQLRelation.scala:46)
at
org.apache.ignite.spark.impl.IgniteSQLRelation$$anonfun$schema$2.apply(IgniteSQLRelation.scala:46)
at scala.Option.getOrElse(Option.scala:121) at
org.apache.ignite.spark.impl.IgniteSQLRelation.schema(IgniteSQLRelation.scala:46)
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:431)
at
org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227) at
org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164) at
com.ignite.examples.spark.SparkIgniteCleanCode$.readThinClientTableUsingSpark(SparkIgniteCleanCode.scala:154)
at
com.ignite.examples.spark.SparkIgniteCleanCode$.main(SparkIgniteCleanCode.scala:216)
at
com.ignite.examples.spark.SparkIgniteCleanCode.main(SparkIgniteCleanCode.scala)*

*Full Code:- (step 7 Fails)*

package com.ignite.examples.spark

import com.ignite.examples.model.Address
import org.apache.ignite.{Ignite, Ignition}
import org.apache.ignite.cache.query.SqlFieldsQuery
import org.apache.ignite.client.{ClientCache, IgniteClient}
import org.apache.ignite.configuration.{CacheConfiguration, ClientConfiguration}
import java.lang.{Long => JLong, String => JString}

import com.ignite.examples.spark.SparkClientConnectionTest.{CACHE_NAME, CONFIG}
import org.apache.ignite.internal.util.IgniteUtils.resolveIgnitePath
import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE,
OPTION_CONFIG_FILE, OPTION_CREATE_TABLE_PARAMETERS,
OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS, OPTION_TABLE}
import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.{SaveMode, SparkSession}

object SparkIgniteCleanCode {

  private val CACHE_NAME = "SparkCache"

  private val CONFIG =
"/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheIgnitePoc/src/main/scala/com/ignite/examples/config/example-ignite.xml"

  def setupExampleData = {

val cfg2 = new ClientConfiguration().setAddresses("18.206.247.40:10800")
val igniteClient:IgniteClient = Ignition.startClient(cfg2)

System.out.format(">>> Created cache [%s].\n", CACHE_NAME)

val cache:ClientCache[Integer, Address] =
igniteClient.getOrCreateCache(CACHE_NAME)

cache.query(new SqlFieldsQuery(String.format("DROP TABLE IF EXISTS Person"))
  .setSchema("PUBLIC")).getAll

cache.query(new SqlFieldsQuery(String.format("CREATE TABLE IF NOT
EXISTS Person (id LONG,street varchar, zip VARCHAR, PRIMARY KEY (id) )
WITH \"VALUE_TYPE=%s\"", classOf[Address].getName))
  .setSchema("PUBLIC")).getAll

cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(1L.asInstanceOf[JLong],"Jameco",
"04074").setSchema("PUBLIC")).getAll
cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(2L.asInstanceOf[JLong],"Bremar road",
"520003").setSchema("PUBLIC")).getAll
cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(3L.asInstanceOf[JLong],"orange road",
"1234").setSchema("PUBLIC")).getAll

System.out.format(">>> Data Inserted into Cache [%s].\n", CACHE_NAME)

val data=cache.query(new SqlFieldsQuery("select * from
Person").setSchema("PUBLIC")).getAll

println(data.toString)
  }

  def sparkReadIgniteWithThinClient(implicit spark: SparkSession)={

val df = spark.read
  .format("jdbc")
  .option("url", "jdbc:ignite:thin://18.206.247.40")
  .option("fetchsize",100)
  //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
  .option("dbtable", "Person").load()

df.printSchema()

df.createOrReplaceTempView("test")

spark.sql("select * from test where id=1").show(10)

spark.sql("select 4,'blah',124232").show(10)

  }

  def sparkWriteIgniteWithThinClient(implicit spark: SparkSession)={

import java.sql.DriverManager
val connection =
DriverManager.getConnection("jdbc:ignite:thin://18.206.247.40")

import java.

Re: Striim support for Ignite

[niamin]

There isn't anything that I can find. For synching data i.e. CDC between
RDBMS and Ignite what would you recommend?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ingite node in SpringBoot App

[Evgeniy Rudenko]

This happens because you create your node inside try () statement. So it is
being stopped as soon as try section ends. If you want to keep it active
change code to

try  {
Ignite ignite = Ignition.start("example-ignite.xml");


On Fri, Oct 18, 2019 at 4:35 PM niamin  wrote:

> springbeans.zip
> 
>
>
> Here you go. I've tried with @PostConstruct and EventListener but in both
> occasions Ignite engine stops immediately after application bootstrap with
> the following:
>
> [09:32:03] Ignite node stopped OK [uptime=00:00:01.168]
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
Best regards,
Evgeniy

Re: nodes in the baseline topology is going to OFFLINE state

[Shiva Kumar]

Hi Ilya,
My goal is to deactivate the cluster and not restart !! There is an issue
in deactivating the cluster in my deployment so I am going with restart.

I have the ignite deployment on kubernetes and during deactivation entire
cluster and even request to deactivate (rest or control.sh) hangs because I
have few applications which connected to this ignite  cluster over JDBC and
try to run some queries and also inserts records to many tables parallelly.
At this time if I issue a deactivate request it hangs for more than 25
minutes. I am in a impression that since there are many clients established
TCP connections and running queries, this is causing the cluster to hang
and thinking of restarting the cluster so that I can proceed with
deactivation easily once restart is done.
Any suggestions is appreciated.

Regards,
Shiva


On Fri, 18 Oct, 2019, 6:37 PM Ilya Kasnacheev, 
wrote:

> Hello!
>
> If cluster is persistent, you can deactivate it and then restart.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 18 окт. 2019 г. в 09:51, shivakumar :
>
>> Hi Ilya Kasnacheev,
>> Is there any other way of gracefully shutting down/restart the entire
>> cluster?
>>
>> regards,
>> shiva
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Ingite node in SpringBoot App

[niamin]

springbeans.zip
  

Here you go. I've tried with @PostConstruct and EventListener but in both
occasions Ignite engine stops immediately after application bootstrap with
the following:

[09:32:03] Ignite node stopped OK [uptime=00:00:01.168]





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

do you mean communication ports 47100-47200 as mentioned (
https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2)
? bare in mind I am running my spark job outside ignite ec2 box (MAC PC).

which option is right in my default.xml file?

Option 1:-




3.88.248.113:4
7100..47200




Option 2:-




3.88.248.113:4
7500..47600




Option 3:-




3.88.248.113






Thanks

On Fri, Oct 18, 2019 at 9:18 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi Stephen ,
>
>  do you mean 3.88.248.113: 47500..47700
> something like this? or just public ip 3.88.248.113
>  I tried all the possibilities none of them
> are getting connected.
>
> Thanks
> Sri
>
> On Fri, Oct 18, 2019 at 6:02 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> You’re trying to connect a thick client (the Spark integration) to the
>> thin client port (10800). Your example-default.xml file needs to have the
>> same configuration as your server node(s).
>>
>> Regards,
>> Stephen
>>
>> On 17 Oct 2019, at 18:12, sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>> Hi Community,
>>
>> I am trying to read and write into the Ignite cluster using apache-spark
>> I am able to do that using JDBC thin client but not native method as
>> mentioned in several spark + ignite examples.
>>
>> Right now all the spark + ignite examples launch a local ignite cluster
>> but I want my code connecting to already existing cluster (client).
>>
>> Question:-
>> *How to pass Ignite connection ip and port (10800)  10800 in
>> example-default.xml ?*
>>
>> Error:-
>> *TcpDiscoverySpi: Failed to connect to any address from IP finder (will
>> retry to join topology every 2000 ms; change 'reconnectDelay' to configure
>> the frequency of retries): [/3.88.248.113:10800
>> ]*
>>
>> Working (Spark + Ignite using JDBC):-
>>
>> val df = spark.read
>> .format("jdbc")
>> .option("url", "jdbc:ignite:thin://3.88.248.113")
>> .option("fetchsize",100)
>> //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
>> .option("dbtable", "Person").load()
>>
>> df.printSchema()
>>
>> df.createOrReplaceTempView("test")
>>
>> spark.sql("select * from test where id=1").show(10)
>>
>> spark.sql("select 4,'blah',124232").show(10)
>>
>> import java.sql.DriverManager
>> val connection = 
>> DriverManager.getConnection("jdbc:ignite:thin://3.88.248.113")
>>
>> import java.util.Properties
>> val connectionProperties = new Properties()
>>
>> connectionProperties.put("url", "jdbc:ignite:thin://3.88.248.113")
>>
>> spark.sql("select 4 as ID,'blah' as STREET,124232 as 
>> ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://3.88.248.113",
>>   "Person",connectionProperties)
>>
>> spark.read
>>   .format("jdbc")
>>   .option("url", "jdbc:ignite:thin://3.88.248.113")
>>   .option("fetchsize",100)
>>   .option("dbtable", "Person").load().show(10,false)
>>
>>
>> Not Working requires a CONFIG file which is example-default.xml:-
>>
>> val igniteDF = spark.read
>>   .format(FORMAT_IGNITE) //Data source type.
>>   .option(OPTION_TABLE, "person") //Table to read.
>>   .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
>>   .load()
>>   .filter(col("id") >= 2) //Filter clause.
>>   .filter(col("name") like "%J%") //Another filter clause.
>>
>>
>> Full Code:- (sparkDSLExample) function fails to connect ignite cluster
>> which I already have
>>
>> package com.ignite.examples.spark
>>
>> import com.ignite.examples.model.Address
>> import org.apache.ignite.{Ignite, Ignition}
>> import org.apache.ignite.cache.query.SqlFieldsQuery
>> import org.apache.ignite.client.{ClientCache, IgniteClient}
>> import org.apache.ignite.configuration.{CacheConfiguration, 
>> ClientConfiguration}
>> import java.lang.{Long => JLong, String => JString}
>>
>> import org.apache.ignite.cache.query.SqlFieldsQuery
>> import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE, 
>> OPTION_CONFIG_FILE, OPTION_TABLE}
>> import org.apache.log4j.{Level, Logger}
>> import org.apache.spark.sql.{SaveMode, SparkSession}
>> import org.apache.spark.sql.functions.col
>>
>> object SparkClientConnectionTest {
>>
>>   private val CACHE_NAME = "SparkCache"
>>
>>   private val CONFIG = 
>> "/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheI

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

Hi Stephen ,

 do you mean 3.88.248.113: 47500..47700
something like this? or just public ip 3.88.248.113
 I tried all the possibilities none of them are
getting connected.

Thanks
Sri

On Fri, Oct 18, 2019 at 6:02 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> You’re trying to connect a thick client (the Spark integration) to the
> thin client port (10800). Your example-default.xml file needs to have the
> same configuration as your server node(s).
>
> Regards,
> Stephen
>
> On 17 Oct 2019, at 18:12, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> Hi Community,
>
> I am trying to read and write into the Ignite cluster using apache-spark I
> am able to do that using JDBC thin client but not native method as
> mentioned in several spark + ignite examples.
>
> Right now all the spark + ignite examples launch a local ignite cluster
> but I want my code connecting to already existing cluster (client).
>
> Question:-
> *How to pass Ignite connection ip and port (10800)  10800 in
> example-default.xml ?*
>
> Error:-
> *TcpDiscoverySpi: Failed to connect to any address from IP finder (will
> retry to join topology every 2000 ms; change 'reconnectDelay' to configure
> the frequency of retries): [/3.88.248.113:10800
> ]*
>
> Working (Spark + Ignite using JDBC):-
>
> val df = spark.read
> .format("jdbc")
> .option("url", "jdbc:ignite:thin://3.88.248.113")
> .option("fetchsize",100)
> //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
> .option("dbtable", "Person").load()
>
> df.printSchema()
>
> df.createOrReplaceTempView("test")
>
> spark.sql("select * from test where id=1").show(10)
>
> spark.sql("select 4,'blah',124232").show(10)
>
> import java.sql.DriverManager
> val connection = 
> DriverManager.getConnection("jdbc:ignite:thin://3.88.248.113")
>
> import java.util.Properties
> val connectionProperties = new Properties()
>
> connectionProperties.put("url", "jdbc:ignite:thin://3.88.248.113")
>
> spark.sql("select 4 as ID,'blah' as STREET,124232 as 
> ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://3.88.248.113",
>   "Person",connectionProperties)
>
> spark.read
>   .format("jdbc")
>   .option("url", "jdbc:ignite:thin://3.88.248.113")
>   .option("fetchsize",100)
>   .option("dbtable", "Person").load().show(10,false)
>
>
> Not Working requires a CONFIG file which is example-default.xml:-
>
> val igniteDF = spark.read
>   .format(FORMAT_IGNITE) //Data source type.
>   .option(OPTION_TABLE, "person") //Table to read.
>   .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
>   .load()
>   .filter(col("id") >= 2) //Filter clause.
>   .filter(col("name") like "%J%") //Another filter clause.
>
>
> Full Code:- (sparkDSLExample) function fails to connect ignite cluster
> which I already have
>
> package com.ignite.examples.spark
>
> import com.ignite.examples.model.Address
> import org.apache.ignite.{Ignite, Ignition}
> import org.apache.ignite.cache.query.SqlFieldsQuery
> import org.apache.ignite.client.{ClientCache, IgniteClient}
> import org.apache.ignite.configuration.{CacheConfiguration, 
> ClientConfiguration}
> import java.lang.{Long => JLong, String => JString}
>
> import org.apache.ignite.cache.query.SqlFieldsQuery
> import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE, 
> OPTION_CONFIG_FILE, OPTION_TABLE}
> import org.apache.log4j.{Level, Logger}
> import org.apache.spark.sql.{SaveMode, SparkSession}
> import org.apache.spark.sql.functions.col
>
> object SparkClientConnectionTest {
>
>   private val CACHE_NAME = "SparkCache"
>
>   private val CONFIG = 
> "/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheIgnitePoc/src/main/scala/com/ignite/examples/config/example-ignite.xml"
>
>   def setupExampleData = {
>
> val cfg2 = new ClientConfiguration().setAddresses("3.88.248.113:10800")
> val igniteClient:IgniteClient = Ignition.startClient(cfg2)
>
> System.out.format(">>> Created cache [%s].\n", CACHE_NAME)
>
> val cache:ClientCache[Integer, Address] = 
> igniteClient.getOrCreateCache(CACHE_NAME)
>
> cache.query(new SqlFieldsQuery(String.format("DROP TABLE IF EXISTS 
> Person"))
>   .setSchema("PUBLIC")).getAll
>
> cache.query(new SqlFieldsQuery(String.format("CREATE TABLE IF NOT EXISTS 
> Person (id LONG,street varchar, zip VARCHAR, PRIMARY KEY (id) ) WITH 
> \"VALUE_TYPE=%s\"", classOf[Address].getName))
>   .setSchema("PUBLIC")).getAll
>
> cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip) 
> VALUES(?,?, ?)").setArgs(1L.asInstanceOf[JLong],"Jameco", 
> "04074").setSchema("PUBLIC")).getAll
> cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip) 
> VALUES(?,?, ?)").setArgs(2L.asInstanceOf[JLong],"Bremar road", 
> "520003").setSchema("PUBLIC")).getAll
> cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)

Re: nodes in the baseline topology is going to OFFLINE state

[Ilya Kasnacheev]

Hello!

If cluster is persistent, you can deactivate it and then restart.

Regards,
-- 
Ilya Kasnacheev


пт, 18 окт. 2019 г. в 09:51, shivakumar :

> Hi Ilya Kasnacheev,
> Is there any other way of gracefully shutting down/restart the entire
> cluster?
>
> regards,
> shiva
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Starvation in striped pool

[Ilya Kasnacheev]

Hello!

We, we had IGNITE_ENABLE_FORCIBLE_NODE_KILL, but the best solution is, in
my opinion, to avoid adding anything unstable to the cluster.

Regards,
-- 
Ilya Kasnacheev


пт, 18 окт. 2019 г. в 08:35, ihalilaltun :

> Hi Ilya,
>
> From time to time, we have faced exactly the same problem. Is there any
> best
> practices for handling network issues? What i mean is, if there is any
> network issues between client/s and server/s we want the cluster keeps
> living. As for the clients, they can be disconnected from servers.
>
> Regards.
>
>
>
> -
> İbrahim Halil Altun
> Senior Software Engineer @ Segmentify
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Node stopped.

[Denis Mekhanikov]

The following documentation page has some useful points on deployment in a 
virtualised environment: https://apacheignite.readme.io/docs/vmware-deployment

Denis
On 17 Oct 2019, 17:41 +0300, John Smith , wrote:
> Ok I have metribeat running on the VM hopefully I will see something...
>
> > On Thu, 17 Oct 2019 at 05:09, Denis Mekhanikov  
> > wrote:
> > > There are no long pauses in the GC logs, so it must be the whole VM pause.
> > >
> > > Denis
> > > On 16 Oct 2019, 23:07 +0300, John Smith , wrote:
> > > > Sorry here is the gc logs for all 3 machines: 
> > > > https://www.dropbox.com/s/chbbxigahd4v9di/gc-logs.zip?dl=0
> > > >
> > > > > On Wed, 16 Oct 2019 at 15:49, John Smith  
> > > > > wrote:
> > > > > > Hi, so it happened again here is my latest gc.log stats: 
> > > > > > https://gceasy.io/diamondgc-report.jsp?oTxnId_value=a215d573-d1cf-4d53-acf1-9001432bb28e
> > > > > >
> > > > > > Everything seems ok to me. I also have Elasticsearch Metricbeat 
> > > > > > running, the CPU usage looked normal at the time.
> > > > > >
> > > > > > > On Thu, 10 Oct 2019 at 13:05, Denis Mekhanikov 
> > > > > > >  wrote:
> > > > > > > > Unfortunately, I don’t.
> > > > > > > > You can ask the VM vendor or the cloud provider (if you use 
> > > > > > > > any) for a proper tooling or logs.
> > > > > > > > Make sure, that there is no such step in the VM’s lifecycle 
> > > > > > > > that makes it freeze for a minute.
> > > > > > > > Also make sure that the physical CPU is not overutilized and no 
> > > > > > > > VMs that run on it are starving.
> > > > > > > >
> > > > > > > > Denis
> > > > > > > > On 10 Oct 2019, 19:03 +0300, John Smith 
> > > > > > > > , wrote:
> > > > > > > > > Do you know of any good tools I can use to check the VM?
> > > > > > > > >
> > > > > > > > > > On Thu, 10 Oct 2019 at 11:38, Denis Mekhanikov 
> > > > > > > > > >  wrote:
> > > > > > > > > > > > Hi Dennis, so are you saying I should enable GC logs + 
> > > > > > > > > > > > the safe point logs as well?
> > > > > > > > > > >
> > > > > > > > > > > Having safepoint statistics in your GC logs may be 
> > > > > > > > > > > useful, so I recommend enabling them for troubleshooting 
> > > > > > > > > > > purposes.
> > > > > > > > > > > Check the lifecycle of your virtual machines. There is a 
> > > > > > > > > > > high chance that the whole machine is frozen, not just 
> > > > > > > > > > > the Ignite node.
> > > > > > > > > > >
> > > > > > > > > > > Denis
> > > > > > > > > > > On 10 Oct 2019, 18:25 +0300, John Smith 
> > > > > > > > > > > , wrote:
> > > > > > > > > > > > Hi Dennis, so are you saying I should enable GC logs + 
> > > > > > > > > > > > the safe point logs as well?
> > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, 10 Oct 2019 at 11:22, John Smith 
> > > > > > > > > > > > >  wrote:
> > > > > > > > > > > > > > You are correct, it is running in a VM.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, 10 Oct 2019 at 10:11, Denis Mekhanikov 
> > > > > > > > > > > > > > >  wrote:
> > > > > > > > > > > > > > > > Hi!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are the following messages in the logs:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [22:26:21,816][WARNING][jvm-pause-detector-worker][IgniteKernal%xx]
> > > > > > > > > > > > > > > >  Possible too long JVM pause: 55705 
> > > > > > > > > > > > > > > > milliseconds.
> > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > [22:26:21,847][SEVERE][ttl-cleanup-worker-#48%xx%][G]
> > > > > > > > > > > > > > > >  Blocked system-critical thread has been 
> > > > > > > > > > > > > > > > detected. This can lead to cluster-wide 
> > > > > > > > > > > > > > > > undefined behaviour 
> > > > > > > > > > > > > > > > [threadName=partition-exchanger, blockedFor=57s]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Looks like the JVM was paused for almost a 
> > > > > > > > > > > > > > > > minute. It doesn’t seem to be caused by a 
> > > > > > > > > > > > > > > > garbage collection, since there is no evidence 
> > > > > > > > > > > > > > > > of GC pressure in the GC log. Usually such big 
> > > > > > > > > > > > > > > > pauses happen in virtualised environments when 
> > > > > > > > > > > > > > > > backups are captured from machines or they just 
> > > > > > > > > > > > > > > > don’t have enough CPU time.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Looking at safepoint statistics may also reveal 
> > > > > > > > > > > > > > > > some interesting details. You can learn about 
> > > > > > > > > > > > > > > > safepoints here: 
> > > > > > > > > > > > > > > > https://blog.gceasy.io/2016/12/22/total-time-for-which-application-threads-were-stopped/
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Denis
> > > > > > > > > > > > > > > > On 9 Oct 2019, 23:14 +0300, John Smith 
> > > > > > > > > > > > > > > > , wrote:
> > > > > > > > > > > > > > > > > So the error sais to set 
> > > > > > > 

Re: Apache Spark + Ignite Connection Issue

[Stephen Darlington]

You’re trying to connect a thick client (the Spark integration) to the thin 
client port (10800). Your example-default.xml file needs to have the same 
configuration as your server node(s).

Regards,
Stephen

> On 17 Oct 2019, at 18:12, sri hari kali charan Tummala 
>  wrote:
> 
> Hi Community, 
> 
> I am trying to read and write into the Ignite cluster using apache-spark I am 
> able to do that using JDBC thin client but not native method as mentioned in 
> several spark + ignite examples.
> 
> Right now all the spark + ignite examples launch a local ignite cluster but I 
> want my code connecting to already existing cluster (client).
> 
> Question:-
> How to pass Ignite connection ip and port (10800)  10800 in 
> example-default.xml ?
> 
> Error:-
> TcpDiscoverySpi: Failed to connect to any address from IP finder (will retry 
> to join topology every 2000 ms; change 'reconnectDelay' to configure the 
> frequency of retries): [/3.88.248.113:10800 ]
> 
> Working (Spark + Ignite using JDBC):-
> val df = spark.read
> .format("jdbc")
> .option("url", "jdbc:ignite:thin://3.88.248.113 ")
> .option("fetchsize",100)
> //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
> .option("dbtable", "Person").load()
> 
> df.printSchema()
> 
> df.createOrReplaceTempView("test")
> 
> spark.sql("select * from test where id=1").show(10)
> 
> spark.sql("select 4,'blah',124232").show(10)
> 
> import java.sql.DriverManager
> val connection = DriverManager.getConnection("jdbc:ignite:thin://3.88.248.113 
> ")
> 
> import java.util.Properties
> val connectionProperties = new Properties()
> 
> connectionProperties.put("url", "jdbc:ignite:thin://3.88.248.113 
> ")
> 
> spark.sql("select 4 as ID,'blah' as STREET,124232 as 
> ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://3.88.248.113 
> ",
>   "Person",connectionProperties)
> 
> spark.read
>   .format("jdbc")
>   .option("url", "jdbc:ignite:thin://3.88.248.113 ")
>   .option("fetchsize",100)
>   .option("dbtable", "Person").load().show(10,false)
> 
> Not Working requires a CONFIG file which is example-default.xml:-
> val igniteDF = spark.read
>   .format(FORMAT_IGNITE) //Data source type.
>   .option(OPTION_TABLE, "person") //Table to read.
>   .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
>   .load()
>   .filter(col("id") >= 2) //Filter clause.
>   .filter(col("name") like "%J%") //Another filter clause.
> 
> Full Code:- (sparkDSLExample) function fails to connect ignite cluster which 
> I already have
> package com.ignite.examples.spark
> 
> import com.ignite.examples.model.Address
> import org.apache.ignite.{Ignite, Ignition}
> import org.apache.ignite.cache.query.SqlFieldsQuery
> import org.apache.ignite.client.{ClientCache, IgniteClient}
> import org.apache.ignite.configuration.{CacheConfiguration, 
> ClientConfiguration}
> import java.lang.{Long => JLong, String => JString}
> 
> import org.apache.ignite.cache.query.SqlFieldsQuery
> import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE, 
> OPTION_CONFIG_FILE, OPTION_TABLE}
> import org.apache.log4j.{Level, Logger}
> import org.apache.spark.sql.{SaveMode, SparkSession}
> import org.apache.spark.sql.functions.col
> 
> object SparkClientConnectionTest {
> 
>   private val CACHE_NAME = "SparkCache"
> 
>   private val CONFIG = 
> "/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheIgnitePoc/src/main/scala/com/ignite/examples/config/example-ignite.xml"
> 
>   def setupExampleData = {
> 
> val cfg2 = new ClientConfiguration().setAddresses("3.88.248.113:10800 
> ")
> val igniteClient:IgniteClient = Ignition.startClient(cfg2)
> 
> System.out.format(">>> Created cache [%s].\n", CACHE_NAME)
> 
> val cache:ClientCache[Integer, Address] = 
> igniteClient.getOrCreateCache(CACHE_NAME)
> 
> cache.query(new SqlFieldsQuery(String.format("DROP TABLE IF EXISTS 
> Person"))
>   .setSchema("PUBLIC")).getAll
> 
> cache.query(new SqlFieldsQuery(String.format("CREATE TABLE IF NOT EXISTS 
> Person (id LONG,street varchar, zip VARCHAR, PRIMARY KEY (id) ) WITH 
> \"VALUE_TYPE=%s\"", classOf[Address].getName))
>   .setSchema("PUBLIC")).getAll
> 
> cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip) 
> VALUES(?,?, ?)").setArgs(1L.asInstanceOf[JLong],"Jameco", 
> "04074").setSchema("PUBLIC")).getAll
> cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip) 
> VALUES(?,?, ?)").setArgs(2L.asInstanceOf[JLong],"Bremar road", 
> "520003").setSchema("PUBLIC")).getAll
> cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip) 
> VALUES(?,?, ?)").setArgs(3L.asInstanceOf[JLong],"orange road", 
> "1234").setSchema("PUBLIC")).getAll
> 
> System.out.format(">>> Data Inserted into Cache [%s].\n", CACHE_NAME)
> 
> va

Re: Starvation in striped pool

[Mahesh Renduchintala]

current setting idleTImeout = LONG_MAX in tcp communication spi has solved the 
problem for us.

Re: Ingite node in SpringBoot App

[Evgeniy Rudenko]

Can you share code of your Bean to check?

On Thu, Oct 17, 2019 at 10:11 PM niamin  wrote:

> I've created a SpringBoot App that includes code to start an Ignite
> standalone node and populates a cache. The logic is wrapped in a
> PostConstruct method on a Bean. However Ignite server stops as soon as app
> returns from the method to start Ignite server and populate cache. I would
> like to be able to connect to the server from a thin client embedded on a
> different app.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
Best regards,
Evgeniy

Re: Getting data[parquet,json,...] from S3 buckets to apache ignite

[viktor]

dmagda wrote
> Hi Viktor,
> 
> Could you please clarify a bit, do you need just to load data
> once/periodically or do you want Ignite to write-back to S3 on updates? If
> the loading is all you need then create a custom Java app/class that pulls
> data from S3 and streams into Ignite IgniteDataStreamer (fastest loading
> technique) [1].
> 
> If the write-back is needed then the 3rd party store (CacheStore) is the
> best way to go. GridGain Web Console (free tool) [2] goes with a model
> importing feature [3] that should be able to read the schema of Drill via
> JDBC and produce an Ignite configuration.
> 
> [1] https://apacheignite.readme.io/docs/data-loading#ignitedatastreamer
> [2]
> https://www.gridgain.com/docs/web-console/latest/web-console-getting-started
> [3] https://apacheignite-tools.readme.io/docs/automatic-rdbms-integration
> 
> -
> Denis
> 
> 
> On Thu, Oct 17, 2019 at 6:45 AM viktor <

> viktor.baert@

> > wrote:
> 
>> Hi,
>>
>> I'm currently working on r&d project where we would like to retrieve data
>> files[parquet, json, ...] from S3 buckets and load the data into apache
>> ignite for machine learning purposes with tensorflow.
>>
>> With the removal of IGFS in the next release I'm having troubles finding
>> a
>> solution.
>> What would be an optimal way to facilitate the data for apache ignite?
>>
>> I'm currently looking into using the 3rd party store features of ignite
>> to
>> integrate with apache drill as it is able to query these s3 bucket data
>> files.
>> However from a glance it doesn't look like as a great solution since
>> every
>> table structure has to be manually defined in the ignite configuration or
>> semi-automatic with the agent.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>

Thanks for the reply our use case goes something like this, data for new ML
projects are 
placed in S3 buckets, BI & ML engineers are able to make use of ignite to
query this data with optimal performance. 
Ideally there shouldn't be much hassle to get ignite in sync with drill
whenever there are structure/data alterations.

I'll play around with the cache store today see how far that gets me, thanks
for the advice.
More suggestions are always welcome.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/