India Scala & Big Data Job Referral

[sri hari kali charan Tummala]

Hi Community,

I was laid off from Apple in February 2023, which led to my relocation from
the USA due to immigration issues related to my H1B visa.


I have over 12 years of experience as a consultant in Big Data, Spark,
Scala, Python, and Flink.


Despite my move to India, I haven't secured a job yet. I am seeking
referrals within product firms (preferably non-consulting) in India that
work with Flink, Spark, Scala, Big Data, or in the fields of ML & AI. Can
someone assist me with this?

Thanks
Sri

Big Data Contract roles ?

[sri hari kali charan Tummala]

Hi Community,

Is anyone hiring contract big Data spark scala corp to corp roles in USA ?
I am open in market looking for work.


Thanks
Sri

Stand-alone Mode Ignite Installation

[sri hari kali charan Tummala]

Hi Ignite Users,

Below we link is not working spark ignite standalone mode installation page.

https://ignite.apache.org/docs/latest/installation/

https://ignite.apache.org/docs/latest/extensions-and-integrations/ignite-for-spark/installation


Thanks
Sri
-- 
Thanks & Regards
Sri Tummala

Re: Looking for feedback on the Ignite 3.0.0 Alpha

[sri hari kali charan Tummala]

Ok thanks , but I am looking forward for the Aws steps.

On Wed, 13 Jan 2021 at 11:39, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Sri,
>
> This will be available in future releases. Current alpha focuses on
> demonstrating how Ignite 3.0 will be delivered, and how you will run
> operations using the new CLI tool.
>
> -Val
>
> On Wed, Jan 13, 2021 at 11:21 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Sure few steps are missing.
>>
>> Creating a ignite cluster is missing (example 3 ec2 instances)
>> Creating a ignite cluster on Aws would be nice step to add
>>
>>
>> Thanks
>> Sri
>>
>> On Wed, 13 Jan 2021 at 04:14, Kseniya Romanova 
>> wrote:
>>
>>> Here's the link for the online gathering:
>>> https://www.meetup.com/Apache-Ignite-Virtual-Meetup/events/275722317/
>>>
>>>
>>> ср, 13 янв. 2021 г. в 13:47, Pavel Tupitsyn :
>>>
>>>> Getting Started Guide:
>>>>
>>>> https://ignite.apache.org/docs/3.0.0-alpha/quick-start/getting-started-guide
>>>>
>>>> On Wed, Jan 13, 2021 at 1:29 PM Stephen Darlington <
>>>> stephen.darling...@gridgain.com> wrote:
>>>>
>>>>> What is the link to the Getting Started Guide?
>>>>>
>>>>> On 13 Jan 2021, at 03:55, Valentin Kulichenko <
>>>>> valentin.kuliche...@gmail.com> wrote:
>>>>>
>>>>> Igniters,
>>>>>
>>>>> I'm excited to announce that the first alpha build of the Ignite 3 is
>>>>> out and available for download!
>>>>>
>>>>> Ignite 3 is the new project that was initiated by the Ignite community
>>>>> last year. Please refer to this page if you want to learn more:
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0
>>>>>
>>>>> The just-released alpha build is a sneak peek into the future of
>>>>> Ignite. It doesn't represent a fully-functional product (no discovery,
>>>>> caches, compute, etc.), but demonstrates major mechanics of how you will
>>>>> interact with Ignite going forward.
>>>>>
>>>>> The main goal of the release is to gather feedback from the community
>>>>> so that we can adjust further development if needed. That said, I would 
>>>>> ask
>>>>> and encourage everyone to go through the Getting Started Guide [1] and 
>>>>> play
>>>>> with the build. If you have any questions, issues, concerns, wishes,
>>>>> requests, or any other thoughts, please reply directly to this thread. We
>>>>> will carefully accumulate all the feedback and make sure it is considered
>>>>> going forward.
>>>>>
>>>>> Another opportunity to share your feedback will come closer to the end
>>>>> of January when we will have a virtual meetup. I will present a quick demo
>>>>> of the alpha build, after which we will have an open discussion. Please
>>>>> stay tuned - I will send a message here when the meetup is scheduled.
>>>>>
>>>>> -Val
>>>>>
>>>>>
>>>>>
>>>>> --
>> Thanks & Regards
>> Sri Tummala
>>
>> --
Thanks & Regards
Sri Tummala

Re: Looking for feedback on the Ignite 3.0.0 Alpha

[sri hari kali charan Tummala]

Sure few steps are missing.

Creating a ignite cluster is missing (example 3 ec2 instances)
Creating a ignite cluster on Aws would be nice step to add


Thanks
Sri

On Wed, 13 Jan 2021 at 04:14, Kseniya Romanova 
wrote:

> Here's the link for the online gathering:
> https://www.meetup.com/Apache-Ignite-Virtual-Meetup/events/275722317/
>
>
> ср, 13 янв. 2021 г. в 13:47, Pavel Tupitsyn :
>
>> Getting Started Guide:
>>
>> https://ignite.apache.org/docs/3.0.0-alpha/quick-start/getting-started-guide
>>
>> On Wed, Jan 13, 2021 at 1:29 PM Stephen Darlington <
>> stephen.darling...@gridgain.com> wrote:
>>
>>> What is the link to the Getting Started Guide?
>>>
>>> On 13 Jan 2021, at 03:55, Valentin Kulichenko <
>>> valentin.kuliche...@gmail.com> wrote:
>>>
>>> Igniters,
>>>
>>> I'm excited to announce that the first alpha build of the Ignite 3 is
>>> out and available for download!
>>>
>>> Ignite 3 is the new project that was initiated by the Ignite community
>>> last year. Please refer to this page if you want to learn more:
>>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0
>>>
>>> The just-released alpha build is a sneak peek into the future of Ignite.
>>> It doesn't represent a fully-functional product (no discovery, caches,
>>> compute, etc.), but demonstrates major mechanics of how you will interact
>>> with Ignite going forward.
>>>
>>> The main goal of the release is to gather feedback from the community so
>>> that we can adjust further development if needed. That said, I would ask
>>> and encourage everyone to go through the Getting Started Guide [1] and play
>>> with the build. If you have any questions, issues, concerns, wishes,
>>> requests, or any other thoughts, please reply directly to this thread. We
>>> will carefully accumulate all the feedback and make sure it is considered
>>> going forward.
>>>
>>> Another opportunity to share your feedback will come closer to the end
>>> of January when we will have a virtual meetup. I will present a quick demo
>>> of the alpha build, after which we will have an open discussion. Please
>>> stay tuned - I will send a message here when the meetup is scheduled.
>>>
>>> -Val
>>>
>>>
>>>
>>> --
Thanks & Regards
Sri Tummala

DataBricks Spark/Scala Opportunity

[sri hari kali charan Tummala]

>
> Hi Apache Ignite Community,
>>
>> Note:- (This is a full time position)
>>
>> I copied Geoff in this email chain who is actively looking for
>> Spark/Scala Developer for a full time position with DataBricks 200k+
>> (Salary/Bonus/Stocks/H1b/GC Process).
>>
>> If you know anyone or your friends who is good in Spark/Scala either
>> American Citizen/GC/H1B ask him to contact Geoff (
>> geoffzun...@databricks.com).
>>
>> 415-328-4879
>>
>>
>> Thanks
>> Sri
>>
>
> --
Thanks & Regards
Sri Tummala

Re: New blog post on Apache Ignite in AWS

[sri hari kali charan Tummala]

Please try to fix the screenshot issue what I mentioned I think it’s a easy
fix.

On Tuesday, January 28, 2020, Sergey Chugunov 
wrote:

> Thank you for the feedback!
>
> Than I'll focus my next post on connecting to AWS cluster with other
> methods like JDBC, thin clients etc.
>
> On Sat, Jan 25, 2020 at 7:22 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> cant open second screenshot I mean cant expand , a simple example to read
>> and write with spark would be nice using thrift jdbc etc ...
>>
>> connecting Ignite from outside ec2 with spark.
>>
>> https://github.com/kali786516/ApacheIgnitePoc/blob/
>> ef95cdd65ef2a91aeef59dd2b4ebafec4da19a17/src/main/scala/com/
>> ignite/examples/spark/SparkClientConnectionTest.scala#L75
>>
>>
>>
>>
>> On Fri, Jan 24, 2020 at 5:20 PM Denis Magda  wrote:
>>
>>> Sergey,
>>>
>>> Thanks for a compact and clear article. I especially enjoyed the part
>>> that explains how to connect from a local laptop to an already running
>>> Ignite cluster on AWS. Looking forward to more content from you!
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Wed, Jan 22, 2020 at 11:35 PM Sergey Chugunov <
>>> sergey.chugu...@gmail.com> wrote:
>>>
>>>> Hello community,
>>>>
>>>> Recently I published a new blog post on getting started with Apache
>>>> Ignite in AWS [1]. I tried to make my example as simple as possible while
>>>> keeping it usable.
>>>>
>>>> Let me know if this post is useful for you.
>>>>
>>>> I plan to write several follow-up posts about AWS-specific things but
>>>> based on feedback may cover other topics in more detail.
>>>>
>>>> Any feedback is welcome, thank you!
>>>>
>>>> [1] https://www.gridgain.com/node/6247
>>>>
>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>

-- 
Thanks & Regards
Sri Tummala

Re: New blog post on Apache Ignite in AWS

[sri hari kali charan Tummala]

cant open second screenshot I mean cant expand , a simple example to read
and write with spark would be nice using thrift jdbc etc ...

connecting Ignite from outside ec2 with spark.

https://github.com/kali786516/ApacheIgnitePoc/blob/ef95cdd65ef2a91aeef59dd2b4ebafec4da19a17/src/main/scala/com/ignite/examples/spark/SparkClientConnectionTest.scala#L75




On Fri, Jan 24, 2020 at 5:20 PM Denis Magda  wrote:

> Sergey,
>
> Thanks for a compact and clear article. I especially enjoyed the part that
> explains how to connect from a local laptop to an already running Ignite
> cluster on AWS. Looking forward to more content from you!
>
> -
> Denis
>
>
> On Wed, Jan 22, 2020 at 11:35 PM Sergey Chugunov <
> sergey.chugu...@gmail.com> wrote:
>
>> Hello community,
>>
>> Recently I published a new blog post on getting started with Apache
>> Ignite in AWS [1]. I tried to make my example as simple as possible while
>> keeping it usable.
>>
>> Let me know if this post is useful for you.
>>
>> I plan to write several follow-up posts about AWS-specific things but
>> based on feedback may cover other topics in more detail.
>>
>> Any feedback is welcome, thank you!
>>
>> [1] https://www.gridgain.com/node/6247
>>
>

-- 
Thanks & Regards
Sri Tummala

Re: Adhoc Reporting use case (Ignite+Spark?)

[sri hari kali charan Tummala]

Check these examples.

https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/

Spark + ignite using thrift connection:-

https://github.com/kali786516/ApacheIgnitePoc/blob/ef95cdd65ef2a91aeef59dd2b4ebafec4da19a17/src/main/scala/com/ignite/examples/spark/SparkClientConnectionTest.scala#L75


https://github.com/kali786516/ApacheIgnitePoc/blob/master/src/main/scala/com/ignite/examples/spark/SparkClientConnectionTest.scala#L75

Thanks
Sri

https://github.com/kali786516/ApacheIgnitePoc/blob/ef95cdd65ef2a91aeef59dd2b4ebafec4da19a17/src/main/scala/com/ignite/examples/spark/SparkClientConnectionTest.scala#L7


On Tuesday, December 10, 2019, Ilya Kasnacheev 
wrote:

> Hello!
>
> I think you will have to provide way more details about your case before
> getting any feedback.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 5 дек. 2019 г. в 00:07, Deepak :
>
>> Hello,
>>
>> I am trying out Ignite+Spark combination for my adhoc reporting use case.
>> Any suggestion/help on similar architecture would be helpful. Does it make
>> sense or I am going totally south ??
>>
>> thanks in advance
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

-- 
Thanks & Regards
Sri Tummala

Re: Ignite with Spark Intergration

[sri hari kali charan Tummala]

I integrated spark and ignite using thrift client will that do ?

https://github.com/kali786516/ApacheIgnitePoc/blob/master/src/main/scala/com/ignite/examples/spark/SparkClientConnectionTest.scala#L73



On Monday, December 9, 2019, Denis Magda  wrote:

> Hi, just ensure that "clientMode" is set in IgniteConfiguration that you
> pass to Spark's IgniteContext object. Spark worker will spin up a client
> node automatically for you and that one will reach out to the server
> (assuming you properly configured Ignite discovery SPI in the same
> IgniteConfiguration).
>
> -
> Denis
>
>
> On Sat, Dec 7, 2019 at 10:46 AM datta  wrote:
>
>> Hi,
>>
>> Then what I have currently implemented is hopefully not embedded mode
>> is it?
>>
>> Also I wanted to know if should install client nodes on spark worker nodes
>> if spark is going to start a client node itself ?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

-- 
Thanks & Regards
Sri Tummala

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

I got it so I have to build a jar move to aws ignite cluster node and run
it on the node.

Thanks
Sri

On Thursday, October 24, 2019, Denis Magda  wrote:

> Sri,
>
> That's wrong to assume that the Ignite thin client is the only way to
> access data stored in AWS or other cloud environments. The problems you are
> experiencing are caused by the fact that your application is running on the
> laptop while the cluster (server nodes) is within an AWS network.
>
> If you use a thick/regular client, then servers running on AWS need to
> have a way to *open* network connections to the application. This won't
> happen if the laptop or dev environment doesn't have public IPs reachable
> from AWS. Plus, additional network configuration might be needed.
>
> Overall, thick clients work with AWS, but either an application needs to
> be deployed to AWS during development, or you need to start a local cluster
> in your network/premises first, accomplish development and move on with
> testing in AWS. A network configuration is also an option but a tricky one
> and might not be doable.
>
> Thin clients are a solution as well if key-value and SQL APIs are all you
> need.
>
> -
> Denis
>
>
> On Sun, Oct 20, 2019 at 2:06 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> so I got an answer from Grid Grain computing, JDBC thin client is the only
> way yo connect the ignite cluster which is running on aws.
>
> https://forums.gridgain.com/community-home/digestviewer/
> viewthread?MessageKey=13f5e836-1569-486a-8475-84c70fc141e0=
> 3b551477-7e2d-462f-bc5f-d6d10ccbbe35=digestviewer&
> SuccessMsg=Thank+you+for+submitting+your+message.
>
> Thanks
> Sri
>
> On Fri, Oct 18, 2019 at 1:38 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> Ok I created a 2 node ec2 instance ignite cluster is below is the right
> way to create cache? the code still using my laptop resources unable to
> connect ignite cluster.
>
> Out put:-
> [13:36:36] Ignite node started OK (id=8535f4d3)
> [13:36:36] Topology snapshot [ver=30, locNode=8535f4d3, *servers=1,*
> clients=1, state=ACTIVE, CPUs=8, *offheap=6.4GB, heap=14.0GB*]
> >>> cache acquired
>
> package com.ignite.examples.igniteStartup
>
> import java.util.Arrays
> import java.util.List
> import com.ignite.examples.model.Address
> import org.apache.commons.logging.Log
> import org.apache.commons.logging.LogFactory
> import org.apache.ignite.Ignite
> import org.apache.ignite.IgniteCache
> import org.apache.ignite.Ignition
> import org.apache.ignite.configuration.IgniteConfiguration
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
>
> //remove if not needed
> import scala.collection.JavaConversions._
>
> object IgniteStart2 {
>
>   private var cache: IgniteCache[String, Address] = _
>
>   def main(args: Array[String]): Unit = {
>
> val spi: TcpDiscoverySpi = new TcpDiscoverySpi()
> val ipFinder: TcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder()
> val hostList: List[String] = 
> Arrays.asList("127.0.0.1:47500..47509,18.206.247.40:47500..47509,52.207.217.31:47500..47509".split(","):
>  _*)
> ipFinder.setAddresses(hostList)
> spi.setIpFinder(ipFinder)
> val cfg: IgniteConfiguration = new IgniteConfiguration()
> cfg.setDiscoverySpi(spi)
> cfg.setClientMode(true)
>     System.out.println(">>> I am here")
>
> cfg.setPeerClassLoadingEnabled(true)
> val ignite: Ignite = Ignition.start(cfg)
> cache = Ignition.ignite().cache("test")
> System.out.println(">>> cache acquired")
>
> System.exit(0)
>
>
>   }
>
> }
>
>
> On Fri, Oct 18, 2019 at 12:56 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> Hi Stephen ,
>
> I followed below steps and created one node ec2 instance with Ignite on it
> now I am connecting to the ignite cluster using spark in two ways 1) thin
> client 2) I don't know you have to tell me
>
> my question is what is the value I have to pass for option 2 is it ec2
> instance public IP with port 47500..47509  in my example-default.xml file ?
>
> https://www.gridgain.com/docs/8.7.6//installation-guide/
> manual-install-on-ec2
>
> Thanks
> Sri
>
> On Fri, Oct 18, 2019 at 12:47 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
> You’re still pointing your Spark node (*thick* client) at port 10800 (the
> *thin* client port). This is not going to work.
>
> You can create a table using sqlline and read it using 

Apache Ignite + DynamoDB Streams on AWS

[sri hari kali charan Tummala]

Hi All,

this email is for info sake I put together a document that covers how to
set up Ignite on AWS and fixed missing code (
https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/)
in this blog, which can be useful for Quick POC.

Working Code to get DynamodB Streams and write to Ignite:-
https://github.com/kali786516/ApacheIgnitePoc/tree/master/src/main/scala/com/ignite/examples/WorkingOrderDetailsDynStreams



-- 
Thanks & Regards
Sri Tummala


IgniteonAWS.docx
Description: MS-Word 2007 document

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

so I got an answer from Grid Grain computing, JDBC thin client is the only
way yo connect the ignite cluster which is running on aws.

https://forums.gridgain.com/community-home/digestviewer/viewthread?MessageKey=13f5e836-1569-486a-8475-84c70fc141e0=3b551477-7e2d-462f-bc5f-d6d10ccbbe35=digestviewer=Thank+you+for+submitting+your+message.

Thanks
Sri

On Fri, Oct 18, 2019 at 1:38 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Ok I created a 2 node ec2 instance ignite cluster is below is the right
> way to create cache? the code still using my laptop resources unable to
> connect ignite cluster.
>
> Out put:-
> [13:36:36] Ignite node started OK (id=8535f4d3)
> [13:36:36] Topology snapshot [ver=30, locNode=8535f4d3, *servers=1,*
> clients=1, state=ACTIVE, CPUs=8, *offheap=6.4GB, heap=14.0GB*]
> >>> cache acquired
>
> package com.ignite.examples.igniteStartup
>
> import java.util.Arrays
> import java.util.List
> import com.ignite.examples.model.Address
> import org.apache.commons.logging.Log
> import org.apache.commons.logging.LogFactory
> import org.apache.ignite.Ignite
> import org.apache.ignite.IgniteCache
> import org.apache.ignite.Ignition
> import org.apache.ignite.configuration.IgniteConfiguration
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
>
> //remove if not needed
> import scala.collection.JavaConversions._
>
> object IgniteStart2 {
>
>   private var cache: IgniteCache[String, Address] = _
>
>   def main(args: Array[String]): Unit = {
>
> val spi: TcpDiscoverySpi = new TcpDiscoverySpi()
> val ipFinder: TcpDiscoveryVmIpFinder = new TcpDiscoveryVmIpFinder()
> val hostList: List[String] = 
> Arrays.asList("127.0.0.1:47500..47509,18.206.247.40:47500..47509,52.207.217.31:47500..47509".split(","):
>  _*)
> ipFinder.setAddresses(hostList)
> spi.setIpFinder(ipFinder)
> val cfg: IgniteConfiguration = new IgniteConfiguration()
> cfg.setDiscoverySpi(spi)
> cfg.setClientMode(true)
> System.out.println(">>> I am here")
>
> cfg.setPeerClassLoadingEnabled(true)
> val ignite: Ignite = Ignition.start(cfg)
> cache = Ignition.ignite().cache("test")
> System.out.println(">>> cache acquired")
>
> System.exit(0)
>
>
>   }
>
> }
>
>
> On Fri, Oct 18, 2019 at 12:56 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Stephen ,
>>
>> I followed below steps and created one node ec2 instance with Ignite on
>> it now I am connecting to the ignite cluster using spark in two ways 1)
>> thin client 2) I don't know you have to tell me
>>
>> my question is what is the value I have to pass for option 2 is it ec2
>> instance public IP with port 47500..47509  in my example-default.xml file ?
>>
>>
>> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2
>>
>> Thanks
>> Sri
>>
>> On Fri, Oct 18, 2019 at 12:47 PM Stephen Darlington <
>> stephen.darling...@gridgain.com> wrote:
>>
>>> You’re still pointing your Spark node (*thick* client) at port 10800
>>> (the *thin* client port). This is not going to work.
>>>
>>> You can create a table using sqlline and read it using Spark, or vice
>>> versa. But you need a functioning cluster.
>>>
>>> Check out the documentation on clustering concepts and configuration:
>>> https://www.gridgain.com/docs/latest/developers-guide/clustering/clustering
>>>
>>>
>>> On 18 Oct 2019, at 16:32, sri hari kali charan Tummala <
>>> kali.tumm...@gmail.com> wrote:
>>>
>>> Hi Stephen/All,
>>>
>>> got it working somewhat using below, but have an issue table and data
>>> which is created using thin client is failing to read using spark but table
>>> created by spark can be read using a thin client, that means table created
>>> in Ignite using spark are the only ones read using spark in Ignite?
>>>
>>> example-default.xml
>>>
>>> 
>>> 
>>> 
>>> 18.206.247.40:10800
>>> 
>>> 
>>>
>>>
>>> NotWorking Scala Function:-
>>>
>>> def readThinClientTableUsingSpark(implicit spark: SparkSession) = {
>>>   val personDataFrame = spark.read
>>> .format(FORMAT_IGNITE)
>>> .option(OPTION_CONFIG_FILE, CONFIG)
>>> .option(O

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

so spark + jdbc thin client is the only way to go forward?

 Reading:-

val df = spark.read
  .format("jdbc")
  .option("url", "jdbc:ignite:thin://18.206.247.40")
  .option("fetchsize",100)
  //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
  .option("dbtable", "Person2").load()

df.printSchema()

df.createOrReplaceTempView("test")

spark.sql("select * from test where id=1").show(10)

spark.sql("select 4,'blah',124232").show(10)


Writing:-

import java.sql.DriverManager
val connection = DriverManager.getConnection("jdbc:ignite:thin://18.206.247.40")

import java.util.Properties
val connectionProperties = new Properties()

connectionProperties.put("url", "jdbc:ignite:thin://18.206.247.40")

spark.sql("select 4 as ID,'blah' as STREET,124232 as
ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://18.206.247.40",
  "Person2",connectionProperties)


On Fri, Oct 18, 2019 at 12:00 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> unfortunately, tables that are created by spark don't exist in ignite
> when I try to query using sqlline jdbc thin client, so the spark is still
> running locally the tables which are created by spark exists only for that
> session.
>
> did anyone come across this issue? how to resolve it?
>
>
>
>
>
> On Fri, Oct 18, 2019 at 11:32 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Stephen/All,
>>
>> got it working somewhat using below, but have an issue table and data
>> which is created using thin client is failing to read using spark but table
>> created by spark can be read using a thin client, that means table created
>> in Ignite using spark are the only ones read using spark in Ignite?
>>
>> example-default.xml
>>
>> 
>> 
>> 
>> 18.206.247.40:10800
>> 
>> 
>>
>>
>> NotWorking Scala Function:-
>>
>> def readThinClientTableUsingSpark(implicit spark: SparkSession) = {
>>   val personDataFrame = spark.read
>> .format(FORMAT_IGNITE)
>> .option(OPTION_CONFIG_FILE, CONFIG)
>> .option(OPTION_TABLE, "person")
>> .load()
>>
>>   println()
>>   println("Data frame thin connection content:person")
>>   println()
>>
>>   //Printing content of data frame to console.
>>   personDataFrame.show()
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Exception in thread "main" class org.apache.ignite.IgniteException:
>> Unknown table person at
>> org.apache.ignite.spark.impl.IgniteSQLRelation$$anonfun$schema$2.apply(IgniteSQLRelation.scala:46)
>> at
>> org.apache.ignite.spark.impl.IgniteSQLRelation$$anonfun$schema$2.apply(IgniteSQLRelation.scala:46)
>> at scala.Option.getOrElse(Option.scala:121) at
>> org.apache.ignite.spark.impl.IgniteSQLRelation.schema(IgniteSQLRelation.scala:46)
>> at
>> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:431)
>> at
>> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227) at
>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164) at
>> com.ignite.examples.spark.SparkIgniteCleanCode$.readThinClientTableUsingSpark(SparkIgniteCleanCode.scala:154)
>> at
>> com.ignite.examples.spark.SparkIgniteCleanCode$.main(SparkIgniteCleanCode.scala:216)
>> at
>> com.ignite.examples.spark.SparkIgniteCleanCode.main(SparkIgniteCleanCode.scala)*
>>
>> *Full Code:- (step 7 Fails)*
>>
>> package com.ignite.examples.spark
>>
>> import com.ignite.examples.model.Address
>> import org.apache.ignite.{Ignite, Ignition}
>> import org.apache.ignite.cache.query.SqlFieldsQuery
>> import org.apache.ignite.client.{ClientCache, IgniteClient}
>> import org.apache.ignite.configuration.{CacheConfiguration, 
>> ClientConfiguration}
>> import java.lang.{Long => JLong, String => JString}
>>
>> import com.ignite.examples.spark.SparkClientConnectionTest.{CACHE_NAME, 
>> CONFIG}
>> import org.apache.ignite.internal.util.IgniteUtils.resolveIgnitePath
>> import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE, 
>> OPTION_CONFIG_FILE, OPTION_CREATE_TABLE_PARAMETERS, 
>> OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS, OPTION_TABLE}
>> import org.apache.log4j.{Level, Logger}
>> import org.apache.spark.sql.

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

ma/util;
   xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd;>

































18.206.247.40:10800










Thanks
Sri

On Fri, Oct 18, 2019 at 9:23 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> do you mean communication ports 47100-47200 as mentioned (
> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2)
> ? bare in mind I am running my spark job outside ignite ec2 box (MAC PC).
>
> which option is right in my default.xml file?
>
> Option 1:-
>
> 
> 
> 
> 3.88.248.113:4 
> <http://3.88.248.113:10800/>7100..47200
> 
> 
>
>
> Option 2:-
>
> 
> 
> 
> 3.88.248.113:4 
> <http://3.88.248.113:10800/>7500..47600
> 
> 
>
>
> Option 3:-
>
> 
>         
>         
> 3.88.248.113 
> <http://3.88.248.113:10800/>
> 
> 
>
>
>
> Thanks
>
> On Fri, Oct 18, 2019 at 9:18 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Stephen ,
>>
>>  do you mean 3.88.248.113: <http://3.88.248.113:10800/>47500..47700
>> something like this? or just public ip 3.88.248.113
>> <http://3.88.248.113:10800/> I tried all the possibilities none of them
>> are getting connected.
>>
>> Thanks
>> Sri
>>
>> On Fri, Oct 18, 2019 at 6:02 AM Stephen Darlington <
>> stephen.darling...@gridgain.com> wrote:
>>
>>> You’re trying to connect a thick client (the Spark integration) to the
>>> thin client port (10800). Your example-default.xml file needs to have the
>>> same configuration as your server node(s).
>>>
>>> Regards,
>>> Stephen
>>>
>>> On 17 Oct 2019, at 18:12, sri hari kali charan Tummala <
>>> kali.tumm...@gmail.com> wrote:
>>>
>>> Hi Community,
>>>
>>> I am trying to read and write into the Ignite cluster using apache-spark
>>> I am able to do that using JDBC thin client but not native method as
>>> mentioned in several spark + ignite examples.
>>>
>>> Right now all the spark + ignite examples launch a local ignite cluster
>>> but I want my code connecting to already existing cluster (client).
>>>
>>> Question:-
>>> *How to pass Ignite connection ip and port (10800)  10800 in
>>> example-default.xml ?*
>>>
>>> Error:-
>>> *TcpDiscoverySpi: Failed to connect to any address from IP finder (will
>>> retry to join topology every 2000 ms; change 'reconnectDelay' to configure
>>> the frequency of retries): [/3.88.248.113:10800
>>> <http://3.88.248.113:10800/>]*
>>>
>>> Working (Spark + Ignite using JDBC):-
>>>
>>> val df = spark.read
>>> .format("jdbc")
>>> .option("url", "jdbc:ignite:thin://3.88.248.113")
>>> .option("fetchsize",100)
>>> //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
>>> .option("dbtable", "Person").load()
>>>
>>> df.printSchema()
>>>
>>> df.createOrReplaceTempView("test")
>>>
>>> spark.sql("select * from test where id=1").show(10)
>>>
>>> spark.sql("select 4,'blah',124232").show(10)
>>>
>>> import java.sql.DriverManager
>>> val connection = 
>>> DriverManager.getConnection("

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

do you mean communication ports 47100-47200 as mentioned (
https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2)
? bare in mind I am running my spark job outside ignite ec2 box (MAC PC).

which option is right in my default.xml file?

Option 1:-




3.88.248.113:4
<http://3.88.248.113:10800/>7100..47200




Option 2:-




3.88.248.113:4
<http://3.88.248.113:10800/>7500..47600




Option 3:-




3.88.248.113
<http://3.88.248.113:10800/>





Thanks

On Fri, Oct 18, 2019 at 9:18 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi Stephen ,
>
>  do you mean 3.88.248.113: <http://3.88.248.113:10800/>47500..47700
> something like this? or just public ip 3.88.248.113
> <http://3.88.248.113:10800/> I tried all the possibilities none of them
> are getting connected.
>
> Thanks
> Sri
>
> On Fri, Oct 18, 2019 at 6:02 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> You’re trying to connect a thick client (the Spark integration) to the
>> thin client port (10800). Your example-default.xml file needs to have the
>> same configuration as your server node(s).
>>
>> Regards,
>> Stephen
>>
>> On 17 Oct 2019, at 18:12, sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>> Hi Community,
>>
>> I am trying to read and write into the Ignite cluster using apache-spark
>> I am able to do that using JDBC thin client but not native method as
>> mentioned in several spark + ignite examples.
>>
>> Right now all the spark + ignite examples launch a local ignite cluster
>> but I want my code connecting to already existing cluster (client).
>>
>> Question:-
>> *How to pass Ignite connection ip and port (10800)  10800 in
>> example-default.xml ?*
>>
>> Error:-
>> *TcpDiscoverySpi: Failed to connect to any address from IP finder (will
>> retry to join topology every 2000 ms; change 'reconnectDelay' to configure
>> the frequency of retries): [/3.88.248.113:10800
>> <http://3.88.248.113:10800/>]*
>>
>> Working (Spark + Ignite using JDBC):-
>>
>> val df = spark.read
>> .format("jdbc")
>> .option("url", "jdbc:ignite:thin://3.88.248.113")
>> .option("fetchsize",100)
>> //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
>> .option("dbtable", "Person").load()
>>
>> df.printSchema()
>>
>> df.createOrReplaceTempView("test")
>>
>> spark.sql("select * from test where id=1").show(10)
>>
>> spark.sql("select 4,'blah',124232").show(10)
>>
>> import java.sql.DriverManager
>> val connection = 
>> DriverManager.getConnection("jdbc:ignite:thin://3.88.248.113")
>>
>> import java.util.Properties
>> val connectionProperties = new Properties()
>>
>> connectionProperties.put("url", "jdbc:ignite:thin://3.88.248.113")
>>
>> spark.sql("select 4 as ID,'blah' as STREET,124232 as 
>> ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://3.88.248.113",
>>   "Person",connectionProperties)
>>
>> spark.read
>>   .format("jdbc")
>>   .option("url", "jdbc:ignite:thin://3.88.248.113")
>>   .option("fetchsize",100)
>>   .option("dbtable", "Person").load().show(10,false)
>>
>>
>> Not Working requires a CONFIG file which is example-default.xml:-
>>
>> val igniteDF = spark.read
>>   .format(FORMAT_IGNITE) //Data source type.
>>   .option(OPTION_TABLE, "person") //Table to read.
>>   .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
>>   .load()
>>   .filter(col("id") >= 2) //Filter clause.
>>   .filter(col("name") like "%J%") //Another filter clause.
>>
>>
>> Full Code:- (sparkDSLExample) function fails to connect ignite cluster
>> which I already have
>>
>> package com.ignite.examples.spark
>>
>> import com.ignite.examples.model.A

Re: Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

Hi Stephen ,

 do you mean 3.88.248.113: <http://3.88.248.113:10800/>47500..47700
something like this? or just public ip 3.88.248.113
<http://3.88.248.113:10800/> I tried all the possibilities none of them are
getting connected.

Thanks
Sri

On Fri, Oct 18, 2019 at 6:02 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> You’re trying to connect a thick client (the Spark integration) to the
> thin client port (10800). Your example-default.xml file needs to have the
> same configuration as your server node(s).
>
> Regards,
> Stephen
>
> On 17 Oct 2019, at 18:12, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> Hi Community,
>
> I am trying to read and write into the Ignite cluster using apache-spark I
> am able to do that using JDBC thin client but not native method as
> mentioned in several spark + ignite examples.
>
> Right now all the spark + ignite examples launch a local ignite cluster
> but I want my code connecting to already existing cluster (client).
>
> Question:-
> *How to pass Ignite connection ip and port (10800)  10800 in
> example-default.xml ?*
>
> Error:-
> *TcpDiscoverySpi: Failed to connect to any address from IP finder (will
> retry to join topology every 2000 ms; change 'reconnectDelay' to configure
> the frequency of retries): [/3.88.248.113:10800
> <http://3.88.248.113:10800/>]*
>
> Working (Spark + Ignite using JDBC):-
>
> val df = spark.read
> .format("jdbc")
> .option("url", "jdbc:ignite:thin://3.88.248.113")
> .option("fetchsize",100)
> //.option("driver", "org.apache.ignite.IgniteJdbcDriver")
> .option("dbtable", "Person").load()
>
> df.printSchema()
>
> df.createOrReplaceTempView("test")
>
> spark.sql("select * from test where id=1").show(10)
>
> spark.sql("select 4,'blah',124232").show(10)
>
> import java.sql.DriverManager
> val connection = 
> DriverManager.getConnection("jdbc:ignite:thin://3.88.248.113")
>
> import java.util.Properties
> val connectionProperties = new Properties()
>
> connectionProperties.put("url", "jdbc:ignite:thin://3.88.248.113")
>
> spark.sql("select 4 as ID,'blah' as STREET,124232 as 
> ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://3.88.248.113",
>   "Person",connectionProperties)
>
> spark.read
>   .format("jdbc")
>   .option("url", "jdbc:ignite:thin://3.88.248.113")
>   .option("fetchsize",100)
>   .option("dbtable", "Person").load().show(10,false)
>
>
> Not Working requires a CONFIG file which is example-default.xml:-
>
> val igniteDF = spark.read
>   .format(FORMAT_IGNITE) //Data source type.
>   .option(OPTION_TABLE, "person") //Table to read.
>   .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
>   .load()
>   .filter(col("id") >= 2) //Filter clause.
>   .filter(col("name") like "%J%") //Another filter clause.
>
>
> Full Code:- (sparkDSLExample) function fails to connect ignite cluster
> which I already have
>
> package com.ignite.examples.spark
>
> import com.ignite.examples.model.Address
> import org.apache.ignite.{Ignite, Ignition}
> import org.apache.ignite.cache.query.SqlFieldsQuery
> import org.apache.ignite.client.{ClientCache, IgniteClient}
> import org.apache.ignite.configuration.{CacheConfiguration, 
> ClientConfiguration}
> import java.lang.{Long => JLong, String => JString}
>
> import org.apache.ignite.cache.query.SqlFieldsQuery
> import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE, 
> OPTION_CONFIG_FILE, OPTION_TABLE}
> import org.apache.log4j.{Level, Logger}
> import org.apache.spark.sql.{SaveMode, SparkSession}
> import org.apache.spark.sql.functions.col
>
> object SparkClientConnectionTest {
>
>   private val CACHE_NAME = "SparkCache"
>
>   private val CONFIG = 
> "/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheIgnitePoc/src/main/scala/com/ignite/examples/config/example-ignite.xml"
>
>   def setupExampleData = {
>
> val cfg2 = new ClientConfiguration().setAddresses("3.88.248.113:10800")
> val igniteClient:IgniteClient = Ignition.startClient(cfg2)
>
> System.out.format(">>> Created cache [%s].\n", CACHE_NAME)
>
> val cache:ClientCache[Integer, Address] = 
> igniteClient.getOrCreateCache(CACHE_NAME)
>
> cache.query(new SqlFieldsQuery(String.format("DROP TABLE IF EXISTS 
> Per

Apache Spark + Ignite Connection Issue

[sri hari kali charan Tummala]

Hi Community,

I am trying to read and write into the Ignite cluster using apache-spark I
am able to do that using JDBC thin client but not native method as
mentioned in several spark + ignite examples.

Right now all the spark + ignite examples launch a local ignite cluster but
I want my code connecting to already existing cluster (client).

Question:-
*How to pass Ignite connection ip and port (10800)  10800 in
example-default.xml ?*

Error:-
*TcpDiscoverySpi: Failed to connect to any address from IP finder (will
retry to join topology every 2000 ms; change 'reconnectDelay' to configure
the frequency of retries): [/3.88.248.113:10800
]*

Working (Spark + Ignite using JDBC):-

val df = spark.read
.format("jdbc")
.option("url", "jdbc:ignite:thin://3.88.248.113")
.option("fetchsize",100)
//.option("driver", "org.apache.ignite.IgniteJdbcDriver")
.option("dbtable", "Person").load()

df.printSchema()

df.createOrReplaceTempView("test")

spark.sql("select * from test where id=1").show(10)

spark.sql("select 4,'blah',124232").show(10)

import java.sql.DriverManager
val connection = DriverManager.getConnection("jdbc:ignite:thin://3.88.248.113")

import java.util.Properties
val connectionProperties = new Properties()

connectionProperties.put("url", "jdbc:ignite:thin://3.88.248.113")

spark.sql("select 4 as ID,'blah' as STREET,124232 as
ZIP").write.mode(SaveMode.Append).jdbc("jdbc:ignite:thin://3.88.248.113",
  "Person",connectionProperties)

spark.read
  .format("jdbc")
  .option("url", "jdbc:ignite:thin://3.88.248.113")
  .option("fetchsize",100)
  .option("dbtable", "Person").load().show(10,false)


Not Working requires a CONFIG file which is example-default.xml:-

val igniteDF = spark.read
  .format(FORMAT_IGNITE) //Data source type.
  .option(OPTION_TABLE, "person") //Table to read.
  .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
  .load()
  .filter(col("id") >= 2) //Filter clause.
  .filter(col("name") like "%J%") //Another filter clause.


Full Code:- (sparkDSLExample) function fails to connect ignite cluster
which I already have

package com.ignite.examples.spark

import com.ignite.examples.model.Address
import org.apache.ignite.{Ignite, Ignition}
import org.apache.ignite.cache.query.SqlFieldsQuery
import org.apache.ignite.client.{ClientCache, IgniteClient}
import org.apache.ignite.configuration.{CacheConfiguration, ClientConfiguration}
import java.lang.{Long => JLong, String => JString}

import org.apache.ignite.cache.query.SqlFieldsQuery
import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE,
OPTION_CONFIG_FILE, OPTION_TABLE}
import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.{SaveMode, SparkSession}
import org.apache.spark.sql.functions.col

object SparkClientConnectionTest {

  private val CACHE_NAME = "SparkCache"

  private val CONFIG =
"/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheIgnitePoc/src/main/scala/com/ignite/examples/config/example-ignite.xml"

  def setupExampleData = {

val cfg2 = new ClientConfiguration().setAddresses("3.88.248.113:10800")
val igniteClient:IgniteClient = Ignition.startClient(cfg2)

System.out.format(">>> Created cache [%s].\n", CACHE_NAME)

val cache:ClientCache[Integer, Address] =
igniteClient.getOrCreateCache(CACHE_NAME)

cache.query(new SqlFieldsQuery(String.format("DROP TABLE IF EXISTS Person"))
  .setSchema("PUBLIC")).getAll

cache.query(new SqlFieldsQuery(String.format("CREATE TABLE IF NOT
EXISTS Person (id LONG,street varchar, zip VARCHAR, PRIMARY KEY (id) )
WITH \"VALUE_TYPE=%s\"", classOf[Address].getName))
  .setSchema("PUBLIC")).getAll

cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(1L.asInstanceOf[JLong],"Jameco",
"04074").setSchema("PUBLIC")).getAll
cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(2L.asInstanceOf[JLong],"Bremar road",
"520003").setSchema("PUBLIC")).getAll
cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(3L.asInstanceOf[JLong],"orange road",
"1234").setSchema("PUBLIC")).getAll

System.out.format(">>> Data Inserted into Cache [%s].\n", CACHE_NAME)

val data=cache.query(new SqlFieldsQuery("select * from
Person").setSchema("PUBLIC")).getAll

println(data.toString)
  }

  def sparkDSLExample(implicit spark: SparkSession): Unit = {
println("Querying using Spark DSL.")
println


val igniteDF = spark.read
  .format(FORMAT_IGNITE) //Data source type.
  .option(OPTION_TABLE, "person") //Table to read.
  .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
  .load()
  .filter(col("id") >= 2) //Filter clause.
  .filter(col("name") like "%J%") //Another filter clause.

println("Data frame schema:")

igniteDF.printSchema() //Printing query schema to console.

println("Data frame 

Re: Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

Hi Stephen ,

in your example or any spark example I only see creating local ignite node
how to connect spark to ignite client which I already have?

b.filter('href is not null') \
.drop('hash', 'meta') \
.write.format('ignite') \
.option('config',*'default-config.xml') \*
.option('table','bookmarks') \
.option('primaryKeyFields','href') \
.mode('overwrite') \
.save()

My code:-

package com.ignite.examples.spark

import com.ignite.examples.model.Address
import org.apache.ignite.{Ignite, Ignition}
import org.apache.ignite.cache.query.SqlFieldsQuery
import org.apache.ignite.client.{ClientCache, IgniteClient}
import org.apache.ignite.configuration.{CacheConfiguration, ClientConfiguration}
import java.lang.{Long => JLong, String => JString}
import org.apache.ignite.cache.query.SqlFieldsQuery
import org.apache.ignite.spark.IgniteDataFrameSettings.{FORMAT_IGNITE,
OPTION_CONFIG_FILE, OPTION_TABLE}
import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.col

object SparkClientConnectionTest {

  private val CACHE_NAME = "SparkCache"

  //private val CONFIG =
"/Users/kalit_000/Downloads/designing-event-driven-applications-apache-kafka-ecosystem/05/demos/kafka-streams-after/ApacheIgnitePoc/src/main/scala/com/ignite/examples/config/example-ignite.xml"


  def setupExampleData = {

val cfg2 = new ClientConfiguration().setAddresses("3.88.248.113:10800")
val igniteClient:IgniteClient = Ignition.startClient(cfg2)

System.out.format(">>> Created cache [%s].\n", CACHE_NAME)

val cache:ClientCache[Integer, Address] =
igniteClient.getOrCreateCache(CACHE_NAME)

cache.query(new SqlFieldsQuery(String.format("DROP TABLE IF EXISTS Person"))
  .setSchema("PUBLIC")).getAll

cache.query(new SqlFieldsQuery(String.format("CREATE TABLE IF NOT
EXISTS Person (id LONG,street varchar, zip VARCHAR, PRIMARY KEY (id) )
WITH \"VALUE_TYPE=%s\"", classOf[Address].getName))
  .setSchema("PUBLIC")).getAll

cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(1L.asInstanceOf[JLong],"Jameco",
"04074").setSchema("PUBLIC")).getAll
cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(2L.asInstanceOf[JLong],"Bremar road",
"520003").setSchema("PUBLIC")).getAll
cache.query(new SqlFieldsQuery("INSERT INTO Person(id,street, zip)
VALUES(?,?, ?)").setArgs(3L.asInstanceOf[JLong],"orange road",
"1234").setSchema("PUBLIC")).getAll

System.out.format(">>> Data Inserted into Cache [%s].\n", CACHE_NAME)

val data=cache.query(new SqlFieldsQuery("select * from
Person").setSchema("PUBLIC")).getAll

println(data.toString)
  }

  def sparkDSLExample(implicit spark: SparkSession): Unit = {
println("Querying using Spark DSL.")
println

val igniteDF = spark.read
  .format(FORMAT_IGNITE) //Data source type.
  .option(OPTION_TABLE, "person") //Table to read.
  .option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
  .load()
  .filter(col("id") >= 2) //Filter clause.
  .filter(col("name") like "%J%") //Another filter clause.

println("Data frame schema:")

igniteDF.printSchema() //Printing query schema to console.

println("Data frame content:")

igniteDF.show() //Printing query results to console.
  }


  def main(args: Array[String]): Unit = {

setupExampleData

//Creating spark session.
implicit val spark = SparkSession.builder()
  .appName("Spark Ignite data sources example")
  .master("local")
  .config("spark.executor.instances", "2")
  .getOrCreate()

// Adjust the logger to exclude the logs of no interest.
Logger.getRootLogger.setLevel(Level.ERROR)
Logger.getLogger("org.apache.ignite").setLevel(Level.INFO)

sparkDSLExample




  }

}





On Thu, Oct 17, 2019 at 7:09 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> You have to tell it *where* to connect:
>
> ./sqlline -u jdbc:ignite:thin://127.0.0.1/
>
> I also wrote this showing a few ways to load data without firing up an IDE:
>
> https://medium.com/@sdarlington/loading-data-into-apache-ignite-c0cb7c065a7
>
> My current favourite “no Java” method of playing around is a Scala REPL.
>
> Regards,
> Stephen
>
> On 16 Oct 2019, at 19:38, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> what are the next steps after creating a cluster, I want to run basic sql
> create some tables and load some data right now I dont want to code right
> away in 

Re: Connection Refused in Scala Ignite Client (Help!)

[sri hari kali charan Tummala]

worked like a charm, thank you.

On Thu, Oct 17, 2019 at 2:54 AM Artem Budnikov 
wrote:

> Hi,
>
> Looks like you are trying to connect to the wrong port. The default port
> for client connection is 10800.
>
> Change:
>
> val cfg2 = new ClientConfiguration().setAddresses("100.25.173.220:47100
> ..47700")
>
> to
>
> val cfg2 = new ClientConfiguration().setAddresses("100.25.173.220:10800")
>
>
> -Artem
> On 17.10.2019 03:39, alexanderko...@gmail.com wrote:
>
> You need to replace the ip finder on all nodes to use the s3 one.
>
> add the libraries from libs/optional/ignite-aws to your application's
> classpath.
>
> No file is required.
>
>
>
> *From:* sri hari kali charan Tummala 
> 
> *Sent:* Wednesday, October 16, 2019 8:10 PM
> *To:* user@ignite.apache.org; alexanderko...@gmail.com
> *Subject:* Re: Connection Refused in Scala Ignite Client (Help!)
>
>
>
> I this below property is while launching ignite on aws
>
> I already have ignite cluster running I just need to connect using java or
> Scala code which is failing with  connection refused error.
>
>
>
> https://apacheignite-mix.readme.io/docs/amazon-aws#section-amazon-s3-based-discovery
>
>
>
>
>
>
>
> On Wednesday, October 16, 2019, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> Do I need to keep any file inside the s3 bucket or is it just a empty
> bucket ? With aws credentials and bucket name ?
>
>
>
>
>
> On Wednesday, October 16, 2019,  wrote:
>
> Hi,
>
>Try using the S3 IP Finder:
> https://apacheignite-mix.readme.io/docs/amazon-aws#section-amazon-s3-based-discovery
>
>
>
>   
>
> 
>
>   
>
>  "org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
>
>   
>
>   
>
> 
>
>   
>
> 
>
>   
>
>
>
> Thanks, Alex
>
>
>
>
>
> *From:* sri hari kali charan Tummala 
> *Sent:* Wednesday, October 16, 2019 5:55 PM
> *To:* user@ignite.apache.org
> *Subject:* Connection Refused in Scala Ignite Client (Help!)
>
>
>
> Hi All,
>
>
>
> I am trying to connect my ignite cluster which is created on AWS (), now I
> am suing scala code to connect to the client which is failing with
> connection refused error.
>
>
>
> In my security group I opened all the ports still I am having issues can
> anyone help.
>
>
>
> Error:-
>
>
> *Exception in thread "main"
> org.apache.ignite.client.ClientConnectionException: Ignite cluster is
> unavailable at org.apache.i*
> gnite.internal.client.thin.TcpClientChannel.(TcpClientChannel.java:114)
> at
> org.apache.ignite.internal.client.thin.TcpIgniteClient.lambda$new$0(TcpIgniteClient.java:79)
> at
> org.apache.ignite.internal.client.thin.ReliableChannel.(ReliableChannel.java:84)
> at
> org.apache.ignite.internal.client.thin.TcpIgniteClient.(TcpIgniteClient.java:86)
> at
> org.apache.ignite.internal.client.thin.TcpIgniteClient.start(TcpIgniteClient.java:205)
> at org.apache.ignite.Ignition.startClient(Ignition.java:586)
> at
> com.ignite.examples.igniteStartup.IgniteClientConnection$.main(IgniteClientConnection.scala:36)
> at
> com.ignite.examples.igniteStartup.IgniteClientConnection.main(IgniteClientConnection.scala)
> *Caused by: java.net.ConnectException: Connection refused (Connection
> refused)*
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net
> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at java.net.Socket.connect(Socket.java:538)
> at java.net.Socket.(Socket.java:434)
> at java.net.Socket.(Socket.java:211)
> at
> org.apache.ignite.internal.client.thin.TcpClientChannel.createSocket(TcpClientChannel.java:216)
> at
> org.apache.ignite.internal.client.thin.TcpClientChannel.(TcpClientChannel.java:108)
> ... 7 more
>
>
>
> Scala Code:-
>
> package com.ignite.examples.igniteStartup
> import org.apache.ignite.Igniteimport org.apache.ignite.IgniteCacheimport 
> org.apache.ignite.Ignitionimport 
> org.apache.ignite.configuration.{ClientConfiguration, 
> IgniteConfiguration}import 
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinderimport 
> org.apache.ignite.configuration.IgniteConfigurationimport 
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi

Re: Ignite Node Connection Issue

[sri hari kali charan Tummala]

Thank you, worked like a charm!

On Thu, Oct 17, 2019 at 5:14 AM Denis Mekhanikov 
wrote:

> I answered to you on stackoverflow:
> https://stackoverflow.com/questions/58422096/failing-to-connect-apache-ignite-cluster
> You need to use the port 10800 for thin clients instead of 47xxx
>
> Denis
> On 17 Oct 2019, 00:08 +0300, sri hari kali charan Tummala <
> kali.tumm...@gmail.com>, wrote:
>
> I even tried with simple scala code still no luck!
>
> Error:-
> Failed to connect to any address from IP finder (will retry to join
> topology every 2000 ms; change 'reconnectDelay' to configure the frequency
> of retries):
>
> package com.ignite.examples.igniteStartup
>
> import org.apache.ignite.Ignite
> import org.apache.ignite.IgniteCache
> import org.apache.ignite.Ignition
> import org.apache.ignite.configuration.IgniteConfiguration
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
> import org.apache.ignite.configuration.IgniteConfiguration
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
> import java.util.Arrays
> import java.util.List
> import scala.collection.JavaConversions._
>
> object IgniteClientConnection {
>   def main(args: Array[String]): Unit = {
>
> val spi = new TcpDiscoverySpi
> val ipFinder = new TcpDiscoveryVmIpFinder
> val hostList: List[String] = 
> Arrays.asList(("ec2-100-25-173-220:47500..47509," +
>   "ec2-100-25-173-220.compute-1.amazonaws.com:47500..47509," +
>   "3.86.250.240:47500..47509," +
>   "172.31.81.211:47500..47509," +
>   "100.25.173.220:47500..47509").split(","): _*)
>
> ipFinder.setAddresses(hostList)
> spi.setIpFinder(ipFinder)
> val cfg = new IgniteConfiguration
> cfg.setDiscoverySpi(spi)
> cfg.setClientMode(true)
> cfg.setPeerClassLoadingEnabled(true)
>
> val ignite: Ignite = Ignition.start(cfg)
> Ignition.ignite().cache("test")
> //LOG.info(">>> cache acquired")
>
>
>   }
>
> }
>
>
> On Wed, Oct 16, 2019 at 3:41 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi All,
>>
>> I was able to create a working ignite cluster (
>> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2)
>> on AWS I opened all the ports in my security group but when I try to
>> connect ignite cluster from my Local PC Scala Code ignite fails with below
>> error.
>>
>> I am using example-default.xml file
>>
>> 
>> 
>> 
>> 
>> 
>> > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>> 
>> 
>> 
>> 
>> 3.86.250.240:47100..47200
>> 3.84.154.193:47100..47200
>> 127.0.0.1:47100..47200
>> 
>> 
>> 
>> 
>> 
>> 
>>
>>
>> *Failed to connect to any address from IP finder (make sure IP finder
>> addresses are correct and firewalls are disabled on all host machines)*
>>
>> *Error on Ignite Ec2 instance:-*
>>
>> class org.apache.ignite.internal.util.nio.GridNioException: Invalid
>> message type: -4692
>>
>> at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2437)
>>
>> at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2178)
>>
>> at
>> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1819)
>>
>> at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Caused by: class org.apache.ignite.IgniteException: Invalid message type:
>> -4692
>>
>> at
>> org.apache.ignite.internal.managers.communication.GridIoMessageFactory.create(GridIoMessageFactory.java:1151)
>>
>> at
>> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$6.create(TcpCommunicationSpi.java:2336)
>>
>> at
>> org.apache.ignite.internal.util.nio.GridDirectParser.decode(GridDirectParser.java:80)
>>
>> at
>> org.apache.

Re: Connection Refused in Scala Ignite Client (Help!)

[sri hari kali charan Tummala]

I this below property is while launching ignite on aws
I already have ignite cluster running I just need to connect using java or
Scala code which is failing with  connection refused error.

https://apacheignite-mix.readme.io/docs/amazon-aws#section-
amazon-s3-based-discovery



On Wednesday, October 16, 2019, sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Do I need to keep any file inside the s3 bucket or is it just a empty
> bucket ? With aws credentials and bucket name ?
>
>
>
> On Wednesday, October 16, 2019,  wrote:
>
>> Hi,
>>
>>Try using the S3 IP Finder: https://apacheignite-mix.readm
>> e.io/docs/amazon-aws#section-amazon-s3-based-discovery
>>
>>
>>
>>   
>>
>> 
>>
>>   
>>
>> 
>>
>>       
>>
>>   
>>
>> 
>>
>>   
>>
>> 
>>
>>   
>>
>>
>>
>> Thanks, Alex
>>
>>
>>
>>
>>
>> *From:* sri hari kali charan Tummala 
>> *Sent:* Wednesday, October 16, 2019 5:55 PM
>> *To:* user@ignite.apache.org
>> *Subject:* Connection Refused in Scala Ignite Client (Help!)
>>
>>
>>
>> Hi All,
>>
>>
>>
>> I am trying to connect my ignite cluster which is created on AWS (), now
>> I am suing scala code to connect to the client which is failing with
>> connection refused error.
>>
>>
>>
>> In my security group I opened all the ports still I am having issues can
>> anyone help.
>>
>>
>>
>> Error:-
>>
>>
>> *Exception in thread "main"
>> org.apache.ignite.client.ClientConnectionException: Ignite cluster is
>> unavailableat org.apache.i*gnite.internal.cli
>> ent.thin.TcpClientChannel.(TcpClientChannel.java:114)
>> at org.apache.ignite.internal.client.thin.TcpIgniteClient.lambd
>> a$new$0(TcpIgniteClient.java:79)
>> at org.apache.ignite.internal.client.thin.ReliableChannel.> >(ReliableChannel.java:84)
>> at org.apache.ignite.internal.client.thin.TcpIgniteClient.> >(TcpIgniteClient.java:86)
>> at org.apache.ignite.internal.client.thin.TcpIgniteClient.start
>> (TcpIgniteClient.java:205)
>> at org.apache.ignite.Ignition.startClient(Ignition.java:586)
>> at com.ignite.examples.igniteStartup.IgniteClientConnection$.
>> main(IgniteClientConnection.scala:36)
>> at com.ignite.examples.igniteStartup.IgniteClientConnection.
>> main(IgniteClientConnection.scala)
>> *Caused by: java.net.ConnectException: Connection refused (Connection
>> refused)*
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
>> etImpl.java:350)
>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPl
>> ainSocketImpl.java:206)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
>> Impl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at java.net.Socket.connect(Socket.java:538)
>> at java.net.Socket.(Socket.java:434)
>> at java.net.Socket.(Socket.java:211)
>> at org.apache.ignite.internal.client.thin.TcpClientChannel.crea
>> teSocket(TcpClientChannel.java:216)
>> at org.apache.ignite.internal.client.thin.TcpClientChannel.> t>(TcpClientChannel.java:108)
>> ... 7 more
>>
>>
>>
>> Scala Code:-
>>
>> package com.ignite.examples.igniteStartup
>>
>> import org.apache.ignite.Ignite
>> import org.apache.ignite.IgniteCache
>> import org.apache.ignite.Ignition
>> import org.apache.ignite.configuration.{ClientConfiguration, 
>> IgniteConfiguration}
>> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
>> import org.apache.ignite.configuration.IgniteConfiguration
>> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
>> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
>> import java.util.Arrays
>> import java.util.List
>> import com.ignite.examples.model.Address
>> import org.apache.ignite.client.{ClientCache, IgniteClient}
>>
>> import scala.collection.JavaConversions._
>>
>> object IgniteClientConnection {
>>   def main(args: Array[String]): Unit = {
>>
>> System.*out*.println()
>> System.*out*.println(">>> Thin client put-get example started.")
>> System.*out*.println(">>> I am here.")
>>
>> val cfg2 = new 
>> ClientConfiguration().setAddresses("100.25.173.220:47100..47700")
>> val igniteClient:IgniteClient = Ignition.*startClient*(cfg2)
>> val CACHE_NAME = "put-get-example";
>> //val cache:ClientCache[Integer, Address] = 
>> igniteClient.getOrCreateCache(CACHE_NAME)
>>
>> System.*out*.format(">>> Created cache [%s].\n", CACHE_NAME)
>>
>>
>>
>>   }
>>
>> }
>>
>>
>>
>> --
>>
>> Thanks & Regards
>>
>> Sri Tummala
>>
>>
>>
>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
>

-- 
Thanks & Regards
Sri Tummala

Re: Connection Refused in Scala Ignite Client (Help!)

[sri hari kali charan Tummala]

Do I need to keep any file inside the s3 bucket or is it just a empty
bucket ? With aws credentials and bucket name ?



On Wednesday, October 16, 2019,  wrote:

> Hi,
>
>Try using the S3 IP Finder: https://apacheignite-mix.
> readme.io/docs/amazon-aws#section-amazon-s3-based-discovery
>
>
>
>   
>
> 
>
>   
>
> 
>
>   
>
>   
>
> 
>
>   
>
>     
>
>   
>
>
>
> Thanks, Alex
>
>
>
>
>
> *From:* sri hari kali charan Tummala 
> *Sent:* Wednesday, October 16, 2019 5:55 PM
> *To:* user@ignite.apache.org
> *Subject:* Connection Refused in Scala Ignite Client (Help!)
>
>
>
> Hi All,
>
>
>
> I am trying to connect my ignite cluster which is created on AWS (), now I
> am suing scala code to connect to the client which is failing with
> connection refused error.
>
>
>
> In my security group I opened all the ports still I am having issues can
> anyone help.
>
>
>
> Error:-
>
>
> *Exception in thread "main"
> org.apache.ignite.client.ClientConnectionException: Ignite cluster is
> unavailableat org.apache.i*gnite.internal.client.thin.TcpClientChannel.<
> init>(TcpClientChannel.java:114)
> at org.apache.ignite.internal.client.thin.TcpIgniteClient.
> lambda$new$0(TcpIgniteClient.java:79)
> at org.apache.ignite.internal.client.thin.ReliableChannel.<
> init>(ReliableChannel.java:84)
> at org.apache.ignite.internal.client.thin.TcpIgniteClient.<
> init>(TcpIgniteClient.java:86)
> at org.apache.ignite.internal.client.thin.TcpIgniteClient.
> start(TcpIgniteClient.java:205)
> at org.apache.ignite.Ignition.startClient(Ignition.java:586)
> at com.ignite.examples.igniteStartup.IgniteClientConnection$.main(
> IgniteClientConnection.scala:36)
> at com.ignite.examples.igniteStartup.IgniteClientConnection.main(
> IgniteClientConnection.scala)
> *Caused by: java.net.ConnectException: Connection refused (Connection
> refused)*
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:350)
> at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
> 188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at java.net.Socket.connect(Socket.java:538)
> at java.net.Socket.(Socket.java:434)
> at java.net.Socket.(Socket.java:211)
> at org.apache.ignite.internal.client.thin.TcpClientChannel.
> createSocket(TcpClientChannel.java:216)
> at org.apache.ignite.internal.client.thin.TcpClientChannel.<
> init>(TcpClientChannel.java:108)
> ... 7 more
>
>
>
> Scala Code:-
>
> package com.ignite.examples.igniteStartup
>
> import org.apache.ignite.Ignite
> import org.apache.ignite.IgniteCache
> import org.apache.ignite.Ignition
> import org.apache.ignite.configuration.{ClientConfiguration, 
> IgniteConfiguration}
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
> import org.apache.ignite.configuration.IgniteConfiguration
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
> import java.util.Arrays
> import java.util.List
> import com.ignite.examples.model.Address
> import org.apache.ignite.client.{ClientCache, IgniteClient}
>
> import scala.collection.JavaConversions._
>
> object IgniteClientConnection {
>   def main(args: Array[String]): Unit = {
>
> System.*out*.println()
> System.*out*.println(">>> Thin client put-get example started.")
> System.*out*.println(">>> I am here.")
>
> val cfg2 = new 
> ClientConfiguration().setAddresses("100.25.173.220:47100..47700")
> val igniteClient:IgniteClient = Ignition.*startClient*(cfg2)
> val CACHE_NAME = "put-get-example";
> //val cache:ClientCache[Integer, Address] = 
> igniteClient.getOrCreateCache(CACHE_NAME)
>
> System.*out*.format(">>> Created cache [%s].\n", CACHE_NAME)
>
>
>
>   }
>
> }
>
>
>
> --
>
> Thanks & Regards
>
> Sri Tummala
>
>
>


-- 
Thanks & Regards
Sri Tummala

Re: Ignite Node Connection Issue

[sri hari kali charan Tummala]

I even tried with simple scala code still no luck!

Error:-
Failed to connect to any address from IP finder (will retry to join
topology every 2000 ms; change 'reconnectDelay' to configure the frequency
of retries):

package com.ignite.examples.igniteStartup

import org.apache.ignite.Ignite
import org.apache.ignite.IgniteCache
import org.apache.ignite.Ignition
import org.apache.ignite.configuration.IgniteConfiguration
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
import org.apache.ignite.configuration.IgniteConfiguration
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder
import java.util.Arrays
import java.util.List
import scala.collection.JavaConversions._

object IgniteClientConnection {
  def main(args: Array[String]): Unit = {

val spi = new TcpDiscoverySpi
val ipFinder = new TcpDiscoveryVmIpFinder
val hostList: List[String] =
Arrays.asList(("ec2-100-25-173-220:47500..47509," +
  "ec2-100-25-173-220.compute-1.amazonaws.com:47500..47509," +
  "3.86.250.240:47500..47509," +
  "172.31.81.211:47500..47509," +
  "100.25.173.220:47500..47509").split(","): _*)

ipFinder.setAddresses(hostList)
spi.setIpFinder(ipFinder)
val cfg = new IgniteConfiguration
cfg.setDiscoverySpi(spi)
cfg.setClientMode(true)
cfg.setPeerClassLoadingEnabled(true)

val ignite: Ignite = Ignition.start(cfg)
Ignition.ignite().cache("test")
//LOG.info(">>> cache acquired")


  }

}


On Wed, Oct 16, 2019 at 3:41 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi All,
>
> I was able to create a working ignite cluster (
> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2)
> on AWS I opened all the ports in my security group but when I try to
> connect ignite cluster from my Local PC Scala Code ignite fails with below
> error.
>
> I am using example-default.xml file
>
> 
> 
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> 
> 
> 3.86.250.240:47100..47200
> 3.84.154.193:47100..47200
> 127.0.0.1:47100..47200
> 
> 
> 
> 
> 
> 
>
>
> *Failed to connect to any address from IP finder (make sure IP finder
> addresses are correct and firewalls are disabled on all host machines)*
>
> *Error on Ignite Ec2 instance:-*
>
> class org.apache.ignite.internal.util.nio.GridNioException: Invalid
> message type: -4692
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2437)
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2178)
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1819)
>
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: class org.apache.ignite.IgniteException: Invalid message type:
> -4692
>
> at
> org.apache.ignite.internal.managers.communication.GridIoMessageFactory.create(GridIoMessageFactory.java:1151)
>
> at
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$6.create(TcpCommunicationSpi.java:2336)
>
> at
> org.apache.ignite.internal.util.nio.GridDirectParser.decode(GridDirectParser.java:80)
>
> at
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:113)
>
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:108)
>
> at
> org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:132)
>
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:108)
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3575)
>
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:174)
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1312)
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2411
>
>
> --
> Thanks & Regards
> Sri Tummala
>
>

-- 
Thanks & Regards
Sri Tummala

Re: Spring XML Issue

[sri hari kali charan Tummala]

that worked thanks.

On Wed, Oct 16, 2019 at 1:41 PM Evgeniy Rudenko 
wrote:

> Just add
>
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd;>
>
> to the start of the file and
>
> 
>
> to the end.
>
> On Wed, Oct 16, 2019 at 8:22 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi All,
>>
>> I am trying out steps mentioned in this blog (
>> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2)
>> I created Ec2 nodes and .xml file when I try to run I get this bean xml
>> error can anyone help with resolving this am I missing something in the
>> file?
>>
>> Run Command:-
>>
>> ./gridgain-community-8.7.6/bin/ignite.sh aws-static-ip-finder_2.xml
>>
>> File aws-static-ip-finder.xml has below content:-
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> > >
>>
>> 
>>
>> > "org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>
>> 
>>
>> 
>>
>> 172.31.81.211
>>
>> 172.31.82.21
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>>
>>
>> Error:-
>>
>>
>> class org.apache.ignite.IgniteException: *Failed to instantiate Spring
>> XML application context
>> [springUrl=file:/home/ec2-user/aws-static-ip-finder_2.xml, err=Line 1 in
>> XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is
>> invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
>> columnNumber: 67; cvc-elt.1: Cannot find the declaration of element
>> 'bean'.]*
>>
>> * at
>> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1052)*
>>
>> * at org.apache.ignite.Ignition.start(Ignition.java:350)*
>>
>> * at
>> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)*
>>
>> *Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
>> instantiate Spring XML application context
>> [springUrl=file:/home/ec2-user/aws-static-ip-finder_2.xml, err=Line 1 in
>> XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is
>> invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
>> columnNumber: 67; cvc-elt.1: Cannot find the declaration of element
>> 'bean'.]*
>>
>> * at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)*
>>
>> * at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:103)*
>>
>> * at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:97)*
>>
>> * at
>> org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:750)*
>>
>> * at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:951)*
>>
>> * at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:860)*
>>
>> * at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:730)*
>>
>> * at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:699)*
>>
>> * at org.apache.ignite.Ignition.start(Ignition.java:347)*
>>
>> * ... 1 more*
>>
>> *Caused by:
>> org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line
>> 1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml]
>> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
>> 1; columnNumber: 67; cvc-elt.1: Cannot find the declaration of element
>> 'bean'.*
>>
>> * at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:399)*
>>
>> * at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)*
>>
>> * at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBea

Re: Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

what are the next steps after creating a cluster, I want to run basic sql
create some tables and load some data right now I dont want to code right
away in Java will sqlline.sh works? I did launched it fails with no current
connection.



On Wed, Oct 16, 2019 at 2:24 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Working Gridgain Ignite cluster on AWS, followed below steps and changed a
> bit with aws_static_ip file.
>
>
> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2
>
> aws_static_ip:-
>
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd;>
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> 172.31.81.211
> 172.31.82.21
> 
>         
>     
> 
> 
> 
> 
> 
>
> On Wed, Oct 16, 2019 at 1:14 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> I am getting bean invalid exception can you help? values in my
>> aws-static-ip-finder.xml file is below.
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> > >
>>
>> 
>>
>> > "org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>
>> 
>>
>> 
>>
>> 172.31.81.211
>>
>> 172.31.82.21
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>>
>>
>>
>> class org.apache.ignite.IgniteException: Failed to instantiate Spring XML
>> application context
>> [springUrl=file:/home/ec2-user/aws-static-ip-finder.xml, err=Line 1 in XML
>> document from URL [file:/home/ec2-user/aws-static-ip-finder.xml] is
>> invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
>> columnNumber: 68; cvc-elt.1: Cannot find the declaration of element 'bean'.]
>>
>> at
>> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1052)
>>
>> at org.apache.ignite.Ignition.start(Ignition.java:350)
>>
>> at
>> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
>>
>> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
>> instantiate Spring XML application context
>> [springUrl=file:/home/ec2-user/aws-static-ip-finder.xml, err=Line 1 in XML
>> document from URL [file:/home/ec2-user/aws-static-ip-finder.xml] is
>> invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
>> columnNumber: 68; cvc-elt.1: Cannot find the declaration of element 'bean'.]
>>
>> at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
>>
>> at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:103)
>>
>> at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:97)
>>
>> at
>> org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:750)
>>
>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:951)
>>
>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:860)
>>
>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:730)
>>
>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:699)
>>
>> at org.apache.ignite.Ignition.start(Ignition.java:347)
>>
>> ... 1 more
>>
>> Caused by:
>> org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line
>> 1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder.xml]
>> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
>> 1; columnNumber: 68; cvc-elt.1: Cannot find the declaration of element
>> 'bean'.
>>
>> at
>> org.springframework.beans.factory

Re: Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

Working Gridgain Ignite cluster on AWS, followed below steps and changed a
bit with aws_static_ip file.

https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2

aws_static_ip:-

http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;>









172.31.81.211
172.31.82.21









On Wed, Oct 16, 2019 at 1:14 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> I am getting bean invalid exception can you help? values in my
> aws-static-ip-finder.xml file is below.
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
>  "org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>
> 
>
> 
>
> 172.31.81.211
>
> 172.31.82.21
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
>
>
>
> class org.apache.ignite.IgniteException: Failed to instantiate Spring XML
> application context
> [springUrl=file:/home/ec2-user/aws-static-ip-finder.xml, err=Line 1 in XML
> document from URL [file:/home/ec2-user/aws-static-ip-finder.xml] is
> invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
> columnNumber: 68; cvc-elt.1: Cannot find the declaration of element 'bean'.]
>
> at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1052)
>
> at org.apache.ignite.Ignition.start(Ignition.java:350)
>
> at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
>
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> instantiate Spring XML application context
> [springUrl=file:/home/ec2-user/aws-static-ip-finder.xml, err=Line 1 in XML
> document from URL [file:/home/ec2-user/aws-static-ip-finder.xml] is
> invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
> columnNumber: 68; cvc-elt.1: Cannot find the declaration of element 'bean'.]
>
> at
> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
>
> at
> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:103)
>
> at
> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:97)
>
> at
> org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:750)
>
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:951)
>
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:860)
>
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:730)
>
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:699)
>
> at org.apache.ignite.Ignition.start(Ignition.java:347)
>
> ... 1 more
>
> Caused by:
> org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line
> 1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder.xml]
> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
> 1; columnNumber: 68; cvc-elt.1: Cannot find the declaration of element
> 'bean'.
>
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:399)
>
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
>
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
>
> at
> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:378)
>
> ... 9 more
>
> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 68;
> cvc-elt.1: Cannot find the declaration of element 'bean'.
>
> at
> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
>
> at
> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:134)
>
> at
> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:396)
>
> at
> com

Spring XML Issue

[sri hari kali charan Tummala]

Hi All,

I am trying out steps mentioned in this blog (
https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2)
I created Ec2 nodes and .xml file when I try to run I get this bean xml
error can anyone help with resolving this am I missing something in the
file?

Run Command:-

./gridgain-community-8.7.6/bin/ignite.sh aws-static-ip-finder_2.xml

File aws-static-ip-finder.xml has below content:-



















172.31.81.211

172.31.82.21

















Error:-


class org.apache.ignite.IgniteException: *Failed to instantiate Spring XML
application context
[springUrl=file:/home/ec2-user/aws-static-ip-finder_2.xml, err=Line 1 in
XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is
invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
columnNumber: 67; cvc-elt.1: Cannot find the declaration of element
'bean'.]*

* at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1052)*

* at org.apache.ignite.Ignition.start(Ignition.java:350)*

* at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)*

*Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
instantiate Spring XML application context
[springUrl=file:/home/ec2-user/aws-static-ip-finder_2.xml, err=Line 1 in
XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is
invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1;
columnNumber: 67; cvc-elt.1: Cannot find the declaration of element
'bean'.]*

* at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)*

* at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:103)*

* at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:97)*

* at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:750)*

* at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:951)*

* at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:860)*

* at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:730)*

* at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:699)*

* at org.apache.ignite.Ignition.start(Ignition.java:347)*

* ... 1 more*

*Caused by:
org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line
1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml]
is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
1; columnNumber: 67; cvc-elt.1: Cannot find the declaration of element
'bean'.*

* at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:399)*

* at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)*

* at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)*

* at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:378)*

* ... 9 more*

*Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 67;
cvc-elt.1: Cannot find the declaration of element 'bean'.*

* at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)*

* at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:134)*

* at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:396)*

* at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)*

* at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:284)*

* at
com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:1901)*

* at
com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(XMLSchemaValidator.java:741)*

* at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:374)*

* at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl$NSContentDriver.scanRootElementHook(XMLNSDocumentScannerImpl.java:613)*

* at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3132)*

* at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:852)*

* at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)*

* at

Re: Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

also can you send me full config file? so I dont have any typos etc



















 IP1

 IP2 

 IP3 

















For Question 2

In this approach, you don’t need to give the list of IP’s
in ignite config file, instead you just have to create an s3 bucket and
mention the s3 bucket name and access key in the ignite config. See below.



































On Wed, Oct 16, 2019 at 9:18 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi ,
>
> someone sent me this I will give a try.
>
>
> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2
>
> Thanks
> Sri
>
> On Wed, Oct 16, 2019 at 12:26 AM Muhammed Favas <
> favas.muham...@expeedsoftware.com> wrote:
>
>> Hi Sri,
>>
>>
>>
>> For Question 1
>>
>> In the ignite config file you use to launch ignite, you
>> can give list of IP’s like below. The same config file should be placed in
>> all the Ec2 servers. Once it is done, you can start ignite by using command
>> $IGNITE_HOME/bin.ignite.sh.
>>
>>
>>
>>
>> 
>>
>> 
>>
>>
>>
>> > "org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>
>>
>>
>> 
>>
>> > "org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>
>> 
>>
>> 
>>
>>  IP1
>>
>>  IP2 
>>
>>  IP3 
>>
>> 
>>
>> 
>>
>> 
>>
>>
>>
>> 
>>
>> 
>>
>> 
>>
>>
>>
>> For Question 2
>>
>> In this approach, you don’t need to give the list of IP’s
>> in ignite config file, instead you just have to create an s3 bucket and
>> mention the s3 bucket name and access key in the ignite config. See below.
>>
>>
>>
>> 
>>
>> 
>>
>> > >
>>
>> 
>>
>> > "org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
>>
>> > />
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>>     
>>
>> 
>>
>> 
>>
>>
>>
>> Hope it will help you
>>
>>
>>
>> *Regards,*
>>
>> *Favas  *
>>
>>
>>
>> *From:* Denis Magda 
>> *Sent:* Tuesday, October 15, 2019 11:21 PM
>> *To:* user@ignite.apache.org
>> *Subject:* Re: Does any one have working Ignite cluster on AWS
>>
>>
>>
>> Refer to GridGain documentation:
>> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2
>> <https://www.gridgain.com/docs/8.7.6/installation-guide/manual-install-on-ec2>
>>
>>
>>
>> Just swap GridGain with Ignite artifacts, the rest is identical.
>>
>>
>> -
>>
>> Denis
>>
>>
>>
>>
>>
>> On Tue, Oct 15, 2019 at 8:23 AM sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>> documentation is not covering step by step it's not that helpful.
>>
>>
>>
>> On Tue, Oct 15, 2019 at 10:35 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>> Hello!
>>
>>
>>
>> Please refer to the docs:
>>
>>
>>
>>
>> https://apacheignite.readme.io/docs/tcpip-discovery#section-static-ip-finder
>>
>>
>>
>> Regards,
>>
>> --
>>
>> Ilya Kasnac

Re: Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

Hi ,

someone sent me this I will give a try.

https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2

Thanks
Sri

On Wed, Oct 16, 2019 at 12:26 AM Muhammed Favas <
favas.muham...@expeedsoftware.com> wrote:

> Hi Sri,
>
>
>
> For Question 1
>
> In the ignite config file you use to launch ignite, you
> can give list of IP’s like below. The same config file should be placed in
> all the Ec2 servers. Once it is done, you can start ignite by using command
> $IGNITE_HOME/bin.ignite.sh.
>
>
>
>
> 
>
> 
>
>
>
>  "org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>
>
>
> 
>
>  "org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>
> 
>
> 
>
>  IP1
>
>  IP2 
>
>  IP3 
>
> 
>
> 
>
> 
>
>
>
> 
>
> 
>
> 
>
>
>
> For Question 2
>
> In this approach, you don’t need to give the list of IP’s
> in ignite config file, instead you just have to create an s3 bucket and
> mention the s3 bucket name and access key in the ignite config. See below.
>
>
>
> 
>
> 
>
> 
>
> 
>
>  "org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
>
>  />
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
>
>
> Hope it will help you
>
>
>
> *Regards,*
>
> *Favas  *
>
>
>
> *From:* Denis Magda 
> *Sent:* Tuesday, October 15, 2019 11:21 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: Does any one have working Ignite cluster on AWS
>
>
>
> Refer to GridGain documentation:
> https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2
> <https://www.gridgain.com/docs/8.7.6/installation-guide/manual-install-on-ec2>
>
>
>
> Just swap GridGain with Ignite artifacts, the rest is identical.
>
>
> -
>
> Denis
>
>
>
>
>
> On Tue, Oct 15, 2019 at 8:23 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> documentation is not covering step by step it's not that helpful.
>
>
>
> On Tue, Oct 15, 2019 at 10:35 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
> Hello!
>
>
>
> Please refer to the docs:
>
>
>
>
> https://apacheignite.readme.io/docs/tcpip-discovery#section-static-ip-finder
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> пн, 30 сент. 2019 г. в 17:57, sri hari kali charan Tummala <
> kali.tumm...@gmail.com>:
>
> thanks for replying but sorry not getting it please dumb it down, below
> are my questions.
>
>
>
> Statically giving the public IP list in the IP configuration section in
> ignite config file.
>
> Question:- wherein the config file should I give the IP address, imagine I
> launch 3 ec2 instances with public IP should I create a file in S3 bucket
> with file containing IP address of ec2 instance?
>
>
>
> Use s3 bucket to configure the IP, and it will automatically discover the
> related nodes from s3 bucket.
>
> Question:- ok I will give the s3 bucket name does the bucket need to have
> a file containing IP address or just empty bucket?
>
>
>
> T
> <https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json>
> hanks
>
> Sri
>
>
>
> On Mon, Sep 30, 2019 at 1:28 AM Muhammed Favas <
> favas.muham...@expeedsoftware.com> wrote:
>
> Hi Sri,
>
>
>
> If your nodes created in EC2, it is very simple to start the ignite
> cluster. You have two option to configure the IP for all nodes auto
> discover the IP.
>
>1. Statically giving the public IP list in the IP configuration
>section in ignite config file.
>2. Use s3 bucket to configure the IP, and it will automatically
>discover the related nodes from s3 bucket.
>
>
>
> I am using the 2nd method, and below is the configuration I have given
> for my cluster
>
>
>
> 
>

Re: Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

documentation is not covering step by step it's not that helpful.

On Tue, Oct 15, 2019 at 10:35 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> Please refer to the docs:
>
>
> https://apacheignite.readme.io/docs/tcpip-discovery#section-static-ip-finder
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 30 сент. 2019 г. в 17:57, sri hari kali charan Tummala <
> kali.tumm...@gmail.com>:
>
>> thanks for replying but sorry not getting it please dumb it down, below
>> are my questions.
>>
>> Statically giving the public IP list in the IP configuration section in
>> ignite config file.
>> Question:- wherein the config file should I give the IP address, imagine
>> I launch 3 ec2 instances with public IP should I create a file in S3 bucket
>> with file containing IP address of ec2 instance?
>>
>> Use s3 bucket to configure the IP, and it will automatically discover the
>> related nodes from s3 bucket.
>> Question:- ok I will give the s3 bucket name does the bucket need to have
>> a file containing IP address or just empty bucket?
>>
>> T
>> <https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json>
>> hanks
>> Sri
>>
>> On Mon, Sep 30, 2019 at 1:28 AM Muhammed Favas <
>> favas.muham...@expeedsoftware.com> wrote:
>>
>>> Hi Sri,
>>>
>>>
>>>
>>> If your nodes created in EC2, it is very simple to start the ignite
>>> cluster. You have two option to configure the IP for all nodes auto
>>> discover the IP.
>>>
>>>1. Statically giving the public IP list in the IP configuration
>>>section in ignite config file.
>>>2. Use s3 bucket to configure the IP, and it will automatically
>>>discover the related nodes from s3 bucket.
>>>
>>>
>>>
>>> I am using the 2nd method, and below is the configuration I have given
>>> for my cluster
>>>
>>>
>>>
>>> 
>>>
>>> 
>>>
>>> >> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>>
>>>         
>>>
>>> >> class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>>
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>> 
>>>
>>>
>>>
>>>
>>>
>>> *Regards,*
>>>
>>> *Favas  *
>>>
>>>
>>>
>>> *From:* sri hari kali charan Tummala 
>>> *Sent:* Friday, September 27, 2019 10:02 PM
>>> *To:* user@ignite.apache.org
>>> *Subject:* Does any one have working Ignite cluster on AWS
>>>
>>>
>>>
>>> Hi All,
>>>
>>>
>>>
>>> can someone help me run a working ignite cuter on AWS , did anyone able
>>> to figure out steps required for setting up working ignite cluster on AWS.
>>>
>>>
>>>
>>> Ignite documentation doesnt make sense it's just launching a docker
>>> instance on Ec2.
>>>
>>>
>>>
>>> --
>>>
>>> Thanks & Regards
>>>
>>> Sri Tummala
>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>

-- 
Thanks & Regards
Sri Tummala

Re: Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

thanks for replying but sorry not getting it please dumb it down, below are
my questions.

Statically giving the public IP list in the IP configuration section in
ignite config file.
Question:- wherein the config file should I give the IP address, imagine I
launch 3 ec2 instances with public IP should I create a file in S3 bucket
with file containing IP address of ec2 instance?

Use s3 bucket to configure the IP, and it will automatically discover the
related nodes from s3 bucket.
Question:- ok I will give the s3 bucket name does the bucket need to have a
file containing IP address or just empty bucket?

T
<https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json>
hanks
Sri

On Mon, Sep 30, 2019 at 1:28 AM Muhammed Favas <
favas.muham...@expeedsoftware.com> wrote:

> Hi Sri,
>
>
>
> If your nodes created in EC2, it is very simple to start the ignite
> cluster. You have two option to configure the IP for all nodes auto
> discover the IP.
>
>1. Statically giving the public IP list in the IP configuration
>section in ignite config file.
>2. Use s3 bucket to configure the IP, and it will automatically
>discover the related nodes from s3 bucket.
>
>
>
> I am using the 2nd method, and below is the configuration I have given
> for my cluster
>
>
>
> 
>
> 
>
> 
>
> 
>
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
>
> 
>
> 
>
> 
>
> 
>
> 
>
>            
>
> 
>
> 
>
> 
>
> 
>
> 
>
>
>
>
>
> *Regards,*
>
> *Favas  *
>
>
>
> *From:* sri hari kali charan Tummala 
> *Sent:* Friday, September 27, 2019 10:02 PM
> *To:* user@ignite.apache.org
> *Subject:* Does any one have working Ignite cluster on AWS
>
>
>
> Hi All,
>
>
>
> can someone help me run a working ignite cuter on AWS , did anyone able to
> figure out steps required for setting up working ignite cluster on AWS.
>
>
>
> Ignite documentation doesnt make sense it's just launching a docker
> instance on Ec2.
>
>
>
> --
>
> Thanks & Regards
>
> Sri Tummala
>
>
>


-- 
Thanks & Regards
Sri Tummala

Does any one have working Ignite cluster on AWS

[sri hari kali charan Tummala]

Hi All,

can someone help me run a working ignite cuter on AWS , did anyone able to
figure out steps required for setting up working ignite cluster on AWS.

Ignite documentation doesnt make sense it's just launching a docker
instance on Ec2.

-- 
Thanks & Regards
Sri Tummala

Re: Apache Ignite Cloud Formation Template

[sri hari kali charan Tummala]

I found one example of cloud formation after launching the template and
started to execute ignite.sh it is failing with class not found exception
AWSCredentialsProviderWrapper.

I sued:-
Ignitefabric.1.5.0
aws java sdk 1.3.21.1


https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json

ConfireIgnite.sh:-

#!/bin/bash

#
# This is a modified version of the file stored at
s3://publicbucketbabupe/ignitelibrary/configureIgnite.sh
# which changes the config to use the instance provided credentials
rather than requiring access/secret to be passed in
#
# Parameters are
# 1 - Cache Name
# 2 - Number of replicas
# 3 - S3 Bucket Name
#

echo "



http://www.springframework.org/schema/beans\;
   xmlns:util=\"http://www.springframework.org/schema/util\;
   xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\;
   xsi:schemaLocation=\"
   http://www.springframework.org/schema/beans
   http://www.springframework.org/schema/beans/spring-beans.xsd
   http://www.springframework.org/schema/util
   http://www.springframework.org/schema/util/spring-util.xsd\;>

  

  

  " > /tmp/igniteconfig.xml

echo "
  
  " >> /tmp/igniteconfig.xml

availfreeMemory=$(cat /proc/meminfo|grep MemTotal|awk '{print $2}')
memoryOverhead=$((availfreeMemory/1024/1024/10))
availfreeMemoryinGB=$((availfreeMemory/1024/1024 - memoryOverhead))
if [[ $availfreeMemoryinGB -gt 8 ]]; then
offheapmemoryinGB=$((availfreeMemoryinGB-8))
echo "
  " >> /tmp/igniteconfig.xml
echo "8g" > /tmp/heapsize.log
else
echo "${availfreeMemoryinGB}g" > /tmp/heapsize.log
fi
echo "



  " >> /tmp/igniteconfig.xml
echo "" >>
/tmp/igniteconfig.xml
echo "" >>
/tmp/igniteconfig.xml
echo "" >> /tmp/igniteconfig.xml
echo "  " >> /tmp/igniteconfig.xml
echo " " >> /tmp/igniteconfig.xml
echo "" >> /tmp/igniteconfig.xml
echo "
    
  
    
  
  
 
  




  

  
  
  
  
  
  
  

" >> /tmp/igniteconfig.xml


On Thu, Sep 26, 2019 at 5:34 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> looks complex and hard to get it right, it would be nice if Ignite
> community builds one working cloud formation template to launch ignite
> cluster and in the cloud formation template output a JDBC connection for
> connecting ignite cluster which would be awesome.
>
> Many would leverage this stack to build Ignite cluster on AWS or how about
> ignite on EMR , while launching emr add ingite bootstrap script which
> launches ignite in emr which would be lot easier also JDBC connection for
> connecting ignite, this would change things for Ignite world.
>
> Thanks
> Sri
>
>
>
> On Thu, Sep 26, 2019 at 3:51 PM Denis Magda  wrote:
>
>> Hello Sri,
>>
>> You’re right, you need to start several aws instances to create a
>> distributed cluster if you follow the guide below. Ignite configuration for
>> all the nodes/instances has to have a properly configured IPFinder. You can
>> use a static IP finder if IPs are know in advance or AWS IPfinder if IPs
>> are assigned dynamically.
>> https://apacheignite.readme.io/docs/tcpip-discovery
>>
>> Let us know if it helps and we’ll update the docs.
>>
>> Denis
>>
>> On Thursday, September 26, 2019, sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>>> Hi Community,
>>>
>>> this doesn't launch a multinode ignite cluster am I right? it just
>>> launches a single node ignite cluster on one ec2 instance.
>>>
>>> https://apacheignite.readme.io/docs/aws-deployment
>>>
>>> Thanks
>>> Sri
>>>
>>> On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
>>> kali.tumm...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> did anyone have an cloud formation example for deploying Apache Ignite
>>>> on AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>>>>
>>>> this below blog has missing artifacts so not working.
>>>>
>>>>
>>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Sri Tummala
>>>>
>>>>
>>>
>>> --
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
>>
>> --
>> -
>> Denis
>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>

-- 
Thanks & Regards
Sri Tummala

Re: Apache Ignite Cloud Formation Template

[sri hari kali charan Tummala]

looks complex and hard to get it right, it would be nice if Ignite
community builds one working cloud formation template to launch ignite
cluster and in the cloud formation template output a JDBC connection for
connecting ignite cluster which would be awesome.

Many would leverage this stack to build Ignite cluster on AWS or how about
ignite on EMR , while launching emr add ingite bootstrap script which
launches ignite in emr which would be lot easier also JDBC connection for
connecting ignite, this would change things for Ignite world.

Thanks
Sri



On Thu, Sep 26, 2019 at 3:51 PM Denis Magda  wrote:

> Hello Sri,
>
> You’re right, you need to start several aws instances to create a
> distributed cluster if you follow the guide below. Ignite configuration for
> all the nodes/instances has to have a properly configured IPFinder. You can
> use a static IP finder if IPs are know in advance or AWS IPfinder if IPs
> are assigned dynamically.
> https://apacheignite.readme.io/docs/tcpip-discovery
>
> Let us know if it helps and we’ll update the docs.
>
> Denis
>
> On Thursday, September 26, 2019, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi Community,
>>
>> this doesn't launch a multinode ignite cluster am I right? it just
>> launches a single node ignite cluster on one ec2 instance.
>>
>> https://apacheignite.readme.io/docs/aws-deployment
>>
>> Thanks
>> Sri
>>
>> On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> did anyone have an cloud formation example for deploying Apache Ignite
>>> on AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>>>
>>> this below blog has missing artifacts so not working.
>>>
>>>
>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>
>>> --
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>
>
> --
> -
> Denis
>
>

-- 
Thanks & Regards
Sri Tummala

Re: Apache Ignite Cloud Formation Template

[sri hari kali charan Tummala]

Hi Community,

this doesn't launch a multinode ignite cluster am I right? it just launches
a single node ignite cluster on one ec2 instance.

https://apacheignite.readme.io/docs/aws-deployment

Thanks
Sri

On Thu, Sep 26, 2019 at 8:49 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Hi All,
>
> did anyone have an cloud formation example for deploying Apache Ignite on
> AWS or steps to deploy or some successful deployment of Ignite onto AWS.
>
> this below blog has missing artifacts so not working.
>
>
> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>
> --
> Thanks & Regards
> Sri Tummala
>
>

-- 
Thanks & Regards
Sri Tummala

Apache Ignite Cloud Formation Template

[sri hari kali charan Tummala]

Hi All,

did anyone have an cloud formation example for deploying Apache Ignite on
AWS or steps to deploy or some successful deployment of Ignite onto AWS.

this below blog has missing artifacts so not working.

https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/

-- 
Thanks & Regards
Sri Tummala

Re: Professional Ignite Course

[sri hari kali charan Tummala]

I did went trough the course and I asked pluralsight to add more, in the
next course I would like to see ignite installation on aws and spark +
ignite integration.
Real time analytics with spark + ignite.


On Wednesday, September 25, 2019, Denis Magda  wrote:

> Igniters,
>
> Recently, I've come across a professionally crafted course about Apache
> Ignite. It will be interesting for some of you:
> https://www.pluralsight.com/courses/apache-ignite-getting-started
>
> Edward, thanks for the course! I've added it to Ignite Doc's menu right
> below the book.
>
> -
> Denis
>


-- 
Thanks & Regards
Sri Tummala

Re: Ignite Spark Example Question

[sri hari kali charan Tummala]

can I run ignite and spark on cluster mode ? in the github example what I
see is just local mode, if I use grid cloud ignite cluster how would I
install spark distributed mode is it comes with the ignite cluster ?

https://github.com/apache/ignite/blob/1f8cf042f67f523e23f795571f609a9c81726258/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameWriteExample.scala#L89

On Tue, Aug 13, 2019 at 6:53 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> As I say, there’s nothing "out of the box” — you’d have to write it
> yourself. Exactly how you architect it would depend on what you’re trying
> to do.
>
> Regards,
> Stephen
>
> On 12 Aug 2019, at 19:59, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> Thanks Stephen , last question so I have to keep looping to find new data
> files in S3 and write to cache real time or is it already built in ?
>
> On Mon, Aug 12, 2019 at 5:43 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> I don’t think there’s anything “out of the box,” but you could write a
>> custom CacheStore to do that.
>>
>> See here for more details:
>> https://apacheignite.readme.io/docs/3rd-party-store#section-custom-cachestore
>>
>> Regards,
>> Stephen
>>
>> On 9 Aug 2019, at 21:50, sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>> one last question, is there an S3 connector for Ignite which can load s3
>> objects in realtime to ignite cache and data updates directly back to S3? I
>> can use spark as one alternative but is there another approach of doing?
>>
>> Let's say I want to build in-memory near real-time data lake files which
>> get loaded to S3 automatically gets loaded to Ignite (I can use spark
>> structured streaming jobs but is there a direct approach ?)
>>
>> On Fri, Aug 9, 2019 at 4:34 PM sri hari kali charan Tummala <
>> kali.tumm...@gmail.com> wrote:
>>
>>> Thank you, I got it now I have to change the id values to see the same
>>> data as extra results (this is just for testing) amazing.
>>>
>>> val df = spark.sql(SELECT monolitically_id() as id, name, department
>>> FROM json_person)
>>>
>>> df.write(append)... to ignite
>>>
>>> Thanks
>>> Sri
>>>
>>>
>>> On Fri, Aug 9, 2019 at 6:08 AM Andrei Aleksandrov <
>>> aealexsand...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Spark contains several *SaveModes *that will be applied if the table
>>>> that you are going to use exists:
>>>>
>>>> * *Overwrite *- with this option you *will try to re-create* existed
>>>> table or create new and load data there using IgniteDataStreamer
>>>> implementation
>>>> * *Append *- with this option you *will not try to re-create* existed
>>>> table or create new table and just load the data to existed table
>>>>
>>>> * *ErrorIfExists *- with this option you will get the exception if the
>>>> table that you are going to use exists
>>>>
>>>> * *Ignore *- with this option nothing will be done in case if the
>>>> table that you are going to use exists. If table already exists, the save
>>>> operation is expected to not save the contents of the DataFrame and to not
>>>> change the existing data.
>>>> According to your question:
>>>>
>>>> You should use the *Append *SaveMode for your spark integration in
>>>> case if you are going to store new data to cache and save the previous
>>>> stored data.
>>>>
>>>> Note, that in case if you will store the data for the same Primary Keys
>>>> then with data will be overwritten in Ignite table. For example:
>>>>
>>>> 1)Add person {id=1, name=Vlad, age=19} where id is the primary key
>>>> 2)Add person {id=1, name=Nikita, age=26} where id is the primary key
>>>>
>>>> In Ignite you will see only {id=1, name=Nikita, age=26}.
>>>>
>>>> Also here you can see the code sample for you and other information
>>>> about SaveModes:
>>>>
>>>>
>>>> https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-saving-dataframes
>>>>
>>>> BR,
>>>> Andrei
>>>>
>>>> On 2019/08/08 17:33:39, sri hari kali charan Tummala 
>>>>  wrote:
>>>> > Hi All,>
>>>> >
>>>> > I am new t

Re: Ignite Spark Example Question

[sri hari kali charan Tummala]

Thanks Stephen , last question so I have to keep looping to find new data
files in S3 and write to cache real time or is it already built in ?

On Mon, Aug 12, 2019 at 5:43 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> I don’t think there’s anything “out of the box,” but you could write a
> custom CacheStore to do that.
>
> See here for more details:
> https://apacheignite.readme.io/docs/3rd-party-store#section-custom-cachestore
>
> Regards,
> Stephen
>
> On 9 Aug 2019, at 21:50, sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
> one last question, is there an S3 connector for Ignite which can load s3
> objects in realtime to ignite cache and data updates directly back to S3? I
> can use spark as one alternative but is there another approach of doing?
>
> Let's say I want to build in-memory near real-time data lake files which
> get loaded to S3 automatically gets loaded to Ignite (I can use spark
> structured streaming jobs but is there a direct approach ?)
>
> On Fri, Aug 9, 2019 at 4:34 PM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Thank you, I got it now I have to change the id values to see the same
>> data as extra results (this is just for testing) amazing.
>>
>> val df = spark.sql(SELECT monolitically_id() as id, name, department FROM
>> json_person)
>>
>> df.write(append)... to ignite
>>
>> Thanks
>> Sri
>>
>>
>> On Fri, Aug 9, 2019 at 6:08 AM Andrei Aleksandrov <
>> aealexsand...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Spark contains several *SaveModes *that will be applied if the table
>>> that you are going to use exists:
>>>
>>> * *Overwrite *- with this option you *will try to re-create* existed
>>> table or create new and load data there using IgniteDataStreamer
>>> implementation
>>> * *Append *- with this option you *will not try to re-create* existed
>>> table or create new table and just load the data to existed table
>>>
>>> * *ErrorIfExists *- with this option you will get the exception if the
>>> table that you are going to use exists
>>>
>>> * *Ignore *- with this option nothing will be done in case if the table
>>> that you are going to use exists. If table already exists, the save
>>> operation is expected to not save the contents of the DataFrame and to not
>>> change the existing data.
>>> According to your question:
>>>
>>> You should use the *Append *SaveMode for your spark integration in case
>>> if you are going to store new data to cache and save the previous stored
>>> data.
>>>
>>> Note, that in case if you will store the data for the same Primary Keys
>>> then with data will be overwritten in Ignite table. For example:
>>>
>>> 1)Add person {id=1, name=Vlad, age=19} where id is the primary key
>>> 2)Add person {id=1, name=Nikita, age=26} where id is the primary key
>>>
>>> In Ignite you will see only {id=1, name=Nikita, age=26}.
>>>
>>> Also here you can see the code sample for you and other information
>>> about SaveModes:
>>>
>>>
>>> https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-saving-dataframes
>>>
>>> BR,
>>> Andrei
>>>
>>> On 2019/08/08 17:33:39, sri hari kali charan Tummala 
>>>  wrote:
>>> > Hi All,>
>>> >
>>> > I am new to Apache Ignite community I am testing out ignite for
>>> knowledge>
>>> > sake in the below example the code reads a json file and writes to
>>> ingite>
>>> > in-memory table is it overwriting can I do append mode I did try
>>> spark>
>>> > append mode .mode(org.apache.spark.sql.SaveMode.Append)>
>>> > without stopping one ignite application inginte.stop which keeps the
>>> cache>
>>> > alive and tried to insert data to cache twice but I am still getting
>>> 4>
>>> > records I was expecting 8 records , what would be the reason ?>
>>> >
>>> >
>>> https://github.com/apache/ignite/blob/1f8cf042f67f523e23f795571f609a9c81726258/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameWriteExample.scala#L89>
>>>
>>> >
>>> > -- >
>>> > Thanks & Regards>
>>> > Sri Tummala>
>>> >
>>>
>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
>
>

-- 
Thanks & Regards
Sri Tummala

Re: Ignite Spark Example Question

[sri hari kali charan Tummala]

one last question, is there an S3 connector for Ignite which can load s3
objects in realtime to ignite cache and data updates directly back to S3? I
can use spark as one alternative but is there another approach of doing?

Let's say I want to build in-memory near real-time data lake files which
get loaded to S3 automatically gets loaded to Ignite (I can use spark
structured streaming jobs but is there a direct approach ?)

On Fri, Aug 9, 2019 at 4:34 PM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> Thank you, I got it now I have to change the id values to see the same
> data as extra results (this is just for testing) amazing.
>
> val df = spark.sql(SELECT monolitically_id() as id, name, department FROM
> json_person)
>
> df.write(append)... to ignite
>
> Thanks
> Sri
>
>
> On Fri, Aug 9, 2019 at 6:08 AM Andrei Aleksandrov 
> wrote:
>
>> Hi,
>>
>> Spark contains several *SaveModes *that will be applied if the table
>> that you are going to use exists:
>>
>> * *Overwrite *- with this option you *will try to re-create* existed
>> table or create new and load data there using IgniteDataStreamer
>> implementation
>> * *Append *- with this option you *will not try to re-create* existed
>> table or create new table and just load the data to existed table
>>
>> * *ErrorIfExists *- with this option you will get the exception if the
>> table that you are going to use exists
>>
>> * *Ignore *- with this option nothing will be done in case if the table
>> that you are going to use exists. If table already exists, the save
>> operation is expected to not save the contents of the DataFrame and to not
>> change the existing data.
>> According to your question:
>>
>> You should use the *Append *SaveMode for your spark integration in case
>> if you are going to store new data to cache and save the previous stored
>> data.
>>
>> Note, that in case if you will store the data for the same Primary Keys
>> then with data will be overwritten in Ignite table. For example:
>>
>> 1)Add person {id=1, name=Vlad, age=19} where id is the primary key
>> 2)Add person {id=1, name=Nikita, age=26} where id is the primary key
>>
>> In Ignite you will see only {id=1, name=Nikita, age=26}.
>>
>> Also here you can see the code sample for you and other information about
>> SaveModes:
>>
>>
>> https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-saving-dataframes
>>
>> BR,
>> Andrei
>>
>> On 2019/08/08 17:33:39, sri hari kali charan Tummala 
>>  wrote:
>> > Hi All,>
>> >
>> > I am new to Apache Ignite community I am testing out ignite for
>> knowledge>
>> > sake in the below example the code reads a json file and writes to
>> ingite>
>> > in-memory table is it overwriting can I do append mode I did try spark>
>> > append mode .mode(org.apache.spark.sql.SaveMode.Append)>
>> > without stopping one ignite application inginte.stop which keeps the
>> cache>
>> > alive and tried to insert data to cache twice but I am still getting 4>
>> > records I was expecting 8 records , what would be the reason ?>
>> >
>> >
>> https://github.com/apache/ignite/blob/1f8cf042f67f523e23f795571f609a9c81726258/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameWriteExample.scala#L89>
>>
>> >
>> > -- >
>> > Thanks & Regards>
>> > Sri Tummala>
>> >
>>
>
>
> --
> Thanks & Regards
> Sri Tummala
>
>

-- 
Thanks & Regards
Sri Tummala

Re: Unable to access Ignite Ec2 Instance from Local PC

[sri hari kali charan Tummala]

the Blog has issue's the code which is in public s3 bucket doesn't exist so
the ec2 instance doesn't have to ignite at all I raised a concern with AWS
Blog team, I will try community ami Ignite ec2 instance or try to install
ignite on my own manually.

On Fri, Aug 9, 2019 at 11:20 AM sri hari kali charan Tummala <
kali.tumm...@gmail.com> wrote:

> please check.
>
> On Fri, Aug 9, 2019 at 10:58 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> give me couple of minutes I think I opened 0 to 65000 ports for inbound
>> thats it , if I have to open all these ports separately I have to give a
>> try.
>>
>> On Fri, Aug 9, 2019 at 10:55 AM Himanshu Gupta 
>> wrote:
>>
>>> Can you share screenshot of your inbound traffic setting, I think it
>>> need to be something like this
>>>
>>> Example:
>>>
>>> Inbound:
>>> Custom TCP Rule TCP 10800 - 10900 0.0.0.0/0 client
>>> Custom TCP Rule TCP 47500 - 47600 0.0.0.0/0 discovery
>>> Custom TCP Rule TCP 47100 - 47200 0.0.0.0/0 communication
>>>
>>> Outbound:
>>> All traffic All All 0.0.0.0/0
>>> All traffic All All ::/0
>>>
>>> as mentioned in
>>> http://apache-ignite-users.70518.x6.nabble.com/AWS-Cluster-td23903.html
>>>
>>> On Fri, Aug 9, 2019 at 9:52 AM sri hari kali charan Tummala <
>>> kali.tumm...@gmail.com> wrote:
>>>
>>>> yes, all traffic inbound and outbound (TCP), I am able to do ssh also I
>>>> am able to ping public ip address from my laptop and the Ec2 instance is
>>>> able to ping the www.amazon.com.
>>>>
>>>>
>>>>
>>>> On Fri, Aug 9, 2019 at 10:46 AM Himanshu Gupta 
>>>> wrote:
>>>>
>>>>> Did you open the ports on EC2 at which your ignite cluster is running?
>>>>>
>>>>> On Fri, Aug 9, 2019 at 8:50 AM sri hari kali charan Tummala <
>>>>> kali.tumm...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> thanks for replying I am following the below blog as per some expert
>>>>>> suggestion he asked me to use my laptop ip address and elastic ip address
>>>>>> of Ec2 instance (
>>>>>> https://github.com/aws-samples/aws-big-data-blog/blob/9eb7cc30a2dfb02c15252a5bad03603a21451572/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/src/main/java/com/amazon/dynamostreams/clientlibrary/StreamsToIgnite.properties#L4
>>>>>> )
>>>>>>
>>>>>>
>>>>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>>>>
>>>>>> Correct me if wrong I will give a try.
>>>>>>
>>>>>> Thanks
>>>>>> Sri
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 9, 2019 at 8:27 AM Ilya Kasnacheev <
>>>>>> ilya.kasnach...@gmail.com> wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> In my experience, on EC2 nodes usually use internal network to talk
>>>>>>> to each other. They will advertise these to any nodes trying to connect,
>>>>>>> which will cause issues.
>>>>>>>
>>>>>>> Is it an option for you to use some flavor of Thin Client to talk to
>>>>>>> your cluster? We do not recommend having heterogenous, non-collocated
>>>>>>> clusters, and this includes thick client nodes.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> пт, 9 авг. 2019 г. в 03:35, sri hari kali charan Tummala <
>>>>>>> kali.tumm...@gmail.com>:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I am trying to connect a ignite ec2 instance (one node) remotely
>>>>>>>> from local PC Mac Intellij Idea I am using TcpDicoverySpi to connect 
>>>>>>>> ignite
>>>>>>>> ec2 instance public IP but its failing to connect I did open all the 
>>>>>>>> ports
>>>>>>>> I was able to do ssh and ping the host from my local PC even on the 
>>>>&g

Re: Unable to access Ignite Ec2 Instance from Local PC

[sri hari kali charan Tummala]

give me couple of minutes I think I opened 0 to 65000 ports for inbound
thats it , if I have to open all these ports separately I have to give a
try.

On Fri, Aug 9, 2019 at 10:55 AM Himanshu Gupta 
wrote:

> Can you share screenshot of your inbound traffic setting, I think it need
> to be something like this
>
> Example:
>
> Inbound:
> Custom TCP Rule TCP 10800 - 10900 0.0.0.0/0 client
> Custom TCP Rule TCP 47500 - 47600 0.0.0.0/0 discovery
> Custom TCP Rule TCP 47100 - 47200 0.0.0.0/0 communication
>
> Outbound:
> All traffic All All 0.0.0.0/0
> All traffic All All ::/0
>
> as mentioned in
> http://apache-ignite-users.70518.x6.nabble.com/AWS-Cluster-td23903.html
>
> On Fri, Aug 9, 2019 at 9:52 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> yes, all traffic inbound and outbound (TCP), I am able to do ssh also I
>> am able to ping public ip address from my laptop and the Ec2 instance is
>> able to ping the www.amazon.com.
>>
>>
>>
>> On Fri, Aug 9, 2019 at 10:46 AM Himanshu Gupta 
>> wrote:
>>
>>> Did you open the ports on EC2 at which your ignite cluster is running?
>>>
>>> On Fri, Aug 9, 2019 at 8:50 AM sri hari kali charan Tummala <
>>> kali.tumm...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> thanks for replying I am following the below blog as per some expert
>>>> suggestion he asked me to use my laptop ip address and elastic ip address
>>>> of Ec2 instance (
>>>> https://github.com/aws-samples/aws-big-data-blog/blob/9eb7cc30a2dfb02c15252a5bad03603a21451572/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/src/main/java/com/amazon/dynamostreams/clientlibrary/StreamsToIgnite.properties#L4
>>>> )
>>>>
>>>>
>>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>>
>>>> Correct me if wrong I will give a try.
>>>>
>>>> Thanks
>>>> Sri
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Aug 9, 2019 at 8:27 AM Ilya Kasnacheev <
>>>> ilya.kasnach...@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> In my experience, on EC2 nodes usually use internal network to talk to
>>>>> each other. They will advertise these to any nodes trying to connect, 
>>>>> which
>>>>> will cause issues.
>>>>>
>>>>> Is it an option for you to use some flavor of Thin Client to talk to
>>>>> your cluster? We do not recommend having heterogenous, non-collocated
>>>>> clusters, and this includes thick client nodes.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> пт, 9 авг. 2019 г. в 03:35, sri hari kali charan Tummala <
>>>>> kali.tumm...@gmail.com>:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I am trying to connect a ignite ec2 instance (one node) remotely from
>>>>>> local PC Mac Intellij Idea I am using TcpDicoverySpi to connect ignite 
>>>>>> ec2
>>>>>> instance public IP but its failing to connect I did open all the ports I
>>>>>> was able to do ssh and ping the host from my local PC even on the remote
>>>>>> host I am able to access internet, ignite right now fails with below 
>>>>>> error.
>>>>>>
>>>>>> [20:14:54] Failed to connect to any address from IP finder (will
>>>>>> retry to join topology every 2000 ms; change 'reconnectDelay' to 
>>>>>> configure
>>>>>> the frequency of retries): [/52.204.222.254:47500, /
>>>>>> 52.204.222.254:47501, /52.204.222.254:47502, /52.204.222.254:47503, /
>>>>>> 52.204.222.254:47504, /52.204.222.254:47505, /52.204.222.254:47506, /
>>>>>> 52.204.222.254:47507, /52.204.222.254:47508, /52.204.222.254:47509, /
>>>>>> 52.204.222.254:47510]
>>>>>>
>>>>>> I am following below blog , can any one help how to access ignite
>>>>>> cluster which is setup remotely do I need to build my code and run in the
>>>>>> cluster by ssh to the box ?
>>>>>>
>>>>>>
>>>>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>>>>
>>>>>> 4.222.254:47507, /52.204.222.254:47508, /52.204.222.254:47509, /
>>>>>> 52.204.222.254:47510]
>>>>>>
>>>>>> [image: Thumbnail]
>>>>>> <https://uploads.disquscdn.com/images/23e365052ba57ed2be6399b06c036ada661bc92584046760584c85afefe4a4e4.png>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards
>>>>>> Sri Tummala
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Sri Tummala
>>>>
>>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>

-- 
Thanks & Regards
Sri Tummala

Re: Unable to access Ignite Ec2 Instance from Local PC

[sri hari kali charan Tummala]

yes, all traffic inbound and outbound (TCP), I am able to do ssh also I am
able to ping public ip address from my laptop and the Ec2 instance is able
to ping the www.amazon.com.



On Fri, Aug 9, 2019 at 10:46 AM Himanshu Gupta 
wrote:

> Did you open the ports on EC2 at which your ignite cluster is running?
>
> On Fri, Aug 9, 2019 at 8:50 AM sri hari kali charan Tummala <
> kali.tumm...@gmail.com> wrote:
>
>> Hi,
>>
>> thanks for replying I am following the below blog as per some expert
>> suggestion he asked me to use my laptop ip address and elastic ip address
>> of Ec2 instance (
>> https://github.com/aws-samples/aws-big-data-blog/blob/9eb7cc30a2dfb02c15252a5bad03603a21451572/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/src/main/java/com/amazon/dynamostreams/clientlibrary/StreamsToIgnite.properties#L4
>> )
>>
>>
>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>
>> Correct me if wrong I will give a try.
>>
>> Thanks
>> Sri
>>
>>
>>
>>
>> On Fri, Aug 9, 2019 at 8:27 AM Ilya Kasnacheev 
>> wrote:
>>
>>> Hello!
>>>
>>> In my experience, on EC2 nodes usually use internal network to talk to
>>> each other. They will advertise these to any nodes trying to connect, which
>>> will cause issues.
>>>
>>> Is it an option for you to use some flavor of Thin Client to talk to
>>> your cluster? We do not recommend having heterogenous, non-collocated
>>> clusters, and this includes thick client nodes.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 9 авг. 2019 г. в 03:35, sri hari kali charan Tummala <
>>> kali.tumm...@gmail.com>:
>>>
>>>> Hi All,
>>>>
>>>> I am trying to connect a ignite ec2 instance (one node) remotely from
>>>> local PC Mac Intellij Idea I am using TcpDicoverySpi to connect ignite ec2
>>>> instance public IP but its failing to connect I did open all the ports I
>>>> was able to do ssh and ping the host from my local PC even on the remote
>>>> host I am able to access internet, ignite right now fails with below error.
>>>>
>>>> [20:14:54] Failed to connect to any address from IP finder (will retry
>>>> to join topology every 2000 ms; change 'reconnectDelay' to configure the
>>>> frequency of retries): [/52.204.222.254:47500, /52.204.222.254:47501, /
>>>> 52.204.222.254:47502, /52.204.222.254:47503, /52.204.222.254:47504, /
>>>> 52.204.222.254:47505, /52.204.222.254:47506, /52.204.222.254:47507, /
>>>> 52.204.222.254:47508, /52.204.222.254:47509, /52.204.222.254:47510]
>>>>
>>>> I am following below blog , can any one help how to access ignite
>>>> cluster which is setup remotely do I need to build my code and run in the
>>>> cluster by ssh to the box ?
>>>>
>>>>
>>>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>>>
>>>> 4.222.254:47507, /52.204.222.254:47508, /52.204.222.254:47509, /
>>>> 52.204.222.254:47510]
>>>>
>>>> [image: Thumbnail]
>>>> <https://uploads.disquscdn.com/images/23e365052ba57ed2be6399b06c036ada661bc92584046760584c85afefe4a4e4.png>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards
>>>> Sri Tummala
>>>>
>>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>

-- 
Thanks & Regards
Sri Tummala

Re: Unable to access Ignite Ec2 Instance from Local PC

[sri hari kali charan Tummala]

Hi,

thanks for replying I am following the below blog as per some expert
suggestion he asked me to use my laptop ip address and elastic ip address
of Ec2 instance (
https://github.com/aws-samples/aws-big-data-blog/blob/9eb7cc30a2dfb02c15252a5bad03603a21451572/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/src/main/java/com/amazon/dynamostreams/clientlibrary/StreamsToIgnite.properties#L4
)

https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/

Correct me if wrong I will give a try.

Thanks
Sri




On Fri, Aug 9, 2019 at 8:27 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> In my experience, on EC2 nodes usually use internal network to talk to
> each other. They will advertise these to any nodes trying to connect, which
> will cause issues.
>
> Is it an option for you to use some flavor of Thin Client to talk to your
> cluster? We do not recommend having heterogenous, non-collocated clusters,
> and this includes thick client nodes.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 9 авг. 2019 г. в 03:35, sri hari kali charan Tummala <
> kali.tumm...@gmail.com>:
>
>> Hi All,
>>
>> I am trying to connect a ignite ec2 instance (one node) remotely from
>> local PC Mac Intellij Idea I am using TcpDicoverySpi to connect ignite ec2
>> instance public IP but its failing to connect I did open all the ports I
>> was able to do ssh and ping the host from my local PC even on the remote
>> host I am able to access internet, ignite right now fails with below error.
>>
>> [20:14:54] Failed to connect to any address from IP finder (will retry to
>> join topology every 2000 ms; change 'reconnectDelay' to configure the
>> frequency of retries): [/52.204.222.254:47500, /52.204.222.254:47501, /
>> 52.204.222.254:47502, /52.204.222.254:47503, /52.204.222.254:47504, /
>> 52.204.222.254:47505, /52.204.222.254:47506, /52.204.222.254:47507, /
>> 52.204.222.254:47508, /52.204.222.254:47509, /52.204.222.254:47510]
>>
>> I am following below blog , can any one help how to access ignite cluster
>> which is setup remotely do I need to build my code and run in the cluster
>> by ssh to the box ?
>>
>>
>> https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/
>>
>> 4.222.254:47507, /52.204.222.254:47508, /52.204.222.254:47509, /
>> 52.204.222.254:47510]
>>
>> [image: Thumbnail]
>> <https://uploads.disquscdn.com/images/23e365052ba57ed2be6399b06c036ada661bc92584046760584c85afefe4a4e4.png>
>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>

-- 
Thanks & Regards
Sri Tummala

Unable to access Ignite Ec2 Instance from Local PC

[sri hari kali charan Tummala]

Hi All,

I am trying to connect a ignite ec2 instance (one node) remotely from local
PC Mac Intellij Idea I am using TcpDicoverySpi to connect ignite ec2
instance public IP but its failing to connect I did open all the ports I
was able to do ssh and ping the host from my local PC even on the remote
host I am able to access internet, ignite right now fails with below error.

[20:14:54] Failed to connect to any address from IP finder (will retry to
join topology every 2000 ms; change 'reconnectDelay' to configure the
frequency of retries): [/52.204.222.254:47500, /52.204.222.254:47501, /
52.204.222.254:47502, /52.204.222.254:47503, /52.204.222.254:47504, /
52.204.222.254:47505, /52.204.222.254:47506, /52.204.222.254:47507, /
52.204.222.254:47508, /52.204.222.254:47509, /52.204.222.254:47510]

I am following below blog , can any one help how to access ignite cluster
which is setup remotely do I need to build my code and run in the cluster
by ssh to the box ?

https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/

4.222.254:47507, /52.204.222.254:47508, /52.204.222.254:47509, /
52.204.222.254:47510]

[image: Thumbnail]



-- 
Thanks & Regards
Sri Tummala

Ignite Spark Example Question

[sri hari kali charan Tummala]

Hi All,

I am new to Apache Ignite community I am testing out ignite for knowledge
sake in the below example the code reads a json file and writes to ingite
in-memory table is it overwriting can I do append mode I did try spark
append mode .mode(org.apache.spark.sql.SaveMode.Append)
without stopping one ignite application inginte.stop which keeps the cache
alive and tried to insert data to cache twice but I am still getting 4
records I was expecting 8 records , what would be the reason ?

https://github.com/apache/ignite/blob/1f8cf042f67f523e23f795571f609a9c81726258/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameWriteExample.scala#L89

-- 
Thanks & Regards
Sri Tummala