Re: Number of partitions of IgniteRDD

[vijayendra bhati]

Hi Val,
I am creating DataFrame using below code - 
 public DataFrame getStockSimulationReturnsDataFrame(LocalDate 
businessDate,String stock){ /*  * If we use sql query , we are assuming that 
data is in cache.  * */ String sql = "select simulationUUID,stockReturn from 
STOCKSIMULATIONRETURNSVAL where businessDate = ? and symbol = ?"; DataFrame df 
=jic.fromCache(PARTITIONED_CACHE_NAME).sql(sql, businessDate,stock); return df; 
}

And to check partitions I am doing - 
private JavaRDD getSimulationsForStock(String stock,LocalDate 
businessDate) { DataFrame df =  
StockSimulationsReaderFactory.getStockSimulationStore(jsc, businessDate, 
businessDate).getStockSimulationReturnsDataFrame(businessDate, stock); 
System.out.println(""+df.javaRDD().getNumPartitions()); return 
df.javaRDD(); }
Regards,Vij 

On Friday, April 29, 2016 3:09 AM, vkulichenko 
 wrote:
 

 Hi Vij,

How do you check the number of partitions and what are you trying to
achieve? Can you show the code?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Number-of-partitions-of-IgniteRDD-tp4644p4671.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


  

Re: Ignite cache data size problem.

[vkulichenko]

Hi Kevin,

Ignite Visor [1] provides 'gc' command that performs GC on all nodes. You
can also use any other tool, e.g. VisualVM.

[1] https://apacheignite.readme.io/docs/command-line-interface

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4680.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Client Blocks On Ignite Server Restart

[vkulichenko]

Hi Colin,

I'm not sure that you have the same issue that Ken has. Is it possible for
you to share your test with us?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Client-Blocks-On-Ignite-Server-Restart-tp4554p4679.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to validate that what was retrieved from Ignite cache is not NULL or empty?

[vkulichenko]

Hi,

Your expectation is correct. The second IgniteRDD will connect to the same
Ignite cache that will already have all the data. This was you can share the
data between applications.

Zero in 'IgniteRDD[0]' string is just an ID within SparkContext, so it
doesn't tell anything about the cached data. I would recommend you to go
through the example described in [1], it demonstrates exactly the same use
case that you're trying to implement.

[1]
https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-validate-that-what-was-retrieved-from-Ignite-cache-is-not-NULL-or-empty-tp4668p4678.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite near cache Problem

[vkulichenko]

Hi,

Can you please properly subscribe to the mailing list so that the community
can receive email notifications? Here is the instruction:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


suhuadong wrote
> hi team,
> Version of igntie is 1.5.0.final.
> 
> ClientMode of Ignite is true.
> 
> I'm trying to run near cache example codes,but I'm having a strange
> Exception.
> 
>
> example code:
> // Create distributed cache on the server nodes, called
> "myCache".
> ignite.getOrCreateCache(new CacheConfigurationString,
> String("myCache"));
> 
> // Create near-cache configuration for "myCache".
> NearCacheConfigurationString, String nearCfg = new
> NearCacheConfiguration<>();
> 
> // Use LRU eviction policy to automatically evict entries
> // from near-cache, whenever it reaches 100_000 in size.
> nearCfg.setNearEvictionPolicy(new
> LruEvictionPolicy<>(100_000));
> 
> // Create near-cache for "myCache".
> IgniteCacheString, String cache =
> ignite.getOrCreateNearCache("myCache", nearCfg);
> 
> https://apacheignite.readme.io/docs/near-caches

This happens because getOrCreateCache also creates a client handler without
the near cache. After that it's impossible to dynamically add near cache.

In your code you should simply remove the first line. getOrCreateNearCache
method will automatically start the distributed server cache and then create
a local near cache.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-near-cache-Problem-tp4657p4677.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Running gridgain yardstick

[vkulichenko]

Yardstick is designed to run benchmarks in distributed environments. Server
hosts, driver hosts and benchmark run configurations are provided in a
configuration file (see [1] for example). You can then run
benchmark-run-all.sh and provide this file, everything will happen
automatically. All participating hosts including local should allow SSH
access, it's used to start the required processes. 

[1]
https://github.com/apacheignite/yardstick-ignite/blob/master/config/benchmark.properties

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Running-gridgain-yardstick-tp4559p4676.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Client fails to connect - joinTimeout vs networkTimeout

[vkulichenko]

Hi Binti,

Yes, you should set 'excludeNeighbors' property on the affinity function:







-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Client-fails-to-connect-joinTimeout-vs-networkTimeout-tp4419p4675.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: how to fetch data when its loaded using cache.loadCache(null,100_00)?

[vkulichenko]

Hi Ravi,

I believe you already had the similar issue. You should always make sure
that Hibernate entity is fully initializes before you close the Hibernate
session and store the value in cache.

If this doesn't help, please show the whole trace (looks like it's
truncated).

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/how-to-fetch-data-when-its-loaded-using-cache-loadCache-null-100-00-tp4654p4673.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Continuous Queries

[vkulichenko]

Hi,

As far as I know, it's very close and should be sent for a vote soon. I
would recommend to monitor the dev list [1] for the latest news.

[1] http://ignite.apache.org/community/resources.html#mail-lists

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Continuous-Queries-tp4440p4672.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Number of partitions of IgniteRDD

[vkulichenko]

Hi Vij,

How do you check the number of partitions and what are you trying to
achieve? Can you show the code?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Number-of-partitions-of-IgniteRDD-tp4644p4671.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Raffle - Free Ticket to In-Memory Computing Summit, May 23-24, San Fran

[alisabaum]

We are raffling off a limited number of conference tickets to the In-Memory
Computing Summit which is May 23-24 at the Grand Hyatt San Francisco.

Link to Raffle Registration Page:
http://go.gridgain.com/IMCS-2016_Raffle.html

Winners will be randomly selected each week leading up to the conference,
and notified by email. You only need to register once to participate in the
weekly drawing.

The conference is the only industry-wide event of its kind, tailored to
in-memory computing related technologies and solutions. This multi-track
conference brings together the data scientists, engineers, developers,
architects and techs responsible for choosing and implementing solutions
involving in-memory computing, Big Data, IoT, and HPC for the purpose of
education, discussion and networking. 

Agenda includes:
-Keynotes by industry visionaries from 451 Research, Barclays, GridGain,
ScaleMP, and SNIA
-Breakout Sessions by speakers from Target, Workday, MemSQL, Oracle, Intel,
GridGain, Hazelcast, and many more

Sponsors include: GridGain, Neeve Research, VoltDB, ScaleMP, ScaleOut
Software, Hazelcast, MemSQL, Plexistor, Third Eye, EPAM, YADRO, The Apache
Software Foundation, and more

For more info about the conference, visit: http://imcsummit.org/




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Raffle-Free-Ticket-to-In-Memory-Computing-Summit-May-23-24-San-Fran-tp4669.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Running gridgain yardstick

[akritibahal91]

Ok got it. So, say, I want to run the IgnitePutGetBenchmark.java to test the
Ignite PutGet, so how do I run it??



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Running-gridgain-yardstick-tp4559p4664.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Ignite & Kubernetes

[Christos Erotocritou]

Hi all,

Is anyone working with Ignite & kubernetes?

Moreover I’d like to understand how it would be possible to do auto discovery 
of new Ignite nodes.

Thanks,

Christos

Re: Client fails to connect - joinTimeout vs networkTimeout

[bintisepaha]

Is there a way to configure backups node on a different physical host in such
a scenario? I do not want the primary and back up on the same host in the
event that host crashes. 

Thanks,
Binti



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Client-fails-to-connect-joinTimeout-vs-networkTimeout-tp4419p4661.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: how to fetch data when its loaded using cache.loadCache(null,100_00)?

[Ravi Puri]

i am getting this error while fetching

cache.get(1); at client side

Exception in thread "main" org.hibernate.LazyInitializationException: could
not initialize proxy - no Session
at
org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:186)
at
org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:545)
at
org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:124)
at
org.hibernate.collection.internal.PersistentMap.toString(PersistentMap.java:270)
at java.lang.String.valueOf(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/how-to-fetch-data-when-its-loaded-using-cache-loadCache-null-100-00-tp4654p4655.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Continuous Queries

[ght230]

Do you know when version 1.6 will release?
This function is important to me.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Continuous-Queries-tp4440p4651.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: SQL Aliases are not interpreted correctly

[Alexey Kuznetsov]

Jun, I implemented support for aliases generation in Schema Import Utility
and pushed to master.

Please try with latest nightly-build

https://ignite.apache.org/community/contribute.html#nightly-builds


On Wed, Apr 20, 2016 at 2:02 PM, jan.swaelens 
wrote:

> Superb, that would really do the trick for my use cases!
>
> best regards
> jan
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/SQL-Aliases-are-not-interpreted-correctly-tp4281p4356.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

Re: Ignite Installation with Spark under CDH

[Vladimir Ozerov]

Hi Michael,

Did you have a chance to analyze heap dump to understand what caused OOME?
As per *IgniteConfiguration*, it is made non-serialziable intentionally,
because we do not expect it to be passed over the wire. Could you please
provide a stack trace where you see it is being serialized?

Vladimir.

On Wed, Apr 27, 2016 at 10:31 PM, mdolgonos 
wrote:

> Vladimir,
>
> Update - I think I solved the ClassNotFound exception. It looks like the
> Ignite installation document for Spark and CDH is outdated and doesn't
> contain complete information on integrating Ignite with Spark running in a
> 'Yarn' (cluster) mode on CDH which I have. This is what I have done and now
> am able to run my Spark program (however, with exception described below):
> went to
> Cloudera Manager -> YARN (MR2 Included) -> Configuration -> Service Wide ->
> Advanced -> Spark Client Advanced Configuration Snippet (Safety Valve) for
> spark-conf/spark-defaults.conf
> Added etc/ignite-fabric-1.5.0/libs/* to already present
>
> spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar.
> The combined line looks like this:
>
> spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar:/etc/ignite-fabric-1.5.0/libs/*
>
> However, after starting Ignite and submitting my Spark program (in another
> window) I see the following exception:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> The only line of code where I put something into the Ignite cache where I'm
> trying to save RDD there:
> cacheIgRDD.savePairs(partRDD, true)
>
> One thing to note is it looks like IgniteConfigurationis not Serializable
> so
> I had to create MyIgniteConfiguration extends IgniteConfiguration with
> Serializable
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4624.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Ignite Client Blocks On Ignite Server Restart

[colinc]

In the case of the test that I am executing, high contention caused by the
test running as an Ignite client is causing not only the client but the
whole cluster to become unresponsive upon subsequent destruction of the
cache. The only way to get it to respond again seems to be to kill the
client JVM - but that depends on identifying which client is causing the
problem. Looking at the processor activity etc. doesn't seem to help because
the cluster is not busy at this point - rather it is just hung along with
everything else.

Because it's just a test that I'm running, I have the luxury of being able
to kill the old clients and then test the cache for responsiveness by
running a new client. As noted, this is a big improvement on v1.5 where the
only way to recover was to restart the cluster. I'd be interested to know if
v1.6 solved Ken's issue - which is probably a more common situation in
production than mine.

Regards,
Colin.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Client-Blocks-On-Ignite-Server-Restart-tp4554p4646.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Number of partitions of IgniteRDD

[vijayendra bhati]

Hi,
Can anybody let me know the criteria on which number of partitions are decided 
for IgniteRDD ?From the IgniteRDD code, as per getPartitions() implementation 
it is dependent upon number of partitions of underlying cachebut I am not aware 
how the number of partitions of underlying cache is decided.
Regards,Vij