Re: Number of servers and lambdas ?

[vkulichenko]

Hi,

Can you please properly subscribe to the mailing list so that the community
receives email notifications? Here is the instruction:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


limabean wrote
> I have a question about lambdas and Ignite servers.
> Running one of the lambda example from the Ignite tutorials:
> 
>   String a[] = new String[]{"abc","klm","xyz","pqr"};
>   List lst = Arrays.asList(a);
>   
> 
>while (true)
> {
> Collection
> 
>  coll = compute.apply(
> (String w) -> {
> System.out.println("Executing word: " + w);
> return w.length();
> }, lst);
> 
> 
> int sum = coll.stream().mapToInt(i -> i).sum();
> System.out.println("Sum is: " + sum);
> 
> Thread.sleep(2000);
> }
> 
> causes 4 servers to be started - one for each word in the string array
> passed in.
> 
> I already had a cluster of 2 servers running and then the program running
> the code pasted above starts a 3rd server.  I expected one of the servers
> to handle 2 words, similar to the "fault tolerance" demo where shutting
> down 1 server causes work to be shifted between the remaining servers.
> 
> Instead, what happens is a fourth server is started for the 4th word:
> 
> [13:55:58] Topology snapshot [ver=26, servers=4, clients=0, CPUs=8,
> heap=9.1GB]
> 
> Is this expected and could someone explain why the initial start up
> scenario 
> behaves this way instead of the way the  "fault tolerant" scenario is
> handled 
> with respect to how words are handed to servers ?
> 
> Thank you,

A node (either server or client) can be started only if you explicitly
execute ignite.sh script or call Ignition.start() within the application.
Nodes are never started automatically for the closure execution. Can you
double-check your code?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Number-of-servers-and-lambdas-tp3370p3372.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Unable to cache beyond RAM memory. Help me PLease!!!

[vkulichenko]

Hi Bhargav,

Can you please properly subscribe to the mailing list so that community
receives email notifications? Follow the instructions here:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


bhargav wrote
> I am trying to check the option of ignite, to store data beyond physical
> RAM. I have started a ignite server  in amazon instance (specifications
> 1gb RAM, disk space - 60gb). Through another application started ignite as
> CLIENT Only and started storing data using cache.put(k,v).

Several points to consider:

1. Besides the data, Ignite creates several internal structures which also
require memory. I would give it at least 2-4GB of heap memory just to make
in fully functional.
2. Usually it makes sense to switch to OFFHEAP_TIERED when you data set is
large and goes beyond 8-10GB per node, because large heaps can cause large
GC pauses. For small data sets ONHEAP_TIERED is enough.
3. Most important: Ignite is in-memory system and having all the data in
swap space will most likely slow you down instead of giving performance
benefits. What are you trying to achieve with this configuration?

-Val
-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Unable-to-cache-beyond-RAM-memory-Help-me-PLease-tp3366p3371.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: index and query org.apache.ignite.spark.IgniteRDD[String,org.apache.spark.sql.Row]

[Alexey Goncharuk]

Yep, BinaryObjectBuilder should definitely be a solution for this. You can
obtain an instance of Ignite from IgniteContext and use the IgniteBinary
interface to get an instance of BinaryObjectBuilder to build object
structures dynamically. And you can use QueryEntity class to describe the
index configuration for the binary objects being stored in cache. Once the
binary object is built, you can save it to Ignite using the same
savePairs() method.

I am surprised we do not have any examples for BinaryObjectBuilder,
however, you can take a look at the CacheJdbcPojoStore source code [1] to
get an idea how the builder works. You can also take a look at the
documentation [2].

Hope this helps!

[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStore.java
[2] https://apacheignite.readme.io/docs/binary-marshaller

Re: @QuerySqlField for collections

[Vladimir Ozerov]

Hi DmitryB,

Could you please provide the full class name of "List"? Is it
java.util.List or some Scala collection? Can you try putting
java.util.ArrayList and see if it works?

Vladimir.

On Fri, Mar 4, 2016 at 6:59 AM, DmitryB  wrote:

> Hi team,
>
> I found that fields with collection type are not properly handled by sql
> query;
>
> here is my code:
>
> import annotation.meta.field
> import org.apache.ignite.cache.query.annotations.QuerySqlField
>
> case class TRow1(
> @(QuerySqlField @field)(index = true) strP: String,
> @(QuerySqlField @field)(index = true) intP: Int,
> @(QuerySqlField @field)(index = true) doubleP: Double,
> @(QuerySqlField @field)(index = false) listP: List[String]
> ) extends Serializable {}
>
> // gen test data
> val rdd = sc.parallelize(1 to 10).map(x => TRow1("str-"+x, x, x*10, (1 to
> x).map(x => "val-" + x).toList ))
> val pair_rdd = rdd.map(r => (r.intP, r))
>
> // prep cache
> val igniteContext = new IgniteContext[Int, TRow1](sc, () => new
> IgniteConfiguration().setPeerClassLoadingEnabled(true), false)
> val cacheCfg = new CacheConfiguration[Int, TRow1]()
> cacheCfg.setName("cache01")
> cacheCfg.setIndexedTypes(classOf[Int], classOf[TRow1])
> val cache = igniteContext.fromCache(cacheCfg)
> >> res6: org.apache.ignite.spark.IgniteRDD[Int,sandbox.TRow1] =
> IgniteRDD[4]
> >> at RDD at IgniteAbstractRDD.scala:27
>
> // get results
> scala> val count = cache.count
> count: Long = 10
>
> scala> cache.take(1)
> res5: Array[(Int, sandbox.TRow1)] =
> Array((1,TRow1(str-1,1,10.0,List(val-1
>
> *scala> val result = cache.sql("select strP, intP, doubleP, listP from
> TRow1").take(1)
> result: Array[org.apache.spark.sql.Row] = Array([str-1,1,10.0,[]])*
>
> *So, field with collection type (List) is not returned back by sql query  *
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/QuerySqlField-for-collections-tp3364.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: index and query org.apache.ignite.spark.IgniteRDD[String,org.apache.spark.sql.Row]

[Andrey Gura]

Dmitry,

case classes are needed only for Sprak sqlContext.createDataFrame method
because type parameter should be scala.Product. Ignite doesn't imply such
limitations to user types.

I have no ideas how to use dynamic structures.

Alexey Goncharuk,

can we use BinaryObject for this case? May be you have some ideas?

On Fri, Mar 4, 2016 at 6:47 AM, DmitryB  wrote:

> Hi Andrey,
>
> Thanks a lots for your help.
> Unfortunately, i can not use case classes, because a schema information is
> only available at runtime;
> to make it more clear let me add more details. suppose that i have a very
> big data set (~500 Tb) which is stored in AWS s3 in a parquet format; Using
> spark, i can process (filter + join) it and reduce size down to ~200 -500
> Gb; resulted dataset i would like to save in ignite cache using IgniteRdd
> and create indexes for a particular set of fields which will be used later
> for running queries (filter, join, aggregations); My assumption is that
> having this result dataset in ignite + indexes would help to improve the
> performance comparing to using spark DataFrame (persisted);
> Unfortunately, the resulted dataset schema can vary with great number of
> variations; So, it seems impossible to describe all of them with case
> classes;
> This is why an approach to store spark.sql.row + describe query fields and
> indexes using QueryEntity would be preferable;
> Thanks to your explanation, i see that this approach doesn't works;
> Another solutions that is spinning in my head is to generate case classes
> dynamically (at runtime) based on spark data frame schema, then map
> sql.rows
> to RDD[generated_case_class], describe ignite query and index fields using
> QueryEntity, create IgniteContext for generated case class; Im not sure
> that
> this approach is even possible, so i would like to ask for your opinion
> before i go deeper;
> Will be very grateful for advice
>
> Best regards,
> Dmitry
>
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/index-and-query-org-apache-ignite-spark-IgniteRDD-String-org-apache-spark-sql-Row-tp3343p3363.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Re: Version issue with concurrent cache updates (EntryProcessor)

[Myron Chelyada]

Thanks Alexey,

I will consider an option to get familiar more deeply with product and
contribute some patch.

2016-03-04 1:03 GMT+02:00 Alexey Goncharuk :

> Myron,
>
> I believe IGNITE-2645 should be fixed in the near future since the issue
> is critical, and will definitely be included to 1.6.
>
> As for the IGNITE-1018, I will not speculate on the timelines because the
> issue has some workarounds, even though it is possible that it will be
> fixed for 1.6 if someone in the community picks it up.
>
> You can track the discussion on the dev list regarding the 1.6 release and
> planned features for it. Needless to say that you are always welcome to
> contribute a patch for any of the tickets you would like to be released :)
>
> --
> AG
>