Re: How to configure a QueryEntity for a BinaryObject
I failed to configure the binaryobject ignitecache. My cache configuration as
following:
name
java.lang.String
org.apache.ignite.binary.BinaryObject
I can apply BinaryObjectBuilder to build a binaryobject with type
name:"com.example.Foo" and put it in the cache. But when i apply SqlQuery, a
IgniteSQLException is thrown with message:"Failed to find SQL table for
type: com.example.Foo". But when i change the cache configuration's
indexedTypes with com.example.Foo, The ignite fail to create such a cache,
cause of "com.example.Foo" doesn't exist. My binary object builder code as
following:
final Collection result = new ArrayList<>(numbers);
IntStream.range(1,numbers).forEach((i)->{
BinaryObjectBuilder fooBuilder
=ignite.binary().builder("com.example.Foo");
fooBuilder.setField("name","foo"+i).setField("age",i);
result.add(fooBuilder.build());
});
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: How to configure a QueryEntity for a BinaryObject
If you're configuring QueryEntity, you don't need to add to the
configuration indexedTypes too. It's just different ways to configure
Indexes and queryable fields.
Evgenii
2017-09-20 10:10 GMT+03:00 Savagearts :
> I failed to configure the binaryobject ignitecache. My cache configuration
> as
> following:
> class="org.apache.ignite.configuration.CacheConfiguration">
>
>
>
>
>
>
>
>
> />
> value="java.lang.String" />
>
>
>
>
> name
>
>
>
>
>
>
> java.lang.String
> org.apache.ignite.
> binary.BinaryObject
>
>
>
>
> I can apply BinaryObjectBuilder to build a binaryobject with type
> name:"com.example.Foo" and put it in the cache. But when i apply SqlQuery,
> a
> IgniteSQLException is thrown with message:"Failed to find SQL table for
> type: com.example.Foo". But when i change the cache configuration's
> indexedTypes with com.example.Foo, The ignite fail to create such a cache,
> cause of "com.example.Foo" doesn't exist. My binary object builder code as
> following:
>final Collection result = new ArrayList<>(numbers);
> IntStream.range(1,numbers).forEach((i)->{
> BinaryObjectBuilder fooBuilder
> =ignite.binary().builder("com.example.Foo");
> fooBuilder.setField("name","foo"+i).setField("age",i);
> result.add(fooBuilder.build());
> });
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: How to configure a QueryEntity for a BinaryObject
Thanks, Evgenii. I removed the indexedTypes configuration according to your suggestion. But it still doesn't work. The ignite throws a exception:"Failed to find SQL table for type: com.example.Foo". (There is a error in the configuration of my previous post, i change the valType from "com.Foo.Bar" to "com.example.Foo"). -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Problem with transactions in Apache Ignite!
Hi!
I am use of contributors of project "Hibernate OGM". I am preparing my PR
for the project.
I have a question.
Problem is that part of tests is failed after migration from Ignite 1.9 to
Ignite 2.1. These tests uses one feature (session.flush()).
em.getTransaction().begin();
Poem poem = new Poem();
poem.setName("L'albatros");
em.persist(poem);
em.getTransaction().commit();
em.clear();
em.getTransaction().begin();
Poem poem2 = new Poem();
poem2.setName("Waza");
em.persist(poem2);
em.flush(); //*data not flushed to Ignite's cache*
Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(2L);
em.getTransaction().rollback();
Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(1L);
Please comment situation with em.flush()
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: How to configure a QueryEntity for a BinaryObject
Check QueryEntity class, it contains tableName property. Evgenii -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Join with subquery in 1.9 or 2.0
Hi, Looks like a bug. Would you please share a full stacktrace and a reproducer if possible? You can try to rewrite query without join to smth like this: Select .. from A, B Where A.id = B.id; On Mon, Sep 18, 2017 at 7:36 PM, acet wrote: > Hello, > I was looking to do something similar to: > > SELECT a.customerid, h.name, h.address > FROM > "customer_cache".CUSTOMER as a > JOIN > (select min(id) as id, name, address, cust_id from "second_cache".DETAILS > group by name, address, cust_id) as h on a.customerid = h.cust_id > > This seems to work fine in the debug console but when trying it with ignite > I get: > > javax.cache.CacheException: class org.apache.ignite.IgniteException: > org.apache.ignite.internal.processors.query.h2.sql.GridSqlJoin cannot be > cast to org.apache.ignite.internal.processors.query.h2.sql.GridSqlAlias > > > Is there any way to achieve this? > Thanks > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > -- Best regards, Andrey V. Mashenkov
Re: REST API response json is empty.
Hi Anji!
Parameters of constructor of College class are not marked with val keyword,
so @BeanProperty annotations have no effect.
Adding val keyword before every constructor parameter should solve your
second problem.
Denis
ср, 20 сент. 2017 г. в 6:37, ANJANEYA PRASAD NIDUBROLU :
> Any luck with my query? What am i missing? Why REST response is blank
> though the cache has data, i tried scan on visor i can see the data over
> there.
>
> Thanks,
> Anji.
> On 19 Sep 2017 00:26, "ANJANEYA PRASAD NIDUBROLU"
> wrote:
>
>> Hello All,
>>
>> Hope you are doing great!.
>>
>> I have tried Ignite's REST API via postman. It is not throwing any
>> errors, but the response json's value part has nothing in it.
>>
>> Here I am pasting cache config (piece of xml file), bean class and main
>> class where I am saving the sparkRDD to cache. Also, the attached document
>> has REST requests and responses along with respective logs.
>>
>> As the Spark RDD/ DF I am using has more columns, i have created scala
>> bean class so that I can save it on to IgniteCache as Key, Value.
>>
>> Ignite server and clients are able to talk to each other. Cache is
>> created and loaded successfully.
>> So far so good, trouble started when I am trying to trigger from REST API
>> (the attached notepad has REST APIs i tried to test and their response).
>>
>> 1) Though the Bean class I created has 8 columns - the cache created has
>> only 7 columns, what happened to final one? [even "*cache -c=<> -scan*"
>> command from "*visor*" results has 7 columns.]
>> 2) The REST API responses says it is success but the response json's
>> value part is empty.
>>
>> Not sure what went wrong. Happy to provide more details if required.
>> Many Thanks,
>> Anji.
>>
>> *ignite-config.xml*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> =
>> *College.scala*
>>
>> package org.anjaneya.prasad.loadbean
>> import scala.beans.BeanProperty
>>
>> class College(@BeanProperty register_number :String,
>> @BeanProperty current_city: String,
>> @BeanProperty date2: String,
>> @BeanProperty date_of_birth: String,
>> @BeanProperty student_code: String,
>> @BeanProperty native_city: String,
>> @BeanProperty college_end_date_1: String,
>> @BeanProperty college_start_date_1: String
>> ) extends Serializable{
>> override def toString: String = s"College: $register_number,
>> $current_city, $date2, $date_of_birth, $student_code, $native_city,
>> $college_end_date_1, college_start_date_1"
>>
>> //return format("%s, %s, %s, %s, %s, %s, %s, %s", register_number ,
>> native_city , current_city , student_code,college_end_date_1,
>> date_of_birth, date2)
>> }
>>
>>
>>
>> *MainProcess.scala*
>>
>> val ic = new IgniteContext(sc,
>> "/home/ops/College/src/main/resources/ignite-config.xml", true)
>>
>> var sharedRDDCollege: IgniteRDD[String, College] =
>> ic.fromCache("CollegeCache")
>> //sharedRDDCollege.collect().foreach(print)
>>
>> var CollegeCache = test2.rdd.map(x => (x.getString(0),
>>new College(x.getString(0) , x.getString(1) , x.getString(2) ,
>> x.getString(3) , x.getString(4) , x.getString(5) , x.getString(6) ,
>> x.getString(7
>>
>> //CollegeCache.collect.foreach(print)
>> sharedRDDCollege.savePairs(CollegeCache)
>>
>>
>>>
Re: CacheStoreAdapter#loadAll and ContinuousQuery
Hi Matt! Please provide your implementation of CacheStore so we could tell what is wrong with it. Denis ср, 20 сент. 2017 г. в 4:05, matt : > I've got an implementation of CacheStoreAdapter that appears to be working > (it's persisting items etc..). I also have a ContinuousQuery setup and an > initialQuery that runs after the impls loadAll(). Before I started using my > own impl of CacheStoreAdapter - the ContinuousQuery worked as expected, but > now with my impl, it's not. The initialQuery cursor actually doesn't ever > yield anything. Is there something I'm missing with making these two > components work together properly? Anything special I need to do with the > impl or config? > > Thanks, > - Matt > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Problem with transactions in Apache Ignite!
Hi!
Could you please provide a pure Ignite reproducer that won't involve
Hibernate OGM?
Right now it's hard to tell whether the problem is in Ignite or in
Hibernate.
Thanks!
Denis
ср, 20 сент. 2017 г. в 12:52, schernolys :
> Hi!
> I am use of contributors of project "Hibernate OGM". I am preparing my PR
> for the project.
> I have a question.
> Problem is that part of tests is failed after migration from Ignite 1.9 to
> Ignite 2.1. These tests uses one feature (session.flush()).
>
> em.getTransaction().begin();
> Poem poem = new Poem();
> poem.setName("L'albatros");
> em.persist(poem);
> em.getTransaction().commit();
> em.clear();
> em.getTransaction().begin();
> Poem poem2 = new Poem();
> poem2.setName("Waza");
> em.persist(poem2);
> em.flush(); //*data not flushed to Ignite's cache*
>
> Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(2L);
> em.getTransaction().rollback();
>
> Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(1L);
>
>
> Please comment situation with em.flush()
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite YARN deployment mode issues
Hello Ray! -- Ilya Kasnacheev 2017-09-19 10:51 GMT+03:00 Ray : > I'm using ignite 2.1, my ignite config xml is as follows. > > I tried to put the log4j2.xml file under IGNITE_USERS_LIBS, still not > working. > I guess it should be relative to IGNITE_HOME, or absolute path (I expect it's no-no with YARN). What is IGNITE_HOME exactly in case of YARN, I will try to find out. -- Ilya Kasnacheev
Re: Ignite YARN deployment mode issues
Thanks for the reply, Ilya. The IGNITE_HOME for yarn is /yarn/nm/usercache/username/appcache/application_appid/container_containerID/ignite/ The log4j2.xml is under IGNITE_HOME/bin folder too. If I launch ignite using ignist.sh, it's working fine and the log4j2.xml under IGNITE_HOME/bin folder can be picked up by ignite. But for YARN deployment, the issue still exists. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite YARN deployment mode issues
Hello Ray, Did you try specifying "bin/log4j2.xml" as constructor-arg then? In stand-alone deployment, it could save you that Ignite also looks in current directory in addition to IGNITE_HOME -- Ilya Kasnacheev 2017-09-20 16:57 GMT+03:00 Ray : > Thanks for the reply, Ilya. > > The IGNITE_HOME for yarn is > /yarn/nm/usercache/username/appcache/application_appid/ > container_containerID/ignite/ > > The log4j2.xml is under IGNITE_HOME/bin folder too. > If I launch ignite using ignist.sh, it's working fine and the log4j2.xml > under IGNITE_HOME/bin folder can be picked up by ignite. > But for YARN deployment, the issue still exists. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Ignite YARN deployment mode issues
Hi Ilya Do you mean modify the original config from to if my understanding is correct? Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: WAL log folder issue in YARN mode
Hello Ray, After a brief glance, I have figured out that YARN expects all its tasks to use HDFS for data storage and not the local system, which is considered ephemeral. You can use Ignite *with* HDFS but I doubt you can use Ignite native persistence persist to HDFS, really. Please tell me what you are trying to accomplish by this setup? Do you want to use YARN just as a launcher for Ignite? Regards, -- Ilya Kasnacheev 2017-09-20 9:58 GMT+03:00 Ray : > When I deployed ignite as a YARN application with persistent store enabled. > The WAL logs are under the > /yarn/nm/usercache/username/appcache/application_appid/ > container_containerID/ignite/apache-ignite-fabric-2.1.0-bin/work/db/wal/. > But when ignite is restarted using YARN, a new appid will be created so the > WAL in the old app will not be copied to the new app. > I tried setting IGNITE_WORK_DIR to a local directory and > IGNITE_RELEASES_DIR > to a hdfs directory hoping ignite will save the WAL logs to these folders, > but it does not. > Please advice me how to solve this issue. > > Thanks > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: AWS Apache Ignite AMI startup.sh reports spurrious errors if options have blanks
Sorry for the delay, spam filter. I haven't been able to find an explanation for startup.sh, but when I login to an EC2 instance created using the AWS community AMI for Ignite (which only has docker and pulls ignite by the version you specify), I find startup.sh in the current directory, and it is clearly the script run at least the first time the instance boots.When I search for pieces of the contents, nothing shows up. When searching for startup.sh with various AWS/ignite qualifiers, I get back to this post, or get a large list of things that don't seem relevant.When I run this script, it loads user-data (which is what is specified in Advanced Options on the AWS AMI launch GUI - a set of environment variables), and prints errors if any of those lines have spaces. This is the entire script: #!/bin/bash if [ ! -f ./user-data ]; then wget http://169.254.169.254/latest/user-data sed -i -e '$a\ ' ./user-data fi ENV_OPTIONS="" while read p; do if [ ! -z "$p" ]; then ENV_OPTIONS="$ENV_OPTIONS -e \"$p\"" export $p fi done < ./user-data eval "docker pull apacheignite/ignite:$IGNITE_VERSION" eval "docker run -d --net=host $ENV_OPTIONS apacheignite/ignite:$IGNITE_VERSION" -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Problem with transactions in Apache Ignite!
Sergey!
After offline discussion I managed to reproduce your problem.
It seems that IgniteCache.size(...) doesn't take into account changes made
during ongoing transaction.
It looks like a bug to me. I'll try to figure out what behavior is expected
here and whether it is a known issue.
Maybe it is worth to file a ticket in JIRA.
For now you may check whether records are written to cache by using
Cache.get(...) method instead of checking the size.
Make sure to invoke get(...) in the thread that started the transaction,
otherwise you risk to get a deadlock.
Denis
ср, 20 сент. 2017 г. в 16:23, Denis Mekhanikov :
> Hi!
>
> Could you please provide a pure Ignite reproducer that won't involve
> Hibernate OGM?
> Right now it's hard to tell whether the problem is in Ignite or in
> Hibernate.
>
> Thanks!
> Denis
>
> ср, 20 сент. 2017 г. в 12:52, schernolys :
>
>> Hi!
>> I am use of contributors of project "Hibernate OGM". I am preparing my PR
>> for the project.
>> I have a question.
>> Problem is that part of tests is failed after migration from Ignite 1.9 to
>> Ignite 2.1. These tests uses one feature (session.flush()).
>>
>> em.getTransaction().begin();
>> Poem poem = new Poem();
>> poem.setName("L'albatros");
>> em.persist(poem);
>> em.getTransaction().commit();
>> em.clear();
>> em.getTransaction().begin();
>> Poem poem2 = new Poem();
>> poem2.setName("Waza");
>> em.persist(poem2);
>> em.flush(); //*data not flushed to Ignite's cache*
>>
>> Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(2L);
>> em.getTransaction().rollback();
>>
>> Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(1L);
>>
>>
>> Please comment situation with em.flush()
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: WAL log folder issue in YARN mode
Hi Ilya, I want to use YARN to launch Ignite because YARN is easy to manage. And there's other service in our cluster and I want to use YARN as a unified resource manager. So what you're telling me is that if I deploy Ignite on YARN there's no way to utilize the native persistent store feature, right? I think if user want to deploy Ignite on YARN, it's only reasonable to move the Ignite native persistent store from local file system to HDFS. Otherwise Ignite on YARN lacks this feature and might be very important to a lot of users. Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite/yardstick benchmarking from within Docker Image
As I understand more, I changed the subject. The Docker image for Ignite 2.1.0 contains the ignite-yardstick benchmarks, which include a ReadMe file that provides instructions that *do not work* if you simply try to run them from inside the Docker Container where Ignite is already running. I had been trying to work from config/benchmark-sample.properties, and thrashed on this for quite a while. I have gotten the benchmark to actually execute inside the container, after setting SERVER_HOSTS="" bin/benchmark-run-all.sh config/benchmark-remote.properties -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Problem with transactions in Apache Ignite!
I found this thread that contains discussion of a similar problem:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-Transactions-and-non-committed-entries-td6451.html
Size is simply not a transactional operation and it is not guaranteed the
value returned from it will be consistent with ongoing transactions. So
aparently it is not a bug.
Denis
ср, 20 сент. 2017 г. в 17:37, Denis Mekhanikov :
> Sergey!
>
> After offline discussion I managed to reproduce your problem.
> It seems that IgniteCache.size(...) doesn't take into account changes made
> during ongoing transaction.
>
> It looks like a bug to me. I'll try to figure out what behavior is
> expected here and whether it is a known issue.
> Maybe it is worth to file a ticket in JIRA.
>
> For now you may check whether records are written to cache by using
> Cache.get(...) method instead of checking the size.
> Make sure to invoke get(...) in the thread that started the transaction,
> otherwise you risk to get a deadlock.
>
> Denis
>
> ср, 20 сент. 2017 г. в 16:23, Denis Mekhanikov :
>
>> Hi!
>>
>> Could you please provide a pure Ignite reproducer that won't involve
>> Hibernate OGM?
>> Right now it's hard to tell whether the problem is in Ignite or in
>> Hibernate.
>>
>> Thanks!
>> Denis
>>
>> ср, 20 сент. 2017 г. в 12:52, schernolys :
>>
>>> Hi!
>>> I am use of contributors of project "Hibernate OGM". I am preparing my PR
>>> for the project.
>>> I have a question.
>>> Problem is that part of tests is failed after migration from Ignite 1.9
>>> to
>>> Ignite 2.1. These tests uses one feature (session.flush()).
>>>
>>> em.getTransaction().begin();
>>> Poem poem = new Poem();
>>> poem.setName("L'albatros");
>>> em.persist(poem);
>>> em.getTransaction().commit();
>>> em.clear();
>>> em.getTransaction().begin();
>>> Poem poem2 = new Poem();
>>> poem2.setName("Waza");
>>> em.persist(poem2);
>>> em.flush(); //*data not flushed to Ignite's cache*
>>>
>>> Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(2L);
>>> em.getTransaction().rollback();
>>>
>>> Assertions.assertThat(TestHelper.getNumberOfEntities(em)).isEqualTo(1L);
>>>
>>>
>>> Please comment situation with em.flush()
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
Re: Cannot start/stop cache within lock or transaction
Hi All. I've updated reproducer and provided some investigation results at issue. On Fri, Sep 15, 2017 at 7:12 PM, rajivgandhi wrote: > Hi Yakov, > The test will pass if you comment the line: > "IgniteCache cache3 = > ignite.getOrCreateCache(getConfig(*"cache3"*));" > > Question is why is creation of cache3 causing deadlock? > > Please note: > 1. Lock is being acquired on cache2 > 2. clear is being called on cache1 > > The error being reported and reproduced is nasty enough that it has kind of > eroded our confidence in the library. > > thanks, > Rajeev > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Unsubscribe
Please unsubscribe me from the mailing list. I tried several times but still receive emails. What's right way to do that? thank you. Jessie On Fri, Sep 8, 2017 at 4:47 AM, 谢宁 wrote: > > > > -- Original -- > *From:* waterg > *Date:* Fri,Sep 8,2017 0:03 PM > *To:* user > *Subject:* Re: unsubscribe > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Unsubscribe
Hi Jessie. Sorry to see you go. To unsubscribe from the user mailing list send a letter to user-unsubscr...@ignite.apache.org with a word "Unsubscribe" without quotes as a topic. If you have a mailing client, follow an unsubscribe link here: https://ignite.apache.org/community/resources.html#mail-lists Denis ср, 20 сент. 2017 г. в 18:22, Jessie Lin : > > Please unsubscribe me from the mailing list. > I tried several times but still receive emails. > What's right way to do that? thank you. > > Jessie > > > On Fri, Sep 8, 2017 at 4:47 AM, 谢宁 wrote: > >> >> >> >> -- Original -- >> *From:* waterg >> *Date:* Fri,Sep 8,2017 0:03 PM >> *To:* user >> *Subject:* Re: unsubscribe >> >> >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
Re: Ignite/yardstick benchmarking from within Docker Image
Hi Dave! Yardstick starts Ignite nodes itself. Any nodes visible by discovery SPI will interfere with running benchmarks. Denis -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Unsubscribe
Thank you Denis. I've seen great work from Ignite. Keep up! On Wed, Sep 20, 2017 at 8:33 AM, Denis Mekhanikov wrote: > Hi Jessie. > > Sorry to see you go. > To unsubscribe from the user mailing list send a letter to > user-unsubscr...@ignite.apache.org with a word "Unsubscribe" without > quotes as a topic. > If you have a mailing client, follow an unsubscribe link here: > https://ignite.apache.org/community/resources.html#mail-lists > > Denis > > ср, 20 сент. 2017 г. в 18:22, Jessie Lin : > >> >> Please unsubscribe me from the mailing list. >> I tried several times but still receive emails. >> What's right way to do that? thank you. >> >> Jessie >> >> >> On Fri, Sep 8, 2017 at 4:47 AM, 谢宁 wrote: >> >>> >>> >>> >>> -- Original -- >>> *From:* waterg >>> *Date:* Fri,Sep 8,2017 0:03 PM >>> *To:* user >>> *Subject:* Re: unsubscribe >>> >>> >>> >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> >>
Re: Ignite/yardstick benchmarking from within Docker Image
Yes. When I figured that out, my immediate reaction was to kill the running ignite instance. But since that process was what was keeping the docker container running, the container exits. So instructions on how to run the benchmarks from inside the docker container they are delivered in would be helpful. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Client Mode and client to client communication
Hi All, We currently have an application which uses Ignite and many of its features including, but not limited to, execution services, distributed map, and map queries. Ignite is integrated into our application as a client and we deploy standalone Ignite servers. We recently starting looking at the Pivotal Cloud Foundry model and how we might be able to deploy our application in that environment. In PCF, containers do not have direct access to one another. Packets are wrapped and routed via a Silk VXLAN component (more information here: https://github.com/cloudfoundry-incubator/cf-networking-release/blob/develop/docs/arch.md). In a test to use PCF, we attempted to deploy our Ignite servers in BOSH (where it is essentially a VM with a public IP). Our client nodes can communicate with the server nodes in this model but we ran into some other errors. We saw the error: “org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues)”. This issue arises from our use of the ExecutorService and we believe it’s caused by the lack of allowed communication between clients (since clients cannot communicate directly with one another because the containers do not have access to each other). A few questions: 1. Is there a reason why clients communicate to other clients directly and not through the servers? 2. Is there a setting we can change in Ignite to facilitate client to client communication through the servers? Essentially route the requests through the servers for processing instead of direct communication? 3. What other client to client communication is built into Ignite? Is there other client to client functionality provided by Ignite that would break even if we weren’t using the executor? 4. Other limitations of deploying an application using Ignite in PCF? We found this post: https://stackoverflow.com/questions/40372570/running-apache-ignite-cluster-on-pivotal-cloud-foundry-environment Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Compute Grid Request Tracing
Hi, We wish to implement request tracing in our Ignite Compute Grid. Typically we handle all of this at the Jersey/Servlet level. Where we pass a Header to the webapp, causing a Filter to then write a flag into the Logger's MDC, which, in turn, we use to tell Logback to enable TRACE logging on that particular Thread. Done using Jersey's ContainerRequestFilter and ContainerResponseFilter. And Logback's MDCFilter. So before I start building something on top of our ComputeTask and ComputeJob, is there any built-in mechanism for request tracing in Ignite?? (there is nothing I could find) And is there any Filter mechanism that fits this sort of concept?? So that I can do this out-of-band. (what I've found is not really a fit) Thanks, -- Chris -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: REST API response json is empty.
Hello Denis,
Many thanks. Its working now and it solved both the problems.
Thanks,
Anji.
On Wed, Sep 20, 2017 at 4:41 PM, Denis Mekhanikov
wrote:
> Hi Anji!
>
> Parameters of constructor of College class are not marked with val
> keyword, so @BeanProperty annotations have no effect.
>
> Adding val keyword before every constructor parameter should solve your
> second problem.
>
> Denis
>
> ср, 20 сент. 2017 г. в 6:37, ANJANEYA PRASAD NIDUBROLU <
> anjaney...@gmail.com>:
>
>> Any luck with my query? What am i missing? Why REST response is blank
>> though the cache has data, i tried scan on visor i can see the data over
>> there.
>>
>> Thanks,
>> Anji.
>> On 19 Sep 2017 00:26, "ANJANEYA PRASAD NIDUBROLU"
>> wrote:
>>
>>> Hello All,
>>>
>>> Hope you are doing great!.
>>>
>>> I have tried Ignite's REST API via postman. It is not throwing any
>>> errors, but the response json's value part has nothing in it.
>>>
>>> Here I am pasting cache config (piece of xml file), bean class and main
>>> class where I am saving the sparkRDD to cache. Also, the attached document
>>> has REST requests and responses along with respective logs.
>>>
>>> As the Spark RDD/ DF I am using has more columns, i have created scala
>>> bean class so that I can save it on to IgniteCache as Key, Value.
>>>
>>> Ignite server and clients are able to talk to each other. Cache is
>>> created and loaded successfully.
>>> So far so good, trouble started when I am trying to trigger from REST
>>> API (the attached notepad has REST APIs i tried to test and their response).
>>>
>>> 1) Though the Bean class I created has 8 columns - the cache created has
>>> only 7 columns, what happened to final one? [even "*cache -c=<> -scan*"
>>> command from "*visor*" results has 7 columns.]
>>> 2) The REST API responses says it is success but the response json's
>>> value part is empty.
>>>
>>> Not sure what went wrong. Happy to provide more details if required.
>>> Many Thanks,
>>> Anji.
>>>
>>> *ignite-config.xml*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> =
>>> *College.scala*
>>>
>>> package org.anjaneya.prasad.loadbean
>>> import scala.beans.BeanProperty
>>>
>>> class College(@BeanProperty register_number :String,
>>> @BeanProperty current_city: String,
>>> @BeanProperty date2: String,
>>> @BeanProperty date_of_birth: String,
>>> @BeanProperty student_code: String,
>>> @BeanProperty native_city: String,
>>> @BeanProperty college_end_date_1: String,
>>> @BeanProperty college_start_date_1: String
>>> ) extends Serializable{
>>> override def toString: String = s"College: $register_number,
>>> $current_city, $date2, $date_of_birth, $student_code, $native_city,
>>> $college_end_date_1, college_start_date_1"
>>>
>>> //return format("%s, %s, %s, %s, %s, %s, %s, %s", register_number ,
>>> native_city , current_city , student_code,college_end_date_1,
>>> date_of_birth, date2)
>>> }
>>>
>>>
>>>
>>> *MainProcess.scala*
>>>
>>> val ic = new IgniteContext(sc,
>>> "/home/ops/College/src/main/resources/ignite-config.xml",
>>> true)
>>>
>>> var sharedRDDCollege: IgniteRDD[String, College] =
>>> ic.fromCache("CollegeCache")
>>> //sharedRDDCollege.collect().foreach(print)
>>>
>>> var CollegeCache = test2.rdd.map(x => (x.getString(0),
>>>new College(x.getString(0) , x.getString(1) , x.getString(2) ,
>>> x.getString(3) , x.getString(4) , x.getString(5) , x.getString(6) ,
>>> x.getString(7
>>>
>>> //CollegeCache.collect.foreach(print)
>>> sharedRDDCollege.savePairs(CollegeCache)
>>>
>>>
exception in transaction, how to rollback all put?
In the code, tx.commit() is not reached. Is there a way to rollback the
cache.put("test1", "1"); cache.put("test2", "2");
public IgniteTest() {
CacheConfiguration cacheCfg = new CacheConfiguration();
cacheCfg.setName("cacheName");
cacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setCacheConfiguration(cacheCfg);
TransactionConfiguration txCfg = new TransactionConfiguration();
cfg.setTransactionConfiguration(txCfg);
cfg.setClientMode(true);
Ignite ignite = Ignition.start(cfg);
transactions = ignite.transactions();
cache = ignite.getOrCreateCache("txn test cache");
}
public void run() {
Transaction tx =
transactions.txStart(TransactionConcurrency.OPTIMISTIC,
TransactionIsolation.SERIALIZABLE);
try{
cache.put("test1", "1");
cache.put("test2", "2");
String[] bug = null;
bug[0] = "hello";
cache.put("test3", "3");
System.out.println("tx.commit");
tx.commit();
}catch (Exception ex) {
ex.printStackTrace();
System.out.println("tx.rollback");
tx.rollback();
}
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: WAL log folder issue in YARN mode
Hi Ray, Sure you can manage an Ignite cluster with the persistence enabled in YARN. Change a path for storage files and its WALs using PersistentStoreConfiguration. This example shows how to achieve this: https://apacheignite.readme.io/docs/durable-memory-tuning#section-separate-disk-device-for-wal Please let me know if the suggestion works, I’ll update the YARN deployment configuration on Ignite side. — Denis > On Sep 20, 2017, at 7:40 AM, Ray wrote: > > Hi Ilya, > > I want to use YARN to launch Ignite because YARN is easy to manage. > And there's other service in our cluster and I want to use YARN as a unified > resource manager. > > So what you're telling me is that if I deploy Ignite on YARN there's no way > to utilize the native persistent store feature, right? > > > I think if user want to deploy Ignite on YARN, it's only reasonable to move > the Ignite native persistent store from local file system to HDFS. > Otherwise Ignite on YARN lacks this feature and might be very important to a > lot of users. > > Thanks > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: exception in transaction, how to rollback all put?
Your code looks correct, tx.rollback() will rollback the whole transaction. Is there anything in particular that is not working? -Val -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: exception in transaction, how to rollback all put?
the test.print() before test.run() will printout three nulls, which is
correct.
However, after test.run(), the print in finally block will print out: 1 2
null, which means cache.put("test1", "1"); cache.put("test2", "2"); are
successfully written into cache, even though the exception happens. I
thought the tx.rollback() should revert back the values of "test1" and
"test2". It would expect to print out 3 nulls after tx.rollback().
null
null
null
java.lang.NullPointerException
at test.IgniteTest.run(IgniteTest.java:48)
at test.IgniteTest.main(IgniteTest.java:71)
tx.rollback
1
2
null
public void print() {
System.out.println(cache.get("test1"));
System.out.println(cache.get("test2"));
System.out.println(cache.get("test3"));
}
public static void main(String[] args) {
IgniteTest test = new IgniteTest();
test.print();
System.out.println("");
try {
test.run();
} catch (Exception ex) {
} finally {
test.print();
}
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ignite Context : java.lang.NoSuchMethodError: org.apache.curator.framework.api.CreateBuilder.creatingParentContainersIfNeeded
Hitting the following exception when starting the Ignite context. Here is the snippet of the code. Also double checked that no other curator jars are present in the class path except from the one from ignite. Here are the versions I am using. Apache Ignite : 2.2.0 Apache Spark: 2.1.0 Can someone help with this error? Thanks !! Also I am hitting this error only when I try to launch IgniteContext. When I try to launch Ignite in client mode (Ignite.start()) I don't hit this error. In both cases I use TcpDiscoveryZookeeperIpFinder as ipFinder. 17/09/20 22:47:08 ERROR internal.IgniteKernal: Got exception while starting (will rollback startup routine). java.lang.NoSuchMethodError: org.apache.curator.framework.api.CreateBuilder.creatingParentContainersIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModePathAndBytesable; at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalRegisterService(ServiceDiscoveryImpl.java:222) at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.registerService(ServiceDiscoveryImpl.java:188) at org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder.registerAddresses(TcpDiscoveryZookeeperIpFinder.java:237) at org.apache.ignite.spi.discovery.tcp.ipfinder.TcpDiscoveryIpFinderAdapter.initializeLocalAddresses(TcpDiscoveryIpFinderAdapter.java:61) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryImpl.registerLocalNodeAddress(TcpDiscoveryImpl.java:317) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:341) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1834) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:842) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1786) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:978) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:536) at org.apache.ignite.Ignition.getOrStart(Ignition.java:414) at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:143) at org.apache.ignite.spark.IgniteContext$$anonfun$1.apply(IgniteContext.scala:54) at org.apache.ignite.spark.IgniteContext$$anonfun$1.apply(IgniteContext.scala:54) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
