Re: grouped index sort vs filter

[Sergi Vladykin]

Actually we allow to enforce join order in SQL (see [1]). I hope in the
next release we will support index hints as well (they are already merged
to H2 master).

[1]
http://ignite.apache.org/releases/1.8.0/javadoc/org/apache/ignite/cache/query/SqlFieldsQuery.html#setEnforceJoinOrder(boolean)

Sergi

2017-01-27 5:40 GMT+03:00 lawrencefinn :

> Sorry for the delay, been working hard on this.  We have changed the data
> model a lot but here it is from the git history:
> Here is the class: LineItem - http://pastebin.com/hTuXa5E3
>
> The query is something like select li._val from lineitem li where member_id
> = 123 order by member_id, name asc.
>
> Our solution around this works but is a bit of a pain in the butt.  We
> created "fake" boolean fields/keys for each of the composite keys with the
> value of false.  So in this example we created a key called "f_name".  We
> updated the composite key "member_name" to actually take in 3 fields -
> member_id, f_name, and name.  Then our query becomes something like
> select li._val from lineitem li where member_id = 123 and f_name = false
> order by member_id, f_name, name
>
> Including the fake field in the where clause forces the correct index to be
> used.  We actually did this to support descending and ascending indexes for
> the same fields too.
>
> Here is an example of what it looks like in the end:
> @QuerySqlField(index = true, name = "member_id", orderedGroups = {
> @QuerySqlField.Group(name = "member_name", order = 0),
> @QuerySqlField.Group(descending = true, name =
> "member_name_desc", order = 0),
> })
> private Long memberId;
>
> @QuerySqlField(index = true, orderedGroups = {
> @QuerySqlField.Group(name = MEMBER_NAME, order = 2),
> @QuerySqlField.Group(descending = true, name =
> "member_name_desc", order = 2)
> })
> private String name;
> @QuerySqlField(orderedGroups = {@QuerySqlField.Group(name =
> MEMBER_NAME,
> order = 1)})
> protected boolean f_name = false;
> @QuerySqlField(orderedGroups = {@QuerySqlField.Group(descending =
> true,
> name = "member_name_desc", order = 1)})
> protected boolean f_name_desc = false;
>
>
> I REALLY wish H2 had the ability to force join order or hint indexes.  It
> looks like that is WAY out in their roadmap :(
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/grouped-index-sort-vs-filter-tp9885p10277.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: insert/update/delete issue in Version 1.8

[Andrey Novikov]

Denis,

Ticket https://issues.apache.org/jira/browse/IGNITE-4622 was fixed in master
branch.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/insert-update-delete-issue-in-Version-1-8-tp10244p10280.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: backup to swap

[Anil]

Thanks Val.

That feature does not exist for now :). and we cannot use any persistence
store for our usecase.

it will be very good feature. Can we raise a feature request for the same ?
Thanks.



On 27 January 2017 at 00:43, vkulichenko 
wrote:

> Hi Anil,
>
> Take a look at this discussion:
> http://stackoverflow.com/questions/40403598/can-you-
> evict-ignite-cache-backups-to-disk
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/backup-to-swap-tp10255p10266.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: ApacheCon CFP closing soon (11 February)

[Denis Magda]

Igniters,

If you’re use Apache projects (not only Ignite) for IoT purposes please go 
ahead and submit your IoT related talks to ApacheCon and IoTFuse.

 - 3 Feb: Submit a presentation proposal for IoTFuse 
https://iotfuse.com/call-for-proposals-2017-iotfuse-conference/ 


 - 11 Feb: Submit a presentation proposal for ApacheCon 
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp 



—
Denis

> On Jan 20, 2017, at 12:19 PM, Dmitriy Setrakyan  wrote:
> 
> I have submitted for a couple of talks myself.
> 
> I would like to encourage everyone in Ignite community, especially if you
> don't mind visiting Miami, to submit a proposal. The topic may include
> Ignite architecture, projects based on top of Ignite, as well as real
> production use cases and deployments.
> 
> Here is the link to submit speaking proposals:
> http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp
> 
> D.
> 
> On Thu, Jan 19, 2017 at 4:58 PM, Denis Magda  wrote:
> 
>> + user list
>> 
>> 
>>> On Jan 19, 2017, at 3:07 PM, Denis Magda  wrote:
>>> 
>>> Igniters,
>>> 
>>> Is there anyone of you who want to attend this conference as a speaker
>> presenting Apache Ignite in some form or sharing your working experience
>> with it?
>>> 
>>> —
>>> Denis
>>> 
 Begin forwarded message:
 
 From: Rich Bowen 
 Subject: ApacheCon CFP closing soon (11 February)
 Date: January 18, 2017 at 8:45:41 AM PST
 To: comdev 
 Reply-To: d...@ignite.apache.org
 Reply-To: comdev 
 
 Hello, fellow Apache enthusiast. Thanks for your participation, and
 interest in, the projects of the Apache Software Foundation.
 
 I wanted to remind you that the Call For Papers (CFP) for ApacheCon
 North America, and Apache: Big Data North America, closes in less than a
 month. If you've been putting it off because there was lots of time
 left, it's time to dig for that inspiration and get those talk
>> proposals in.
 
 It's also time to discuss with your developer and user community whether
 there's a track of talks that you might want to propose, so that you
 have more complete coverage of your project than a talk or two.
 
 We're looking for talks directly, and indirectly, related to projects at
 the Apache Software Foundation. These can be anything from in-depth
 technical discussions of the projects you work with, to talks about
 community, documentation, legal issues, marketing, and so on. We're also
 very interested in talks about projects and services built on top of
 Apache projects, and case studies of how you use Apache projects to
 solve real-world problems.
 
 We are particularly interested in presentations from Apache projects
 either in the Incubator, or recently graduated. ApacheCon is where
 people come to find out what technology they'll be using this time next
 year.
 
 Important URLs are:
 
 To submit a talk for Apache: Big Data -
 http://events.linuxfoundation.org/events/apache-big-data-
>> north-america/program/cfp
 To submit a talk for ApacheCon -
 http://events.linuxfoundation.org/events/apachecon-north-
>> america/program/cfp
 
 To register for Apache: Big Data -
 http://events.linuxfoundation.org/events/apache-big-data-
>> north-america/attend/register-
 To register for ApacheCon -
 http://events.linuxfoundation.org/events/apachecon-north-
>> america/attend/register-
 
 Early Bird registration rates end March 12th, but if you're a committer
 on an Apache project, you get the low committer rate, which is less than
 half of the early bird rate!
 
 For further updated about ApacheCon, follow us on Twitter, @ApacheCon,
 or drop by our IRC channel, #apachecon on the Freenode IRC network. Or
 contact me - rbo...@apache.org - with any questions or concerns.
 
 Thanks!
 
 Rich Bowen, VP Conferences, Apache Software Foundation
 
 --
 (You've received this email because you're on a dev@ or users@ mailing
 list of an Apache Software Foundation project. For subscription and
 unsubscription information, consult the headers of this email message,
 as this varies from one list to another.)
>>> 
>> 
>> 

Re: grouped index sort vs filter

[lawrencefinn]

Sorry for the delay, been working hard on this.  We have changed the data
model a lot but here it is from the git history:
Here is the class: LineItem - http://pastebin.com/hTuXa5E3 

The query is something like select li._val from lineitem li where member_id
= 123 order by member_id, name asc.

Our solution around this works but is a bit of a pain in the butt.  We
created "fake" boolean fields/keys for each of the composite keys with the
value of false.  So in this example we created a key called "f_name".  We
updated the composite key "member_name" to actually take in 3 fields -
member_id, f_name, and name.  Then our query becomes something like
select li._val from lineitem li where member_id = 123 and f_name = false
order by member_id, f_name, name

Including the fake field in the where clause forces the correct index to be
used.  We actually did this to support descending and ascending indexes for
the same fields too.

Here is an example of what it looks like in the end:
@QuerySqlField(index = true, name = "member_id", orderedGroups = {
@QuerySqlField.Group(name = "member_name", order = 0),
@QuerySqlField.Group(descending = true, name =
"member_name_desc", order = 0),
})
private Long memberId;

@QuerySqlField(index = true, orderedGroups = {
@QuerySqlField.Group(name = MEMBER_NAME, order = 2),
@QuerySqlField.Group(descending = true, name =
"member_name_desc", order = 2)
})
private String name;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = MEMBER_NAME,
order = 1)})
protected boolean f_name = false;
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(descending = true,
name = "member_name_desc", order = 1)})
protected boolean f_name_desc = false;


I REALLY wish H2 had the ability to force join order or hint indexes.  It
looks like that is WAY out in their roadmap :(



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/grouped-index-sort-vs-filter-tp9885p10277.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Fetch by contains on a field as Array or List

[javastuff....@gmail.com]

Hi,

I have an object with one of the field as Array or List. Array/List field
holds Long values. Size could be 10s or 100s or few 1000s, not expecting it
to grow more than 5000. Object won't be updated frequently, mostly read
only.

Object implements Binarylizable.

Requirement is to fetch all the cached objects where Array/List field
contains a give value. Is there any type of SQL query I can use to achieve
this usecase? 

Thanks,
-Sam



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Fetch-by-contains-on-a-field-as-Array-or-List-tp10276.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: eagerTtl

[vkulichenko]

Surprisingly, the name of the thread is 'ttl-cleanup-worker' :) It's started
only if there is at least one cache using it.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/eagerTtl-tp10249p10275.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Newbie: Questions on Ignite over cassandra

[vkulichenko]

I meant that Cassandra itself will be involved only when you load the data
into caches, which is a separate step that should happen prior to query
execution. When Ignite query is executed, Cassandra is not touched.

The answer on your question is yes - any joins are possible, similar to any
relational database. However, for good performance you should consider
collocation and indexing. See the documentation for details:
https://apacheignite.readme.io/docs/sql-grid

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Newbie-Questions-on-Ignite-over-cassandra-tp10264p10274.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: eagerTtl

[javastuff....@gmail.com]

Hi Val,

JavaDoc does not talk about threads per cache or node. To get more info on
this property, came across below post, I see similar memory and thread
pattern with my application but not yet investigated specifically for
eagerTtl or its related threads.

http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-High-Memory-Overhead-caused-by-GridCircularBuffer-Item-instances-td3682.html#a3691

Can you please point to thread name related to egarTtl which I can take a
look at in thread dump?

Secondly, If none of the cache is configured for expiry, will this thread be
present and running? If it s a single thread per node then I am not worried. 

Thank,
-Sam




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/eagerTtl-tp10249p10272.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: OFFHEAP_TIERED and memory fragmentation

[Peter Schmitt]

Hi Alexey,

a modern OS won't give you a physical address. Instead you get virtual
memory which allows the OS to support features like defragmentation.
"fine-grained per-entry" sounds like a byte-array for every entry.
We tested something like that and the result is again higher GC pauses.
In our case a FGC with off-heap caches takes e.g. up to 1s.
With the same data and using on-heap caches it takes up to 25s.
With a byte-array per entry we get about 4s for a FGC.
With some optimizations you might get down a bit, however, in view of the
FGC time I fear that such an approach causes a significant impact for
applications which need to keep huge caches.

Kind regards,
Peter



2017-01-26 15:57 GMT+01:00 Alexey Goncharuk :

> Hi Peter,
>
> Leaving defragmentation to Ignite is one of the reasons we are trying
> PageMemory approach. In Ignite 1.x we basically use OS memory allocator to
> place a value off-heap. Once the OS has given us a pointer, the memory
> cannot be moved around unless we free this region, thus the fragmentation.
> On the other hand, with PageMemory approach we can define the page layout
> ourselves and, given fine-grained per-entry locking, move values around in
> memory. This makes memory management more complicated and usually defines
> some restrictions on the data structures, but we expect this will give us
> more control over the data.
>
> For example, one of the tricks we can do in the future is cluster-wide
> cache snapshots: we simply dump the continuous region of memory on disk and
> get a node-local caches snapshot. This is impossible to do using current
> off-heap approach.
>
> --AG
>
> 2017-01-25 12:32 GMT+03:00 Peter Schmitt :
>
>> Hi Val,
>>
>> fair enough :-)
>> However, in the source-code it looks like "Flexible and precise
>> per-cache memory limit" is supported already.
>> At least each GridUnsafeMemory instance (in each GridCacheContext)
>> receives the value from CacheConfiguration.getOffHeapMaxMemory).
>> I haven't debugged it, but the instances are created per cache.
>>
>> Furthermore, I'm not sure about the memory-page concept, because you
>> would get pages on top of pages (used by a modern OS).
>> Sounds like even more fragmentation and in this case Ignite would need to
>> do the defragmentation.
>>
>> Kind regards,
>> Peter
>>
>>
>>
>> 2017-01-25 1:56 GMT+01:00 vkulichenko :
>>
>>> I don't think I will be able to explain better than it's done in the
>>> ticket
>>> :) Current approach is naive and not always effective for multiple
>>> reasons.
>>>
>>> -Val
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-ignite-users.705
>>> 18.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp1
>>> 0218p10235.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>
>>
>>
>

Re: How to get On heap memory usage for a map?

[hitendrapratap]

I have followed http://apacheignite.gridgain.org/v1.5/docs/capacity-planning
for capacity planning. But I have a question regarding "Add around 20MB per
cache." line.  Does it mean per cache entry, per node, per map?

I have inserted only 1 entry in the cluster. Following are the stats:

Size of 1 entry =108KB
1 Node internal memory = 200-300MB. So we have 4 servers and 2 clients.
4 Node internal memory = 300MB*4 = 1.2GB
Not sure what does "Add around 20MB per cache" means?

Heap memory used across the cluster = 11745159288 = 11GB
[igniteInstance.cluster().metrics() -> ClusterMetrics.getHeapMemoryUsed]

I am trying to understand why the heap memory is 11GB? 

Finally I want to measure amount of heap memory (ONHEAP) in bytes used by
this map if insert 1million entries. 

I was looking into igniteInstance.cache(mapName).metrics() which return
CacheMetrics for that mapName. 




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-get-On-heap-memory-usage-for-a-map-tp10263p10270.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Ignite 1.7.0 Session caching issue with update session attributes

[vkulichenko]

Binary marshaller is default internal format, there is no need to set it
explicitly in configuration. If you remove the 'marshaller' property from
configuration, it will be used. If it doesn't work this way as well, please
attach configuration and the exception trace.

-Val





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Apache-Ignite-1-7-0-Session-caching-issue-with-update-session-attributes-tp10220p10269.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Newbie: Questions on Ignite over cassandra

[vkulichenko]

Hi,

Please properly subscribe to the mailing list so that the community can
receive email notifications for your messages. To subscribe, send empty
email to user-subscr...@ignite.apache.org and follow simple instructions in
the reply.


Jenny B. wrote
> I am exploring Apache Ignite on top of Cassandra as a possible tool to be
> able to give ad-hoc queries on cassandra tables. Using Ignite is it
> possible to able to search or query on any column in the underlying
> cassandra tables, like a RDBMS? Or can the join columns and search columns
> only be partition and clustering columns ?
> 
> If using Ignite, is there still need to create indexes on cassandra ? Also
> how does ignite treat materialized views ? Will there be a need to create
> materialized views ?
> 
> Also any insights into how updates to cassandra release can/will be
> handled by Ignite would be very helpful.

If you execute in-memory SQL queries using Ignite API, you have to load the
data from the store first:
https://apacheignite.readme.io/docs/data-loading#ignitecacheloadcache

Read-through works only for key based access. With queries you don't know
set of required keys in advance, thus only data which is already in memory
is used to execute them.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Newbie-Questions-on-Ignite-over-cassandra-tp10264p10268.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: How to get On heap memory usage for a map?

[vkulichenko]

What do you mean by usage? Which metrics are you looking for exactly?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-get-On-heap-memory-usage-for-a-map-tp10263p10267.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: backup to swap

[vkulichenko]

Hi Anil,

Take a look at this discussion:
http://stackoverflow.com/questions/40403598/can-you-evict-ignite-cache-backups-to-disk

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/backup-to-swap-tp10255p10266.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Ignite 1.7.0 Session caching issue with update session attributes

[Siva Annapareddy]

We were using default set up for Web Sessions caching in Groovy
Application. No specific Ignite code except reading default-config.xml that
specifies :
Cache Name
Ignite Cluster IP Addresses to be discovered

Under Spring Security A session created is cached in Ignite JVMs. After
Authentication, a new session is created and Session gets updated. During
Session update, Application puts some Groovy objects in Session Object.
Groovy Object contains fields without types specified. Some of the types
seemed to have been implemented as LazyMap that is not serializable. Hence
the Session stored on Ignite JVMs are not updated. Next time Application
references the Session, it would not find necessary data in the Session
retrived from Ignite JVMs



On Thu, Jan 26, 2017 at 1:00 PM, Denis Magda-2 [via Apache Ignite Users] <
ml-node+s70518n10261...@n6.nabble.com> wrote:

> Binary Marshaller is used by default. So, don’t override marshaller in
> your configuration.
>
> By the why are you trying to serialize this LazyMap? How does Ignite code
> look like?
>
> -
> Denis
>
> > On Jan 26, 2017, at 6:57 AM, Siva Annapareddy <[hidden email]
> > wrote:
> >
> > Since OptimizedMarshaller is default we have run into issues:
> >
> > End of stream reached: java.io.ByteArrayInputStream@438f0960.
> Stacktrace
> > follows:
> > Message: Failed to unmarshal object with optimized marshaller
> >
> > Hence, tried with jdkmarshaller and ran into Object Not Serializable
> >
> > We understand that Groovy 2.5.4 the default underlying Map  is LazyMap
> and
> > that is not Serializable.
> >
> > We did not find org.apache.ignite.marshaller.binary.BinaryMarshaller in
> > Ignite 1.7.0. Am I missing something?
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Apache-Ignite-1-7-0-Session-caching-
> issue-with-update-session-attributes-tp10220p10257.html
> > Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Apache-
> Ignite-1-7-0-Session-caching-issue-with-update-session-
> attributes-tp10220p10261.html
> To unsubscribe from Apache Ignite 1.7.0 Session caching issue with update
> session attributes, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Apache-Ignite-1-7-0-Session-caching-issue-with-update-session-attributes-tp10220p10265.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

How to get On heap memory usage for a map?

[hitendrapratap]

How to get On heap memory usage for a map? I have looked into CacheMetrics
but it doesn't have this info. Is there any other way to get the same? 



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/How-to-get-On-heap-memory-usage-for-a-map-tp10263.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: insert/update/delete issue in Version 1.8

[Denis Magda]

Please see below 

> On Jan 26, 2017, at 8:56 AM, minisoft_rm  wrote:
> 
> Hi Denis, thank you. it works.
> 
> I manually... added these line... something like "
> 
> Set keyFlds = new HashSet<>();
> keyFlds.add("pk");
> qryEntity.setKeyFields(keyFlds);
> 
> "
> 
> after that , no more exception there. I am wondering... so I reviewed the
> "model" section of web console. it generates everything for us, right? and I
> found it is a little strange...  
> 
> in WEB CONSOLE:
> the auto generated code is "
> qryEntity.setKeyFields(new JdbcTypeField(Types.BIGINT, "PK",
> long.class, "pk"));
> "
> in DOWNLOADED zip file, code is "
> type.setKeyFields(new JdbcTypeField(Types.BIGINT, "PK", long.class,
> "pk"));
> " the type is "JdbcType"... not the "qryEntity"
> 
> so is it a minor bug?
> 
> 
Yes, this is a bug. Thanks for reporting. The ticket is ready:
https://issues.apache.org/jira/browse/IGNITE-4622 


Andrey N, Alex K, please take a look at it.

> 
> after that, May I ask one more question the update sql returns no more
> exception, it is good. But, the data doesn't be udpatedin ignite and
> Db... do you have any comments? thanks :-)
> 
> 

This sounds weird and unbelievable :) Please share the code snippet that does 
the update and that validates that the data has been updated.

—
Denis

> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/insert-update-delete-issue-in-Version-1-8-tp10244p10260.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Apache Ignite 1.7.0 Session caching issue with update session attributes

[Denis Magda]

Binary Marshaller is used by default. So, don’t override marshaller in your 
configuration.

By the why are you trying to serialize this LazyMap? How does Ignite code look 
like?

-
Denis

> On Jan 26, 2017, at 6:57 AM, Siva Annapareddy  
> wrote:
> 
> Since OptimizedMarshaller is default we have run into issues:
> 
> End of stream reached: java.io.ByteArrayInputStream@438f0960. Stacktrace
> follows:
> Message: Failed to unmarshal object with optimized marshaller
> 
> Hence, tried with jdkmarshaller and ran into Object Not Serializable 
> 
> We understand that Groovy 2.5.4 the default underlying Map  is LazyMap and
> that is not Serializable.
> 
> We did not find org.apache.ignite.marshaller.binary.BinaryMarshaller in
> Ignite 1.7.0. Am I missing something?
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Apache-Ignite-1-7-0-Session-caching-issue-with-update-session-attributes-tp10220p10257.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: insert/update/delete issue in Version 1.8

[minisoft_rm]

Hi Denis, thank you. it works.

I manually... added these line... something like "

Set keyFlds = new HashSet<>();
keyFlds.add("pk");
qryEntity.setKeyFields(keyFlds);

"

after that , no more exception there. I am wondering... so I reviewed the
"model" section of web console. it generates everything for us, right? and I
found it is a little strange...  

in WEB CONSOLE:
 the auto generated code is "
 qryEntity.setKeyFields(new JdbcTypeField(Types.BIGINT, "PK",
long.class, "pk"));
"
in DOWNLOADED zip file, code is "
 type.setKeyFields(new JdbcTypeField(Types.BIGINT, "PK", long.class,
"pk"));
" the type is "JdbcType"... not the "qryEntity"

so is it a minor bug?



after that, May I ask one more question the update sql returns no more
exception, it is good. But, the data doesn't be udpatedin ignite and
Db... do you have any comments? thanks :-)




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/insert-update-delete-issue-in-Version-1-8-tp10244p10260.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Reloading from Persistent Store after Losing a Node

[Cody Yancey]

Hello Ignite users!

I have a use case where I am doing SQL queries on a sharded cache, and I
need to ensure that SQL queries always return The Right Answer even if some
nodes in the ring are lost. As I have rigorously confirmed, SQL queries
only apply to data in the cache (as opposed to in the write-through
persistent store but lost from the cache). Also, when you lose a node, you
don't lose persisted data, but data IS now gone from the cache (unless
there is an in-cache backup of the relevant cache partitions).

Now, I *could* do this by just increasing the backup factor for the cache
equal to the number of nodes I can stand to lose, and then setting a
TopologyValidator on the cache to ensure I always have more nodes in the
ring than that number. If the TopologyValidator ever returns a number of
nodes below this survivability threshold, I crash the app and let
everything get reloaded from the persistent store when the nodes
automatically start back up.

This technique has a lot of false positives, where we lose too many nodes,
but slowly enough that Ignite is well-able to shift the data around to
avoid data loss and so we shouldn't have had to crash the app.

Therefore, I would rather be a little smarter about this for the sake of
uptime.

Ideally, in the TopologyValidator logic, while reads and writes to the
cache are blocked, I would be able to:

1.) Detect when a lost partition has no viable backup,
2.) Reload from the persistent store.

The problem I am facing is, I can't find a clean and efficient way of
figuring out #1 from the information the ToplogyValidator gives you.

And even if I could, #2 hangs forever, which makes sense because the cache
isn't readable or writeable until AFTER the topology has been validated.

Has anyone faced a similar challenge and has some wisdom to share? Am I
making this way more complicated than it needs to be?

Thanks in advance,
Cody

Re: OFFHEAP_TIERED and memory fragmentation

[Alexey Goncharuk]

Hi Peter,

Leaving defragmentation to Ignite is one of the reasons we are trying
PageMemory approach. In Ignite 1.x we basically use OS memory allocator to
place a value off-heap. Once the OS has given us a pointer, the memory
cannot be moved around unless we free this region, thus the fragmentation.
On the other hand, with PageMemory approach we can define the page layout
ourselves and, given fine-grained per-entry locking, move values around in
memory. This makes memory management more complicated and usually defines
some restrictions on the data structures, but we expect this will give us
more control over the data.

For example, one of the tricks we can do in the future is cluster-wide
cache snapshots: we simply dump the continuous region of memory on disk and
get a node-local caches snapshot. This is impossible to do using current
off-heap approach.

--AG

2017-01-25 12:32 GMT+03:00 Peter Schmitt :

> Hi Val,
>
> fair enough :-)
> However, in the source-code it looks like "Flexible and precise per-cache
> memory limit" is supported already.
> At least each GridUnsafeMemory instance (in each GridCacheContext)
> receives the value from CacheConfiguration.getOffHeapMaxMemory).
> I haven't debugged it, but the instances are created per cache.
>
> Furthermore, I'm not sure about the memory-page concept, because you would
> get pages on top of pages (used by a modern OS).
> Sounds like even more fragmentation and in this case Ignite would need to
> do the defragmentation.
>
> Kind regards,
> Peter
>
>
>
> 2017-01-25 1:56 GMT+01:00 vkulichenko :
>
>> I don't think I will be able to explain better than it's done in the
>> ticket
>> :) Current approach is naive and not always effective for multiple
>> reasons.
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-
>> tp10218p10235.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Apache Ignite 1.7.0 Session caching issue with update session attributes

[Siva Annapareddy]

Since OptimizedMarshaller is default we have run into issues:

End of stream reached: java.io.ByteArrayInputStream@438f0960. Stacktrace
follows:
Message: Failed to unmarshal object with optimized marshaller

Hence, tried with jdkmarshaller and ran into Object Not Serializable 

We understand that Groovy 2.5.4 the default underlying Map  is LazyMap and
that is not Serializable.

We did not find org.apache.ignite.marshaller.binary.BinaryMarshaller in
Ignite 1.7.0. Am I missing something?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Apache-Ignite-1-7-0-Session-caching-issue-with-update-session-attributes-tp10220p10257.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: IGFS Questions

[Vladimir Ozerov]

Hi,

1. Durability depends on IGFS mode. In PRIMARY there is no durability, In
other modes IGFS will propagate writes to underlying file system (e.g. to
HDFS).
2. Files stored in IGFS are always partitioned. You can specify number of
backups and/or REPLICATED mode in data cache configuration.
3. Yes, as long as you have Hadoop-compliant implementation of S3 file
system (e.g. org.apache.hadoop.fs.s3.S3FileSystem).
4. You can configure evictions from data cache. Please refer to
org.apache.ignite.cache.eviction.igfs.IgfsPerBlockLruEvictionPolicy class.
5. Underlying file system must be shared between all nodes in cluster. If
it is true, then you can use
org.apache.ignite.igfs.secondary.local.LocalIgfsSecondaryFileSystem.
6. You can define pinned files before node start. But you cannot change it
in runtime. Please refer to IgfsPerBlockLruEvictionPolicy.setExcludePaths()
method.
7. PROXY mode doesn't cache any data, but simply delegates to secondary
file system. It is useful when you do not want to cache certain part of
data at all. For example, if you access data only once.
8. No. Write-behind speeds up user operations at the cost of consistency
guarantees.


On Tue, Jan 24, 2017 at 5:00 PM, pragmaticbigdata 
wrote:

> I have some questions of deploying IGFS as a cache layer given that ignite
> could be deployed both as a key-value store and as a file system
>
> 1. How does IGFS behave when deployed in standalone mode? I wanted to
> confirm that there is no durability in this mode. Assuming I persist a
> parquet file on IGFS, if the cluster goes down I lose the file, right?
> 2. Do we get the ability to specify the fact that the file stored in IGFS
> could be both partitioned (with backup nodes) or replicated?
> 3. IGFS can act as a cache layer over HDFS and local file system, can it
> act
> as a caching layer over S3 store?
> 4. As with the key-value store, can I configure tiered storage in IGFS i.e.
> given that IGFS is configured with local file system as the secondary store
> and the ignite cluster of 3 server nodes configured with 5GB memory each,
> would the data spill over to the local disk if I try to load a 25GB file
> into IGFS? If so, what is the configuration needed?
> 5. Can I configure local SSD disks as the secondary store for IGFS?
> 6. I browsed through the documentation but I didn't find the capability of
> pinning and unpinning files in IGFS. I am looking something similar to what
> alluxio
>  Alluxio.html#pinning-files>
> provides. Can it be implemented?
> 7. Could you elaborate a bit on how IGFS Proxy Mode works? What is its
> recommended use case?
> 8. With DUAL_ASYNC (write-behind mode), does ignite have failover
> guarantees
> which is lacking in the key-value store -
> https://issues.apache.org/jira/browse/IGNITE-1897?
>
> Thanks in advance.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/IGFS-Questions-tp10217.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>