[jira] [Created] (IGNITE-11335) Data Center Affinity on Reads for Performance Increase
Gabriel Jimenez created IGNITE-11335: Summary: Data Center Affinity on Reads for Performance Increase Key: IGNITE-11335 URL: https://issues.apache.org/jira/browse/IGNITE-11335 Project: Ignite Issue Type: Improvement Reporter: Gabriel Jimenez *Problem Statement:* When working with an ignite grid deployed across multiple data centers, a functionality could not be identified that would guarantee no unnecessary cross data center communication on read requests. *Solution:* We decided to add filtering logic to GridCacheContext selectAffinityNodeBalanced(...) dependent on an expected node attribute - 'DATA_CENTER' (no functional change if attribute not present). The logic attempts to filter the incoming 'affNodes' parameter so that it includes only nodes whose attribute value match the local node's attribute value, maintaining the original nodes if none match. Relevant Function: https://github.com/apache/ignite/blob/ignite-2.6/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheContext.java#L2190 *Additional Questions:* Was there an existing solution/approach to our '*Problem Statement'* that did not involve changing the codebase? Is there another preferred solution for our '*Problem Statement'*? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: New committer: Ilya Lantukh
Thanks! On Fri, Feb 15, 2019 at 3:03 PM Andrey Mashenkov wrote: > Congrats, Ilya. > > On Fri, Feb 15, 2019 at 2:33 PM Anton Vinogradov wrote: > > > Congrats! > > > > On Fri, Feb 15, 2019 at 2:23 PM Павлухин Иван > wrote: > > > > > Ilya, > > > > > > My congratulations! Hope to hear from you about hair-splitting in > > > context of affinity and topology. > > > > > > пт, 15 февр. 2019 г. в 13:44, Dmitriy Pavlov : > > > > > > > > Congrats, Ilya. I'm glad that it eventually happened. > > > > > > > > пт, 15 февр. 2019 г. в 11:04, Alexey Goncharuk < > agoncha...@apache.org > > >: > > > > > > > > > Dear Ignite Developers, > > > > > > > > > > > > > > > > > > > > The Project Management Committee (PMC) for Apache Ignite has > invited > > > Ilya > > > > > Lantukh to become a committer and we are pleased to announce that > he > > > has > > > > > accepted. > > > > > > > > > > > > > > > > > > > > Ilya has a long history contributing to Apache Ignite, including a > > lot > > > of > > > > > complex changes allowed us to enable and support native > persistence. > > > > > > > > > > Being a committer enables easier contribution to the project since > > > there is > > > > > no need to go via the patch submission process. This should enable > > > better > > > > > productivity. > > > > > > > > > > > > > > > > > > > > Please join me in welcoming Ilya, and congratulating him on the new > > > role in > > > > > the Apache Ignite Community. > > > > > > > > > > Best Regards, > > > > > > > > > > Alexey Goncharuk > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > > > -- > Best regards, > Andrey V. Mashenkov >
[jira] [Created] (IGNITE-11334) SQL: Deprecate SqlQuery
Vladimir Ozerov created IGNITE-11334:
Summary: SQL: Deprecate SqlQuery
Key: IGNITE-11334
URL: https://issues.apache.org/jira/browse/IGNITE-11334
Project: Ignite
Issue Type: Task
Components: sql
Reporter: Vladimir Ozerov
Assignee: Taras Ledkov
This API is very limited comparing to {{SqlFieldsQuery}}. Let's deprecate it
with proper links to {{SqlFieldsQuery}}. This should be not only deprecation on
public API, but removal from examples as well.
Separate ticket for documentation is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-11333) SQL: Deprecate H2 console
Vladimir Ozerov created IGNITE-11333: Summary: SQL: Deprecate H2 console Key: IGNITE-11333 URL: https://issues.apache.org/jira/browse/IGNITE-11333 Project: Ignite Issue Type: Task Reporter: Vladimir Ozerov Assignee: Taras Ledkov This functional is not tested, not supported and may fail with unexpected errors. This affects user experience. Need to disable it and create ticket for relevant documentation update. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11332) Add jmx ability to exclude node from topology.
Stanilovsky Evgeny created IGNITE-11332:
---
Summary: Add jmx ability to exclude node from topology.
Key: IGNITE-11332
URL: https://issues.apache.org/jira/browse/IGNITE-11332
Project: Ignite
Issue Type: Improvement
Components: general
Affects Versions: 2.7
Reporter: Stanilovsky Evgeny
Assignee: Stanilovsky Evgeny
Fix For: 2.8
In very rare corner cases may occur situation that physically unreachable host
still present in topology that may leads to hanging overall cluster operations.
Add ability to exclude with specific warning.
{code:java}
/**
* Exclude node from cluster.
*
* @param nodeId Node id.
*/
@MXBeanDescription("Exclude node from cluster.")
public void excludeNode(UUID nodeId);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: beforeTestsStarted() was deprecated, what should we use instead?
Oleg, Still, see beforeTestsStarted is deprecated. Since we found this deprecation was incorrect, we have to roll it back. Please fix the issue. On Thu, Feb 7, 2019 at 5:18 PM oignatenko wrote: > Hi Ivan, > > For the sake of precision, startGrid() is currently overridden in one of > subclasses of GridAbstractTest. This means that we would have to somehow > handle that when changing this method to static. Same thing is with another > solid candidate to making static, stopAllGrids(boolean), which is also > currently overridden in another subclass. > > That said, above issues look relatively minor and manageable and overall I > think it would be better to redesign GridAbstractTest as you suggested. > > regards, Oleg > > > Павлухин Иван wrote > > Hi, > > > > AFAIK various startGrid* methods of GridAbstractTest are not tied to > > concrete instance of test class. I think that a we can make them > > static. Also same can be done for some other methods and all they can > > be extracted into static utility class like GridTestSupport in order > > to reduce GridAbstractTest class size and separate concerns. > > > > What do you think? > > > > чт, 7 февр. 2019 г. в 09:44, oignatenko < > > > oignatenko@ > > > >: > >> > >> Hi Anton, > >> > >> You can start and stop grid from static context required by mentioned > >> annotations using code like: > >> > >> MyTestClass.class.newInstance().startGrid(); > >> > >> This way doesn't look very convenient and I created JIRA task to address > >> that: > >> > >> https://issues.apache.org/jira/browse/IGNITE-11240 > >> > >> Does that address your question? > >> > >> regards, Oleg > >> > >> > >> > >> -- > >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Re: [DISCUSSION] [IGNITE-11141] The new java method to create a cache from a template
Stanislav, thank you for reply!
As a result of discussion, I propose the following plan:
1) Create the new method getCacheConfiguration(String cacheName) for ignite
2.8+ version, which will return the cache template or the cache
configuration.
2) Create the new ticket about changing method names and adding new
property for storing cache templates for ignite 3+ version.
Any comments or additions?
Thank you,
Sergey
чт, 14 февр. 2019 г. в 17:18, :
> Sergey, Ed,
> On method naming/deprecation/etc.
> I would actually like the new method to work for both templates and
> regular caches.
> For templates it would return a copy of the template.
> For existing caches it would return a copy of the cache configuration.
> In other words, it would be a shortcut for
> `new
> CacheConfiguration(ignite.cache("foo").configuration(CacheConfiguration.class))`
> I wouldn't expect it to be widely used but I think this adds some
> uniformity to the behavior.
>
> Ilya,
> On asterisk usage.
> If you have a cache config "foo*" in your static cache configurations
> then you will be able to get that configuration via
> `ignite.cacheConfiguration("foo")` (no asterisk).
> Asterisk is being dropped when the template is created. This is how it
> works now, no changes here.
>
> All,
> Regarding reshuffling all this in 3.0.
> I would
> - change method name addCacheConfiguration to addCacheTemplate
> - would add a new property `CacheConfiguration[] cacheTemplates` to put
> templates to - instead of putting them
> into `cacheConfigutations` with an asterisk
>
> But that's a different topic, I think.
>
> Thanks,
> Stan
>
> > -Original Message-
> > From: Ilya Kasnacheev
> > Sent: Thursday, February 14, 2019 3:32 PM
> > To: dev@ignite.apache.org
> > Subject: Re: [DISCUSSION] [IGNITE-11141] The new java method to create a
> > cache from a template
> >
> > Hello!
> >
> > What about existing convention of using asterisk to mark templates? How
> > does it integerate with this one?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 14 февр. 2019 г. в 13:04, Sergey Moldachev
> > :
> >
> > > Sounds good, I agree with naming and marking *addCacheConfiguration*
> > method
> > > as deprecated with replacing on *addCacheConfigurationTemplate*.
> > >
> > > Stanislav, could you please look at this?
> > >
> > > Thank you,
> > > Sergey
> > >
> > > чт, 14 февр. 2019 г. в 01:49, Eduard Shangareev <
> > > eduard.shangar...@gmail.com
> > > >:
> > >
> > > > CacheConfiguration cfg = ignite.cacheConfiguration("myTemplate");
> > > > cfg.setName("myCacheFromTemplate");
> > > > ignite.createCache(cfg);
> > > >
> > > > Ok, I got it. We already have addCacheConfiguration and
> > > cacheConfiguration
> > > > should be a getter-like counterpart for it.
> > > >
> > > > So, I would suggest deprecating this addCacheConfiguration method and
> > add
> > > > new one Ignite.addCacheTemplate and its counterpart
> > getCacheTemplate.
> > > > Because cacheConfiguration looks very weird, I would expect that it
> > > should
> > > > return cache configuration for any given cache name.
> > > >
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Feb 14, 2019 at 12:01 AM Sergey Moldachev <
> > > > sergeymoldac...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, Eduard, thank you for your reply!
> > > > >
> > > > > You can find example and full description in the Jira task
> > > > > . Also you can
> > > find
> > > > > simple implementation in comments.
> > > > >
> > > > > Thank you,
> > > > > Sergey!
> > > > >
> > > > > + update subject (fixed a typo)
> > > > >
> > > > > ср, 13 февр. 2019 г. в 19:55, Eduard Shangareev <
> > > > > eduard.shangar...@gmail.com
> > > > > >:
> > > > >
> > > > > > Hi, Sergey!
> > > > > >
> > > > > > Could you give some example of how it is supposed to use?
> > > > > >
> > > > > > On Wed, Feb 13, 2019 at 6:02 PM Sergey Moldachev <
> > > > > > sergeymoldac...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi, Igniters!
> > > > > > >
> > > > > > > I want to add the new java method *cacheConfiguration(String
> > > > > cacheName)*
> > > > > > in
> > > > > > > to *Ignite* core interface as part of the task
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-11141.
> > > > > > >
> > > > > > > I'll be glad to see the comments about this feature.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Sergey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
Re: New committer: Ilya Lantukh
Congrats, Ilya. On Fri, Feb 15, 2019 at 2:33 PM Anton Vinogradov wrote: > Congrats! > > On Fri, Feb 15, 2019 at 2:23 PM Павлухин Иван wrote: > > > Ilya, > > > > My congratulations! Hope to hear from you about hair-splitting in > > context of affinity and topology. > > > > пт, 15 февр. 2019 г. в 13:44, Dmitriy Pavlov : > > > > > > Congrats, Ilya. I'm glad that it eventually happened. > > > > > > пт, 15 февр. 2019 г. в 11:04, Alexey Goncharuk >: > > > > > > > Dear Ignite Developers, > > > > > > > > > > > > > > > > The Project Management Committee (PMC) for Apache Ignite has invited > > Ilya > > > > Lantukh to become a committer and we are pleased to announce that he > > has > > > > accepted. > > > > > > > > > > > > > > > > Ilya has a long history contributing to Apache Ignite, including a > lot > > of > > > > complex changes allowed us to enable and support native persistence. > > > > > > > > Being a committer enables easier contribution to the project since > > there is > > > > no need to go via the patch submission process. This should enable > > better > > > > productivity. > > > > > > > > > > > > > > > > Please join me in welcoming Ilya, and congratulating him on the new > > role in > > > > the Apache Ignite Community. > > > > > > > > Best Regards, > > > > > > > > Alexey Goncharuk > > > > > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > > -- Best regards, Andrey V. Mashenkov
Re: New committer: Ilya Lantukh
Congrats! On Fri, Feb 15, 2019 at 2:23 PM Павлухин Иван wrote: > Ilya, > > My congratulations! Hope to hear from you about hair-splitting in > context of affinity and topology. > > пт, 15 февр. 2019 г. в 13:44, Dmitriy Pavlov : > > > > Congrats, Ilya. I'm glad that it eventually happened. > > > > пт, 15 февр. 2019 г. в 11:04, Alexey Goncharuk : > > > > > Dear Ignite Developers, > > > > > > > > > > > > The Project Management Committee (PMC) for Apache Ignite has invited > Ilya > > > Lantukh to become a committer and we are pleased to announce that he > has > > > accepted. > > > > > > > > > > > > Ilya has a long history contributing to Apache Ignite, including a lot > of > > > complex changes allowed us to enable and support native persistence. > > > > > > Being a committer enables easier contribution to the project since > there is > > > no need to go via the patch submission process. This should enable > better > > > productivity. > > > > > > > > > > > > Please join me in welcoming Ilya, and congratulating him on the new > role in > > > the Apache Ignite Community. > > > > > > Best Regards, > > > > > > Alexey Goncharuk > > > > > > > -- > Best regards, > Ivan Pavlukhin >
Re: New committer: Ilya Lantukh
Ilya, My congratulations! Hope to hear from you about hair-splitting in context of affinity and topology. пт, 15 февр. 2019 г. в 13:44, Dmitriy Pavlov : > > Congrats, Ilya. I'm glad that it eventually happened. > > пт, 15 февр. 2019 г. в 11:04, Alexey Goncharuk : > > > Dear Ignite Developers, > > > > > > > > The Project Management Committee (PMC) for Apache Ignite has invited Ilya > > Lantukh to become a committer and we are pleased to announce that he has > > accepted. > > > > > > > > Ilya has a long history contributing to Apache Ignite, including a lot of > > complex changes allowed us to enable and support native persistence. > > > > Being a committer enables easier contribution to the project since there is > > no need to go via the patch submission process. This should enable better > > productivity. > > > > > > > > Please join me in welcoming Ilya, and congratulating him on the new role in > > the Apache Ignite Community. > > > > Best Regards, > > > > Alexey Goncharuk > > -- Best regards, Ivan Pavlukhin
Re: MVCC and IgniteDataStreamer
Hi, It is time to continue DataStreamer for MVCC caches discussion. Main focus is on allowOverwrite=false mode. Currently there is a problem related to partition update counters. BACKGROUND As you might know MVCC transactions update partition counters on transaction finish phase and on backups counters are applied as intervals (low, high). Basically, transaction on primary partition counts number of updates it done and increments a partition counter locally by that number during finish stage. So, it happens that primary transaction updates counter from _low_ to _high_. And that (low, high) interval is sent to backups. If a counter on particular backup is equal to _low_ value than counter is incremented to _high_. Otherwise if current value is lesser than an interval is put into a queue. It will be applied when current value becomes equal to _low_. This technique leads us to a situation when partition counters are incremented on backups in the same order as on primary. Let's consider a simple example. Assume that we have partition counter 10 at some point and 2 transactions finish concurrently. Each have made e.g. 5 updates. Partition counter is updated in some order on primary and backup receives messages from primary in reversed order. Primary [10] | Backup [10] Tx1 (10 -> 15) [15]| Tx2 (15 -> 20) [20]| | Receives (15, 20) [10] | // (15, 20) enqueued | Receives (10, 15) [20] | // (10, 15) applied, (15, 20) dequeued and applied (Partition counter value in square brackets) But in contrast data streamer updates counters right at a time of inserting an entry into cache. And it totally breaks the idea of interval counters application. If we have data steamer and transaction modifying the same partition counter concurrently we can get following situation (initially the counter is 10): 1. Tx updated counter (10 -> 15) on primary and send a (10, 15) to backup. 2. Streamer inserted an entry and updated counter (15 -> 16) on primary. 3. Streamer inserted an entry and updated counter (10 -> 11) on backup. 4. Backup receives (10, 15) from tx and does not know what to do with it as it has counter equal to 11 now. And we can have other unexpected effects, e.g. in case of 2 transactions and a streamer the order of counter application by transactions might be reordered. PROPOSAL It looks like that streamer should apply counters by intervals to resolve the inconsistency. To do so we need stream through primary partition because counters should be "reserved" in a single place. So, following could be done when we are working with MVCC cache: 1. Send batches from streamer only to primary partition owners. 2. Remember partition counter updates made on primary. 3. Forward batches to backups along with counter intervals. What do you think? вт, 14 авг. 2018 г. в 15:21, Dmitriy Setrakyan : > > On Tue, Aug 14, 2018 at 4:30 AM, Vladimir Ozerov > wrote: > > > Bypassing WAL will make the whole cache data vulnerable to complete loss in > > case of node failure. I would not do this automatically. > > > > Well, in this case I would expect a log message suggesting that there is an > option to turn off WAL which could significantly improve performance. We > could just print it out once. -- Best regards, Ivan Pavlukhin
[jira] [Created] (IGNITE-11331) SQL: Remove unnecessary parameters binding
Vladimir Ozerov created IGNITE-11331:
Summary: SQL: Remove unnecessary parameters binding
Key: IGNITE-11331
URL: https://issues.apache.org/jira/browse/IGNITE-11331
Project: Ignite
Issue Type: Task
Components: sql
Reporter: Vladimir Ozerov
Fix For: 2.8
See usages of {{H2Utils#bindParameters}}. Note that it is used both in SELECT
and DML planners without any reason. Let's remove it from there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: New committer: Ilya Lantukh
Congrats, Ilya. I'm glad that it eventually happened. пт, 15 февр. 2019 г. в 11:04, Alexey Goncharuk : > Dear Ignite Developers, > > > > The Project Management Committee (PMC) for Apache Ignite has invited Ilya > Lantukh to become a committer and we are pleased to announce that he has > accepted. > > > > Ilya has a long history contributing to Apache Ignite, including a lot of > complex changes allowed us to enable and support native persistence. > > Being a committer enables easier contribution to the project since there is > no need to go via the patch submission process. This should enable better > productivity. > > > > Please join me in welcoming Ilya, and congratulating him on the new role in > the Apache Ignite Community. > > Best Regards, > > Alexey Goncharuk >
[jira] [Created] (IGNITE-11330) Partition demander should skip entries for invalid partition.
Andrew Mashenkov created IGNITE-11330: - Summary: Partition demander should skip entries for invalid partition. Key: IGNITE-11330 URL: https://issues.apache.org/jira/browse/IGNITE-11330 Project: Ignite Issue Type: Bug Components: cache Reporter: Andrew Mashenkov Fix For: 2.8 For now Partition demander tries to process all entries for invalid partition instead of switching to the next partition. Looks like, this was introduced with a fix IGNITE-8955 to make checkpoint locks fine-grained. A "for" loop under checkpoint readlock in GridDhtPartitionDemander.handleSupplyMessage() causes the issue. A flow can breaks inner "for" loop in case of invalid partition, but seems an outer "while" loop expected. To resolve this we can e.g. move CacheEntryInfo collection processing into separate method and just exit from it if invalid partition is detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-11329) Support data page scan for JDBC v2 driver
Pavel Kuznetsov created IGNITE-11329:
Summary: Support data page scan for JDBC v2 driver
Key: IGNITE-11329
URL: https://issues.apache.org/jira/browse/IGNITE-11329
Project: Ignite
Issue Type: Improvement
Components: jdbc, sql
Reporter: Pavel Kuznetsov
We need to fix compatibility issue in compute task of jdbc v2 driver and add
support of this flag.
Currently jdbc thick driver doen't ask cluster nodes for theirs versions.
Guess we have node with only {{JdbcQueryTaskV2}} and v3 is missing in that
version of the node. If we use new feature, introduced in v3, driver performs
v3 with no regards what versions is supported on the node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-11328) Ignite binary build is too big
Sergey Kozlov created IGNITE-11328:
--
Summary: Ignite binary build is too big
Key: IGNITE-11328
URL: https://issues.apache.org/jira/browse/IGNITE-11328
Project: Ignite
Issue Type: Bug
Affects Versions: 2.8
Reporter: Sergey Kozlov
Fix For: 2.8
I built Apache Ignite and get zip file is ~ 800MB.
+400MB added by 4 ML modules in {{libs/optional}}
Looks like it should be redesigned (join in a single module and at least it
will remove same deps)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSSION] Usage of system properties in tests
++1 from my side. I guess it will be a more reliable way for works with
properties in the tests. From time to time somebody forgot clear property
after test and next tests may be failed or flaky failed completion.
On Thu, Feb 14, 2019 at 6:56 PM Dmitriy Pavlov wrote:
> I find it absolutely positive and modern approach and good contribution.
> Count on my support if you will need any assistance with applying this
> patch.
>
> чт, 14 февр. 2019 г. в 18:53, Ivan Bessonov :
>
> > Hello Igniters,
> >
> > I'd like to discuss the way we treat system properties in our test
> > codebase.
> > It's a common pattern where we set system property in
> "beforeTestsStarted"
> > and clear it in "afterTestsStopped". Purest example that I've found is
> > class
> > "RedisProtocolGetAllAsArrayTest".
> >
> > There are similar things with "beforeTest"/"afterTest" or huge
> > "try/finally" blocks
> > right in test methods.
> >
> > I think that all this code can be avoided and solution might look like
> > this:
> >
> > @Test
> > @WithSystemProperty(key = IGNITE_PROPERTY_NAME, value = "true")
> > public void testSomething() throws Exception {
> > ...
> > }
> >
> > Same annotation might be used on class, this way new system property will
> > be applied to all test methods in the class.
> >
> > I already created the issue for this change [1] and PR with demo [2]. It
> > contains
> > implementation of annotation processing and a few migrated tests. If you
> > like
> > the idea then I will migrate all the other tests on the same mechanism.
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11323
> > [2] https://github.com/apache/ignite/pull/6109
> >
> > --
> > Sincerely yours,
> > Ivan Bessonov
> >
>
New committer: Ilya Lantukh
Dear Ignite Developers, The Project Management Committee (PMC) for Apache Ignite has invited Ilya Lantukh to become a committer and we are pleased to announce that he has accepted. Ilya has a long history contributing to Apache Ignite, including a lot of complex changes allowed us to enable and support native persistence. Being a committer enables easier contribution to the project since there is no need to go via the patch submission process. This should enable better productivity. Please join me in welcoming Ilya, and congratulating him on the new role in the Apache Ignite Community. Best Regards, Alexey Goncharuk
