Re: [DISCUSSION] Ignite 3.0 and to be removed list

[Roman Shtykh]

Probably it makes sense to have a survey on users mailing list to have some 
idea what is being used before deciding what to bury, support in a separate 
repository or master branch.

-- Roman
 

On Thursday, June 20, 2019, 10:26:37 p.m. GMT+9, Ilya Kasnacheev 
 wrote:  
 
 Hello!

I think that Hibernate and MongoDB support should go not to cemetery but to
separate repositories to be supported. They see quite a few demand.

Regards,
-- 
Ilya Kasnacheev


вт, 18 июн. 2019 г. в 21:28, Denis Magda :

> Alex,
>
> I've separated all to-be-removed points from existing
> > Ignite 3.0 wishlist [1] to a dedicated block and also added a few more
> > things that look right to be dropped.
>
>
> Could you please share a reference to the wishlist? It's not in your
> original email nor anywhere else in the discussion.
>
> Generally speaking, I would group to-be-removed capabilities by Components,
> Integrations, and APIs. Here is how my list looks like:
>
>    1. Components - IGFS and In-Memory Hadoop Accelerator. Strongly suggest
>    we to not putting off this procedure until 3.0 but execute the decision
> in
>    the next Ignite release. Refer to this thread [1] for details.
>    2. Integrations (aka. modules or plugins):
>      - Remove completely OR move to Github cemetery and no longer support
>      for every Ignite releases: Twitter, ZeroMQ, RocketMQ, Storm,
> Flume, Flink,
>      MQTT, Camel, Hibernate, JMS, OSGi, YARN, Mesos, AOP-Based Grid
>      ,
> ignite-clients
>      module  >
>      - Preserve and move to separate repositories (to be supported by the
>      community). The goal is to separate Ignite core from
> modules/plugins: Spark
>      Integration, TensorFlow, Cassandra Integration (ING), SpringData,
>      SpringBoot, Spring Caching, Kafka Integration.
>      - Move to separate repositories: thin clients (at least non-Java
> ones)
>    3. APIs:
>      - Remove: Redis and Memcached protocols support, compute
>      checkpointing SPI, geospatial support
>      - Should we remove the following or invest our time in better
>      support? - full-text search, Ignite messaging
>
>
> I would put all of the suggestions on the wiki or on that wishlist and
> track everything there while the discussion continues.
>
> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html
>
> -
> Denis
>
>
> On Mon, Jun 17, 2019 at 5:18 AM Alexey Goncharuk 
> wrote:
>
> > Igniters,
> >
> > Even though we are still planning the Ignite 2.8 release, I would like to
> > kick-off a discussion related to Ignite 3.0, because the efforts for AI
> 3.0
> > will be significantly larger than for AI 2.8, better to start early.
> >
> > As a first step, I would like to discuss the list of things to be removed
> > in Ignite 3.0 (partially this thread is inspired by Denis Magda's IGFS
> > removal thread). I've separated all to-be-removed points from existing
> > Ignite 3.0 wishlist [1] to a dedicated block and also added a few more
> > things that look right to be dropped.
> >
> > Please share your thoughts, probably, there are more outdated things we
> > need to add to the wishlist.
> >
> > As a side question: I think it makes sense to create tickets for such
> > improvements, how do we track them. Will the 3.0 version suffice or
> should
> > we add a separate label?
> >
> > --AG
> >
>  

Connection pool for Java thin client?

[Shane Duan]

Hi Igniters,

I have a multithread application in which I plan to use the Ignite Java
thin client. Is it safe to share the client between threads? Or should each
thread create its own client connection? Is there a connection pool kind of
implementation for the Java thin client , like the one for JDBC? In the
context, multiple threads will try put/get cached values concurrently.

Thanks!

Shane

[jira] [Created] (IGNITE-11938) SQL: Support java.time.Instant type as native time type

[Pavel Kuznetsov (JIRA)]

Pavel Kuznetsov created IGNITE-11938:


 Summary: SQL: Support java.time.Instant type as native time type
 Key: IGNITE-11938
 URL: https://issues.apache.org/jira/browse/IGNITE-11938
 Project: Ignite
  Issue Type: Improvement
Reporter: Pavel Kuznetsov


Currently Instant type is treated by Ignite as JAVA_OBJECT sql 
type(IGNITE-10690). 
But it can be possibly supported as one of the internal sql types. This 
probably breaks backward compatibility, since indexes should be rebuild



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11937) Fix MVCC PDS flaky suites timeout

[Alexei Scherbakov (JIRA)]

Alexei Scherbakov created IGNITE-11937:
--

 Summary: Fix MVCC PDS flaky suites timeout
 Key: IGNITE-11937
 URL: https://issues.apache.org/jira/browse/IGNITE-11937
 Project: Ignite
  Issue Type: Bug
Reporter: Alexei Scherbakov


Currently we have non-zero failure rate for some MVCC PDS suites in master.

Seems this is due to failure [1] in testRebalancingDuringLoad* tests group, 
which leads to dumping WAL and lock states at the time proportional to current 
WAL length increasing test duration for random time depending on WAL length.

Worse thing the test remains green despite throwing a critical exception.

[1]  Stacktrace
{noformat}
[2019-06-19 
15:56:53,386][ERROR][sys-stripe-6-#134%persistence.IgnitePdsContinuousRestartTestWithSharedGroupAndIndexes3%][IgniteTestResources]
 Critical system error detected. Will be handled accordingly to configured 
handler [hnd=NoOpFailureHandler [super=AbstractFailure
Handler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, 
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext 
[type=CRITICAL_ERROR, err=class 
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is 
corrupted [pages(groupId, page
Id)=[IgniteBiTuple [val1=81227264, val2=844420635164676]], msg=Runtime failure 
on search row: TxKey [major=1560948946388, minor=17286
class 
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
 B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=81227264, 
val2=844420635164676]], msg=Runtime failure on search row: TxKey 
[major=1560948946388, minor=17286]]
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:5909)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1859)
at 
org.apache.ignite.internal.processors.cache.mvcc.txlog.TxLog.put(TxLog.java:293)
at 
org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl.updateState(MvccProcessorImpl.java:699)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.setMvccState(IgniteTxManager.java:2570)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1228)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.state(IgniteTxAdapter.java:1070)
at 
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:421)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1837)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1198)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:118)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:224)
at 
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:222)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1141)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1558)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1186)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at 
org.apache.ignite.internal.managers.communication.GridIoManager$8.run(GridIoManager.java:1083)
at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:559)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Unexpected new transaction state. 
[currState=2, newState=1, cntr=17286]
at 
org.apache.ignite.internal.processors.cache.mvcc.txlog.TxLog$TxLogUpdateClosure.invalid(TxLog.java:629)
at 

Re: Thin client test suites failure

[Павлухин Иван]

Igniters,

I found one thing which might be reason of an observed behavior. The
problem does not appear for maven-based tests, while e.g. Python thin
client tests use ignite.sh for starting a cluster. So, I compared JVM
launch options and observed a following extra option in maven-based
runs:
--patch-module=java.transaction=~/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar

I missed when this trick was employed? Can anybody help with it? Does
it mean that ignite.sh does not work for all supported Java versions?

чт, 20 июн. 2019 г. в 13:30, Павлухин Иван :
>
> By the way, what about Java 9? Do we run tests using it?
>
> чт, 20 июн. 2019 г. в 13:28, Dmitriy Pavlov :
> >
> > Actually, TC Bot is not actively developed. I would like to say, moreover,
> > help is needed there.
> >
> > I agree that 10095 is (intentionally) vague. Will separate notification
> > based on JDK version be supported or not, depends on the time available to
> > complete this task.
> >
> > чт, 20 июн. 2019 г. в 12:02, Павлухин Иван :
> >
> > > Dmitriy,
> > >
> > >  It is good to know that TC Bot is being developed actively. Just to
> > > double check. Will the mentioned issue [1] allow us to see alerts when
> > > a particular test fails only on a specific Java version?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-10095
> > >
> > > чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov :
> > > >
> > > > Hi,
> > > >
> > > > About TC Bot: I was going to support filtering of builds from same JVM
> > > for
> > > > TC Bot visa
> > > > https://issues.apache.org/jira/browse/IGNITE-10095
> > > >
> > > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do
> > > > investigations on how to retrieve/build run history faster than it is
> > > now.
> > > >
> > > > Only nightly runs are random, and yes, Ignite is tested using different
> > > VMs
> > > > because Ignite 2.7.5 declared support of, at least, Java 11.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван :
> > > >
> > > > > Maxim,
> > > > >
> > > > > Well, it is an interesting point. First of all I want to believe that
> > > > > issues occurring only on a specific Java version are rare. If so,
> > > > > running tests on a single version for a visa should be enough.
> > > > >
> > > > > On the other hand, a uniform test environment sounds a good idea
> > > > > (especially for a visa), preventing a test status change by "magic".
> > > > >
> > > > > So, I think that it can be done as follows:
> > > > > 1. Run tests for a visa on a baseline Java version (11?).
> > > > > 2. Run Nightly builds on all supported versions, choosing version at
> > > > > random is a one of options for that. (BTW, do we need some changes in
> > > > > TC bot to send a proper alert when test fails on some versions and
> > > > > fails on others?).
> > > > >
> > > > > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov :
> > > > > >
> > > > > > Folks,
> > > > > >
> > > > > > Maybe I'm missing something, but what is the reason for setting
> > > > > > JAVA_HOME randomly? For instance, some commit is working under jdk8
> > > > > > but fails under jdk9 (and we declare that 9-th version is supported
> > > by
> > > > > > Ignite). Should we merge this commit to the master branch (all test
> > > > > > suites can be OK under jdk8)?
> > > > > >
> > > > > > I know, that running all tests under all java version leads us to
> > > > > > Hell, but maybe we can have separated versions nightly builds?
> > > > > >
> > > > > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван 
> > > wrote:
> > > > > > >
> > > > > > > Dmitriy,
> > > > > > >
> > > > > > > Thank you for the hint! Java version seems to be the reason. I see
> > > > > > > that tests pass with Java 8 and 11 and fail with Java 10. See the
> > > > > > > exception below. Btw, do we test with Java 9? Have not seen that
> > > > > > > version in recent runs.
> > > > > > > java.lang.NoClassDefFoundError: javax/transaction/SystemException
> > > > > > > at java.base/java.lang.Class.forName0(Native Method)
> > > > > > > at java.base/java.lang.Class.forName(Class.java:291)
> > > > > > > at
> > > > >
> > > org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249)
> > > > > > > at
> > > > >
> > > org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235)
> > > > > > > at
> > > > >
> > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427)
> > > > > > > at
> > > > >
> > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869)
> > > > > > > at
> > > > >
> > > org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886)
> > > > > > > at
> > > > > org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135)
> > > > > > > at
> > > > >
> > > 

Re: [DISCUSSION] Ignite 3.0 and to be removed list

[Ilya Kasnacheev]

Hello!

I think that Hibernate and MongoDB support should go not to cemetery but to
separate repositories to be supported. They see quite a few demand.

Regards,
-- 
Ilya Kasnacheev


вт, 18 июн. 2019 г. в 21:28, Denis Magda :

> Alex,
>
> I've separated all to-be-removed points from existing
> > Ignite 3.0 wishlist [1] to a dedicated block and also added a few more
> > things that look right to be dropped.
>
>
> Could you please share a reference to the wishlist? It's not in your
> original email nor anywhere else in the discussion.
>
> Generally speaking, I would group to-be-removed capabilities by Components,
> Integrations, and APIs. Here is how my list looks like:
>
>1. Components - IGFS and In-Memory Hadoop Accelerator. Strongly suggest
>we to not putting off this procedure until 3.0 but execute the decision
> in
>the next Ignite release. Refer to this thread [1] for details.
>2. Integrations (aka. modules or plugins):
>   - Remove completely OR move to Github cemetery and no longer support
>   for every Ignite releases: Twitter, ZeroMQ, RocketMQ, Storm,
> Flume, Flink,
>   MQTT, Camel, Hibernate, JMS, OSGi, YARN, Mesos, AOP-Based Grid
>   ,
> ignite-clients
>   module  >
>   - Preserve and move to separate repositories (to be supported by the
>   community). The goal is to separate Ignite core from
> modules/plugins: Spark
>   Integration, TensorFlow, Cassandra Integration (ING), SpringData,
>   SpringBoot, Spring Caching, Kafka Integration.
>   - Move to separate repositories: thin clients (at least non-Java
> ones)
>3. APIs:
>   - Remove: Redis and Memcached protocols support, compute
>   checkpointing SPI, geospatial support
>   - Should we remove the following or invest our time in better
>   support? - full-text search, Ignite messaging
>
>
> I would put all of the suggestions on the wiki or on that wishlist and
> track everything there while the discussion continues.
>
> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html
>
> -
> Denis
>
>
> On Mon, Jun 17, 2019 at 5:18 AM Alexey Goncharuk 
> wrote:
>
> > Igniters,
> >
> > Even though we are still planning the Ignite 2.8 release, I would like to
> > kick-off a discussion related to Ignite 3.0, because the efforts for AI
> 3.0
> > will be significantly larger than for AI 2.8, better to start early.
> >
> > As a first step, I would like to discuss the list of things to be removed
> > in Ignite 3.0 (partially this thread is inspired by Denis Magda's IGFS
> > removal thread). I've separated all to-be-removed points from existing
> > Ignite 3.0 wishlist [1] to a dedicated block and also added a few more
> > things that look right to be dropped.
> >
> > Please share your thoughts, probably, there are more outdated things we
> > need to add to the wishlist.
> >
> > As a side question: I think it makes sense to create tickets for such
> > improvements, how do we track them. Will the 3.0 version suffice or
> should
> > we add a separate label?
> >
> > --AG
> >
>

Re: [RESULT] [VOTE] Release Apache Ignite 2.7.5-rc4

[Dmitriy Pavlov]

Hi Denis,
Artem explained to me what to do, but I need permissions for all
supplementary docs to switch version.

Could you please grant it for
C#/.NET
C++
SQL
Integrations*
Ignite for Spark
Tools
?

вт, 11 июн. 2019 г. в 23:21, Dmitriy Pavlov :

> Thank you, Mauricio,
>
> I'm going to apply this patch on Monday. I will be traveling with limited
> access to the Internet.
>
> I'll also add more explanation on why we should update these tags to the
> wiki.
>
> вт, 11 июн. 2019 г. в 20:57, Mauricio Stekl :
>
>> Hi Dmitriy,
>> Yes, that would be enough to set the URL for latest docs.
>>
>> Additionally I usually do some SEO updates to new/old .html files, like
>> adding NOINDEX metatag to older version; add canonical URL to latest docs;
>> and add GA code.
>>
>> I am attaching a svn patch which includes all these updates, including
>> the .htaccess one.
>>
>>
>> Thanks.
>> Mauricio Stekl
>>
>>
>>
>>
>>
>> On Tue, Jun 11, 2019 at 1:38 PM Dmitriy Pavlov 
>> wrote:
>>
>>> Ok, Denis, thank you.
>>>
>>> Garrett, could you please advice me on how to do a copy of 2.7.0 docs to
>>> 2.7.5? I have an account on the readme.io, so I would like to do it and
>>> describe in the Release process
>>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process
>>>
>>> Mauricio, is my understanding correct?
>>> in the htaccess I can replace
>>>
>>> RewriteRule ^releases/latest/(.*)$ /releases/2.7.0/$1 [L]
>>>
>>> to
>>>
>>> RewriteRule ^releases/latest/(.*)$ /releases/2.7.5/$1 [L]
>>>
>>> and that's it?
>>>
>>> Sincerely
>>> Dmitriy Pavlov
>>>
>>> вт, 11 июн. 2019 г. в 19:26, Denis Magda :
>>>
>>> > Igniters,
>>> >
>>> > We have to create version 2.7.5 for readme.io even if the changes are
>>> > minimal. That's a general practice. Copying Garrett and Artem who can
>>> > facilitate on the documentation end.
>>> >
>>> > Also, feel free to reach out Mauricio (copied) who can help to run the
>>> > scripts that will make JavaDocs 2.7.5 latest for Ignite website.
>>> >
>>> > -
>>> > Denis
>>> >
>>> >
>>> > On Tue, Jun 11, 2019 at 6:00 AM Dmitriy Pavlov 
>>> wrote:
>>> >
>>> > > Sorry for the second email. I want to clarify: I've already uploaded
>>> > > Javadoc,scaladoc,c++ docs (generated). Question is related only to
>>> > > readme.io.
>>> > > We have links named as 2.7 there, so both 2.7.5 & 2.7.0 seems to be
>>> > > applicable.
>>> > >
>>> > > There were no changes in the public API/parameters.
>>> > >
>>> > > вт, 11 июн. 2019 г. в 15:44, Nikolay Izhikov :
>>> > >
>>> > > > Hello, Dmitriy.
>>> > > >
>>> > > > I think we should write down diff between 2.7.0 and 2.7.5 on some
>>> page
>>> > in
>>> > > > 2.7 documentation.
>>> > > > Also, we should make this page very noticable.
>>> > > >
>>> > > > В Вт, 11/06/2019 в 15:22 +0300, Dmitriy Pavlov пишет:
>>> > > > > Hi Igniters, PMCs,
>>> > > > >
>>> > > > > could you please advice me about step
>>> > > > >
>>> > > > >1. Release all the documentation (Java, .NET, C++, etc.) on
>>> > > > >apacheignite.readme.io.
>>> > > > >
>>> > > > > from the release process.
>>> > > > >
>>> > > > > Do we have some documentation to publish for 2.7.5? Since it is
>>> minor
>>> > > > > release I propose to keep documentation for 2.7.0.
>>> > > > >
>>> > > > > Sincerely,
>>> > > > > Dmitriy Pavlov
>>> > > > >
>>> > > > > пн, 10 июн. 2019 г. в 14:30, Dmitriy Pavlov >> >:
>>> > > > >
>>> > > > > > The vote for a new release candidate is closed, now
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > Vote result: Vote passes with 9 votes +1 (4 binding +1 votes),
>>> no 0
>>> > > > and no
>>> > > > > > -1.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > +1 votes:
>>> > > > > >
>>> > > > > > - Ilya Kasnacheev
>>> > > > > >
>>> > > > > > - Nikolay Izhikov (binding)
>>> > > > > >
>>> > > > > > - Denis Magda (binding)
>>> > > > > >
>>> > > > > > - Andrey Gura (binding)
>>> > > > > >
>>> > > > > > - Alexey Goncharuk (binding)
>>> > > > > >
>>> > > > > > - Igor Sapego
>>> > > > > >
>>> > > > > > - Ivan Pavlukhin
>>> > > > > >
>>> > > > > > - Yuriy Babak
>>> > > > > >
>>> > > > > > - Vyacheslav Daradur
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > Vote thread
>>> > > > > >
>>> > > >
>>> > >
>>> >
>>> https://lists.apache.org/thread.html/35cbc2d4c5b769155dc8aec15edd808a25c5cf48a5e12637528e931d@%3Cdev.ignite.apache.org%3E
>>> > > > > >
>>> > > > > >
>>> > > >
>>> > >
>>> >
>>>
>>

Re: Read Repair (ex. Consistency Check) - review request #2

[Anton Vinogradov]

Nikolay,

>> Should we consider it's removal in Ignite 3
Don't think so.

My initial ReadRepair implementation uses version to detect inconsistency.
Strategy can be changed later (most likely it will) or even provided
ability to use own strategy.
Data streamer's and cache.load's cases can be supported by simple values
check as the second phase of checks.
As I mentioned privately - all limitations are limitations for the initial
version and can be eliminated in the future.

Currently, the main goal is to merge the solution to continue the work.
* After the merge, it will be possible to split work on this task (metrics,
whole partitions fix, data streamers, ... etc).
* Seems, I found the consistency issue checking my feature (a backup can be
outdated for some time after tx finish in full_sync).
Merge will provide the ability to investigate bug deeper and solve if
confirmed.

On Thu, Jun 20, 2019 at 2:22 PM Nikolay Izhikov  wrote:

> Anton.
>
> I worried about this limitation:
>
> > Entries streamed using data streamer (using not a "cache.put" based
> updater) and loaded by cache.load.
>
> As we discussed privately in this modes
>
> *ALL ENTRIES ON ALL OWNERS WILL HAVE DIFFERENT VERSIONS*
>
> Why we need this modes, in the first place?
> Should we consider it's removal in Ignite 3 or should we fix them?
>
> В Чт, 20/06/2019 в 14:15 +0300, Anton Vinogradov пишет:
> > Igniters,
> > I'm glad to introduce Read Repair feature [0] provides additional
> > consistency guarantee for Ignite.
> >
> > 1) Why we need it?
> > The detailed explanation can be found at IEP-31 [1].
> > In short, because of bugs, it's possible to gain an inconsistent state.
> > We need additional features to handle this case.
> >
> > Currently we able to check cluster using Idle_verify [2] feature, but it
> > will not fix the data, will not even tell which entries are broken.
> > Read Repair is a feature to understand which entries are broken and to
> fix
> > them.
> >
> > 1) How it works?
> > IgniteCache now able to provide special proxy [3] withReadRepair().
> > This proxy guarantee that data will be gained from all owners and
> compared.
> > In the case of consistency violation situation, data will be recovered
> and
> > a special event recorded.
> >
> > 3) Naming?
> > Feature name based on Cassandra's Read Repair feature [4], which is
> pretty
> > similar.
> >
> > 4) Limitations which can be fixed in the future?
> >   * MVCC and Near caches are not supported.
> >   * Atomic caches can be checked (false positive case is possible on this
> > check), but can't be recovered.
> >   * Partial entry removal can't be recovered.
> >   * Entries streamed using data streamer (using not a "cache.put" based
> > updater) and loaded by cache.load
> >   are perceived as inconsistent since they may have different versions
> for
> > same keys.
> >   * Only explicit get operations are supported (getAndReplace, getAndPut,
> > etc can be supported in future).
> >
> > 5) What's left?
> >   * SQL/ThinClient/etc support.
> >   * Metrics (found/repaired).
> >   * Simple per-partition recovery feature able to work in the background
> in
> > addition to per-entry recovery feature.
> >
> > 6) Is code checked?
> >   * Pull Request #5656 [5] (feature) - has green TC.
> >   * Pull Request #6575 [6] (RunAll with the feature enabled for every
> get()
> > request) - has a limited amount of failures (because of data streamer,
> > cache.load, etc).
> >
> > Thoughts?
> >
> > [0] https://issues.apache.org/jira/browse/IGNITE-10663
> > [1]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-31+Consistency+check+and+fix
> > [2]
> >
> https://apacheignite-tools.readme.io/docs/control-script#section-verification-of-partition-checksums
> > [3]
> >
> https://github.com/apache/ignite/blob/27b6105ecc175b61e0aef59887830588dfc388ef/modules/core/src/main/java/org/apache/ignite/IgniteCache.java#L140
> > [4]
> >
> https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsRepairNodesReadRepair.html
> > [5] https://github.com/apache/ignite/pull/5656
> > [6] https://github.com/apache/ignite/pull/6575
>

Re: Read Repair (ex. Consistency Check) - review request #2

[Nikolay Izhikov]

Anton.

I worried about this limitation:

> Entries streamed using data streamer (using not a "cache.put" based updater) 
> and loaded by cache.load.

As we discussed privately in this modes 

*ALL ENTRIES ON ALL OWNERS WILL HAVE DIFFERENT VERSIONS*

Why we need this modes, in the first place?
Should we consider it's removal in Ignite 3 or should we fix them?

В Чт, 20/06/2019 в 14:15 +0300, Anton Vinogradov пишет:
> Igniters,
> I'm glad to introduce Read Repair feature [0] provides additional
> consistency guarantee for Ignite.
> 
> 1) Why we need it?
> The detailed explanation can be found at IEP-31 [1].
> In short, because of bugs, it's possible to gain an inconsistent state.
> We need additional features to handle this case.
> 
> Currently we able to check cluster using Idle_verify [2] feature, but it
> will not fix the data, will not even tell which entries are broken.
> Read Repair is a feature to understand which entries are broken and to fix
> them.
> 
> 1) How it works?
> IgniteCache now able to provide special proxy [3] withReadRepair().
> This proxy guarantee that data will be gained from all owners and compared.
> In the case of consistency violation situation, data will be recovered and
> a special event recorded.
> 
> 3) Naming?
> Feature name based on Cassandra's Read Repair feature [4], which is pretty
> similar.
> 
> 4) Limitations which can be fixed in the future?
>   * MVCC and Near caches are not supported.
>   * Atomic caches can be checked (false positive case is possible on this
> check), but can't be recovered.
>   * Partial entry removal can't be recovered.
>   * Entries streamed using data streamer (using not a "cache.put" based
> updater) and loaded by cache.load
>   are perceived as inconsistent since they may have different versions for
> same keys.
>   * Only explicit get operations are supported (getAndReplace, getAndPut,
> etc can be supported in future).
> 
> 5) What's left?
>   * SQL/ThinClient/etc support.
>   * Metrics (found/repaired).
>   * Simple per-partition recovery feature able to work in the background in
> addition to per-entry recovery feature.
> 
> 6) Is code checked?
>   * Pull Request #5656 [5] (feature) - has green TC.
>   * Pull Request #6575 [6] (RunAll with the feature enabled for every get()
> request) - has a limited amount of failures (because of data streamer,
> cache.load, etc).
> 
> Thoughts?
> 
> [0] https://issues.apache.org/jira/browse/IGNITE-10663
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-31+Consistency+check+and+fix
> [2]
> https://apacheignite-tools.readme.io/docs/control-script#section-verification-of-partition-checksums
> [3]
> https://github.com/apache/ignite/blob/27b6105ecc175b61e0aef59887830588dfc388ef/modules/core/src/main/java/org/apache/ignite/IgniteCache.java#L140
> [4]
> https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsRepairNodesReadRepair.html
> [5] https://github.com/apache/ignite/pull/5656
> [6] https://github.com/apache/ignite/pull/6575


signature.asc
Description: This is a digitally signed message part

Read Repair (ex. Consistency Check) - review request #2

[Anton Vinogradov]

Igniters,
I'm glad to introduce Read Repair feature [0] provides additional
consistency guarantee for Ignite.

1) Why we need it?
The detailed explanation can be found at IEP-31 [1].
In short, because of bugs, it's possible to gain an inconsistent state.
We need additional features to handle this case.

Currently we able to check cluster using Idle_verify [2] feature, but it
will not fix the data, will not even tell which entries are broken.
Read Repair is a feature to understand which entries are broken and to fix
them.

1) How it works?
IgniteCache now able to provide special proxy [3] withReadRepair().
This proxy guarantee that data will be gained from all owners and compared.
In the case of consistency violation situation, data will be recovered and
a special event recorded.

3) Naming?
Feature name based on Cassandra's Read Repair feature [4], which is pretty
similar.

4) Limitations which can be fixed in the future?
  * MVCC and Near caches are not supported.
  * Atomic caches can be checked (false positive case is possible on this
check), but can't be recovered.
  * Partial entry removal can't be recovered.
  * Entries streamed using data streamer (using not a "cache.put" based
updater) and loaded by cache.load
  are perceived as inconsistent since they may have different versions for
same keys.
  * Only explicit get operations are supported (getAndReplace, getAndPut,
etc can be supported in future).

5) What's left?
  * SQL/ThinClient/etc support.
  * Metrics (found/repaired).
  * Simple per-partition recovery feature able to work in the background in
addition to per-entry recovery feature.

6) Is code checked?
  * Pull Request #5656 [5] (feature) - has green TC.
  * Pull Request #6575 [6] (RunAll with the feature enabled for every get()
request) - has a limited amount of failures (because of data streamer,
cache.load, etc).

Thoughts?

[0] https://issues.apache.org/jira/browse/IGNITE-10663
[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-31+Consistency+check+and+fix
[2]
https://apacheignite-tools.readme.io/docs/control-script#section-verification-of-partition-checksums
[3]
https://github.com/apache/ignite/blob/27b6105ecc175b61e0aef59887830588dfc388ef/modules/core/src/main/java/org/apache/ignite/IgniteCache.java#L140
[4]
https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsRepairNodesReadRepair.html
[5] https://github.com/apache/ignite/pull/5656
[6] https://github.com/apache/ignite/pull/6575

Re: Thin client test suites failure

[Павлухин Иван]

By the way, what about Java 9? Do we run tests using it?

чт, 20 июн. 2019 г. в 13:28, Dmitriy Pavlov :
>
> Actually, TC Bot is not actively developed. I would like to say, moreover,
> help is needed there.
>
> I agree that 10095 is (intentionally) vague. Will separate notification
> based on JDK version be supported or not, depends on the time available to
> complete this task.
>
> чт, 20 июн. 2019 г. в 12:02, Павлухин Иван :
>
> > Dmitriy,
> >
> >  It is good to know that TC Bot is being developed actively. Just to
> > double check. Will the mentioned issue [1] allow us to see alerts when
> > a particular test fails only on a specific Java version?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-10095
> >
> > чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov :
> > >
> > > Hi,
> > >
> > > About TC Bot: I was going to support filtering of builds from same JVM
> > for
> > > TC Bot visa
> > > https://issues.apache.org/jira/browse/IGNITE-10095
> > >
> > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do
> > > investigations on how to retrieve/build run history faster than it is
> > now.
> > >
> > > Only nightly runs are random, and yes, Ignite is tested using different
> > VMs
> > > because Ignite 2.7.5 declared support of, at least, Java 11.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван :
> > >
> > > > Maxim,
> > > >
> > > > Well, it is an interesting point. First of all I want to believe that
> > > > issues occurring only on a specific Java version are rare. If so,
> > > > running tests on a single version for a visa should be enough.
> > > >
> > > > On the other hand, a uniform test environment sounds a good idea
> > > > (especially for a visa), preventing a test status change by "magic".
> > > >
> > > > So, I think that it can be done as follows:
> > > > 1. Run tests for a visa on a baseline Java version (11?).
> > > > 2. Run Nightly builds on all supported versions, choosing version at
> > > > random is a one of options for that. (BTW, do we need some changes in
> > > > TC bot to send a proper alert when test fails on some versions and
> > > > fails on others?).
> > > >
> > > > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov :
> > > > >
> > > > > Folks,
> > > > >
> > > > > Maybe I'm missing something, but what is the reason for setting
> > > > > JAVA_HOME randomly? For instance, some commit is working under jdk8
> > > > > but fails under jdk9 (and we declare that 9-th version is supported
> > by
> > > > > Ignite). Should we merge this commit to the master branch (all test
> > > > > suites can be OK under jdk8)?
> > > > >
> > > > > I know, that running all tests under all java version leads us to
> > > > > Hell, but maybe we can have separated versions nightly builds?
> > > > >
> > > > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван 
> > wrote:
> > > > > >
> > > > > > Dmitriy,
> > > > > >
> > > > > > Thank you for the hint! Java version seems to be the reason. I see
> > > > > > that tests pass with Java 8 and 11 and fail with Java 10. See the
> > > > > > exception below. Btw, do we test with Java 9? Have not seen that
> > > > > > version in recent runs.
> > > > > > java.lang.NoClassDefFoundError: javax/transaction/SystemException
> > > > > > at java.base/java.lang.Class.forName0(Native Method)
> > > > > > at java.base/java.lang.Class.forName(Class.java:291)
> > > > > > at
> > > >
> > org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249)
> > > > > > at
> > > >
> > org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235)
> > > > > > at
> > > >
> > org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427)
> > > > > > at
> > > >
> > org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869)
> > > > > > at
> > > >
> > org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886)
> > > > > > at
> > > > org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135)
> > > > > > at
> > > >
> > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1992)
> > > > > > at
> > > >
> > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1683)
> > > > > > at
> > > > org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1109)
> > > > > > at
> > > >
> > org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1027)
> > > > > > at
> > org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:913)
> > > > > > at
> > org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:812)
> > > > > > at
> > org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
> > > > > > at
> > org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:651)
> > > > > > at org.apache.ignite.Ignition.start(Ignition.java:346)
> > > > > > 

Re: Thin client test suites failure

[Dmitriy Pavlov]

Actually, TC Bot is not actively developed. I would like to say, moreover,
help is needed there.

I agree that 10095 is (intentionally) vague. Will separate notification
based on JDK version be supported or not, depends on the time available to
complete this task.

чт, 20 июн. 2019 г. в 12:02, Павлухин Иван :

> Dmitriy,
>
>  It is good to know that TC Bot is being developed actively. Just to
> double check. Will the mentioned issue [1] allow us to see alerts when
> a particular test fails only on a specific Java version?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-10095
>
> чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov :
> >
> > Hi,
> >
> > About TC Bot: I was going to support filtering of builds from same JVM
> for
> > TC Bot visa
> > https://issues.apache.org/jira/browse/IGNITE-10095
> >
> > Now it is displayed only in UI as JDK8,JDK11 tags. I still do
> > investigations on how to retrieve/build run history faster than it is
> now.
> >
> > Only nightly runs are random, and yes, Ignite is tested using different
> VMs
> > because Ignite 2.7.5 declared support of, at least, Java 11.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван :
> >
> > > Maxim,
> > >
> > > Well, it is an interesting point. First of all I want to believe that
> > > issues occurring only on a specific Java version are rare. If so,
> > > running tests on a single version for a visa should be enough.
> > >
> > > On the other hand, a uniform test environment sounds a good idea
> > > (especially for a visa), preventing a test status change by "magic".
> > >
> > > So, I think that it can be done as follows:
> > > 1. Run tests for a visa on a baseline Java version (11?).
> > > 2. Run Nightly builds on all supported versions, choosing version at
> > > random is a one of options for that. (BTW, do we need some changes in
> > > TC bot to send a proper alert when test fails on some versions and
> > > fails on others?).
> > >
> > > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov :
> > > >
> > > > Folks,
> > > >
> > > > Maybe I'm missing something, but what is the reason for setting
> > > > JAVA_HOME randomly? For instance, some commit is working under jdk8
> > > > but fails under jdk9 (and we declare that 9-th version is supported
> by
> > > > Ignite). Should we merge this commit to the master branch (all test
> > > > suites can be OK under jdk8)?
> > > >
> > > > I know, that running all tests under all java version leads us to
> > > > Hell, but maybe we can have separated versions nightly builds?
> > > >
> > > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван 
> wrote:
> > > > >
> > > > > Dmitriy,
> > > > >
> > > > > Thank you for the hint! Java version seems to be the reason. I see
> > > > > that tests pass with Java 8 and 11 and fail with Java 10. See the
> > > > > exception below. Btw, do we test with Java 9? Have not seen that
> > > > > version in recent runs.
> > > > > java.lang.NoClassDefFoundError: javax/transaction/SystemException
> > > > > at java.base/java.lang.Class.forName0(Native Method)
> > > > > at java.base/java.lang.Class.forName(Class.java:291)
> > > > > at
> > >
> org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249)
> > > > > at
> > >
> org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235)
> > > > > at
> > >
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427)
> > > > > at
> > >
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869)
> > > > > at
> > >
> org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886)
> > > > > at
> > > org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135)
> > > > > at
> > >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1992)
> > > > > at
> > >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1683)
> > > > > at
> > > org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1109)
> > > > > at
> > >
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1027)
> > > > > at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:913)
> > > > > at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:812)
> > > > > at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
> > > > > at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:651)
> > > > > at org.apache.ignite.Ignition.start(Ignition.java:346)
> > > > > at
> > >
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
> > > > > Caused by: java.lang.ClassNotFoundException:
> > > javax.transaction.SystemException
> > > > > at
> > >
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
> > > > > at
> > >
> 

Re: Thin client test suites failure

[Павлухин Иван]

Dmitriy,

 It is good to know that TC Bot is being developed actively. Just to
double check. Will the mentioned issue [1] allow us to see alerts when
a particular test fails only on a specific Java version?

[1] https://issues.apache.org/jira/browse/IGNITE-10095

чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov :
>
> Hi,
>
> About TC Bot: I was going to support filtering of builds from same JVM for
> TC Bot visa
> https://issues.apache.org/jira/browse/IGNITE-10095
>
> Now it is displayed only in UI as JDK8,JDK11 tags. I still do
> investigations on how to retrieve/build run history faster than it is now.
>
> Only nightly runs are random, and yes, Ignite is tested using different VMs
> because Ignite 2.7.5 declared support of, at least, Java 11.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 19 июн. 2019 г. в 22:13, Павлухин Иван :
>
> > Maxim,
> >
> > Well, it is an interesting point. First of all I want to believe that
> > issues occurring only on a specific Java version are rare. If so,
> > running tests on a single version for a visa should be enough.
> >
> > On the other hand, a uniform test environment sounds a good idea
> > (especially for a visa), preventing a test status change by "magic".
> >
> > So, I think that it can be done as follows:
> > 1. Run tests for a visa on a baseline Java version (11?).
> > 2. Run Nightly builds on all supported versions, choosing version at
> > random is a one of options for that. (BTW, do we need some changes in
> > TC bot to send a proper alert when test fails on some versions and
> > fails on others?).
> >
> > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov :
> > >
> > > Folks,
> > >
> > > Maybe I'm missing something, but what is the reason for setting
> > > JAVA_HOME randomly? For instance, some commit is working under jdk8
> > > but fails under jdk9 (and we declare that 9-th version is supported by
> > > Ignite). Should we merge this commit to the master branch (all test
> > > suites can be OK under jdk8)?
> > >
> > > I know, that running all tests under all java version leads us to
> > > Hell, but maybe we can have separated versions nightly builds?
> > >
> > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван  wrote:
> > > >
> > > > Dmitriy,
> > > >
> > > > Thank you for the hint! Java version seems to be the reason. I see
> > > > that tests pass with Java 8 and 11 and fail with Java 10. See the
> > > > exception below. Btw, do we test with Java 9? Have not seen that
> > > > version in recent runs.
> > > > java.lang.NoClassDefFoundError: javax/transaction/SystemException
> > > > at java.base/java.lang.Class.forName0(Native Method)
> > > > at java.base/java.lang.Class.forName(Class.java:291)
> > > > at
> > org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249)
> > > > at
> > org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235)
> > > > at
> > org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427)
> > > > at
> > org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869)
> > > > at
> > org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886)
> > > > at
> > org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135)
> > > > at
> > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1992)
> > > > at
> > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1683)
> > > > at
> > org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1109)
> > > > at
> > org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1027)
> > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:913)
> > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:812)
> > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
> > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:651)
> > > > at org.apache.ignite.Ignition.start(Ignition.java:346)
> > > > at
> > org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
> > > > Caused by: java.lang.ClassNotFoundException:
> > javax.transaction.SystemException
> > > > at
> > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
> > > > at
> > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190)
> > > > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499)
> > > > ... 18 more
> > > >
> > > > вт, 18 июн. 2019 г. в 23:27, Dmitriy Pavlov :
> > > > >
> > > > > Hi Ivan,
> > > > >
> > > > > Can these failures be related to Java version? Java home is set
> > randomly by
> > > > > TC Bot.
> > > > >
> > > > > Dmitriy Pavlov
> > > > >
> > > > > вт, 18 июн. 2019 г. в 22:11, Павлухин Иван :
> > > > >
> > > > > > Hi igniters,
> > > > > >
> > > > > 

Re: [VOTE] Complete Discontinuation of IGFS and Hadoop Accelerator

[Nikita Amelchev]

+1

чт, 20 июн. 2019 г. в 11:28, Anton Vinogradov :
>
> +1 (binding)
>
> On Thu, Jun 20, 2019 at 10:08 AM Dmitriy Pavlov  wrote:
>
> > +1
> >
> > чт, 20 июн. 2019 г. в 09:15, Nikolay Izhikov :
> >
> > > +1
> > >
> > > чт, 20 июня 2019 г., 8:28 Павлухин Иван :
> > >
> > > > +1
> > > >
> > > > чт, 20 июн. 2019 г. в 04:18, Valentin Kulichenko
> > > > :
> > > > >
> > > > > +1
> > > > >
> > > > > On Wed, Jun 19, 2019 at 5:58 PM Roman Shtykh
> >  > > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > On Thursday, June 20, 2019, 7:34:47 a.m. GMT+9, Andrey Gura <
> > > > > > ag...@apache.org> wrote:
> > > > > >
> > > > > >  +1
> > > > > >
> > > > > > On Thu, Jun 20, 2019 at 12:58 AM Denis Magda 
> > > > wrote:
> > > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Based on our earlier discussion [1], let's formalize our decision
> > > and
> > > > > > vote
> > > > > > > for the following:
> > > > > > >
> > > > > > >- IGFS and In-Memory Hadoop Accelerator components are to be
> > > > > > >discontinued and no longer supported by the community
> > > > > > >- The existing source code of IGFS and In-Memory Hadoop
> > > > Accelerator is
> > > > > > >to be removed from Ignite master. Before that, a special
> > branch
> > > > like
> > > > > > >"ignite-igfs-and-hadoop-accelerator" to be forked off the
> > master
> > > > in
> > > > > > order
> > > > > > >to preserve the sources in Git history for those who might
> > need
> > > > it.
> > > > > > >
> > > > > > > Please cast your vote:
> > > > > > >
> > > > > > > +1 - agree
> > > > > > > 0 - indifferent
> > > > > > > -1 - disagree, explain why.
> > > > > > >
> > > > > > > The vote lasts for the next 72 hours.
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > >
> > >
> > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html#a42326
> > > > > > >
> > > > > > > -
> > > > > > > Denis
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> >



-- 
Best wishes,
Amelchev Nikita

Re: [VOTE] Complete Discontinuation of IGFS and Hadoop Accelerator

[Anton Vinogradov]

+1 (binding)

On Thu, Jun 20, 2019 at 10:08 AM Dmitriy Pavlov  wrote:

> +1
>
> чт, 20 июн. 2019 г. в 09:15, Nikolay Izhikov :
>
> > +1
> >
> > чт, 20 июня 2019 г., 8:28 Павлухин Иван :
> >
> > > +1
> > >
> > > чт, 20 июн. 2019 г. в 04:18, Valentin Kulichenko
> > > :
> > > >
> > > > +1
> > > >
> > > > On Wed, Jun 19, 2019 at 5:58 PM Roman Shtykh
>  > >
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > >
> > > > > On Thursday, June 20, 2019, 7:34:47 a.m. GMT+9, Andrey Gura <
> > > > > ag...@apache.org> wrote:
> > > > >
> > > > >  +1
> > > > >
> > > > > On Thu, Jun 20, 2019 at 12:58 AM Denis Magda 
> > > wrote:
> > > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > Based on our earlier discussion [1], let's formalize our decision
> > and
> > > > > vote
> > > > > > for the following:
> > > > > >
> > > > > >- IGFS and In-Memory Hadoop Accelerator components are to be
> > > > > >discontinued and no longer supported by the community
> > > > > >- The existing source code of IGFS and In-Memory Hadoop
> > > Accelerator is
> > > > > >to be removed from Ignite master. Before that, a special
> branch
> > > like
> > > > > >"ignite-igfs-and-hadoop-accelerator" to be forked off the
> master
> > > in
> > > > > order
> > > > > >to preserve the sources in Git history for those who might
> need
> > > it.
> > > > > >
> > > > > > Please cast your vote:
> > > > > >
> > > > > > +1 - agree
> > > > > > 0 - indifferent
> > > > > > -1 - disagree, explain why.
> > > > > >
> > > > > > The vote lasts for the next 72 hours.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html#a42326
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
>

[IEP-35] GridJobProcessorMetrics migration

[Nikolay Izhikov]

Hello, Igniters.

Especially, Ignite veterans.

I've prepared PR [1] for the ticket IGNITE-11926 [2].

I found that we don't have any tests for the current GridJobMetrics 
implementation.
So I added basic tests for the current implementation in the PR.

Guys, do we have real-world usages of numbers from these metrics?

Back to my PR: I think we should migrate only a few of the existing 
GridJobProcessor metrics.
And that's why:

1. We shouldn't migrate aggregate metrics - max*, avg*
Aggregation should be done with the metric collect system(Prometheus, Graphite, 
etc.).

2. We shouldn't migrate `cpuLoadAvg`
Metrics for CPU should be available from separate sources(OS sensors or 
similar).

3. We shouldn't migrate `curidleTime`, `totalIdleTime`.
Idle metrics doesn't make sense for me.

They can be obtained from regularly scrapped `activeJobs` value.
Seems, they can't be used in the real world. Imagine 32 CPU server with only 
one active job. 
Idle time will be 0 for this scenario.

4. Execution(waiting) time should be available per job in the job list.

So my PR contains counters for the following numbers.
All the code belongs to the GridJobProcessor becomes deprecated.

Can someone do the review?

```
/** Number of started jobs. */
final LongMetricImpl startedJobsMetric;

 /** Number of active jobs currently executing. */
final LongMetricImpl activeJobsMetric;

 /** Number of currently queued jobs waiting to be executed. */
final LongMetricImpl waitingJobsMetric;

 /** Number of cancelled jobs that are still running. */
final LongMetricImpl canceledJobsMetric;

 /** Number of jobs rejected after more recent collision resolution 
operation. */
final LongMetricImpl rejectedJobsMetric;

 /** Number of finished jobs. */
final LongMetricImpl finishedJobsMetric;
```




[1] https://github.com/apache/ignite/pull/6622
[2] https://issues.apache.org/jira/browse/IGNITE-11926


signature.asc
Description: This is a digitally signed message part

Re: Thin client test suites failure

[Dmitriy Pavlov]

Hi,

About TC Bot: I was going to support filtering of builds from same JVM for
TC Bot visa
https://issues.apache.org/jira/browse/IGNITE-10095

Now it is displayed only in UI as JDK8,JDK11 tags. I still do
investigations on how to retrieve/build run history faster than it is now.

Only nightly runs are random, and yes, Ignite is tested using different VMs
because Ignite 2.7.5 declared support of, at least, Java 11.

Sincerely,
Dmitriy Pavlov

ср, 19 июн. 2019 г. в 22:13, Павлухин Иван :

> Maxim,
>
> Well, it is an interesting point. First of all I want to believe that
> issues occurring only on a specific Java version are rare. If so,
> running tests on a single version for a visa should be enough.
>
> On the other hand, a uniform test environment sounds a good idea
> (especially for a visa), preventing a test status change by "magic".
>
> So, I think that it can be done as follows:
> 1. Run tests for a visa on a baseline Java version (11?).
> 2. Run Nightly builds on all supported versions, choosing version at
> random is a one of options for that. (BTW, do we need some changes in
> TC bot to send a proper alert when test fails on some versions and
> fails on others?).
>
> ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov :
> >
> > Folks,
> >
> > Maybe I'm missing something, but what is the reason for setting
> > JAVA_HOME randomly? For instance, some commit is working under jdk8
> > but fails under jdk9 (and we declare that 9-th version is supported by
> > Ignite). Should we merge this commit to the master branch (all test
> > suites can be OK under jdk8)?
> >
> > I know, that running all tests under all java version leads us to
> > Hell, but maybe we can have separated versions nightly builds?
> >
> > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван  wrote:
> > >
> > > Dmitriy,
> > >
> > > Thank you for the hint! Java version seems to be the reason. I see
> > > that tests pass with Java 8 and 11 and fail with Java 10. See the
> > > exception below. Btw, do we test with Java 9? Have not seen that
> > > version in recent runs.
> > > java.lang.NoClassDefFoundError: javax/transaction/SystemException
> > > at java.base/java.lang.Class.forName0(Native Method)
> > > at java.base/java.lang.Class.forName(Class.java:291)
> > > at
> org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249)
> > > at
> org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235)
> > > at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427)
> > > at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869)
> > > at
> org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886)
> > > at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135)
> > > at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1992)
> > > at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1683)
> > > at
> org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1109)
> > > at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1027)
> > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:913)
> > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:812)
> > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682)
> > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:651)
> > > at org.apache.ignite.Ignition.start(Ignition.java:346)
> > > at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
> > > Caused by: java.lang.ClassNotFoundException:
> javax.transaction.SystemException
> > > at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
> > > at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190)
> > > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499)
> > > ... 18 more
> > >
> > > вт, 18 июн. 2019 г. в 23:27, Dmitriy Pavlov :
> > > >
> > > > Hi Ivan,
> > > >
> > > > Can these failures be related to Java version? Java home is set
> randomly by
> > > > TC Bot.
> > > >
> > > > Dmitriy Pavlov
> > > >
> > > > вт, 18 июн. 2019 г. в 22:11, Павлухин Иван :
> > > >
> > > > > Hi igniters,
> > > > >
> > > > > Does anyone know why python, php and nodejs [1, 2, 3] suites fail
> so
> > > > > frequently on TC? Is there any activity to deal with it?
> > > > >
> > > > > [1]
> > > > >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPython_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv
> > > > > [2]
> > > > >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientNodeJs_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv
> > > > > [3]
> > > > >
> 

Re: [VOTE] Complete Discontinuation of IGFS and Hadoop Accelerator

[Dmitriy Pavlov]

+1

чт, 20 июн. 2019 г. в 09:15, Nikolay Izhikov :

> +1
>
> чт, 20 июня 2019 г., 8:28 Павлухин Иван :
>
> > +1
> >
> > чт, 20 июн. 2019 г. в 04:18, Valentin Kulichenko
> > :
> > >
> > > +1
> > >
> > > On Wed, Jun 19, 2019 at 5:58 PM Roman Shtykh  >
> > > wrote:
> > >
> > > > +1
> > > >
> > > >
> > > > On Thursday, June 20, 2019, 7:34:47 a.m. GMT+9, Andrey Gura <
> > > > ag...@apache.org> wrote:
> > > >
> > > >  +1
> > > >
> > > > On Thu, Jun 20, 2019 at 12:58 AM Denis Magda 
> > wrote:
> > > > >
> > > > > Igniters,
> > > > >
> > > > > Based on our earlier discussion [1], let's formalize our decision
> and
> > > > vote
> > > > > for the following:
> > > > >
> > > > >- IGFS and In-Memory Hadoop Accelerator components are to be
> > > > >discontinued and no longer supported by the community
> > > > >- The existing source code of IGFS and In-Memory Hadoop
> > Accelerator is
> > > > >to be removed from Ignite master. Before that, a special branch
> > like
> > > > >"ignite-igfs-and-hadoop-accelerator" to be forked off the master
> > in
> > > > order
> > > > >to preserve the sources in Git history for those who might need
> > it.
> > > > >
> > > > > Please cast your vote:
> > > > >
> > > > > +1 - agree
> > > > > 0 - indifferent
> > > > > -1 - disagree, explain why.
> > > > >
> > > > > The vote lasts for the next 72 hours.
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html#a42326
> > > > >
> > > > > -
> > > > > Denis
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>

Re: [VOTE] Complete Discontinuation of IGFS and Hadoop Accelerator

[Nikolay Izhikov]

+1

чт, 20 июня 2019 г., 8:28 Павлухин Иван :

> +1
>
> чт, 20 июн. 2019 г. в 04:18, Valentin Kulichenko
> :
> >
> > +1
> >
> > On Wed, Jun 19, 2019 at 5:58 PM Roman Shtykh 
> > wrote:
> >
> > > +1
> > >
> > >
> > > On Thursday, June 20, 2019, 7:34:47 a.m. GMT+9, Andrey Gura <
> > > ag...@apache.org> wrote:
> > >
> > >  +1
> > >
> > > On Thu, Jun 20, 2019 at 12:58 AM Denis Magda 
> wrote:
> > > >
> > > > Igniters,
> > > >
> > > > Based on our earlier discussion [1], let's formalize our decision and
> > > vote
> > > > for the following:
> > > >
> > > >- IGFS and In-Memory Hadoop Accelerator components are to be
> > > >discontinued and no longer supported by the community
> > > >- The existing source code of IGFS and In-Memory Hadoop
> Accelerator is
> > > >to be removed from Ignite master. Before that, a special branch
> like
> > > >"ignite-igfs-and-hadoop-accelerator" to be forked off the master
> in
> > > order
> > > >to preserve the sources in Git history for those who might need
> it.
> > > >
> > > > Please cast your vote:
> > > >
> > > > +1 - agree
> > > > 0 - indifferent
> > > > -1 - disagree, explain why.
> > > >
> > > > The vote lasts for the next 72 hours.
> > > >
> > > >
> > > > [1]
> > > >
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html#a42326
> > > >
> > > > -
> > > > Denis
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>