Re: [MTCGA]: new failures in builds [2635316] needs to be handled
This is my fault, introduced within https://issues.apache.org/jira/browse/IGNITE-10715 I have no idea why it had not been failed during testing on TC. I will prepare the fix quickly. On Tue, Dec 25, 2018 at 2:37 AM wrote: > > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were a > volunteer to make the contribution to this project, but things change and you > may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and fix > test failures or step down and some committer may revert you commit. > > *New stable failure of a flaky test in master > IgniteClientConnectTest.testClientConnectToBigTopology > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=9021926721143358489=%3Cdefault%3E=testDetails > Changes may lead to failure were done by > - ilya.kasnacheev > https://ci.ignite.apache.org/viewModification.html?modId=850683 > - daradurvs > https://ci.ignite.apache.org/viewModification.html?modId=850663 > - antonovsergey93 > https://ci.ignite.apache.org/viewModification.html?modId=850675 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 02:36:58 25-12-2018 -- Best Regards, Vyacheslav D.
[jira] [Created] (IGNITE-10809) IgniteClusterActivateDeactivateTestWithPersistence.testActivateFailover3 fails in master
Sergey Chugunov created IGNITE-10809: Summary: IgniteClusterActivateDeactivateTestWithPersistence.testActivateFailover3 fails in master Key: IGNITE-10809 URL: https://issues.apache.org/jira/browse/IGNITE-10809 Project: Ignite Issue Type: Bug Reporter: Sergey Chugunov Assignee: Sergey Chugunov Fix For: 2.8 Test logic involves independent activation two sets of nodes and then their join into a single cluster. After introducing BaselineTopology concept in 2.4 version this action became prohibited to enforce data integrity. Test should be refactored to take this into account. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [MTCGA]: new failures in builds [2634856] needs to be handled
The failure has been introduced within task: https://issues.apache.org/jira/browse/IGNITE-10671 Double initialization of segmentAware and FileArchiver lead to race breaking file compression. The cause is a missed '@Test' annotation. Here is the patch to fix [1], TC test is green [2]. [1] https://github.com/apache/ignite/pull/5737 [2] https://ci.ignite.apache.org/viewLog.html?buildId=2638777 On Mon, Dec 24, 2018 at 8:52 PM wrote: > > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were a > volunteer to make the contribution to this project, but things change and you > may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and fix > test failures or step down and some committer may revert you commit. > > *Recently contributed test failed in master > WalCompactionSwitchOnTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6257556032945731756=%3Cdefault%3E=testDetails > Changes may lead to failure were done by > - pvoronkin > https://ci.ignite.apache.org/viewModification.html?modId=850576 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 20:51:57 24-12-2018 -- Best Regards, Vyacheslav D.
[MTCGA]: new failures in builds [2635316] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New stable failure of a flaky test in master IgniteClientConnectTest.testClientConnectToBigTopology https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=9021926721143358489=%3Cdefault%3E=testDetails Changes may lead to failure were done by - ilya.kasnacheev https://ci.ignite.apache.org/viewModification.html?modId=850683 - daradurvs https://ci.ignite.apache.org/viewModification.html?modId=850663 - antonovsergey93 https://ci.ignite.apache.org/viewModification.html?modId=850675 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 02:36:58 25-12-2018
Re: Continuous queries and duplicates
>
> In my case, values are immutable - I never change them, I just add new
> entry for newer versions. Does it mean that I won't have any duplicates
> between the initial query and listener entries when using continuous
> queries on caches supporting MVCC?
I'm afraid there still might be a race. Val, Vladimir, other Ignite
experts, please confirm.
After reading the related thread (
>
> http://apache-ignite-developers.2346864.n4.nabble.com/Continuous-queries-and-MVCC-td33972.html
> )
> I'm now concerned about the ordering. My case assumes that there are groups
> of entries which belong to a business aggregate object and I would like to
> make sure that if I commit two records in two serial transactions then I
> have notifications in the same order. Those entries will have different
> keys so based on what you said ("we'd better to leave things as is and
> guarantee only per-key ordering"), it would seem that the order is not
> guaranteed. But do you think it would possible to guarantee order when
> those entries share the same affinity key and they belong to the same
> partition?
The order should be the same for key-value transactions. Vladimir, could
you clear out MVCC based behavior?
--
Denis
On Mon, Dec 17, 2018 at 9:55 AM Piotr Romański
wrote:
> Hi all, sorry for answering so late.
>
> I would like to use SqlQuery because I can leverage indexes there.
>
> As it was already mentioned earlier, the partition update counter is
> exposed through CacheQueryEntryEvent. Initially, I thought that the
> partition update counter is something what's persisted together with the
> data but I'm guessing now that this is only a part of the notification
> mechanism.
>
> I imagined that I would be able to implement my own deduplicaton by having
> 3 stages on the client side: 1. Keep processing initial query results,
> store their keys in memory, 2. When initial query is over, then process
> listener entries but before that check if they have been already delivered
> in the first stage, 3. When we are sure that we are already processing
> notifications for commits executed after initial query was done, then we
> can process listener entries without any additional checks (so our key set
> from stage 1 can be removed from memory). The problem is that I have no way
> to say that I can move from stage 2 to 3. Another problem is that we need
> to stash listener entries while still processing initial query results
> causing an excessive memory pressure on our client.
>
> In my case, values are immutable - I never change them, I just add new
> entry for newer versions. Does it mean that I won't have any duplicates
> between the initial query and listener entries when using continuous
> queries on caches supporting MVCC?
>
> After reading the related thread (
>
> http://apache-ignite-developers.2346864.n4.nabble.com/Continuous-queries-and-MVCC-td33972.html
> )
> I'm now concerned about the ordering. My case assumes that there are groups
> of entries which belong to a business aggregate object and I would like to
> make sure that if I commit two records in two serial transactions then I
> have notifications in the same order. Those entries will have different
> keys so based on what you said ("we'd better to leave things as is and
> guarantee only per-key ordering"), it would seem that the order is not
> guaranteed. But do you think it would possible to guarantee order when
> those entries share the same affinity key and they belong to the same
> partition?
>
> Piotr
>
> pt., 14 gru 2018, 19:31: Denis Magda napisał(a):
>
> > Vladimir,
> >
> > Thanks for referring to the MVCC and Continuous Queries discussion, I
> knew
> > that saw us discussing a solution of the duplication problem. Let me copy
> > and paste it in here for others:
> >
> > 2) *Initial query*. We implemented it so that user can get some initial
> > > data snapshot and then start receiving events. Without MVCC we have no
> > > guarantees of visibility. E.g. if key is updated from V1 to V2, it is
> > > possible to see V2 in initial query and in event. With MVCC it is now
> > > technically possible to query data on certain snapshot and then receive
> > > only events happened after this snapshot. So that we never see V2
> twice.
> > > Do
> > > you think we this feature will be interesting for our users?
> >
> >
> > Am I right that this would be a generic solution - whether you use Scan
> or
> > SQL query as an initial one? Have we planned it for the transactional SQL
> > GA or it's out of scope for now?
> >
> > --
> > Denis
> >
> > On Thu, Dec 13, 2018 at 12:40 PM Vladimir Ozerov
> > wrote:
> >
> > > [1]
> > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Continuous-queries-and-MVCC-td33972.html
> > >
> > > On Thu, Dec 13, 2018 at 11:38 PM Vladimir Ozerov >
> > > wrote:
> > >
> > > > Denis,
> > > >
> > > > Not really. They are used to ensure that ordering of notifications is
> > > > consistent with ordering of updates, so that when a key
Re: Query history statistics API
Yuri, Vladimir, How expensive will it be to have the history enabled by default? For instance, let's take the running queries mechanics as an example, what is the performance impact the did? As for the running queries, Yuri, if it's completed please send a summary to the relevant discussion explaining how it will be used by the end user. Plus, it might be a right time to restart our KILL command conversation, we haven't come to an agreement in regards the syntax. -- Denis On Fri, Dec 21, 2018 at 8:34 AM Юрий wrote: > Vladimir, thanks for your expert opinion. > > I have some thoughts about 5 point. > I tried to find how it works for Oracle and PG: > > *PG*: keep by default 1000 (can be configured) statements without and > discard the least-executed statements. Update statistics is asynchronous > process and statistics may have lag. > > *Oracle*: use shared pool for historical data and can evict records with > min time of last execution in case free space at shared pool is not enough > for a data which can be related not only historical statistics. So seems > also separate asynchronous process (information about it so small). > > > Unfortunately I could not find information about big workload and how it > handled for these databases. However We could see that both of vendors use > asynchronous statistic processing. > > > I see few variants how we can handle very high workload. > > First part of variants use asynchronous model with separate thread which > should take elements to update stats from a queue: > 1) We blocking on overlimited queue and wait when capacity will be enough > to put new element. > > + We have all actual statistics > - End of our query execution can be blocked. > > 2) Discard statistics for ended query in case queue is full. > > + Very fast for current query > - We lose part of statistics. > > 3) Do full clean of statistic's queue. > > + Fast and freespace for further elements > - We lose big number of statistic elements. > > > Second part of variants use current approach for queryMetrics. When we have > some additional capacity for CHM with history + periodical cleanup the Map. > In case even the additional space is not enough we can : > 1) Discard statistics for ended query > 2) Do full clean CHM and discard all gathered information. > > First part of variants potentially should work faster due to we can update > history Map in single thread without contention and put to queue should be > faster. > > > What do you think? Which of the variant will be prefer or may be you can > suggest another way to handle potential huge workload? > > Also there is one initial question which stay not clear to me - it is right > place for new API. > > > пт, 21 дек. 2018 г. в 13:05, Vladimir Ozerov : > > > Hi, > > > > I'd propose the following approach: > > 1) Enable history by default. Becuase otherwise users will have to > restart > > the node to enable it, or we will have to implement dynamic history > enable, > > which is complex thing. Default value should be relatively small yet > > allowing to accommodate typical workloads. E.g. 1000 entries. This should > > not put any serious pressure to GC. > > 2) Split queries by: schema, query, local flag > > 3) Track only growing values: execution count, error count, minimum > > duration, maximum duration > > 4) Implement ability to clear history - JMX, SQL command, whatever (may > be > > this is different ticket) > > 5) History cleanup might be implemented similarly to current approach: > > store everything in CHM. Periodically check it's size. If it is too big - > > evict oldest entries. But this should be done with care - under some > > workloads new queries will be generated very quickly. In this case we > > should either fallback to synchronous evicts, or do not log history at > all. > > > > Thoughts? > > > > Vladimir. > > - > > > > On Fri, Dec 21, 2018 at 11:22 AM Юрий > wrote: > > > > > Alexey, > > > > > > Yes, such property to configuration history size will be added. I think > > > default value should be 0 and history by default shouldn't be gather at > > > all, and can be switched on by property in case when it required. > > > > > > Currently I planned use the same way to evicting old data as for > > > queryMetrics - scheduled task will evict will old data by oldest start > > time > > > of query. > > > > > > Will be gathered statistics for only initial clients queries, so > internal > > > queries will not including. For the same queries we will have one > record > > in > > > history with merged statistics. > > > > > > All above points just my proposal. Please revert back in case you think > > > anything should be implemented by another way. > > > > > > > > > > > > > > > > > > чт, 20 дек. 2018 г. в 18:23, Alexey Kuznetsov : > > > > > > > Yuriy, > > > > > > > > I have several questions: > > > > > > > > Are we going to add some properties to cluster configuration for > > history > > > > size? > > > > > > > > And what will be default history
Re: Dealing with changing class definitions in Ignite
Gert, All,
That's not a peer-class-loading issue but rather the binary objects
limitation. It's impossible to change the *type* of an object field. You
can add new fields or remove existing ones. The same limitation applies to
SQL - now way to change the type in runtime. That latter should be
addressed in 2019.
Vladimir, shouldn't it be easier to support fields type change for compute
runnables which are not stored in caches/tables?
--
Denis
On Thu, Dec 20, 2018 at 4:24 AM Gert Dubois
wrote:
> For now we can work around the issue by overriding the
> BinaryNameMarshaller and ensuring our clients append a unique identifier
> behind the class name. This ensures that every client gets their own Binary
> Metadata records in the Marshaller for every class they serialize in the
> cluster.
>
> .setNameMapper(new BinaryNameMapper() {
> @Override
> public String typeName(String clsName) {
> if(shouldAddSuffix) {
> return clsName + "_clientVersionId";
> }
> return clsName;
> }
>
> @Override
> public String fieldName(String fieldName) {
> return fieldName;
> }
> });
>
> But we expected the ISOLATED deployment mode would have solved this issue
> for us. It also forces us to maintain some kind of client versioning system
> so when the classes in our jobs change they get a fresh record in the
> Marshaller.
>
> On Wed, Dec 19, 2018 at 6:30 PM Ilya Kasnacheev
> wrote:
>
>> Hello!
>>
>> I have found that you can avoid compute job type incompatibility problem
>> (please see history below) by setting
>>
>> .setMarshaller(new OptimizedMarshaller())
>>
>> in your Ignite configuration on all nodes.
>>
>> However, it is not clear at all why this is needed. Anybody can help? Why
>> compute jobs are processed by capricious BinaryMarshaller?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 19 дек. 2018 г. в 02:29, Gert Dubois :
>>
>>> Hey Ilya,
>>>
>>> I opened a ticket on the ignite Jira about it.
>>> https://issues.apache.org/jira/browse/IGNITE-10717
>>>
>>> I attached a zip file containing a maven project with sample code that
>>> reproduces our issue. Reproducing the issue is rather easy though
>>>
>>> 1. Have a client + server, with peer class loading enabled
>>> 2. Create a simple class that implements IgniteRunnable with some class
>>> field
>>> 3. Execute an instance of this class on the ignite cluster
>>> 4. Change the type of the field
>>> 5. Recompile and execute again. Now it breaks because the class can't be
>>> serialized using the binary marshaller.
>>>
>>> Code to execute these steps is included in the maven project.
>>>
>>> On Tue, Dec 18, 2018 at 12:04 PM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
Hello!
Can you show an example of "Ignite Runnables conflicting on the Binary
Marshaller"? As a small code snippet perhaps?
Maybe I could recommend something but I lack understanding of your use
case.
Regards,
--
Ilya Kasnacheev
пн, 17 дек. 2018 г. в 18:12, Gert Dubois :
> Thanks for the reply.
>
> Everything related to the cache we understand the current
> architecture, in our code base we'll probably treat everything cache
> related the same as schema migrations in a DB (migration scripts, etc.).
> Our real issue is with Ignite Runnables conflicting on the Binary
> Marshaller. Every class that gets executed on Ignite as a job gets a
> stored
> definition, this means we can't refactor classes that get used internally
> by our Ignite Runnables, because doing so would prevent the updated code
> from running. Even worse, if we update our libraries and they changed
> class
> definitions we might run into the same issue, without changing a letter in
> our own code. From the documentation it looked like Deployment Mode could
> provide a solution for this issue but the Binary Marshaller seems to run
> completely separate from the Deployment Mode.
>
> On Mon, Dec 17, 2018 at 3:18 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>> Hello!
>>
>> As far as my understanding goes:
>> You can't peer class load your Key/Value types.
>> You also can't redeploy your Key/Value types.
>> They even survive node restart via WORKDIR/marshaller directory, and
>> come back to haunt you.
>>
>> There are plans to maybe ease some of those limitations in 3.0, but
>> nothing concrete yet. It is not a bug rather a pillar of current Ignite
>> architecture. You will have to route around it, such as introducing new
>> fields instead of changing types. And maybe avoid having those types on
>> server nodes at all, and relying on
Re: Need help on Apache Ignite tests
Hello! As far as my understanding goes it checks some basic invariants of events() API - e.g. that it does not crash if you try to de-register inexistent listener or that it returns a valid handle if you register a new one. Regards, -- Ilya Kasnacheev пн, 24 дек. 2018 г. в 20:14, Chandranana Naik < chandranana_n...@persistent.com>: > Hi Team, > I have recently started working with Apache Ignite. Build on x86 Ubuntu > 16.04 is complete. I wanted some information on the tests I am running. > > Could you please give some information on what the test does - > GridEventConsumeSelfTest#testApi, I want to understand the functionality of > the test . > > Thanks in advance. > > Regards, > Chandranana > > DISCLAIMER > == > This e-mail may contain privileged and confidential information which is > the property of Persistent Systems Ltd. It is intended only for the use of > the individual or entity to which it is addressed. If you are not the > intended recipient, you are not authorized to read, retain, copy, print, > distribute or use this message. If you have received this communication in > error, please notify the sender and delete all copies of this message. > Persistent Systems Ltd. does not accept any liability for virus infected > mails. >
[MTCGA]: new failures in builds [2634856] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *Recently contributed test failed in master WalCompactionSwitchOnTest.initializationError https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6257556032945731756=%3Cdefault%3E=testDetails Changes may lead to failure were done by - pvoronkin https://ci.ignite.apache.org/viewModification.html?modId=850576 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 20:51:57 24-12-2018
Re: Service grid redesign
Hello, Igniters. Please, let us know, if someone want to do additional review of this PR. В Пн, 24/12/2018 в 20:23 +0300, Vyacheslav Daradur пишет: > Igniters, especially future reviewers, > > Discovery listener registered by 'IgniteServiceProcessor' become > implemented 'HighPriorityListener', seems it's best lock-free > solutions discussed during the review. This change is covered by > `ServiceDeploymentDiscoveryListenerNotificationOrderTest` which should > protect us if the order of listeners will be changed. > > It's about the problem of custom messages which are nullified by PME > [1] and are listened by service deployment to manage the lifecycle of > affinity services. This guarantees that service deployment discovery > listener will be notified earlier than PME's discovery listener and > will be able to capture custom messages which may be nullified in PME > process. > > Looks like we do not have any controversial questions now. > > Thanks! > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/Danger-change-of-DiscoveryCustomEvent-in-GridDhtPartitionsExchangeFuture-onDone-td35946.html > > > On Mon, Dec 24, 2018 at 4:23 PM Vyacheslav Daradur > wrote: > > > > Stanislav, thank you for the notes, most of them have been resolved. I > > answered on GitHub. > > > > > > On Sun, Dec 23, 2018 at 9:34 PM Stanislav Lukyanov > > wrote: > > > > > > I’ve done a quick superficial review. Didn’t look at the tests, didn’t > > > dive into the design, etc, just the code. > > > I’ve left some comments – almost all are about minor issues, grammar and > > > code style. > > > > > > Stan > > > > > > From: Vyacheslav Daradur > > > Sent: 21 декабря 2018 г. 14:58 > > > To: dev@ignite.apache.org > > > Subject: Re: Service grid redesign > > > > > > Igniters, > > > > > > Please, let us know if someone is going to do an additional review? > > > > > > We should know can we merge the PR since it has been approved by > > > Nikolay Izhikov and Denis Mekhanikov or we should wait for other > > > community members. > > > > > > On Thu, Dec 20, 2018 at 7:52 PM Vyacheslav Daradur > > > wrote: > > > > > > > > I think I found names which should satisfy me and Denis, and possibly > > > > Nikolay ) > > > > > > > > See the following names (Actual name <- Previously used): > > > > > > > > - ServiceDeploymentManager <- ServicesDeploymentManager > > > > - ServiceDeploymentActions <- ServicesDeploymentActions > > > > - ServiceDeploymentProcessId <- ServicesDeploymentProcessId > > > > - ServiceDeploymentTask <- ServicesDeploymentTask > > > > > > > > - ServiceDeploymentRequest <- ServiceDeploymentChange > > > > - ServiceUndeploymentRequest <- ServiceUndeploymentChange > > > > - ServiceChangeAbstractRequest <- ServiceAbstractChange > > > > > > > > - ServiceSingleNodeDeploymentResult <- ServiceSingleDeploymentsResults > > > > - ServiceSingleNodeDeploymentResultBatch <- > > > > ServicesSingleDeploymentsMessage > > > > > > > > - ServiceClusterDeploymentResult <- ServiceFullDeploymentsResults > > > > - ServiceClusterDeploymentResultBatch <- ServicesFullDeploymentsMessage > > > > > > > > - ServiceProcessorCommonDiscoveryData <- ServicesCommonDiscoveryData > > > > - ServiceProcessorJoinNodeDiscoveryData <- ServicesJoinNodeDiscoveryData > > > > > > > > Also, I had a short talk with Alexey Goncharuk about the problem of > > > > nullified custom messages. I changed the implementation to a lock-free > > > > solution which allows us to nullify messages depend on an using > > > > counter. > > > > > > > > In comparison with high priority listener, this allows us to not copy > > > > custom discovery event in service deployment manager and work with the > > > > original object. > > > > > > > > On Thu, Dec 20, 2018 at 8:57 AM Nikolay Izhikov > > > > wrote: > > > > > > > > > > Denis, great news! > > > > > > > > > > Alexey, Vova, Yakov, do you want to take a look at this PR? > > > > > > > > > > > > > > > > > > > > В Ср, 19/12/2018 в 18:47 +0300, Denis Mekhanikov пишет: > > > > > > Guys, > > > > > > > > > > > > I finished my code review. The pool request looks good to me. > > > > > > > > > > > > Does anybody else want to look at the changes? > > > > > > There are a few points, that we didn't meet an agreement on, > > > > > > though they don't affect the behaviour in any way: > > > > > > > > > > > >- *Class naming. * See the discussion above. > > > > > >- *Unnecessary task object cleaning. * > > > > > >IMO, ServicesDeploymentTask#clear() method doesn't do anything > > > > > > useful, > > > > > >and it should be removed. > > > > > >By the moment, when this method is called, the task object is > > > > > > removed > > > > > >from all collections anyway, so it's ready for garbage > > > > > > collection. > > > > > >Removing data from it doesn't help anybody. > > > > > >- > > > > > > *Unnecessary tests. *ServiceInfoSelfTest and > > > > > >ServicesDeploymentProcessIdSelfTest look
Re: Service grid redesign
Igniters, especially future reviewers, Discovery listener registered by 'IgniteServiceProcessor' become implemented 'HighPriorityListener', seems it's best lock-free solutions discussed during the review. This change is covered by `ServiceDeploymentDiscoveryListenerNotificationOrderTest` which should protect us if the order of listeners will be changed. It's about the problem of custom messages which are nullified by PME [1] and are listened by service deployment to manage the lifecycle of affinity services. This guarantees that service deployment discovery listener will be notified earlier than PME's discovery listener and will be able to capture custom messages which may be nullified in PME process. Looks like we do not have any controversial questions now. Thanks! [1] http://apache-ignite-developers.2346864.n4.nabble.com/Danger-change-of-DiscoveryCustomEvent-in-GridDhtPartitionsExchangeFuture-onDone-td35946.html On Mon, Dec 24, 2018 at 4:23 PM Vyacheslav Daradur wrote: > > Stanislav, thank you for the notes, most of them have been resolved. I > answered on GitHub. > > > On Sun, Dec 23, 2018 at 9:34 PM Stanislav Lukyanov > wrote: > > > > I’ve done a quick superficial review. Didn’t look at the tests, didn’t dive > > into the design, etc, just the code. > > I’ve left some comments – almost all are about minor issues, grammar and > > code style. > > > > Stan > > > > From: Vyacheslav Daradur > > Sent: 21 декабря 2018 г. 14:58 > > To: dev@ignite.apache.org > > Subject: Re: Service grid redesign > > > > Igniters, > > > > Please, let us know if someone is going to do an additional review? > > > > We should know can we merge the PR since it has been approved by > > Nikolay Izhikov and Denis Mekhanikov or we should wait for other > > community members. > > > > On Thu, Dec 20, 2018 at 7:52 PM Vyacheslav Daradur > > wrote: > > > > > > I think I found names which should satisfy me and Denis, and possibly > > > Nikolay ) > > > > > > See the following names (Actual name <- Previously used): > > > > > > - ServiceDeploymentManager <- ServicesDeploymentManager > > > - ServiceDeploymentActions <- ServicesDeploymentActions > > > - ServiceDeploymentProcessId <- ServicesDeploymentProcessId > > > - ServiceDeploymentTask <- ServicesDeploymentTask > > > > > > - ServiceDeploymentRequest <- ServiceDeploymentChange > > > - ServiceUndeploymentRequest <- ServiceUndeploymentChange > > > - ServiceChangeAbstractRequest <- ServiceAbstractChange > > > > > > - ServiceSingleNodeDeploymentResult <- ServiceSingleDeploymentsResults > > > - ServiceSingleNodeDeploymentResultBatch <- > > > ServicesSingleDeploymentsMessage > > > > > > - ServiceClusterDeploymentResult <- ServiceFullDeploymentsResults > > > - ServiceClusterDeploymentResultBatch <- ServicesFullDeploymentsMessage > > > > > > - ServiceProcessorCommonDiscoveryData <- ServicesCommonDiscoveryData > > > - ServiceProcessorJoinNodeDiscoveryData <- ServicesJoinNodeDiscoveryData > > > > > > Also, I had a short talk with Alexey Goncharuk about the problem of > > > nullified custom messages. I changed the implementation to a lock-free > > > solution which allows us to nullify messages depend on an using > > > counter. > > > > > > In comparison with high priority listener, this allows us to not copy > > > custom discovery event in service deployment manager and work with the > > > original object. > > > > > > On Thu, Dec 20, 2018 at 8:57 AM Nikolay Izhikov > > > wrote: > > > > > > > > Denis, great news! > > > > > > > > Alexey, Vova, Yakov, do you want to take a look at this PR? > > > > > > > > > > > > > > > > В Ср, 19/12/2018 в 18:47 +0300, Denis Mekhanikov пишет: > > > > > Guys, > > > > > > > > > > I finished my code review. The pool request looks good to me. > > > > > > > > > > Does anybody else want to look at the changes? > > > > > There are a few points, that we didn't meet an agreement on, > > > > > though they don't affect the behaviour in any way: > > > > > > > > > >- *Class naming. * See the discussion above. > > > > >- *Unnecessary task object cleaning. * > > > > >IMO, ServicesDeploymentTask#clear() method doesn't do anything > > > > > useful, > > > > >and it should be removed. > > > > >By the moment, when this method is called, the task object is > > > > > removed > > > > >from all collections anyway, so it's ready for garbage collection. > > > > >Removing data from it doesn't help anybody. > > > > >- > > > > > *Unnecessary tests. *ServiceInfoSelfTest and > > > > >ServicesDeploymentProcessIdSelfTest look excessive to me. > > > > >I don't see any point in testing an interface implementation, that > > > > > only > > > > >saves some objects and returns them from certain methods. > > > > >- Interface for events with servicesDeploymentActions() method. > > > > >Take a look at the discussion: > > > > > > > > > > https://github.com/apache/ignite/pull/4434/files/30e69d9a53ce6ea16c4e9d15354e94360caa719d#r239442342 >
Need help on Apache Ignite tests
Hi Team, I have recently started working with Apache Ignite. Build on x86 Ubuntu 16.04 is complete. I wanted some information on the tests I am running. Could you please give some information on what the test does - GridEventConsumeSelfTest#testApi, I want to understand the functionality of the test . Thanks in advance. Regards, Chandranana DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
[jira] [Created] (IGNITE-10808) Discovery message queue may build up with TcpDiscoveryMetricsUpdateMessage
Stanislav Lukyanov created IGNITE-10808: --- Summary: Discovery message queue may build up with TcpDiscoveryMetricsUpdateMessage Key: IGNITE-10808 URL: https://issues.apache.org/jira/browse/IGNITE-10808 Project: Ignite Issue Type: Bug Reporter: Stanislav Lukyanov Attachments: IgniteMetricsOverflowTest.java A node receives a new metrics update message every `metricsUpdateFrequency` milliseconds, and the message will be put at the top of the queue (because it is a high priority message). If processing one message takes more than `metricsUpdateFrequency` then multiple `TcpDiscoveryMetricsUpdateMessage` will be in the queue. A long enough delay (e.g. caused by a network glitch or GC) may lead to the queue building up tens of metrics update messages which are essentially useless to be processed. Finally, if processing a message on average takes a little more than `metricsUpdateFrequency` (even for a relatively short period of time, say, for a minute due to network issues) then the message worker will end up processing only the metrics updates and the cluster will essentially hang. Reproducer is attached. In the test, the queue first builds up and then very slowly being teared down, causing "Failed to wait for PME" messages. Need to change ServerImpl's SocketReader not to put another metrics update message to the top of the queue if it already has one (or replace the one on the top with new one). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Replace Cron4J with Quartz for ignite-schedule module.
HI, Igniters! I've updated and rebased implementation to master branch and made some fixes. Also I have a question regarding current implementation. As I found Cron4J source code this implementation checks schedule every minute (seconds not supported) but spawns a thread for every task which scheduling pattern matches the current time. There no any limits to the number of tasks launched silmultaneously. New implementation is based on org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler with its default parameters currently, i.e thread pool size is 1. Could you advise me, do we need to add some system property or introduce some attribute to IgniteConfiguration to configure Scheduler thread pool size? And what should be default value? Best regards, Sergey Kosarev. чт, 10 мая 2018 г. в 20:23, Dmitry Pavlov : > Hi Anton, > > Thank you for joining and review. > I hope all proposals will be applied. > > Sincerely, > Dmitriy Pavlov > > пт, 4 мая 2018 г. в 16:04, Anton Vinogradov : > > > Folks, > > > > How can it be at PATCH AVAILABLE since *none* of my latest comments (made > > Feb 8) are resolved at Upsource? > > Changed state to IP. > > > > пн, 23 апр. 2018 г. в 20:00, Dmitry Pavlov : > > > > > Hi Andrey, > > > > > > Could you please pick up review? > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > пн, 23 апр. 2018 г. в 17:39, Dmitriy Setrakyan >: > > > > > > > Dmitriy, who is a good candidate within the community to review this > > > > ticket? > > > > > > > > On Mon, Apr 23, 2018 at 6:10 AM, Dmitry Pavlov < > dpavlov@gmail.com> > > > > wrote: > > > > > > > > > Hi Igniters, > > > > > > > > > > it seems ticket https://issues.apache.org/jira/browse/IGNITE-5565 > is > > > > still > > > > > in PA state. What are our next steps? > > > > > > > > > > Who did review of this patch? > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > ср, 28 июн. 2017 г. в 1:40, Denis Magda : > > > > > > > > > > > Yakov, > > > > > > > > > > > > No, the mentioned discussion didn’t turn into a JIRA ticket. > > > > > > > > > > > > Alex K., please follow to some thoughts from there and wrap them > up > > > in > > > > a > > > > > > form of the ticket. > > > > > > > > > > > > — > > > > > > Denis > > > > > > > > > > > > > On Jun 26, 2017, at 2:58 AM, Yakov Zhdanov < > yzhda...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > Guys, I remember we discussed this some time ago. > > > > > > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble. > > > > > com/Tasks-Scheduling-and-Chaining-td14293.html > > > > > > > > > > > > > > Denis, do you have any ticket or SoW? > > > > > > > > > > > > > > --Yakov > > > > > > > > > > > > > > > > > > > > > > > > > > >
[MTCGA]: new failures in builds [2636079] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *Recently contributed test failed in master CheckpointReadLockFailureTest.initializationError https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5099637610936041665=%3Cdefault%3E=testDetails Changes may lead to failure were done by - stkuzma https://ci.ignite.apache.org/viewModification.html?modId=850850 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 19:36:56 24-12-2018
Re: Broken layout of Ignite javadoc on web
Hi,
Mentioned javadoc.css references to some image resources. They present
for 2.3.0 release [1]. But similar url returns 404 for 2.7.0 [2]. Who
knows how to upload resources?
[1] https://ignite.apache.org/releases/2.3.0/javadoc/resources
[2] https://ignite.apache.org/releases/2.7.0/javadoc/resources
пн, 17 дек. 2018 г. в 16:38, Павлухин Иван :
>
> Dmitriy,
>
> Yep, it looks like the problem is inside assembly/docfiles/javadoc.css
>
> I will try to dig into some time later on a spare time. Of course, if
> nobody fixes the problem earlier.
> пн, 17 дек. 2018 г. в 15:18, Dmitriy Pavlov :
> >
> > I see the same, browsers checked: Edge & Chrome. I guess it is not a layout
> > is broken, but a style missing.
> >
> > Unfortunately, I don't know how to fix. Probably we should start research
> > from ignite/pom.xml:191
> >
> > ${basedir}/assembly/docfiles/javadoc.css
> >
> >
> > пн, 17 дек. 2018 г. в 14:32, Павлухин Иван :
> >
> > > Hi,
> > >
> > > I noticed that Ignite javadoc layout on web for latest versions has
> > > some problems. For example, I see no links near the "Class" at heading
> > > of the page [1]. Here is a screenshot [2]. Do you see the same?
> > >
> > > I was able to build javadoc on my machine. With current configuration
> > > I see the same broken layout. After using commenting out customized
> > > style sheet configuration (referring to assembly/docfiles/javadoc.css)
> > > I was able to obtain an html which renders fine for me.
> > >
> > > Can anyone suggest a simple solution for this?
> > >
> > > [1]
> > > https://ignite.apache.org/releases/2.7.0/javadoc/org/apache/ignite/cache/eviction/EvictionPolicy.html
> > > [2]
> > > https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/1b257ad73f81cb4698f6105a9d1646295ba55795/javadoc_layout.png
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
--
Best regards,
Ivan Pavlukhin
Re: Service grid redesign
Stanislav, thank you for the notes, most of them have been resolved. I answered on GitHub. On Sun, Dec 23, 2018 at 9:34 PM Stanislav Lukyanov wrote: > > I’ve done a quick superficial review. Didn’t look at the tests, didn’t dive > into the design, etc, just the code. > I’ve left some comments – almost all are about minor issues, grammar and code > style. > > Stan > > From: Vyacheslav Daradur > Sent: 21 декабря 2018 г. 14:58 > To: dev@ignite.apache.org > Subject: Re: Service grid redesign > > Igniters, > > Please, let us know if someone is going to do an additional review? > > We should know can we merge the PR since it has been approved by > Nikolay Izhikov and Denis Mekhanikov or we should wait for other > community members. > > On Thu, Dec 20, 2018 at 7:52 PM Vyacheslav Daradur > wrote: > > > > I think I found names which should satisfy me and Denis, and possibly > > Nikolay ) > > > > See the following names (Actual name <- Previously used): > > > > - ServiceDeploymentManager <- ServicesDeploymentManager > > - ServiceDeploymentActions <- ServicesDeploymentActions > > - ServiceDeploymentProcessId <- ServicesDeploymentProcessId > > - ServiceDeploymentTask <- ServicesDeploymentTask > > > > - ServiceDeploymentRequest <- ServiceDeploymentChange > > - ServiceUndeploymentRequest <- ServiceUndeploymentChange > > - ServiceChangeAbstractRequest <- ServiceAbstractChange > > > > - ServiceSingleNodeDeploymentResult <- ServiceSingleDeploymentsResults > > - ServiceSingleNodeDeploymentResultBatch <- ServicesSingleDeploymentsMessage > > > > - ServiceClusterDeploymentResult <- ServiceFullDeploymentsResults > > - ServiceClusterDeploymentResultBatch <- ServicesFullDeploymentsMessage > > > > - ServiceProcessorCommonDiscoveryData <- ServicesCommonDiscoveryData > > - ServiceProcessorJoinNodeDiscoveryData <- ServicesJoinNodeDiscoveryData > > > > Also, I had a short talk with Alexey Goncharuk about the problem of > > nullified custom messages. I changed the implementation to a lock-free > > solution which allows us to nullify messages depend on an using > > counter. > > > > In comparison with high priority listener, this allows us to not copy > > custom discovery event in service deployment manager and work with the > > original object. > > > > On Thu, Dec 20, 2018 at 8:57 AM Nikolay Izhikov wrote: > > > > > > Denis, great news! > > > > > > Alexey, Vova, Yakov, do you want to take a look at this PR? > > > > > > > > > > > > В Ср, 19/12/2018 в 18:47 +0300, Denis Mekhanikov пишет: > > > > Guys, > > > > > > > > I finished my code review. The pool request looks good to me. > > > > > > > > Does anybody else want to look at the changes? > > > > There are a few points, that we didn't meet an agreement on, > > > > though they don't affect the behaviour in any way: > > > > > > > >- *Class naming. * See the discussion above. > > > >- *Unnecessary task object cleaning. * > > > >IMO, ServicesDeploymentTask#clear() method doesn't do anything > > > > useful, > > > >and it should be removed. > > > >By the moment, when this method is called, the task object is removed > > > >from all collections anyway, so it's ready for garbage collection. > > > >Removing data from it doesn't help anybody. > > > >- > > > > *Unnecessary tests. *ServiceInfoSelfTest and > > > >ServicesDeploymentProcessIdSelfTest look excessive to me. > > > >I don't see any point in testing an interface implementation, that > > > > only > > > >saves some objects and returns them from certain methods. > > > >- Interface for events with servicesDeploymentActions() method. > > > >Take a look at the discussion: > > > > > > > > https://github.com/apache/ignite/pull/4434/files/30e69d9a53ce6ea16c4e9d15354e94360caa719d#r239442342 > > > > > > > > Also solution with *DiscoveryCustomEvent#nullifyingCustomMsgLock* looks > > > > clumsy to me. > > > > The problem with nullifying of *DiscoveryCustomEvent#customMsg* field > > > > can > > > > be solved > > > > by making *ServiceDiscoveryListener* a high priority listener. > > > > > > > > Or *DiscoveryCustomEvent#customMessage()* method could be marked > > > > synchronized and > > > > *GridEventStorageManager#notifyListeners(..)* method could synchronize > > > > on > > > > the event object. > > > > But this solution is the same, it's just a matter of taste. > > > > > > > > If anybody wants to look the the code of the PR, please consider these > > > > points as well. > > > > > > > > Denis > > > > > > > > ср, 19 дек. 2018 г. в 17:37, Nikolay Izhikov : > > > > > > > > > Denis, > > > > > > > > > > I don't think that differences with your and my naming is huge :) > > > > > And, it's definetely a matter of taste. > > > > > > > > > > If there is no any other issues with PR let's rename and move on! :) > > > > > > > > > > ср, 19 дек. 2018 г. в 17:32, Vyacheslav Daradur : > > > > > > > > > > > > We have IgniteServiceProcessor and GridServiceProcessor with > > > > > > > singular > > > > >
Re: Suggestion to improve deadlock detection
Igor, thanks for your feedback! Let's do according to your proposal. пн, 24 дек. 2018 г. в 14:57, Seliverstov Igor : > > Ivan, > > We have to design configuration carefully. > There are possible issues with compatibility and this API is public, it > might be difficult to redesign it after while. > > Since there is a ticket about MVCC related configuration parts [1] (TxLog > memory region/size, vacuum thread count and intervals, etc) > I suggest solving the issue in scope of that ticket. > > At now, let's introduce a couple of system variables for an ability to > change the behavior while development phase. > > [1] https://issues.apache.org/jira/browse/IGNITE-7966 > > Regards, > Igor > > чт, 20 дек. 2018 г. в 16:29, Павлухин Иван : > > > Hi, > > > > I prepared a patch with deadlock detection algorithm implementation > > (you can find it from ticket [1]). > > > > Now I would like to discuss options for configuring deadlock > > detection. From a usability standpoint following need to be supported: > > 1. Turn deadlock detection on/off (on by default). > > 2. Configure a delay between starting waiting for a lock and sending > > first deadlock detection message (e.g. 1 second by default). > > 3. Something else? > > > > I can suggest following options: > > 1. Use a single long for configuration. Values < 0 means disabled > > deadlock detection. 0 means starting detection without a delay. Values > > > 0 specifies a delay in milliseconds before starting detection. > > 2. Use a boolean flag for turning detection on/off. Use a long value > > for configuring delay. > > > > Also there are number of ways to pass this values to Ignite. I do no > > have much experience in adding new configuration options to Ignite and > > I need an advice here. For example, it could be done by adding > > additional properties to TransactionConfiguration class. Another way > > is using java system properties. > > > > Igor, Vladimir and others what do you think? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9322 > > ср, 19 дек. 2018 г. в 15:51, Павлухин Иван : > > > > > > Igor, > > > > > > I see your points. And I agree to start with "lightweight > > > implementation". Today we have 2PL and there is no activity on > > > implementing rollback to savepoint. And if we do it in the future we > > > will have to return to the subject of deadlock detection anyway. > > > > > > I will proceed with "forward-only" approach. > > > вт, 18 дек. 2018 г. в 18:33, Seliverstov Igor : > > > > > > > > Ivan, > > > > > > > > I would prefer forward-only implementation even knowing it allows false > > > > positive check results. > > > > > > > > Why I think so: > > > > > > > > 1) From my experience, when we have any future is waiting for reply, we > > > > have to take a failover into consideration. > > > > Usually failover implementations are way more complex than an initial > > > > algorithm itself. > > > > Forward-only version doesn't need any failover implementation as it > > expects > > > > failed probes (the probes didn't face a deadlock ). > > > > > > > > 2) Simple lightweight feature implementation is a really good point to > > > > start - it may be extended if needed but really often such extending > > > > doesn't need at all. > > > > > > > > 3) Any implementation allow false positive result - we are working with > > > > distributed system, there are a bunch of processes happens at the same > > time > > > > and, > > > > for example, concurrently happened rollback on timeout may solve a > > deadlock > > > > but probe is finished at this time, so- there is a rollback on deadlock > > > > also as a result. > > > > > > > > 4) The described case (when false positive result is probable) isn't > > usual > > > > but, potentially, extremely rare, I don't think we should solve it > > since it > > > > doesn't break the grid. > > > > > > > > Regards, > > > > Igor > > > > > > > > вт, 18 дек. 2018 г. в 17:57, Павлухин Иван : > > > > > > > > > Hi folks, > > > > > > > > > > During implementation of edge-chasing deadlock detection algorithm in > > > > > scope of [1] it has been realized that basically we have 2 options > > for > > > > > "chasing" strategy. I will use terms Near when GridNearTxLocal is > > > > > assumed and Dht when GridDhtTxLocal (tx which updates primary > > > > > partition). So, here is 2 mentioned strategies: > > > > > 1. Send initial probe when Dht tx blocks waiting for another tx to > > > > > release a key. Initial probe is sent from waiting Dht tx to Near tx > > > > > holding a lock. If receiving Near tx is blocked as well then it > > relays > > > > > the probe to Dht tx it awaits response from. So, the probe is > > > > > traveling like Dht -> Near -> Dht -> Near -> ... Let's call the > > > > > approach "forward-only". > > > > > 2. Send probes only between Near transactions. This approach requires > > > > > additional request-response call which Near tx issues to Dht node to > > > > > check if tx sending a request is waiting for another
[jira] [Created] (IGNITE-10807) Support long-running tx rollback on PME
Ivan Pavlukhin created IGNITE-10807:
---
Summary: Support long-running tx rollback on PME
Key: IGNITE-10807
URL: https://issues.apache.org/jira/browse/IGNITE-10807
Project: Ignite
Issue Type: Bug
Components: mvcc
Reporter: Ivan Pavlukhin
Currently long-running tx rollback on PME does not work for MVCC transactions.
{{org.apache.ignite.internal.processors.cache.transactions.TxRollbackOnTopologyChangeTest#testRollbackOnTopologyChange}}
can be used as starting point.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-10806) Test failure on Big Endian
Namrata Bhave created IGNITE-10806:
--
Summary: Test failure on Big Endian
Key: IGNITE-10806
URL: https://issues.apache.org/jira/browse/IGNITE-10806
Project: Ignite
Issue Type: Bug
Affects Versions: 2.7, 2.6
Environment: Linux Ubuntu 16.04 Big Endian
Reporter: Namrata Bhave
{{Hi,}}
{{`*GridMessageCollectionTest.testMarshal->doTestMarshal*` from
`*IgniteBasicTestSuite*` fails with below error on *big endian platform*:}}
_{{[ERROR] testMarshal(org.apache.ignite.util.GridMessageCollectionTest) Time
elapsed: 0 s <<< FAILURE!}}_
_{{junit.framework.AssertionFailedError: expected:<115> but was:<29440>}}_
_{{at
org.apache.ignite.util.GridMessageCollectionTest.doTestMarshal(GridMessageCollectionTest.java:120)}}_
_{{at
org.apache.ignite.util.GridMessageCollectionTest.doTestMarshal(GridMessageCollectionTest.java:89)}}_
_{{at
org.apache.ignite.util.GridMessageCollectionTest.testMarshal(GridMessageCollectionTest.java:71)}}_
{{As per my analysis, this error occurs when proto is set to 1 in
[GridMessageCollectionTest.java|}}{{https://github.com/apache/ignite/blob/2.7.0/modules/core/src/test/java/org/apache/ignite/util/GridMessageCollectionTest.java#L74}}{{]
and test passes for proto=2.}}
{{While checking the implementation of
[DirectByteBufferStreamImplV2.java|}}{{https://github.com/apache/ignite/blob/2.7.0/modules/core/src/main/java/org/apache/ignite/internal/direct/stream/v2/DirectByteBufferStreamImplV2.java#L1422}}{{],
observed write/read methods are defined based on endianness and hence test
passes for proto=2. I tried making those changes in
DirectByteBufferStreamImplV1.java(for proto=1) and test passed. }}
{{I am not aware about the differences in the two protocol versions and their
implementation. Can the changes based on endianness be incorporated in v1 as
well? Could someone please help to resolve this? }}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Set 'TcpDiscoveryVmIpFinder' as default IP finder for tests instead of 'TcpDiscoveryMulticastIpFinder'
Folks, The important thing here is that 99% of "final static" ip-finders were removed. (~800 pcs.) Non-static, mostly, kept as is since removal may or cause a test failure. (~130 pcs.) In case someone interested in the continuation of cleanup, please feel free to create an issue and provide PR, I'm ready to review it. On Mon, Dec 24, 2018 at 3:04 PM Vyacheslav Daradur wrote: > The second step [2] "removing related boilerplate in tests" - has been > done. It is expected that a bit memory will be released at tests TC > agents, which could not be cleaned by GC because of static final > fields. > > Guys, please, do not generate a new one ) > > [1] https://issues.apache.org/jira/browse/IGNITE-10715 > > On Tue, Dec 18, 2018 at 6:29 PM Anton Vinogradov wrote: > > > > Folks, > > > > Now I see 5-10% speedup at all TC suites right after the merge. > > Great fix! > > > > > > On Mon, Dec 17, 2018 at 2:39 PM Vyacheslav Daradur > > wrote: > > > > > Andrey Mashenkov, at first sight, I have not seen any problems with > > > .NET platform. > > > > > > I believe we need carefully configure ports in platform's examples, > > > additional actions should not be required. > > > > > > On Mon, Dec 17, 2018 at 2:35 PM Vyacheslav Daradur < > daradu...@gmail.com> > > > wrote: > > > > > > > > The task [1] is done. > > > > > > > > 'TcpDiscoveryVmIpFinder' is default IP finder in tests now. > > > > > > > > The IP finder is initialized on per tests class level to avoid hidden > > > > affecting among tests, it means the test methods in the common test > > > > class will use the same IP finder. > > > > > > > > You don't need to set up 'TcpDiscoveryVmIpFinder' in your tests > > > > explicitly anymore, also task [2] has been filled to remove related > > > > boilerplate if nobody minds. > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10555 > > > > [2] https://issues.apache.org/jira/browse/IGNITE-10715 > > > > > > > > > > > > On Wed, Dec 5, 2018 at 7:17 PM Dmitriy Pavlov > > > wrote: > > > > > > > > > > ++1 > > > > > > > > > > ср, 5 дек. 2018 г. в 18:38, Denis Mekhanikov < > dmekhani...@gmail.com>: > > > > > > > > > > > Andrey, > > > > > > > > > > > > Multi-JVM tests may also use a static IP finder, but it should > use > > > some > > > > > > specific port range instead of being shared. > > > > > > Something like 127.0.0.1:48500..48509 would do. > > > > > > > > > > > > Denis > > > > > > > > > > > > ср, 5 дек. 2018 г. в 18:34, Vyacheslav Daradur < > daradu...@gmail.com > > > >: > > > > > > > > > > > > > I filled a task [1]. > > > > > > > > > > > > > > >> Slava, do you think Platforms tests can be fixed as well or > one > > > more > > > > > > > ticket > > > > > > > should be created? > > > > > > > > > > > > > > I'll try to fix them within one ticket, it should be > investigated > > > a bit > > > > > > > deeper. > > > > > > > > > > > > > > I'll inform about the task's progress in this thread later. > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10555 > > > > > > > On Wed, Dec 5, 2018 at 6:28 PM Andrey Mashenkov > > > > > > > wrote: > > > > > > > > > > > > > > > > Slava, > > > > > > > > +1 for your proposal. > > > > > > > > Is there any ticket for this? > > > > > > > > > > > > > > > > Denis, > > > > > > > > I've just read in nabble thread you suggest to allow > multicast > > > finder > > > > > > for > > > > > > > > multiJVM tests > > > > > > > > and I'd think we shouldn't use multicast in test at all > (excepts > > > > > > > multicast > > > > > > > > Ip finder self tests of course), > > > > > > > > but e.g. add an assertion to force user to create ipfinder > > > properly. > > > > > > > > > > > > > > > > > > > > > > > > Also, we have a ticket for similar issue in 'examples' > module. > > > > > > > > Seems, there are some issues with Platforms module > integration. > > > > > > > > Slava, do you think Platforms tests can be fixed as well or > one > > > more > > > > > > > ticket > > > > > > > > should be created? > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6826 > > > > > > > > > > > > > > > > On Wed, Dec 5, 2018 at 5:55 PM Denis Mekhanikov < > > > dmekhani...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Slava, > > > > > > > > > > > > > > > > > > These are exactly my thoughts, so I fully support you here. > > > > > > > > > I already wrote about it: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/IP-finder-in-tests-td33322.html > > > > > > > > > But I kind of abandoned this activity. Feel free to take > over > > > it. > > > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > ср, 5 дек. 2018 г. в 17:22, Vladimir Ozerov < > > > voze...@gridgain.com>: > > > > > > > > > > > > > > > > > > > Huge +1 > > > > > > > > > > > > > > > > > > > > On Wed, Dec 5, 2018 at 5:09 PM Vyacheslav Daradur < > > > > > > >
[jira] [Created] (IGNITE-10805) [TC Bot] Gather info from investigations for page with muted tests
Ryabov Dmitrii created IGNITE-10805: --- Summary: [TC Bot] Gather info from investigations for page with muted tests Key: IGNITE-10805 URL: https://issues.apache.org/jira/browse/IGNITE-10805 Project: Ignite Issue Type: Task Reporter: Ryabov Dmitrii Some mutes have no ticket in the description [1], but its investigations [2] could have it. 1. We should periodically gather investigations like mutes and save them in a separate cache. 2. We should combine mutes and investigations before sending as response for mutes page. For this, we need to separate mutes to keep at most one test (e.g. send 7 mutes with 1 test in each instead of 1 mute with 7 tests). [1] https://ci.ignite.apache.org/app/rest/mutes/id:6626 [2] https://ci.ignite.apache.org/app/rest/investigations/test:(id:1284555489623943735),assignmentProject:(id:IgniteTests24Java8) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Set 'TcpDiscoveryVmIpFinder' as default IP finder for tests instead of 'TcpDiscoveryMulticastIpFinder'
The second step [2] "removing related boilerplate in tests" - has been done. It is expected that a bit memory will be released at tests TC agents, which could not be cleaned by GC because of static final fields. Guys, please, do not generate a new one ) [1] https://issues.apache.org/jira/browse/IGNITE-10715 On Tue, Dec 18, 2018 at 6:29 PM Anton Vinogradov wrote: > > Folks, > > Now I see 5-10% speedup at all TC suites right after the merge. > Great fix! > > > On Mon, Dec 17, 2018 at 2:39 PM Vyacheslav Daradur > wrote: > > > Andrey Mashenkov, at first sight, I have not seen any problems with > > .NET platform. > > > > I believe we need carefully configure ports in platform's examples, > > additional actions should not be required. > > > > On Mon, Dec 17, 2018 at 2:35 PM Vyacheslav Daradur > > wrote: > > > > > > The task [1] is done. > > > > > > 'TcpDiscoveryVmIpFinder' is default IP finder in tests now. > > > > > > The IP finder is initialized on per tests class level to avoid hidden > > > affecting among tests, it means the test methods in the common test > > > class will use the same IP finder. > > > > > > You don't need to set up 'TcpDiscoveryVmIpFinder' in your tests > > > explicitly anymore, also task [2] has been filled to remove related > > > boilerplate if nobody minds. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10555 > > > [2] https://issues.apache.org/jira/browse/IGNITE-10715 > > > > > > > > > On Wed, Dec 5, 2018 at 7:17 PM Dmitriy Pavlov > > wrote: > > > > > > > > ++1 > > > > > > > > ср, 5 дек. 2018 г. в 18:38, Denis Mekhanikov : > > > > > > > > > Andrey, > > > > > > > > > > Multi-JVM tests may also use a static IP finder, but it should use > > some > > > > > specific port range instead of being shared. > > > > > Something like 127.0.0.1:48500..48509 would do. > > > > > > > > > > Denis > > > > > > > > > > ср, 5 дек. 2018 г. в 18:34, Vyacheslav Daradur > >: > > > > > > > > > > > I filled a task [1]. > > > > > > > > > > > > >> Slava, do you think Platforms tests can be fixed as well or one > > more > > > > > > ticket > > > > > > should be created? > > > > > > > > > > > > I'll try to fix them within one ticket, it should be investigated > > a bit > > > > > > deeper. > > > > > > > > > > > > I'll inform about the task's progress in this thread later. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10555 > > > > > > On Wed, Dec 5, 2018 at 6:28 PM Andrey Mashenkov > > > > > > wrote: > > > > > > > > > > > > > > Slava, > > > > > > > +1 for your proposal. > > > > > > > Is there any ticket for this? > > > > > > > > > > > > > > Denis, > > > > > > > I've just read in nabble thread you suggest to allow multicast > > finder > > > > > for > > > > > > > multiJVM tests > > > > > > > and I'd think we shouldn't use multicast in test at all (excepts > > > > > > multicast > > > > > > > Ip finder self tests of course), > > > > > > > but e.g. add an assertion to force user to create ipfinder > > properly. > > > > > > > > > > > > > > > > > > > > > Also, we have a ticket for similar issue in 'examples' module. > > > > > > > Seems, there are some issues with Platforms module integration. > > > > > > > Slava, do you think Platforms tests can be fixed as well or one > > more > > > > > > ticket > > > > > > > should be created? > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6826 > > > > > > > > > > > > > > On Wed, Dec 5, 2018 at 5:55 PM Denis Mekhanikov < > > dmekhani...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Slava, > > > > > > > > > > > > > > > > These are exactly my thoughts, so I fully support you here. > > > > > > > > I already wrote about it: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/IP-finder-in-tests-td33322.html > > > > > > > > But I kind of abandoned this activity. Feel free to take over > > it. > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > ср, 5 дек. 2018 г. в 17:22, Vladimir Ozerov < > > voze...@gridgain.com>: > > > > > > > > > > > > > > > > > Huge +1 > > > > > > > > > > > > > > > > > > On Wed, Dec 5, 2018 at 5:09 PM Vyacheslav Daradur < > > > > > > daradu...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > I've found that the project's test framework uses > > > > > > > > > > 'TcpDiscoveryMulticastIpFinder' as default IP finder for > > tests > > > > > and > > > > > > > > > > there are a lot of tests written by Ignite's experts that > > > > > override > > > > > > it > > > > > > > > > > to 'TcpDiscoveryVmIpFinder'. > > > > > > > > > > > > > > > > > > > > Most of our tests starting Ignite nodes in the same JVM, > > that > > > > > > allows > > > > > > > > > > us using shared 'TcpDiscoveryVmIpFinder'. > > > > > > > > > > > > > > > > > > > > I think that using of 'TcpDiscoveryMulticastIpFinder' may > > be > > >
Re: Suggestion to improve deadlock detection
Ivan, We have to design configuration carefully. There are possible issues with compatibility and this API is public, it might be difficult to redesign it after while. Since there is a ticket about MVCC related configuration parts [1] (TxLog memory region/size, vacuum thread count and intervals, etc) I suggest solving the issue in scope of that ticket. At now, let's introduce a couple of system variables for an ability to change the behavior while development phase. [1] https://issues.apache.org/jira/browse/IGNITE-7966 Regards, Igor чт, 20 дек. 2018 г. в 16:29, Павлухин Иван : > Hi, > > I prepared a patch with deadlock detection algorithm implementation > (you can find it from ticket [1]). > > Now I would like to discuss options for configuring deadlock > detection. From a usability standpoint following need to be supported: > 1. Turn deadlock detection on/off (on by default). > 2. Configure a delay between starting waiting for a lock and sending > first deadlock detection message (e.g. 1 second by default). > 3. Something else? > > I can suggest following options: > 1. Use a single long for configuration. Values < 0 means disabled > deadlock detection. 0 means starting detection without a delay. Values > > 0 specifies a delay in milliseconds before starting detection. > 2. Use a boolean flag for turning detection on/off. Use a long value > for configuring delay. > > Also there are number of ways to pass this values to Ignite. I do no > have much experience in adding new configuration options to Ignite and > I need an advice here. For example, it could be done by adding > additional properties to TransactionConfiguration class. Another way > is using java system properties. > > Igor, Vladimir and others what do you think? > > [1] https://issues.apache.org/jira/browse/IGNITE-9322 > ср, 19 дек. 2018 г. в 15:51, Павлухин Иван : > > > > Igor, > > > > I see your points. And I agree to start with "lightweight > > implementation". Today we have 2PL and there is no activity on > > implementing rollback to savepoint. And if we do it in the future we > > will have to return to the subject of deadlock detection anyway. > > > > I will proceed with "forward-only" approach. > > вт, 18 дек. 2018 г. в 18:33, Seliverstov Igor : > > > > > > Ivan, > > > > > > I would prefer forward-only implementation even knowing it allows false > > > positive check results. > > > > > > Why I think so: > > > > > > 1) From my experience, when we have any future is waiting for reply, we > > > have to take a failover into consideration. > > > Usually failover implementations are way more complex than an initial > > > algorithm itself. > > > Forward-only version doesn't need any failover implementation as it > expects > > > failed probes (the probes didn't face a deadlock ). > > > > > > 2) Simple lightweight feature implementation is a really good point to > > > start - it may be extended if needed but really often such extending > > > doesn't need at all. > > > > > > 3) Any implementation allow false positive result - we are working with > > > distributed system, there are a bunch of processes happens at the same > time > > > and, > > > for example, concurrently happened rollback on timeout may solve a > deadlock > > > but probe is finished at this time, so- there is a rollback on deadlock > > > also as a result. > > > > > > 4) The described case (when false positive result is probable) isn't > usual > > > but, potentially, extremely rare, I don't think we should solve it > since it > > > doesn't break the grid. > > > > > > Regards, > > > Igor > > > > > > вт, 18 дек. 2018 г. в 17:57, Павлухин Иван : > > > > > > > Hi folks, > > > > > > > > During implementation of edge-chasing deadlock detection algorithm in > > > > scope of [1] it has been realized that basically we have 2 options > for > > > > "chasing" strategy. I will use terms Near when GridNearTxLocal is > > > > assumed and Dht when GridDhtTxLocal (tx which updates primary > > > > partition). So, here is 2 mentioned strategies: > > > > 1. Send initial probe when Dht tx blocks waiting for another tx to > > > > release a key. Initial probe is sent from waiting Dht tx to Near tx > > > > holding a lock. If receiving Near tx is blocked as well then it > relays > > > > the probe to Dht tx it awaits response from. So, the probe is > > > > traveling like Dht -> Near -> Dht -> Near -> ... Let's call the > > > > approach "forward-only". > > > > 2. Send probes only between Near transactions. This approach requires > > > > additional request-response call which Near tx issues to Dht node to > > > > check if tx sending a request is waiting for another tx. The call > > > > returns identifier of a Near tx blocking tx sending a request (if > > > > sending tx is in fact blocked). Then waiting Near tx relays a probe > to > > > > blocked Near tx. Let's call that approach "lock-checking". > > > > > > > > I would like to define several points to consider while comparing > > > > approaches (please
[jira] [Created] (IGNITE-10804) [ML] Add ability to load LinReg model from Spark to Ignite via PMML
Aleksey Zinoviev created IGNITE-10804: - Summary: [ML] Add ability to load LinReg model from Spark to Ignite via PMML Key: IGNITE-10804 URL: https://issues.apache.org/jira/browse/IGNITE-10804 Project: Ignite Issue Type: Sub-task Components: ml Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev 1) Write simple ML pipeline for Spark 2) Convert to PMML model 3) Load to Ignite 4) Predict on Ignite -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10803) [ML] Add prototype LinearRegression loading from PMML format
Aleksey Zinoviev created IGNITE-10803: - Summary: [ML] Add prototype LinearRegression loading from PMML format Key: IGNITE-10803 URL: https://issues.apache.org/jira/browse/IGNITE-10803 Project: Ignite Issue Type: Sub-task Components: ml Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Generate or get existing PMML model for known dataset to load and predict new data in Ignite -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10802) Documentation for option IGNITE_DISCOVERY_DISABLE_CACHE_METRICS_UPDATE
Alex Volkov created IGNITE-10802: Summary: Documentation for option IGNITE_DISCOVERY_DISABLE_CACHE_METRICS_UPDATE Key: IGNITE-10802 URL: https://issues.apache.org/jira/browse/IGNITE-10802 Project: Ignite Issue Type: Task Components: documentation Affects Versions: 2.8 Reporter: Alex Volkov After implementation -IGNITE-10172- it's not clear how cluster should behave in some cases. For example, if I have cluster with option IGNITE_DISCOVERY_DISABLE_CACHE_METRICS_UPDATE=True and one additional node with flag IGNITE_DISCOVERY_DISABLE_CACHE_METRICS_UPDATE=False comes to this cluster. What behaviour should I expect from the cluster? In general, please provide documentation for this option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10801) Update H2 verstion to 1.4.198
Sergey Antonov created IGNITE-10801: --- Summary: Update H2 verstion to 1.4.198 Key: IGNITE-10801 URL: https://issues.apache.org/jira/browse/IGNITE-10801 Project: Ignite Issue Type: Improvement Components: sql Affects Versions: 2.7 Reporter: Sergey Antonov Fix For: 2.8 After h2 1.4.198 release we should update h2 version in AI, because of important bugs will be fixed there. For example https://github.com/h2database/h2database/issues/1057 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10800) Add wal_mode parameter to properties file
Ilya Suntsov created IGNITE-10800: - Summary: Add wal_mode parameter to properties file Key: IGNITE-10800 URL: https://issues.apache.org/jira/browse/IGNITE-10800 Project: Ignite Issue Type: Bug Components: yardstick Affects Versions: 2.7 Reporter: Ilya Suntsov As I understand we can enable persistence with properties file parameter. I guess we need to add a parameter for WAL mode. Expected behavior: * When we have in configuration region with persistence should be used this configuration and wal_mode value from properties should be ignored. Also should be added warnings in all nodes logs with configuration details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10799) Optimize affinity initialization/re-calculation
Pavel Kovalenko created IGNITE-10799:
Summary: Optimize affinity initialization/re-calculation
Key: IGNITE-10799
URL: https://issues.apache.org/jira/browse/IGNITE-10799
Project: Ignite
Issue Type: Improvement
Components: cache
Affects Versions: 2.1
Reporter: Pavel Kovalenko
Assignee: Pavel Kovalenko
Fix For: 2.8
In case of persistence enabled and a baseline is set we have 2 main approaches
to recalculate affinity:
{noformat}
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager#onServerJoinWithExchangeMergeProtocol
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager#onServerLeftWithExchangeMergeProtocol
{noformat}
Both of them following the same approach of recalculating:
1) Take a current baseline (ideal assignment).
2) Filter out offline nodes from it.
3) Choose new primary nodes if previous went away.
4) Place temporal primary nodes to late affinity assignment set.
Looking at implementation details we may notice that we do a lot of unnecessary
online nodes cache lookups and array list copies. The performance becomes too
slow if we do recalculate affinity for replicated caches (It takes P * N on
each node, where P - partitions count, N - the number of nodes in the cluster).
In case of large partitions count or large cluster, it may take few seconds,
which is unacceptable, because this process happens during PME and freezes
ongoing cluster operations.
We should investigate possible bottlenecks and improve the performance of
affinity recalculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-10798) Data page scan for ScanQuery, SqlQuery and SqlFieldsQuery
Sergi Vladykin created IGNITE-10798: --- Summary: Data page scan for ScanQuery, SqlQuery and SqlFieldsQuery Key: IGNITE-10798 URL: https://issues.apache.org/jira/browse/IGNITE-10798 Project: Ignite Issue Type: Improvement Reporter: Sergi Vladykin Assignee: Sergi Vladykin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
