[jira] [Created] (IGNITE-12623) ClassCastException on Thin client when get cache value with List and Map
HAOFENG DENG created IGNITE-12623:
-
Summary: ClassCastException on Thin client when get cache value
with List and Map
Key: IGNITE-12623
URL: https://issues.apache.org/jira/browse/IGNITE-12623
Project: Ignite
Issue Type: Bug
Components: cache, clients
Affects Versions: 2.7.6
Environment: Ubuntu
Java 11
Ignite 2.7.6
Reporter: HAOFENG DENG
I use the thin java client
I Create a ArrayList and put the Person object into it.
{code:java}
try (IgniteClient igniteClient = Ignition.startClient(new
ClientConfiguration().setAddresses("127.0.0.1:10800"))) {
final ClientCache> demo =
igniteClient.getOrCreateCache("demo");
final List personList = new java.util.ArrayList<>();
personList.add(new Person("apache", 100));
personList.add(new Person("Ignite", 13));
demo.put("test", personList);
}
{code}
And when I take it out of the list
{code:java}
try (IgniteClient igniteClient = Ignition.startClient(new
ClientConfiguration().setAddresses("127.0.0.1:10800"))) {
final ClientCache> demo =
igniteClient.getOrCreateCache("demo");
final List personList = demo.get("test");
final Person person = personList.get(0);
}{code}
I get the following exceptions:
{code:java}
java.lang.ClassCastException: class
org.apache.ignite.internal.binary.BinaryObjectImpl cannot be cast to class
com.peeandgee.Person (org.apache.ignite.internal.binary.BinaryObjectImpl and
com.peeandgee.Person are in unnamed module of loader 'app')
{code}
However, there is no exception if I modify the code as follows:
{code:java}
BinaryObject bo = (BinaryObject) map.get(0);
Person person = bo.deserialize();
{code}
Same issue in Map [ClassCastException on thinClient in Apache
Ignite|[https://stackoverflow.com/questions/59299316/classcastexception-on-thinclient-in-apache-ignite]]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Nick
Can we retain only single cancel(long queryId) method in QueryMXBean ?
I think query type can be determined by queryId.
I also think we need similar "cancellation" API without the need to use mx
beans.
вт, 4 февр. 2020 г. в 18:07, Nikolay Izhikov :
> Hello, Igniters.
>
> I propose to create management API to dance user provided tasks and
> queries.
> Below my proposal in the IEP [1] form.
>
> Please, share your feedback.
>
>
> Motivation:
>
> Ignite provides many API to deploy and execute user-provided code on
> the server nodes
> inside the sam JVM as Ignite process runs.
>
> We also has a many APIs that allocate many resources on the server
> nodes:
>
> In case of some buggy code that consumes many system resources(CPU,
> RAM, flood network) or heavy query
> the whole cluster can becomes instable.
> We should provide to the cluster administator the ability to stop any
> user deployed task.
>
> Description:
> JMX beans to cancel listed tasks should be introduced:
> * Compute task
> * Service
> * Continuous query
> * Transactions
> * Queries(SQL, Scan, Text)
>
> In the scope of IEP-35 System view API introduced.
> New API should use the same identifier that is used in corresponding
> System View.
>
> Public API changes:
> * Cancel of Scan(text) queries should work similar to the SQL.
> * Cancel of compute task should invoke
> `ComputeTaskInternalFuture#cancel` for the corresponding task.
> * Cancel of service should be the same as `IgniteServices#cancel`
> * Cancel of transaction should works the same as in
> `TransactionMXBean#getActiveTransaction(kill=true)`.
> * Cancel of a continuous query should work the same as
> `QueryCursor#close` execution.
>
> ```
> QueryMXBean {
> boolean cancelSQL(Long queryId); //Already implemented in
> IGNITE-4436.
>
> boolean cancelScan(Long queryId);
>
> boolean cancelText(Long queryId);
> }
>
> ComputeMXBean {
> boolean cancel(IgniteUuid id);
> }
>
> ServiceMXBean {
> boolean cancel(IgniteUuid id);
> }
>
> TransactionMXBean {
> boolean cancel(IgniteUuid xid);
> }
>
> interface ContinuousQueryMXBean {
> boolean cancel(UUID id);
> }
> ```
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=145724615
>
>
>
> > 17 янв. 2020 г., в 22:43, Николай Ижиков
> написал(а):
> >
> > Hello, Andrey.
> >
> > Thanks for positive feedback.
> > Appreciate it.
> >
> >> we can't cancel service or service's method
> >
> > I understand it.
> > Seems, we have several obvious options here:
> >
> > * Try to do it and if fail answer to the user: «I’ve tried and fail.
> Sorry»
> > * Kill hanging thread.
> >
> >> This invocations not register anywhere and can't be tracked and killed.
> >
> > Yes.
> > So we should invent tracking for those invocation to be able to kill it
> :)
> >
> >
> >> 17 янв. 2020 г., в 21:03, Andrey Gura написал(а):
> >>
> >> It would be grate.
> >>
> >> Only one comment: we can't cancel service or service's method
> >> execution because service grid API doesn't imply any requirement to
> >> service implementation. So if user do something like infinite cycle or
> >> blocking but non-interruptible operation it's impossible to interrupt
> >> it. Also service method invocation itself isn't cluster wide or local
> >> node specific operation. This invocations not register anywhere and
> >> can't be tracked and killed.
> >>
> >> Unfortunately, the same is valid for some jobs in sense of infinite or
> >> non-interruptible operations.
> >>
> >> On Wed, Jan 15, 2020 at 9:30 PM Николай Ижиков
> wrote:
> >>>
> >>> Hello, Igniters.
> >>>
> >>> As you may know, we put a lot of effort to improve Ignite metric and
> diagnostic API.
> >>> We have created the following API:
> >>> * Metric manager
> >>> * System view manager
> >>> As far as I know, we would have tracing capabilities soon.
> >>>
> >>> I think it's time to take the next step.
> >>> We should provide to the administrator the API to cancel(stop, kill)
> arbitrary user started computation.
> >>>
> >>> Right now we have API to do it for:
> >>> * transactions `TransactionsMXBean#getActiveTransactions`.
> >>> * SQL queries: `KILL QUERY` sql command and visor task -
> `VisorQueryCancelTask`
> >>>
> >>> Please, note, these features are implemented via different API.
> >>>
> >>> I think we should introduce uniform Cancel API for the following
> computations:
> >>>
> >>> * service.
> >>> * specific service method execution.
> >>> * compute job(compute task).
> >>> * query(scan, continuous, text).
> >>>
> >>> We should also rework kill transaction and kill SQL queries API to
> make them similar to each other.
> >>> I have plans to write an IEP for it and implement it.
> >>> What do you think?
> >>>
> >
>
>
--
Best regards,
Alexei Scherbakov
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Hello all. Denis, which changes exactly? In current implementation of ticket [2] flag [1] is checked before requiring --force flag and showing any warnings. Do we need to set reuse-memory-on-deactivate to true by default? [1] org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE [2] https://issues.apache.org/jira/browse/IGNITE-12614 вт, 4 февр. 2020 г. в 22:45, Denis Magda : > That's the best solution for this scenario. Should we readjust the already > created ticket [1] suggesting to implement the changes of Alex Scherbakov > instead? > > [1] https://issues.apache.org/jira/browse/IGNITE-12614 > > - > Denis > > > On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov < > alexey.scherbak...@gmail.com> wrote: > > > Folks, > > > > For a long time we have a flag [1] > > > > It does almost what we want here. > > > > I suggest to make this behavior default and rework it to keep data in > > memory as well (we already have special "recovery" mode for caches). > > > > [1] > > > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > > > > > > > пн, 3 февр. 2020 г. в 18:47, Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > > > I do not mind making this change if we explicitly and clearly define > what > > > 'new inactive state' means. What should happen if a partition is lost > in > > > inactive state? What if such node joins back the cluster after? Etc. > > > > > > пт, 31 янв. 2020 г. в 20:57, Denis Magda : > > > > > > > Back up Ivan's opinion here. Initially, the activation/deactivation > was > > > > created for the baseline topology designed for cases with native > > > > persistence. My thinking was that the mechanism itended to prevent > data > > > > inconsistencies while nodes with data on the disk will be in the > > process > > > of > > > > joining the cluster. > > > > > > > > Artem, could you please update the docs bringing this to the > attention > > of > > > > the user community? > > > > https://issues.apache.org/jira/browse/IGNITE-12615 > > > > > > > > AG, what if we don't purge data from memory at least for the caches > not > > > > backed by the native persistence? Is this a big deal? We can > certainly > > > put > > > > this off by my guts feel we'll return to this question sooner or > later. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Fri, Jan 31, 2020 at 2:17 AM Ivan Pavlukhin > > > > wrote: > > > > > > > > > For me it looks like some coincidence effect. I understand that we > > get > > > > > such behavior because deactivation works the same way as for > > > > > persistent caches. Was cluster activation/deactivation designed and > > > > > described for in-memory caches? Current behavior sounds for me a as > > > > > big risk. I expect a lot of upset users unexpectedly purged all > their > > > > > data. > > > > > > > > > > пт, 31 янв. 2020 г. в 00:00, Alexey Goncharuk < > > > > alexey.goncha...@gmail.com > > > > > >: > > > > > > > > > > > > Because originally the sole purpose of deactivation was resource > > > > > > deallocation. > > > > > > > > > > > > чт, 30 янв. 2020 г. в 22:13, Denis Magda : > > > > > > > > > > > > > Such a revelation for me that data is purged from RAM if > someone > > > > > > > deactivates the cluster. Alex, do you remember why we decided > to > > > > > implement > > > > > > > it this way initially? > > > > > > > > > > > > > > - > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 30, 2020 at 2:09 AM Alexey Goncharuk < > > > > > > > alexey.goncha...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > I agree on CLI and JMX because those interfaces can be used > by > > a > > > > > system > > > > > > > > administrator and can be invoked by mistake. > > > > > > > > > > > > > > > > As for the Java API, personally, I find it strange to add > > 'force' > > > > or > > > > > > > > 'confirm' flags to it because it is very unlikely that such > an > > > > > invocation > > > > > > > > is done by mistake. Such mistakes are caught during the > testing > > > > > phase and > > > > > > > > developers will end up hard-coding 'true' as a flag anyways. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best regards, > > > > > Ivan Pavlukhin > > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > Alexei Scherbakov > > >
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Folks, Shouldn't we unify identifiers for all ongoing tasks in Ignite and move cancel to a single method in a single bean? Right now different types of IDs (long, UUID, IgniteUuid) look confusing and messy. I understand that such API is enforced by the IDs implementation in the corresponding Ignite subsystems. However, given that lately we developed a unified approach to metrics and system views with Nikolay's help, shouldn't we take one more step forward and unify this part as well?
Re: Internal classes are exposed in public API
Denis, I just realized that this vote would not be a regular Apache vote we used to run. Usually we vote for a change to be accepted or rejected. Here, on the other hand, we need to choose between two alternatives, and I am not sure yet how this should be formally handled. Dmitriy, perhaps, you can shed some light on this? Meanwhile, I'll brush up on the voting process as well.
Re: Internal classes are exposed in public API
Sorry for the rush, I think I got it - it's right in the voting process [1]. This would be a vote on a procedural issue of how to handle code deprecation. I'll start the discussion soon, let's agree on the alternatives and then start the vote. [1] https://www.apache.org/foundation/voting.html
Re: Forbid mixed cache groups with both atomic and transactional caches
Folks, A bit of offtop. Do we have some recommendations in the community how long should we wait until treating something as "a Community conclusion"? It worries me a little bit that I see a discussion for a first time and there is already a conclusion. And the discussion was started lesser than 24 hours ago. I suppose we should allow everyone interested to share an opinion (here I agree with the proposal) and it usually requires some time in open-source communities. ср, 5 февр. 2020 г. в 10:58, Ivan Rakov : > > Folks, > > Thanks for your feedback. > I've created a JIRA issue on this change: > https://issues.apache.org/jira/browse/IGNITE-12622 > > On Tue, Feb 4, 2020 at 10:43 PM Denis Magda wrote: > > > +1 from my end. It doesn't sound like a big deal if Ignite users need to > > define separate groups for atomic and transactional caches. > > > > - > > Denis > > > > > > On Tue, Feb 4, 2020 at 3:28 AM Ivan Rakov wrote: > > > > > Igniters, > > > > > > Apparently it's possible in Ignite to configure a cache group with both > > > ATOMIC and TRANSACTIONAL caches. > > > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. > > > In my opinion, it would be better to remove such possibility from the > > > product. There are several reasons: > > > > > > 1) The original idea of grouping caches was optimizing storage overhead > > and > > > PME time by joining data of similar caches into the same partitions. > > ATOMIC > > > and TRANSACTIONAL caches provide different guarantees and are designed > > for > > > different use cases, thus they can hardly be called "similar". > > > > > > 2) Diving deeper: synchronization protocols and possible reasons for > > > primary-backup divergences are conceptually different for ATOMIC and > > > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery > > protocol > > > allows to recover consistency if any participating node will fail, but > > for > > > ATOMIC caches there's possible scenario with failure of primary node > > where > > > neither of backups will contain the most recent state of the data. > > Example: > > > one backup have received updates 1, 3, 5 while another have received 2, 4 > > > (which is possible due to message reordering), and even tracking counters > > > [1] won't restore the consistency. The problem is that we can't > > distinguish > > > what kind of conflict we have faced in case update counters have diverged > > > in a mixed group. > > > > > > 3) Mixed groups are poorly tested. I can't find any tests except a couple > > > of smoke tests in IgniteCacheGroupsTest. We can't be sure that different > > > synchronization protocols will work correctly for such configurations, > > > especially under load and with a variety of dependent configuration > > > parameters. > > > > > > 4) I have never heard of any feedback on mixed groups. I have asked > > > different people on this and no one recalled any attempts to configure > > such > > > groups. I believe that in fact no one has ever tried to do it. > > > > > > Please let me know if you are aware of any cases where mixed groups are > > > used or reasons to keep them. Otherwise I'll create a ticket to prohibit > > > mixed configurations. > > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > > > > > -- > > > Best Regards, > > > Ivan Rakov > > > > > -- Best regards, Ivan Pavlukhin
Re: Contribution
Andrey,
> > Andrey, then I suggest firstly check if old annotations are present,
and if
> > not - are there any new annotations. Do you agree?
> I'm afraid I don't understand. Old annotations are present and new
> annotations don't exist yet :)
I talked about implementation of methods *getDescription* /
*getParameterName* in *IgniteStandardMXBean *class. To support backward
compatibility I need to check if there are old annotations above method, or
new one before parameters. So, my question was the order of searching
annotations (first - old, after - new).
вт, 4 февр. 2020 г. в 21:10, Andrey Gura :
> > Andrey, then I suggest firstly check if old annotations are present, and
> if
> > not - are there any new annotations. Do you agree?
>
> I'm afraid I don't understand. Old annotations are present and new
> annotations don't exist yet :)
>
> > And should I also replace usage of old annotations or leave it as it is?
>
> It would be great to replace old annotations by new ones.
>
> On Tue, Feb 4, 2020 at 6:09 PM Лев Киселев
> wrote:
> >
> > Andrey, then I suggest firstly check if old annotations are present, and
> if
> > not - are there any new annotations. Do you agree?
> > And should I also replace usage of old annotations or leave it as it is?
> >
> > вт, 4 февр. 2020 г. в 16:57, Andrey Gura :
> >
> > > Lev,
> > >
> > > > I'm confused about your first comment. Does it mean, that the only
> thing
> > > I
> > > can do is to create new annotation?
> > >
> > > Yes, it does. But in this case you can mark old annotations as
> > > deprecated because new annotations are real alternative.
> > >
> > > > So, I can't change implementation of *getDescription /
> getParameterName
> > > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods
> from*
> > > > IgniteStandardMXBean* class,
> > > > and replace current annotations with new one.
> > >
> > > You have to support both old and new annotations due to a backward
> > > compatibility.
> > >
> > > On Tue, Feb 4, 2020 at 3:14 PM Лев Киселев >
> > > wrote:
> > > >
> > > > Thanks for answer, Andrey.
> > > > I'm confused about your first comment. Does it mean, that the only
> thing
> > > I
> > > > can do is to create new annotation?
> > > > Because, as you said, I must leave unchanged all related to old
> > > annotations
> > > > functionality.
> > > > So, I can't change implementation of *getDescription /
> getParameterName
> > > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods
> from*
> > > > IgniteStandardMXBean* class,
> > > > and replace current annotations with new one.
> > > >
> > > > вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
> > > >
> > > > > Hi, Lev!
> > > > >
> > > > > I think you understand task correctly.
> > > > >
> > > > > A couple of comments:
> > > > >
> > > > > - Existing annotations and related functionality must not be
> deleted
> > > > > because it is part of public API and we committed to have stable
> API
> > > > > between major releases.
> > > > > - I would suggest introduce @MXBeanParameter annotation instead of
> > > > > @MXBeanParameterInformation. Just shorten name.
> > > > >
> > > > > On Thu, Jan 30, 2020 at 8:53 PM Denis Magda
> wrote:
> > > > > >
> > > > > > Nikolay, Andrey,
> > > > > >
> > > > > > Would you be the best committers to help out here? You are
> already
> > > > > deeply involved in metrics development and can quickly suggest Lev
> how
> > > to
> > > > > proceed with this task.
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев <
> > > lev.kiselev.1...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Hello everyone, I have question about following task:
> > > > > >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> > > > > >> Solution proposed in task description is seem to be logical.
> > > > > >> So, I need to every replace @MXBeanParametersNames and
> > > > > >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> > > > > something
> > > > > >> like:
> > > > > >> void methodName(@MXBeanParameterInformation(name = "name",
> > > description =
> > > > > >> "description") firstParameter, ...) {}.
> > > > > >> And, of course, need to change processing logic at
> > > > > >> getParameterName/getDescription methods from
> IgniteStandardMXBean.
> > > > > >> Do I understand correctly what needs to be done?
> > > > >
> > >
>
Re: Forbid mixed cache groups with both atomic and transactional caches
Ivan. It seems we don’t have a format definition for a «community decision» We, for sure, should fill this gap. Me and Andrey Gura, have certain proposals for our development process based on metrics API discussion. We will share those proposals after 2.8 release and some additional discussion. > 5 февр. 2020 г., в 13:11, Ivan Pavlukhin написал(а): > > Folks, > > A bit of offtop. Do we have some recommendations in the community how > long should we wait until treating something as "a Community > conclusion"? It worries me a little bit that I see a discussion for a > first time and there is already a conclusion. And the discussion was > started lesser than 24 hours ago. I suppose we should allow everyone > interested to share an opinion (here I agree with the proposal) and it > usually requires some time in open-source communities. > > ср, 5 февр. 2020 г. в 10:58, Ivan Rakov : >> >> Folks, >> >> Thanks for your feedback. >> I've created a JIRA issue on this change: >> https://issues.apache.org/jira/browse/IGNITE-12622 >> >> On Tue, Feb 4, 2020 at 10:43 PM Denis Magda wrote: >> >>> +1 from my end. It doesn't sound like a big deal if Ignite users need to >>> define separate groups for atomic and transactional caches. >>> >>> - >>> Denis >>> >>> >>> On Tue, Feb 4, 2020 at 3:28 AM Ivan Rakov wrote: >>> Igniters, Apparently it's possible in Ignite to configure a cache group with both ATOMIC and TRANSACTIONAL caches. Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. In my opinion, it would be better to remove such possibility from the product. There are several reasons: 1) The original idea of grouping caches was optimizing storage overhead >>> and PME time by joining data of similar caches into the same partitions. >>> ATOMIC and TRANSACTIONAL caches provide different guarantees and are designed >>> for different use cases, thus they can hardly be called "similar". 2) Diving deeper: synchronization protocols and possible reasons for primary-backup divergences are conceptually different for ATOMIC and TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery >>> protocol allows to recover consistency if any participating node will fail, but >>> for ATOMIC caches there's possible scenario with failure of primary node >>> where neither of backups will contain the most recent state of the data. >>> Example: one backup have received updates 1, 3, 5 while another have received 2, 4 (which is possible due to message reordering), and even tracking counters [1] won't restore the consistency. The problem is that we can't >>> distinguish what kind of conflict we have faced in case update counters have diverged in a mixed group. 3) Mixed groups are poorly tested. I can't find any tests except a couple of smoke tests in IgniteCacheGroupsTest. We can't be sure that different synchronization protocols will work correctly for such configurations, especially under load and with a variety of dependent configuration parameters. 4) I have never heard of any feedback on mixed groups. I have asked different people on this and no one recalled any attempts to configure >>> such groups. I believe that in fact no one has ever tried to do it. Please let me know if you are aware of any cases where mixed groups are used or reasons to keep them. Otherwise I'll create a ticket to prohibit mixed configurations. [1]: https://issues.apache.org/jira/browse/IGNITE-11797 -- Best Regards, Ivan Rakov >>> > > > > -- > Best regards, > Ivan Pavlukhin
Re: Contribution
Different approaches are possible. For me it would be logical if
MXBeanParameter override MXBeanParametersNames and
MXBeanParametersDescriptions.
In such case the order of annotation processing is fixed: old on
method then new on method's parameters if exists.
On Wed, Feb 5, 2020 at 1:11 PM Лев Киселев wrote:
>
> Andrey,
> > > Andrey, then I suggest firstly check if old annotations are present,
> and if
> > > not - are there any new annotations. Do you agree?
>
> > I'm afraid I don't understand. Old annotations are present and new
> > annotations don't exist yet :)
>
>
> I talked about implementation of methods *getDescription* /
> *getParameterName* in *IgniteStandardMXBean *class. To support backward
> compatibility I need to check if there are old annotations above method, or
> new one before parameters. So, my question was the order of searching
> annotations (first - old, after - new).
>
> вт, 4 февр. 2020 г. в 21:10, Andrey Gura :
>
> > > Andrey, then I suggest firstly check if old annotations are present, and
> > if
> > > not - are there any new annotations. Do you agree?
> >
> > I'm afraid I don't understand. Old annotations are present and new
> > annotations don't exist yet :)
> >
> > > And should I also replace usage of old annotations or leave it as it is?
> >
> > It would be great to replace old annotations by new ones.
> >
> > On Tue, Feb 4, 2020 at 6:09 PM Лев Киселев
> > wrote:
> > >
> > > Andrey, then I suggest firstly check if old annotations are present, and
> > if
> > > not - are there any new annotations. Do you agree?
> > > And should I also replace usage of old annotations or leave it as it is?
> > >
> > > вт, 4 февр. 2020 г. в 16:57, Andrey Gura :
> > >
> > > > Lev,
> > > >
> > > > > I'm confused about your first comment. Does it mean, that the only
> > thing
> > > > I
> > > > can do is to create new annotation?
> > > >
> > > > Yes, it does. But in this case you can mark old annotations as
> > > > deprecated because new annotations are real alternative.
> > > >
> > > > > So, I can't change implementation of *getDescription /
> > getParameterName
> > > > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods
> > from*
> > > > > IgniteStandardMXBean* class,
> > > > > and replace current annotations with new one.
> > > >
> > > > You have to support both old and new annotations due to a backward
> > > > compatibility.
> > > >
> > > > On Tue, Feb 4, 2020 at 3:14 PM Лев Киселев > >
> > > > wrote:
> > > > >
> > > > > Thanks for answer, Andrey.
> > > > > I'm confused about your first comment. Does it mean, that the only
> > thing
> > > > I
> > > > > can do is to create new annotation?
> > > > > Because, as you said, I must leave unchanged all related to old
> > > > annotations
> > > > > functionality.
> > > > > So, I can't change implementation of *getDescription /
> > getParameterName
> > > > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods
> > from*
> > > > > IgniteStandardMXBean* class,
> > > > > and replace current annotations with new one.
> > > > >
> > > > > вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
> > > > >
> > > > > > Hi, Lev!
> > > > > >
> > > > > > I think you understand task correctly.
> > > > > >
> > > > > > A couple of comments:
> > > > > >
> > > > > > - Existing annotations and related functionality must not be
> > deleted
> > > > > > because it is part of public API and we committed to have stable
> > API
> > > > > > between major releases.
> > > > > > - I would suggest introduce @MXBeanParameter annotation instead of
> > > > > > @MXBeanParameterInformation. Just shorten name.
> > > > > >
> > > > > > On Thu, Jan 30, 2020 at 8:53 PM Denis Magda
> > wrote:
> > > > > > >
> > > > > > > Nikolay, Andrey,
> > > > > > >
> > > > > > > Would you be the best committers to help out here? You are
> > already
> > > > > > deeply involved in metrics development and can quickly suggest Lev
> > how
> > > > to
> > > > > > proceed with this task.
> > > > > > >
> > > > > > > -
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев <
> > > > lev.kiselev.1...@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Hello everyone, I have question about following task:
> > > > > > >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> > > > > > >> Solution proposed in task description is seem to be logical.
> > > > > > >> So, I need to every replace @MXBeanParametersNames and
> > > > > > >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> > > > > > something
> > > > > > >> like:
> > > > > > >> void methodName(@MXBeanParameterInformation(name = "name",
> > > > description =
> > > > > > >> "description") firstParameter, ...) {}.
> > > > > > >> And, of course, need to change processing logic at
> > > > > > >> getParameterName/getDescription methods from
> > IgniteStandardMXBean.
> > > > > > >> Do I understand correctly what needs to be done?
> > > > > >
> > > >
> >
Re: Forbid mixed cache groups with both atomic and transactional caches
Ivan, Thanks for pointing this out. Less than one day is indeed too early to treat this discussion thread as a "community conclusion". Still, the consensus among the current participants made me feel that a conclusion will be reached. We'll surely get back to the discussion if opposite opinions will arise. On Wed, Feb 5, 2020 at 1:11 PM Ivan Pavlukhin wrote: > Folks, > > A bit of offtop. Do we have some recommendations in the community how > long should we wait until treating something as "a Community > conclusion"? It worries me a little bit that I see a discussion for a > first time and there is already a conclusion. And the discussion was > started lesser than 24 hours ago. I suppose we should allow everyone > interested to share an opinion (here I agree with the proposal) and it > usually requires some time in open-source communities. > > ср, 5 февр. 2020 г. в 10:58, Ivan Rakov : > > > > Folks, > > > > Thanks for your feedback. > > I've created a JIRA issue on this change: > > https://issues.apache.org/jira/browse/IGNITE-12622 > > > > On Tue, Feb 4, 2020 at 10:43 PM Denis Magda wrote: > > > > > +1 from my end. It doesn't sound like a big deal if Ignite users need > to > > > define separate groups for atomic and transactional caches. > > > > > > - > > > Denis > > > > > > > > > On Tue, Feb 4, 2020 at 3:28 AM Ivan Rakov > wrote: > > > > > > > Igniters, > > > > > > > > Apparently it's possible in Ignite to configure a cache group with > both > > > > ATOMIC and TRANSACTIONAL caches. > > > > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* > tests. > > > > In my opinion, it would be better to remove such possibility from the > > > > product. There are several reasons: > > > > > > > > 1) The original idea of grouping caches was optimizing storage > overhead > > > and > > > > PME time by joining data of similar caches into the same partitions. > > > ATOMIC > > > > and TRANSACTIONAL caches provide different guarantees and are > designed > > > for > > > > different use cases, thus they can hardly be called "similar". > > > > > > > > 2) Diving deeper: synchronization protocols and possible reasons for > > > > primary-backup divergences are conceptually different for ATOMIC and > > > > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery > > > protocol > > > > allows to recover consistency if any participating node will fail, > but > > > for > > > > ATOMIC caches there's possible scenario with failure of primary node > > > where > > > > neither of backups will contain the most recent state of the data. > > > Example: > > > > one backup have received updates 1, 3, 5 while another have received > 2, 4 > > > > (which is possible due to message reordering), and even tracking > counters > > > > [1] won't restore the consistency. The problem is that we can't > > > distinguish > > > > what kind of conflict we have faced in case update counters have > diverged > > > > in a mixed group. > > > > > > > > 3) Mixed groups are poorly tested. I can't find any tests except a > couple > > > > of smoke tests in IgniteCacheGroupsTest. We can't be sure that > different > > > > synchronization protocols will work correctly for such > configurations, > > > > especially under load and with a variety of dependent configuration > > > > parameters. > > > > > > > > 4) I have never heard of any feedback on mixed groups. I have asked > > > > different people on this and no one recalled any attempts to > configure > > > such > > > > groups. I believe that in fact no one has ever tried to do it. > > > > > > > > Please let me know if you are aware of any cases where mixed groups > are > > > > used or reasons to keep them. Otherwise I'll create a ticket to > prohibit > > > > mixed configurations. > > > > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > > > > > > > -- > > > > Best Regards, > > > > Ivan Rakov > > > > > > > > > > > -- > Best regards, > Ivan Pavlukhin >
[jira] [Created] (IGNITE-12624) Java thin client: Wrong typeId generation for system types
Aleksey Plekhanov created IGNITE-12624:
--
Summary: Java thin client: Wrong typeId generation for system types
Key: IGNITE-12624
URL: https://issues.apache.org/jira/browse/IGNITE-12624
Project: Ignite
Issue Type: Bug
Components: binary, thin client
Reporter: Aleksey Plekhanov
Assignee: Aleksey Plekhanov
Thin client generates wrong {{typeId}} for system types. This is caused by
{{ClientMarshallerContext}} implementation, which always returns {{false}} for
{{isSystemType}} method. These leads to {{typeId}} duplication for the same
class and assertions when trying to get object by thick client. Reproducer:
{code:java}
thinClient.cache(DEFAULT_CACHE_NAME).put(1, CacheAtomicityMode.ATOMIC);
thickClient.cache(DEFAULT_CACHE_NAME).get(1);
{code}
Assertion:
{noformat}
java.lang.AssertionError: Duplicate typeId [typeId=1984564222, cls=class
org.apache.ignite.cache.CacheAtomicityMode, desc=BinaryClassDescriptor
[cls=class org.apache.ignite.cache.CacheAtomicityMode, serializer=null,
initialSerializer=null, mapper=BinaryInternalMapper
[nameMapper=BinaryBaseNameMapper [isSimpleName=true],
idMapper=BinaryBaseIdMapper [isLowerCase=true], checkOnZeroId=false],
mode=OPTIMIZED, userType=false, typeId=329149470,
typeName=org.apache.ignite.cache.CacheAtomicityMode, affKeyFieldName=null,
ctor=null, writeReplacer=null, readResolveMtd=null, stableSchema=null,
schemaReg=org.apache.ignite.internal.binary.BinarySchemaRegistry@167f45f8,
registered=true, useOptMarshaller=true, excluded=false,
stableSchemaPublished=false]]
at
org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:775)
at
org.apache.ignite.internal.binary.BinaryUtils.resolveClass(BinaryUtils.java:1669)
at
org.apache.ignite.internal.binary.BinaryEnumObjectImpl.deserialize(BinaryEnumObjectImpl.java:178)
at
org.apache.ignite.internal.binary.BinaryEnumObjectImpl.value(BinaryEnumObjectImpl.java:284)
at
org.apache.ignite.internal.processors.cache.CacheObjectUtils.unwrapBinary(CacheObjectUtils.java:176)
at
org.apache.ignite.internal.processors.cache.CacheObjectUtils.unwrapBinaryIfNeeded(CacheObjectUtils.java:67)
at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:136)
at
org.apache.ignite.internal.processors.cache.GridCacheContext.unwrapBinaryIfNeeded(GridCacheContext.java:1814)
{noformat}
If we perform "put" operation with value of the same type from thick client
before "get" operation, there is no assertion anymore, but {{typeId}} is still
duplicated. And there is another issue: different marshallers can be used to
marshal the same class for thin and thick clients, wrong class descriptor is
returned for class name and there is assertion again. Reproducer:
{code:java}
thickClient.cache(DEFAULT_CACHE_NAME).put(3, Collections.emptyList());
thinClient.cache(DEFAULT_CACHE_NAME).put(2, Collections.emptyList());
thickClient.cache(DEFAULT_CACHE_NAME).get(2);
{code}
Assertion:
{noformat}
java.lang.AssertionError: OptimizedMarshaller should not be used here:
java.util.Collections$EmptyList
at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:865)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
at
org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:792)
at
org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:142)
at
org.apache.ignite.internal.processors.cache.CacheObjectUtils.unwrapBinary(CacheObjectUtils.java:176)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Igniters, thanks for the feedback. > Can we retain only single cancel(long queryId) method in QueryMXBean ? > I think query type can be determined by queryId. It seems the answer is no we can’t do it for now. Different types of queries are handled by different managers and have different sets of identifiers. Please, take a look, into `ScanQueryView` [1], `RunningQueryManager` [2] > Right now different types of IDs (long, UUID, IgniteUuid) look confusing and > messy. Yes. Let’s, fix it. > Shouldn't we unify identifiers for all ongoing tasks in Ignite and move > cancel to a single method in a single bean? I think we should: 1. create separate beans for each API - TransactionMXBean, ComputeMXBean, etc. * The single method will be very error-prone: Consider copy-pasting wrong id from log to its parameters(killing not the buggy compute task, but *VERY* important transaction. How users even know about this type of error with the single method approach? * When the user wants to kill some task(from a script or by hand) he knows the type of task for sure. 2. Implement IEP-39 with the current identifiers. 3. Improve our API with unified identifier types. Bringing our API to the unified identifiers types approach is a very hard task to do and it will take a long time to implement. We shouldn’t leave our users without the proposed API until this moment. [1] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/spi/systemview/view/ScanQueryView.java [2] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/query/RunningQueryManager.java > 5 февр. 2020 г., в 12:07, Alexey Goncharuk > написал(а): > > Folks, > > Shouldn't we unify identifiers for all ongoing tasks in Ignite and move > cancel to a single method in a single bean? > > Right now different types of IDs (long, UUID, IgniteUuid) look confusing > and messy. I understand that such API is enforced by the IDs implementation > in the corresponding Ignite subsystems. However, given that lately we > developed a unified approach to metrics and system views with Nikolay's > help, shouldn't we take one more step forward and unify this part as well?
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Nikolay, > Consider copy-pasting wrong id from log to its > parameters(killing not the buggy compute task, but *VERY* important > transaction. > How users even know about this type of error with the > single method approach? > > I thought that the identifiers would never intersect (meaning that a transaction and a task would never share the same ID) I agree that change ID types for all objects would be a hard task, so probably it's worth discussing a single cancel entry on phase 3.
Re: [DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Alexey.
I’m talking the following scenario:
1. Consider we have unified bean to kill tasks:
CancelMXBean {
public void cancel(long id);
}
2. And we have the following log:
```
Transaction with ID=42 started.
Compute task with ID=43 started.
```
3. We want to kill compute task and by mistake executing:
cancelMxBean.cancel(42); //This will kill transaction not compute task.
The user doesn’t have a chance to know, what type of object he is killing.
I think we should prevent this type of error by the API design.
> 5 февр. 2020 г., в 14:43, Alexey Goncharuk
> написал(а):
>
> Nikolay,
>
>
>>Consider copy-pasting wrong id from log to its
>> parameters(killing not the buggy compute task, but *VERY* important
>> transaction.
>>How users even know about this type of error with the
>> single method approach?
>>
>> I thought that the identifiers would never intersect (meaning that a
> transaction and a task would never share the same ID)
>
> I agree that change ID types for all objects would be a hard task, so
> probably it's worth discussing a single cancel entry on phase 3.
Question: different setting for nodes.
Hi, folks. I recently found that one node might be started with flag reuse-memory-on-deactivate [1] set off while another node might be configured with the flag enabled. This ability hinders prediction of cluster behavior on deactivation/activation. One node keeps data in memory, other doesn’t. Should we check this flag is set equally on all nodes? Is this an issue? We might prevent node start if it has different setting. [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
[jira] [Created] (IGNITE-12625) Thin client: Marshaling/unmarshaling of objects performed twice
Aleksey Plekhanov created IGNITE-12625:
--
Summary: Thin client: Marshaling/unmarshaling of objects performed
twice
Key: IGNITE-12625
URL: https://issues.apache.org/jira/browse/IGNITE-12625
Project: Ignite
Issue Type: Bug
Components: binary, thin client
Reporter: Aleksey Plekhanov
For each thin client cache operation marshaling/unmarshaling of objects
performed twice. For example, cache "put" operation marshal object on the
client-side, then unmarshal object (with keep binaries) on the server-side and
marshal it again before putting it to the cache. It causes some undesirable
effects. For example, references to the same binary object in a collection (
{{new ArrayList(Arrays.asList(person, person))}} ) deserialized as different
objects. Also, with binary object full byte array of this object is marshaled,
but object takes only some part of this array, it causes size overhead for
storing these objects.
To avoid double marshalling we could pass byte array from request content to
cache directly (for example using {{CacheObject}}), but we don't have object
size in thin client protocol, so in any case, we need to traverse the objects.
Also, we don't have the ability to get {{CacheObject}} from the cache now, so
such an approach will only work in one way, for "put" operations, but not for
"get" operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Question: different setting for nodes.
Vladimir. This looks like a bug to me. Can you, please, prepare the simple reproducer for this issue and it’s consequences? > 5 февр. 2020 г., в 17:08, Vladimir Steshin написал(а): > > Hi, folks. > > > > I recently found that one node might be started with flag > reuse-memory-on-deactivate [1] set off while another node might be > configured with the flag enabled. This ability hinders prediction of > cluster behavior on deactivation/activation. One node keeps data in memory, > other doesn’t. > > Should we check this flag is set equally on all nodes? Is this an issue? We > might prevent node start if it has different setting. > > > > [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE
Re: Orphaned, duplicate, and main-class tests!
Hello! Just to resurrect this old thread: I have uncommented another batch of tests, would appreciate a review of PR: https://issues.apache.org/jira/browse/IGNITE-9216 Regards, -- Ilya Kasnacheev ср, 31 окт. 2018 г. в 15:22, Ilya Kasnacheev : > Hello! > > So we have uncommented 4 batches out of 10! 6 to go. Some broken > functionality were exposed. > > There is still work to do, so do not hesitate to assign a subtask to > yourself. > > Regards, > -- > Ilya Kasnacheev > > > ср, 15 авг. 2018 г. в 19:42, Ilya Kasnacheev : > >> Hello! >> >> So we have enabled a first batch of tests: >> https://github.com/apache/ignite/pull/4504 >> >> How it was done: I have uncommented classes. Some of these were absent in >> code base, so I have checked if we didn't lose anything important - they >> were testing CLOCK mode which isn't with us for some time, so I removed >> their entries. >> Then I have ran them, some were broken. Most of those were testing >> on-heap caching with copy=false, which now requires setOnheapCaching(true), >> which I did. After that, cache.invoke() still didn't work, so I commented >> this part out. >> The remaining test was broken due to dependence on hash map iteration >> order, which was changed in Java 8. So I have got the remaining tests >> working, checking important parts of our system. >> >> Please do not hesitate to assign subtasks of >> https://issues.apache.org/jira/browse/IGNITE-9210 to yourself, dabble >> with tests. IMO it's the best way for a novice developer to become >> acquainted with Ignite code base, tests and history, while helping the >> project. >> >> Thanks, >> >> -- >> Ilya Kasnacheev >> >> 2018-08-07 16:54 GMT+03:00 Ilya Kasnacheev : >> >>> Hello! >>> >>> Thank you Dmitriy, and thanks to everybody who participated in >>> discussions. >>> >>> I have created tickets for next steps: >>> https://issues.apache.org/jira/browse/IGNITE-9210 (with subtasks) >>> https://issues.apache.org/jira/browse/IGNITE-9222 >>> https://issues.apache.org/jira/browse/IGNITE-9223 >>> >>> As usual, feedback will be very welcome. >>> >>> Regards, >>> >>> -- >>> Ilya Kasnacheev >>> >>> 2018-08-07 13:58 GMT+03:00 Dmitriy Pavlov : >>> Hi Igniters, I've merged chages for following tickets IGNITE-7615: Find orphaned tests without test suites, create separate test suite for them; IGNITE-8344: Remove duplicate tests and suites; IGNITE-8345: Streamline tests' class names: mark Abstract and Load tests obviously so; After including these suites we have now more than 100 occurrences of //suite.addTest These tests were created early but not executed on TeamCity. If you are interseted in test coverage increase and can contribute each of these suite actualization, please feel free to create ticket for such suites resurrection (or group of suites). Ilya, thank you for contribution and for your efforts to make this happen. Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 12:52, Dmitriy Pavlov : > Hi Ilya, > > could you please actualize this PR. TC Bot can now detect newly > contributed tests' failures, so I think it is best point to apply you > change. > > Sincerely, > Dmitriy Pavlov > > пт, 25 мая 2018 г. в 18:16, Eduard Shangareev < eduard.shangar...@gmail.com > >: > >> Igniters, >> >> While making review I checked next main-method tests: >> >> org.apache.ignite.loadtests.mapper.GridContinuousMapperLoadTest1 >> org.apache.ignite.loadtests.mapper.GridContinuousMapperLoadTest2 >> >> And I have found that they are totally outdated! >> They use config which was changed a long time ago. >> And use localPeek with parameters which don't make sense now. >> >> So, I suggest to delete them. >> >> If there wouldn't be any objection I will do it myself. >> >> >> >> >> On Tue, May 22, 2018 at 6:59 PM, Ilya Kasnacheev < >> ilya.kasnach...@gmail.com> >> wrote: >> >> > Hello, Igniters! >> > >> > One moment more of your time. One, we seem to have a consensus now that >> > tests should be added to suites, but commented out. They should be >> > uncommented out later, for which numerous tickets will be created. This >> way >> > we can tackle. >> > >> > Another issue sprang up, just now I have discovered an 'ignored-tests' >> > module. My proposal thus is to: >> > - Move tests from this suite to relevant suites, comment them out. >> > - Kill this module (with fire). >> > >> > Would be glad to her your input, >> > >> > >> > >> > -- >> > Ilya Kasnacheev >> > >> > 2018-05-03 20:03 GMT+03:00 Ilya Kasnacheev < ilya.kasnach...@gmail.com>: >> > >> > > Hello Dmitry, igniters! >> > > >> > > S
Explicit plugin providers configuration issue
Anton, Mikhail, I stumbled upon impossibility to configure plugin with out adding appropriate class names to org.apache.ignite.plugin.PluginProvider file and found change where this possibility was added [1]. Thanks a lot for this change. I noticed also that IgniteConfiguration.setPluginProviders() overrides plugins that usually will be loaded by ServiceLoader. Is it deliberate decision? It seems that better approach is using plugins from both sources: ServiceLoader provided and user provided via IgniteConfiguration. WDYT? [1] https://issues.apache.org/jira/browse/IGNITE-11744
[DISCUSSION] ignite-extension - ignite-spring-boot-autoconfigurer release.
Hello, Igniters. We don't have a release process for newly created ignite-extensions modules. I want to release two modules: * ignite-spring-boot-autoconfigure * ignite-client-spring-boot-autoconfigure Let's discuss it. Any objections to it? What should be done before release?
Re: [DISCUSSION] ignite-extension - ignite-spring-boot-autoconfigurer release.
Alex, did you have a chance to review Saikat’s changes related to the extensions repository organization and release approach? Denis On Wednesday, February 5, 2020, Nikolay Izhikov wrote: > Hello, Igniters. > > We don't have a release process for newly created ignite-extensions > modules. > I want to release two modules: > > * ignite-spring-boot-autoconfigure > * ignite-client-spring-boot-autoconfigure > > Let's discuss it. > > Any objections to it? > What should be done before release? > -- - Denis
[jira] [Created] (IGNITE-12626) Add RELEASE_NOTES about 2.8
Maxim Muzafarov created IGNITE-12626: Summary: Add RELEASE_NOTES about 2.8 Key: IGNITE-12626 URL: https://issues.apache.org/jira/browse/IGNITE-12626 Project: Ignite Issue Type: Task Reporter: Maxim Muzafarov Assignee: Maxim Muzafarov Fix For: 2.8 Update RELEASE_NOTES for the 2.8 release -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Question: different setting for nodes.
Folks, I think it's a common problem for such system properties (which may be different on different nodes and which may lead to unpredictable cluster behaviour). The same mentioned by Ivan here [1]. Is it better to validate (somehow) all system properties, for instance, started with IGNITE_ prefix, on all nodes the same way as we do it for IgniteConfiguration? [1] https://issues.apache.org/jira/browse/IGNITE-12470?focusedCommentId=17004207&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17004207 On Wed, 5 Feb 2020 at 17:20, Nikolay Izhikov wrote: > > Vladimir. > > This looks like a bug to me. > > Can you, please, prepare the simple reproducer for this issue and it’s > consequences? > > > > 5 февр. 2020 г., в 17:08, Vladimir Steshin написал(а): > > > > Hi, folks. > > > > > > > > I recently found that one node might be started with flag > > reuse-memory-on-deactivate [1] set off while another node might be > > configured with the flag enabled. This ability hinders prediction of > > cluster behavior on deactivation/activation. One node keeps data in memory, > > other doesn’t. > > > > Should we check this flag is set equally on all nodes? Is this an issue? We > > might prevent node start if it has different setting. > > > > > > > > [1] IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE >
Re: Explicit plugin providers configuration issue
Hi, Andrey. The approach of setting plugins via IgniteConfiguration provides an additional feature to control the order in which plugins will be loaded. It's important in the case when several plugins provide different implementations of the same GridComponent while providing some other unique implementations (now the first found plugin that provides required component will be used for its obtaining). The ServiceLoader approach gives no mentioned above order guarantees. So I think that we can't mix these two approaches. Also, it's not clear for me -- are there any reasons to continue support of the ServiceLoader approach, given the fact that configurations needed for a plugin, in this case, are also specified via IgniteConfiguration? Moreover, why someone should mix those approaches? On 05.02.2020 17:35, Andrey Gura wrote: Anton, Mikhail, I stumbled upon impossibility to configure plugin with out adding appropriate class names to org.apache.ignite.plugin.PluginProvider file and found change where this possibility was added [1]. Thanks a lot for this change. I noticed also that IgniteConfiguration.setPluginProviders() overrides plugins that usually will be loaded by ServiceLoader. Is it deliberate decision? It seems that better approach is using plugins from both sources: ServiceLoader provided and user provided via IgniteConfiguration. WDYT? [1] https://issues.apache.org/jira/browse/IGNITE-11744
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Hi Vladimir, Yes, I'm suggesting us to enable this flag by org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE default instead of introducing --force flag and showing any warnings. - Denis On Wed, Feb 5, 2020 at 2:33 AM Vladimir Steshin wrote: > Hello all. > > Denis, which changes exactly? In current implementation of ticket [2] flag > [1] is checked before requiring --force flag and showing any warnings. Do > we need to set reuse-memory-on-deactivate to true by default? > > [1] > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > [2] https://issues.apache.org/jira/browse/IGNITE-12614 > > > вт, 4 февр. 2020 г. в 22:45, Denis Magda : > > > That's the best solution for this scenario. Should we readjust the > already > > created ticket [1] suggesting to implement the changes of Alex Scherbakov > > instead? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12614 > > > > - > > Denis > > > > > > On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov < > > alexey.scherbak...@gmail.com> wrote: > > > > > Folks, > > > > > > For a long time we have a flag [1] > > > > > > It does almost what we want here. > > > > > > I suggest to make this behavior default and rework it to keep data in > > > memory as well (we already have special "recovery" mode for caches). > > > > > > [1] > > > > > > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > > > > > > > > > > > пн, 3 февр. 2020 г. в 18:47, Alexey Goncharuk < > > alexey.goncha...@gmail.com > > > >: > > > > > > > I do not mind making this change if we explicitly and clearly define > > what > > > > 'new inactive state' means. What should happen if a partition is lost > > in > > > > inactive state? What if such node joins back the cluster after? Etc. > > > > > > > > пт, 31 янв. 2020 г. в 20:57, Denis Magda : > > > > > > > > > Back up Ivan's opinion here. Initially, the activation/deactivation > > was > > > > > created for the baseline topology designed for cases with native > > > > > persistence. My thinking was that the mechanism itended to prevent > > data > > > > > inconsistencies while nodes with data on the disk will be in the > > > process > > > > of > > > > > joining the cluster. > > > > > > > > > > Artem, could you please update the docs bringing this to the > > attention > > > of > > > > > the user community? > > > > > https://issues.apache.org/jira/browse/IGNITE-12615 > > > > > > > > > > AG, what if we don't purge data from memory at least for the caches > > not > > > > > backed by the native persistence? Is this a big deal? We can > > certainly > > > > put > > > > > this off by my guts feel we'll return to this question sooner or > > later. > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Fri, Jan 31, 2020 at 2:17 AM Ivan Pavlukhin < > vololo...@gmail.com> > > > > > wrote: > > > > > > > > > > > For me it looks like some coincidence effect. I understand that > we > > > get > > > > > > such behavior because deactivation works the same way as for > > > > > > persistent caches. Was cluster activation/deactivation designed > and > > > > > > described for in-memory caches? Current behavior sounds for me a > as > > > > > > big risk. I expect a lot of upset users unexpectedly purged all > > their > > > > > > data. > > > > > > > > > > > > пт, 31 янв. 2020 г. в 00:00, Alexey Goncharuk < > > > > > alexey.goncha...@gmail.com > > > > > > >: > > > > > > > > > > > > > > Because originally the sole purpose of deactivation was > resource > > > > > > > deallocation. > > > > > > > > > > > > > > чт, 30 янв. 2020 г. в 22:13, Denis Magda : > > > > > > > > > > > > > > > Such a revelation for me that data is purged from RAM if > > someone > > > > > > > > deactivates the cluster. Alex, do you remember why we decided > > to > > > > > > implement > > > > > > > > it this way initially? > > > > > > > > > > > > > > > > - > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 30, 2020 at 2:09 AM Alexey Goncharuk < > > > > > > > > alexey.goncha...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I agree on CLI and JMX because those interfaces can be used > > by > > > a > > > > > > system > > > > > > > > > administrator and can be invoked by mistake. > > > > > > > > > > > > > > > > > > As for the Java API, personally, I find it strange to add > > > 'force' > > > > > or > > > > > > > > > 'confirm' flags to it because it is very unlikely that such > > an > > > > > > invocation > > > > > > > > > is done by mistake. Such mistakes are caught during the > > testing > > > > > > phase and > > > > > > > > > developers will end up hard-coding 'true' as a flag > anyways. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > > > > Ivan Pavlukhin > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > Alexei Scherbakov > > > > > >
[jira] [Created] (IGNITE-12627) Control utility does not show corrupted indexes
Vladimir Malinovskiy created IGNITE-12627:
-
Summary: Control utility does not show corrupted indexes
Key: IGNITE-12627
URL: https://issues.apache.org/jira/browse/IGNITE-12627
Project: Ignite
Issue Type: Bug
Reporter: Vladimir Malinovskiy
In 2.5.11/8.5.11 control utility does not show corrupted indexes:
{noformat}
{{[avolkov@lab43 bin]$ ./control.sh --host 172.25.1.43 --port 11211 --cache
validate_indexes
Control utility [ver. 2.5.11#20191004-sha1:565c778e]
2019 Copyright(C) Apache Software Foundation
User: avolkov
Time: 2019-10-09T11:10:56.519
Command [CACHE] started
Arguments: --host 172.25.1.43 --port 11211 --cache validate_indexes
no issues found.
Command [CACHE] finished with code: 0
Control utility has completed execution at: 2019-10-09T11:10:56.916
Execution time: 397 ms}}
{noformat}{{}}
but in logs I can see:
\{noformat}
{{11:10:56,745][WARNING][pool-6-thread-3][ValidateIndexesClosure] Current
progress of ValidateIndexesClosure: checked integrity of 1 index partitions of
14 cache groups
[11:10:56,850][SEVERE][pool-6-thread-14][ValidateIndexesClosure] Find in index
failed: IndexValidationIssue [key=null, cacheName=cache_group_1_028,
idxName=EXAMPLE_IDX], class org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[], cacheId=1586135659,
cacheName=cache_group_1_028, indexName=EXAMPLE_IDX, msg=Runtime failure on
bounds: [lower=null, upper=null]]" [5-195]
[11:10:56,850][SEVERE][pool-6-thread-12][ValidateIndexesClosure] Find in index
failed: IndexValidationIssue [key=null, cacheName=cache_group_1_028,
idxName=_key_PK], class org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[], cacheId=1586135659,
cacheName=cache_group_1_028, indexName=_key_PK, msg=Runtime failure on bounds:
[lower=null, upper=null]]" [5-195]
[11:10:56,851][SEVERE][pool-6-thread-11][ValidateIndexesClosure] Find in index
failed: IndexValidationIssue [key=null, cacheName=cache_group_1_028,
idxName=ALLTYPESINDEXED_LONGCOL_IDX], class org.h2.message.DbException: General
error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[], cacheId=1586135659,
cacheName=cache_group_1_028, indexName=ALLTYPESINDEXED_LONGCOL_IDX, msg=Runtime
failure on bounds: [lower=null, upper=null]]" [5-195]
[11:10:56,853][SEVERE][pool-6-thread-3][ValidateIndexesClosure] Find in index
failed: IndexValidationIssue [key=null, cacheName=cache_group_1_028,
idxName=EXAMPLE_IDX], class org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[], cacheId=1586135659,
cacheName=cache_group_1_028, indexName=EXAMPLE_IDX, msg=Runtime failure on
bounds: [lower=null, upper=null]]" [5-195]
[11:10:56,853][SEVERE][pool-6-thread-1][ValidateIndexesClosure] Find in index
failed: IndexValidationIssue [key=null, cacheName=cache_group_1_028,
idxName=ALLTYPESINDEXED_LONGCOL_IDX], class org.h2.message.DbException: General
error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[], cacheId=1586135659,
cacheName=cache_group_1_028, indexName=ALLTYPESINDEXED_LONGCOL_IDX, msg=Runtime
failure on bounds: [lower=null, upper=null]]" [5-195]
[11:10:56,856][SEVERE][pool-6-thread-6][ValidateIndexesClosure] Find in index
failed: IndexValidationIssue [key=null, cacheName=cache_group_1_028,
idxName=_key_PK], class org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[], cacheId=1586135659,
cacheName=cache_group_1_028, indexName=_key_PK, msg=Runtime failure on bounds:
[lower=null, upper=null]]" [5-195]
[11:10:56,861][WARNING][mgmt-#131][ValidateIndexesClosure]
ValidateIndexesClosure finished: processed 2648 partitions and 6 indexes.}}
{noformat}{{}}
This information isn't reflected in control utility output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Denis, but why reuse-data-on-deactivate was disabled by default? There should be a reason for that. Any data consistency issues when node gets activated anew? We may use both solutions because the flag can be switched off again. ср, 5 февр. 2020 г. в 20:47, Denis Magda : > Hi Vladimir, > > Yes, I'm suggesting us to enable this flag by > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > default instead of introducing --force flag and showing any warnings. > > - > Denis > > > On Wed, Feb 5, 2020 at 2:33 AM Vladimir Steshin > wrote: > > > Hello all. > > > > Denis, which changes exactly? In current implementation of ticket [2] > flag > > [1] is checked before requiring --force flag and showing any warnings. Do > > we need to set reuse-memory-on-deactivate to true by default? > > > > [1] > > > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12614 > > > > > > вт, 4 февр. 2020 г. в 22:45, Denis Magda : > > > > > That's the best solution for this scenario. Should we readjust the > > already > > > created ticket [1] suggesting to implement the changes of Alex > Scherbakov > > > instead? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12614 > > > > > > - > > > Denis > > > > > > > > > On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov < > > > alexey.scherbak...@gmail.com> wrote: > > > > > > > Folks, > > > > > > > > For a long time we have a flag [1] > > > > > > > > It does almost what we want here. > > > > > > > > I suggest to make this behavior default and rework it to keep data in > > > > memory as well (we already have special "recovery" mode for caches). > > > > > > > > [1] > > > > > > > > > > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > > > > > > > > > > > > > > > пн, 3 февр. 2020 г. в 18:47, Alexey Goncharuk < > > > alexey.goncha...@gmail.com > > > > >: > > > > > > > > > I do not mind making this change if we explicitly and clearly > define > > > what > > > > > 'new inactive state' means. What should happen if a partition is > lost > > > in > > > > > inactive state? What if such node joins back the cluster after? > Etc. > > > > > > > > > > пт, 31 янв. 2020 г. в 20:57, Denis Magda : > > > > > > > > > > > Back up Ivan's opinion here. Initially, the > activation/deactivation > > > was > > > > > > created for the baseline topology designed for cases with native > > > > > > persistence. My thinking was that the mechanism itended to > prevent > > > data > > > > > > inconsistencies while nodes with data on the disk will be in the > > > > process > > > > > of > > > > > > joining the cluster. > > > > > > > > > > > > Artem, could you please update the docs bringing this to the > > > attention > > > > of > > > > > > the user community? > > > > > > https://issues.apache.org/jira/browse/IGNITE-12615 > > > > > > > > > > > > AG, what if we don't purge data from memory at least for the > caches > > > not > > > > > > backed by the native persistence? Is this a big deal? We can > > > certainly > > > > > put > > > > > > this off by my guts feel we'll return to this question sooner or > > > later. > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Fri, Jan 31, 2020 at 2:17 AM Ivan Pavlukhin < > > vololo...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > For me it looks like some coincidence effect. I understand that > > we > > > > get > > > > > > > such behavior because deactivation works the same way as for > > > > > > > persistent caches. Was cluster activation/deactivation designed > > and > > > > > > > described for in-memory caches? Current behavior sounds for me > a > > as > > > > > > > big risk. I expect a lot of upset users unexpectedly purged all > > > their > > > > > > > data. > > > > > > > > > > > > > > пт, 31 янв. 2020 г. в 00:00, Alexey Goncharuk < > > > > > > alexey.goncha...@gmail.com > > > > > > > >: > > > > > > > > > > > > > > > > Because originally the sole purpose of deactivation was > > resource > > > > > > > > deallocation. > > > > > > > > > > > > > > > > чт, 30 янв. 2020 г. в 22:13, Denis Magda >: > > > > > > > > > > > > > > > > > Such a revelation for me that data is purged from RAM if > > > someone > > > > > > > > > deactivates the cluster. Alex, do you remember why we > decided > > > to > > > > > > > implement > > > > > > > > > it this way initially? > > > > > > > > > > > > > > > > > > - > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 30, 2020 at 2:09 AM Alexey Goncharuk < > > > > > > > > > alexey.goncha...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I agree on CLI and JMX because those interfaces can be > used > > > by > > > > a > > > > > > > system > > > > > > > > > > administrator and can be invoked by mistake. > > > > > > > > > > > > > > > > > > > > As for the Java API, personally, I find it strange to add > > > > 'force' > > > > > >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
I believe there might be a consistency-related reason why this flag was disabled by default for caches that store data in Ignite native persistence. I hope, Alex Goncharuk or Scherbakov can shed some light on this. As for the memory-only caches or caches backed up by a CacheStore such as an RDBMS, enabling of the flag should be harmless. Once we do this, we'll eliminate the need to load the data back into the cluster which can be a time-consuming operation depending on the data volume. - Denis On Wed, Feb 5, 2020 at 11:58 AM Vladimir Steshin wrote: > Denis, but why reuse-data-on-deactivate was disabled by default? There > should be a reason for that. Any data consistency issues when node gets > activated anew? We may use both solutions because the flag can be switched > off again. > > ср, 5 февр. 2020 г. в 20:47, Denis Magda : > > > Hi Vladimir, > > > > Yes, I'm suggesting us to enable this flag by > > > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > default instead of introducing --force flag and showing any warnings. > > > > - > > Denis > > > > > > On Wed, Feb 5, 2020 at 2:33 AM Vladimir Steshin > > wrote: > > > > > Hello all. > > > > > > Denis, which changes exactly? In current implementation of ticket [2] > > flag > > > [1] is checked before requiring --force flag and showing any warnings. > Do > > > we need to set reuse-memory-on-deactivate to true by default? > > > > > > [1] > > > > > > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-12614 > > > > > > > > > вт, 4 февр. 2020 г. в 22:45, Denis Magda : > > > > > > > That's the best solution for this scenario. Should we readjust the > > > already > > > > created ticket [1] suggesting to implement the changes of Alex > > Scherbakov > > > > instead? > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12614 > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov < > > > > alexey.scherbak...@gmail.com> wrote: > > > > > > > > > Folks, > > > > > > > > > > For a long time we have a flag [1] > > > > > > > > > > It does almost what we want here. > > > > > > > > > > I suggest to make this behavior default and rework it to keep data > in > > > > > memory as well (we already have special "recovery" mode for > caches). > > > > > > > > > > [1] > > > > > > > > > > > > > > > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > > > > > > > > > > > > > > > > > > > пн, 3 февр. 2020 г. в 18:47, Alexey Goncharuk < > > > > alexey.goncha...@gmail.com > > > > > >: > > > > > > > > > > > I do not mind making this change if we explicitly and clearly > > define > > > > what > > > > > > 'new inactive state' means. What should happen if a partition is > > lost > > > > in > > > > > > inactive state? What if such node joins back the cluster after? > > Etc. > > > > > > > > > > > > пт, 31 янв. 2020 г. в 20:57, Denis Magda : > > > > > > > > > > > > > Back up Ivan's opinion here. Initially, the > > activation/deactivation > > > > was > > > > > > > created for the baseline topology designed for cases with > native > > > > > > > persistence. My thinking was that the mechanism itended to > > prevent > > > > data > > > > > > > inconsistencies while nodes with data on the disk will be in > the > > > > > process > > > > > > of > > > > > > > joining the cluster. > > > > > > > > > > > > > > Artem, could you please update the docs bringing this to the > > > > attention > > > > > of > > > > > > > the user community? > > > > > > > https://issues.apache.org/jira/browse/IGNITE-12615 > > > > > > > > > > > > > > AG, what if we don't purge data from memory at least for the > > caches > > > > not > > > > > > > backed by the native persistence? Is this a big deal? We can > > > > certainly > > > > > > put > > > > > > > this off by my guts feel we'll return to this question sooner > or > > > > later. > > > > > > > > > > > > > > - > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 31, 2020 at 2:17 AM Ivan Pavlukhin < > > > vololo...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > For me it looks like some coincidence effect. I understand > that > > > we > > > > > get > > > > > > > > such behavior because deactivation works the same way as for > > > > > > > > persistent caches. Was cluster activation/deactivation > designed > > > and > > > > > > > > described for in-memory caches? Current behavior sounds for > me > > a > > > as > > > > > > > > big risk. I expect a lot of upset users unexpectedly purged > all > > > > their > > > > > > > > data. > > > > > > > > > > > > > > > > пт, 31 янв. 2020 г. в 00:00, Alexey Goncharuk < > > > > > > > alexey.goncha...@gmail.com > > > > > > > > >: > > > > > > > > > > > > > > > > > > Because originally the sole purpose of deactivation was > > > resource > > > > > > > > > deallocation. > > > > > > > > > > > > > > > > > > чт, 30 ян
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Igniters, I've prepared RELEASE_NOTES pull-request [1] to the 2.8 release. Currently, IEP-35 monitoring issues are not included in this PR. Will do it soon. Please, take a look. [1] https://github.com/apache/ignite/pull/7367/files On Mon, 3 Feb 2020 at 14:38, Maxim Muzafarov wrote: > > Igniters, > > > Let me share the current status of the release. > > 1. > Waiting for the issues [1] [2] (discussed previously this thread) to > be tested by TC.Bot and merged to the 2.8 release branch. > > 2. > Only 2 release BLOCKER issues left. I'm planning to move these issues > to 2.8.1 release. > The issue [4] (Error during purges by expiration: Unknown page type) > will be covered by [1] [2]. > The issue [3] (Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes > getting down) probably require additional info to reproduce the issue. > > 3. > A potential performance drop on `putAll` operations on an in-memory > cluster (see [5] for details). > I'll try to reproduce in another test environment. > > > Will keep you posted. > > > [1] https://issues.apache.org/jira/browse/IGNITE-12593 > [2] https://issues.apache.org/jira/browse/IGNITE-12594 > [3] https://issues.apache.org/jira/browse/IGNITE-12398 > [4] https://issues.apache.org/jira/browse/IGNITE-12489 > [5] > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks(LATEST) > > On Thu, 30 Jan 2020 at 15:02, Alexey Goncharuk > wrote: > > > > Sounds good, will do!
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Maxim, A couple of questions: 1. We added an annotation to designate experimental API. Should not we state in release notes what new experimental API was added? Perhaps in a separate block. 2. As Flink integration was moved to external repository how Ignite 2.8 users will be able to use that integration? ср, 5 февр. 2020 г. в 21:21, Maxim Muzafarov : > > Igniters, > > > I've prepared RELEASE_NOTES pull-request [1] to the 2.8 release. > > Currently, IEP-35 monitoring issues are not included in this PR. Will > do it soon. > Please, take a look. > > > [1] https://github.com/apache/ignite/pull/7367/files > > On Mon, 3 Feb 2020 at 14:38, Maxim Muzafarov wrote: > > > > Igniters, > > > > > > Let me share the current status of the release. > > > > 1. > > Waiting for the issues [1] [2] (discussed previously this thread) to > > be tested by TC.Bot and merged to the 2.8 release branch. > > > > 2. > > Only 2 release BLOCKER issues left. I'm planning to move these issues > > to 2.8.1 release. > > The issue [4] (Error during purges by expiration: Unknown page type) > > will be covered by [1] [2]. > > The issue [3] (Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes > > getting down) probably require additional info to reproduce the issue. > > > > 3. > > A potential performance drop on `putAll` operations on an in-memory > > cluster (see [5] for details). > > I'll try to reproduce in another test environment. > > > > > > Will keep you posted. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12593 > > [2] https://issues.apache.org/jira/browse/IGNITE-12594 > > [3] https://issues.apache.org/jira/browse/IGNITE-12398 > > [4] https://issues.apache.org/jira/browse/IGNITE-12489 > > [5] > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks(LATEST) > > > > On Thu, 30 Jan 2020 at 15:02, Alexey Goncharuk > > wrote: > > > > > > Sounds good, will do! -- Best regards, Ivan Pavlukhin
[jira] [Created] (IGNITE-12628) Add tests for jmx metrics return types
Vladimir Malinovskiy created IGNITE-12628: - Summary: Add tests for jmx metrics return types Key: IGNITE-12628 URL: https://issues.apache.org/jira/browse/IGNITE-12628 Project: Ignite Issue Type: Bug Reporter: Vladimir Malinovskiy Add tests than check if JMX metrics comply with Oracle best practices([https://www.oracle.com/technetwork/java/javase/tech/best-practices-jsp-136021.html]) -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Ivan, > Should not we state in release notes what new experimental API was added? I think we should. Will do. Just not to miss anything that we should mark with @IgniteExperimental: Consistency Check [1], Monitoring [2] anything else? > As Flink integration was moved to external repository how Ignite 2.8 users > will be able to use that integration? Since ignite-extension has a separate release cycle (right?), it is better to release ignite-extension rather than cherry-pick this change back to 2.8. I also think it is not a blocker for the release, but we should do our best make the first ignite-extension release as earlier as possible. [1] https://issues.apache.org/jira/browse/IGNITE-10663 [2] https://issues.apache.org/jira/browse/IGNITE-11848 On Wed, 5 Feb 2020 at 22:07, Ivan Pavlukhin wrote: > > Maxim, > > A couple of questions: > 1. We added an annotation to designate experimental API. Should not we > state in release notes what new experimental API was added? Perhaps in > a separate block. > 2. As Flink integration was moved to external repository how Ignite > 2.8 users will be able to use that integration? > > ср, 5 февр. 2020 г. в 21:21, Maxim Muzafarov : > > > > Igniters, > > > > > > I've prepared RELEASE_NOTES pull-request [1] to the 2.8 release. > > > > Currently, IEP-35 monitoring issues are not included in this PR. Will > > do it soon. > > Please, take a look. > > > > > > [1] https://github.com/apache/ignite/pull/7367/files > > > > On Mon, 3 Feb 2020 at 14:38, Maxim Muzafarov wrote: > > > > > > Igniters, > > > > > > > > > Let me share the current status of the release. > > > > > > 1. > > > Waiting for the issues [1] [2] (discussed previously this thread) to > > > be tested by TC.Bot and merged to the 2.8 release branch. > > > > > > 2. > > > Only 2 release BLOCKER issues left. I'm planning to move these issues > > > to 2.8.1 release. > > > The issue [4] (Error during purges by expiration: Unknown page type) > > > will be covered by [1] [2]. > > > The issue [3] (Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes > > > getting down) probably require additional info to reproduce the issue. > > > > > > 3. > > > A potential performance drop on `putAll` operations on an in-memory > > > cluster (see [5] for details). > > > I'll try to reproduce in another test environment. > > > > > > > > > Will keep you posted. > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12593 > > > [2] https://issues.apache.org/jira/browse/IGNITE-12594 > > > [3] https://issues.apache.org/jira/browse/IGNITE-12398 > > > [4] https://issues.apache.org/jira/browse/IGNITE-12489 > > > [5] > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks(LATEST) > > > > > > On Thu, 30 Jan 2020 at 15:02, Alexey Goncharuk > > > wrote: > > > > > > > > Sounds good, will do! > > > > -- > Best regards, > Ivan Pavlukhin
[DISCUSS] Public API deprecation rules
Igniters, We would like to add some formal requirements to the API deprecation process. So far the API deprecation was more like intuitive-driven which recently led to some disagreement and delaying the AI 2.8 release [1][2]. During the discussion [1] we agreed to add an annotation @IgniteExperimental [3] which marks APIs which are yet not stable, dangerous, partially implemented or are allowed to be changed in a future. The reason to release such APIs is to expose the APIs to users and collect feedback on the usability and possible bugs. The argument we did not manage to resolve is whether we are allowed to deprecate the old APIs _before_ the new ones get stable and get released without @IgniteExperimental annotation. We decided to resolve the uncertainty with a vote. The vote we are going to have is reduced to two choices so far: * Never deprecate the old APIs unless the new APIs are stable and released without @IgniteExperimental. The old APIs javadoc may be updated with a reference to new APIs to encourage users to evaluate new APIs * Allow to deprecate the old APIs even when new APIs are marked with @IgniteExperimental to explicitly notify users that the new APIs are available Nikolay, Andrey, please let us know if we should correct the choices articulation or add another option for the vote. --AG [1] http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html [2] http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html [3] https://issues.apache.org/jira/browse/IGNITE-12559
Random Forest, trying to evaluate accuracy but getting exception
I have attempted to add this call to a RandomForestModel in order to obtain
accuracy:
*double accuracy = Evaluator.evaluate(
dataCache,
randomForestMdl,
vectorizer,
new Accuracy<>()
);
System.out.println("\n>>> Accuracy " + accuracy);
System.out.println("\n>>> Test Error " + (1 - accuracy));*
But I get this exception (editted to remove extra detail). Am I using the
wrong parameters?
Exception in thread "main" java.lang.RuntimeException: class
/org.apache.ignite.ml.math.exceptions.IndexException: Invalid (out of bound)
index: 16
at
org.apache.ignite.ml.selection.scoring.evaluator.Evaluator.calculateMetric(Evaluator.java:330)
at
org.apache.ignite.ml.selection.scoring.evaluator.Evaluator.evaluate(Evaluator.java:57)
at com.gg.TrainRandomForest2.main(TrainRandomForest2.java:111)
Caused by: class org.apache.ignite.ml.math.exceptions.IndexException:
Invalid (out of bound) index: 16
at
org.apache.ignite.ml.math.primitives.vector.AbstractVector.checkIndex(AbstractVector.java:174)
at
org.apache.ignite.ml.math.primitives.vector.AbstractVector.get(AbstractVector.java:179)
at
org.apache.ignite.ml.tree.randomforest.data.TreeNode.predict(TreeNode.java:91)
at
org.apache.ignite.ml.tree.randomforest.data.TreeNode.predict(TreeNode.java:92)
at
org.apache.ignite.ml.tree.randomforest.data.TreeRoot.predict(TreeRoot.java:52)
at
org.apache.ignite.ml.tree.randomforest.data.TreeRoot.predict(TreeRoot.java:29)
at
org.apache.ignite.ml.composition.ModelsComposition.predict(ModelsComposition.java:64)
at
org.apache.ignite.ml.composition.ModelsComposition.predict(ModelsComposition.java:32)
at
org.apache.ignite.ml.selection.scoring.cursor.CacheBasedLabelPairCursor$TruthWithPredictionIterator.next(CacheBasedLabelPairCursor.java:145)
at
org.apache.ignite.ml.selection.scoring.cursor.CacheBasedLabelPairCursor$TruthWithPredictionIterator.next(CacheBasedLabelPairCursor.java:121)
at
org.apache.ignite.ml.selection.scoring.metric.classification.Accuracy.score(Accuracy.java:36)
at
org.apache.ignite.ml.selection.scoring.evaluator.Evaluator.calculateMetric(Evaluator.java:328)
... 2 more
/
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSS] Public API deprecation rules
The justification and choices look crystal clear to me. - Denis On Wed, Feb 5, 2020 at 1:36 PM Alexey Goncharuk wrote: > Igniters, > > We would like to add some formal requirements to the API deprecation > process. So far the API deprecation was more like intuitive-driven which > recently led to some disagreement and delaying the AI 2.8 release [1][2]. > > During the discussion [1] we agreed to add an > annotation @IgniteExperimental [3] which marks APIs which are yet not > stable, dangerous, partially implemented or are allowed to be changed in a > future. The reason to release such APIs is to expose the APIs to users and > collect feedback on the usability and possible bugs. > > The argument we did not manage to resolve is whether we are allowed to > deprecate the old APIs _before_ the new ones get stable and get released > without @IgniteExperimental annotation. We decided to resolve the > uncertainty with a vote. > > The vote we are going to have is reduced to two choices so far: > * Never deprecate the old APIs unless the new APIs are stable and released > without @IgniteExperimental. The old APIs javadoc may be updated with a > reference to new APIs to encourage users to evaluate new APIs > * Allow to deprecate the old APIs even when new APIs are marked > with @IgniteExperimental to explicitly notify users that the new APIs are > available > > Nikolay, Andrey, please let us know if we should correct the choices > articulation or add another option for the vote. > > --AG > > [1] > > http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html > [2] > > http://apache-ignite-developers.2346864.n4.nabble.com/Internal-classes-are-exposed-in-public-API-td45146.html > [3] https://issues.apache.org/jira/browse/IGNITE-12559 >
[jira] [Created] (IGNITE-12629) Fit all metrics into single standard
Vladimir Malinovskiy created IGNITE-12629: - Summary: Fit all metrics into single standard Key: IGNITE-12629 URL: https://issues.apache.org/jira/browse/IGNITE-12629 Project: Ignite Issue Type: Bug Reporter: Vladimir Malinovskiy All ignite metrics should be fitted to single standard. This standard should be discussed and documented. Oracle best practices [https://www.oracle.com/technetwork/java/javase/tech/best-practices-jsp-136021.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Random Forest, trying to evaluate accuracy but getting exception
Yes, the known issue in the Random Forest, it throws the strange math
exception in the case, when the preprocessed vector has the
different length with the vector which could be handled by the model.
I didn't see your another code, but I could suppose, that you have another
preprocessing stages like one-hot-encoding that adds new columns and you
missed the link here
double accuracy = Evaluator.evaluate(
dataCache,
randomForestMdl,
vectorizer, <-- should be correct
(the last in the chain and the same like in the fit() call ) preprocesser
new Accuracy<>()
);
I'll add the ticket to throw another exception with the correct message to
explain this situation for the user.
чт, 6 февр. 2020 г. в 00:29, kencottrell :
> I have attempted to add this call to a RandomForestModel in order to obtain
> accuracy:
>
> *double accuracy = Evaluator.evaluate(
> dataCache,
> randomForestMdl,
> vectorizer,
> new Accuracy<>()
> );
>
> System.out.println("\n>>> Accuracy " + accuracy);
> System.out.println("\n>>> Test Error " + (1 - accuracy));*
>
>
> But I get this exception (editted to remove extra detail). Am I using the
> wrong parameters?
>
> Exception in thread "main" java.lang.RuntimeException: class
> /org.apache.ignite.ml.math.exceptions.IndexException: Invalid (out of
> bound)
> index: 16
> at
> org.apache.ignite.ml
> .selection.scoring.evaluator.Evaluator.calculateMetric(Evaluator.java:330)
> at
> org.apache.ignite.ml
> .selection.scoring.evaluator.Evaluator.evaluate(Evaluator.java:57)
> at com.gg.TrainRandomForest2.main(TrainRandomForest2.java:111)
> Caused by: class org.apache.ignite.ml.math.exceptions.IndexException:
> Invalid (out of bound) index: 16
> at
>
> org.apache.ignite.ml.math.primitives.vector.AbstractVector.checkIndex(AbstractVector.java:174)
> at
>
> org.apache.ignite.ml.math.primitives.vector.AbstractVector.get(AbstractVector.java:179)
> at
>
> org.apache.ignite.ml.tree.randomforest.data.TreeNode.predict(TreeNode.java:91)
> at
>
> org.apache.ignite.ml.tree.randomforest.data.TreeNode.predict(TreeNode.java:92)
> at
>
> org.apache.ignite.ml.tree.randomforest.data.TreeRoot.predict(TreeRoot.java:52)
> at
>
> org.apache.ignite.ml.tree.randomforest.data.TreeRoot.predict(TreeRoot.java:29)
> at
> org.apache.ignite.ml
> .composition.ModelsComposition.predict(ModelsComposition.java:64)
> at
> org.apache.ignite.ml
> .composition.ModelsComposition.predict(ModelsComposition.java:32)
> at
> org.apache.ignite.ml
> .selection.scoring.cursor.CacheBasedLabelPairCursor$TruthWithPredictionIterator.next(CacheBasedLabelPairCursor.java:145)
> at
> org.apache.ignite.ml
> .selection.scoring.cursor.CacheBasedLabelPairCursor$TruthWithPredictionIterator.next(CacheBasedLabelPairCursor.java:121)
> at
> org.apache.ignite.ml
> .selection.scoring.metric.classification.Accuracy.score(Accuracy.java:36)
> at
> org.apache.ignite.ml
> .selection.scoring.evaluator.Evaluator.calculateMetric(Evaluator.java:328)
> ... 2 more
>
> /
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
