Re: Adding experimental support for Intel Optane DC Persistent Memory

[Ivan Pavlukhin]

Hope that we will be better prepared for new iteration.

чт, 17 окт. 2019 г. в 02:29, Denis Magda :
>
> Ivan,
>
> There is no argument here, I see what you mean. All we know is that Optane
> is a new class of a storage device that is slower than RAM but faster than
> disk. Plus, it's denser than RAM, and non-volatile that let us store
> many-more terabytes of data on a single machine that can be accessed via
> the memory bus and don't worry about data preloading upon restarts. So,
> there definitely will be a benefit for certain use cases once Optane is
> supported natively - much more data can be stored and accessed locally at
> (almost) speed of RAM.
>
> The current integration is not finished yet (based on the review) but once
> we get hardware, agree, that the benchmarks can be executed even over what
> was developed so far.
>
>
> -
> Denis
>
>
> On Tue, Oct 15, 2019 at 10:44 PM Ivan Pavlukhin  wrote:
>
> > Denis,
> >
> > I suppose we are using not optimal strategy here. And motivation is simple:
> > 1. Code review is "expensive", benchmarking (usually) are "cheap".
> > 2. Code review is "subjective", benchmarking are "objective".
> >
> > And my personal opinion here is that there could be a performance win
> > with Optane in proposed implementation in particular cases. It depends
> > on how much slower is Optane performance comparing to RAM. But we do
> > not know it.
> >
> > ср, 16 окт. 2019 г. в 01:10, Denis Magda :
> > >
> > > Alex, thanks for an extensive summary and proposing different
> > > implementation trails. Presently, the JEP-352 approach seems the most
> > > reasonable - JDK folks will take care of the integration and will be
> > > testing/maintaining the feature going forward, while the Ignite community
> > > will reuse what JDK provides. However, let's see what Eric and Mulugeta
> > of
> > > Intel (both copied) think about this and other approaches.
> > >
> > > Ivan, I'm not sure if any performance testing was done for the given
> > > pull-request. Anyway, it might be premature considering Alex's summary.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Mon, Oct 14, 2019 at 11:50 PM Ivan Pavlukhin 
> > wrote:
> > >
> > > > By the way, did we have a change to run some existing benchmarks
> > > > against proposed implementation?
> > > >
> > > > пт, 11 окт. 2019 г. в 17:49, Alexey Goncharuk <
> > alexey.goncha...@gmail.com
> > > > >:
> > > > >
> > > > > Hello Mulugeta, Igniters.
> > > > >
> > > > > Thanks for your interest and efforts in integrating the persistent
> > memory
> > > > > to Ignite. Here are my thoughts on the PR:
> > > > >
> > > > > First of all I was questioning whether we want to use the integration
> > > > with
> > > > > the pmem library via JNI. Java 14 will have native support for NVME
> > via
> > > > > native mapped byte buffers [1] since the corresponding tickets are
> > > > already
> > > > > resolved [2],[3]. The suggested integration uses JNI calls quite
> > often
> > > > (if
> > > > > I read the PR right, there will be tens of JNI calls per a single
> > > > > operation), which may and most likely will undermine the benefits of
> > > > using
> > > > > PM. On the other hand, there is a PoC/project [4] which successfully
> > > > > demonstrates how the new integration can be used to natively access
> > > > > persistent memory from Java. Additionally, new Unsafe have several
> > > > > performance issues - each put* methods has a map lookup, and CAS
> > method
> > > > > uses a global lock. Removing these issues will change the PR and
> > > > > architecture dramatically.
> > > > >
> > > > > Other than performance and an extra library dependency, there are
> > > > > additional reasons to use custom implementation of PM persistence
> > > > > primitives:
> > > > >
> > > > >- Ignite significantly benefits from having WAL as a single point
> > of
> > > > >data for both failure recovery and historical rebalance. As far
> > as I
> > > > can
> > > > >guess, the suggested library also uses some sort of journaling in
> > > > order to
> > > > >support crash recovery. Since we do not want give up on the
> > historical
> > > > >rebalance, we would need to write an additional journal, which
> > then
> > > > raises
> > > > >a question of how to transact between this journal.
> > > > >- Ignite is supposed to work with large indexes, a I not aware how
> > > > LLPL
> > > > >handles them. Several researches [5],[6] show that since PM
> > memory has
> > > > >larger access latency than regular RAM, it is beneficial to buffer
> > > > index
> > > > >pages in regular memory and store leaf pages in PM. There is a
> > generic
> > > > >approach to convert in-memory indexes to PM ones [7], I think it
> > makes
> > > > >sense to investigate whether our existing indexes can be
> > converted to
> > > > PM.
> > > > >- There is an active IEP which is targeted for a file-based
> > > > rebalancing.
> > > > >In the suggested implementation this 

Re: [DISCUSS] Pub/Sub Streamer Implementation

[Saikat Maitra]

Hi Denis, Emmanouil

Thank you for your email. I think creating a separate integration project
in github and also proposing it as an apache incubator project will be a
good move. The other separate integration modules
https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
can
be moved to this new apache incubator project.

There are similar integration available for Flink and Spark in Apache Bahir
https://bahir.apache.org/

Do you think we should reach out to Apache Bahir community with a proposal
or we should plan to create a separate Apache Incubator project?

Regards,
Saikat



On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras 
wrote:

> Hi all!
>
> Based on the discussions most probably this is going to be on another
> GitHub repo. Therefore I will prepare a standalone project with the Pub/Sub
> module and once the repository is created a pull request shall be issued.
> If there is anything else I can do in order to facilitate this please let
> me know.
>
> If a ticket is created for this issue, I shall be able to create a pull
> request and thus a build will take place on ignite's CI.
> If everyone is aligned with adding a Pub/Sub implementation may I create a
> ticket on JIRA?
>
> My next question has to do on how Ignite configures the build jobs in Team
> City. The unit tests for my implementation need a local Pub/Sub server up
> and running (they provide one for testing purposes). Will this take effect
> in a form of a build script inside my implementation or is it something
> that should be configured on Team City?
>
> Kind regards,
> Emmanouil
>
>
> *Emmanouil Gkatziouras*
> https://egkatzioura.com/ |
> https://www.linkedin.com/in/gkatziourasemmanouil/
> https://github.com/gkatzioura
>
>
> On Tue, 15 Oct 2019 at 23:27, Denis Magda  wrote:
>
> > Emmanouil, Saikat,
> >
> > After contemplating on this, let me propose another way with this
> > particular integration that is aligned with our modularization endeavor.
> > The modularization [1] defines the Ignite core, Ignite modules, and
> > independent integrations. The first two are to be located in Ignite
> > repositories and to be tested/updated by the community for every release.
> > The modules are a subset of existing integrations that are adopted widely
> > and require community attention.
> >
> > As for the independent integrations [1], those are supposed to be located
> > in a separate GitHub repo. I would advise us to start implementing this
> > concept with the Pub/Sub integration. Even though it will be located in a
> > separate Github repo, it will be listed among officially available
> > integrations on the Ignite website featuring Emmanouil as a contributor.
> >
> > What do you think?
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >
> >
> >
> > -
> > Denis
> >
> >
> > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra 
> > wrote:
> >
> > > Hi Emmanouil,
> > >
> > > The changes looks good to me. I wanted to check if you are also able to
> > > validate that the once data is added to Ignite cluster you are also
> able
> > to
> > > access it using another Ignite client or using rest endpoints.
> > >
> > > We use Teamcity for CI process, the details are as follows
> > >
> > > Trigger validation of those test suites that have been affected by your
> > > changes on TeamCity : <
> > http://204.14.53.153/
> > > >
> > >
> > >- Locate a test suite you have to check, press button named "..."
> that
> > >is located on the left of "Run" button. "Run custom build" dialog
> will
> > >appear;
> > >- Go to "Changes" tab and choose "pull//head"
> in
> > >"Build branch" dropdown list;
> > >- Press "Run build" button and monitor tests results.
> > >
> > >
> > >
> > > @Roman rsht...@yahoo.com - can you also please take a look?
> > >
> > > Regards,
> > > Saikat
> > >
> > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > gkatzio...@gmail.com>
> > > wrote:
> > >
> > > > Hi @Saikat Maitra
> > > >
> > > > I have my implementation on an Ignite fork on my personal GitHub
> > account
> > > > [1]
> > > > It is based on the Kafka implementation which is of a polling nature.
> > > > GCP does not provide a service like Kafka. Instead it provides us
> with
> > > > Pub/Sub which is just like Kafka.
> > > > Therefore using the Kafka streamer as a guideline made more sense to
> > me.
> > > >
> > > > Apart from the implementation I have also added a unit test.
> > > > It definitely needs some polishing and a double check on my side to
> > make
> > > > the review smooth.
> > > > For the tests a local Pub/Sub [2] server has to be setup however I am
> > not
> > > > sure which is the best way to do so with regards to your CI/CD.
> > > >
> > > > [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > > [2] 

Re: Adding experimental support for Intel Optane DC Persistent Memory

[Denis Magda]

Ivan,

There is no argument here, I see what you mean. All we know is that Optane
is a new class of a storage device that is slower than RAM but faster than
disk. Plus, it's denser than RAM, and non-volatile that let us store
many-more terabytes of data on a single machine that can be accessed via
the memory bus and don't worry about data preloading upon restarts. So,
there definitely will be a benefit for certain use cases once Optane is
supported natively - much more data can be stored and accessed locally at
(almost) speed of RAM.

The current integration is not finished yet (based on the review) but once
we get hardware, agree, that the benchmarks can be executed even over what
was developed so far.


-
Denis


On Tue, Oct 15, 2019 at 10:44 PM Ivan Pavlukhin  wrote:

> Denis,
>
> I suppose we are using not optimal strategy here. And motivation is simple:
> 1. Code review is "expensive", benchmarking (usually) are "cheap".
> 2. Code review is "subjective", benchmarking are "objective".
>
> And my personal opinion here is that there could be a performance win
> with Optane in proposed implementation in particular cases. It depends
> on how much slower is Optane performance comparing to RAM. But we do
> not know it.
>
> ср, 16 окт. 2019 г. в 01:10, Denis Magda :
> >
> > Alex, thanks for an extensive summary and proposing different
> > implementation trails. Presently, the JEP-352 approach seems the most
> > reasonable - JDK folks will take care of the integration and will be
> > testing/maintaining the feature going forward, while the Ignite community
> > will reuse what JDK provides. However, let's see what Eric and Mulugeta
> of
> > Intel (both copied) think about this and other approaches.
> >
> > Ivan, I'm not sure if any performance testing was done for the given
> > pull-request. Anyway, it might be premature considering Alex's summary.
> >
> > -
> > Denis
> >
> >
> > On Mon, Oct 14, 2019 at 11:50 PM Ivan Pavlukhin 
> wrote:
> >
> > > By the way, did we have a change to run some existing benchmarks
> > > against proposed implementation?
> > >
> > > пт, 11 окт. 2019 г. в 17:49, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > > >:
> > > >
> > > > Hello Mulugeta, Igniters.
> > > >
> > > > Thanks for your interest and efforts in integrating the persistent
> memory
> > > > to Ignite. Here are my thoughts on the PR:
> > > >
> > > > First of all I was questioning whether we want to use the integration
> > > with
> > > > the pmem library via JNI. Java 14 will have native support for NVME
> via
> > > > native mapped byte buffers [1] since the corresponding tickets are
> > > already
> > > > resolved [2],[3]. The suggested integration uses JNI calls quite
> often
> > > (if
> > > > I read the PR right, there will be tens of JNI calls per a single
> > > > operation), which may and most likely will undermine the benefits of
> > > using
> > > > PM. On the other hand, there is a PoC/project [4] which successfully
> > > > demonstrates how the new integration can be used to natively access
> > > > persistent memory from Java. Additionally, new Unsafe have several
> > > > performance issues - each put* methods has a map lookup, and CAS
> method
> > > > uses a global lock. Removing these issues will change the PR and
> > > > architecture dramatically.
> > > >
> > > > Other than performance and an extra library dependency, there are
> > > > additional reasons to use custom implementation of PM persistence
> > > > primitives:
> > > >
> > > >- Ignite significantly benefits from having WAL as a single point
> of
> > > >data for both failure recovery and historical rebalance. As far
> as I
> > > can
> > > >guess, the suggested library also uses some sort of journaling in
> > > order to
> > > >support crash recovery. Since we do not want give up on the
> historical
> > > >rebalance, we would need to write an additional journal, which
> then
> > > raises
> > > >a question of how to transact between this journal.
> > > >- Ignite is supposed to work with large indexes, a I not aware how
> > > LLPL
> > > >handles them. Several researches [5],[6] show that since PM
> memory has
> > > >larger access latency than regular RAM, it is beneficial to buffer
> > > index
> > > >pages in regular memory and store leaf pages in PM. There is a
> generic
> > > >approach to convert in-memory indexes to PM ones [7], I think it
> makes
> > > >sense to investigate whether our existing indexes can be
> converted to
> > > PM.
> > > >- There is an active IEP which is targeted for a file-based
> > > rebalancing.
> > > >In the suggested implementation this optimization would not work.
> A
> > > better
> > > >integration should allow to take a snapshot of a partition and
> > > transfer it
> > > >to another node for fast rebalancing.
> > > >
> > > > Overall, I think we may take the following path:
> > > >
> > > >- Target JEP352 APIs; for now keep using the whole region msync
> for
> 

Re: Binary object format KB article

[Ivan Pavlukhin]

Sergey,

Thank you for a review!

> It seems to me that document tries to focus on details of the format itself 
> but other aspects of this functionality leak into the explanation and 
> confuses reader.

You are absolutely right, it was an original idea. I will try to
define a terminology and clarify things about schemas.

ср, 16 окт. 2019 г. в 16:49, Sergey Chugunov :
>
> Then I would suggest to define good terminology at the very beginning of
> the article.
>
> Right in introduction section I see a lot of terms like "Binary object
> format", "Binary object container format" (is it the same thing?), "Binary
> serialization format". In the next section "binary type" pops up. What are
> the relations between them?
>
> Schemes part needs more examples. What is scheme? How it is related to
> binary type? Is it a one-to-one relationship? One-to-many? When a new
> scheme is created? Why type and scheme should be registered on a receiver
> side? And if the receiver exists then who is the sender?
>
> It seems to me that document tries to focus on details of the format itself
> but other aspects of this functionality leak into the explanation and
> confuses reader.
>
> On Wed, Oct 16, 2019 at 2:52 PM Ivan Pavlukhin  wrote:
>
> > Pavel, Sergey,
> >
> > Thank you for your feedback!
> >
> > To be exact the document does not describe broad picture (including
> > metadata exchange) and is not a formal format specification
> > intentionally. I wanted to create a lightweight article giving an
> > intuition about binary object structure to a reader. And yes,
> > intuition about metadata registration is definitely an important,
> > related but slightly different subject.
> >
> > ср, 16 окт. 2019 г. в 14:23, Sergey Chugunov :
> > >
> > > Ivan, thank you for documenting this functionality, agree with Pavel
> > here.
> > >
> > > I think this document is a good starting point and contains a lot of
> > > low-level details and great examples but from my perspective it doesn't
> > > show how binary objects fit into a broader picture.
> > >
> > > It worth adding higher-level details and restructure the document into a
> > > top-down article starting from where binary format is used
> > (representation
> > > of objects in cache, binary protocol for communications with thin
> > clients)
> > > and down to lower details like binary metadata exchange and serialization
> > > and container formats.
> > >
> > > Another option would be to leave the document focused on a low-level
> > > details as it is now but build around it drafts for documents describing
> > > other aspects of Binary Objects.
> > > This will make our documentation much more solid and useful for readers.
> > >
> > > On Wed, Oct 16, 2019 at 2:07 PM Pavel Tupitsyn 
> > wrote:
> > >
> > > > Ivan, great job, thanks for putting this together.
> > > >
> > > > I think we also need a more formal description of the format, including
> > > > binary metadata exchange mechanics.
> > > > It was done (partially) for IEP-9 Thin Client Protocol, we should
> > probably
> > > > copy from there:
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol#IEP-9ThinClientProtocol-BinaryObjects
> > > >
> > > >
> > > >
> > > > On Wed, Oct 16, 2019 at 11:49 AM Ivan Pavlukhin 
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > I published a document about Binary format in cwiki [1]. Please share
> > > > > your feedback. I feel that there is a lack of pictures on the page.
> > > > > Need to figure out what aspects will be more clear with pictures.
> > > > >
> > > > > [1]
> > > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Binary+object+format
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >



-- 
Best regards,
Ivan Pavlukhin

Re: Binary object format KB article

[Sergey Chugunov]

Then I would suggest to define good terminology at the very beginning of
the article.

Right in introduction section I see a lot of terms like "Binary object
format", "Binary object container format" (is it the same thing?), "Binary
serialization format". In the next section "binary type" pops up. What are
the relations between them?

Schemes part needs more examples. What is scheme? How it is related to
binary type? Is it a one-to-one relationship? One-to-many? When a new
scheme is created? Why type and scheme should be registered on a receiver
side? And if the receiver exists then who is the sender?

It seems to me that document tries to focus on details of the format itself
but other aspects of this functionality leak into the explanation and
confuses reader.

On Wed, Oct 16, 2019 at 2:52 PM Ivan Pavlukhin  wrote:

> Pavel, Sergey,
>
> Thank you for your feedback!
>
> To be exact the document does not describe broad picture (including
> metadata exchange) and is not a formal format specification
> intentionally. I wanted to create a lightweight article giving an
> intuition about binary object structure to a reader. And yes,
> intuition about metadata registration is definitely an important,
> related but slightly different subject.
>
> ср, 16 окт. 2019 г. в 14:23, Sergey Chugunov :
> >
> > Ivan, thank you for documenting this functionality, agree with Pavel
> here.
> >
> > I think this document is a good starting point and contains a lot of
> > low-level details and great examples but from my perspective it doesn't
> > show how binary objects fit into a broader picture.
> >
> > It worth adding higher-level details and restructure the document into a
> > top-down article starting from where binary format is used
> (representation
> > of objects in cache, binary protocol for communications with thin
> clients)
> > and down to lower details like binary metadata exchange and serialization
> > and container formats.
> >
> > Another option would be to leave the document focused on a low-level
> > details as it is now but build around it drafts for documents describing
> > other aspects of Binary Objects.
> > This will make our documentation much more solid and useful for readers.
> >
> > On Wed, Oct 16, 2019 at 2:07 PM Pavel Tupitsyn 
> wrote:
> >
> > > Ivan, great job, thanks for putting this together.
> > >
> > > I think we also need a more formal description of the format, including
> > > binary metadata exchange mechanics.
> > > It was done (partially) for IEP-9 Thin Client Protocol, we should
> probably
> > > copy from there:
> > >
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol#IEP-9ThinClientProtocol-BinaryObjects
> > >
> > >
> > >
> > > On Wed, Oct 16, 2019 at 11:49 AM Ivan Pavlukhin 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I published a document about Binary format in cwiki [1]. Please share
> > > > your feedback. I feel that there is a lack of pictures on the page.
> > > > Need to figure out what aspects will be more clear with pictures.
> > > >
> > > > [1]
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Binary+object+format
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[jira] [Created] (IGNITE-12297) Detect lost partitions is not happened during cluster activation

[Pavel Kovalenko (Jira)]

Pavel Kovalenko created IGNITE-12297:


 Summary: Detect lost partitions is not happened during cluster 
activation
 Key: IGNITE-12297
 URL: https://issues.apache.org/jira/browse/IGNITE-12297
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.4
Reporter: Pavel Kovalenko
 Fix For: 2.8


We invoke `detectLostPartitions` during PME only if there is a server join or 
server left.
However,  we can activate a persistent cluster where a partition may have 
MOVING status on all nodes. In this case, a partition may stay in MOVING state 
forever before any other topology event. 





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Binary object format KB article

[Ivan Pavlukhin]

Pavel, Sergey,

Thank you for your feedback!

To be exact the document does not describe broad picture (including
metadata exchange) and is not a formal format specification
intentionally. I wanted to create a lightweight article giving an
intuition about binary object structure to a reader. And yes,
intuition about metadata registration is definitely an important,
related but slightly different subject.

ср, 16 окт. 2019 г. в 14:23, Sergey Chugunov :
>
> Ivan, thank you for documenting this functionality, agree with Pavel here.
>
> I think this document is a good starting point and contains a lot of
> low-level details and great examples but from my perspective it doesn't
> show how binary objects fit into a broader picture.
>
> It worth adding higher-level details and restructure the document into a
> top-down article starting from where binary format is used (representation
> of objects in cache, binary protocol for communications with thin clients)
> and down to lower details like binary metadata exchange and serialization
> and container formats.
>
> Another option would be to leave the document focused on a low-level
> details as it is now but build around it drafts for documents describing
> other aspects of Binary Objects.
> This will make our documentation much more solid and useful for readers.
>
> On Wed, Oct 16, 2019 at 2:07 PM Pavel Tupitsyn  wrote:
>
> > Ivan, great job, thanks for putting this together.
> >
> > I think we also need a more formal description of the format, including
> > binary metadata exchange mechanics.
> > It was done (partially) for IEP-9 Thin Client Protocol, we should probably
> > copy from there:
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol#IEP-9ThinClientProtocol-BinaryObjects
> >
> >
> >
> > On Wed, Oct 16, 2019 at 11:49 AM Ivan Pavlukhin 
> > wrote:
> >
> > > Igniters,
> > >
> > > I published a document about Binary format in cwiki [1]. Please share
> > > your feedback. I feel that there is a lack of pictures on the page.
> > > Need to figure out what aspects will be more clear with pictures.
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/IGNITE/Binary+object+format
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >



-- 
Best regards,
Ivan Pavlukhin

Re: Binary object format KB article

[Sergey Chugunov]

Ivan, thank you for documenting this functionality, agree with Pavel here.

I think this document is a good starting point and contains a lot of
low-level details and great examples but from my perspective it doesn't
show how binary objects fit into a broader picture.

It worth adding higher-level details and restructure the document into a
top-down article starting from where binary format is used (representation
of objects in cache, binary protocol for communications with thin clients)
and down to lower details like binary metadata exchange and serialization
and container formats.

Another option would be to leave the document focused on a low-level
details as it is now but build around it drafts for documents describing
other aspects of Binary Objects.
This will make our documentation much more solid and useful for readers.

On Wed, Oct 16, 2019 at 2:07 PM Pavel Tupitsyn  wrote:

> Ivan, great job, thanks for putting this together.
>
> I think we also need a more formal description of the format, including
> binary metadata exchange mechanics.
> It was done (partially) for IEP-9 Thin Client Protocol, we should probably
> copy from there:
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol#IEP-9ThinClientProtocol-BinaryObjects
>
>
>
> On Wed, Oct 16, 2019 at 11:49 AM Ivan Pavlukhin 
> wrote:
>
> > Igniters,
> >
> > I published a document about Binary format in cwiki [1]. Please share
> > your feedback. I feel that there is a lack of pictures on the page.
> > Need to figure out what aspects will be more clear with pictures.
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/IGNITE/Binary+object+format
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>

Re: [DISCUSS] Pub/Sub Streamer Implementation

[Emmanouil Gkatziouras]

Hi all!

Based on the discussions most probably this is going to be on another
GitHub repo. Therefore I will prepare a standalone project with the Pub/Sub
module and once the repository is created a pull request shall be issued.
If there is anything else I can do in order to facilitate this please let
me know.

If a ticket is created for this issue, I shall be able to create a pull
request and thus a build will take place on ignite's CI.
If everyone is aligned with adding a Pub/Sub implementation may I create a
ticket on JIRA?

My next question has to do on how Ignite configures the build jobs in Team
City. The unit tests for my implementation need a local Pub/Sub server up
and running (they provide one for testing purposes). Will this take effect
in a form of a build script inside my implementation or is it something
that should be configured on Team City?

Kind regards,
Emmanouil


*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura


On Tue, 15 Oct 2019 at 23:27, Denis Magda  wrote:

> Emmanouil, Saikat,
>
> After contemplating on this, let me propose another way with this
> particular integration that is aligned with our modularization endeavor.
> The modularization [1] defines the Ignite core, Ignite modules, and
> independent integrations. The first two are to be located in Ignite
> repositories and to be tested/updated by the community for every release.
> The modules are a subset of existing integrations that are adopted widely
> and require community attention.
>
> As for the independent integrations [1], those are supposed to be located
> in a separate GitHub repo. I would advise us to start implementing this
> concept with the Pub/Sub integration. Even though it will be located in a
> separate Github repo, it will be listed among officially available
> integrations on the Ignite website featuring Emmanouil as a contributor.
>
> What do you think?
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>
>
>
> -
> Denis
>
>
> On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra 
> wrote:
>
> > Hi Emmanouil,
> >
> > The changes looks good to me. I wanted to check if you are also able to
> > validate that the once data is added to Ignite cluster you are also able
> to
> > access it using another Ignite client or using rest endpoints.
> >
> > We use Teamcity for CI process, the details are as follows
> >
> > Trigger validation of those test suites that have been affected by your
> > changes on TeamCity : <
> http://204.14.53.153/
> > >
> >
> >- Locate a test suite you have to check, press button named "..." that
> >is located on the left of "Run" button. "Run custom build" dialog will
> >appear;
> >- Go to "Changes" tab and choose "pull//head" in
> >"Build branch" dropdown list;
> >- Press "Run build" button and monitor tests results.
> >
> >
> >
> > @Roman rsht...@yahoo.com - can you also please take a look?
> >
> > Regards,
> > Saikat
> >
> > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > gkatzio...@gmail.com>
> > wrote:
> >
> > > Hi @Saikat Maitra
> > >
> > > I have my implementation on an Ignite fork on my personal GitHub
> account
> > > [1]
> > > It is based on the Kafka implementation which is of a polling nature.
> > > GCP does not provide a service like Kafka. Instead it provides us with
> > > Pub/Sub which is just like Kafka.
> > > Therefore using the Kafka streamer as a guideline made more sense to
> me.
> > >
> > > Apart from the implementation I have also added a unit test.
> > > It definitely needs some polishing and a double check on my side to
> make
> > > the review smooth.
> > > For the tests a local Pub/Sub [2] server has to be setup however I am
> not
> > > sure which is the best way to do so with regards to your CI/CD.
> > >
> > > [1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra 
> > > wrote:
> > >
> > > > Hello Emmanouil,
> > > >
> > > > Can you please share more info about the implementation?
> > > >
> > > > We have similar implementation for data streamers with Kafka and
> > RocketMQ
> > > >
> > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > >
> > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > >
> > > > Please review and share if you have any questions.
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > >
> > > >
> > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> > > > gkatzio...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > By mentioning Pub/Sub I mean Google Cloud's 

Re: Binary object format KB article

[Pavel Tupitsyn]

Ivan, great job, thanks for putting this together.

I think we also need a more formal description of the format, including
binary metadata exchange mechanics.
It was done (partially) for IEP-9 Thin Client Protocol, we should probably
copy from there:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol#IEP-9ThinClientProtocol-BinaryObjects



On Wed, Oct 16, 2019 at 11:49 AM Ivan Pavlukhin  wrote:

> Igniters,
>
> I published a document about Binary format in cwiki [1]. Please share
> your feedback. I feel that there is a lack of pictures on the page.
> Need to figure out what aspects will be more clear with pictures.
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Binary+object+format
>
> --
> Best regards,
> Ivan Pavlukhin
>

[jira] [Created] (IGNITE-12296) Implement Log Throttle in .Net and print warning for unsafe PutAll ops.

[Ilya Kasnacheev (Jira)]

Ilya Kasnacheev created IGNITE-12296:


 Summary: Implement Log Throttle in .Net and print warning for 
unsafe PutAll ops.
 Key: IGNITE-12296
 URL: https://issues.apache.org/jira/browse/IGNITE-12296
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Ilya Kasnacheev
Assignee: Pavel Tupitsyn


Right now it's not possible to do one-time developer warnings in .Net as there 
is no Log Throttle. Please implement one. Please also issue warning if 
random-order collection is passed to PutAll, InvokeAll, RemoveAll, as it is 
prime source for deadlocks and developer frustration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Best Effort Affinity for thin clients

[Pavel Tupitsyn]

Did a partial review, left one comment on the PR.
We need another pair of eyes on this for sure.

On Wed, Oct 16, 2019 at 10:34 AM Alex Plehanov 
wrote:

> Hello guys,
>
>
> I've implemented affinity awareness support for java thin client [1]. There
> is only client-side affected by the patch. Can anyone review the change?
>
>
> 1: https://issues.apache.org/jira/browse/IGNITE-11898
>
>
>
> ср, 13 мар. 2019 г. в 22:54, Pavel Tupitsyn :
>
> > Default value for boolean is false, and I though we'll have the feature
> > enabled by default.
> > But I agree with you. Let's disable by default and name the config
> property
> > EnableAffinityAwareness.
> >
> > On Wed, Mar 13, 2019 at 12:52 PM Igor Sapego  wrote:
> >
> > > For the "false" I mean "disable" here.
> > >
> > > BTW, I believe we should name this parameter in a positive way:
> > > EnableAffinityAwareness, not disable.
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Wed, Mar 13, 2019 at 12:50 PM Igor Sapego 
> wrote:
> > >
> > > > Well, yes, this looks like a simplest solution. Let's implement it
> for
> > > the
> > > > beginning and set this feature to "false" by default, as this feature
> > > looks
> > > > complex, probably error-prone, and should be considered in a "beta"
> > > > state for the first release.
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > > >
> > > > On Mon, Mar 11, 2019 at 8:04 PM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > >> My suggestion is a boolean flag in client configuration:
> > > >> DisableAffinityAwareness
> > > >> And use old random/round-robin behavior with only one active
> > connection.
> > > >>
> > > >> On Mon, Mar 11, 2019 at 1:36 PM Igor Sapego 
> > wrote:
> > > >>
> > > >> > Pavel,
> > > >> >
> > > >> > That's right. Do you have other suggestions or objections?
> > > >> >
> > > >> > Best Regards,
> > > >> > Igor
> > > >> >
> > > >> >
> > > >> > On Fri, Mar 8, 2019 at 11:37 AM Pavel Tupitsyn <
> > ptupit...@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> > > >  maxConnectionNumber parameter
> > > >> > > What's the idea? Follow the Best Effor Affinity logic, but
> > establish
> > > >> up
> > > >> > to
> > > >> > > N connections?
> > > >> > >
> > > >> > > On Thu, Mar 7, 2019 at 1:23 PM Igor Sapego 
> > > >> wrote:
> > > >> > >
> > > >> > > > I can propose two improvements here:
> > > >> > > >
> > > >> > > > 1. A simple one. Lets introduce maxConnectionNumber parameter
> > > >> > > > in ClientConfiguration. As it is easy to implement it may be
> > > >> introduced
> > > >> > > > together with the new feature to give user an additional
> > control.
> > > >> > > >
> > > >> > > > 2. Asynchronous connection establishment. In this case startup
> > > >> method
> > > >> > > > of a client returns control to user once it have established
> at
> > > >> least
> > > >> > one
> > > >> > > > connection. Other connections established in background by a
> > > >> separate
> > > >> > > > thread. This one is harder to implement and maybe it makes
> sense
> > > to
> > > >> add
> > > >> > > > it as a separate feature.
> > > >> > > >
> > > >> > > > Best Regards,
> > > >> > > > Igor
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Mar 6, 2019 at 9:43 PM Pavel Tupitsyn <
> > > ptupit...@apache.org
> > > >> >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi,
> > > >> > > > >
> > > >> > > > > I'm in progress of implementing this IEP for Ignite.NET,
> and I
> > > >> have
> > > >> > > > > concerns about the following:
> > > >> > > > >
> > > >> > > > > > On thin client startup it connects to all nodes provided
> by
> > > >> client
> > > >> > > > > configuration
> > > >> > > > >
> > > >> > > > > Should we, at least, make this behavior optional?
> > > >> > > > >
> > > >> > > > > One of the benefits of thin client is quick startup/connect
> > time
> > > >> and
> > > >> > > low
> > > >> > > > > resource usage.
> > > >> > > > > Adding "connect all" behavior can negate those benefits,
> > > >> especially
> > > >> > on
> > > >> > > > > large clusters.
> > > >> > > > >
> > > >> > > > > Thoughts?
> > > >> > > > >
> > > >> > > > > On Thu, Feb 14, 2019 at 5:39 PM Igor Sapego <
> > isap...@apache.org
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Guys, I've updated the IEP page [1] once again.
> > > >> > > > > >
> > > >> > > > > > Please, pay attention to sections Cache affinity mapping
> > > >> acquiring
> > > >> > > > > > (4.a, format of Cache Partitions Request) and Changes to
> > cache
> > > >> > > > > > operations with single key (3 and 4, algorithm).
> > > >> > > > > >
> > > >> > > > > > Long story short, I've decided to add some additional data
> > to
> > > >> Cache
> > > >> > > > > > Partitions Response, so that client can determine how to
> > > >> calculate
> > > >> > > > > > partition for a given key properly.
> > > >> > > > > >
> > > >> > > > > > [1] -
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> 

[jira] [Created] (IGNITE-12295) Faster index eviction

[Sergey Kalashnikov (Jira)]

Sergey Kalashnikov created IGNITE-12295:
---

 Summary: Faster index eviction
 Key: IGNITE-12295
 URL: https://issues.apache.org/jira/browse/IGNITE-12295
 Project: Ignite
  Issue Type: Sub-task
Reporter: Sergey Kalashnikov
Assignee: Sergey Kalashnikov


For the file-based rebalancing approach, it seems feasible to avoid iterating 
the old partition data in order to clear the indexes.
One can independently clear the shared index structures of all the rows 
referencing entries from moving partitions by deducing partition id from the 
links in the leaf pages.

The proposed algorithm is simple and takes the set of integer partition ids as 
an input:
1. Iterate over leaf pages of the index and remove items attributed to any of 
indicated partitions, unless it is the only or the rightmost item on a page.
2. If the rightmost item (or the only item) on a page happens to belong to any 
of the indicated partitions, employ a regular remove algorithm (descending from 
the root) so that inner pages get correctly updated.
Restart iteration from the leaf page where the removed item would be inserted 
(descend from the root to find it).

The use of such algorithm can be justified (as having performance advantage) 
when the number of keys that'd be removed is bigger than the number of leaf 
pages in the index.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Binary object format KB article

[Ivan Pavlukhin]

Igniters,

I published a document about Binary format in cwiki [1]. Please share
your feedback. I feel that there is a lack of pictures on the page.
Need to figure out what aspects will be more clear with pictures.

[1] https://cwiki.apache.org/confluence/display/IGNITE/Binary+object+format

-- 
Best regards,
Ivan Pavlukhin

[jira] [Created] (IGNITE-12294) .NET Inspections failure on TC

[Pavel Tupitsyn (Jira)]

Pavel Tupitsyn created IGNITE-12294:
---

 Summary: .NET Inspections failure on TC
 Key: IGNITE-12294
 URL: https://issues.apache.org/jira/browse/IGNITE-12294
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.8
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.8


New R# version detected a new problem:

{code}
BinaryUtils.cs (1)
625:  Actual shift count equals zero. Left operand will remain unchanged
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Best Effort Affinity for thin clients

[Alex Plehanov]

Hello guys,


I've implemented affinity awareness support for java thin client [1]. There
is only client-side affected by the patch. Can anyone review the change?


1: https://issues.apache.org/jira/browse/IGNITE-11898



ср, 13 мар. 2019 г. в 22:54, Pavel Tupitsyn :

> Default value for boolean is false, and I though we'll have the feature
> enabled by default.
> But I agree with you. Let's disable by default and name the config property
> EnableAffinityAwareness.
>
> On Wed, Mar 13, 2019 at 12:52 PM Igor Sapego  wrote:
>
> > For the "false" I mean "disable" here.
> >
> > BTW, I believe we should name this parameter in a positive way:
> > EnableAffinityAwareness, not disable.
> >
> > Best Regards,
> > Igor
> >
> >
> > On Wed, Mar 13, 2019 at 12:50 PM Igor Sapego  wrote:
> >
> > > Well, yes, this looks like a simplest solution. Let's implement it for
> > the
> > > beginning and set this feature to "false" by default, as this feature
> > looks
> > > complex, probably error-prone, and should be considered in a "beta"
> > > state for the first release.
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Mon, Mar 11, 2019 at 8:04 PM Pavel Tupitsyn 
> > > wrote:
> > >
> > >> My suggestion is a boolean flag in client configuration:
> > >> DisableAffinityAwareness
> > >> And use old random/round-robin behavior with only one active
> connection.
> > >>
> > >> On Mon, Mar 11, 2019 at 1:36 PM Igor Sapego 
> wrote:
> > >>
> > >> > Pavel,
> > >> >
> > >> > That's right. Do you have other suggestions or objections?
> > >> >
> > >> > Best Regards,
> > >> > Igor
> > >> >
> > >> >
> > >> > On Fri, Mar 8, 2019 at 11:37 AM Pavel Tupitsyn <
> ptupit...@apache.org>
> > >> > wrote:
> > >> >
> > >> > > >  maxConnectionNumber parameter
> > >> > > What's the idea? Follow the Best Effor Affinity logic, but
> establish
> > >> up
> > >> > to
> > >> > > N connections?
> > >> > >
> > >> > > On Thu, Mar 7, 2019 at 1:23 PM Igor Sapego 
> > >> wrote:
> > >> > >
> > >> > > > I can propose two improvements here:
> > >> > > >
> > >> > > > 1. A simple one. Lets introduce maxConnectionNumber parameter
> > >> > > > in ClientConfiguration. As it is easy to implement it may be
> > >> introduced
> > >> > > > together with the new feature to give user an additional
> control.
> > >> > > >
> > >> > > > 2. Asynchronous connection establishment. In this case startup
> > >> method
> > >> > > > of a client returns control to user once it have established at
> > >> least
> > >> > one
> > >> > > > connection. Other connections established in background by a
> > >> separate
> > >> > > > thread. This one is harder to implement and maybe it makes sense
> > to
> > >> add
> > >> > > > it as a separate feature.
> > >> > > >
> > >> > > > Best Regards,
> > >> > > > Igor
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Mar 6, 2019 at 9:43 PM Pavel Tupitsyn <
> > ptupit...@apache.org
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > I'm in progress of implementing this IEP for Ignite.NET, and I
> > >> have
> > >> > > > > concerns about the following:
> > >> > > > >
> > >> > > > > > On thin client startup it connects to all nodes provided by
> > >> client
> > >> > > > > configuration
> > >> > > > >
> > >> > > > > Should we, at least, make this behavior optional?
> > >> > > > >
> > >> > > > > One of the benefits of thin client is quick startup/connect
> time
> > >> and
> > >> > > low
> > >> > > > > resource usage.
> > >> > > > > Adding "connect all" behavior can negate those benefits,
> > >> especially
> > >> > on
> > >> > > > > large clusters.
> > >> > > > >
> > >> > > > > Thoughts?
> > >> > > > >
> > >> > > > > On Thu, Feb 14, 2019 at 5:39 PM Igor Sapego <
> isap...@apache.org
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Guys, I've updated the IEP page [1] once again.
> > >> > > > > >
> > >> > > > > > Please, pay attention to sections Cache affinity mapping
> > >> acquiring
> > >> > > > > > (4.a, format of Cache Partitions Request) and Changes to
> cache
> > >> > > > > > operations with single key (3 and 4, algorithm).
> > >> > > > > >
> > >> > > > > > Long story short, I've decided to add some additional data
> to
> > >> Cache
> > >> > > > > > Partitions Response, so that client can determine how to
> > >> calculate
> > >> > > > > > partition for a given key properly.
> > >> > > > > >
> > >> > > > > > [1] -
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > >> > > > > >
> > >> > > > > > Best Regards,
> > >> > > > > > Igor
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Mon, Feb 4, 2019 at 8:24 PM Pavel Tupitsyn <
> > >> > ptupit...@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Looks good to me.
> > >> > > > > > >
> > >> > > > > > > On Mon, Feb 4, 2019 at 6:30 PM Igor Sapego <
> > >> isap...@apache.org>
> > >> > > > wrote:
> > >> >