Re: Add cluster (de)activation events IGNITE-8376

[Ken Cheng]

I checked the code again, seems there is no such event.


Thanks,
Ken Cheng


On Tue, Jul 3, 2018 at 7:41 AM Dmitriy Setrakyan 
wrote:

> Do we really not have events in Ignite for cluster activation?
>
> Alexey Goncharuk, can you please comment?
>
> D.
>
> On Mon, Jul 2, 2018 at 1:34 AM, kcheng.mvp  wrote:
>
> > Dear igniters,I am going to pick up
> > https://issues.apache.org/jira/browse/IGNITE-8376, and did some research
> > based on the comments.based on my understanding, if we want to know this
> > action is triggered by which way(rest, mbean, auto or visocmd)  then we
> > need
> > to change the core method's signature.I am not sure my understanding is
> > correct or not. Can anybody help to clarify this? Thank you very much.By
> > the
> > way, I commented this in jira as well.Thanks youkcvmp
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: A quick question on cluster topology and partitions?

[John Wilson]

Regarding 1. I'm referring to the documentation here,
https://apacheignite.readme.io/docs/topology-validation, which states
"*Topology
validator is used to verify that cluster topology is valid for further
cache operations*." and "*If topology validator is not configured, then the
cluster topology is always considered to be valid.*"

Thanks,

On Mon, Jul 2, 2018 at 4:21 PM, Dmitriy Setrakyan 
wrote:

> On Mon, Jul 2, 2018 at 4:03 PM, John Wilson 
> wrote:
>
> > Hi,
> >
> >
> >1. What exactly is a cluster topology? What makes a cluster topology
> >invalid for further cache operations?
> >
>
> Cluster topology is a set of Ignite nodes in the cluster. I do not think a
> cluster topology could be invalid on its own. Perhaps you are asking about
> a situation when after a certain number of node failures/stops we can be in
> a situation where all primary and backup copies become inaccessible. In
> that case, the cluster should enter a read-only state for the lost
> partitions.
>
>
> >2.  Why do we have the concept of partitions in Ignite? Why don't we
> >have a key-to-node mapping rather than a key-to-partition and a
> >partition-to-node mapping?
> >
>
> Main reason is because there is a finite number of partitions and there is
> an infinite number of keys. Whenever ignite topology changes, Ignite must
> rebalance data to the new nodes (or to the existing nodes). In this case,
> Ignite needs to know when a certain partition is moved to another node. If
> there were no partitions, then it would be impossible to tell when to
> finish the rebalancing process.
>
> D.
>

Re: Add cluster (de)activation events IGNITE-8376

[Dmitriy Setrakyan]

Do we really not have events in Ignite for cluster activation?

Alexey Goncharuk, can you please comment?

D.

On Mon, Jul 2, 2018 at 1:34 AM, kcheng.mvp  wrote:

> Dear igniters,I am going to pick up
> https://issues.apache.org/jira/browse/IGNITE-8376, and did some research
> based on the comments.based on my understanding, if we want to know this
> action is triggered by which way(rest, mbean, auto or visocmd)  then we
> need
> to change the core method's signature.I am not sure my understanding is
> correct or not. Can anybody help to clarify this? Thank you very much.By
> the
> way, I commented this in jira as well.Thanks youkcvmp
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: A quick question on cluster topology and partitions?

[Dmitriy Setrakyan]

On Mon, Jul 2, 2018 at 4:03 PM, John Wilson  wrote:

> Hi,
>
>
>1. What exactly is a cluster topology? What makes a cluster topology
>invalid for further cache operations?
>

Cluster topology is a set of Ignite nodes in the cluster. I do not think a
cluster topology could be invalid on its own. Perhaps you are asking about
a situation when after a certain number of node failures/stops we can be in
a situation where all primary and backup copies become inaccessible. In
that case, the cluster should enter a read-only state for the lost
partitions.


>2.  Why do we have the concept of partitions in Ignite? Why don't we
>have a key-to-node mapping rather than a key-to-partition and a
>partition-to-node mapping?
>

Main reason is because there is a finite number of partitions and there is
an infinite number of keys. Whenever ignite topology changes, Ignite must
rebalance data to the new nodes (or to the existing nodes). In this case,
Ignite needs to know when a certain partition is moved to another node. If
there were no partitions, then it would be impossible to tell when to
finish the rebalancing process.

D.

A quick question on cluster topology and partitions?

[John Wilson]

Hi,


   1. What exactly is a cluster topology? What makes a cluster topology
   invalid for further cache operations?
   2.  Why do we have the concept of partitions in Ignite? Why don't we
   have a key-to-node mapping rather than a key-to-partition and a
   partition-to-node mapping?

Thanks,
John

Re: NodeJS client: npm publishing instruction

[Denis Magda]

Alexey,

Thanks for the instructions. Please put them on Ignite wiki as a sub-page
of Development Process:
https://cwiki.apache.org/confluence/display/IGNITE/Development+Process

Tell me your Wiki ID and I'll grant required permissions.

--
Denis

On Mon, Jul 2, 2018 at 1:35 AM Igor Sapego  wrote:

> I believe, we need to put this instruction on wiki page.
>
> Best Regards,
> Igor
>
>
> On Sun, Jul 1, 2018 at 3:58 PM Alexey Kosenchuk <
> alexey.kosenc...@nobitlost.com> wrote:
>
> > Denis,
> >
> > below is instruction how to publish/update npm module with Ignite NodeJS
> > client on npmjs.
> > Use it when you decide to publish the client.
> >
> > Another short instruction is how to generate the API spec using jsdoc.
> >
> > Pls place the instructions to appropriate locations.
> >
> > Thanks,
> > -Alexey
> >
> >
> > How to publish Ignite NodeJS Client on npmjs.com:
> > -
> > 1. Install NodeJS npm (https://nodejs.org/en/), if not installed yet.
> > 2. Register an account at npmjs (https://www.npmjs.com/signup), if not
> > registered yet.
> > 3. Execute `npm login` command and provide your npmjs account
> credentials.
> > 4. Clone or download Ignite repository
> > https://github.com/apache/ignite.git to `local_ignite_path`
> > 5. Go to `local_ignite_path/modules/platforms/nodejs`
> > 6. Prepare/update
> > `local_ignite_path/modules/platforms/nodejs/package.json` file. Pay
> > attention to:
> >- "name" - name of the npm module
> >- "version" - version of the npm module, increment it if you update
> > the module
> >- "description" - description of the npm module
> >- "repository" - type and link to the repository with the source code
> >- "keywords" - keywords for the search of the module on npmjs
> >- "license" - license type
> >- other properties depend on the implementation/tests, do not touch
> them
> > Example of the package.json file is attached.
> > 7. Prepare/update `local_ignite_path/modules/platforms/nodejs/README.md`
> > file. It should exist and should not be empty. Eg. add a link to a place
> > with the documentation.
> > 8. Execute `npm publish` command from the
> > `local_ignite_path/modules/platforms/nodejs` folder.
> > 9. Check the module is published and well-described at
> > https://www.npmjs.com/package/apache-ignite-client (assuming
> > `apache-ignite-client` is the name of the module)
> >
> > Common instruction about npm publishing:
> > https://docs.npmjs.com/getting-started/publishing-npm-packages
> >
> > How to generate Ignite NodeJS Client API specification:
> > ---
> > It should be done if a public API class/method has been changed.
> > 1. Execute `npm install -g jsdoc` to install jsdoc
> > (https://www.npmjs.com/package/jsdoc)
> > 2. Clone or download Ignite repository
> > https://github.com/apache/ignite.git to `local_ignite_path`
> > 3. Go to `local_ignite_path/modules/platforms/nodejs/api_spec`
> > 4. Only if a class has been removed from the public API, remove all
> > files from `local_ignite_path/modules/platforms/nodejs/api_spec` except
> > conf.json file.
> > 5. Execute `jsdoc -c conf.json` command.
> >
> > Note: `local_ignite_path/modules/platforms/nodejs/api_spec/conf.json` is
> > a file with jsdoc configuration.
> >
>

Re: Ignite as distributed file storage

[Dmitriy Setrakyan]

To be honest, I am not sure if we need to kick off another file system
storage discussion in Ignite. It sounds like a huge effort and likely will
not be productive.

However, I think an ability to store large objects will make sense. For
example, how do I store a 10GB blob in Ignite cache? Most likely we have to
have a separate memory or disk space, allocated for blobs only. We also
need to be able to efficiently transfer a 10GB Blob object over the network
and store it off-heap right away, without bringing it into main heap memory
(otherwise we would run out of memory).

I suggest that we create an IEP about this use case alone and leave the
file system for the future discussions.

D.

On Mon, Jul 2, 2018 at 6:50 AM, Vladimir Ozerov 
wrote:

> Pavel,
>
> Thank you. I'll wait for feature comparison and concrete use cases, because
> for me this feature still sounds too abstract to judge whether product
> would benefit from it.
>
> On Mon, Jul 2, 2018 at 3:15 PM Pavel Kovalenko  wrote:
>
> > Dmitriy,
> >
> > I think we have a little miscommunication here. Of course, I meant
> > supporting large entries / chunks of binary data. Internally it will be
> > BLOB storage, which can be accessed through various interfaces.
> > "File" is just an abstraction for an end user for convenience, a wrapper
> > layer to have user-friendly API to directly store BLOBs. We shouldn't
> > support full file protocol support with file system capabilities. It can
> be
> > added later, but now it's absolutely unnecessary and introduces extra
> > complexity.
> >
> > We can implement our BLOB storage step by step. The first thing is
> > core functionality and support to save large parts of binary objects to
> it.
> > "File" layer, Web layer, etc. can be added later.
> >
> > The initial IGFS design doesn't have good capabilities to have a
> > persistence layer. I think we shouldn't do any changes to it, this
> project
> > as for me is almost outdated. We will drop IGFS after implementing File
> > System layer over our BLOB storage.
> >
> > Vladimir,
> >
> > I will prepare a comparison with other existing distributed file storages
> > and file systems in a few days.
> >
> > About usage data grid, I never said, that we need transactions, sync
> backup
> > and etc. We need just a few core things - Atomic cache with persistence,
> > Discovery, Baseline, Affinity, and Communication.
> > Other things we can implement by ourselves. So this feature can develop
> > independently of other non-core features.
> > For me Ignite way is providing to our users a fast and convenient way to
> > solve their problems with good performance and durability. We have the
> > problem with storing large data, we should solve it.
> > About other things see my message to Dmitriy above.
> >
> > вс, 1 июл. 2018 г. в 9:48, Dmitriy Setrakyan :
> >
> > > Pavel,
> > >
> > > I have actually misunderstood the use case. To be honest, I thought
> that
> > > you were talking about the support of large values in Ignite caches,
> e.g.
> > > objects that are several megabytes in cache.
> > >
> > > If we are tackling the distributed file system, then in my view, we
> > should
> > > be talking about IGFS and adding persistence support to IGFS (which is
> > > based on HDFS API). It is not clear to me that you are talking about
> > IGFS.
> > > Can you confirm?
> > >
> > > D.
> > >
> > >
> > > On Sat, Jun 30, 2018 at 10:59 AM, Pavel Kovalenko 
> > > wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > Yes, I have approximate design in my mind. The main idea is that we
> > > already
> > > > have distributed cache for files metadata (our Atomic cache), the
> data
> > > flow
> > > > and distribution will be controlled by our AffinityFunction and
> > Baseline.
> > > > We're already have discovery and communication to make such local
> files
> > > > storages to be synced. The files data will be separated to large
> blocks
> > > > (64-128Mb) (which looks very similar to our WAL). Each block can
> > contain
> > > > one or more file chunks. The tablespace (segment ids, offsets and
> etc.)
> > > > will be stored to our regular page memory. This is key ideas to
> > implement
> > > > first version of such storage. We already have similiar components in
> > our
> > > > persistence, so this experience can be reused to develop such
> storage.
> > > >
> > > > Denis,
> > > >
> > > > Nothing significant should be changed at our memory level. It will be
> > > > separate, pluggable component over cache. Most of the functions which
> > > give
> > > > performance boost can be delegated to OS level (Memory mapped files,
> > DMA,
> > > > Direct write from Socket to disk and vice versa). Ignite and File
> > Storage
> > > > can develop independetly of each other.
> > > >
> > > > Alexey Stelmak, which has a great experience with developing such
> > systems
> > > > can provide more low level information about how it should look.
> > > >
> > > > сб, 30 июн. 2018 г. в 19:40, Dmitriy Setrakyan <
> dsetrak...@apache.org

[GitHub] ignite pull request #4292: Ignite 2.4.7.b1 no 8900

[mcherkasov]

GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/4292

Ignite 2.4.7.b1 no 8900



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.7.b1-no-8900

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4292


commit debc906a25d3e2d65db58e16307fae6f08460eeb
Author: devozerov 
Date:   2018-02-27T09:13:52Z

Revert "IGNITE-7253: JDBC thin streaming: fixed default local buffer size, 
improved error messages in case of unsupported SQL statements."

This reverts commit d53504adb1d4276f79ede2401e2d1512fe0287ec.

commit d8cf9e042c0e9afe65508c006d9d1c39779d78b9
Author: devozerov 
Date:   2018-02-27T09:14:21Z

Revert "IGNITE-7253: Streaming mode for JDBC thin driver. This closes 
#3499."

This reverts commit bc331f9de716c30d6f733e28821ab44da7ed0cf7.

commit 975a7cdb68cb89da74ec78c3cf23f644050918ff
Author: devozerov 
Date:   2018-02-27T09:49:44Z

Merge branch 'ignite-2.4' into ignite-2.4.3

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java
#   parent/pom.xml

commit 74a54d23913bd7195c525d8e222b4e4047515843
Author: Ilya Lantukh 
Date:   2018-02-28T07:08:54Z

IGNITE-7836 Fixed handling of state change message when forceReassignment 
is false

commit 2a70ede048f59753061973495f83927f47452d66
Author: Andrey Novikov 
Date:   2018-01-19T03:05:03Z

IGNITE-6920 Web Console: Create default account for direct-install package.

(cherry picked from commit e5005d9)

commit 2f1ea2cdb76863008d4514f26845457bdeb7d6ed
Author: Andrey Novikov 
Date:   2018-01-19T04:35:30Z

IGNITE-6920 Minor fix.

(cherry picked from commit 9cc7cbf)

commit 62652f3fb1563ba149dcbccb80928d50b822ff36
Author: alexdel 
Date:   2018-01-25T08:49:28Z

IGNITE-7064 Web Console: Implemented basic E2E tests.

(cherry picked from commit ce96e4f)

commit 06e891f1161af598e0aa4665f7a6047637d1c476
Author: Dmitriy Shabalin 
Date:   2018-01-25T09:51:44Z

IGNITE-7522 Web Console: Fixed cluster selector state after cluster restart.

(cherry picked from commit ac3cfb8)

commit 291cb2c88118ccffebcf3383db629647faec1eee
Author: Dmitriy Shabalin 
Date:   2018-01-25T10:33:13Z

IGNITE-7529 Web Console: Refactor UIGrid column filters.

(cherry picked from commit 08658ea)

commit 443eafc718685e66c9a60058bd0ab56d88f9f0a6
Author: alexdel 
Date:   2018-01-25T14:38:36Z

IGNITE-7064 Web Console: Minor test fix.

(cherry picked from commit 4b6d9ad)

commit 6f1df5c40100363b5922734223a774ff1d6a008e
Author: Ilya Borisov 
Date:   2018-01-26T09:07:47Z

IGNITE-7031 Web Console: Refactored confirmation cancellation logic.

(cherry picked from commit 92ae3fe)

commit 16982825fb06ff2724ba4583781cc15443c4f46d
Author: Alexey Kuznetsov 
Date:   2018-02-02T10:07:57Z

IGNITE-7610 Web Console: Profile page refactored to component.

(cherry picked from commit 958975e)

commit 9c6a52250e058c4546aef0d0ecee977c07076a1a
Author: Alexey Kuznetsov 
Date:   2018-02-02T10:09:37Z

IGNITE-7612 Web Console: Refactored mongoose schemas to separate module.

(cherry picked from commit 71db707)

commit 89e15830dedcb46f24d9cc9b922ba3b013331a18
Author: Vasiliy Sisko 
Date:   2018-02-12T10:22:10Z

Web Agent: Fixed wrong config of IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE in 
demo startup.

(cherry picked from commit 1a6e544)

commit 18966673570425192e1b89fbb2c63d164b47eaca
Author: Vasiliy Sisko 
Date:   2018-02-12T13:24:30Z

IGNITE-7578 Actualized client connector configuration.

(cherry picked from commit 819d746)

commit 237063efa35c54bb9e9800ecf63ea223ec20a9ef
Author: alexdel 
Date:   2018-02-19T04:25:24Z

IGNITE-7650 Extracted signin/signup form to separate page, improved landing 
page.

(cherry picked from commit 1925674)

commit 679aeca7a3ff60a9dd478966d3949107d302d5db
Author: Andrey Novikov 
Date:   2018-02-19T07:56:07Z

IGNITE-7650 Fixed headers.

(cherry picked from commit 67922b3)

commit 5d5a6a05ec49f895e8a5edd505e496dcfe58a208
Author: Vasiliy Sisko 
Date:   2018-02-21T04:21:02Z

IGNITE-6094 Web Agent: Enabled persistent in demo mode.

(cherry picked from commit 3c35900)

commit e35d8cfb06f52765959fc2e1883bf70fe0259f45
Author: Alexander Kalinin 
Date:   2018-02-21T07:03:20Z

IGNITE-7320 Web Console - Fixed table headers for Safari.

(cherry picked from commit 04a025b)

commit 20cb1439f48b9a3c985f65784af36ef2c6f45e4f
Author: Andrey Novikov 
Date:   2018-02-22T02:54:05Z

IGNITE-7650 Fixed counties codes.

(cherry picked from commit fc40261)

[GitHub] ignite pull request #4291: Ignite-8911

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/4291

Ignite-8911



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8911

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4291


commit 7bf8bba177daddac672bdcd4a4839a7a3b1e4f39
Author: Alexey Stelmak 
Date:   2018-06-19T12:08:20Z

Update locks

commit ab15222118b1024c09fd53376af6be9b2a4ed3ee
Author: Alexey Stelmak 
Date:   2018-06-19T12:37:43Z

Update locks

commit 5b3aaf37ba338494425b0ac1742f4b55ccbbd061
Author: Alexey Stelmak 
Date:   2018-06-20T11:44:30Z

ignite-8831 Update locks

commit 65022505aa22d28b7467380025ed034a0bcdfe6b
Author: Eduard Shangareev 
Date:   2018-06-21T22:30:31Z

Merge remote-tracking branch 'origin/ignite-8831' into ignite-gg-13873-2

commit 19bae0a6cfcad91317a98c81ced44625d16b58e1
Author: EdShangGG 
Date:   2018-06-27T19:18:03Z

GG-13772 NPE during simultaneous cache access and active snapshot RESTORE 
operation.

Signed-off-by: EdShangGG 

commit 09b65743cdff67a82b207992c5a711bd630d9775
Author: EdShangGG 
Date:   2018-06-29T16:37:15Z

GG-13772 NPE during simultaneous cache access and active snapshot RESTORE 
operation.

Signed-off-by: EdShangGG 

commit 210b55d1a527e485da4e6a7031883f03ad62dc3b
Author: EdShangGG 
Date:   2018-07-02T17:09:17Z

GG-13772 NPE during simultaneous cache access and active snapshot RESTORE 
operation.

Signed-off-by: EdShangGG 

commit f9f433ee12e6c19043a0d9b48a665359acdffefc
Author: EdShangGG 
Date:   2018-07-02T18:41:35Z

Merge branch 'ignite-gg-13873-2' into ignite-gg-13772




---

[jira] [Created] (IGNITE-8911) While cache is restarting it's possible to start new cache with this name

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-8911:
-

 Summary: While cache is restarting it's possible to start new 
cache with this name
 Key: IGNITE-8911
 URL: https://issues.apache.org/jira/browse/IGNITE-8911
 Project: Ignite
  Issue Type: Bug
Reporter: Eduard Shangareev


We have the state "restarting" for caches when we certainly now that these 
caches would start at some moment in future. But we could start new cache with 
the same name.

Plus, NPE is throwing when we try to get proxy for such caches (in "restarting" 
state).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Automatic Handling of Long Stop-the-World Pauses

[Denis Magda]

Pavel,

We already can monitor the state of individual nodes and show it through
metrics. Now I'd like to see how to go further and automate a decision on
if a node should be kicked off from the cluster or not.

--
Denis

On Mon, Jul 2, 2018 at 12:28 PM Pavel Kovalenko  wrote:

> Denis,
>
> I think, JVM can't easily help to itself if it's in SW pause. Most
> solutions what I saw about handling such situations are checking heartbeats
> on other nodes or run in parallel supervisor process which can detect that
> JVM with Ignite in SW.
>
> 2018-07-02 20:54 GMT+03:00 Denis Magda :
>
> > Igniters,
> >
> > Pulling this discussion up. Any thoughts?
> >
> > --
> > Denis
> >
> > On Thu, Jun 21, 2018 at 3:52 PM Denis Magda  wrote:
> >
> > > Igniters,
> > >
> > > It's a pleasure to see how our project is evolving in a directing of
> > being
> > > a self-healing solution:
> > >
> > >- Ignite can already handle critical failures such as OOM, File I/O
> > >issues, etc. [1]
> > >- There is an endeavor to fix cluster lock-ins due to partition map
> > >exchange issues. [2]
> > >
> > > There is one more notorious problem that might affect Ignite
> deployments
> > > which is long stop-the-world GC pauses.
> > >
> > > I know we did a little progress in this direction [3] by providing
> > > particular metrics that help to monitor the pauses. Why don't we keep
> the
> > > pace and teach Ignite to help itself if it sees there is a node that
> > brings
> > > down overall cluster performance due to an STP?
> > >
> > > I would create policies similar to the critical failures policies [4]
> or
> > > just add a long STP to the list of critical failures and reuse existing
> > > functionality.
> > >
> > > Thoughts? Anyone who'd like to implement the feature?
> > >
> > > [1] https://apacheignite.readme.io/docs/critical-failures-handling
> > > [2]
> > > http://apache-ignite-developers.2346864.n4.nabble.
> > com/IEP-25-Partition-Map-Exchange-hangs-resolving-td31819.html
> > > [3] https://issues.apache.org/jira/browse/IGNITE-6171
> > > [4]
> > > https://apacheignite.readme.io/docs/critical-failures-
> > handling#section-failure-handling
> > >
> >
>

Re: Automatic Handling of Long Stop-the-World Pauses

[Pavel Kovalenko]

Denis,

I think, JVM can't easily help to itself if it's in SW pause. Most
solutions what I saw about handling such situations are checking heartbeats
on other nodes or run in parallel supervisor process which can detect that
JVM with Ignite in SW.

2018-07-02 20:54 GMT+03:00 Denis Magda :

> Igniters,
>
> Pulling this discussion up. Any thoughts?
>
> --
> Denis
>
> On Thu, Jun 21, 2018 at 3:52 PM Denis Magda  wrote:
>
> > Igniters,
> >
> > It's a pleasure to see how our project is evolving in a directing of
> being
> > a self-healing solution:
> >
> >- Ignite can already handle critical failures such as OOM, File I/O
> >issues, etc. [1]
> >- There is an endeavor to fix cluster lock-ins due to partition map
> >exchange issues. [2]
> >
> > There is one more notorious problem that might affect Ignite deployments
> > which is long stop-the-world GC pauses.
> >
> > I know we did a little progress in this direction [3] by providing
> > particular metrics that help to monitor the pauses. Why don't we keep the
> > pace and teach Ignite to help itself if it sees there is a node that
> brings
> > down overall cluster performance due to an STP?
> >
> > I would create policies similar to the critical failures policies [4] or
> > just add a long STP to the list of critical failures and reuse existing
> > functionality.
> >
> > Thoughts? Anyone who'd like to implement the feature?
> >
> > [1] https://apacheignite.readme.io/docs/critical-failures-handling
> > [2]
> > http://apache-ignite-developers.2346864.n4.nabble.
> com/IEP-25-Partition-Map-Exchange-hangs-resolving-td31819.html
> > [3] https://issues.apache.org/jira/browse/IGNITE-6171
> > [4]
> > https://apacheignite.readme.io/docs/critical-failures-
> handling#section-failure-handling
> >
>

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Hi Igor,

I totally agree with the point that auto-generated things does not 
belong to the source tree.


I already made short instructions on how to generate HTML documents with 
Sphinx in README file. The instructions assume that the user is able to 
use the common tools (Python, pip, virtualenv) and have them installed. 
From the beginning of the development we found it inconvenient for 
those who wish to just read the documents and not fire up the whole 
development process. That's why I included the prebuilt documents in the 
repository.


Note also, that Python programs, unlike Java ones, do not have the 
additional build step and always distributed as text. So each user will 
have to build the documents from the source to read it.


To leverage this condition, most of Python package developers use 
readthedocs.org service (freemium, engine is under MIT license) to 
publish their documentation online. Unfortunately, the service can not 
be used by our project because of the directory structure. RTD treats 
the git root as a home to `docs` directory to build Sphinx docs out of 
it. In our case it would build anything but the documents we need from 
upstream git source. This situation can be fixed by developing Python 
thin client in a separate git repository and including it in upstream as 
a git sub-module, but I see this solution is too radical.


Anyway, I am removing the prebuilt docs from now on, just as you 
suggested. Hope that potential usability problems might be addressed in 
the future.


On 07/02/2018 09:41 PM, Igor Sapego wrote:

By the way, guys,

I can see generated docs under source control. We do not store them
this way in Ignite. Instead, we have separate step during release, where
we generate them and include in binary distribution.

So you should remove documents from Git and provide us with instructions
on how to generate docs during release.

Best Regards,
Igor

[GitHub] ignite pull request #4290: Ignite 8900 repro

[mcherkasov]

GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/4290

Ignite 8900 repro



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/ignite ignite-8900-repro

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4290


commit b3707b44c89b3bb4cb30d07cb1c9936264b0666e
Author: devozerov 
Date:   2018-06-29T20:01:26Z

Reproducer for IGNITE-8900 issue with broken links.

commit 1e05b9963592e20150d6006752da8a7b9fc87b09
Author: Alexey Goncharuk 
Date:   2018-06-30T11:08:45Z

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/ignite 
into ignite-8900-repro

commit 1c9c746c9ebae59a83f030689d0d7e90a429152e
Author: Alexey Goncharuk 
Date:   2018-06-30T13:23:21Z

IGNITE-8900 Attempting to fix the link issue




---

Re: Automatic Handling of Long Stop-the-World Pauses

[Denis Magda]

Igniters,

Pulling this discussion up. Any thoughts?

--
Denis

On Thu, Jun 21, 2018 at 3:52 PM Denis Magda  wrote:

> Igniters,
>
> It's a pleasure to see how our project is evolving in a directing of being
> a self-healing solution:
>
>- Ignite can already handle critical failures such as OOM, File I/O
>issues, etc. [1]
>- There is an endeavor to fix cluster lock-ins due to partition map
>exchange issues. [2]
>
> There is one more notorious problem that might affect Ignite deployments
> which is long stop-the-world GC pauses.
>
> I know we did a little progress in this direction [3] by providing
> particular metrics that help to monitor the pauses. Why don't we keep the
> pace and teach Ignite to help itself if it sees there is a node that brings
> down overall cluster performance due to an STP?
>
> I would create policies similar to the critical failures policies [4] or
> just add a long STP to the list of critical failures and reuse existing
> functionality.
>
> Thoughts? Anyone who'd like to implement the feature?
>
> [1] https://apacheignite.readme.io/docs/critical-failures-handling
> [2]
> http://apache-ignite-developers.2346864.n4.nabble.com/IEP-25-Partition-Map-Exchange-hangs-resolving-td31819.html
> [3] https://issues.apache.org/jira/browse/IGNITE-6171
> [4]
> https://apacheignite.readme.io/docs/critical-failures-handling#section-failure-handling
>

[jira] [Created] (IGNITE-8910) PagesList.takeEmptyPage may fail with AssertionError: type = 1

[Ivan Rakov (JIRA)]

Ivan Rakov created IGNITE-8910:
--

 Summary: PagesList.takeEmptyPage may fail with AssertionError: 
type = 1
 Key: IGNITE-8910
 URL: https://issues.apache.org/jira/browse/IGNITE-8910
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Rakov
Assignee: Dmitriy Sorokin


Even after IGNITE-8769 fix, we sometimes get an AssertionError from free list 
during update operation. Page with type PageIO#T_DATA appears in free list for 
some reason.
Example hang on TC: 
https://ci.ignite.apache.org/viewLog.html?buildId=1442664=IgniteTests24Java8_PdsIndexingWalRecovery
Example stacktrace:
{noformat}
[15:59:26]W: [org.apache.ignite:ignite-indexing] 
java.lang.AssertionError: Assertion error on search row: 
org.apache.ignite.internal.processors.cache.tree.SearchRow@1e76dfc5
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1643)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1272)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1603)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:370)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1755)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2436)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1898)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1740)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1630)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1119)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:609)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2428)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2405)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:812)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.persistence.db.wal.IgniteWalRecoveryTest$2.call(IgniteWalRecoveryTest.java:551)
[15:59:26]W: [org.apache.ignite:ignite-indexing]at 
org.apache.ignite.internal.processors.cache.persistence.db.wal.IgniteWalRecoveryTest$2.call(IgniteWalRecoveryTest.java:546)
[15:59:26]W:

[GitHub] ignite pull request #4242: IGNITE-8746 EVT_CACHE_REBALANCE_PART_DATA_LOST ev...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4242


---

[GitHub] ignite pull request #4218: IGNITE-8821: Huge logs on BPlusTreeSelfTest put/r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4218


---

Re: data extractor

[Alexey Goncharuk]

Dmitriy,

A few questions regarding the user cases for the utility:
1) Would I be able to read the extracted data from the dumped file without
Ignite node binary/marshaller metadata? In other words, will I be able to
move only the dumped file to another grid or will I need to move the
metadata as well?
2) Are you planning to add a public API version of this utility as a part
of Ignite? For example, if I am planning to run some statistics on a
checkpointed data, will I be able to get some sort of an iterator to
process this data?
3) How a user will choose which caches (cache groups) to process? Will the
user need to provide a cache or cache ID (or either of them)? Will the
utility be able to extract a single cache data from a cache group?
4) I think the upload part of the utility is missing some input parameters
- for example, what cluster to connect to, what caches to upload to, etc.

сб, 30 июн. 2018 г. в 22:38, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com>:

> Igniters,
>
> I am working on IGNITE-7644
>  (export all key-value
> data from a persisted partition),
> it will be command line tool for extracting data from Ignite partition
> file without the need to start node.
> The main motivation is to have a lifebuoy in case if a file has damage for
> some reason.
>
> I suggest simple API and two commands for the first implementation:
>
> -c
> --CRC [srcPath] - check CRC for all(or by type) pages in partition
>
> -e
> --extract [srcPath] [outPath] - dump all survey data from partition to
> another file with raw key/value pair format
> (required graceful stop for a node, not necessary after --restore will be
> implemented)
>
> Output file format see in attached, this format does not contain any index
> inside but it is very simple and
> flexible for future works with raw key/value data.
>
> Future features:
> -u
> --upload - reload raw key/value pairs to node
>
> -s
> --status - check current node file status, need binary recovery or not
> (node crash on the middle of a checkpoint)
>
> -r
> --restore - restore binary consistency (finish checkpoint, required WAL
> file for recovery)
>
> Let's start a discussion, any comments are welcome.
>
>

[GitHub] ignite pull request #4289: Ignite 2.4.7.b1

[mcherkasov]

GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/4289

Ignite 2.4.7.b1



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.7.b1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4289


commit 6d43b6a31d0e7da0a1c245ed7c4ae85ea6889ddc
Author: Slava Koptilin 
Date:   2018-02-15T11:57:40Z

IGNITE-5804 ScanQuery transformer should be applied to all result pages - 
Fixes #3470.

Signed-off-by: Alexey Goncharuk 

commit d53504adb1d4276f79ede2401e2d1512fe0287ec
Author: devozerov 
Date:   2018-02-22T07:07:19Z

IGNITE-7253: JDBC thin streaming: fixed default local buffer size, improved 
error messages in case of unsupported SQL statements.

commit debc906a25d3e2d65db58e16307fae6f08460eeb
Author: devozerov 
Date:   2018-02-27T09:13:52Z

Revert "IGNITE-7253: JDBC thin streaming: fixed default local buffer size, 
improved error messages in case of unsupported SQL statements."

This reverts commit d53504adb1d4276f79ede2401e2d1512fe0287ec.

commit d8cf9e042c0e9afe65508c006d9d1c39779d78b9
Author: devozerov 
Date:   2018-02-27T09:14:21Z

Revert "IGNITE-7253: Streaming mode for JDBC thin driver. This closes 
#3499."

This reverts commit bc331f9de716c30d6f733e28821ab44da7ed0cf7.

commit 975a7cdb68cb89da74ec78c3cf23f644050918ff
Author: devozerov 
Date:   2018-02-27T09:49:44Z

Merge branch 'ignite-2.4' into ignite-2.4.3

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java
#   parent/pom.xml

commit 74a54d23913bd7195c525d8e222b4e4047515843
Author: Ilya Lantukh 
Date:   2018-02-28T07:08:54Z

IGNITE-7836 Fixed handling of state change message when forceReassignment 
is false

commit 2a70ede048f59753061973495f83927f47452d66
Author: Andrey Novikov 
Date:   2018-01-19T03:05:03Z

IGNITE-6920 Web Console: Create default account for direct-install package.

(cherry picked from commit e5005d9)

commit 2f1ea2cdb76863008d4514f26845457bdeb7d6ed
Author: Andrey Novikov 
Date:   2018-01-19T04:35:30Z

IGNITE-6920 Minor fix.

(cherry picked from commit 9cc7cbf)

commit 62652f3fb1563ba149dcbccb80928d50b822ff36
Author: alexdel 
Date:   2018-01-25T08:49:28Z

IGNITE-7064 Web Console: Implemented basic E2E tests.

(cherry picked from commit ce96e4f)

commit 06e891f1161af598e0aa4665f7a6047637d1c476
Author: Dmitriy Shabalin 
Date:   2018-01-25T09:51:44Z

IGNITE-7522 Web Console: Fixed cluster selector state after cluster restart.

(cherry picked from commit ac3cfb8)

commit 291cb2c88118ccffebcf3383db629647faec1eee
Author: Dmitriy Shabalin 
Date:   2018-01-25T10:33:13Z

IGNITE-7529 Web Console: Refactor UIGrid column filters.

(cherry picked from commit 08658ea)

commit 443eafc718685e66c9a60058bd0ab56d88f9f0a6
Author: alexdel 
Date:   2018-01-25T14:38:36Z

IGNITE-7064 Web Console: Minor test fix.

(cherry picked from commit 4b6d9ad)

commit 6f1df5c40100363b5922734223a774ff1d6a008e
Author: Ilya Borisov 
Date:   2018-01-26T09:07:47Z

IGNITE-7031 Web Console: Refactored confirmation cancellation logic.

(cherry picked from commit 92ae3fe)

commit 16982825fb06ff2724ba4583781cc15443c4f46d
Author: Alexey Kuznetsov 
Date:   2018-02-02T10:07:57Z

IGNITE-7610 Web Console: Profile page refactored to component.

(cherry picked from commit 958975e)

commit 9c6a52250e058c4546aef0d0ecee977c07076a1a
Author: Alexey Kuznetsov 
Date:   2018-02-02T10:09:37Z

IGNITE-7612 Web Console: Refactored mongoose schemas to separate module.

(cherry picked from commit 71db707)

commit 89e15830dedcb46f24d9cc9b922ba3b013331a18
Author: Vasiliy Sisko 
Date:   2018-02-12T10:22:10Z

Web Agent: Fixed wrong config of IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE in 
demo startup.

(cherry picked from commit 1a6e544)

commit 18966673570425192e1b89fbb2c63d164b47eaca
Author: Vasiliy Sisko 
Date:   2018-02-12T13:24:30Z

IGNITE-7578 Actualized client connector configuration.

(cherry picked from commit 819d746)

commit 237063efa35c54bb9e9800ecf63ea223ec20a9ef
Author: alexdel 
Date:   2018-02-19T04:25:24Z

IGNITE-7650 Extracted signin/signup form to separate page, improved landing 
page.

(cherry picked from commit 1925674)

commit 679aeca7a3ff60a9dd478966d3949107d302d5db
Author: Andrey Novikov 
Date:   2018-02-19T07:56:07Z

IGNITE-7650 Fixed headers.

(cherry picked from commit 67922b3)

commit 5d5a6a05ec49f895e8a5edd505e496dcfe58a208
Author: Vasiliy Sisko 
Date:   2018-02-21T04:21:02Z

IGNITE-6094 Web Agent: Enabled persistent in demo 

[GitHub] ignite pull request #4211: IGNITE-8203 Interrupting task can cause node fail...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4211


---

Re: Ignite as distributed file storage

[Vladimir Ozerov]

Pavel,

Thank you. I'll wait for feature comparison and concrete use cases, because
for me this feature still sounds too abstract to judge whether product
would benefit from it.

On Mon, Jul 2, 2018 at 3:15 PM Pavel Kovalenko  wrote:

> Dmitriy,
>
> I think we have a little miscommunication here. Of course, I meant
> supporting large entries / chunks of binary data. Internally it will be
> BLOB storage, which can be accessed through various interfaces.
> "File" is just an abstraction for an end user for convenience, a wrapper
> layer to have user-friendly API to directly store BLOBs. We shouldn't
> support full file protocol support with file system capabilities. It can be
> added later, but now it's absolutely unnecessary and introduces extra
> complexity.
>
> We can implement our BLOB storage step by step. The first thing is
> core functionality and support to save large parts of binary objects to it.
> "File" layer, Web layer, etc. can be added later.
>
> The initial IGFS design doesn't have good capabilities to have a
> persistence layer. I think we shouldn't do any changes to it, this project
> as for me is almost outdated. We will drop IGFS after implementing File
> System layer over our BLOB storage.
>
> Vladimir,
>
> I will prepare a comparison with other existing distributed file storages
> and file systems in a few days.
>
> About usage data grid, I never said, that we need transactions, sync backup
> and etc. We need just a few core things - Atomic cache with persistence,
> Discovery, Baseline, Affinity, and Communication.
> Other things we can implement by ourselves. So this feature can develop
> independently of other non-core features.
> For me Ignite way is providing to our users a fast and convenient way to
> solve their problems with good performance and durability. We have the
> problem with storing large data, we should solve it.
> About other things see my message to Dmitriy above.
>
> вс, 1 июл. 2018 г. в 9:48, Dmitriy Setrakyan :
>
> > Pavel,
> >
> > I have actually misunderstood the use case. To be honest, I thought that
> > you were talking about the support of large values in Ignite caches, e.g.
> > objects that are several megabytes in cache.
> >
> > If we are tackling the distributed file system, then in my view, we
> should
> > be talking about IGFS and adding persistence support to IGFS (which is
> > based on HDFS API). It is not clear to me that you are talking about
> IGFS.
> > Can you confirm?
> >
> > D.
> >
> >
> > On Sat, Jun 30, 2018 at 10:59 AM, Pavel Kovalenko 
> > wrote:
> >
> > > Dmitriy,
> > >
> > > Yes, I have approximate design in my mind. The main idea is that we
> > already
> > > have distributed cache for files metadata (our Atomic cache), the data
> > flow
> > > and distribution will be controlled by our AffinityFunction and
> Baseline.
> > > We're already have discovery and communication to make such local files
> > > storages to be synced. The files data will be separated to large blocks
> > > (64-128Mb) (which looks very similar to our WAL). Each block can
> contain
> > > one or more file chunks. The tablespace (segment ids, offsets and etc.)
> > > will be stored to our regular page memory. This is key ideas to
> implement
> > > first version of such storage. We already have similiar components in
> our
> > > persistence, so this experience can be reused to develop such storage.
> > >
> > > Denis,
> > >
> > > Nothing significant should be changed at our memory level. It will be
> > > separate, pluggable component over cache. Most of the functions which
> > give
> > > performance boost can be delegated to OS level (Memory mapped files,
> DMA,
> > > Direct write from Socket to disk and vice versa). Ignite and File
> Storage
> > > can develop independetly of each other.
> > >
> > > Alexey Stelmak, which has a great experience with developing such
> systems
> > > can provide more low level information about how it should look.
> > >
> > > сб, 30 июн. 2018 г. в 19:40, Dmitriy Setrakyan  >:
> > >
> > > > Pavel, it definitely makes sense. Do you have a design in mind?
> > > >
> > > > D.
> > > >
> > > > On Sat, Jun 30, 2018, 07:24 Pavel Kovalenko 
> > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > I would like to start a discussion about designing a new feature
> > > because
> > > > I
> > > > > think it's time to start making steps towards it.
> > > > > I noticed, that some of our users have tried to store large
> > homogenous
> > > > > entries (> 1, 10, 100 Mb/Gb/Tb) to our caches, but without big
> > success.
> > > > >
> > > > > IGFS project has the possibility to do it, but as for me it has one
> > big
> > > > > disadvantage - it's in-memory only, so users have a strict size
> limit
> > > of
> > > > > their data and have data loss problem.
> > > > >
> > > > > Our durable memory has a possibility to persist a data that doesn't
> > fit
> > > > to
> > > > > RAM to disk, but page structure of it is not supposed to store
> large
> > > > pieces
> > > > > of 

[jira] [Created] (IGNITE-8909) Visor shows unimplemented off-heap memory cache metric

[Denis Mekhanikov (JIRA)]

Denis Mekhanikov created IGNITE-8909:


 Summary: Visor shows unimplemented off-heap memory cache metric
 Key: IGNITE-8909
 URL: https://issues.apache.org/jira/browse/IGNITE-8909
 Project: Ignite
  Issue Type: Task
  Components: visor
Affects Versions: 2.5
Reporter: Denis Mekhanikov


"Off-Heap Memory" metric in result of *cache -a* command is always 0.

Cache memory metrics were replaced with the ones for data regions, so this 
metric is unlikely to be implemented.

Off-Heap Memory metric should be removed from Visor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Ignite as distributed file storage

[Pavel Kovalenko]

Dmitriy,

I think we have a little miscommunication here. Of course, I meant
supporting large entries / chunks of binary data. Internally it will be
BLOB storage, which can be accessed through various interfaces.
"File" is just an abstraction for an end user for convenience, a wrapper
layer to have user-friendly API to directly store BLOBs. We shouldn't
support full file protocol support with file system capabilities. It can be
added later, but now it's absolutely unnecessary and introduces extra
complexity.

We can implement our BLOB storage step by step. The first thing is
core functionality and support to save large parts of binary objects to it.
"File" layer, Web layer, etc. can be added later.

The initial IGFS design doesn't have good capabilities to have a
persistence layer. I think we shouldn't do any changes to it, this project
as for me is almost outdated. We will drop IGFS after implementing File
System layer over our BLOB storage.

Vladimir,

I will prepare a comparison with other existing distributed file storages
and file systems in a few days.

About usage data grid, I never said, that we need transactions, sync backup
and etc. We need just a few core things - Atomic cache with persistence,
Discovery, Baseline, Affinity, and Communication.
Other things we can implement by ourselves. So this feature can develop
independently of other non-core features.
For me Ignite way is providing to our users a fast and convenient way to
solve their problems with good performance and durability. We have the
problem with storing large data, we should solve it.
About other things see my message to Dmitriy above.

вс, 1 июл. 2018 г. в 9:48, Dmitriy Setrakyan :

> Pavel,
>
> I have actually misunderstood the use case. To be honest, I thought that
> you were talking about the support of large values in Ignite caches, e.g.
> objects that are several megabytes in cache.
>
> If we are tackling the distributed file system, then in my view, we should
> be talking about IGFS and adding persistence support to IGFS (which is
> based on HDFS API). It is not clear to me that you are talking about IGFS.
> Can you confirm?
>
> D.
>
>
> On Sat, Jun 30, 2018 at 10:59 AM, Pavel Kovalenko 
> wrote:
>
> > Dmitriy,
> >
> > Yes, I have approximate design in my mind. The main idea is that we
> already
> > have distributed cache for files metadata (our Atomic cache), the data
> flow
> > and distribution will be controlled by our AffinityFunction and Baseline.
> > We're already have discovery and communication to make such local files
> > storages to be synced. The files data will be separated to large blocks
> > (64-128Mb) (which looks very similar to our WAL). Each block can contain
> > one or more file chunks. The tablespace (segment ids, offsets and etc.)
> > will be stored to our regular page memory. This is key ideas to implement
> > first version of such storage. We already have similiar components in our
> > persistence, so this experience can be reused to develop such storage.
> >
> > Denis,
> >
> > Nothing significant should be changed at our memory level. It will be
> > separate, pluggable component over cache. Most of the functions which
> give
> > performance boost can be delegated to OS level (Memory mapped files, DMA,
> > Direct write from Socket to disk and vice versa). Ignite and File Storage
> > can develop independetly of each other.
> >
> > Alexey Stelmak, which has a great experience with developing such systems
> > can provide more low level information about how it should look.
> >
> > сб, 30 июн. 2018 г. в 19:40, Dmitriy Setrakyan :
> >
> > > Pavel, it definitely makes sense. Do you have a design in mind?
> > >
> > > D.
> > >
> > > On Sat, Jun 30, 2018, 07:24 Pavel Kovalenko 
> wrote:
> > >
> > > > Igniters,
> > > >
> > > > I would like to start a discussion about designing a new feature
> > because
> > > I
> > > > think it's time to start making steps towards it.
> > > > I noticed, that some of our users have tried to store large
> homogenous
> > > > entries (> 1, 10, 100 Mb/Gb/Tb) to our caches, but without big
> success.
> > > >
> > > > IGFS project has the possibility to do it, but as for me it has one
> big
> > > > disadvantage - it's in-memory only, so users have a strict size limit
> > of
> > > > their data and have data loss problem.
> > > >
> > > > Our durable memory has a possibility to persist a data that doesn't
> fit
> > > to
> > > > RAM to disk, but page structure of it is not supposed to store large
> > > pieces
> > > > of data.
> > > >
> > > > There are a lot of projects of distributed file systems like HDFS,
> > > > GlusterFS, etc. But all of them concentrate to implement high-grade
> > file
> > > > protocol, rather than user-friendly API which leads to high entry
> > > threshold
> > > > to start implementing something over it.
> > > > We shouldn't go in this way. Our main goal should be providing to
> user
> > > easy
> > > > and fast way to use file storage and processing here and now.
> > 

[GitHub] ignite pull request #4272: new IgnitePredicate filtering credential attribut...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4272


---

Re: Thin Client lib: Python

[Igor Sapego]

By the way, guys,

I can see generated docs under source control. We do not store them
this way in Ignite. Instead, we have separate step during release, where
we generate them and include in binary distribution.

So you should remove documents from Git and provide us with instructions
on how to generate docs during release.

Best Regards,
Igor


On Fri, Jun 29, 2018 at 5:01 PM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Dmitry,
>
> I have mentioned Jira task in my reply earlier today. Sorry, but the
> task description was a bit messy and less than informative. I just
> updated it. Please have a look.
>
> https://issues.apache.org/jira/browse/IGNITE-7782
>
> Please let me know if this information is supposed to be somewhere in
> Wiki. I'll post it right away.
>
> On 06/29/2018 07:53 AM, Dmitriy Setrakyan wrote:
> > Is there an IEP for it where the community can read about the feature set
> > that will be supported by the Python client?
> >
> > D.
>

[GitHub] ignite pull request #4288: IGNITE-8905 incorrect assertion was replaced with...

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/4288

IGNITE-8905 incorrect assertion was replaced with if-check



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8905

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4288


commit f2194169e9eda7232bc642bf3b1b5a1ad4967af0
Author: Sergey Chugunov 
Date:   2018-07-02T10:54:55Z

IGNITE-8905 incorrect assertion was replaced with if-check




---

[jira] [Created] (IGNITE-8908) NPE in BinaryMetadataTransport

[Mikhail Cherkasov (JIRA)]

Mikhail Cherkasov created IGNITE-8908:
-

 Summary: NPE in BinaryMetadataTransport
 Key: IGNITE-8908
 URL: https://issues.apache.org/jira/browse/IGNITE-8908
 Project: Ignite
  Issue Type: Bug
  Components: general
Reporter: Mikhail Cherkasov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8907) [ML] Using vectors in featureExtractor

[Yury Babak (JIRA)]

Yury Babak created IGNITE-8907:
--

 Summary: [ML] Using vectors in featureExtractor
 Key: IGNITE-8907
 URL: https://issues.apache.org/jira/browse/IGNITE-8907
 Project: Ignite
  Issue Type: Improvement
  Components: ml
Reporter: Yury Babak
Assignee: Alexey Platonov
 Fix For: 2.7






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8906) SQL TX: Immediate rollback for Near transactions during topology changing.

[Roman Kondakov (JIRA)]

Roman Kondakov created IGNITE-8906:
--

 Summary: SQL TX: Immediate rollback for Near transactions during 
topology changing.
 Key: IGNITE-8906
 URL: https://issues.apache.org/jira/browse/IGNITE-8906
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Roman Kondakov


At the moment all near transaction that was active during PME, but hadn't 
locked topology (i.e. read transactions), are marked as rollback only in 
GridDhtPartitionsExchangeFuture#onDone. But it is better to stop transaction 
execution immediately to prevent useless updates.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8905) Incorrect assertion in GridDhtPartitionsExchangeFuture

[Sergey Chugunov (JIRA)]

Sergey Chugunov created IGNITE-8905:
---

 Summary: Incorrect assertion in GridDhtPartitionsExchangeFuture
 Key: IGNITE-8905
 URL: https://issues.apache.org/jira/browse/IGNITE-8905
 Project: Ignite
  Issue Type: Bug
Reporter: Sergey Chugunov
Assignee: Sergey Chugunov


Assertion was added as part IGNITE-8657 into GridDhtPartitionsExchangeFuture 
which is correct only in situation when client has to reconnect due to too 
short EXCHANGE_HISTORY.

Exceptions from other situations like not being able to acquire file lock are 
also passed to client node in FullMessage.

This assertion should be removed and check should be introduced instead: if 
this exception is intended to be thrown on current client node, we should do 
this, otherwise old program flow should be executed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: is these correct?

[kcheng.mvp]

Hi dkarachentsev,


oh, my fault. I did not notice this part when I look through the document.

I am supposed to post such question to user mail list.


Thank you very much.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: is these correct?

[Dmitry Karachentsev]

Hi,

Please clarify what's wrong in your opinion?

If you mean that you do not see hits or misses - metrics are disabled by 
default [1].


[1] https://apacheignite.readme.io/v2.0/docs/memory-and-cache-metrics

Thanks!
-Dmitry

30.06.2018 05:12, kcheng.mvp пишет:

here is the environment : 1 server node + 1 client node

executed below code
===
IgniteCache igniteCache =
igniteSpringBean.getOrCreateCache("HappyFlow");
 igniteCache.put("hello", "hello1");
 Assert.assertEquals("hello1",igniteCache.get("hello"));

 for (int i = 0;i < 100; i ++) {
 igniteCache.get("hello");
 }
===

I got below screenshot in ignitevisorcmd.sh






--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-8904) Add rebalanceThreadPoolSize to nodes configuration consistency check

[Pavel Kovalenko (JIRA)]

Pavel Kovalenko created IGNITE-8904:
---

 Summary: Add rebalanceThreadPoolSize to nodes configuration 
consistency check
 Key: IGNITE-8904
 URL: https://issues.apache.org/jira/browse/IGNITE-8904
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.5, 2.4
Reporter: Pavel Kovalenko
Assignee: Pavel Kovalenko
 Fix For: 2.7


If supplier node has less thread-pool size than demander node, rebalance 
process between them will hang indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: NodeJS client: npm publishing instruction

[Igor Sapego]

I believe, we need to put this instruction on wiki page.

Best Regards,
Igor


On Sun, Jul 1, 2018 at 3:58 PM Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Denis,
>
> below is instruction how to publish/update npm module with Ignite NodeJS
> client on npmjs.
> Use it when you decide to publish the client.
>
> Another short instruction is how to generate the API spec using jsdoc.
>
> Pls place the instructions to appropriate locations.
>
> Thanks,
> -Alexey
>
>
> How to publish Ignite NodeJS Client on npmjs.com:
> -
> 1. Install NodeJS npm (https://nodejs.org/en/), if not installed yet.
> 2. Register an account at npmjs (https://www.npmjs.com/signup), if not
> registered yet.
> 3. Execute `npm login` command and provide your npmjs account credentials.
> 4. Clone or download Ignite repository
> https://github.com/apache/ignite.git to `local_ignite_path`
> 5. Go to `local_ignite_path/modules/platforms/nodejs`
> 6. Prepare/update
> `local_ignite_path/modules/platforms/nodejs/package.json` file. Pay
> attention to:
>- "name" - name of the npm module
>- "version" - version of the npm module, increment it if you update
> the module
>- "description" - description of the npm module
>- "repository" - type and link to the repository with the source code
>- "keywords" - keywords for the search of the module on npmjs
>- "license" - license type
>- other properties depend on the implementation/tests, do not touch them
> Example of the package.json file is attached.
> 7. Prepare/update `local_ignite_path/modules/platforms/nodejs/README.md`
> file. It should exist and should not be empty. Eg. add a link to a place
> with the documentation.
> 8. Execute `npm publish` command from the
> `local_ignite_path/modules/platforms/nodejs` folder.
> 9. Check the module is published and well-described at
> https://www.npmjs.com/package/apache-ignite-client (assuming
> `apache-ignite-client` is the name of the module)
>
> Common instruction about npm publishing:
> https://docs.npmjs.com/getting-started/publishing-npm-packages
>
> How to generate Ignite NodeJS Client API specification:
> ---
> It should be done if a public API class/method has been changed.
> 1. Execute `npm install -g jsdoc` to install jsdoc
> (https://www.npmjs.com/package/jsdoc)
> 2. Clone or download Ignite repository
> https://github.com/apache/ignite.git to `local_ignite_path`
> 3. Go to `local_ignite_path/modules/platforms/nodejs/api_spec`
> 4. Only if a class has been removed from the public API, remove all
> files from `local_ignite_path/modules/platforms/nodejs/api_spec` except
> conf.json file.
> 5. Execute `jsdoc -c conf.json` command.
>
> Note: `local_ignite_path/modules/platforms/nodejs/api_spec/conf.json` is
> a file with jsdoc configuration.
>

Add cluster (de)activation events IGNITE-8376

[kcheng.mvp]

Dear igniters,I am going to pick up
https://issues.apache.org/jira/browse/IGNITE-8376, and did some research
based on the comments.based on my understanding, if we want to know this
action is triggered by which way(rest, mbean, auto or visocmd)  then we need
to change the core method's signature.I am not sure my understanding is
correct or not. Can anybody help to clarify this? Thank you very much.By the
way, I commented this in jira as well.Thanks youkcvmp



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #4286: IGNITE-8898

[vldpyatkov]

GitHub user vldpyatkov opened a pull request:

https://github.com/apache/ignite/pull/4286

IGNITE-8898

rename command argument '--force' to '--yes' for control.sh

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8898

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4286


commit d6c3de9e0e18a61286500614f8a8f9414a9ce2a1
Author: vd-pyatkov 
Date:   2018-06-29T14:50:02Z

IGNITE-8898
rename command argument '--force' to '--yes' for control.sh




---

[GitHub] ignite pull request #4283: GG-13963

[vldpyatkov]

Github user vldpyatkov closed the pull request at:

https://github.com/apache/ignite/pull/4283


---