Re: How a new index is built in runtime?

[Yakov Zhdanov]

Of course, iteration should have an option to be run from more than 1
thread. What will really help, IMO, is ability to insert presorted batches
in a single tree operation.

--
Yakov Zhdanov

[GitHub] ignite pull request #2476: Ignite-5983: Memory policy metrics should be avai...

[shroman]

GitHub user shroman opened a pull request:

https://github.com/apache/ignite/pull/2476

Ignite-5983: Memory policy metrics should be available via REST.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shroman/ignite IGNITE-5983

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2476


commit da7d1aa650022f42bef215fedb9a0facbad48d08
Author: shroman 
Date:   2017-08-17T08:57:13Z

IGNITE-5983: Memory policy metrics should be available via REST.

commit fd218e3ec415948adfd472505877b0d8a210a399
Author: shroman 
Date:   2017-08-18T03:54:38Z

IGNITE-5983: Memory policy metrics should be available via REST.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6105) Web console: Cache name is missed in preview for cache checkpoint SPI

[Vasiliy Sisko (JIRA)]

Vasiliy Sisko created IGNITE-6105:
-

 Summary: Web console: Cache name is missed in preview for cache 
checkpoint SPI
 Key: IGNITE-6105
 URL: https://issues.apache.org/jira/browse/IGNITE-6105
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Affects Versions: 2.1
Reporter: Vasiliy Sisko
Assignee: Vasiliy Sisko






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Default page size must be changed to 4k. Should it be backwards compatible?

[Dmitriy Setrakyan]

On Tue, Aug 15, 2017 at 10:09 PM, Serge Puchnin 
wrote:

> #1 option looks more predictable. But it's possible to add a message "for
> better IO performance please migrate to 4K-pages".
>

Agree. Can we update the ticket to make sure that this suggestion is
printed out?


> BR,
> Serge
>
>
> On Wed, 16 Aug 2017 at 03:46, Dmitriy Setrakyan 
> wrote:
>
> > I like #1 if possible. Of course, if the LFS is empty, then the new
> default
> > should be 4k.
> >
> > On Tue, Aug 15, 2017 at 12:01 PM, Ivan Rakov 
> > wrote:
> >
> > > Guys,
> > >
> > > We have benchmarked how checkpoint write speed on SSD disk depends on
> > > various parameters. It became absolutely obvious that using 4K pages in
> > > durable memory instead of 2K brings considerable, significant
> speed-up. I
> > > think, we must set 4K as default page size.
> > > Ticket with detailed explanation: https://issues.apache.org/jira
> > > /browse/IGNITE-5884
> > > Spoiler: it depends on write order and alignment, but writing 4K is at
> > > least *3x faster* than writing 2K when other parameters are the same.
> > >
> > > The question is backwards compatibility. If pageSize is not explicitly
> > set
> > > in user configuration, attempt to start "4k default" Ignite node from
> "2k
> > > default" LFS files will fail with exception:
> > >
> > > class org.apache.ignite.IgniteCheckedException: Failed to verify store
> > >> file (invalid page size) [expectedPageSize=4096, filePageSize=2048]
> > >> at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStore.checkFile(FilePageStore.java:206)
> > >> at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStore.init(FilePageStore.java:416)
> > >> at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStore.read(FilePageStore.java:315)
> > >> at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStoreManager.read(FilePageStoreManager.java:287)
> > >> at org.apache.ignite.internal.processors.cache.persistence.file
> > >> .FilePageStoreManager.read(FilePageStoreManager.java:272)
> > >> at org.apache.ignite.internal.processors.cache.persistence.page
> > >> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:569)
> > >> at org.apache.ignite.internal.processors.cache.persistence.page
> > >> mem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:487)
> > >> at org.apache.ignite.internal.processors.cache.persistence.Grid
> > >> CacheOffheapManager.getOrAllocateCacheMetas(GridCacheOffheap
> > >> Manager.java:515)
> > >> at org.apache.ignite.internal.processors.cache.persistence.Grid
> > >> CacheOffheapManager.initDataStructures(GridCacheOffheapManager.java:
> 86)
> > >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> > >> apManagerImpl.start(IgniteCacheOffheapManagerImpl.java:139)
> > >> at org.apache.ignite.internal.processors.cache.CacheGroupContex
> > >> t.start(CacheGroupContext.java:868)
> > >>
> > >
> > > I think, we have two options here:
> > >
> > > 1) Obvious and safe - provide silent backwards compatibility. We can
> > > implement a task which will find any LFS file, check its pageSize and
> use
> > > it as default.
> > > 2) Less user-friendly, but in my opinion still better option - crash
> > node,
> > > but make error message more informative. We'll let user know that
> default
> > > pageSize was changed to 4k due to discovered performance boost on most
> > > UNIX-based enviroments with SSD (which is for sure most popular
> > enviroment
> > > among users), and recommend user to migrate to 4K-page LFS. If user
> still
> > > wants to work with 2k pages, he can always set it explicitly in
> > > MemoryConfiguration and start node.
> > >
> > > Thoughts?
> > >
> > > --
> > > Best Regards,
> > > Ivan Rakov
> > >
> > >
> >
>

Re: Batch service deployment

[Dmitriy Setrakyan]

Sounds good! Thanks for the detailed info. Can you please provide the
updated API in the ticket?

On Thu, Aug 17, 2017 at 12:41 AM, Denis Mekhanikov 
wrote:

> > Can we add an "allOrNone" flag to the deployment method?
>
> Sounds good, I think we can.
>
> > However, hot do you ensure atomicity here?
>
> We can guarantee that if some of configurations are invalid, or a
> transaction, that writes configuration to the internal cache, fails, then
> no services will be deployed.
>
> Currently we don't track failures on the server side and services are
> considered successfully deployed once their configurations are written to
> the cache. So, it's not possible that all configurations are valid, but
> only a part of the services fail to deploy. If we change this behavior and
> start tracking failures during deployment and initialization on the server,
> then we could automatically cancel services that are already deployed in a
> batch.
>
> чт, 17 авг. 2017 г. в 8:34, Dmitriy Setrakyan :
>
> > On Wed, Aug 16, 2017 at 8:17 AM, Denis Mekhanikov  >
> > wrote:
> >
> > > I've had a few off-line conversations with other Igniters regarding
> this
> > > question and almost all of them think that services should be deployed
> > with
> > > "all-or-none" failing policy.
> > > We have a similar functionality for caches: Ignite#createCaches method
> > > don't allow partial deployments, and I think, we should also stick to
> > this
> > > principle for services.
> > >
> >
> > Can we add an "allOrNone" flag to the deployment method? If true, then
> all
> > services will have to either be deployed or failed. However, hot do you
> > ensure atomicity here? If you are deploying 10 services, and only 1
> fails,
> > what do you do with the other 9, given that they have already been
> deployed
> > and may have started serving API requests?
> >
> >
> > >
> > > Another question that I'd like to discuss here is that currently
> > > IgniteServices#deployAsync method may fail with an exception instead of
> > > returning a future. Shouldn't we change this behavior to make async
> > > operations always return a future whose get() method would throw an
> > > exception?
> > >
> >
> > Makes sense to me. I think throwing exception from async method is plain
> > wrong.
> >
> > >
> > > вт, 15 авг. 2017 г. в 11:42, Dmitriy Setrakyan  >:
> > >
> > > > Denis,
> > > >
> > > > I don't think we need a king deployment result.
> > > >
> > > > The "deployAllAsync" method should never throw an exception, it
> should
> > > > always return the future. However, the IgniteFuture.get(...) method
> > does
> > > > throw an exception, and in this exception you should provide the info
> > > about
> > > > the failures.
> > > >
> > > > D.
> > > >
> > > > On Tue, Aug 15, 2017 at 1:31 AM, Denis Mekhanikov <
> > dmekhani...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Dmitriy, thank you for your reply!
> > > > >
> > > > > I see a possibility of a bad scenario here. If we use
> deployAllAsync
> > > > method
> > > > > and it throws an exception, then the constructed future won't be
> > > returned
> > > > > and we won't have a way to wait for the rest of the services to
> > deploy.
> > > > > Maybe we should return some king of deployment result, containing a
> > > > future
> > > > > along with a collection of failed services, instead of throwing an
> > > > > exception?
> > > > >
> > > > > пн, 14 авг. 2017 г. в 18:03, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >:
> > > > >
> > > > > > Hi Denis, I agree, we should have an API for batch service
> > > deployment.
> > > > My
> > > > > > comments are inline...
> > > > > >
> > > > > > On Mon, Aug 14, 2017 at 2:22 AM, Denis Mekhanikov <
> > > > dmekhani...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Igniters!
> > > > > > >
> > > > > > > Currently Ignite doesn't have support for batch service
> > deployment,
> > > > but
> > > > > > it
> > > > > > > may be a very useful feature in case of a big number of nodes
> in
> > a
> > > > > > cluster
> > > > > > > and services to be deployed. Each deployment includes write
> into
> > an
> > > > > > > internal transactional cache, which is the longest part of the
> > > > > procedure.
> > > > > > >
> > > > > > > I propose to optimize it by performing multiple writes in a
> > single
> > > > > > > transaction. It implies an introduction of a few new methods in
> > > > > > > IgniteServices interface.
> > > > > > > I am thinking about the following signatures:
> > > > > > >
> > > > > > >   void deployAll(Iterable cfgs) throws
> > > > > > > IgniteException;
> > > > > > >   IgniteFuture
> > deployAllAsync(Iterable
> > > > > > > cfgs) throws IgniteException;
> > > > > > >
> > > > > > > I'd like to know your opinion on the following questions:
> > > > > > >
> > > > > > >- Do you agree with the proposed signatures?
> > > > > > >
> > > > > >
> > > > > > Yes, but Iterable should be changed to Collection to be
> consistent
> > > with
> > > > > > other similar APIs

Re: Cluster auto activation design proposal

[Dmitriy Setrakyan]

On Thu, Aug 17, 2017 at 7:34 AM, Sergey Chugunov 
wrote:

> Dmitriy,
>
> As automatic creation of Baseline Topology is just another use case about
> the whole concept, lets discuss it in this thread and forget about parallel
> one.
>
> *initialActivationNodes* for desired configuration of Baseline Topology
> sounds good to me, I would like to hear from other Igniters.
>

I don't think this sounds right. It should be either "inititialActiveNodes"
or "initialNodes". I am also open to other suggestions.


>
> However other options to create Baseline Topology were proposed:
>
>- Prepare it manually with WebConsole or visor-console (no
>initialActivationNodes configuration is involved).
>- Create it automatically on the first manual activation (again no
>initial configuration is involved).
>

> Thus we still need Baseline Topology entity not only in documentation.
>

In either case, wouldn't it involve creation of initialActiveNodes
implicitly?


>
> Also please take into account that at some point in time user may want to
> recreate BLT (e.g. to add nodes to existing cluster). User will have to
> interact with some entity; so we need it not only internally but on public
> API.
>

What would the public API look like and who would user interact with it?

Re: Ignite: configuration changes at runtime

[Dmitriy Setrakyan]

On Thu, Aug 17, 2017 at 10:37 AM, Yakov Zhdanov  wrote:

> >I don't like the idea of intermixing changeable properties with constant
> >ones in the configuration. Moreover, the standard configuration class is
> >local to the node, while changing the config properties at runtime may be
> a
> >distributed operation that needs to be propagated across the cluster.
>
> I don't 100% agree here. Let's discuss.
>

I will take your 99% :)


>
> 1. I would not mess with propagation for now. Propagation can be handled by
> special methods or by Visor/WebConsole
>

I think providing API is important for users who want to integrate Ignite
with their own management tools.


> 2. If we choose to create separate managing objects we end up in creating
> that for each configuration type we have now. And we have quite great
> number of config objects. For me personally, it seems more convenient to
> have configuration object a place to change and retrieve current
> configuration. We can have separate method to apply the changes (on
> configuration object or object that is configured by that configuration
> object).
>

I think it is a design decision. Do we have enough config properties that
can be changed at runtime to merit a separate config object? Or can we
group them all in one runtime config?


>
> >How about we add getters and setters to IgniteConfigurationRuntime
> >interface. BTW, this interface should also become an MBean, so it can be
> >managed by external tools.
>
> Again, if we introduce separate runtime configurable object this seems to
> be very confusing. After I change the property via that object, should
> corresponding property change in let's say IgniteConfiguration?
>

No, the Ignite configuration is what Ignite receives on startup. Runtime
changes are outside of configuration, especially given that if cluster
restarts, they are likely lost.


>
> --Yakov
>

Re: ODBC doc needs to be updated

[Dmitriy Setrakyan]

Denis, it will be helpful to also file a ticket on this.

On Thu, Aug 17, 2017 at 1:20 PM, Denis Magda  wrote:

> Hi Igor,
>
> Could you spend some time walking through the doc and updating it
> considering the latest features and changes?
> https://apacheignite.readme.io/docs/quering-data <
> https://apacheignite.readme.io/docs/quering-data>
>
> For instance:
>
> 1. The configuration section [1] can be switched to the usage of DDL
> statements for tables creation and indexes definition.
>
> 2. Information about “_key” and “_value” should be wiped out because
> presently there is a way to set unique names for the whole key and value.
>
>
> [1] https://apacheignite.readme.io/docs/quering-data#
> configuring-ignite-cluster  io/docs/quering-data#configuring-ignite-cluster>
>
> —
> Denis

Re: Ignite2.1: Page eviction is not compatible with persistence when startup

[Dmitriy Setrakyan]

Agree. If we remove the exception though, we need to make sure to print out
the warning that the eviction policy will be ignored with Ignite native
persistence enabled.

On Thu, Aug 17, 2017 at 1:35 PM, Denis Magda  wrote:

> Dmitriy,
>
> >> Developers,
> >>
> >> Let me bring this to your attention. Why do we throw an exception if the
> >> user has both an eviction policy and the Ignite persistence configured?
> Why
> >> don’t we simply ignore the eviction policy printing a warning and
> proceed
> >> with the node startup?
> >>
> >
> > Denis, any reason one approach is better than another?
>
> The user doesn’t need to struggle with a bout of failures once he enable
> the Ignite persistence. This specific user had the memory policy configured
> before and once he enabled the disk he got extra exception he has to deal
> with. It shouldn’t work this way.
>
> In any case, the exception’s message doesn’t explain how to overcome the
> issue and has to be improved:
>
> Caused by: class org.apache.ignite.IgniteCheckedException: Page eviction
> is
> not compatible with persistence: 1G_Region
>
> —
> Denis

Re: Redesigning Ignite's site Look & Feel

[Dmitriy Setrakyan]

On Thu, Aug 17, 2017 at 1:45 PM, Denis Magda  wrote:

> Trying to request a staging server from INFRA:
> https://issues.apache.org/jira/browse/INFRA-14899 <
> https://issues.apache.org/jira/browse/INFRA-14899>
>

What is wrong with using Christos' laptop? :)


>
> —
> Denis
>
> > On Aug 17, 2017, at 1:11 AM, Christos Erotocritou 
> wrote:
> >
> > Hello, try again now. This is hosted currently on my local machine until
> we move onto a staging server.
> >
> >
> >> On 17 Aug 2017, at 02:58, Alexey Kuznetsov 
> wrote:
> >>
> >> I cannot open this url.
> >> It is working?
> >>
> >> On Thu, Aug 17, 2017 at 8:16 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> >> wrote:
> >>
> >>> Looks very cool!
> >>>
> >>> My first comment is that the home page looks different in Windows Edge
> from
> >>> Mac. For example, the font of the "Memory-Centric Data Platform" is
> >>> different and bold is not visible in Windows.
> >>>
> >>> D.
> >>>
> >>> On Wed, Aug 16, 2017 at 4:30 PM, Denis Magda 
> wrote:
> >>>
>  Igniters,
> 
>  Some of us have been working on the new design for the site. That’s an
>  early draft we have. Feel free to share the suggestions:
>  http://81.156.86.10:8081/index.html  index.html
> 
> 
>  In the meanwhile, is it possible to request a staging environment for
> the
>  site from INFRA? There are so many changes we need to apply to CSS
> files,
>  menus, pages, etc. that we have to create a branch to incorporate them
> >>> and
>  check the results on the staging environment.
> 
>  —
>  Denis
> >>>
> >>
> >>
> >>
> >> --
> >> Alexey Kuznetsov
> >
>
>

Re: How a new index is built in runtime?

[Dmitriy Setrakyan]

On Thu, Aug 17, 2017 at 1:47 PM, Yakov Zhdanov  wrote:

> Denis, you are absolutely right. Iterating through millions of objects is
> indeed a very complex task. Ignite casts a secret spell instead.\
>

Well, Ignite should cast a secret spell and do it in parallel, across
multiple threads. This will make index creation faster, but will occupy
more cores. Perhaps we should have both, single-threaded and multi-threaded
options.


>
> --Yakov
>
> 2017-08-17 21:26 GMT+01:00 Denis Magda :
>
> > Alex P., Vladimir,
> >
> > Having CREATE INDEX command we can define indexes in runtime. However,
> > it’s unclear how a new index is built.
> >
> > Let’s imagine I have a field “name” that was in Person’s model for a
> while
> > and there are millions of such objects in the cluster. Now I turned the
> > field into the index in runtime. How Ignite is going to built the index?
> Do
> > we iterate over the millions of objects with the field in background or
> at
> > the time of the CREATE INDEX execution blocking the latter? Or is there
> > more sophisticated process?
> >
> > —
> > Denis
>

Re: SQL usability issues

[Dmitriy Setrakyan]

On Thu, Aug 17, 2017 at 2:22 PM, Denis Magda  wrote:

> No I see what you tried to convey. Here it is the doc:
> https://apacheignite.readme.io/docs/sql-tooling <
> https://apacheignite.readme.io/docs/sql-tooling>
>

Looks very nice. I think we should separate all the SQL-related
documentation into its own separate SQL section.


> —
> Denis
>
> > On Aug 16, 2017, at 6:06 PM, Dmitriy Setrakyan 
> wrote:
> >
> > Denis,
> >
> > I agree with Val. I am OK with using DBeaver as an example, but the name
> of
> > the doc should be generic and the doc should be updated stating that
> > DBeaver is just one example of how Ignite can be used with generic SQL
> > viewer tools.
> >
> > D.
> >
> > On Wed, Aug 16, 2017 at 6:01 PM, Denis Magda  wrote:
> >
> >> Val,
> >>
> >> This article is generic:
> >> https://apacheignite.readme.io/docs/getting-started-sql <
> >> https://apacheignite.readme.io/docs/getting-started-sql>
> >>
> >> DBeaver, Pentaho and Tableau are given as examples.
> >>
> >> If the description or introduction of the tool should sound clearer
> please
> >> suggest your version.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 16, 2017, at 5:55 PM, Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com> wrote:
> >>>
> >>> Denis,
> >>>
> >>> I think this article should be more generic, describing how to connect
> a
> >>> generic JDBC tool to Ignite (probably using DBeaver as an example).
> >>>
> >>> Right now it looks like out of all such tools we support only DBeaver
> for
> >>> some reason :)
> >>>
> >>> What do you think?
> >>>
> >>> -Val
> >>>
> >>> On Wed, Aug 16, 2017 at 1:49 PM, Denis Magda 
> wrote:
> >>>
>  Igniters,
> 
>  I’ve prepared the getting started basing on DBeaver tool:
>  https://apacheignite-tools.readme.io/docs/dbeaver <
>  https://apacheignite-tools.readme.io/docs/dbeaver>
> 
>  That guide is based on the generic one made out by Akmal:
>  https://apacheignite.readme.io/docs/getting-started-sql <
>  https://apacheignite.readme.io/docs/getting-started-sql>
> 
>  During the work on the guide we faced one more issue that prevents us
> >> from
>  using the bulk inserts:
>  https://issues.apache.org/jira/browse/IGNITE-6092 <
>  https://issues.apache.org/jira/browse/IGNITE-6092>
> 
>  *Alexander P.*, *Vladimir*, how difficult is to support this feature?
> 
>  —
>  Denis
> 
> > On Aug 13, 2017, at 9:20 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
>  wrote:
> >
> > On Fri, Aug 11, 2017 at 7:31 PM, Denis Magda 
>  wrote:
> >
> >> Dmitriy,
> >>
> >> That's the documentation that shows how to configure Pentaho via
> JDBC:
> >> https://apacheignite-tools.readme.io/v2.1/docs/pentaho
> >
> >
> > I am not sure I agree. Why would I look at Pentaho integration when
>  trying
> > to configure a generic SQL viewer tool? We need a generic
> documentation
> > section for configuring JDBC/ODBC drivers with 3rd party tools.
> >
> >
> >>
> >>
> >> Plus on the tools' domain you can see Tableau based ODBC example.
> >>
> >> So, I'm not sure we need something else from the documentation
>  standpoint.
> >> I would rather add a direct reference from JDBC/ODBC docs to Pentaho
> >> and
> >> Tablea.
> >>
> >> Denis
> >>
> >> On Friday, August 11, 2017, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> >> wrote:
> >>
> >>> Igniters,
> >>>
> >>> I have tried to connect to Ignite from DBeaver [1], and realized
> that
> >> there
> >>> are some usability issues we need to address before the next
> release:
> >>>
> >>> 1. We need to have documentation on how to configure JDBC and ODBC
> >>> drivers with external SQL tools [2]
> >>> 2. You cannot highlight multiple SQL statements and run them
> >> together
> >>> [3]
> >>> 3. Commands like *DESCRIBE* or *SHOW* do not work [4]
> >>> 4. Schema, index, and table metadata is not displayed [5]. Looks
> >> like
> >>> this fix was already implemented.
> >>>
> >>> The links to the tickets are below.
> >>>
> >>> [1] http://dbeaver.jkiss.org/
> >>> [2] https://issues.apache.org/jira/browse/IGNITE-6048
> >>> [3] https://issues.apache.org/jira/browse/IGNITE-6046
> >>> [4] https://issues.apache.org/jira/browse/IGNITE-6047
> >>> [5] https://issues.apache.org/jira/browse/IGNITE-5233
> >>>
> >>> D.
> >>>
> >>
> 
> 
> >>
> >>
>
>

Re: ERROR: Heuristic transaction failure.

[Valentin Kulichenko]

Hi Usein,

Which Java version do you have? There was a similar thread already where
this exception was fixed by upgrading to the latest one:
http://apache-ignite-users.70518.x6.nabble.com/Caused-by-org-h2-jdbc-JdbcSQLException-General-error-quot-java-lang-IllegalMonitorStateException-Attt-td15684.html

Can you try to upgrade as well? If you confirm that it indeed helps, then
it needs to be documented.

-Val

On Thu, Aug 17, 2017 at 3:58 AM, Usein Faradzhev 
wrote:

> Hello.
>
>
>
> We are trying to use the Ignite Memory File System and sometimes Ignite
> can’t write file to IGFS and can’t read. What is this happens?
>
> Below is an example for Cloudera Quick Start VM 5.10.0 and error, also
> configuration and full log in attachments. This problems arise on our
> cluster with CentOS 7 and CDH 5.11.1 too.
>
>
>
> In-Memory Hadoop Accelerator:
>
> Version2.1.0
>
> Date  2017-07-27
>
> File http://apache-mirror.rbc.ru/pub/apache//ignite/2.1.0/
> apache-ignite-hadoop-2.1.0-bin.zip
> 
>
>
>
> [cloudera@quickstart ~]$ ls -l dtm_ekp_scoring_plan_oper75.csv
>
> -rw-r--r-- 1 cloudera cloudera 19579883 Aug 16 03:53
> dtm_ekp_scoring_plan_oper75.csv
>
>
>
> [cloudera@quickstart ~]$ hdfs dfs -mkdir -p igfs://igfs@/user/cloudera/
> dtm_ekp_scoring_plan_oper/
>
> [cloudera@quickstart ~]$ hdfs dfs -put dtm_ekp_scoring_plan_oper75.csv
> igfs://igfs@/user/cloudera/dtm_ekp_scoring_plan_oper/
>
> put: Failed to flush data during stream close [path=/user/cloudera/dtm_ekp_
> scoring_plan_oper/dtm_ekp_scoring_plan_oper75.csv._COPYING_,
> fileInfo=IgfsFileInfo [len=0, blockSize=65536, 
> lockId=4600eafed51-15b0cff9-0c6e-459c-8c1e-1d8f59d102e6,
> affKey=null, fileMap=IgfsFileMap [ranges=null], evictExclude=true]]
>
>
>
>
>
> [2017-08-17 03:13:07,951][ERROR][igfs-#47%null%][GridNearTxLocal]
> Heuristic transaction failure.
>
> class org.apache.ignite.internal.transactions.
> IgniteTxHeuristicCheckedException: Failed to locally write to cache (all
> transaction entries will be invalidated, however there was a window when
> entries for this transaction were visible to others): GridNearTxLocal
> [mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping
> [entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
> [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
> affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114,
> txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
> [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
> affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114],
> val=[op=CREATE, val=CacheObjectByteArrayImpl [arrLen=65536]],
> prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
> entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
> explicitVer=null, dhtVer=null, filters=[], filtersPassed=false,
> filtersSet=true, entry=GridDhtCacheEntry [rdrs=[], part=954, 
> super=GridDistributedCacheEntry
> [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=954,
> val=IgfsBlockKey [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be,
> blockId=130, affKey=null, evictExclude=true], hasValBytes=true], val=null,
> startVer=1502964754897, ver=GridCacheVersion [topVer=11755,
> order=1502964754897, nodeOrder=1], hash=236544549, 
> extras=GridCacheMvccEntryExtras
> [mvcc=GridCacheMvcc [locs=[GridCacheMvccCandidate
> [nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, ver=GridCacheVersion
> [topVer=11755, order=1502964754896, nodeOrder=1], threadId=69, id=152,
> topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], reentry=null,
> otherNodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6,
> otherVer=GridCacheVersion [topVer=11755, order=1502964754896,
> nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null,
> serOrder=null, key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
> [fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
> affKey=null, evictExclude=true], hasValBytes=true],
> masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_
> implicit=1|dht_local=1|near_local=0|removed=0|read=0, prevVer=null,
> nextVer=null]], rmts=null]], flags=2]]], prepared=1, locked=false,
> nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, locMapped=false,
> expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
> serReadVer=null, xidVer=GridCacheVersion [topVer=11755,
> order=1502964754896, nodeOrder=1]]], explicitLock=false, dhtVer=null,
> last=false, nearEntries=0, clientFirst=false, 
> node=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6]],
> nearLocallyMapped=false, colocatedLocallyMapped=true, needCheckBackup=null,
> hasRemoteLocks=false, thread=igfs-#47%null%, 
> mappings=IgniteTxMappingsSingleImpl
> [mapping=GridDistributedTxMapping [entries=[IgniteTxEntry
> [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
> [fileId=1600eafed51-cd65

Re: Failure to deserialize simple model object

[Valentin Kulichenko]

Guys,

Does anyone has ideas?

-Val

On Mon, Aug 14, 2017 at 4:33 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Cross-posting to dev
>
> Folks,
>
> I'm confused by the issue discussed in this thread.
>
> Here is the scenario:
> - Start server node with a cache with POJO store configured. There is one
> type declared, read-through enabled.
> - Start client node and execute get() for a key that exists in underlying
> DB.
> - During deserialization on the client, 'Requesting mapping from grid
> failed for' exception is thrown.
>
> Specifying the type explicitly in BinaryConfiguration solves the issue,
> and I think I understand technical reasons for this. But is this really
> expected? Is it possible to fix the issue without requiring to provide this
> configuration?
>
> I thought we do not require to provide types in configuration as long as
> there is only one platform involved, am I wrong? If yes, we need to
> identify scenarios when this documentation is required and document them.
>
> -Val
>
> On Mon, Aug 14, 2017 at 4:23 AM, franck102  wrote:
>
>> My bad, here is the whole project.
>>
>> Franck ignite-binary-sample.zip
>> > ignite-binary-sample.zip>
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Failure-to-deserialize-simple-model-object-
>> tp15440p16158.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: SQL usability issues

[Denis Magda]

No I see what you tried to convey. Here it is the doc:
https://apacheignite.readme.io/docs/sql-tooling 


—
Denis

> On Aug 16, 2017, at 6:06 PM, Dmitriy Setrakyan  wrote:
> 
> Denis,
> 
> I agree with Val. I am OK with using DBeaver as an example, but the name of
> the doc should be generic and the doc should be updated stating that
> DBeaver is just one example of how Ignite can be used with generic SQL
> viewer tools.
> 
> D.
> 
> On Wed, Aug 16, 2017 at 6:01 PM, Denis Magda  wrote:
> 
>> Val,
>> 
>> This article is generic:
>> https://apacheignite.readme.io/docs/getting-started-sql <
>> https://apacheignite.readme.io/docs/getting-started-sql>
>> 
>> DBeaver, Pentaho and Tableau are given as examples.
>> 
>> If the description or introduction of the tool should sound clearer please
>> suggest your version.
>> 
>> —
>> Denis
>> 
>>> On Aug 16, 2017, at 5:55 PM, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>> 
>>> Denis,
>>> 
>>> I think this article should be more generic, describing how to connect a
>>> generic JDBC tool to Ignite (probably using DBeaver as an example).
>>> 
>>> Right now it looks like out of all such tools we support only DBeaver for
>>> some reason :)
>>> 
>>> What do you think?
>>> 
>>> -Val
>>> 
>>> On Wed, Aug 16, 2017 at 1:49 PM, Denis Magda  wrote:
>>> 
 Igniters,
 
 I’ve prepared the getting started basing on DBeaver tool:
 https://apacheignite-tools.readme.io/docs/dbeaver <
 https://apacheignite-tools.readme.io/docs/dbeaver>
 
 That guide is based on the generic one made out by Akmal:
 https://apacheignite.readme.io/docs/getting-started-sql <
 https://apacheignite.readme.io/docs/getting-started-sql>
 
 During the work on the guide we faced one more issue that prevents us
>> from
 using the bulk inserts:
 https://issues.apache.org/jira/browse/IGNITE-6092 <
 https://issues.apache.org/jira/browse/IGNITE-6092>
 
 *Alexander P.*, *Vladimir*, how difficult is to support this feature?
 
 —
 Denis
 
> On Aug 13, 2017, at 9:20 AM, Dmitriy Setrakyan 
 wrote:
> 
> On Fri, Aug 11, 2017 at 7:31 PM, Denis Magda 
 wrote:
> 
>> Dmitriy,
>> 
>> That's the documentation that shows how to configure Pentaho via JDBC:
>> https://apacheignite-tools.readme.io/v2.1/docs/pentaho
> 
> 
> I am not sure I agree. Why would I look at Pentaho integration when
 trying
> to configure a generic SQL viewer tool? We need a generic documentation
> section for configuring JDBC/ODBC drivers with 3rd party tools.
> 
> 
>> 
>> 
>> Plus on the tools' domain you can see Tableau based ODBC example.
>> 
>> So, I'm not sure we need something else from the documentation
 standpoint.
>> I would rather add a direct reference from JDBC/ODBC docs to Pentaho
>> and
>> Tablea.
>> 
>> Denis
>> 
>> On Friday, August 11, 2017, Dmitriy Setrakyan 
>> wrote:
>> 
>>> Igniters,
>>> 
>>> I have tried to connect to Ignite from DBeaver [1], and realized that
>> there
>>> are some usability issues we need to address before the next release:
>>> 
>>> 1. We need to have documentation on how to configure JDBC and ODBC
>>> drivers with external SQL tools [2]
>>> 2. You cannot highlight multiple SQL statements and run them
>> together
>>> [3]
>>> 3. Commands like *DESCRIBE* or *SHOW* do not work [4]
>>> 4. Schema, index, and table metadata is not displayed [5]. Looks
>> like
>>> this fix was already implemented.
>>> 
>>> The links to the tickets are below.
>>> 
>>> [1] http://dbeaver.jkiss.org/
>>> [2] https://issues.apache.org/jira/browse/IGNITE-6048
>>> [3] https://issues.apache.org/jira/browse/IGNITE-6046
>>> [4] https://issues.apache.org/jira/browse/IGNITE-6047
>>> [5] https://issues.apache.org/jira/browse/IGNITE-5233
>>> 
>>> D.
>>> 
>> 
 
 
>> 
>> 

[GitHub] ignite pull request #2475: added throw runtime exception when mvcc addLocal

[voipp]

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2475

added throw runtime exception when mvcc addLocal



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite ignite-5714-3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2475


commit 47b9f668b7ab76d07b01cd89f833bd34d99112c2
Author: voipp 
Date:   2017-08-17T14:25:42Z

added throw runtime exception when mvcc addLocal




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2474: added throw runtime exception when mvcc addLocal

[voipp]

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2474

added throw runtime exception when mvcc addLocal



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite ignite-5714-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2474.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2474


commit 284b1235168f470ba1bbd462bb52b467a94a8a35
Author: voipp 
Date:   2017-08-17T14:21:17Z

added throw runtime exception when mvcc addLocal




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2473: added throw runtime exception when candidate add

[voipp]

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2473

added throw runtime exception when candidate add



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite ignite-5714-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2473


commit ed22986a9e10a33adc04e41df0b409caa522d871
Author: voipp 
Date:   2017-08-17T14:17:24Z

added throw runtime exception when candidate add




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: How a new index is built in runtime?

[Yakov Zhdanov]

Denis, you are absolutely right. Iterating through millions of objects is
indeed a very complex task. Ignite casts a secret spell instead.

--Yakov

2017-08-17 21:26 GMT+01:00 Denis Magda :

> Alex P., Vladimir,
>
> Having CREATE INDEX command we can define indexes in runtime. However,
> it’s unclear how a new index is built.
>
> Let’s imagine I have a field “name” that was in Person’s model for a while
> and there are millions of such objects in the cluster. Now I turned the
> field into the index in runtime. How Ignite is going to built the index? Do
> we iterate over the millions of objects with the field in background or at
> the time of the CREATE INDEX execution blocking the latter? Or is there
> more sophisticated process?
>
> —
> Denis

Re: Redesigning Ignite's site Look & Feel

[Denis Magda]

Trying to request a staging server from INFRA:
https://issues.apache.org/jira/browse/INFRA-14899 


—
Denis

> On Aug 17, 2017, at 1:11 AM, Christos Erotocritou  
> wrote:
> 
> Hello, try again now. This is hosted currently on my local machine until we 
> move onto a staging server.
> 
> 
>> On 17 Aug 2017, at 02:58, Alexey Kuznetsov  wrote:
>> 
>> I cannot open this url.
>> It is working?
>> 
>> On Thu, Aug 17, 2017 at 8:16 AM, Dmitriy Setrakyan 
>> wrote:
>> 
>>> Looks very cool!
>>> 
>>> My first comment is that the home page looks different in Windows Edge from
>>> Mac. For example, the font of the "Memory-Centric Data Platform" is
>>> different and bold is not visible in Windows.
>>> 
>>> D.
>>> 
>>> On Wed, Aug 16, 2017 at 4:30 PM, Denis Magda  wrote:
>>> 
 Igniters,
 
 Some of us have been working on the new design for the site. That’s an
 early draft we have. Feel free to share the suggestions:
 http://81.156.86.10:8081/index.html >> and
 check the results on the staging environment.
 
 —
 Denis
>>> 
>> 
>> 
>> 
>> -- 
>> Alexey Kuznetsov
> 

Re: Ignite Add-on/Extension Request: Please add GA Grid (beta) to Ignite website

[Denis Magda]

Hi Turik,

Sure, we will do this with pleasure.

Prachi, could you add the component to the mentioned page?

—
Denis

> On Aug 16, 2017, at 9:16 PM, techbysample  wrote:
> 
> Forum Admin,
> 
> Please add project:
> 
> GA Grid (beta): Distributive in Memory Genetic Algorithm component for
> Ignite
> 
> to "Addons and Extensions" section of Apache Ignite website:
> 
> https://ignite.apache.org/addons.html
> 
> 
>  
> 
> (F)itness Calculation, (C)rossover, and (M)utation operations are modeled as
> a ComputeTask for distributive behavior. The ComputeTask is split into
> multiple ComputeJobs, (ie: Fn,Cn,Mn) assigned to respective nodes, and
> executed in parallel.
> 
> All of these ComputeTasks leverage Apache Ignite's Affinity Colocation to
> route ComputeJobs to respective nodes where Chromosomes are stored in cache.
> 
> Additional information about GA Grid (beta) found here:
> 
> https://github.com/techbysample/gagrid 
> 
> Please advise.
> 
> Best,
> Turik Campbell
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Add-on-Extension-Request-Please-add-GA-Grid-beta-to-Ignite-website-tp20947.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: Ignite2.1: Page eviction is not compatible with persistence when startup

[Denis Magda]

Dmitriy,

>> Developers,
>> 
>> Let me bring this to your attention. Why do we throw an exception if the
>> user has both an eviction policy and the Ignite persistence configured? Why
>> don’t we simply ignore the eviction policy printing a warning and proceed
>> with the node startup?
>> 
> 
> Denis, any reason one approach is better than another?

The user doesn’t need to struggle with a bout of failures once he enable the 
Ignite persistence. This specific user had the memory policy configured before 
and once he enabled the disk he got extra exception he has to deal with. It 
shouldn’t work this way. 

In any case, the exception’s message doesn’t explain how to overcome the issue 
and has to be improved:

Caused by: class org.apache.ignite.IgniteCheckedException: Page eviction is
not compatible with persistence: 1G_Region

—
Denis

Re: .sha Release Distribution Policy

[Denis Magda]

Guys, 

Thanks for the confirmation and taking care of this.

—
Denis

> On Aug 17, 2017, at 1:32 AM, Sergey Kozlov  wrote:
> 
> Denis
> 
> Also we don't use .sha extension so we already follow that rules
> 
> On Thu, Aug 17, 2017 at 10:57 AM, Oleg Ostanin 
> wrote:
> 
>> Hi, Denis
>> 
>> Yes, we have a ticket that already takes this into account:
>> https://issues.apache.org/jira/browse/IGNITE-5817
>> I think we can create both sha-256 and sha-512 checksums.
>> 
>> Best regards
>> Oleg
>> 
>> On Thu, Aug 17, 2017 at 1:51 AM, Denis Magda  wrote:
>> 
>>> Igniters, especially the release managers,
>>> 
>>> Please consider these changes and recommendations for the next release.
>> Do
>>> we have any ticket that already takes this into account?
>>> 
>>> —
>>> Denis
>>> 
 Begin forwarded message:
 
 From: "Henk P. Penning" 
 Subject: .sha Release Distribution Policy
 Date: August 16, 2017 at 1:55:57 AM PDT
 To: 
 Reply-To: priv...@ignite.apache.org
 
 Hi PMC,
 
  The Release Distribution Policy[1] changed regarding .sha files.
  See under "Cryptographic Signatures and Checksums Requirements" [2].
 
 Old policy :
 
   -- use extension .sha for any SHA checksum (SHA-1, SHA-256, SHA-512)
 
 New policy :
 
-- use .sha1 for a SHA-1 checksum
-- use .sha256 for a SHA-256 checksum
-- use .sha512 for a SHA-512 checksum
-- [*] .sha should contain a SHA-1
 
 Why this change ?
 
-- Verifying a checksum under the old policy is/was not handy.
   You have to inspect the .sha to find out which algorithm
   should be used ; or try them all (SHA-1, SHA256, etc).
   The new scheme avoids this ambiguity.
-- The last point[*] was only added for clarity. Most of the
   old, stale .sha's contain a SHA-1. The relatively new .sha's
   contain a SHA-512. The expectation is that the last catagory
>> will
   disappear, when active projects adapt to the 'new' convention.
 
 Impact :
 
-- Should be none ; many projects already use the 'new' convention.
-- Please ask your release managers to use .sha1, .sha256, .sha512
   instead of the .sha extension.
-- Please fix your build-tools if you have any.
 
 Piggyback :
 
-- The policy requires a .md5 for every package ;
   providing a .sha512 is recommended.
   Since MD5 is essentially broken, it is to be expected that
   in the future a .sha512 will be required.
   Perhaps it is wize to start providing .sha512's
   with your releases if you do not already do so.
 
-- Visit http://mirror-vm.apache.org/checker/
   to check the health of your /dist/-area ;
   my stuff ; any feedback is most welcome.
 
 Thanks ; regards,
 
 Henk Penning
 
  [1] http://www.apache.org/dev/release-distribution
  [2] http://www.apache.org/dev/release-distribution#sigs-and-sums
 
 
 Henk P. Penning ; apache.org infrastructure volunteer.
 he...@apache.org ; http://mirror-vm.apache.org/~henkp/
>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com

Re: new website tickets

[Denis Magda]

Thanks Akmal, I’ve provide more details in the ticket. Let us know if you have 
any questions.

—
Denis

> On Aug 17, 2017, at 11:51 AM, Akmal Chaudhri  
> wrote:
> 
> I shall get started on this tomorrow. Hopefully finish it fairly quickly.
> 
> Thank you.
> 
> On 11 August 2017 at 20:45, Denis Magda  wrote:
> 
>> Akmal,
>> 
>> Thanks for stepping in! Considering that you’ve been working on the SQL
>> getting started, I’ve reassigned the general getting guide task on you and
>> provide more details and suggestions for it:
>> https://issues.apache.org/jira/browse/IGNITE-6041 <
>> https://issues.apache.org/jira/browse/IGNITE-6041>
>> 
>> Please let me know if you have any questions.
>> 
>> I’ll take care of the rest.
>> 
>> —
>> Denis
>> 
>>> On Aug 11, 2017, at 10:40 AM, Dmitriy Setrakyan 
>> wrote:
>>> 
>>> On Fri, Aug 11, 2017 at 10:38 AM, Akmal Chaudhri <
>>> akmal.chaud...@gridgain.com> wrote:
>>> 
 Dimitriy,
 
 I will take a look at those and pick-up the documentation tickets early
 next week.
 
 
>>> Awesome! Feel free to assign any tickets to yourself.
>>> 
>>> 
 Thanks.
 
 On 11 August 2017 at 02:34, Dmitriy Setrakyan 
 wrote:
 
> Igniters,
> 
> I have filed several website tickets to reflect the addition of
 persistence
> in the latest 2.1 release:
> 
> https://issues.apache.org/jira/browse/IGNITE-6036
> https://issues.apache.org/jira/browse/IGNITE-6037
> https://issues.apache.org/jira/browse/IGNITE-6038
> https://issues.apache.org/jira/browse/IGNITE-6039
> https://issues.apache.org/jira/browse/IGNITE-6039
> https://issues.apache.org/jira/browse/IGNITE-6040
> https://issues.apache.org/jira/browse/IGNITE-6041
> 
> The idea behind this ticket to to improve user experience,
>> documentation
> and feature descriptions. If anyone can think of more changes or
 additions,
> please reply in this thread.
> 
> Denis, given your natural affinity and love towards the website changes
> (pun intended) I have assigned all of them to you, but feel free to
> unassign yourself.
> 
> D.
> 
 
>> 
>> 

How a new index is built in runtime?

[Denis Magda]

Alex P., Vladimir,

Having CREATE INDEX command we can define indexes in runtime. However, it’s 
unclear how a new index is built.

Let’s imagine I have a field “name” that was in Person’s model for a while and 
there are millions of such objects in the cluster. Now I turned the field into 
the index in runtime. How Ignite is going to built the index? Do we iterate 
over the millions of objects with the field in background or at the time of the 
CREATE INDEX execution blocking the latter? Or is there more sophisticated 
process?

—
Denis 

ODBC doc needs to be updated

[Denis Magda]

Hi Igor, 

Could you spend some time walking through the doc and updating it considering 
the latest features and changes?
https://apacheignite.readme.io/docs/quering-data 


For instance:

1. The configuration section [1] can be switched to the usage of DDL statements 
for tables creation and indexes definition.

2. Information about “_key” and “_value” should be wiped out because presently 
there is a way to set unique names for the whole key and value.


[1] https://apacheignite.readme.io/docs/quering-data#configuring-ignite-cluster 


—
Denis

Re: SQL Getting Started Guide

[Denis Magda]

Igor,

Would you mind helping us to test the snippets? I do believe it will take you a 
couple of minutes since the environment is set up on your side.

—
Denis

> On Aug 17, 2017, at 5:37 AM, Igor Sapego  wrote:
> 
> Akmal,
> 
> I work on the ODBC driver and would be glad to help. You can send
> mails with your requests to devlist and I will try to help as much as I
> can.
> 
> Best Regards,
> Igor
> 
> On Thu, Aug 17, 2017 at 3:28 PM, Akmal Chaudhri > wrote:
> 
>> When I have the opportunity to test the ODBC code and ensure that it
>> performs, I will add it to GH. At the moment, just snippets exist. If
>> anyone can assist with this, I would appreciate it, as I have no experience
>> with ODBC. I made some efforts to build the ODBC driver and have had some
>> success on Windows and Linux, but have been unable to run any C++ code.
>> 
>> On 16 August 2017 at 21:41, Denis Magda  wrote:
>> 
>>> Igniters,
>>> 
>>> The SQL getting started guide is ready and published:
>>> https://apacheignite.readme.io/v2.1/docs/getting-started-sql
>>> 
>>> Akmal thanks for your efforts and please apply the following minor
>> changes:
>>> * Add ODBC source files to your GitHub project.
>>> * Add a flat SQL file (script) to the same project that will include all
>>> the pure SQL statements used in the guide.
>>> 
>>> Basing on this guide I’ve prepared and release a guide for DBeaver SQL
>>> tool:
>>> https://apacheignite-tools.readme.io/docs/dbeaver
>>> 
>>> —
>>> Denis
>>> 
>>> On Aug 11, 2017, at 3:17 PM, Denis Magda  wrote:
>>> 
>>> Akmal,
>>> 
>>> Good start! Please consider the following feedback:
>>> 
>>> 1. Connectivity section. Emphasize that ignite-core.jar has to be copied
>>> to the classpath of an app or tool and give a link to more advanced JDBC
>>> Thin driver documentation if the one needs more details. Probably,
>> similar
>>> steps should be brought up for the ODBC once it’s ready.
>>> 
>>> 2. Make the first letters of city and people tables uppercase: city ->
>>> City, people -> People
>>> 
>>> 3. Rename INSERT section to “Inserting Data”, SELECT to “Querying Data”,
>>> UPDATE -> “Modifying Data”, DELETE to “Removing Data” or to better
>>> alternatives.
>>> 
>>> 4. Show how to insert at least 2 cities and 5 people in the INSERT
>> section.
>>> 
>>> —
>>> Denis
>>> 
>>> On Aug 11, 2017, at 10:42 AM, Akmal Chaudhri <
>> akmal.chaud...@gridgain.com>
>>> wrote:
>>> 
>>> Denis, All
>>> 
>>> I have made some progress with the documentation:
>>> 
>>> https://apacheignite.readme.io/v2.1/docs/getting-started-sql <
>>> https://apacheignite.readme.io/v2.1/docs/getting-started-sql>
>>> 
>>> ODBC connectivity and examples are missing, but should be completed very
>>> soon.
>>> 
>>> Community feedback welcome.
>>> 
>>> Thanks.
>>> 
>>> On 31 July 2017 at 19:02, Denis Magda >> mailto:dma...@apache.org >> wrote:
>>> Igniters,
>>> 
>>> There is a lot of SQL related documentation available for Ignite.
>> However,
>>> the one can get lost in it when he/she does her first steps on a learning
>>> path. It’s time to simplify these are first steps with a clear and
>>> straightforward getting started guide. I’ve put the requirements into
>> this
>>> JIRA ticket:
>>> https://issues.apache.org/jira/browse/IGNITE-5886 <
>>> https://issues.apache.org/jira/browse/IGNITE-5886>
>>> 
>>> Please share your thoughts in JIRA or in this discussion.
>>> 
>>> Akmal, who perfected his technical writer skills working for IBM,
>>> volunteered to take over this task. Akmal, please create an account in
>>> Ignite JIRA and share it with us. You’ll be able to assign the task on
>>> yourself afterwards.
>>> https://issues.apache.org/jira/projects/IGNITE <
>> https://issues.apache.org/
>>> jira/projects/IGNITE>
>>> 
>>> Moreover, once the guide is ready Prachi will use it as a basis for the
>>> new SQL screencast we need to produce.
>>> 
>>> —
>>> Denis
>>> 
>>> 
>>> 
>>> 
>> 

Re: Ignite not friendly for Monitoring

[Denis Magda]

Vladimir,

I would disagree. In IGNITE-5620 we’re going to introduce some constant error 
codes and prepare a sheet that will elaborate on every error. That’s a part of 
bigger endeavor when the whole platform should be covered by special unique IDs 
for errors, warning and events. 

Now, we need to agree at least on the IDs range for SQL.

—
Denis

> On Aug 15, 2017, at 11:10 PM, Vladimir Ozerov  wrote:
> 
> Denis,
> 
> IGNITE-5620 is completely different thing. Let's do not mix cluster
> monitoring and parser errors.
> 
> ср, 16 авг. 2017 г. в 2:57, Denis Magda :
> 
>> Alexey,
>> 
>> Didn’t know that such an improvement as consistent IDs for errors and
>> events can be used as an integration point with the DevOps tools. Thanks
>> for sharing your experience with us.
>> 
>> Would you step in as a architect for this task and make out a JIRA ticket
>> with all the required information.
>> 
>> In general, we’ve already planned to do something around this starting
>> with SQL:
>> https://issues.apache.org/jira/browse/IGNITE-5620 <
>> https://issues.apache.org/jira/browse/IGNITE-5620>
>> 
>> It makes sense to consider your input before the work on IGNITE-5620 is
>> started.
>> 
>> —
>> Denis
>> 
>>> On Aug 15, 2017, at 10:56 AM, Alexey Kukushkin <
>> alexeykukush...@yahoo.com.INVALID> wrote:
>>> 
>>> Hi Alexey,
>>> A nice thing about delegating alerting to 3rd party enterprise systems
>> is that those systems already deal with lots of things including
>> distributed apps.
>>> What is needed from Ignite is to consistently write to log files (again
>> that means stable event IDs, proper event granularity, no repetition,
>> documentation). This would be 3rd party monitoring system's responsibility
>> to monitor log files on all nodes, filter, aggregate, process, visualize
>> and notify on events.
>>> How a monitoring tool would deal with an event like "node left":
>>> The only thing needed from Ignite is to write an entry like below to log
>> files on all Ignite servers. In this example 3300 identifies this "node
>> left" event and will never change in the future even if text description
>> changes:
>>> [2017-09-01 10:00:14] [WARN] 3300 Node DF2345F-XCVDS4-34ETJH left the
>> cluster
>>> Then we document somewhere on the web that Ignite has event 3300 and it
>> means a node left the cluster. Maybe provide documentation how to deal with
>> it. Some examples:Oracle Web Cache events:
>> https://docs.oracle.com/cd/B14099_19/caching.1012/b14046/event.htm#sthref2393MS
>> SQL Server events:
>> https://msdn.microsoft.com/en-us/library/cc645603(v=sql.105).aspx
>>> That is all for Ignite! Everything else is handled by specific
>> monitoring system configured by DevOps on the customer side.
>>> Basing on the Ignite documentation similar to above, DevOps of a company
>> where Ignite is going to be used will configure their monitoring system to
>> understand Ignite events. Consider the "node left" event as an example.
>>> - This event is output on every node but DevOps do not want to be
>> notified many times. To address this, they will build an "Ignite model"
>> where there will be a parent-child dependency between components "Ignite
>> Cluster" and "Ignite Node". For example, this is how you do it in Nagios:
>> https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/dependencies.html
>> and this is how you do it in Microsoft SCSM:
>> https://docs.microsoft.com/en-us/system-center/scsm/auth-classes. Then
>> DevOps will configure "node left" monitors in SCSM (or a "checks" in
>> Nagios) for parent "Ignite Cluster" and child "Ignite Service" components.
>> State change (OK -> WARNING) and notification (email, SMS, whatever) will
>> be configured only for the "Ignite Cluster"'s "node left" monitor.- Now
>> suppose a node left. The "node left" monitor (that uses log file monitoring
>> plugin) on "Ignite Node" will detect the event and pass it to the parent.
>> This will trigger "Ignite Cluster" state change from OK to WARNING and send
>> a notification. No more notification will be sent unless the "Ignite
>> Cluster" state is reset back to OK, which happens either manually or on
>> timeout or automatically on "node joined".
>>> This was just FYI. We, Ignite developers, do not care about how
>> monitoring works - this is responsibility of customer's DevOps. Our
>> responsibility is consistent event logging.
>>> Thank you!
>>> 
>>> 
>>> Best regards, Alexey
>>> 
>>> 
>>> On Tuesday, August 15, 2017, 6:16:25 PM GMT+3, Alexey Kuznetsov <
>> akuznet...@apache.org> wrote:
>>> 
>>> Alexey,
>>> 
>>> How you are going to deal with distributed nature of Ignite cluster?
>>> And how do you propose handle nodes restart / stop?
>>> 
>>> On Tue, Aug 15, 2017 at 9:12 PM, Alexey Kukushkin <
>>> alexeykukush...@yahoo.com.invalid> wrote:
>>> 
 Hi Denis,
 Monitoring tools simply watch event logs for patterns (regex in case of
 unstructured logs like text files). A stable (not changing in new
>> releases)
 event ID identif

Re: new website tickets

[Akmal Chaudhri]

I shall get started on this tomorrow. Hopefully finish it fairly quickly.

Thank you.

On 11 August 2017 at 20:45, Denis Magda  wrote:

> Akmal,
>
> Thanks for stepping in! Considering that you’ve been working on the SQL
> getting started, I’ve reassigned the general getting guide task on you and
> provide more details and suggestions for it:
> https://issues.apache.org/jira/browse/IGNITE-6041 <
> https://issues.apache.org/jira/browse/IGNITE-6041>
>
> Please let me know if you have any questions.
>
> I’ll take care of the rest.
>
> —
> Denis
>
> > On Aug 11, 2017, at 10:40 AM, Dmitriy Setrakyan 
> wrote:
> >
> > On Fri, Aug 11, 2017 at 10:38 AM, Akmal Chaudhri <
> > akmal.chaud...@gridgain.com> wrote:
> >
> >> Dimitriy,
> >>
> >> I will take a look at those and pick-up the documentation tickets early
> >> next week.
> >>
> >>
> > Awesome! Feel free to assign any tickets to yourself.
> >
> >
> >> Thanks.
> >>
> >> On 11 August 2017 at 02:34, Dmitriy Setrakyan 
> >> wrote:
> >>
> >>> Igniters,
> >>>
> >>> I have filed several website tickets to reflect the addition of
> >> persistence
> >>> in the latest 2.1 release:
> >>>
> >>> https://issues.apache.org/jira/browse/IGNITE-6036
> >>> https://issues.apache.org/jira/browse/IGNITE-6037
> >>> https://issues.apache.org/jira/browse/IGNITE-6038
> >>> https://issues.apache.org/jira/browse/IGNITE-6039
> >>> https://issues.apache.org/jira/browse/IGNITE-6039
> >>> https://issues.apache.org/jira/browse/IGNITE-6040
> >>> https://issues.apache.org/jira/browse/IGNITE-6041
> >>>
> >>> The idea behind this ticket to to improve user experience,
> documentation
> >>> and feature descriptions. If anyone can think of more changes or
> >> additions,
> >>> please reply in this thread.
> >>>
> >>> Denis, given your natural affinity and love towards the website changes
> >>> (pun intended) I have assigned all of them to you, but feel free to
> >>> unassign yourself.
> >>>
> >>> D.
> >>>
> >>
>
>

Re: Ignite: configuration changes at runtime

[Yakov Zhdanov]

>I don't like the idea of intermixing changeable properties with constant
>ones in the configuration. Moreover, the standard configuration class is
>local to the node, while changing the config properties at runtime may be a
>distributed operation that needs to be propagated across the cluster.

I don't 100% agree here. Let's discuss.

1. I would not mess with propagation for now. Propagation can be handled by
special methods or by Visor/WebConsole
2. If we choose to create separate managing objects we end up in creating
that for each configuration type we have now. And we have quite great
number of config objects. For me personally, it seems more convenient to
have configuration object a place to change and retrieve current
configuration. We can have separate method to apply the changes (on
configuration object or object that is configured by that configuration
object).

>How about we add getters and setters to IgniteConfigurationRuntime
>interface. BTW, this interface should also become an MBean, so it can be
>managed by external tools.

Again, if we introduce separate runtime configurable object this seems to
be very confusing. After I change the property via that object, should
corresponding property change in let's say IgniteConfiguration?

--Yakov

[GitHub] ignite pull request #2472: Ignite 1.9.6

[ntikhonov]

GitHub user ntikhonov opened a pull request:

https://github.com/apache/ignite/pull/2472

Ignite 1.9.6



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-1.9.6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2472.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2472


commit 5dfe16f7e91374008b9f6dfbb899364f5b2e1164
Author: Denis Magda 
Date:   2017-02-14T06:03:30Z

IGNITE-4159: using logger instead of system.out.println
(cherry picked from commit b9bf77c)

commit 6e596d1ef426b66abd866d011a8f5cf5c5c25124
Author: Andrey V. Mashenkov 
Date:   2017-04-06T11:43:50Z

IGNITE-4832: Prevent service deployment on client by default when 
configuration is provided on startup. This closes #1748.

(cherry picked from commit b7ab273)

commit 443ac9a7aa82af1359a03bcfc8f9212b108300e4
Author: Andrey V. Mashenkov 
Date:   2017-04-05T12:01:02Z

IGNITE-4917: Fixed failure when accessing BinaryObjectBuilder field value 
serialized with OptimizedMarshaller . This closes #1736.

commit 05f3c747921aed6838804d2f5f2c8d2bd7985337
Author: Andrey V. Mashenkov 
Date:   2017-04-05T12:01:02Z

IGNITE-4917: Fixed failure when accessing BinaryObjectBuilder field value 
serialized with OptimizedMarshaller . This closes #1736.

(cherry picked from commit 443ac9a)

commit b8e3d1b6f972b6d30657b7c85c8b34506dc29b88
Author: Andrey V. Mashenkov 
Date:   2017-04-05T12:01:02Z

IGNITE-4917: Fixed failure when accessing BinaryObjectBuilder field value 
serialized with OptimizedMarshaller . This closes #1736.

(cherry picked from commit 443ac9a)

commit a0392605f39c23fcd20c98d852c4cab749cd059b
Author: Andrey V. Mashenkov 
Date:   2017-04-06T11:43:50Z

IGNITE-4832: Prevent service deployment on client by default when 
configuration is provided on startup. This closes #1748.

(cherry picked from commit b7ab273)

commit 220db882b466c03eadd148b3a19a0bf70d82d4a6
Author: dkarachentsev 
Date:   2017-04-10T07:28:15Z

IGNITE-2466 - Use current NIO back pressure mechanism to limit received 
messages. Mark them process only when backups acknowledged.

commit 3d616799efb472227b3b313516e6b40729654631
Author: dkarachentsev 
Date:   2017-04-10T07:36:11Z

IGNITE-2466 - Use current NIO back pressure mechanism to limit received 
messages. Mark them process only when backups acknowledged.

(backport from 1.9.2)

(cherry picked from commit 220db882b466c03eadd148b3a19a0bf70d82d4a6)

commit 2a88a7a7581465ff0a6f8733550e6d42d7f71a6c
Author: dkarachentsev 
Date:   2017-04-10T07:54:37Z

IGNITE-4667 - Throw exception on starting client cache when indexed types 
cannot be loaded

commit ba6227be49c8a395a5632e9841a6acb65ae340b6
Author: dkarachentsev 
Date:   2017-04-10T08:40:17Z

IGNITE-2466 - Disable back-pressure for sender data nodes to avoid deadlock.

commit bb3ff120e6995431d10439243d8b163712de0e0e
Author: dkarachentsev 
Date:   2017-04-10T08:40:17Z

IGNITE-2466 - Disable back-pressure for sender data nodes to avoid deadlock.

(cherry picked from commit ba6227b)

commit 315ff38eeef96f12954d6ff39c84d58b2b959667
Author: Andrey V. Mashenkov 
Date:   2017-04-06T11:43:50Z

IGNITE-4879: Fixed System pool starvation while partition evicting.

commit 01ceeb13420b68edf12b0262fe0991e84c085dd8
Author: Andrey V. Mashenkov 
Date:   2017-04-06T11:43:50Z

IGNITE-4863: Disallow change RootLogger log-level if it can have negative 
effect on other loggers. This closes #1687.

commit 0f7ef74216fab64f5d1d2b6d432b552b7fe40d2f
Author: Andrey V. Mashenkov 
Date:   2017-04-12T10:01:25Z

IGNITE-4907: Fixed excessive service instances can be started with dynamic 
deployment. This closes #1766.

commit 465084da5b00dcfc056d338f5d0a24875ca2af08
Author: Andrey V. Mashenkov 
Date:   2017-04-12T10:01:25Z

IGNITE-4907: Fixed excessive service instances can be started with dynamic 
deployment. This closes #1766.

(cherry picked from commit 0f7ef74)

commit a20c307df1dd000309a273ef93231fdc41a2a81c
Author: dkarachentsev 
Date:   2017-04-13T06:31:17Z

IGNITE-4891 - Fix. Key is deserialized during transactional get() even if 
withKeepBinary is set

(Backport from master)

commit 630558dfeb373f237057e394e8f2f63230d59dab
Author: vladisav 
Date:   2017-04-13T10:24:42Z

ignite-4173 IgniteSemaphore with failoverSafe enabled doesn't release 
permits in case permits owner node left topology

Backport from master.

(cherry picked from commit 76485fc)

commit 870b752c095ed3776e91a65b99763142b9f2ebc0
Author: Vladisav Jelisavcic 
Date:   2017-04-11T11:09:12Z

ignite-1977 - fixed IgniteSemaphore fault tolerance.

Backport from master.

(cherry picked from commit 902bf42)

commit 241e9291

[GitHub] ignite pull request #2471: 8.0.3.ea15

[agoncharuk]

GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/2471

8.0.3.ea15



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-8.0.3.ea15

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2471.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2471


commit 52556f4bf6d544a44bfd49b02d84aa32f741813f
Author: Ilya Lantukh 
Date:   2017-04-28T16:40:31Z

Multiple optimizations from ignite-gg-8.0.3.ea5-atomicbench.

commit 58b6e05e82978c68ba9e2e53a3b9b866e2c474ca
Author: Ilya Lantukh 
Date:   2017-04-28T16:57:32Z

Optimizations from ignite-5068.

commit 925370c3c6c587870e19f31f8660139884847e06
Author: sboikov 
Date:   2017-05-06T09:15:21Z

client join race

(cherry picked from commit 682ca8e)

commit 643e6d49d0e0fb1d66b18dcfe74b570c15924036
Author: Alexey Goncharuk 
Date:   2017-05-10T09:14:42Z

GG-12170 - Added assertions for tag value, skip failed-to-write-lock-pages 
on checkpoint begin

commit f00785611e1d785fe8c2247770bf07e07ddb1ebe
Author: Ivan Rakov 
Date:   2017-05-12T16:39:52Z

GG-12194: Backport GG-12184 into 8.0.3.ea8

commit b39f7c8d4f486f25e48a812e4a9d5aa4bbb5932f
Author: Alexey Kuznetsov 
Date:   2017-05-13T15:37:25Z

GG-12190 Backported to 803ea8.

commit 8f1398f43038700cdf66a4538d24c4dfd227cb0e
Author: Alexey Goncharuk 
Date:   2017-05-15T09:47:38Z

Additional debug

commit c35dbf4ec3ba6b01932c76f14ad4c45fed402391
Author: Yakov Zhdanov 
Date:   2017-05-15T15:03:07Z

Added IO latency test + made it available from MBean

(cherry picked from commit 8195ae0)

commit b1116069549be224d59983b93d2ee22cab8402b8
Author: sboikov 
Date:   2017-05-16T08:24:11Z

Improved exchange timeout debug logging.

commit 560ef60bf90643dfc4a329a760714d652c73e9a8
Author: sboikov 
Date:   2017-05-16T08:30:29Z

DirectByteBufferStreamImpl: converted asserts into exceptions.

commit 250b4e04606d95821ee9e87c40225c45f3432d3c
Author: Yakov Zhdanov 
Date:   2017-05-17T13:36:27Z

Results printout for IO latency test

(cherry picked from commit a0fc6ee)

commit 46cba2a46966759e6d658f3ba991f15722a6634f
Author: Alexey Goncharuk 
Date:   2017-05-17T16:04:10Z

Do not re-create node2part map on every singleMap message

commit d4c999795917b7695cc13f2fea3e69cd1a3d5078
Author: Alexey Goncharuk 
Date:   2017-05-17T17:58:21Z

MetaPageInitRecord fix

commit 7b545fa9029ba9f3d90828cd38611f6a2988cb25
Author: EdShangGG 
Date:   2017-05-16T16:07:03Z

GG-12140 We will lose data if we cancel snapshot restore
(cherry picked from commit 8e3ad6d)

commit e590a8110b1ce7f8140f06ae4a504f60777847e6
Author: Eduard Shangareev 
Date:   2017-05-17T23:12:44Z

GG-12140 We will lose data if we cancel snapshot restore
(cherry picked from commit 7721838)

commit db84a921920ff6a4ebe55af4cf8047e37e0addb1
Author: EdShangGG 
Date:   2017-05-18T14:37:30Z

fixing compilation after cherry-picking

commit afbade50e151d2aa793e9762637853ec6d6f2f93
Author: EdShangGG 
Date:   2017-05-18T16:30:00Z

GG-12192 Concurrent node join and snapshot restore cause to coordinator fail
-fixing tests and compilation

commit b29b918aece6a99a12c8bf3f21c5419a9d97de25
Author: Alexey Kuznetsov 
Date:   2017-05-19T07:18:23Z

Added Affinity topology version and Pending exchanges to Visor data 
collector task.
(cherry picked from commit 7402ea1)

commit 096404d36c1cc3a1d9da1db3a2b0a2b7fcdd702d
Author: Yakov Zhdanov 
Date:   2017-05-19T17:33:36Z

Results printout for IO latency test

commit 33f1c3376a05e728a63df5cf8802d5df6f9e02f5
Author: Alexey Goncharuk 
Date:   2017-05-22T16:46:39Z

Fixed assertion in checkpointer on cache stop

commit e7edb38fdd42008fd358e320dfe003a47b22cbe6
Author: EdShangGG 
Date:   2017-05-23T12:31:55Z

GG-12192 Concurrent node join and snapshot restore cause to coordinator fail
-adding test
-fixing problem with coordinator left

commit 34a0d5f5f7c97c4f401c14b4211af35eaa34e850
Author: Vladislav Pyatkov 
Date:   2017-05-23T12:33:39Z

ignite-5212 Allow custom affinity function for data structures cache
(cherry picked from commit f353faf)

commit f169a1898aacdd08584210fe3377767d12be563d
Author: Yakov Zhdanov 
Date:   2017-05-23T14:39:37Z

Results printout for IO latency test and new metrics

commit 767fe9abd578302159caaa59502f5ffb6f53336c
Author: Ilya Lantukh 
Date:   2017-05-23T15:56:07Z

Renting primary node - fix.

commit 2256fe195da7f82006bc68de1d38cd5e2c4bbbe2
Author: Ilya Lantukh 
Date:   2017-05-23T15:58:25Z

Merge branch 'ignite-gg-8.0.3.ea9' of 
https://github.com/gridgain/apache-ignite into ignite-gg-8.0.3.ea9

commit 6cf5d59c07a86c9bb0c78d2447970ba11977621d
Author: EdShangGG 
Date:   2017-05-23T15:02:29Z

GG-12210 Don't allow joi

[jira] [Created] (IGNITE-6104) Web Console: move "Download Web Agent" link to footer.

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-6104:


 Summary: Web Console: move "Download Web Agent" link to footer.
 Key: IGNITE-6104
 URL: https://issues.apache.org/jira/browse/IGNITE-6104
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov


Having big red button in header is a little bit annoying.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2470: IGNITE-6102 Implement checks that node that joins...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2470


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2463: IGNITE-6096 : Fixed races on partition evict.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2463


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2437: IGNITE-5991

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2437


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6103) Handle -1 partition ID during WAL replay

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-6103:


 Summary: Handle -1 partition ID during WAL replay
 Key: IGNITE-6103
 URL: https://issues.apache.org/jira/browse/IGNITE-6103
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Affects Versions: 2.1
Reporter: Alexey Goncharuk
 Fix For: 2.2


We might encounter a -1 partition ID in DataRecords, which will lead to 
recovery failure. To make code more robust, we should handle it and 
re-calculate partition ID.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2470: IGNITE-6102 Implement checks that node that joins...

[glukos]

GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/2470

IGNITE-6102 Implement checks that node that joins topology has consistent 
database configuration



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6102

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2470.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2470


commit 1b7f782b7e0c7384e48e00993de970a9842b29ba
Author: Ivan Rakov 
Date:   2017-08-17T14:48:49Z

IGNITE-6102 Implement checks that node that joins topology has consistent 
database configuration

commit bd06439fc92228e15f14f0391672c8d0c7d16b0e
Author: Dmitriy Govorukhin 
Date:   2017-08-17T14:56:57Z

ignite-6102 minor java doc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2469: Ignite 6102

[glukos]

Github user glukos closed the pull request at:

https://github.com/apache/ignite/pull/2469


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2469: Ignite 6102

[glukos]

GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/2469

Ignite 6102



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6102

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2469.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2469


commit 1b7f782b7e0c7384e48e00993de970a9842b29ba
Author: Ivan Rakov 
Date:   2017-08-17T14:48:49Z

IGNITE-6102 Implement checks that node that joins topology has consistent 
database configuration

commit bd06439fc92228e15f14f0391672c8d0c7d16b0e
Author: Dmitriy Govorukhin 
Date:   2017-08-17T14:56:57Z

ignite-6102 minor java doc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2468: IGNITE-3987: ODBC: Improved error output when que...

[isapego]

GitHub user isapego opened a pull request:

https://github.com/apache/ignite/pull/2468

IGNITE-3987: ODBC: Improved error output when query parsing failed.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3987

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2468.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2468


commit 3530143ab318f2e810cf39a7752afed146c1f49e
Author: Igor Sapego 
Date:   2017-08-17T14:41:08Z

IGNITE-3987: ODBC: Improved error output when query parsing failed.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Cluster auto activation design proposal

[Sergey Chugunov]

Dmitriy,

As automatic creation of Baseline Topology is just another use case about
the whole concept, lets discuss it in this thread and forget about parallel
one.

*initialActivationNodes* for desired configuration of Baseline Topology
sounds good to me, I would like to hear from other Igniters.

However other options to create Baseline Topology were proposed:

   - Prepare it manually with WebConsole or visor-console (no
   initialActivationNodes configuration is involved).
   - Create it automatically on the first manual activation (again no
   initial configuration is involved).

Thus we still need Baseline Topology entity not only in documentation.

Also please take into account that at some point in time user may want to
recreate BLT (e.g. to add nodes to existing cluster). User will have to
interact with some entity; so we need it not only internally but on public
API.

Thanks,
Sergey.



On Sat, Aug 12, 2017 at 4:07 AM, Dmitriy Setrakyan 
wrote:

> Sergey,
>
> As it is becoming clear from another thread, the only configuration users
> will need to provide is the initial set of nodes. Everything else will be
> handled by Ignite automatically.
>
> In this case, the name Baseline Topology will appear only in documentation,
> in which case the name is OK (that is if I am understanding the design
> correctly).
>
> However, the list initial set of nodes can be called
> initialActivationNodes.
>
> Makes sense?
>
> D.
>
> On Fri, Aug 11, 2017 at 2:46 AM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > More options to name the concept (one may put "node set" or "topology" at
> > the end - for me these are interchangeable):
> >
> >
> >1. essential
> >2. basic
> >3. completed
> >4. prepared
> >5. solid
> >6. enduring
> >
> > To me the name must stress the fact that "node set" or "topology"
> contains
> > all nodes user is going to work with for a long run.
> > It is what user plans to work with and base the work on.
> >
> > Any thoughts on this?
> >
> > Thanks,
> > Sergey.
> >
> > On Thu, Aug 10, 2017 at 5:41 PM, Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Going down "node set" road:
> > >
> > > -fixed node set
> > > -established node set
> > > -base node set
> > >
> > > On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > >> Can we brainstorm on the names again, I am not sure we have a
> consensus
> > on
> > >> the name "baseline topology". This will be included in Ignite
> > >> configuration, so the name has to be clear.
> > >>
> > >> Some of the proposals were:
> > >>
> > >> - baseline topology
> > >> - minimal node set
> > >> - node restart set
> > >> - minimal topology
> > >>
> > >> Any other suggestions?
> > >>
> > >> D.
> > >>
> > >> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk <
> > >> alexey.goncha...@gmail.com> wrote:
> > >>
> > >> > Denis,
> > >> >
> > >> > This should be handled by the BT triggers. If I have 3 backups
> > >> configured,
> > >> > I actually won't care if my cluster will live 6 hours without an
> > >> additional
> > >> > backup. If for a partition there is only one backup left - a new BT
> > >> should
> > >> > be triggered automatically.
> > >> >
> > >> > 2017-08-10 0:33 GMT+03:00 Denis Magda :
> > >> >
> > >> > > Sergey,
> > >> > >
> > >> > > That’s the only concern I have:
> > >> > >
> > >> > > * 5. User takes out nodes from cluster (e.g. for maintenance
> > >> purposes):
> > >> > no
> > >> > >   rebalance happens until user recreates BLT on new cluster
> > topology.*
> > >> > >
> > >> > > What if a node is crashed (or some other kind of outage) in the
> > >> middle of
> > >> > > the night and the user has to be sure that survived nodes will
> > >> rearrange
> > >> > > and rebalancing partitions?
> > >> > >
> > >> > > —
> > >> > > Denis
> > >> > >
> > >> > >
> > >> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov <
> > >> sergey.chugu...@gmail.com
> > >> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > Folks,
> > >> > > >
> > >> > > > I've summarized all results from our discussion so far on wiki
> > page:
> > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > >> > > Automatic+activation+design+-+draft
> > >> > > >
> > >> > > > I hope I reflected the most important details and going to add
> API
> > >> > > > suggestions for all use cases soon.
> > >> > > >
> > >> > > > Feel free to give feedback here or in comments under the page.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Sergey.
> > >> > > >
> > >> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov <
> > >> > akuznet...@apache.org>
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Hi,
> > >> > > >>
> > >> > >  1. User creates new BLT using WebConsole or other tool and
> > >> "applies"
> > >> > > it
> > >> > > >> to brand-new cluster.
> > >> > > >>
> > >> > > >> Good idea, but we also should implement *command-line utility*
> > for
> > >> the
> > >> > > same
> > >> > > >> use case.
> > >> 

[jira] [Created] (IGNITE-6102) Implement checks that node that joins topology has consistent database configuration

[Ivan Rakov (JIRA)]

Ivan Rakov created IGNITE-6102:
--

 Summary: Implement checks that node that joins topology has 
consistent database configuration
 Key: IGNITE-6102
 URL: https://issues.apache.org/jira/browse/IGNITE-6102
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.1
Reporter: Ivan Rakov
Assignee: Ivan Rakov
 Fix For: 2.2


We need to check for node misconfiguration: all nodes in cluster should have 
same (or consistent) database settings.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6101) Try to improve local scans performance

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-6101:


 Summary: Try to improve local scans performance
 Key: IGNITE-6101
 URL: https://issues.apache.org/jira/browse/IGNITE-6101
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.1
Reporter: Igor Seliverstov


Cutently local scan queries at least two times slower than direct iteration 
over partition (using BTree cursor). Check if it's possible to decrease the 
speed difference.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2466: IGNITE-6100 fix memory leak

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2466


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2465: IGNITE-6098 fix test IgniteDataIntegrityTests.tes...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2465


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2467: added throw runtime exception when it waits on Gr...

[voipp]

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2467

added throw runtime exception when it waits on GridCacheTxFinishSync



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite master-exception-on-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2467.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2467


commit 5cffb41fc335e75f775703260af44acc538fdb6d
Author: voipp 
Date:   2017-08-17T13:28:03Z

added throw runtime exception when it waits on GridCacheTxFinishSync




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2466: IGNITE-6100 fix memory leak

[DmitriyGovorukhin]

GitHub user DmitriyGovorukhin opened a pull request:

https://github.com/apache/ignite/pull/2466

IGNITE-6100 fix memory leak

IgnitePdsRecoveryAfterFileCorruptionTest.testPageRecoveryAfterFileCorruption

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6100

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2466.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2466


commit 4e1628049beaae4b059637c107e24008b7ad5733
Author: Dmitriy Govorukhin 
Date:   2017-08-17T13:52:30Z

ignite-6100 fix memory leak, 
IgnitePdsRecoveryAfterFileCorruptionTest.testPageRecoveryAfterFileCorruption




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6100) Test fail IgnitePdsRecoveryAfterFileCorruptionTest.testPageRecoveryAfterFileCorruption

[Dmitriy Govorukhin (JIRA)]

Dmitriy Govorukhin created IGNITE-6100:
--

 Summary: Test fail 
IgnitePdsRecoveryAfterFileCorruptionTest.testPageRecoveryAfterFileCorruption 
 Key: IGNITE-6100
 URL: https://issues.apache.org/jira/browse/IGNITE-6100
 Project: Ignite
  Issue Type: Test
Reporter: Dmitriy Govorukhin
Assignee: Dmitriy Govorukhin


Test leads to a memory leak



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6099) ODBC: Implement SQLGetInfo for all info types

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-6099:
---

 Summary: ODBC: Implement SQLGetInfo for all info types
 Key: IGNITE-6099
 URL: https://issues.apache.org/jira/browse/IGNITE-6099
 Project: Ignite
  Issue Type: Improvement
  Components: odbc
Affects Versions: 2.1
Reporter: Igor Sapego
Assignee: Igor Sapego
 Fix For: 2.2


Currently not all the info types supported for {{SQLGetInfo}}. Need to 
implement fully.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IGNITE-5714 Context switching for pessimistic transactions

[ALEKSEY KUZNETSOV]

The problem is that GridCacheTxFinishSync stores thread id. When we call
suspend on transaction, old thread id becames incorrect. So if this sync
class is useless, we can eliminate thread id storage and this class.

чт, 17 авг. 2017 г. в 15:47, ALEKSEY KUZNETSOV :

> I cannot find\create an example test scenario where a thread waits on it.
> A thread can wait only if finish request had been sent and finish response
> had not been received. But it seems, lock cannot proceed mapping in this
> case. So, when lock call awaitFinishAckAsync, it is already done and
> thread doesnt wait.
>
> чт, 17 авг. 2017 г. в 14:17, Alexey Goncharuk  >:
>
>> Aleksey,
>>
>> GridCacheTxFinishSync is used in IgniteTxManager#awaitFinishAckAsync()
>> method (the wait is done before mapping Near or Colocated lock future, see
>> the call hierarchy).
>>
>> --AG
>>
>> 2017-08-11 17:52 GMT+03:00 ALEKSEY KUZNETSOV :
>>
>> > Hi!
>> > There is GridCacheTxFinishSync synchronizer, which used to notify
>> threads
>> > (waiting on GridCacheTxFinishSync.TxFinishSync#pendingFut)
>> > that GridNearTxFinishResponse is received.
>> >
>> > Currently, only GridCacheTxFinishSync uses the synchronizer(but doesn't
>> > wait on it somehow). And it seems the synchronizer is useless. Do we
>> really
>> > need to keep GridCacheTxFinishSync in a project?
>> >
>> >
>> >
>> > ср, 9 авг. 2017 г. в 17:13, ALEKSEY KUZNETSOV > >:
>> >
>> > > Hi, Igntrs!
>> > >
>> > > Currently we have context switching implemented for optimistic
>> > > transactions : https://issues.apache.org/jira/browse/IGNITE-5712.
>> > >
>> > >
>> > >
>> > > So, the next step is to implement it for pessimistic transactions :
>> > > https://issues.apache.org/jira/browse/IGNITE-5714
>> > >
>> > >
>> > >
>> > > The problem with them lies in *IgniteTxAdapter#threadId*. Thread id is
>> > > transferred between nodes by GridDistributedTxPrepareRequest when key
>> is
>> > > got locked.
>> > >
>> > > When we suspend and resume transaction, thread id is got changed
>> locally,
>> > > but not on remote nodes.
>> > >
>> > >
>> > >
>> > > After studying the code, it seemed we can eliminate thread id from
>> > > transactions completely.
>> > >
>> > > For that reason, i want to start implementing additional tests, that
>> will
>> > > cover transaction logic. Tickets would be created for them.
>> > > Later on I will provide test scenarious and send you. *Will appreciate
>> > > any ideas from you on new tests, thanks!*
>> > >
>> > >
>> > >
>> > > It will be the first step. The next one will be refactoring and
>> > > eliminating thread id. What do you think ?
>> > >
>> > > --
>> > >
>> > > *Best Regards,*
>> > >
>> > > *Kuznetsov Aleksey*
>> > >
>> > --
>> >
>> > *Best Regards,*
>> >
>> > *Kuznetsov Aleksey*
>> >
>>
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>
-- 

*Best Regards,*

*Kuznetsov Aleksey*

[GitHub] ignite pull request #2465: IGNITE-6098 fix test IgniteDataIntegrityTests.tes...

[DmitriyGovorukhin]

GitHub user DmitriyGovorukhin opened a pull request:

https://github.com/apache/ignite/pull/2465

IGNITE-6098 fix test IgniteDataIntegrityTests.testExpandBuffer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6098

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2465.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2465


commit d298b8bb6e183a51984e3f8db9a9c6b3b92f8695
Author: Dmitriy Govorukhin 
Date:   2017-08-17T12:57:42Z

ignite-6098 fix test IgniteDataIntegrityTests.testExpandBuffer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2441: IGNITE-6033 Add sorted and multithreaded modes in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2441


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6098) Test fail IgniteDataIntegrityTests.testExpandBuffer

[Dmitriy Govorukhin (JIRA)]

Dmitriy Govorukhin created IGNITE-6098:
--

 Summary: Test fail IgniteDataIntegrityTests.testExpandBuffer
 Key: IGNITE-6098
 URL: https://issues.apache.org/jira/browse/IGNITE-6098
 Project: Ignite
  Issue Type: Test
Reporter: Dmitriy Govorukhin
Assignee: Dmitriy Govorukhin


{code}
junit.framework.AssertionFailedError: expected:<0> but was:<13>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at 
org.apache.ignite.internal.processors.cache.persistence.db.wal.crc.IgniteDataIntegrityTests.testExpandBuffer(IgniteDataIntegrityTests.java:138)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IGNITE-5714 Context switching for pessimistic transactions

[ALEKSEY KUZNETSOV]

I cannot find\create an example test scenario where a thread waits on it. A
thread can wait only if finish request had been sent and finish response
had not been received. But it seems, lock cannot proceed mapping in this
case. So, when lock call awaitFinishAckAsync, it is already done and thread
doesnt wait.

чт, 17 авг. 2017 г. в 14:17, Alexey Goncharuk :

> Aleksey,
>
> GridCacheTxFinishSync is used in IgniteTxManager#awaitFinishAckAsync()
> method (the wait is done before mapping Near or Colocated lock future, see
> the call hierarchy).
>
> --AG
>
> 2017-08-11 17:52 GMT+03:00 ALEKSEY KUZNETSOV :
>
> > Hi!
> > There is GridCacheTxFinishSync synchronizer, which used to notify threads
> > (waiting on GridCacheTxFinishSync.TxFinishSync#pendingFut)
> > that GridNearTxFinishResponse is received.
> >
> > Currently, only GridCacheTxFinishSync uses the synchronizer(but doesn't
> > wait on it somehow). And it seems the synchronizer is useless. Do we
> really
> > need to keep GridCacheTxFinishSync in a project?
> >
> >
> >
> > ср, 9 авг. 2017 г. в 17:13, ALEKSEY KUZNETSOV  >:
> >
> > > Hi, Igntrs!
> > >
> > > Currently we have context switching implemented for optimistic
> > > transactions : https://issues.apache.org/jira/browse/IGNITE-5712.
> > >
> > >
> > >
> > > So, the next step is to implement it for pessimistic transactions :
> > > https://issues.apache.org/jira/browse/IGNITE-5714
> > >
> > >
> > >
> > > The problem with them lies in *IgniteTxAdapter#threadId*. Thread id is
> > > transferred between nodes by GridDistributedTxPrepareRequest when key
> is
> > > got locked.
> > >
> > > When we suspend and resume transaction, thread id is got changed
> locally,
> > > but not on remote nodes.
> > >
> > >
> > >
> > > After studying the code, it seemed we can eliminate thread id from
> > > transactions completely.
> > >
> > > For that reason, i want to start implementing additional tests, that
> will
> > > cover transaction logic. Tickets would be created for them.
> > > Later on I will provide test scenarious and send you. *Will appreciate
> > > any ideas from you on new tests, thanks!*
> > >
> > >
> > >
> > > It will be the first step. The next one will be refactoring and
> > > eliminating thread id. What do you think ?
> > >
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>
-- 

*Best Regards,*

*Kuznetsov Aleksey*

[GitHub] ignite pull request #2345: IGNITE-5837 Test logic fix.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2345


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

ERROR: Heuristic transaction failure.

[Usein Faradzhev]

Hello.

 

We are trying to use the Ignite Memory File System and sometimes Ignite
can't write file to IGFS and can't read. What is this happens?

Below is an example for Cloudera Quick Start VM 5.10.0 and error, also
configuration and full log in attachments. This problems arise on our
cluster with CentOS 7 and CDH 5.11.1 too.

 

In-Memory Hadoop Accelerator:

Version2.1.0

Date  2017-07-27

File
http://apache-mirror.rbc.ru/pub/apache//ignite/2.1.0/apache-ignite-hadoop-2.
1.0-bin.zip
 

 

[cloudera@quickstart ~]$ ls -l dtm_ekp_scoring_plan_oper75.csv 

-rw-r--r-- 1 cloudera cloudera 19579883 Aug 16 03:53
dtm_ekp_scoring_plan_oper75.csv

 

[cloudera@quickstart ~]$ hdfs dfs -mkdir -p
igfs://igfs@/user/cloudera/dtm_ekp_scoring_plan_oper/

[cloudera@quickstart ~]$ hdfs dfs -put dtm_ekp_scoring_plan_oper75.csv
igfs://igfs@/user/cloudera/dtm_ekp_scoring_plan_oper/

put: Failed to flush data during stream close
[path=/user/cloudera/dtm_ekp_scoring_plan_oper/dtm_ekp_scoring_plan_oper75.c
sv._COPYING_, fileInfo=IgfsFileInfo [len=0, blockSize=65536,
lockId=4600eafed51-15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, affKey=null,
fileMap=IgfsFileMap [ranges=null], evictExclude=true]]

 

 

[2017-08-17 03:13:07,951][ERROR][igfs-#47%null%][GridNearTxLocal] Heuristic
transaction failure.

class
org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
Failed to locally write to cache (all transaction entries will be
invalidated, however there was a window when entries for this transaction
were visible to others): GridNearTxLocal
[mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping
[entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
[fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114,
txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
[fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114],
val=[op=CREATE, val=CacheObjectByteArrayImpl [arrLen=65536]],
prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=[], filtersPassed=false,
filtersSet=true, entry=GridDhtCacheEntry [rdrs=[], part=954,
super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
[fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
affKey=null, evictExclude=true], hasValBytes=true], val=null,
startVer=1502964754897, ver=GridCacheVersion [topVer=11755,
order=1502964754897, nodeOrder=1], hash=236544549,
extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc
[locs=[GridCacheMvccCandidate [nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6,
ver=GridCacheVersion [topVer=11755, order=1502964754896, nodeOrder=1],
threadId=69, id=152, topVer=AffinityTopologyVersion [topVer=1,
minorTopVer=0], reentry=null,
otherNodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, otherVer=GridCacheVersion
[topVer=11755, order=1502964754896, nodeOrder=1], mappedDhtNodes=null,
mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl
[part=954, val=IgfsBlockKey
[fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
affKey=null, evictExclude=true], hasValBytes=true],
masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=1|dht_lo
cal=1|near_local=0|removed=0|read=0, prevVer=null, nextVer=null]],
rmts=null]], flags=2]]], prepared=1, locked=false,
nodeId=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
serReadVer=null, xidVer=GridCacheVersion [topVer=11755,
order=1502964754896, nodeOrder=1]]], explicitLock=false, dhtVer=null,
last=false, nearEntries=0, clientFirst=false,
node=15b0cff9-0c6e-459c-8c1e-1d8f59d102e6]], nearLocallyMapped=false,
colocatedLocallyMapped=true, needCheckBackup=null, hasRemoteLocks=false,
thread=igfs-#47%null%, mappings=IgniteTxMappingsSingleImpl
[mapping=GridDistributedTxMapping [entries=[IgniteTxEntry
[key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
[fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114,
txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=954, val=IgfsBlockKey
[fileId=1600eafed51-cd651f8d-10b5-4cc3-9c14-e74963c7c2be, blockId=130,
affKey=null, evictExclude=true], hasValBytes=true], cacheId=-313790114],
val=[op=CREATE, val=CacheObjectByteArrayImpl [arrLen=65536]],
prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=[], filtersPassed=false,
filtersSet=true, entry=GridDhtCacheEntry [rdrs=[], part=954

Re: SQL Getting Started Guide

[Igor Sapego]

Akmal,

I work on the ODBC driver and would be glad to help. You can send
mails with your requests to devlist and I will try to help as much as I
can.

Best Regards,
Igor

On Thu, Aug 17, 2017 at 3:28 PM, Akmal Chaudhri  wrote:

> When I have the opportunity to test the ODBC code and ensure that it
> performs, I will add it to GH. At the moment, just snippets exist. If
> anyone can assist with this, I would appreciate it, as I have no experience
> with ODBC. I made some efforts to build the ODBC driver and have had some
> success on Windows and Linux, but have been unable to run any C++ code.
>
> On 16 August 2017 at 21:41, Denis Magda  wrote:
>
> > Igniters,
> >
> > The SQL getting started guide is ready and published:
> > https://apacheignite.readme.io/v2.1/docs/getting-started-sql
> >
> > Akmal thanks for your efforts and please apply the following minor
> changes:
> > * Add ODBC source files to your GitHub project.
> > * Add a flat SQL file (script) to the same project that will include all
> > the pure SQL statements used in the guide.
> >
> > Basing on this guide I’ve prepared and release a guide for DBeaver SQL
> > tool:
> > https://apacheignite-tools.readme.io/docs/dbeaver
> >
> > —
> > Denis
> >
> > On Aug 11, 2017, at 3:17 PM, Denis Magda  wrote:
> >
> > Akmal,
> >
> > Good start! Please consider the following feedback:
> >
> > 1. Connectivity section. Emphasize that ignite-core.jar has to be copied
> > to the classpath of an app or tool and give a link to more advanced JDBC
> > Thin driver documentation if the one needs more details. Probably,
> similar
> > steps should be brought up for the ODBC once it’s ready.
> >
> > 2. Make the first letters of city and people tables uppercase: city ->
> > City, people -> People
> >
> > 3. Rename INSERT section to “Inserting Data”, SELECT to “Querying Data”,
> > UPDATE -> “Modifying Data”, DELETE to “Removing Data” or to better
> > alternatives.
> >
> > 4. Show how to insert at least 2 cities and 5 people in the INSERT
> section.
> >
> > —
> > Denis
> >
> > On Aug 11, 2017, at 10:42 AM, Akmal Chaudhri <
> akmal.chaud...@gridgain.com>
> > wrote:
> >
> > Denis, All
> >
> > I have made some progress with the documentation:
> >
> > https://apacheignite.readme.io/v2.1/docs/getting-started-sql <
> > https://apacheignite.readme.io/v2.1/docs/getting-started-sql>
> >
> > ODBC connectivity and examples are missing, but should be completed very
> > soon.
> >
> > Community feedback welcome.
> >
> > Thanks.
> >
> > On 31 July 2017 at 19:02, Denis Magda  > mailto:dma...@apache.org >> wrote:
> > Igniters,
> >
> > There is a lot of SQL related documentation available for Ignite.
> However,
> > the one can get lost in it when he/she does her first steps on a learning
> > path. It’s time to simplify these are first steps with a clear and
> > straightforward getting started guide. I’ve put the requirements into
> this
> > JIRA ticket:
> > https://issues.apache.org/jira/browse/IGNITE-5886 <
> > https://issues.apache.org/jira/browse/IGNITE-5886>
> >
> > Please share your thoughts in JIRA or in this discussion.
> >
> > Akmal, who perfected his technical writer skills working for IBM,
> > volunteered to take over this task. Akmal, please create an account in
> > Ignite JIRA and share it with us. You’ll be able to assign the task on
> > yourself afterwards.
> > https://issues.apache.org/jira/projects/IGNITE <
> https://issues.apache.org/
> > jira/projects/IGNITE>
> >
> > Moreover, once the guide is ready Prachi will use it as a basis for the
> > new SQL screencast we need to produce.
> >
> > —
> > Denis
> >
> >
> >
> >
>

[GitHub] ignite pull request #2270: IGNITE-5505 @AffinityKeyMapped annotation is igno...

[kukushal]

Github user kukushal closed the pull request at:

https://github.com/apache/ignite/pull/2270


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: SQL Getting Started Guide

[Akmal Chaudhri]

When I have the opportunity to test the ODBC code and ensure that it
performs, I will add it to GH. At the moment, just snippets exist. If
anyone can assist with this, I would appreciate it, as I have no experience
with ODBC. I made some efforts to build the ODBC driver and have had some
success on Windows and Linux, but have been unable to run any C++ code.

On 16 August 2017 at 21:41, Denis Magda  wrote:

> Igniters,
>
> The SQL getting started guide is ready and published:
> https://apacheignite.readme.io/v2.1/docs/getting-started-sql
>
> Akmal thanks for your efforts and please apply the following minor changes:
> * Add ODBC source files to your GitHub project.
> * Add a flat SQL file (script) to the same project that will include all
> the pure SQL statements used in the guide.
>
> Basing on this guide I’ve prepared and release a guide for DBeaver SQL
> tool:
> https://apacheignite-tools.readme.io/docs/dbeaver
>
> —
> Denis
>
> On Aug 11, 2017, at 3:17 PM, Denis Magda  wrote:
>
> Akmal,
>
> Good start! Please consider the following feedback:
>
> 1. Connectivity section. Emphasize that ignite-core.jar has to be copied
> to the classpath of an app or tool and give a link to more advanced JDBC
> Thin driver documentation if the one needs more details. Probably, similar
> steps should be brought up for the ODBC once it’s ready.
>
> 2. Make the first letters of city and people tables uppercase: city ->
> City, people -> People
>
> 3. Rename INSERT section to “Inserting Data”, SELECT to “Querying Data”,
> UPDATE -> “Modifying Data”, DELETE to “Removing Data” or to better
> alternatives.
>
> 4. Show how to insert at least 2 cities and 5 people in the INSERT section.
>
> —
> Denis
>
> On Aug 11, 2017, at 10:42 AM, Akmal Chaudhri 
> wrote:
>
> Denis, All
>
> I have made some progress with the documentation:
>
> https://apacheignite.readme.io/v2.1/docs/getting-started-sql <
> https://apacheignite.readme.io/v2.1/docs/getting-started-sql>
>
> ODBC connectivity and examples are missing, but should be completed very
> soon.
>
> Community feedback welcome.
>
> Thanks.
>
> On 31 July 2017 at 19:02, Denis Magda  mailto:dma...@apache.org >> wrote:
> Igniters,
>
> There is a lot of SQL related documentation available for Ignite. However,
> the one can get lost in it when he/she does her first steps on a learning
> path. It’s time to simplify these are first steps with a clear and
> straightforward getting started guide. I’ve put the requirements into this
> JIRA ticket:
> https://issues.apache.org/jira/browse/IGNITE-5886 <
> https://issues.apache.org/jira/browse/IGNITE-5886>
>
> Please share your thoughts in JIRA or in this discussion.
>
> Akmal, who perfected his technical writer skills working for IBM,
> volunteered to take over this task. Akmal, please create an account in
> Ignite JIRA and share it with us. You’ll be able to assign the task on
> yourself afterwards.
> https://issues.apache.org/jira/projects/IGNITE  jira/projects/IGNITE>
>
> Moreover, once the guide is ready Prachi will use it as a basis for the
> new SQL screencast we need to produce.
>
> —
> Denis
>
>
>
>

[GitHub] ignite pull request #2393: IGNITE-5738: SQL: Added support for batching for ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2393


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2417: Ignite 5947

[ntikhonov]

Github user ntikhonov closed the pull request at:

https://github.com/apache/ignite/pull/2417


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IGNITE-5714 Context switching for pessimistic transactions

[Alexey Goncharuk]

Aleksey,

GridCacheTxFinishSync is used in IgniteTxManager#awaitFinishAckAsync()
method (the wait is done before mapping Near or Colocated lock future, see
the call hierarchy).

--AG

2017-08-11 17:52 GMT+03:00 ALEKSEY KUZNETSOV :

> Hi!
> There is GridCacheTxFinishSync synchronizer, which used to notify threads
> (waiting on GridCacheTxFinishSync.TxFinishSync#pendingFut)
> that GridNearTxFinishResponse is received.
>
> Currently, only GridCacheTxFinishSync uses the synchronizer(but doesn't
> wait on it somehow). And it seems the synchronizer is useless. Do we really
> need to keep GridCacheTxFinishSync in a project?
>
>
>
> ср, 9 авг. 2017 г. в 17:13, ALEKSEY KUZNETSOV :
>
> > Hi, Igntrs!
> >
> > Currently we have context switching implemented for optimistic
> > transactions : https://issues.apache.org/jira/browse/IGNITE-5712.
> >
> >
> >
> > So, the next step is to implement it for pessimistic transactions :
> > https://issues.apache.org/jira/browse/IGNITE-5714
> >
> >
> >
> > The problem with them lies in *IgniteTxAdapter#threadId*. Thread id is
> > transferred between nodes by GridDistributedTxPrepareRequest when key is
> > got locked.
> >
> > When we suspend and resume transaction, thread id is got changed locally,
> > but not on remote nodes.
> >
> >
> >
> > After studying the code, it seemed we can eliminate thread id from
> > transactions completely.
> >
> > For that reason, i want to start implementing additional tests, that will
> > cover transaction logic. Tickets would be created for them.
> > Later on I will provide test scenarious and send you. *Will appreciate
> > any ideas from you on new tests, thanks!*
> >
> >
> >
> > It will be the first step. The next one will be refactoring and
> > eliminating thread id. What do you think ?
> >
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>

[GitHub] ignite pull request #2462: IGNITE-6091 fix flaky test CacheLateAffinityAssig...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2462


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2458: IGNITE-GG-12549-1.9.6

[asfedotov]

Github user asfedotov closed the pull request at:

https://github.com/apache/ignite/pull/2458


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2079: IGNITE-5233: JDBC thin Driver: implement metadata...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2079


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2464: IGNITE-5849 metastorage

[kdudkov]

GitHub user kdudkov opened a pull request:

https://github.com/apache/ignite/pull/2464

IGNITE-5849 metastorage



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2464


commit 5af0ca0d0211b48f368beaebdf484b2df9ca11b3
Author: Konstantin Dudkov 
Date:   2017-08-17T10:39:28Z

IGNITE-5849 refactor DataPageIO & FreeList




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

The reason behind locking backup copy

[ALEKSEY KUZNETSOV]

Hi, Igntrs! Consider primary node P, backup node B for some key1. When we
create lock on this key (at primary node) :

IgniteCache primaryCache =
primaryIgnite.cache(DEFAULT_CACHE_NAME);

primaryCache.lock("key1").lock()

After lock acquired on primary node, the GridCacheMvccCandidate would be
created on backup node .

What is the point of creating candidate on backup node ? I cannot find any
reason for this.
-- 

*Best Regards,*

*Kuznetsov Aleksey*

[GitHub] ignite pull request #2463: IGNITE-6096 : Fixed races on partition evict.

[ilantukh]

GitHub user ilantukh opened a pull request:

https://github.com/apache/ignite/pull/2463

IGNITE-6096 : Fixed races on partition evict.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6096

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2463


commit 42877048314332ce1691a2f47ef967e129e9ad75
Author: Ilya Lantukh 
Date:   2017-08-17T09:58:23Z

ignite-6096 : Fixed races on partition evict.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6097) GridCacheDatabaseSharedManager.restorePartitionState(...) should not access PageMemory directly

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-6097:


 Summary: GridCacheDatabaseSharedManager.restorePartitionState(...) 
should not access PageMemory directly
 Key: IGNITE-6097
 URL: https://issues.apache.org/jira/browse/IGNITE-6097
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Ilya Lantukh
Assignee: Ilya Lantukh
 Fix For: 2.2


It breaks encapsulation and leads to subtle non-trivial problems (like deadlock 
in IGNITE-6096). We should update state by calling methods of 
GridDhtLocalPartition instead.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6096) Race between partition eviction and re-creation

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-6096:


 Summary: Race between partition eviction and re-creation
 Key: IGNITE-6096
 URL: https://issues.apache.org/jira/browse/IGNITE-6096
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Ilya Lantukh
Assignee: Ilya Lantukh
 Fix For: 2.2


There are a number of cases that aren't handled correctly, leading to assertion 
errors, grid hang-ups and data loss:
- PageMemoryImpl.refreshOutdatedPage(...) - if refreshed page is currently 
scheduled for checkpoint, it will be stored if FileStore filled with zeroes. 
Reading this page later will fail.
- GridCacheDatabaseSharedManager.restorePartitionState(...) - 
grp.offheap().onPartitionInitialCounterUpdated(...) is called under meta page 
write lock. If DataStore requires initialization, it will try to aquire write 
lock for meta page again and hang up.
- GridDhtPartitionTopologyImpl.createPartition(...) and .localPartition0(...) - 
if partition is present but has EVICTED state, we will try to create new 
partition instance. However, DataStore for old partition might still be 
present, and we will get AssertionError when we attempt to create new DataStore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: .sha Release Distribution Policy

[Sergey Kozlov]

Denis

Also we don't use .sha extension so we already follow that rules

On Thu, Aug 17, 2017 at 10:57 AM, Oleg Ostanin 
wrote:

> Hi, Denis
>
> Yes, we have a ticket that already takes this into account:
> https://issues.apache.org/jira/browse/IGNITE-5817
> I think we can create both sha-256 and sha-512 checksums.
>
> Best regards
> Oleg
>
> On Thu, Aug 17, 2017 at 1:51 AM, Denis Magda  wrote:
>
> > Igniters, especially the release managers,
> >
> > Please consider these changes and recommendations for the next release.
> Do
> > we have any ticket that already takes this into account?
> >
> > —
> > Denis
> >
> > > Begin forwarded message:
> > >
> > > From: "Henk P. Penning" 
> > > Subject: .sha Release Distribution Policy
> > > Date: August 16, 2017 at 1:55:57 AM PDT
> > > To: 
> > > Reply-To: priv...@ignite.apache.org
> > >
> > > Hi PMC,
> > >
> > >   The Release Distribution Policy[1] changed regarding .sha files.
> > >   See under "Cryptographic Signatures and Checksums Requirements" [2].
> > >
> > >  Old policy :
> > >
> > >-- use extension .sha for any SHA checksum (SHA-1, SHA-256, SHA-512)
> > >
> > >  New policy :
> > >
> > > -- use .sha1 for a SHA-1 checksum
> > > -- use .sha256 for a SHA-256 checksum
> > > -- use .sha512 for a SHA-512 checksum
> > > -- [*] .sha should contain a SHA-1
> > >
> > >  Why this change ?
> > >
> > > -- Verifying a checksum under the old policy is/was not handy.
> > >You have to inspect the .sha to find out which algorithm
> > >should be used ; or try them all (SHA-1, SHA256, etc).
> > >The new scheme avoids this ambiguity.
> > > -- The last point[*] was only added for clarity. Most of the
> > >old, stale .sha's contain a SHA-1. The relatively new .sha's
> > >contain a SHA-512. The expectation is that the last catagory
> will
> > >disappear, when active projects adapt to the 'new' convention.
> > >
> > >  Impact :
> > >
> > > -- Should be none ; many projects already use the 'new' convention.
> > > -- Please ask your release managers to use .sha1, .sha256, .sha512
> > >instead of the .sha extension.
> > > -- Please fix your build-tools if you have any.
> > >
> > >  Piggyback :
> > >
> > > -- The policy requires a .md5 for every package ;
> > >providing a .sha512 is recommended.
> > >Since MD5 is essentially broken, it is to be expected that
> > >in the future a .sha512 will be required.
> > >Perhaps it is wize to start providing .sha512's
> > >with your releases if you do not already do so.
> > >
> > > -- Visit http://mirror-vm.apache.org/checker/
> > >to check the health of your /dist/-area ;
> > >my stuff ; any feedback is most welcome.
> > >
> > >  Thanks ; regards,
> > >
> > >  Henk Penning
> > >
> > >   [1] http://www.apache.org/dev/release-distribution
> > >   [2] http://www.apache.org/dev/release-distribution#sigs-and-sums
> > >
> > > 
> > > Henk P. Penning ; apache.org infrastructure volunteer.
> > > he...@apache.org ; http://mirror-vm.apache.org/~henkp/
> >
> >
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

[GitHub] ignite pull request #2454: IGNITE-6080

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2454


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Redesigning Ignite's site Look & Feel

[Christos Erotocritou]

Hello, try again now. This is hosted currently on my local machine until we 
move onto a staging server.


> On 17 Aug 2017, at 02:58, Alexey Kuznetsov  wrote:
> 
> I cannot open this url.
> It is working?
> 
> On Thu, Aug 17, 2017 at 8:16 AM, Dmitriy Setrakyan 
> wrote:
> 
>> Looks very cool!
>> 
>> My first comment is that the home page looks different in Windows Edge from
>> Mac. For example, the font of the "Memory-Centric Data Platform" is
>> different and bold is not visible in Windows.
>> 
>> D.
>> 
>> On Wed, Aug 16, 2017 at 4:30 PM, Denis Magda  wrote:
>> 
>>> Igniters,
>>> 
>>> Some of us have been working on the new design for the site. That’s an
>>> early draft we have. Feel free to share the suggestions:
>>> http://81.156.86.10:8081/index.html >> 
>>> 
>>> In the meanwhile, is it possible to request a staging environment for the
>>> site from INFRA? There are so many changes we need to apply to CSS files,
>>> menus, pages, etc. that we have to create a branch to incorporate them
>> and
>>> check the results on the staging environment.
>>> 
>>> —
>>> Denis
>> 
> 
> 
> 
> -- 
> Alexey Kuznetsov

[jira] [Created] (IGNITE-6095) Web console: In demo mode Implement configuration of key fields in domain model for SQL query

[Vasiliy Sisko (JIRA)]

Vasiliy Sisko created IGNITE-6095:
-

 Summary: Web console: In demo mode Implement configuration of key 
fields in domain model for SQL query
 Key: IGNITE-6095
 URL: https://issues.apache.org/jira/browse/IGNITE-6095
 Project: Ignite
  Issue Type: Task
  Components: wizards
Affects Versions: 2.1
Reporter: Vasiliy Sisko
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: .sha Release Distribution Policy

[Oleg Ostanin]

Hi, Denis

Yes, we have a ticket that already takes this into account:
https://issues.apache.org/jira/browse/IGNITE-5817
I think we can create both sha-256 and sha-512 checksums.

Best regards
Oleg

On Thu, Aug 17, 2017 at 1:51 AM, Denis Magda  wrote:

> Igniters, especially the release managers,
>
> Please consider these changes and recommendations for the next release. Do
> we have any ticket that already takes this into account?
>
> —
> Denis
>
> > Begin forwarded message:
> >
> > From: "Henk P. Penning" 
> > Subject: .sha Release Distribution Policy
> > Date: August 16, 2017 at 1:55:57 AM PDT
> > To: 
> > Reply-To: priv...@ignite.apache.org
> >
> > Hi PMC,
> >
> >   The Release Distribution Policy[1] changed regarding .sha files.
> >   See under "Cryptographic Signatures and Checksums Requirements" [2].
> >
> >  Old policy :
> >
> >-- use extension .sha for any SHA checksum (SHA-1, SHA-256, SHA-512)
> >
> >  New policy :
> >
> > -- use .sha1 for a SHA-1 checksum
> > -- use .sha256 for a SHA-256 checksum
> > -- use .sha512 for a SHA-512 checksum
> > -- [*] .sha should contain a SHA-1
> >
> >  Why this change ?
> >
> > -- Verifying a checksum under the old policy is/was not handy.
> >You have to inspect the .sha to find out which algorithm
> >should be used ; or try them all (SHA-1, SHA256, etc).
> >The new scheme avoids this ambiguity.
> > -- The last point[*] was only added for clarity. Most of the
> >old, stale .sha's contain a SHA-1. The relatively new .sha's
> >contain a SHA-512. The expectation is that the last catagory will
> >disappear, when active projects adapt to the 'new' convention.
> >
> >  Impact :
> >
> > -- Should be none ; many projects already use the 'new' convention.
> > -- Please ask your release managers to use .sha1, .sha256, .sha512
> >instead of the .sha extension.
> > -- Please fix your build-tools if you have any.
> >
> >  Piggyback :
> >
> > -- The policy requires a .md5 for every package ;
> >providing a .sha512 is recommended.
> >Since MD5 is essentially broken, it is to be expected that
> >in the future a .sha512 will be required.
> >Perhaps it is wize to start providing .sha512's
> >with your releases if you do not already do so.
> >
> > -- Visit http://mirror-vm.apache.org/checker/
> >to check the health of your /dist/-area ;
> >my stuff ; any feedback is most welcome.
> >
> >  Thanks ; regards,
> >
> >  Henk Penning
> >
> >   [1] http://www.apache.org/dev/release-distribution
> >   [2] http://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> > 
> > Henk P. Penning ; apache.org infrastructure volunteer.
> > he...@apache.org ; http://mirror-vm.apache.org/~henkp/
>
>

[jira] [Created] (IGNITE-6094) Web console: Implement persistent store in demo mode.

[Vasiliy Sisko (JIRA)]

Vasiliy Sisko created IGNITE-6094:
-

 Summary: Web console: Implement persistent store in demo mode.
 Key: IGNITE-6094
 URL: https://issues.apache.org/jira/browse/IGNITE-6094
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Affects Versions: 2.1
Reporter: Vasiliy Sisko
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Batch service deployment

[Denis Mekhanikov]

> Can we add an "allOrNone" flag to the deployment method?

Sounds good, I think we can.

> However, hot do you ensure atomicity here?

We can guarantee that if some of configurations are invalid, or a
transaction, that writes configuration to the internal cache, fails, then
no services will be deployed.

Currently we don't track failures on the server side and services are
considered successfully deployed once their configurations are written to
the cache. So, it's not possible that all configurations are valid, but
only a part of the services fail to deploy. If we change this behavior and
start tracking failures during deployment and initialization on the server,
then we could automatically cancel services that are already deployed in a
batch.

чт, 17 авг. 2017 г. в 8:34, Dmitriy Setrakyan :

> On Wed, Aug 16, 2017 at 8:17 AM, Denis Mekhanikov 
> wrote:
>
> > I've had a few off-line conversations with other Igniters regarding this
> > question and almost all of them think that services should be deployed
> with
> > "all-or-none" failing policy.
> > We have a similar functionality for caches: Ignite#createCaches method
> > don't allow partial deployments, and I think, we should also stick to
> this
> > principle for services.
> >
>
> Can we add an "allOrNone" flag to the deployment method? If true, then all
> services will have to either be deployed or failed. However, hot do you
> ensure atomicity here? If you are deploying 10 services, and only 1 fails,
> what do you do with the other 9, given that they have already been deployed
> and may have started serving API requests?
>
>
> >
> > Another question that I'd like to discuss here is that currently
> > IgniteServices#deployAsync method may fail with an exception instead of
> > returning a future. Shouldn't we change this behavior to make async
> > operations always return a future whose get() method would throw an
> > exception?
> >
>
> Makes sense to me. I think throwing exception from async method is plain
> wrong.
>
> >
> > вт, 15 авг. 2017 г. в 11:42, Dmitriy Setrakyan :
> >
> > > Denis,
> > >
> > > I don't think we need a king deployment result.
> > >
> > > The "deployAllAsync" method should never throw an exception, it should
> > > always return the future. However, the IgniteFuture.get(...) method
> does
> > > throw an exception, and in this exception you should provide the info
> > about
> > > the failures.
> > >
> > > D.
> > >
> > > On Tue, Aug 15, 2017 at 1:31 AM, Denis Mekhanikov <
> dmekhani...@gmail.com
> > >
> > > wrote:
> > >
> > > > Dmitriy, thank you for your reply!
> > > >
> > > > I see a possibility of a bad scenario here. If we use deployAllAsync
> > > method
> > > > and it throws an exception, then the constructed future won't be
> > returned
> > > > and we won't have a way to wait for the rest of the services to
> deploy.
> > > > Maybe we should return some king of deployment result, containing a
> > > future
> > > > along with a collection of failed services, instead of throwing an
> > > > exception?
> > > >
> > > > пн, 14 авг. 2017 г. в 18:03, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > >
> > > > > Hi Denis, I agree, we should have an API for batch service
> > deployment.
> > > My
> > > > > comments are inline...
> > > > >
> > > > > On Mon, Aug 14, 2017 at 2:22 AM, Denis Mekhanikov <
> > > dmekhani...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Igniters!
> > > > > >
> > > > > > Currently Ignite doesn't have support for batch service
> deployment,
> > > but
> > > > > it
> > > > > > may be a very useful feature in case of a big number of nodes in
> a
> > > > > cluster
> > > > > > and services to be deployed. Each deployment includes write into
> an
> > > > > > internal transactional cache, which is the longest part of the
> > > > procedure.
> > > > > >
> > > > > > I propose to optimize it by performing multiple writes in a
> single
> > > > > > transaction. It implies an introduction of a few new methods in
> > > > > > IgniteServices interface.
> > > > > > I am thinking about the following signatures:
> > > > > >
> > > > > >   void deployAll(Iterable cfgs) throws
> > > > > > IgniteException;
> > > > > >   IgniteFuture
> deployAllAsync(Iterable
> > > > > > cfgs) throws IgniteException;
> > > > > >
> > > > > > I'd like to know your opinion on the following questions:
> > > > > >
> > > > > >- Do you agree with the proposed signatures?
> > > > > >
> > > > >
> > > > > Yes, but Iterable should be changed to Collection to be consistent
> > with
> > > > > other similar APIs in Ignite.
> > > > >
> > > > >
> > > > > >- What should happen in case of a failure (some of the
> > > > configurations
> > > > > >don't pass validation, or a service with specified name but
> > > > different
> > > > > >configuration already exists)? Should partial deployments be
> > > > performed
> > > > > > in
> > > > > >case when some of them fail?
> > > > > >
> > > > >
> > > > > Yes, we should allow partial deployme