Re: Data streaming using Apache Ignite and Flink
Thank you so much Denis. Regards, Saikat On Mon, Aug 27, 2018 at 5:00 PM, Denis Magda wrote: > Hello Saikat, > > Thanks for the article and contribution. We'll get it added to: > https://ignite.apache.org/blogs.html > > -- > Denis > > On Sun, Aug 26, 2018 at 2:59 PM Saikat Maitra > wrote: > > > Hello, > > > > I recently published blog on how we can stream data using Apache Ignite > and > > Flink. This uses IgniteSink with recent changes merged (release due in > > 2.7.0) which will allow us to run IgniteSink using Apache Flink in > cluster > > mode. > > > > > > > > https://samaitra.blogspot.com/2018/08/data-streaming-using- > apache-flink-and.html > > > > Please review and let me know if you have feedback. > > > > Regards, > > Saikat > > >
[GitHub] ignite pull request #4629: IGNITE-9382 Fixed data handling for multiple chun...
GitHub user tyabuki opened a pull request: https://github.com/apache/ignite/pull/4629 IGNITE-9382 Fixed data handling for multiple chunks of data You can merge this pull request into a Git repository by running: $ git pull https://github.com/tyabuki/ignite ignite-9382 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4629.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4629 commit 92ef9dfcb779c2a8b171a2dc691bd6d64c5b78ee Author: Toru Yabuki Date: 2018-08-23T10:28:41Z IGNITE-9382 Fixed data handling for multiple chunks of data ---
Re: Hello
Hi Mikhail! Done, welcome to the community! -- Denis On Mon, Aug 27, 2018 at 6:52 AM Mikhail Petrov wrote: > I'm new to Ignite and I would like to join Apache Ignite development. > My JIRA's login: PetrovMikhail >
Re: Data streaming using Apache Ignite and Flink
Hello Saikat, Thanks for the article and contribution. We'll get it added to: https://ignite.apache.org/blogs.html -- Denis On Sun, Aug 26, 2018 at 2:59 PM Saikat Maitra wrote: > Hello, > > I recently published blog on how we can stream data using Apache Ignite and > Flink. This uses IgniteSink with recent changes merged (release due in > 2.7.0) which will allow us to run IgniteSink using Apache Flink in cluster > mode. > > > > https://samaitra.blogspot.com/2018/08/data-streaming-using-apache-flink-and.html > > Please review and let me know if you have feedback. > > Regards, > Saikat >
Re: Service Grid new design overview
Agree with Val. I think all users would expect that a service is restarted upon a node or cluster restart. Let's make sure we preserve this behavior. D. On Fri, Aug 24, 2018 at 4:17 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Guys, > > I believe we should preserve the behavior that we have now. What happens to > services if we restart a persistent cluster running 2.6? Are services > recreated or not? If YES, we should make sure the same happens after > redesign. Would be even better if we preserve compatibility, i.e. allow > seamless upgrade from older version that uses system cache to newer version > that uses disco messages for service deployment. If NO, it's much easier > and we can leave it as is for now. However, eventually would be great to > have an option to persist services and redeploy them after cluster restart. > > -Val > > On Fri, Aug 24, 2018 at 2:51 PM Dmitriy Pavlov > wrote: > > > Denis M. & Val please share your vision about this topic. > > > > пт, 24 авг. 2018 г. в 15:52, Vyacheslav Daradur : > > > > > Nick, Antron, thank you for stepping in. > > > > > > AFAIK, Ignite cluster can move its state to a new version of Ignite > > > using persistence only. > > > > > > Since Ignite v.2.3 persistence is configured on a memory region and > > > system memory region is not persistence, that means the system > > > (utility) cache will not be recovered on cluster restart. > > > > > > Here is a ticket which describes the same issue: > > > https://issues.apache.org/jira/browse/IGNITE-6629 > > > > > > > BTW, Is proposed solution provides the guarantee that services will > be > > > > redeployed after each cluster restart since now we're not using the > > > cache? > > > > > > No, only services described in IgniteConfiguration will be deployed at > > > node startup as well as now. > > > > > > Am I wrong in something? > > > On Thu, Aug 23, 2018 at 5:59 PM Anton Vinogradov > wrote: > > > > > > > > Vyacheslav. > > > > > > > > It looks like we able to restart all services on grid startup from > old > > > > definitions (inside cache) in case persistence turned on. > > > > Se no problems to provide such automated migration case. > > > > Also, we can test it using compatibility framework. > > > > > > > > BTW, Is proposed solution provides the guarantee that services will > be > > > > redeployed after each cluster restart since now we're not using the > > > cache? > > > > > > > > чт, 23 авг. 2018 г. в 15:21, Nikolay Izhikov : > > > > > > > > > Hello, Vyacheslav. > > > > > > > > > > Thanks, for sharing your design. > > > > > > > > > > > I have a question about services migration from AI 2.6 to a new > > > solution > > > > > > > > > > Can you describe consequences of not having migration solution? > > > > > What will happen on the user side? > > > > > > > > > > > > > > > В Чт, 23/08/2018 в 14:44 +0300, Vyacheslav Daradur пишет: > > > > > > Hi, Igniters! > > > > > > > > > > > > I’m working on Service Grid redesign tasks and design seems to be > > > > > finished. > > > > > > > > > > > > The main goal of Service Grid redesign is to provide missed > > > guarantees: > > > > > > - Synchronous services deployment/undeployment; > > > > > > - Failover on coordinator change; > > > > > > - Propagation of deployments errors across the cluster; > > > > > > - Introduce of a deployment failures policy; > > > > > > - Prevention of deployments initiators hangs while deployment; > > > > > > - etc. > > > > > > > > > > > > I’d like to ask the community their thoughts about the proposed > > > design > > > > > > to be sure that all important things have been considered. > > > > > > > > > > > > Also, I have a question about services migration from AI 2.6 to a > > new > > > > > > solution. It’s very hard to provide tools for users migration, > > > because > > > > > > of significant changes. We don’t use utility cache anymore. > Should > > we > > > > > > spend time on this? > > > > > > > > > > > > I’ve prepared a definition of new Service Grid design, it’s > > described > > > > > below: > > > > > > > > > > > > *OVERVIEW* > > > > > > > > > > > > All nodes (servers and clients) are able to host services, but > the > > > > > > client nodes are excluded from service deployment by default. The > > > only > > > > > > way to deploy service on clients nodes is to specify node filter > in > > > > > > ServiceConfiguration. > > > > > > > > > > > > All deployed services are identified internally by “serviceId” > > > > > > (IgniteUuid). This allows us to build a base for such features as > > hot > > > > > > redeployment and service’s versioning. It’s important to have the > > > > > > ability to identify and manage services with the same name, but > > > > > > different version. > > > > > > > > > > > > All actions on service’s state change are processed according to > > > unified > > > > > flow: > > > > > > 1) Initiator sends over disco-spi a request to change service > state > > > > > > [deploy, undeploy]
Re: Table Names in Spark Catalog
Hi Nikolay,
I think it's actually pretty unfortunate that Spark uses term "database"
here, as it essentially refers to a schema in my view. Usually, database is
something you create a physical connection to, and connection is bind to
that database. To connect to another database you need to create a new
connection. In Spark, however, you can switch between "databases" within a
single session, which looks really weird to me because it's usually a
characteristic of a schema. Having said that, I understand your concern,
but I don't think there is an ideal solution.
As for your approach, I still don't understand how it will allow to fully
support schemas in catalog.
- How will you get a list of tables within a particular schema? In other
words, what would listTables() method return?
- How will you switch between the schemas?
- Etc.
I still think assuming database=schema is the best we can do here, but I
would be happy to hear another opinions from other community members.
OPTION_SCHEMA should definitely be introduced though (I thought we already
did, no?). CREATE TABLE will be supported with this ticket:
https://issues.apache.org/jira/browse/IGNITE-5780. For now we will have to
throw an exception if custom schema name is provided when creating a Spark
session, but table does not exist yet.
-Val
On Sun, Aug 26, 2018 at 7:56 AM Nikolay Izhikov wrote:
> Igniters,
>
> Personally, I don't like the solution with database == schema name.
>
> 1. I think we should try to use the right abstractions.
> schema == database doesn't sound right for me.
>
> Do you want to answer to all of our users something like that:
>
> - "How I can change Ignite SQL schema?"
> - "This is obvious, just use setDatabase("MY_SCHEMA_NAME")".
>
> 2. I think we restrict whole solution with that decision.
> If Ignite will support multiple databases in the future we just don't have
> a place for it.
>
> I think we should do the following:
>
> 1. IgniteExternalCatalog should be able to return *ALL* tables
> within Ignite instance.
> We shouldn't restrict tables list by schema by default.
> We should return tables with schema name - `schema.table`
>
> 2. We should introduce `OPTION_SCHEMA` for a dataframe to specify
> a schema.
>
> There is an issue with the second step: We can't use schema name
> in `CREATE TABLE` clause.
> This is restriction of current Ignite SQL.
>
> I propose to make the following:
>
> 1. For all write modes that requires the creation of table we
> should disallow usage of table outside of `SQL_PUBLIC`
> or usage of `OPTION_SCHEMA`. We should throw proper exception for
> this case.
>
> 2. Create a ticket to support `CREATE TABLE` with custom schema
> name.
>
> 3. After resolving ticket from step 2 we can add full support of
> custom schema to Spark integration.
>
> 4. We should throw an exception if user try to use setDatabase.
>
> Is that makes sense for you?
>
> В Вс, 26/08/2018 в 14:09 +0100, Stuart Macdonald пишет:
> > I'll go ahead and make the changes to represent the schema name as the
> > database name for the purposes of the Spark catalog.
> >
> > If anyone knows of an existing way to list all available schemata within
> an
> > Ignite instance please let me know, otherwise the first task will be
> > creating that mechanism.
> >
> > Stuart.
> >
> > On Fri, Aug 24, 2018 at 6:23 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Nikolay,
> > >
> > > If there are multiple configuration in XML, IgniteContext will always
> use
> > > only one of them. Looks like current approach simply doesn't work. I
> > > propose to report schema name as 'database' in Spark. If there are
> multiple
> > > clients, you would create multiple sessions and multiple catalogs.
> > >
> > > Makes sense?
> > >
> > > -Val
> > >
> > > On Fri, Aug 24, 2018 at 12:33 AM Nikolay Izhikov
> > > wrote:
> > >
> > > > Hello, Valentin.
> > > >
> > > > > catalog exist in scope of a single IgniteSparkSession> (and
> therefore
> > > >
> > > > single IgniteContext and single Ignite instance)?
> > > >
> > > > Yes.
> > > > Actually, I was thinking about use case when we have several Ignite
> > > > configuration in one XML file.
> > > > Now I see, may be this is too rare use-case to support.
> > > >
> > > > Stuart, Valentin, What is your proposal?
> > > >
> > > > В Ср, 22/08/2018 в 08:56 -0700, Valentin Kulichenko пишет:
> > > > > Nikolay,
> > > > >
> > > > > Whatever we decide on would be right :) Basically, we need to
> answer
> > >
> > > this
> > > > > question: does the catalog exist in scope of a single
> > >
> > > IgniteSparkSession
> > > > > (and therefore single IgniteContext and single Ignite instance)? In
> > >
> > > other
> > > > > words, in case of a rare use case when a single Spark application
> > > >
> > > > connects
> > > > > to multiple Ignite clusters, would there be a catalog created per
> > > >
> > > > cluster?
> > > >
[GitHub] ignite pull request #4628: IGNITE-9393: Fixed bug in reduce function in data...
GitHub user zaleslaw opened a pull request: https://github.com/apache/ignite/pull/4628 IGNITE-9393: Fixed bug in reduce function in dataset.compute You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9393 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4628.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4628 commit 96fffa56cd0b334a1dd3b678b68bc9224f5b4c02 Author: Zinoviev Alexey Date: 2018-08-27T17:44:07Z IGNITE-8924: Result of merge ---
[GitHub] ignite pull request #4627: IGNITE-7259: tests fixed.
GitHub user AMashenkov opened a pull request: https://github.com/apache/ignite/pull/4627 IGNITE-7259: tests fixed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-7259 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4627.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4627 commit 04f94948bbd89136d4555a2e3fccd9367169 Author: Igor Seliverstov Date: 2017-12-27T14:04:58Z IGNITE-7321 DML operation hangs in case exception is thrown from DHT enlist future commit 3818bc9d74b2bba594ef89cc2d1ac975da816b54 Author: Igor Seliverstov Date: 2017-12-27T14:33:18Z fix tests. commit eb7a615c9d32e648eceeafb09893b226fe276467 Author: Igor Seliverstov Date: 2017-12-27T14:47:39Z fix exception types. commit f4ce065ce863a741e0d14901e087ff69856b7d9f Author: Alexander Paschenko Date: 2017-12-27T16:23:23Z Merge branch 'master' into ignite-5571 commit 73bedeb4979d5474be654c6f8f80c168c9a0606f Author: Alexander Paschenko Date: 2017-12-27T19:00:36Z IGNITE-5571 More tests commit ccf16fcc4d819e33b71a3322ae99d48755ee6b76 Author: Alexander Paschenko Date: 2017-12-28T07:49:25Z IGNITE-7259: Fixed GridIndexRebuildSelfTest. This closes #3304. commit 53466fc059d027edb24b3d7f5da05b6be2e997fc Author: Igor Seliverstov Date: 2017-12-28T11:24:16Z fix partition checking. commit b9aa9ebca0dca79fec278415beb5753ef49a4c60 Author: Igor Seliverstov Date: 2017-12-29T12:18:30Z fix mvcc version checking on lock acquire commit a86af95ea1ee08c2f66751a8e2fc440e4671d686 Author: Igor Seliverstov Date: 2018-01-11T12:45:02Z TODOs, right tickets numbers, minors commit 59ba16afcab00326a60f53b3a7592d2978f19313 Author: Igor Seliverstov Date: 2018-01-11T14:45:53Z Merge branch 'master' into ignite-4191-master commit 79d876bce306802bf097a69030e1fcca61cec859 Author: Igor Seliverstov Date: 2018-01-11T14:49:22Z Merge branch 'master' into ignite-4191-master commit 9bd874c8d132f23842ae4bf8caa2c633b42b0789 Author: Igor Seliverstov Date: 2018-01-11T14:52:15Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java commit cbae5040399f433c0689df76e697df5a637ca1bc Author: Igor Seliverstov Date: 2018-01-11T15:29:13Z Disable DROP COLUMN with enabled mvcc commit c5bb6c68a3faf9f236ae8a1639bfaf495c825eb7 Author: Igor Seliverstov Date: 2018-01-12T09:09:52Z Merge branch 'master' into ignite-4191-master commit 02f4ffa82a6ebaaaeaf96c31f15c4e8191c188d0 Author: Igor Seliverstov Date: 2018-01-12T09:23:00Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/jdbc/thin/JdbcThinTcpIo.java # modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/jdbc/JdbcConnectionContext.java # modules/indexing/src/test/java/org/apache/ignite/testsuites/IgniteCacheQuerySelfTestSuite.java commit 21f9acdee409265f98893a2910e04effa63bcc2d Author: Igor Seliverstov Date: 2018-01-12T10:34:45Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/H2Tree.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/H2TreeIndex.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2ExtrasInnerIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2ExtrasLeafIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2InnerIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2LeafIO.java commit 5684b7224d077e285af35c5741d50332dec31eae Author: Igor Seliverstov Date: 2018-01-12T10:36:04Z Merge branch 'master' into ignite-4191-master commit 284c4335897fc82388baa99c960693b59a460e61 Author: Alexander Paschenko Date: 2018-01-17T09:20:35Z Merge remote-tracking branch 'apache/master' into ignite-5571 # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/jdbc/JdbcRequestHandler.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java # modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/IgnitePersistentStoreSchemaLoadTest.java #
[GitHub] ignite pull request #4626: IGNITE-9389 Avoid deadlocks on cache#close()
GitHub user dmekhanikov opened a pull request: https://github.com/apache/ignite/pull/4626 IGNITE-9389 Avoid deadlocks on cache#close() You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9389 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4626.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4626 commit 18bf109f386ea87470f9371faeafd7e2104e09b4 Author: Denis Mekhanikov Date: 2018-08-27T16:13:57Z IGNITE-9389 Add test. commit 3321f77bd7242ec0a26002bee152bf6522943586 Author: Denis Mekhanikov Date: 2018-08-27T16:16:16Z IGNITE-9389 Avoid deadlock on cache#close(). ---
[GitHub] ignite pull request #4483: IGNITE-9183 Add proper handling of UUID column in...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4483 ---
[GitHub] SomeFire opened a new pull request #4: show critical failures on top of the results page
SomeFire opened a new pull request #4: show critical failures on top of the results page URL: https://github.com/apache/ignite-teamcity-bot/pull/4 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (IGNITE-9393) [ML] KMeans fails on complex data in cache
Aleksey Zinoviev created IGNITE-9393: Summary: [ML] KMeans fails on complex data in cache Key: IGNITE-9393 URL: https://issues.apache.org/jira/browse/IGNITE-9393 Project: Ignite Issue Type: Bug Components: ml Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Described here http://apache-ignite-users.70518.x6.nabble.com/NPE-exception-in-KMeansTrainer-td23504.html#a23512 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4625: IGNITE-9392 : Specified TX timeout in test config...
GitHub user ilantukh opened a pull request: https://github.com/apache/ignite/pull/4625 IGNITE-9392 : Specified TX timeout in test configuration. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9392 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4625.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4625 commit 3ffc1065787ea67f4f07f9882fc9401062cc0f8e Author: Ilya Lantukh Date: 2018-08-27T14:31:37Z IGNITE-9392 : Specified TX timeout in test configuration. ---
[jira] [Created] (IGNITE-9392) CacheAsyncOperationsFailoverTxTest hangs on TC
Ilya Lantukh created IGNITE-9392: Summary: CacheAsyncOperationsFailoverTxTest hangs on TC Key: IGNITE-9392 URL: https://issues.apache.org/jira/browse/IGNITE-9392 Project: Ignite Issue Type: Bug Reporter: Ilya Lantukh Assignee: Ilya Lantukh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: QueryDetailMetrics for cache-less SQL queries
Correct, this also needs to be fixed.
On Mon, Aug 27, 2018 at 5:17 PM Evgenii Zhuravlev
wrote:
> Vladimir,
>
> Then it looks like we send wrong EVT_CACHE_QUERY_EXECUTED events too, since
> we send it for the same cache, as in your example.
>
> Evgenii
>
>
>
> пн, 27 авг. 2018 г. в 13:20, Vladimir Ozerov :
>
> > Folks,
> >
> > Can we leave query details metrics API unchanged, and simply fix how we
> > record them? Historically it worked as follows:
> > 1) IgniteCache.query() is invoked
> > 2) Query is executed, possibly on other caches!
> > 3) Metric is incremented for cache from p.1
> >
> > I.e. our query detail metrics were broken since their first days. I
> propose
> > to fix them as follows:
> > 1) Any SQL query() method is invoked (see GridQueryProcessor)
> > 2) We collect all cache IDs during parsing (this already happens)
> > 3) Record event for all cache IDs through
> > GridCacheQueryManager#collectMetrics
> >
> > This appears to be as the simplest and consistent solution to the
> problem.
> >
> > On Tue, Aug 21, 2018 at 1:09 PM Evgenii Zhuravlev <
> > e.zhuravlev...@gmail.com>
> > wrote:
> >
> > > Hi Alex,
> > >
> > > I agree that we can't move all metrics to ignite.metrics() and SPI
> > metrics
> > > is a good example here. I propose to move at least dataRegionMetrics,
> > > dataStorageMetrics to it, since the main API class is not a good place
> > for
> > > such things. If we will decide to choose option 1 from my first
> message,
> > > then, I will also add QueryDetailMetrics for cacheless queries to this
> > new
> > > class.
> > >
> > > пн, 20 авг. 2018 г. в 23:28, Alex Plehanov :
> > >
> > > > Hi Evgeny,
> > > >
> > > > Do you propose to move into IgniteMetrics absolutely all Ignite
> metrics
> > > or
> > > > just dataRegionMetrics, dataStorageMetrics and queryDetailMetrics? I
> > > think
> > > > you can't move all metrics into one place. Pluggable components and
> > > > different SPI implementations may have their own metric sets, and
> > perhaps
> > > > it's not such a good idea to try to fit them in one common fixed
> > > interface.
> > > >
> > > > 2018-08-20 18:14 GMT+03:00 Evgenii Zhuravlev <
> e.zhuravlev...@gmail.com
> > >:
> > > >
> > > > > As for now, metrics are accumulated for cache only when the query
> is
> > > run
> > > > > directly over this cache, for example, using ignite.cache("Some
> > > > > cache").sqlFieldsQuery("select ... from .."). When a query is
> started
> > > > using
> > > > > other APIs, it doesn't detect cache, to which this table belongs
> and
> > > > > doesn't save any metrics.
> > > > >
> > > > >
> > > > >
> > > > > 2018-08-17 16:44 GMT+03:00 Vladimir Ozerov :
> > > > >
> > > > > > Query is not executed on specific cache. It is executed on many
> > > caches.
> > > > > >
> > > > > > On Fri, Aug 17, 2018 at 6:10 AM Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > But internally the SQL query still runs on some cache, no? What
> > > > happens
> > > > > > to
> > > > > > > the metrics accumulated on that cache?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov <
> > > akuznet...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Dima,
> > > > > > > >
> > > > > > > > "cache-less" means that SQL executed directly on SQL engine.
> > > > > > > >
> > > > > > > > In previous version of Ignite we execute queries via cache:
> > > > > > > >
> > > > > > > > ignite.cache("Some cache").sqlFieldsQuery("select ... from
> ..")
> > > > > > > >
> > > > > > > > In current Ignite we can execute query directly without using
> > > cache
> > > > > as
> > > > > > > > "gateway".
> > > > > > > >
> > > > > > > > And if we execute query directly, metrics not update.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Evgeny, what is a "cache-less" SQL query?
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > > > > > > > > e.zhuravlev...@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Igniters,
> > > > > > > > > >
> > > > > > > > > > I've started to work on adding QueryDetailMetrics for
> > > > cache-less
> > > > > > SQL
> > > > > > > > > > queries(issue
> > > > https://issues.apache.org/jira/browse/IGNITE-6677)
> > > > > > and
> > > > > > > > > found
> > > > > > > > > > that it's required to change API. I don't think that
> adding
> > > > > methods
> > > > > > > > like
> > > > > > > > > > queryDetailMetrics, resetQueryDetailMetrics, as in
> > > IgniteCache
> > > > to
> > > > > > > > Ignite
> > > > > > > > > > class is a good idea. So, I see 2 possible solutions
> here:
> > > > > > > > > >
> > > > > > > > > > 1. Create IgniteMetrics(ignite.metrics()) and move
> metrics
> > >
Re: QueryDetailMetrics for cache-less SQL queries
Vladimir,
Then it looks like we send wrong EVT_CACHE_QUERY_EXECUTED events too, since
we send it for the same cache, as in your example.
Evgenii
пн, 27 авг. 2018 г. в 13:20, Vladimir Ozerov :
> Folks,
>
> Can we leave query details metrics API unchanged, and simply fix how we
> record them? Historically it worked as follows:
> 1) IgniteCache.query() is invoked
> 2) Query is executed, possibly on other caches!
> 3) Metric is incremented for cache from p.1
>
> I.e. our query detail metrics were broken since their first days. I propose
> to fix them as follows:
> 1) Any SQL query() method is invoked (see GridQueryProcessor)
> 2) We collect all cache IDs during parsing (this already happens)
> 3) Record event for all cache IDs through
> GridCacheQueryManager#collectMetrics
>
> This appears to be as the simplest and consistent solution to the problem.
>
> On Tue, Aug 21, 2018 at 1:09 PM Evgenii Zhuravlev <
> e.zhuravlev...@gmail.com>
> wrote:
>
> > Hi Alex,
> >
> > I agree that we can't move all metrics to ignite.metrics() and SPI
> metrics
> > is a good example here. I propose to move at least dataRegionMetrics,
> > dataStorageMetrics to it, since the main API class is not a good place
> for
> > such things. If we will decide to choose option 1 from my first message,
> > then, I will also add QueryDetailMetrics for cacheless queries to this
> new
> > class.
> >
> > пн, 20 авг. 2018 г. в 23:28, Alex Plehanov :
> >
> > > Hi Evgeny,
> > >
> > > Do you propose to move into IgniteMetrics absolutely all Ignite metrics
> > or
> > > just dataRegionMetrics, dataStorageMetrics and queryDetailMetrics? I
> > think
> > > you can't move all metrics into one place. Pluggable components and
> > > different SPI implementations may have their own metric sets, and
> perhaps
> > > it's not such a good idea to try to fit them in one common fixed
> > interface.
> > >
> > > 2018-08-20 18:14 GMT+03:00 Evgenii Zhuravlev >:
> > >
> > > > As for now, metrics are accumulated for cache only when the query is
> > run
> > > > directly over this cache, for example, using ignite.cache("Some
> > > > cache").sqlFieldsQuery("select ... from .."). When a query is started
> > > using
> > > > other APIs, it doesn't detect cache, to which this table belongs and
> > > > doesn't save any metrics.
> > > >
> > > >
> > > >
> > > > 2018-08-17 16:44 GMT+03:00 Vladimir Ozerov :
> > > >
> > > > > Query is not executed on specific cache. It is executed on many
> > caches.
> > > > >
> > > > > On Fri, Aug 17, 2018 at 6:10 AM Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > But internally the SQL query still runs on some cache, no? What
> > > happens
> > > > > to
> > > > > > the metrics accumulated on that cache?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov <
> > akuznet...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Dima,
> > > > > > >
> > > > > > > "cache-less" means that SQL executed directly on SQL engine.
> > > > > > >
> > > > > > > In previous version of Ignite we execute queries via cache:
> > > > > > >
> > > > > > > ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")
> > > > > > >
> > > > > > > In current Ignite we can execute query directly without using
> > cache
> > > > as
> > > > > > > "gateway".
> > > > > > >
> > > > > > > And if we execute query directly, metrics not update.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Evgeny, what is a "cache-less" SQL query?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > > > > > > > e.zhuravlev...@gmail.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Igniters,
> > > > > > > > >
> > > > > > > > > I've started to work on adding QueryDetailMetrics for
> > > cache-less
> > > > > SQL
> > > > > > > > > queries(issue
> > > https://issues.apache.org/jira/browse/IGNITE-6677)
> > > > > and
> > > > > > > > found
> > > > > > > > > that it's required to change API. I don't think that adding
> > > > methods
> > > > > > > like
> > > > > > > > > queryDetailMetrics, resetQueryDetailMetrics, as in
> > IgniteCache
> > > to
> > > > > > > Ignite
> > > > > > > > > class is a good idea. So, I see 2 possible solutions here:
> > > > > > > > >
> > > > > > > > > 1. Create IgniteMetrics(ignite.metrics()) and move metrics
> > from
> > > > > > > > > Ignite(like dataRegionMetrics and dataStorageMetrics) and
> > add a
> > > > new
> > > > > > > > > metric "queryDetailMetrics" to it. Of course, old methods
> > will
> > > be
> > > > > > > > > deprecated.
> > > > > > > > >
> > > > > > > > > 2. Finally create Ignite.sql() API, which was already
> > discussed
> > > > > here:
> > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > > >
[GitHub] ignite pull request #4624: IGNITE-9391 added checking result on waitForCondi...
GitHub user akalash opened a pull request: https://github.com/apache/ignite/pull/4624 IGNITE-9391 added checking result on waitForCondition in testRebalanc⦠â¦eEstimateFinishTime You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9391 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4624.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4624 commit d7fa9496f5976107d4ba96b721170a430f9e1d35 Author: Anton Kalashnikov Date: 2018-08-27T14:14:00Z IGNITE-9391 added checking result on waitForCondition in testRebalanceEstimateFinishTime ---
[jira] [Created] (IGNITE-9391) Incorrect calculated esitmated rabalancing finish time
Anton Kalashnikov created IGNITE-9391: - Summary: Incorrect calculated esitmated rabalancing finish time Key: IGNITE-9391 URL: https://issues.apache.org/jira/browse/IGNITE-9391 Project: Ignite Issue Type: Test Reporter: Anton Kalashnikov Actually looks like test CacheGroupsMetricsRebalanceTest.testRebalanceEstimateFinishTime is incorrect or we have bug in esitmated rabalancing finish time calculation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zzzadruga opened a new pull request #3: IGNITE-9333 Add statistics page
zzzadruga opened a new pull request #3: IGNITE-9333 Add statistics page URL: https://github.com/apache/ignite-teamcity-bot/pull/3 This is a draft version This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Hello
I'm new to Ignite and I would like to join Apache Ignite development. My JIRA's login: PetrovMikhail
[jira] [Created] (IGNITE-9390) MVCC: add MVCC support to .NET configuration.
Andrew Mashenkov created IGNITE-9390: Summary: MVCC: add MVCC support to .NET configuration. Key: IGNITE-9390 URL: https://issues.apache.org/jira/browse/IGNITE-9390 Project: Ignite Issue Type: Improvement Components: mvcc, platforms Reporter: Andrew Mashenkov Fix For: 2.7 We should make MVCC configurable from .NET client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9389) Concurrent Cache#close and put/get operations lead to a deadlock
Denis Mekhanikov created IGNITE-9389: Summary: Concurrent Cache#close and put/get operations lead to a deadlock Key: IGNITE-9389 URL: https://issues.apache.org/jira/browse/IGNITE-9389 Project: Ignite Issue Type: Bug Components: cache Affects Versions: 2.6 Reporter: Denis Mekhanikov Attachments: CacheConcurrentUseAndCloseTest.java When cache is closed on a client node, while it is being used concurrently in other threads, deadlock may occur. Test case, reproducing the problem is attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4552: IGNITE-9267: Deadlock between unsuccessful client...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4552 ---
[GitHub] ignite pull request #4623: IGNITE-9373: Fix mvcc tests.
GitHub user AMashenkov opened a pull request: https://github.com/apache/ignite/pull/4623 IGNITE-9373: Fix mvcc tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite IGNITE-9373 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4623.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4623 commit 6150d94c51b748fff361df32391131f1e68095e5 Author: Igor Seliverstov Date: 2017-12-27T11:32:58Z Fix no query in update plan. commit 2c212c18159848e413927dd05cf330e2be1cd97d Author: Alexander Paschenko Date: 2017-12-27T11:49:11Z IGNITE-7280: JDBC tests. commit e50c81e9ec3597414d8dbe775626654e0dc86eb0 Author: Alexander Paschenko Date: 2017-12-27T13:06:00Z IGNITE-7302: Fixed INFORMATIONAL_SCHEMA processing. This closes #3294. commit 04f94948bbd89136d4555a2e3fccd9367169 Author: Igor Seliverstov Date: 2017-12-27T14:04:58Z IGNITE-7321 DML operation hangs in case exception is thrown from DHT enlist future commit 3818bc9d74b2bba594ef89cc2d1ac975da816b54 Author: Igor Seliverstov Date: 2017-12-27T14:33:18Z fix tests. commit eb7a615c9d32e648eceeafb09893b226fe276467 Author: Igor Seliverstov Date: 2017-12-27T14:47:39Z fix exception types. commit f4ce065ce863a741e0d14901e087ff69856b7d9f Author: Alexander Paschenko Date: 2017-12-27T16:23:23Z Merge branch 'master' into ignite-5571 commit 73bedeb4979d5474be654c6f8f80c168c9a0606f Author: Alexander Paschenko Date: 2017-12-27T19:00:36Z IGNITE-5571 More tests commit ccf16fcc4d819e33b71a3322ae99d48755ee6b76 Author: Alexander Paschenko Date: 2017-12-28T07:49:25Z IGNITE-7259: Fixed GridIndexRebuildSelfTest. This closes #3304. commit 53466fc059d027edb24b3d7f5da05b6be2e997fc Author: Igor Seliverstov Date: 2017-12-28T11:24:16Z fix partition checking. commit b9aa9ebca0dca79fec278415beb5753ef49a4c60 Author: Igor Seliverstov Date: 2017-12-29T12:18:30Z fix mvcc version checking on lock acquire commit a86af95ea1ee08c2f66751a8e2fc440e4671d686 Author: Igor Seliverstov Date: 2018-01-11T12:45:02Z TODOs, right tickets numbers, minors commit 59ba16afcab00326a60f53b3a7592d2978f19313 Author: Igor Seliverstov Date: 2018-01-11T14:45:53Z Merge branch 'master' into ignite-4191-master commit 79d876bce306802bf097a69030e1fcca61cec859 Author: Igor Seliverstov Date: 2018-01-11T14:49:22Z Merge branch 'master' into ignite-4191-master commit 9bd874c8d132f23842ae4bf8caa2c633b42b0789 Author: Igor Seliverstov Date: 2018-01-11T14:52:15Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java commit cbae5040399f433c0689df76e697df5a637ca1bc Author: Igor Seliverstov Date: 2018-01-11T15:29:13Z Disable DROP COLUMN with enabled mvcc commit c5bb6c68a3faf9f236ae8a1639bfaf495c825eb7 Author: Igor Seliverstov Date: 2018-01-12T09:09:52Z Merge branch 'master' into ignite-4191-master commit 02f4ffa82a6ebaaaeaf96c31f15c4e8191c188d0 Author: Igor Seliverstov Date: 2018-01-12T09:23:00Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/jdbc/thin/JdbcThinTcpIo.java # modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/jdbc/JdbcConnectionContext.java # modules/indexing/src/test/java/org/apache/ignite/testsuites/IgniteCacheQuerySelfTestSuite.java commit 21f9acdee409265f98893a2910e04effa63bcc2d Author: Igor Seliverstov Date: 2018-01-12T10:34:45Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/H2Tree.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/H2TreeIndex.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2ExtrasInnerIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2ExtrasLeafIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2InnerIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2LeafIO.java commit 5684b7224d077e285af35c5741d50332dec31eae Author: Igor Seliverstov Date: 2018-01-12T10:36:04Z Merge branch 'master' into ignite-4191-master commit 284c4335897fc82388baa99c960693b59a460e61 Author: Alexander Paschenko Date: 2018-01-17T09:20:35Z Merge
[GitHub] ignite pull request #4572: IGNITE-9246 Transactions can wait for topology fu...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4572 ---
[GitHub] ignite pull request #4574: IGNITE-9147 Race between tx async rollback and lo...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4574 ---
Re: Compression prototype
According to my benchmarks - zstd compression algorithm [1] looks very interesting, it has a high compression ratio with quite good speed. AFAIK it supports external dictionaries, but I'm not sure about using it with "on the fly building" dictionaries. Anyway, have look at (it has ASF 2.0 friendly license). Also, here is data generator / loader [1]. If it will be useful for you we should ask Nikolay Izhikov to share public docs to start. [1] https://github.com/facebook/zstd [2] https://github.com/nizhikov/ignite-cod-data-loader On Mon, Aug 27, 2018 at 2:11 PM Ilya Kasnacheev wrote: > > Hello Vyacheslav! > > Unfortunately I have not found any efficient algorithms that will allow me > to use external dictionary as a pre-processed data structure. If plain gzip > is used without dictionary, the compression is around 0.7, as opposed to > 0.4 that I will get with custom implementation, AFAIR the performance was > also worse. I didn't really try it with dictionary, but I assume > performance will be even worse since it will have to scan dictionary before > getting to actual data. > > We have such a huge array of tests that we can just run them all with > compression enabled, see if there are any new failures. But the impact of > my commit is fairly low, it is only triggered when data is written to page > (maybe to WAL also?), and we don't really do much frivolous stuff to pages. > > Still, I am very much interested in finding existing compression > implementations with support of external dictionary; I am also very much > interested in having different implementations of compression for Apache > Ignite (such as per page compression) and comparing them by benchmark and > by code impact. I am also very interested in large standard datasets for > Apache Ignite (or generators thereof) so that we can run precise benchmarks > on various compression schemes. If you have any of the following, please > get back to me. > > Regards, > -- > Ilya Kasnacheev > > > пн, 27 авг. 2018 г. в 11:35, Vyacheslav Daradur : > > > Hi Igniters! > > > > Ilya, I'm glad to see one more person who is interested in the > > compression feature in Ignite. > > > > I looked through the pull request and want to share following thoughts: > > > > It's very dangerous using a custom algorithm in this way - you store > > serialized data separate from a dictionary and there are a lot of > > points when we may lose data: rebalancing, serialization errors, node > > rebooting and so on. > > > > I'd suggest the following ways to improve reliability: > > - use well know algorithms: zstd, deflater, lzma, gzip e.g. that > > allows us to decompress data in any situation > > - store the dictionary inside page with data > > > > Also, we have a lot of discussions [1] [2] about compression on > > BinaryObject and BinaryMarshaller level and Vladimir Ozerov was > > strictly against a compression on this level. > > If something has changed since then, you may look through [1] [2] [3] > > I've done a lot of research in algorithms comparison it may be useful > > for you. > > > > [1] > > http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-2-0-td10099.html > > [2] > > http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-td20679.html > > [3] https://issues.apache.org/jira/browse/IGNITE-3592 > > [4] https://issues.apache.org/jira/browse/IGNITE-5226 > > [5] https://github.com/daradurvs/ignite-compression > > On Sat, Aug 25, 2018 at 2:51 AM Denis Magda wrote: > > > > > > > > > > > Currently, the dictionary for decompression is only stored on heap. > > After > > > > restart there's compressed data in the PDS, but there's no dictionary > > :) > > > > > > > > > Basically, it means that I've lost my data, right? How about persisting > > > data to disk. > > > > > > Overall, we need Vladimir Ozerov to check the contribution. He was the > > one > > > who sponsored the IEP and knows the area best. > > > > > > -- > > > Denis > > > > > > On Fri, Aug 24, 2018 at 4:31 AM Ilya Kasnacheev < > > ilya.kasnach...@gmail.com> > > > wrote: > > > > > > > Hello! > > > > > > > > It is somewhat a part of IEP-20, since I have updated it with this > > > > particular direction. > > > > > > > > Regards, > > > > > > > > -- > > > > Ilya Kasnacheev > > > > > > > > 2018-08-24 2:56 GMT+03:00 Denis Magda : > > > > > > > > > Hi Ilya, > > > > > > > > > > Sounds terrific! Is this part of the following Ignite enhancement > > > > proposal? > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > > > 20%3A+Data+Compression+in+Ignite > > > > > > > > > > -- > > > > > Denis > > > > > > > > > > On Thu, Aug 23, 2018 at 5:17 AM Ilya Kasnacheev < > > > > ilya.kasnach...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Hello! > > > > > > > > > > > > My plan was to add a compression section to cache configuration, > > where > > > > > you > > > > > > can enable compression, enable key compression (which has heavier > > > > > > performance
[jira] [Created] (IGNITE-9388) IgniteProvider tries to access obolete ignite.run or download from slow archive
Roman Shtykh created IGNITE-9388: Summary: IgniteProvider tries to access obolete ignite.run or download from slow archive Key: IGNITE-9388 URL: https://issues.apache.org/jira/browse/IGNITE-9388 Project: Ignite Issue Type: Bug Affects Versions: 2.6 Reporter: Roman Shtykh Assignee: Roman Shtykh Although it downloads from the archive, for Japan, for instance, it takes 2-3 hours. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Compression prototype
Hello Vyacheslav! Unfortunately I have not found any efficient algorithms that will allow me to use external dictionary as a pre-processed data structure. If plain gzip is used without dictionary, the compression is around 0.7, as opposed to 0.4 that I will get with custom implementation, AFAIR the performance was also worse. I didn't really try it with dictionary, but I assume performance will be even worse since it will have to scan dictionary before getting to actual data. We have such a huge array of tests that we can just run them all with compression enabled, see if there are any new failures. But the impact of my commit is fairly low, it is only triggered when data is written to page (maybe to WAL also?), and we don't really do much frivolous stuff to pages. Still, I am very much interested in finding existing compression implementations with support of external dictionary; I am also very much interested in having different implementations of compression for Apache Ignite (such as per page compression) and comparing them by benchmark and by code impact. I am also very interested in large standard datasets for Apache Ignite (or generators thereof) so that we can run precise benchmarks on various compression schemes. If you have any of the following, please get back to me. Regards, -- Ilya Kasnacheev пн, 27 авг. 2018 г. в 11:35, Vyacheslav Daradur : > Hi Igniters! > > Ilya, I'm glad to see one more person who is interested in the > compression feature in Ignite. > > I looked through the pull request and want to share following thoughts: > > It's very dangerous using a custom algorithm in this way - you store > serialized data separate from a dictionary and there are a lot of > points when we may lose data: rebalancing, serialization errors, node > rebooting and so on. > > I'd suggest the following ways to improve reliability: > - use well know algorithms: zstd, deflater, lzma, gzip e.g. that > allows us to decompress data in any situation > - store the dictionary inside page with data > > Also, we have a lot of discussions [1] [2] about compression on > BinaryObject and BinaryMarshaller level and Vladimir Ozerov was > strictly against a compression on this level. > If something has changed since then, you may look through [1] [2] [3] > I've done a lot of research in algorithms comparison it may be useful > for you. > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-2-0-td10099.html > [2] > http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-td20679.html > [3] https://issues.apache.org/jira/browse/IGNITE-3592 > [4] https://issues.apache.org/jira/browse/IGNITE-5226 > [5] https://github.com/daradurvs/ignite-compression > On Sat, Aug 25, 2018 at 2:51 AM Denis Magda wrote: > > > > > > > > Currently, the dictionary for decompression is only stored on heap. > After > > > restart there's compressed data in the PDS, but there's no dictionary > :) > > > > > > Basically, it means that I've lost my data, right? How about persisting > > data to disk. > > > > Overall, we need Vladimir Ozerov to check the contribution. He was the > one > > who sponsored the IEP and knows the area best. > > > > -- > > Denis > > > > On Fri, Aug 24, 2018 at 4:31 AM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > > wrote: > > > > > Hello! > > > > > > It is somewhat a part of IEP-20, since I have updated it with this > > > particular direction. > > > > > > Regards, > > > > > > -- > > > Ilya Kasnacheev > > > > > > 2018-08-24 2:56 GMT+03:00 Denis Magda : > > > > > > > Hi Ilya, > > > > > > > > Sounds terrific! Is this part of the following Ignite enhancement > > > proposal? > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > > 20%3A+Data+Compression+in+Ignite > > > > > > > > -- > > > > Denis > > > > > > > > On Thu, Aug 23, 2018 at 5:17 AM Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Hello! > > > > > > > > > > My plan was to add a compression section to cache configuration, > where > > > > you > > > > > can enable compression, enable key compression (which has heavier > > > > > performance implications), adjust dictionary gathering settings, > and in > > > > the > > > > > future possibly choose betwen algorithms. In fact I'm not sure, > since > > > my > > > > > assumption is that you can always just use latest, but > maybe > > > we > > > > > can have e.g. very fast and not very strong vs. slower but stronger > > > one. > > > > > > > > > > I'm not sure yet if we should share dictionary between all caches > vs. > > > > > having separate dictionary for every cache. > > > > > > > > > > With regards to data format, of course there will be room for > further > > > > > extension. > > > > > > > > > > Regards, > > > > > > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > 2018-08-23 15:13 GMT+03:00 Sergey Kozlov : > > > > > > > > > > > Hi Ilya > > > > > > > > > > > > Is there a plan to introduce it as
[jira] [Created] (IGNITE-9387) [ML] Model updating
Yury Babak created IGNITE-9387: -- Summary: [ML] Model updating Key: IGNITE-9387 URL: https://issues.apache.org/jira/browse/IGNITE-9387 Project: Ignite Issue Type: Improvement Components: ml Reporter: Yury Babak Assignee: Alexey Platonov Fix For: 2.7 In trainer interface we need to support model updating by batches after first training -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9386) control.sh --tx can produce confusing results when limit is set to small value
Alexei Scherbakov created IGNITE-9386: - Summary: control.sh --tx can produce confusing results when limit is set to small value Key: IGNITE-9386 URL: https://issues.apache.org/jira/browse/IGNITE-9386 Project: Ignite Issue Type: Improvement Reporter: Alexei Scherbakov This is happening because currently the limit is applied to primary and backup transactions, which breaks output post-filtering (removal of primary and backup transactions from output if near is present). Possible solution: apply limit only to near valid transactions. If some txs have no near part (broken tx topology), they should be always visible in output, probably with special "broken" marking. Best way to achieve this - implement tx paging on client side (using continuous mapping) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Binary Client Protocol client hangs in case of OOM on server
I believe, you should file a ticket on this one. Best Regards, Igor On Fri, Aug 24, 2018 at 5:57 PM dmitrievanthony wrote: > I've rebuilt Ignite from master and tried again, but the behaviour is the > same. If the page is very big (compare with heap size) server node fails > with OOM and the client hangs. > > [14:54:06,273][SEVERE][client-connector-#101][ClientListenerProcessor] > Runtime error caught during grid runnable execution: GridWorker > [name=message-received-notify, igniteInstanceName=null, finished=false, > hashCode=967716821, interrupted=false, runner=client-connector-#101] > java.lang.OutOfMemoryError: Java heap space > at > > org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk.reallocate(BinaryMemoryAllocatorChunk.java:69) > at > > org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.ensureCapacity(BinaryHeapOutputStream.java:65) > at > > org.apache.ignite.internal.binary.streams.BinaryAbstractOutputStream.writeByteArray(BinaryAbstractOutputStream.java:41) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteByteArray(BinaryWriterExImpl.java:530) > at > > org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:634) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506) > at > > org.apache.ignite.internal.processors.platform.client.cache.ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.java:44) > at > > org.apache.ignite.internal.processors.platform.client.cache.ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.java:29) > at > > org.apache.ignite.internal.processors.platform.client.cache.ClientCacheQueryCursor.writePage(ClientCacheQueryCursor.java:80) > at > > org.apache.ignite.internal.processors.platform.client.cache.ClientCacheQueryResponse.encode(ClientCacheQueryResponse.java:50) > at > > org.apache.ignite.internal.processors.platform.client.ClientMessageParser.encode(ClientMessageParser.java:379) > at > > org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:172) > at > > org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:45) > at > > org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279) > at > > org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) > at > > org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > > org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Exception in thread "client-connector-#101" java.lang.OutOfMemoryError: > Java > heap space > at > > org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk.reallocate(BinaryMemoryAllocatorChunk.java:69) > at > > org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.ensureCapacity(BinaryHeapOutputStream.java:65) > at > > org.apache.ignite.internal.binary.streams.BinaryAbstractOutputStream.writeByteArray(BinaryAbstractOutputStream.java:41) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteByteArray(BinaryWriterExImpl.java:530) > at > > org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:634) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:223) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151) > at > > org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectDetached(BinaryWriterExImpl.java:1506) > at > > org.apache.ignite.internal.processors.platform.client.cache.ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.java:44) > at > > org.apache.ignite.internal.processors.platform.client.cache.ClientCacheEntryQueryCursor.writeEntry(ClientCacheEntryQueryCursor.java:29) > at > >
[jira] [Created] (IGNITE-9385) Redesign and Refactor UI part 2
Dmitriy Shabalin created IGNITE-9385: Summary: Redesign and Refactor UI part 2 Key: IGNITE-9385 URL: https://issues.apache.org/jira/browse/IGNITE-9385 Project: Ignite Issue Type: Improvement Components: wizards Reporter: Dmitriy Shabalin Assignee: Dmitriy Shabalin We should refactor all screens to use latest modern controls as on "Configuration" screen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4621: IGNITE-9367 CPP: ODBC client crashes after execut...
GitHub user gromtech opened a pull request: https://github.com/apache/ignite/pull/4621 IGNITE-9367 CPP: ODBC client crashes after executing query⦠...by using a closed connection You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9367 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4621.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4621 commit 3898e7ccb16e61039d93720e6f0865acbc9acd34 Author: Roman Guseinov Date: 2018-08-27T10:26:01Z IGNITE-9367 CPP: ODBC client crashes after executing query with closed connection ---
Re: QueryDetailMetrics for cache-less SQL queries
Folks,
Can we leave query details metrics API unchanged, and simply fix how we
record them? Historically it worked as follows:
1) IgniteCache.query() is invoked
2) Query is executed, possibly on other caches!
3) Metric is incremented for cache from p.1
I.e. our query detail metrics were broken since their first days. I propose
to fix them as follows:
1) Any SQL query() method is invoked (see GridQueryProcessor)
2) We collect all cache IDs during parsing (this already happens)
3) Record event for all cache IDs through
GridCacheQueryManager#collectMetrics
This appears to be as the simplest and consistent solution to the problem.
On Tue, Aug 21, 2018 at 1:09 PM Evgenii Zhuravlev
wrote:
> Hi Alex,
>
> I agree that we can't move all metrics to ignite.metrics() and SPI metrics
> is a good example here. I propose to move at least dataRegionMetrics,
> dataStorageMetrics to it, since the main API class is not a good place for
> such things. If we will decide to choose option 1 from my first message,
> then, I will also add QueryDetailMetrics for cacheless queries to this new
> class.
>
> пн, 20 авг. 2018 г. в 23:28, Alex Plehanov :
>
> > Hi Evgeny,
> >
> > Do you propose to move into IgniteMetrics absolutely all Ignite metrics
> or
> > just dataRegionMetrics, dataStorageMetrics and queryDetailMetrics? I
> think
> > you can't move all metrics into one place. Pluggable components and
> > different SPI implementations may have their own metric sets, and perhaps
> > it's not such a good idea to try to fit them in one common fixed
> interface.
> >
> > 2018-08-20 18:14 GMT+03:00 Evgenii Zhuravlev :
> >
> > > As for now, metrics are accumulated for cache only when the query is
> run
> > > directly over this cache, for example, using ignite.cache("Some
> > > cache").sqlFieldsQuery("select ... from .."). When a query is started
> > using
> > > other APIs, it doesn't detect cache, to which this table belongs and
> > > doesn't save any metrics.
> > >
> > >
> > >
> > > 2018-08-17 16:44 GMT+03:00 Vladimir Ozerov :
> > >
> > > > Query is not executed on specific cache. It is executed on many
> caches.
> > > >
> > > > On Fri, Aug 17, 2018 at 6:10 AM Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > But internally the SQL query still runs on some cache, no? What
> > happens
> > > > to
> > > > > the metrics accumulated on that cache?
> > > > >
> > > > > D.
> > > > >
> > > > > On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov <
> akuznet...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Dima,
> > > > > >
> > > > > > "cache-less" means that SQL executed directly on SQL engine.
> > > > > >
> > > > > > In previous version of Ignite we execute queries via cache:
> > > > > >
> > > > > > ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")
> > > > > >
> > > > > > In current Ignite we can execute query directly without using
> cache
> > > as
> > > > > > "gateway".
> > > > > >
> > > > > > And if we execute query directly, metrics not update.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Evgeny, what is a "cache-less" SQL query?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > > > > > > e.zhuravlev...@gmail.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > I've started to work on adding QueryDetailMetrics for
> > cache-less
> > > > SQL
> > > > > > > > queries(issue
> > https://issues.apache.org/jira/browse/IGNITE-6677)
> > > > and
> > > > > > > found
> > > > > > > > that it's required to change API. I don't think that adding
> > > methods
> > > > > > like
> > > > > > > > queryDetailMetrics, resetQueryDetailMetrics, as in
> IgniteCache
> > to
> > > > > > Ignite
> > > > > > > > class is a good idea. So, I see 2 possible solutions here:
> > > > > > > >
> > > > > > > > 1. Create IgniteMetrics(ignite.metrics()) and move metrics
> from
> > > > > > > > Ignite(like dataRegionMetrics and dataStorageMetrics) and
> add a
> > > new
> > > > > > > > metric "queryDetailMetrics" to it. Of course, old methods
> will
> > be
> > > > > > > > deprecated.
> > > > > > > >
> > > > > > > > 2. Finally create Ignite.sql() API, which was already
> discussed
> > > > here:
> > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > > > > com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> > > > > > > > and place "queryDetailMetrics" metric there. Here is the
> ticket
> > > for
> > > > > > this
> > > > > > > > change: https://issues.apache.org/jira/browse/IGNITE-4701
> > > > > > > >
> > > > > > > > Personally, I think that the second solution looks better in
> > this
> > > > > case,
> > > > > > > > however, moving dataRegionMetrics and dataStorageMetrics to
> > > > > > > > ignite.matrics() is still a good idea - IMO, Ignite
[GitHub] ignite pull request #4620: IGNITE-9265: MVCC TX: Two rows with the same key ...
GitHub user AMashenkov opened a pull request: https://github.com/apache/ignite/pull/4620 IGNITE-9265: MVCC TX: Two rows with the same key in one MERGE statement produce an exception Tests added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9265 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4620.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4620 commit 3818bc9d74b2bba594ef89cc2d1ac975da816b54 Author: Igor Seliverstov Date: 2017-12-27T14:33:18Z fix tests. commit eb7a615c9d32e648eceeafb09893b226fe276467 Author: Igor Seliverstov Date: 2017-12-27T14:47:39Z fix exception types. commit f4ce065ce863a741e0d14901e087ff69856b7d9f Author: Alexander Paschenko Date: 2017-12-27T16:23:23Z Merge branch 'master' into ignite-5571 commit 73bedeb4979d5474be654c6f8f80c168c9a0606f Author: Alexander Paschenko Date: 2017-12-27T19:00:36Z IGNITE-5571 More tests commit ccf16fcc4d819e33b71a3322ae99d48755ee6b76 Author: Alexander Paschenko Date: 2017-12-28T07:49:25Z IGNITE-7259: Fixed GridIndexRebuildSelfTest. This closes #3304. commit 53466fc059d027edb24b3d7f5da05b6be2e997fc Author: Igor Seliverstov Date: 2017-12-28T11:24:16Z fix partition checking. commit b9aa9ebca0dca79fec278415beb5753ef49a4c60 Author: Igor Seliverstov Date: 2017-12-29T12:18:30Z fix mvcc version checking on lock acquire commit a86af95ea1ee08c2f66751a8e2fc440e4671d686 Author: Igor Seliverstov Date: 2018-01-11T12:45:02Z TODOs, right tickets numbers, minors commit 59ba16afcab00326a60f53b3a7592d2978f19313 Author: Igor Seliverstov Date: 2018-01-11T14:45:53Z Merge branch 'master' into ignite-4191-master commit 79d876bce306802bf097a69030e1fcca61cec859 Author: Igor Seliverstov Date: 2018-01-11T14:49:22Z Merge branch 'master' into ignite-4191-master commit 9bd874c8d132f23842ae4bf8caa2c633b42b0789 Author: Igor Seliverstov Date: 2018-01-11T14:52:15Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java commit cbae5040399f433c0689df76e697df5a637ca1bc Author: Igor Seliverstov Date: 2018-01-11T15:29:13Z Disable DROP COLUMN with enabled mvcc commit c5bb6c68a3faf9f236ae8a1639bfaf495c825eb7 Author: Igor Seliverstov Date: 2018-01-12T09:09:52Z Merge branch 'master' into ignite-4191-master commit 02f4ffa82a6ebaaaeaf96c31f15c4e8191c188d0 Author: Igor Seliverstov Date: 2018-01-12T09:23:00Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/jdbc/thin/JdbcThinTcpIo.java # modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/jdbc/JdbcConnectionContext.java # modules/indexing/src/test/java/org/apache/ignite/testsuites/IgniteCacheQuerySelfTestSuite.java commit 21f9acdee409265f98893a2910e04effa63bcc2d Author: Igor Seliverstov Date: 2018-01-12T10:34:45Z Merge branch 'master' into ignite-4191-master # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/H2Tree.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/H2TreeIndex.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2ExtrasInnerIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2ExtrasLeafIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2InnerIO.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/io/H2LeafIO.java commit 5684b7224d077e285af35c5741d50332dec31eae Author: Igor Seliverstov Date: 2018-01-12T10:36:04Z Merge branch 'master' into ignite-4191-master commit 284c4335897fc82388baa99c960693b59a460e61 Author: Alexander Paschenko Date: 2018-01-17T09:20:35Z Merge remote-tracking branch 'apache/master' into ignite-5571 # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/jdbc/JdbcRequestHandler.java # modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java # modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/IgnitePersistentStoreSchemaLoadTest.java # modules/indexing/src/test/java/org/apache/ignite/testsuites/IgniteCacheQuerySelfTestSuite.java commit e3341a4b9d94600533fed000c195a81294393f28 Author:
Re: [MTCGA]: new failures in builds [1588345] needs to be handled
Folks, this is a new failure in IGFS/Hadoop domain. Looks like the http://www.us.apache.org/dist/hadoop/core/hadoop-2.5.2/hadoop-2.5.2.tar.gz was moved (the URL gives 404). Vladimir, can you take a look or point community how to take care of this failure? Should we simply upgrade the Hadoop version? пн, 27 авг. 2018 г. в 9:43, : > Hi Ignite Developer, > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. > I hope you can help. > > *Recently contributed test failed in master > org.apache.ignite.testsuites.IgniteIgfsLinuxAndMacOSTestSuite.initializationError > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4034437294452110435=%3Cdefault%3E=testDetails > No changes in build > > - If your changes can led to this failure(s), please create issue > with label MakeTeamCityGreenAgain and assign it to you. > -- If you have fix, please set ticket to PA state and write to dev > list fix is ready > -- For case fix will require some time please mute test and set > label Muted_Test to issue > - If you know which change caused failure please contact change > author directly > - If you don't know which change caused failure please send > message to dev list to find out > Should you have any questions please contact dev@ignite.apache.org > Best Regards, > MTCGA.Bot > Notification generated at Mon Aug 27 09:43:20 MSK 2018 >
Re: [MTCGA]: new failures in builds [1731430, 1470304, 1263230] needs to be handled
This can be ignored for now, last 10 runs are green. сб, 25 авг. 2018 г. в 15:12, : > Hi Ignite Developer, > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. > I hope you can help. > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestPutGetTime > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7658720346544183028=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestGetAndPutIfAbsent > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6996817533011399981=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestContainsKeys > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7285770292804096604=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestPutAll > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6390205996503748689=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestRemoveAllKeysIterVector > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8758275678114310806=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheStoreTestSuite: > LocalLoadCacheSingleNodeNoPredicate > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=9015012699833728261=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestCreateCache > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5095691067384514204=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestLocalClearAllIterArray > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=3166615619754517087=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestPutGetDate > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2606088378994662262=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestLocalEvictIterArray > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7575936322131817559=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheStoreTestSuite: > LoadCacheSeveralNodesNoPredicate > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7743436803082385641=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestContainsKey > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1115225155613585414=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestLocalClearAllIterList > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=2661124177463452745=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheStoreTestSuite: > LoadCacheSingleNodeNoPredicate > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=3359013355795990869=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: TestGet > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5606130551689605135=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestPutIfAbsent > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=4462541653374489660=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestRemoveAllKeysIterArray > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2310399454784119338=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestGetAndReplace > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1749168869106643063=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: > BinaryIdentityResolverTestSuite: IdentityEquilityWithGuid > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3273658252531597324=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: > TestLocalEvictIterSet > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=3524550646347789055=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: TestSizes > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=2370244242758466103=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: > BinaryIdentityResolverTestSuite: IdentityEquilityWithoutGuid > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=2359304547582357827=%3Cdefault%3E=testDetails > > *New test failure in master IgniteCoreTest: CacheTestSuite: >
[jira] [Created] (IGNITE-9384) Transaction state PREPARED may be set too early or too late
Alexey Goncharuk created IGNITE-9384:
Summary: Transaction state PREPARED may be set too early or too
late
Key: IGNITE-9384
URL: https://issues.apache.org/jira/browse/IGNITE-9384
Project: Ignite
Issue Type: Bug
Reporter: Alexey Goncharuk
I see the following issues in the {{GridDhtTxPrepareFutureAdapter}} class:
1) {{PREPARED}} state on near local transaction may be set during the future
completion. I think the {{onComplete(res)}} method should be corrected as
follows:
{code}
if (last || tx.isSystemInvalidate() && !(tx.near() && tx.local()))
tx.state(PREPARED);
...
{code}
2) In {{onDone}} we have the following code:
{code}
if (REPLIED_UPD.compareAndSet(this, 0, 1)) {
GridNearTxPrepareResponse res = createPrepareResponse(this.err);
try {
sendPrepareResponse(res);
}
finally {
// Will call super.onDone().
onComplete(res);
}
return true;
}
...
{code}
This code will send near prepare response to the near node before the local DHT
transaction sets it's state to {{PREPARED}} which violates the invariant that
all transactions are prepared before any of the transactions is committed. I
think these two methods should be swapped, but we need to carefully check the
error handling (note that {{onComplete}} is called in {{finally}} block).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-9383) Add to the documentation that Ignite cluster requires that each nodes have direct connection to any nodes of grid.
Evgenii Zhuravlev created IGNITE-9383: - Summary: Add to the documentation that Ignite cluster requires that each nodes have direct connection to any nodes of grid. Key: IGNITE-9383 URL: https://issues.apache.org/jira/browse/IGNITE-9383 Project: Ignite Issue Type: Improvement Components: documentation Affects Versions: 2.6 Reporter: Evgenii Zhuravlev -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Compression prototype
Hi Igniters! Ilya, I'm glad to see one more person who is interested in the compression feature in Ignite. I looked through the pull request and want to share following thoughts: It's very dangerous using a custom algorithm in this way - you store serialized data separate from a dictionary and there are a lot of points when we may lose data: rebalancing, serialization errors, node rebooting and so on. I'd suggest the following ways to improve reliability: - use well know algorithms: zstd, deflater, lzma, gzip e.g. that allows us to decompress data in any situation - store the dictionary inside page with data Also, we have a lot of discussions [1] [2] about compression on BinaryObject and BinaryMarshaller level and Vladimir Ozerov was strictly against a compression on this level. If something has changed since then, you may look through [1] [2] [3] I've done a lot of research in algorithms comparison it may be useful for you. [1] http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-2-0-td10099.html [2] http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-td20679.html [3] https://issues.apache.org/jira/browse/IGNITE-3592 [4] https://issues.apache.org/jira/browse/IGNITE-5226 [5] https://github.com/daradurvs/ignite-compression On Sat, Aug 25, 2018 at 2:51 AM Denis Magda wrote: > > > > > Currently, the dictionary for decompression is only stored on heap. After > > restart there's compressed data in the PDS, but there's no dictionary :) > > > Basically, it means that I've lost my data, right? How about persisting > data to disk. > > Overall, we need Vladimir Ozerov to check the contribution. He was the one > who sponsored the IEP and knows the area best. > > -- > Denis > > On Fri, Aug 24, 2018 at 4:31 AM Ilya Kasnacheev > wrote: > > > Hello! > > > > It is somewhat a part of IEP-20, since I have updated it with this > > particular direction. > > > > Regards, > > > > -- > > Ilya Kasnacheev > > > > 2018-08-24 2:56 GMT+03:00 Denis Magda : > > > > > Hi Ilya, > > > > > > Sounds terrific! Is this part of the following Ignite enhancement > > proposal? > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > 20%3A+Data+Compression+in+Ignite > > > > > > -- > > > Denis > > > > > > On Thu, Aug 23, 2018 at 5:17 AM Ilya Kasnacheev < > > ilya.kasnach...@gmail.com > > > > > > > wrote: > > > > > > > Hello! > > > > > > > > My plan was to add a compression section to cache configuration, where > > > you > > > > can enable compression, enable key compression (which has heavier > > > > performance implications), adjust dictionary gathering settings, and in > > > the > > > > future possibly choose betwen algorithms. In fact I'm not sure, since > > my > > > > assumption is that you can always just use latest, but maybe > > we > > > > can have e.g. very fast and not very strong vs. slower but stronger > > one. > > > > > > > > I'm not sure yet if we should share dictionary between all caches vs. > > > > having separate dictionary for every cache. > > > > > > > > With regards to data format, of course there will be room for further > > > > extension. > > > > > > > > Regards, > > > > > > > > -- > > > > Ilya Kasnacheev > > > > > > > > 2018-08-23 15:13 GMT+03:00 Sergey Kozlov : > > > > > > > > > Hi Ilya > > > > > > > > > > Is there a plan to introduce it as an option of Ignite configuration? > > > In > > > > > that instead the boolean type I suggest to use the enum and reserve > > the > > > > > ability to extend compressions algorithms in future > > > > > > > > > > On Thu, Aug 23, 2018 at 1:09 PM, Ilya Kasnacheev < > > > > > ilya.kasnach...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hello! > > > > > > > > > > > > I want to share with the developer community my compression > > > prototype. > > > > > > > > > > > > Long story short, it compresses BinaryObject's byte[] as they are > > > > written > > > > > > to Durable Memory page, operating on a pre-built dictionary. > > Typical > > > > > > compression ratio is 0.4 (meaning 2.5x compression) using custom > > > > > > LZW+Huffman. Metadata, indexes and primitive values are unaffected > > > > > > entirely. > > > > > > > > > > > > This is akin to DB2's table-level compression[1] but independently > > > > > > invented. > > > > > > > > > > > > On Yardstick tests performance hit is -6% with PDS and up to -25% > > (in > > > > > > throughput) with In-Memory loads. It also means you can fit ~twice > > as > > > > > much > > > > > > data into the same IM cluster, or have higher ram/disk ratio with > > PDS > > > > > > cluster, saving on hardware or decreasing latency. > > > > > > > > > > > > The code is available as PR 4295[2] (set > > > IGNITE_ENABLE_COMPRESSION=true > > > > > to > > > > > > activate). Note that it will not presently survive a PDS node > > > restart. > > > > > > The impact is very small, the patch should be applicable to most > > 2.x > > > > > > releases. > > > > > > > > > > > > Sure there's a long way
Re: Some problem with execution ignite.bat in master
Dmitry, Not sure I understand the issue, but I see no reason to rollback anything. In case we have some issues, let's just fix them. сб, 25 авг. 2018 г. в 0:53, Dmitriy Pavlov : > Hi Anton, > > Do you have objections about a partial rollback of changes? > > Sincerely, > Dmitriy Pavlov > > пт, 24 авг. 2018 г. в 14:24, ARomantsov : > >> Hello, Igniters. >> >> I'm testing some java keys with ignite.bat and notice that using more that >> four keys lead to internal problem and ignite.bat ignoring them all. >> >> So I created issue https://issues.apache.org/jira/browse/IGNITE-8837 and >> investigated , below what i founded >> >> 1) inside file ignite.bat happen call bin/include/parseargs.bat with next >> code to parse arguments >> set convertArgsCmd="!JAVA_HOME!\bin\java.exe" -cp "%CP%" >> org.apache.ignite.startup.cmdline.CommandLineTransformer %* >> >> 2) call of org.apache.ignite.startup.cmdline.CommandLineTransformer >> correct >> parsing JVM arguments only if they count less five >> >> 3) This problem (2) easy fix by add to set in convertArgsCmd one of next >> Java key ( -Dfile.encoding=IBM866 or -Dfile.encoding=UTF-8) >> >> 4) Like part of issue https://issues.apache.org/jira/browse/IGNITE-7135 >> it >> already had fix (3) and included in Ignite 2.4 >> >> 5) Unfortunately, when fix issue >> https://issues.apache.org/jira/browse/IGNITE-898 that key was removed >> and >> that lead to problem in 2.5 , 2.6, master >> >> What correct steps to back ignite.bat working correct on windows in >> master / >> 2.7? Both of ticket in (4) / (5) fix another problems , is correct to >> reopen >> one of them? >> >> >> >> -- >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >> >
[jira] [Created] (IGNITE-9382) Node.js fails to process large payloads
Roman Shtykh created IGNITE-9382: Summary: Node.js fails to process large payloads Key: IGNITE-9382 URL: https://issues.apache.org/jira/browse/IGNITE-9382 Project: Ignite Issue Type: Bug Components: thin client Reporter: Roman Shtykh Looks like finalizing response is done too early. This can be easily checked with an SQL _limit_ 1 and _limit 100_. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Request for review : IGNITE-3303 Apache Flink Integration - Flink source
Hi, I have updated the PR with additional tests. Please review and share feedback. This PR is related to IgniteSink but allows to stream data from Ignite. PR https://github.com/apache/ignite/pull/870/files Review https://reviews.ignite.apache.org/ignite/review/IGNT-CR-135 Regards, Saikat
[MTCGA]: new failures in builds [1588345] needs to be handled
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *Recently contributed test failed in master org.apache.ignite.testsuites.IgniteIgfsLinuxAndMacOSTestSuite.initializationError https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4034437294452110435=%3Cdefault%3E=testDetails No changes in build - If your changes can led to this failure(s), please create issue with label MakeTeamCityGreenAgain and assign it to you. -- If you have fix, please set ticket to PA state and write to dev list fix is ready -- For case fix will require some time please mute test and set label Muted_Test to issue - If you know which change caused failure please contact change author directly - If you don't know which change caused failure please send message to dev list to find out Should you have any questions please contact dev@ignite.apache.org Best Regards, MTCGA.Bot Notification generated at Mon Aug 27 09:43:20 MSK 2018
