[MTCGA]: new failures in builds [1712385] needs to be handled
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *New test failure in master CacheAbstractTest.TestCacheConfigurationExpiryPolicy https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5800368842121996296=%3Cdefault%3E=testDetails No changes in build - If your changes can led to this failure(s), please create issue with label MakeTeamCityGreenAgain and assign it to you. -- If you have fix, please set ticket to PA state and write to dev list fix is ready -- For case fix will require some time please mute test and set label Muted_Test to issue - If you know which change caused failure please contact change author directly - If you don't know which change caused failure please send message to dev list to find out Should you have any questions please contact dpav...@apache.org or write to dev.list Best Regards, MTCGA.Bot Notification generated at Thu Aug 23 08:37:38 MSK 2018
[GitHub] ignite pull request #4531: IGNITE-9180 IgniteSparkSession Should Copy State ...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4531 ---
[GitHub] ignite pull request #4599: IGNITE-9353: Removed "Known issue, possible deadl...
GitHub user shroman opened a pull request: https://github.com/apache/ignite/pull/4599 IGNITE-9353: Removed "Known issue, possible deadlock in case of low p⦠â¦riority cache rebalancing delayed" comment from GridCacheRebalancingSyncSelfTest#getConfiguration. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shroman/ignite IGNITE-9353 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4599.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4599 commit f4b5e92082a5fc3f5f74a4e2ba71a1d75be8917b Author: shroman Date: 2018-08-23T04:22:23Z IGNITE-9353: Removed "Known issue, possible deadlock in case of low priority cache rebalancing delayed" comment from GridCacheRebalancingSyncSelfTest#getConfiguration. ---
[GitHub] ignite pull request #4598: IGNITE-9353: Removed "Known issue, possible deadl...
Github user shroman closed the pull request at: https://github.com/apache/ignite/pull/4598 ---
[GitHub] ignite pull request #4597: IGNITE-9350 Added checks for empty data in web so...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4597 ---
[GitHub] ignite pull request #4598: IGNITE-9353: Removed "Known issue, possible deadl...
GitHub user shroman opened a pull request: https://github.com/apache/ignite/pull/4598 IGNITE-9353: Removed "Known issue, possible deadlock in case of low p⦠â¦riority cache rebalancing delayed" comment from GridCacheRebalancingSyncSelfTest#getConfiguration. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shroman/ignite master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4598.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4598 commit fa2c8e72e7559803eee250f35c5120e0b7270de2 Author: shroman Date: 2018-08-23T03:45:15Z IGNITE-9353: Removed "Known issue, possible deadlock in case of low priority cache rebalancing delayed" comment from GridCacheRebalancingSyncSelfTest#getConfiguration. ---
[jira] [Created] (IGNITE-9353) Remove "Known issue, possible deadlock in case of low priority cache rebalancing delayed" comment from GridCacheRebalancingSyncSelfTest#getConfiguration
Roman Shtykh created IGNITE-9353: Summary: Remove "Known issue, possible deadlock in case of low priority cache rebalancing delayed" comment from GridCacheRebalancingSyncSelfTest#getConfiguration Key: IGNITE-9353 URL: https://issues.apache.org/jira/browse/IGNITE-9353 Project: Ignite Issue Type: Task Reporter: Roman Shtykh Assignee: Roman Shtykh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9352) Web Console: Investigate how correctly use package-lock.json
Alexey Kuznetsov created IGNITE-9352: Summary: Web Console: Investigate how correctly use package-lock.json Key: IGNITE-9352 URL: https://issues.apache.org/jira/browse/IGNITE-9352 Project: Ignite Issue Type: Bug Components: wizards Reporter: Alexey Kuznetsov Fix For: 2.7 We need to investigate how correctly use package-lock.json and "^" symbol in package.json. Using "^" allows to use latest version of dependencies, but sometimes it can break build or application just before release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4597: IGNITE-9350 Added checks for empty data in web so...
GitHub user akuznetsov-gridgain opened a pull request: https://github.com/apache/ignite/pull/4597 IGNITE-9350 Added checks for empty data in web sockets events. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9350 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4597.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4597 commit 3ae0edf6e6cd740517c0283ab0fed6c03b72362c Author: Alexey Kuznetsov Date: 2018-08-22T16:24:19Z IGNITE-9350 Web Console: Added checks for unexpected response. commit fa42c77d1403bee80b58b2871552e5ba8b29b410 Author: Alexey Kuznetsov Date: 2018-08-22T17:01:09Z IGNITE-9350 Web Console: Added checks for unexpected data in web socket event data. ---
[MTCGA]: new failures in builds [1711030] needs to be handled
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *Recently contributed test failed in master TransactionIntegrityWithPrimaryIndexCorruptionTest.testPrimaryIndexCorruptionDuringCommitOnPrimaryNode3 https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=3475401922666481512=%3Cdefault%3E=testDetails No changes in build - If your changes can led to this failure(s), please create issue with label MakeTeamCityGreenAgain and assign it to you. -- If you have fix, please set ticket to PA state and write to dev list fix is ready -- For case fix will require some time please mute test and set label Muted_Test to issue - If you know which change caused failure please contact change author directly - If you don't know which change caused failure please send message to dev list to find out Should you have any questions please contact dpav...@apache.org or write to dev.list Best Regards, MTCGA.Bot Notification generated at Thu Aug 23 02:37:34 MSK 2018
[GitHub] ignite pull request #4596: Spark Streaming with Ignite: MD POC
GitHub user kukushal opened a pull request: https://github.com/apache/ignite/pull/4596 Spark Streaming with Ignite: MD POC You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-spark-streaming Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4596.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4596 commit 864f0414f909d74bd6202b30bda4b1cd3bbb42ec Author: Alexey Kukushkin Date: 2018-08-22T21:24:03Z Spark Streaming with Ignite: MD POC ---
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Dmitriy, Alex, thanks for clarification, this is that i already mention in ticket. I think this approach correct. Try to provide PR soon. thanks! > >--- Forwarded message --- >From: "Alex Plehanov" < plehanov.a...@gmail.com > >To: dev@ignite.apache.org >Cc: arzamas...@mail.ru >Subject: Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench. >Date: Tue, 21 Aug 2018 23:17:20 +0300 > >Dmitriy, yes, you correctly understood my idea. > >вт, 21 авг. 2018 г. в 22:03, Vladimir Ozerov < voze...@gridgain.com >: > >> Guys, >> >> Do we have any numbers for real workloads? Or at least Yardstick. >> >> вт, 21 авг. 2018 г. в 21:04, Dmitriy Pavlov < dpavlov@gmail.com >: >> >> > Let us double-check current options. Maybe I'm mistaken about Alex >> > Plehanov's idea. >> > >> > Let's introduce the following variables and functions: >> > data - our block. >> > OldImpl(data) = X >> > NewImpl(data) = Y >> > NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR >> operation. >> > And NewImpl(data) ^ C = X = OldImpl(data) >> > >> > So new and old implementations give us X in all cases, we don't need >> any >> > compatibility mode. >> > >> > Let's try to adopt new fast implementation with C constant. >> > >> > Evgeniy, would you like to try? >> > >> > вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov < skoz...@gridgain.com >: >> > >> > > Alex >> > > >> > > In that case the data becomes in unpredictable state (mix of new and >> old >> > > methods) and there's a chance that some partitions will be never >> touched >> > > and never converted. It will force us always support old compression. >> > > I suppose the explicit conversion that clearly say what is happening >> now >> > > and may be warn that the db files should backed up before version >> > upgrade. >> > > >> > > Also I think AI 3.0 will have set of incompatible changes that give >> us >> a >> > > chance to add an upgrade procedure where compression method update >> among >> > > other stuff will take into account. >> > > >> > > On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov < >> plehanov.a...@gmail.com >> > > >> > > wrote: >> > > >> > > > Sergey, converting data also force us to introduce some flag to WAL >> > > > segments/records or convert WAL segments too. WAL segments can be >> > > archived, >> > > > they also can be compressed. >> > > > IMO we better should not introduce any compatibility modes, left >> data >> > as >> > > is >> > > > and always just convert crc value returned by zip.CRC32 to old >> format >> > > (xor >> > > > it) at runtime. >> > > > >> > > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov < skoz...@gridgain.com >: >> > > > >> > > > > Dmitriy >> > > > > >> > > > > Due to significant improvement and to reduce the number supported >> > > > > modes/options would be good to convert the data at the moment of >> > > upgrade. >> > > > > >> > > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan < >> > > > dsetrak...@apache.org >> > > > > > >> > > > > wrote: >> > > > > >> > > > > > Sergey, that was precisely my comment in the ticket: >> > > > > > >> > > > > > Can we add this option without breaking compatibility with >> previous >> > > > > > page/storage formats? If not, then this should support both >> > > > > implementation. >> > > > > > The default should be the new fastest implementation, but if we >> > > > encounter >> > > > > > the older, slower one, then we should print out a warning in >> the >> > log >> > > > and >> > > > > > automatically switch to the older implementation. >> > > > > > >> > > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov < >> > skoz...@gridgain.com >> > > > >> > > > > > wrote: >> > > > > > >> > > > > > > Hi Igniters >> > > > > > > >> > > > > > > I suppose that'll break compatibility for LFS (PDS). >> > > > > > > >> > > > > > > Do we plan to provide a migration guide w/o data loss for >> upgrade >> > > AI >> > > > > 2.x >> > > > > > to >> > > > > > > 3.0? >> > > > > > > >> > > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < >> > > > > > dsetrak...@apache.org >> > > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > I commented in the ticket: https://issues.apache.org/ >> > > > > > > > jira/browse/IGNITE-9272 >> > > > > > > > >> > > > > > > > It if can integrate it correctly, according to my comment, >> in >> > 2.7 >> > > > > > > release, >> > > > > > > > it would be great. Otherwise, let's plan this change for >> 3.0 >> > > > release. >> > > > > > > > >> > > > > > > > D. >> > > > > > > > >> > > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < >> > > > > > > > eduard.shangar...@gmail.com > wrote: >> > > > > > > > >> > > > > > > > > Hi, >> > > > > > > > > >> > > > > > > > > I have checked the benchmark and it shows great >> performance >> > > boost >> > > > > on >> > > > > > my >> > > > > > > > > laptop! >> > > > > > > > > >> > > > > > > > > +1 for this change. >> > > > > > > > > >> > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < >> > > > >
[GitHub] ignite pull request #4594: Ignite gg 14002
Github user alamar closed the pull request at: https://github.com/apache/ignite/pull/4594 ---
[GitHub] ignite pull request #4595: IGNITE-9351 Log restore partition state start and...
GitHub user dspavlov opened a pull request: https://github.com/apache/ignite/pull/4595 IGNITE-9351 Log restore partition state start and finish events with ⦠â¦elapsed time You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9351 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4595.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4595 commit 563d12f34385a9f8ead0046cafb44f7bc1beb797 Author: Dmitriy Pavlov Date: 2018-08-22T16:57:39Z IGNITE-9351 Log restore partition state start and finish events with elapsed time ---
[jira] [Created] (IGNITE-9351) Log restore partition state start and finish events with elapsed time
Dmitriy Pavlov created IGNITE-9351:
--
Summary: Log restore partition state start and finish events with
elapsed time
Key: IGNITE-9351
URL: https://issues.apache.org/jira/browse/IGNITE-9351
Project: Ignite
Issue Type: Task
Reporter: Dmitriy Pavlov
Assignee: Dmitriy Pavlov
During startup of Ignite in addition to applying logical updates, there can be
a huge operation of restoring partition states, which for slow HDD and
relatively big count of partitions can require significant time.
It is suggested to add start and finish messages with elapsed time.
{noformat}
"exchange-worker-#42@3605" prio=5 tid=0x57 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145)
at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.read(AsyncFileIO.java:95)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:351)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:328)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:312)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:142)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:212)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.(AbstractFreeList.java:370)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.(CacheFreeListImpl.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore$1.(GridCacheOffheapManager.java:1203)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1203)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.updateCounter(GridCacheOffheapManager.java:1420)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.updateCounter(GridDhtLocalPartition.java:942)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.(GridDhtLocalPartition.java:222)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.forceCreatePartition(GridDhtPartitionTopologyImpl.java:843)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restorePartitionStates(GridCacheDatabaseSharedManager.java:2380)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2333)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1443)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1183)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1132)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:712)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2419)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2299)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: has the following been resolved since 2.3? (jdbc thin client security)
Vladimir, This ticket [1] can be closed, right? All the thing clients in Ignite 2.6 go with authentication capabilities. However, not sure what's the status of the other task [2]. [1] https://issues.apache.org/jira/browse/IGNITE-6941 [2] https://issues.apache.org/jira/browse/IGNITE-6856 On Wed, Aug 22, 2018 at 4:35 AM wt wrote: > https://issues.apache.org/jira/browse/IGNITE-6856 > > https://issues.apache.org/jira/browse/IGNITE-6941 > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
[GitHub] ignite pull request #4594: Ignite gg 14002
GitHub user alamar opened a pull request: https://github.com/apache/ignite/pull/4594 Ignite gg 14002 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-14002 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4594.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4594 commit 37c8033c72ac4fd1ec1e419ba68142c4fe11ad8b Author: tledkov-gridgain Date: 2018-03-13T09:37:29Z IGNITE-7860: JDBC thin: changed default socket buffer size to 64Kb. This closes #3600. commit 130adcf29ddb61f8e9baa784b81454d3ed7c3b75 Author: Pavel Tupitsyn Date: 2018-01-26T08:48:14Z IGNITE-7530 .NET: Fix GetAll memory usage and performance in binary mode This closes #3436 commit 824004909b23a9a7d599500967af34103acb8c47 Author: Igor Sapego Date: 2018-01-30T12:56:17Z IGNITE-6810: Implemented SSL support for ODBC. This closes #3361 commit 82da0b5e9dc2ee2339c3fb1023e35d415bf1b647 Author: Pavel Kuznetsov Date: 2018-02-07T12:37:52Z IGNITE-6217: Added benchmark to compare JDBC vs native SQL This closes #2558 commit 701c6f141f6812ad7bc050a86266e696cf5863ed Author: tledkov-gridgain Date: 2018-02-08T12:29:42Z IGNITE-6625: JDBC thin: support SSL connection to Ignite node This closes #3233 commit 2d729bf5c6f6fca9d07be2d57850642ba4b55004 Author: tledkov-gridgain Date: 2018-02-09T11:08:15Z IGNITE-6625: SSL support for thin JDBC: additional fix test; change error message commit 8994f847d7f5f15db73706d9210cdccb1cf3fb26 Author: devozerov Date: 2018-02-12T13:34:24Z IGNITE-7003: Fixed faulty test WalModeChangeAdvancedSelfTest.testJoinCoordinator. commit b142712264007d7397d1594541681a4a7e3d4b93 Author: Igor Sapego Date: 2018-02-26T12:02:07Z IGNITE-7362: Fixed PDO ODBC integration issue commit a2b2aee52cc65d01f2ecaf9462adc4bd368438ea Author: Igor Sapego Date: 2018-02-28T10:23:12Z IGNITE-7763: Fixed 32bit tests configurations to prevent OOM This closes #3557 commit 652f3c4cdbaad40f5de25b06f0c13710aa7f2fd9 Author: Pavel Kuznetsov Date: 2018-03-13T12:46:36Z IGNITE-7531: Data load benchmarks. This closes #3571. commit 9337a53d9fcd62af87f6760080d350b43e275105 Author: tledkov-gridgain Date: 2018-03-16T11:38:38Z IGNITE-7879: Fixed splitter logic for DISTINCT with subqueries. This closes #3634. commit 7bec8b13cb373002d2a6b1b268d410338259bac2 Author: Igor Sapego Date: 2018-03-19T11:17:33Z IGNITE-7811: Implemented connection failover for ODBC This closes #3643 commit e512e5e0a2602df0ecfb69b2b5efabce836b04db Author: Igor Sapego Date: 2018-03-20T10:37:02Z IGNITE-7888: Implemented login timeout for ODBC. This closes #3657 commit 211fca3a55e84b78ff0c1af04d91e25d6fc846c4 Author: devozerov Date: 2018-03-20T11:13:46Z IGNITE-7984: SQL: improved deadlock handling in DML. This closes #3655. commit bcd2888d27afe65f1a060e35b99a05ea420979c7 Author: Roman Guseinov Date: 2018-02-16T09:57:26Z IGNITE-7192: Implemented JDBC support FQDN to multiple IPs This closes #3439 commit d2659d0ec9f6e1a0b905fc7bf23b65fd5522c80a Author: Alexander Paschenko Date: 2018-03-14T09:23:37Z IGNITE-7253: JDBC thin driver: implemented streaming. This closes #3499. This closes #3591. commit bc9018ef8b116f81b8e06d2ff7651ba2b6c7beae Author: tledkov-gridgain Date: 2018-03-19T08:01:26Z IGNITE-7029: JDBC thin driver now supports multiple IP addresses with transparent failover. This closes #3585. commit 587022862fd5bdbb076ab6207ae6fd9bc7583c13 Author: Sergey Chugunov Date: 2018-03-16T16:24:17Z IGNITE-7964 rmvId is stored to MetaStorage metapage during operations - Fixes #3645. commit 006ef4d15e4faedb6dfce6ce9637602055b97293 Author: tledkov-gridgain Date: 2018-03-22T11:47:06Z IGNITE-7436: Simple username/password authentication. This closes #3483. commit 1c7f59c90514670e802d9d07544b00b7562fe6d2 Author: Pavel Tupitsyn Date: 2018-01-30T09:48:16Z .NET: Fix build status icon in README commit 162df61b305fccfc55e186d07351727f35b55179 Author: Pavel Tupitsyn Date: 2018-02-01T11:53:16Z IGNITE-7561 .NET: Add IServices.GetDynamicServiceProxy This closes #3457 commit 9a0328ebbc9d52f8e96898a384fa45743d2efa5b Author: Pavel Tupitsyn Date: 2018-02-02T12:01:27Z .NET: Update README regarding C++ interop and thin client commit b804cfea51c87724b45b40de4fd58d300c049be1 Author: Pavel Tupitsyn Date: 2018-01-31T09:39:19Z .NET: Suppress API parity check for SSL in ClientConnectorConfiguration commit 6f8014de7250c4c0d87cbc8764afae4a225f654b Author: apopov Date: 2018-02-13T10:13:15Z IGNITE-3111 .NET can be now configured SSL without Spring This closes #3498 commit
Re: Table Names in Spark Catalog
Nikolay,
Whatever we decide on would be right :) Basically, we need to answer this
question: does the catalog exist in scope of a single IgniteSparkSession
(and therefore single IgniteContext and single Ignite instance)? In other
words, in case of a rare use case when a single Spark application connects
to multiple Ignite clusters, would there be a catalog created per cluster?
If the answer is yes, current logic doesn't make sense.
-Val
On Wed, Aug 22, 2018 at 1:44 AM Nikolay Izhikov wrote:
> Hello, Valentin.
>
> > I believe we should get rid of this logic and use Ignite schema name as
> database name in Spark's catalog.
>
> When I develop Ignite integration with Spark Data Frame I use following
> abstraction described by Vladimir Ozerov:
>
> "1) Let's consider Ignite cluster as a single database ("catalog" in ANSI
> SQL'92 terms)." [1]
>
> Am I was wrong? If yes - let's fix it.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/SQL-usability-catalogs-schemas-and-tables-td17148.html
>
> В Ср, 22/08/2018 в 09:26 +0100, Stuart Macdonald пишет:
> > Hi Val, yes that's correct. I'd be happy to make the change to have the
> > database reference the schema if Nikolay agrees. (I'll first need to do a
> > bit of research into how to obtain the list of all available schemata...)
> >
> > Thanks,
> > Stuart.
> >
> > On Tue, Aug 21, 2018 at 9:43 PM, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Stuart,
> > >
> > > Thanks for pointing this out, I was not aware that we use Spark
> database
> > > concept this way. Actually, this confuses me a lot. As far as I
> understand,
> > > catalog is created in the scope of a particular IgniteSparkSession,
> which
> > > in turn is assigned to a particular IgniteContext and therefore single
> > > Ignite client. If that's the case, I don't think it should be aware of
> > > other Ignite clients that are connected to other clusters. This doesn't
> > > look like correct behavior to me, not to mention that with this
> approach
> > > having multiple databases would be a very rare case. I believe we
> should
> > > get rid of this logic and use Ignite schema name as database name in
> > > Spark's catalog.
> > >
> > > Nikolay, what do you think?
> > >
> > > -Val
> > >
> > > On Tue, Aug 21, 2018 at 8:17 AM Stuart Macdonald
> > > wrote:
> > >
> > > > Nikolay, Val,
> > > >
> > > > The JDBC Spark datasource[1] -- as far as I can tell -- has no
> > > > ExternalCatalog implementation, it just uses the database specified
> in the
> > > > JDBC URL. So I don't believe there is any way to call listTables() or
> > > > listDatabases() for JDBC provider.
> > > >
> > > > The Hive ExternalCatalog[2] makes the distinction between database
> and
> > > > table using the actual database and table mechanisms built into the
> > > > catalog, which is fine because Hive has the clear distinction and
> > > > hierarchy
> > > > of databases and tables.
> > > >
> > > > *However* Ignite already uses the "database" concept in the Ignite
> > > >
> > > > ExternalCatalog[3] to mean the name of an Ignite instance. So in
> Ignite we
> > > > have instances containing schemas containing tables, and Spark only
> has
> > > > the
> > > > concept of databases and tables so it seems like either we ignore
> one of
> > > > the three Ignite concepts or combine two of them into database or
> table.
> > > > The current implementation in the pull request combines Ignite
> schema and
> > > > table attributes into the Spark table attribute.
> > > >
> > > > Stuart.
> > > >
> > > > [1]
> > > > https://github.com/apache/spark/blob/master/sql/core/
> > > > src/main/scala/org/apache/spark/sql/execution/
> > > > datasources/jdbc/JDBCRelation.scala
> > > > [2]
> > > > https://github.com/apache/spark/blob/master/sql/hive/
> > > > src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
> > > > [3]
> > > > https://github.com/apache/ignite/blob/master/modules/
> > > > spark/src/main/scala/org/apache/spark/sql/ignite/
> > > > IgniteExternalCatalog.scala
> > > >
> > > > On Tue, Aug 21, 2018 at 9:31 AM, Nikolay Izhikov <
> nizhi...@apache.org>
> > > > wrote:
> > > >
> > > > > Hello, Stuart.
> > > > >
> > > > > Can you do some research and find out how schema is handled in Data
> > > >
> > > > Frames
> > > > > for a regular RDBMS such as Oracle, MySQL, etc?
> > > > >
> > > > > В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет:
> > > > > > Stuart, Nikolay,
> > > > > >
> > > > > > I see that the 'Table' class (returned by listTables method) has
> a
> > > > >
> > > > > 'database' field. Can we use this one to report schema name?
> > > > > >
> > > > > > In any case, I think we should look into how this is done in data
> > > >
> > > > source
> > > > > implementations for other databases. Any relational database has a
> > > >
> > > > notion
> > > > > of schema, and I'm sure Spark integrations take this into account
> > > >
> > > > somehow.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On
Re: Adding experimental support for Intel Optane DC Persistent Memory
Hi Dmitry, That's a BSD-3-Clause license if to believe this statement "SPDX-License-Identifier: BSD-3-Clause": https://github.com/pmem/llpl/blob/master/LICENSE This license can be used with ASF software: https://www.apache.org/legal/resolved.html#category-a -- Denis On Wed, Aug 22, 2018 at 9:28 AM Dmitriy Pavlov wrote: > Hi Denis, > > Could you please double check if we can refer to any library licensed to > Intel. Can we develop code only version of this support (without shipping > it in release)? > > https://github.com/apache/ignite/pull/4381 is quite huge change, > including 128 files changed, patch review will require resources from > community members to review. I would like to be sure we can include this > patch from the legal point of view. > > Sincerely, > Dmitriy Pavlov > > пт, 3 авг. 2018 г. в 19:23, Dmitriy Pavlov : > >> Hi Mulugeta, >> >> I appologise, I've missed that license is already there. So I guess it is >> not standard open-source license, it is seems it is not listed in >> https://www.apache.org/legal/resolved.html#category-a >> >> So there can be legal concern related to including this lib as dependency >> into Apache product. It should not block review, we can later >> consult Secretary/Legal to find out how we can correctly include reference >> to lib. >> >> Sincerely, >> Dmitriy Pavlov >> >> чт, 2 авг. 2018 г. в 0:24, Mammo, Mulugeta : >> >>> Hi Dmitriy, >>> >>> Do you mean our LLPL library? It has a license, please look here: >>> https://github.com/pmem/llpl >>> >>> Regarding the changes made to Ignite, you may refer to the pull request >>> here: https://github.com/apache/ignite/pull/4381 >>> >>> Thanks, >>> Mulugeta >>> >>> -Original Message- >>> From: Dmitriy Pavlov [mailto:dpavlov@gmail.com] >>> Sent: Wednesday, August 1, 2018 10:49 AM >>> To: dev@ignite.apache.org >>> Subject: Re: Adding experimental support for Intel Optane DC Persistent >>> Memory >>> >>> Hi Mulugeta Mammo, >>> >>> I've just noticed that repository, what you refer is full fork of Ignite. >>> How can I see differences with original Ignite? >>> >>> One more thing, library which you're referencing seems to not contain >>> license, at least github can not parse it. Apache product has limitations >>> which libraries may be used (see >>> https://www.apache.org/legal/resolved.html#category-a and >>> https://www.apache.org/legal/resolved.html#category-b) >>> >>> Could you please comment if there is some legal risk? >>> >>> Sincerely, >>> Dmitriy Pavlov >>> >>> ср, 1 авг. 2018 г. в 20:43, Dmitriy Pavlov : >>> >>> > Hi, >>> > >>> > This link works for me >>> > >>> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-26%3A+Adding+Ex >>> > perimental+Support+for+Intel+Optane+DC+Persistent+Memory >>> > >>> > Sincerely, >>> > Dmitriy Pavlov >>> > >>> > чт, 26 июл. 2018 г. в 15:31, Stanislav Lukyanov < >>> stanlukya...@gmail.com>: >>> > >>> >> Ah, ok, it’s just the ‘.’ at the end of the link. Removed it and it’s >>> >> fine. >>> >> >>> >> From: Stanislav Lukyanov >>> >> Sent: 26 июля 2018 г. 15:12 >>> >> To: dev@ignite.apache.org >>> >> Subject: RE: Adding experimental support for Intel Optane DC >>> >> Persistent Memory >>> >> >>> >> Hi, >>> >> >>> >> The link you’ve shared gives me 404. >>> >> Perhaps you need to add a permission for everyone to access the page? >>> >> >>> >> Thanks, >>> >> Stan >>> >> >>> >> From: Mammo, Mulugeta >>> >> Sent: 26 июля 2018 г. 2:44 >>> >> To: dev@ignite.apache.org >>> >> Subject: Adding experimental support for Intel Optane DC Persistent >>> >> Memory >>> >> >>> >> Hi, >>> >> >>> >> I have added a new proposal to support Intel Optane DC Persistent >>> >> Memory for Ignite here: >>> >> https://cwiki.apache.org/confluence/display/IGNITE/Adding+Experimenta >>> >> l+Support+for+Intel+Optane+DC+Persistent+Memory >>> >> . >>> >> >>> >> I'm looking forward to your feedback and collaboration on this. >>> >> >>> >> Thanks, >>> >> Mulugeta >>> >> >>> >> >>> >> >>> >> >>> >>
[jira] [Created] (IGNITE-9350) Web Console backend should not fail if null received via web sockets
Alexey Kuznetsov created IGNITE-9350:
Summary: Web Console backend should not fail if null received via
web sockets
Key: IGNITE-9350
URL: https://issues.apache.org/jira/browse/IGNITE-9350
Project: Ignite
Issue Type: Bug
Components: wizards
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
Fix For: 2.7
Backend failed with error:
{code}
app/browsersHandler.js:207
sock.on('node:rest', (\{clusterId, params, credentials}, cb) => {
^
TypeError: Cannot destructure property `clusterId` of 'undefined' or 'null'.
{code}
We should check for null data received via web sockets
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ignite pull request #4529: IGNITE-9054 Avoid using OptimizedMarshaller with ...
Github user alamar closed the pull request at: https://github.com/apache/ignite/pull/4529 ---
[GitHub] ignite pull request #4593: IGNITE-9305
GitHub user xtern opened a pull request: https://github.com/apache/ignite/pull/4593 IGNITE-9305 You can merge this pull request into a Git repository by running: $ git pull https://github.com/xtern/ignite IGNITE-9305 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4593.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4593 commit bd7a871d1cc20101bfba68fb05d07cd4f4c31206 Author: pereslegin-pa Date: 2018-08-22T12:19:34Z IGNITE-9305 Non heap mem metrics logging. commit bf8eb41ed66363adffaec56e86c828f6bb14 Author: pereslegin-pa Date: 2018-08-22T13:42:14Z IGNITE-9305 Log mem metrics separately for each region. commit 5014b414969689152dc6fe4d6ea268c17fb8d543 Author: pereslegin-pa Date: 2018-08-22T14:52:01Z IGNITE-9305 Persistence region metrics log format check. ---
[GitHub] ignite pull request #4592: IGNITE-9054 Avoid using OptimizedMarshaller with ...
GitHub user alamar opened a pull request: https://github.com/apache/ignite/pull/4592 IGNITE-9054 Avoid using OptimizedMarshaller with initial ScanQuery. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-14092 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4592.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4592 commit a9e41f07d78f9b64d827303763155bc77da23224 Author: Ilya Kasnacheev Date: 2018-08-22T14:47:52Z IGNITE-9054 Avoid using OptimizedMarshaller with initial ScanQuery. ---
[GitHub] ignite pull request #4578: IGNITE-9188 Unexpected eviction leading to data l...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4578 ---
[GitHub] ignite pull request #4432: IGNITE-8920 Fix transaction hanging on exception ...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4432 ---
[jira] [Created] (IGNITE-9348) ML examples improvements, follow-up to IGNITE-9297
Oleg Ignatenko created IGNITE-9348: -- Summary: ML examples improvements, follow-up to IGNITE-9297 Key: IGNITE-9348 URL: https://issues.apache.org/jira/browse/IGNITE-9348 Project: Ignite Issue Type: Task Components: ml Affects Versions: 2.6 Reporter: Oleg Ignatenko After IGNITE-9297 was resolved, ML examples were discussed with some folks ([~skozlov], [~avolkov], [~chief]) in order to check if some other improvements may be desired. This ticket is to address obtained feedback. List of points that look worth taking care of includes (but is not limited to): * some examples javadocs still looks like missing details (eg random forest) * some mentions of algorithms (eg kNN, gradient boosting etc) are better to be linked to respective introductory articles in order to help less experienced users easier grasp the examples * presence of some examples doesn't look justified (eg LocalDatasetExample) * logging in some examples looks sub-optimal, either too brief or too lengthy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9347) Attempt to start a dynamic cache with invalid configuration leads to exchange worker death
Vladimir Ozerov created IGNITE-9347:
---
Summary: Attempt to start a dynamic cache with invalid
configuration leads to exchange worker death
Key: IGNITE-9347
URL: https://issues.apache.org/jira/browse/IGNITE-9347
Project: Ignite
Issue Type: Bug
Components: cache
Affects Versions: 2.6, 2.5
Reporter: Vladimir Ozerov
Fix For: 2.7
Reproducer - {{SqlIllegalSchemaSelfTest}}. Technically, this test pass. But
note the following:
1) See logs of {{*Dynamic}} tests - instead of normal stop, node gets killed by
failure detector:
{code:java}
[2018-08-22
14:20:39,777][ERROR][exchange-worker-#77%query.SqlIllegalSchemaSelfTest%][IgniteTestResources]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=class o.a.i.failure.NoOpFailureHandler, failureCtx=FailureContext
[type=SYSTEM_WORKER_TERMINATION, err=java.lang.AssertionError: stopping=true,
groupName=null, caches=[]]]
java.lang.AssertionError: stopping=true, groupName=null, caches=[]
at
org.apache.ignite.internal.processors.cache.CacheGroupContext.singleCacheContext(CacheGroupContext.java:371){code}
2) Similar behavior is observed if one tries to start caches with invalid
configuration twice. Pick any {{*Dynamic}} test and just copy/paste
{{GridTestUtils.assertThrows}} logic one after another. Expected - two expected
exceptions, actual - node is killed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Adding experimental support for Intel Optane DC Persistent Memory
Hi Denis, Could you please double check if we can refer to any library licensed to Intel. Can we develop code only version of this support (without shipping it in release)? https://github.com/apache/ignite/pull/4381 is quite huge change, including 128 files changed, patch review will require resources from community members to review. I would like to be sure we can include this patch from the legal point of view. Sincerely, Dmitriy Pavlov пт, 3 авг. 2018 г. в 19:23, Dmitriy Pavlov : > Hi Mulugeta, > > I appologise, I've missed that license is already there. So I guess it is > not standard open-source license, it is seems it is not listed in > https://www.apache.org/legal/resolved.html#category-a > > So there can be legal concern related to including this lib as dependency > into Apache product. It should not block review, we can later > consult Secretary/Legal to find out how we can correctly include reference > to lib. > > Sincerely, > Dmitriy Pavlov > > чт, 2 авг. 2018 г. в 0:24, Mammo, Mulugeta : > >> Hi Dmitriy, >> >> Do you mean our LLPL library? It has a license, please look here: >> https://github.com/pmem/llpl >> >> Regarding the changes made to Ignite, you may refer to the pull request >> here: https://github.com/apache/ignite/pull/4381 >> >> Thanks, >> Mulugeta >> >> -Original Message- >> From: Dmitriy Pavlov [mailto:dpavlov@gmail.com] >> Sent: Wednesday, August 1, 2018 10:49 AM >> To: dev@ignite.apache.org >> Subject: Re: Adding experimental support for Intel Optane DC Persistent >> Memory >> >> Hi Mulugeta Mammo, >> >> I've just noticed that repository, what you refer is full fork of Ignite. >> How can I see differences with original Ignite? >> >> One more thing, library which you're referencing seems to not contain >> license, at least github can not parse it. Apache product has limitations >> which libraries may be used (see >> https://www.apache.org/legal/resolved.html#category-a and >> https://www.apache.org/legal/resolved.html#category-b) >> >> Could you please comment if there is some legal risk? >> >> Sincerely, >> Dmitriy Pavlov >> >> ср, 1 авг. 2018 г. в 20:43, Dmitriy Pavlov : >> >> > Hi, >> > >> > This link works for me >> > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-26%3A+Adding+Ex >> > perimental+Support+for+Intel+Optane+DC+Persistent+Memory >> > >> > Sincerely, >> > Dmitriy Pavlov >> > >> > чт, 26 июл. 2018 г. в 15:31, Stanislav Lukyanov > >: >> > >> >> Ah, ok, it’s just the ‘.’ at the end of the link. Removed it and it’s >> >> fine. >> >> >> >> From: Stanislav Lukyanov >> >> Sent: 26 июля 2018 г. 15:12 >> >> To: dev@ignite.apache.org >> >> Subject: RE: Adding experimental support for Intel Optane DC >> >> Persistent Memory >> >> >> >> Hi, >> >> >> >> The link you’ve shared gives me 404. >> >> Perhaps you need to add a permission for everyone to access the page? >> >> >> >> Thanks, >> >> Stan >> >> >> >> From: Mammo, Mulugeta >> >> Sent: 26 июля 2018 г. 2:44 >> >> To: dev@ignite.apache.org >> >> Subject: Adding experimental support for Intel Optane DC Persistent >> >> Memory >> >> >> >> Hi, >> >> >> >> I have added a new proposal to support Intel Optane DC Persistent >> >> Memory for Ignite here: >> >> https://cwiki.apache.org/confluence/display/IGNITE/Adding+Experimenta >> >> l+Support+for+Intel+Optane+DC+Persistent+Memory >> >> . >> >> >> >> I'm looking forward to your feedback and collaboration on this. >> >> >> >> Thanks, >> >> Mulugeta >> >> >> >> >> >> >> >> >> >
Re: Release policy updates
Issue [1] created. [1] https://issues.apache.org/jira/browse/IGNITE-9346 пн, 20 авг. 2018 г. в 17:27, Denis Magda : > Yes, let’s just remove md5. Will you create the ticket and handle this for > 2.7? > > Denis > > On Monday, August 20, 2018, Anton Vinogradov wrote: > > > Denis, > > > > Currently we provide md5 and sha512 [1]. > > Should we just get rid of md5? > > > > [1] https://www.apache.org/dist/ignite/2.6.0/ > > > > сб, 18 авг. 2018 г. в 3:51, Denis Magda : > > > >> Peter, Anton V, Igniters, > >> > >> The board communicated the following release policy changes: > >> -- for new releases : > >> -- you MUST supply a SHA-256 and/or SHA-512 file > >> -- you SHOULD NOT supply MD5 or SHA-1 files > >> > >> Are we good? More details are below. > >> > >> > >> > >> > >> *2 Release Dist Policy Changes (Q? us...@infra.apache.org) > >> --- > >> > >> The Release Distribution Policy[1] changed regarding checksum files. > >> See under "Cryptographic Signatures and Checksums Requirements" [2]. > >> > >> Note that "MUST", "SHOULD", "SHOULD NOT" are technical terms ; > >> not just emphasized words ; for an explanation see RFC-2119 [3]. > >> > >> Old policy : > >> > >> -- SHOULD supply a SHA checksum file > >> -- SHOULD NOT supply a MD5 checksum file > >> > >> New policy : > >> > >> -- SHOULD supply a SHA-256 and/or SHA-512 checksum file > >> -- SHOULD NOT supply MD5 or SHA-1 checksum files > >> > >> Why this change ? > >> > >> -- Like MD5, SHA-1 is too broken ; we should move away from it. > >> > >> Impact for PMCs : > >> > >> -- for new releases : > >> -- you MUST supply a SHA-256 and/or SHA-512 file > >> -- you SHOULD NOT supply MD5 or SHA-1 files > >> > >> -- for past releases : > >> -- you are not required to change anything ; > >> -- it would be nice if you fixed your dist area ; > >> start with : cleanup ; rename .sha's ; remove .md5's > >> > > >
[jira] [Created] (IGNITE-9346) md5 should be removed from release
Anton Vinogradov created IGNITE-9346: Summary: md5 should be removed from release Key: IGNITE-9346 URL: https://issues.apache.org/jira/browse/IGNITE-9346 Project: Ignite Issue Type: Task Reporter: Anton Vinogradov Assignee: Anton Vinogradov The board communicated the following release policy changes: -- for new releases : -- you MUST supply a SHA-256 and/or SHA-512 file -- you SHOULD NOT supply MD5 or SHA-1 files So, we should get rid of md5 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Exchange stucks while node restoring state from WAL
Hello Maxim, Thank you for your work. I will review your changes within the next several days. 2018-08-22 11:12 GMT+03:00 Maxim Muzafarov : > Pavel, > > Thank you for so detailed introduction into the root of the problem of > ticket IGNITE-7196. > As you mentioned before, we can divide this ticket into two sub-tasks. So, > I will > file new issue for `Logical consistency recovery`. I think it's more > convenient way > for reviewing and merging each of these high level Steps (PRs). > > Currently, let's focus on `Physical(binary) consistency recovery` as the > most critical > part of this issue and synchronize our vision and vision of other community > members > on implementation details of Step 1. From this point everything what I'm > saying is about > only binary recovery. > > > I think PR is almost ready (I need to check TC carefully). > Can you look at my changes? Am I going the right way? > > PR: https://github.com/apache/ignite/pull/4520 > Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-727 > JIRA: https://issues.apache.org/jira/browse/IGNITE-7196 > > > These are my milestones which I've tried to solve: > 1. On first time cluster activation no changes required. We should keep > this behavior. > > 2. `startMemoryPolicies` earlier if need. > Now at the master branch data regions starts at onActivate method called. > If we are trying > to restore binary state on node reconnects to cluster we should start > regions earlier. > > 3. `suspendLogging` method added to switch off handlers. > Keep fast PDS cleanup when joining is not in BLT (or belongs to another > BLT). > Restore memory and metastorage initialization required `WALWriter` and > `FileWriteHandle` > to be started. They can lock directory\files and block cleanups. > > 4. `isBaselineNode` check before resumming logging. > The same as point 3. Local node gets discovery data and determines > #isLocalNodeNotInBaseline. > If node not in current baseline it performs cleanup local PDS and we should > reset state of > previously initialized at startup metastorage instance. > > 5. `cacheProcessorStarted` callback added to restore memory at startup. > We can restore memory only after having info about all > `CacheGroupDescriptor`. To achieve this > we should do restoring at the moment `GridCacheProcessor` started. > > 6. Move `fsync` for `MemoryRecoveryRecord` at node startup. > We are fsync'ing it inside running PME. Suppose, it used for recovery > actions and\or control > cluster state outside Ignite project (methods `nodeStart` and > `nodeStartedPointers` of this > are not used at Ignite project code). I've moved them at the node startup, > but may be we should > dumping timestamp only at the time of node activation. Not sure. > > 7. `testBinaryRecoverBeforeExchange` test added to check restore before > PME. > Not sure that we should repeat wal reading logic as it was in > `testApplyDeltaRecords` test for > checking memory restoring. Suppose, minimally necessary condition is to > check `lastRestored` and > `checkpointHistory` states of database manager's (they should happend > before node joins to cluster). > > I will create separate ticket and refactor these minor issues after we will > finish with 7196: > - simlify registerMBeans methods for `Ignite\ > GridCacheDatabaseSharedManager` > - add @Override annotations if missed > - fix abbreviations regarding IdeaAbbrPlugin > - createDataRegionConfiguration method can be removed and simplified > - startMemoryPolicies rename to startDataRegion > - output messages format fix code style needed > - fix format javadoc's > - make inner classes private > - missed logging for cleanupCheckpointDirectory, > cleanupCheckpointDirectory, cleanupWalDirectories > > On Fri, 3 Aug 2018 at 18:20 Pavel Kovalenko wrote: > > > Maxim, > > > > In the addition to my answer, I propose to divide the whole task into 2 > > steps: > > 1) Physical consistency recovery before PME. > > 2) Logical consistency recovery before PME. > > > > The first step can be done before the discovery manager start, as it no > > needs any information about caches, it operates only with pages binary > > data. On this step, we can do points a) and b) from answer 3) and fsync > the > > state of the page memory. > > The second step is much harder to implement as it requires major changes > in > > Discovery SPI. Discovery node join event is triggered on coordinator > after > > node successfully joins to the cluster. To avoid it, we should introduce > a > > new state of node and fire discovery event only after logical recovery > has > > finished. I doubt that logical recovery consumes a lot of time of > exchange, > > but physical recovery does, as it performs fsync on disk after restore. > > > > > > > > > > 2018-08-03 17:46 GMT+03:00 Pavel Kovalenko : > > > > > Hello Maxim, > > > > > > 1) Yes, Discovery Manager is starting after GridCacheProcessor, which > > > starts GridCacheDatabaseSharedManager which invokes readMetastorage on > > >
[GitHub] ignite pull request #4527: IGNITE-9248: C++ compilation with Clang is suppor...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4527 ---
Re: Ignite CPP client local development
Hello, Igor. Thanks, for an answer. I think we should start with a simple step by step how-to: "How to build current sources?". Actually, I don't know how I can help you with this task? Seems, only you have the knowledge of this process. At this step, no source changes required. I propose to you to do following: 1. Download Microsoft virtual machine from previous mail and wrote your steps for project compilation. OR 2. Download Ubuntu linux and to same steps for a Linux. It will be enough to start with. В Ср, 22/08/2018 в 15:03 +0300, Igor Sapego пишет: > Nikolay, > > You are right, it should be much easier. > Unfortunately, there are very few C++ developers in community. > > I'm planning on fix all these, but I'm always lack time. I'd appreciate > a lot and provide any help if anyone could help me with those tasks. > > Best Regards, > Igor > > > On Wed, Aug 22, 2018 at 12:17 PM Vyacheslav Daradur > wrote: > > > Hi, > > > > I think we should provide an article "Ignite.C++ development" in the > > project's wiki. > > > > Such article as "Ignite.NET Development" [1] helped me to prepare the > > environment step by step. > > > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/Ignite.NET+Development > > > > On Wed, Aug 22, 2018 at 12:06 PM Nikolay Izhikov > > wrote: > > > > > > Hello, Igniters. > > > > > > While working on IGNITE-6055 [1] I need to do some fixes in ODBC. > > > I find out that setup of local CPP development environment is a very > > > > hard task: > > > > > > 1. Linux - > > > - We use a very old version of boost library. > > > Modern Linux distribution uses the new one. > > > There are no clear instructions about how download and setup > > > > boost for an Ignite CPP client. > > > There are no clear instructions about boost version. > > > > > > - Many of the required steps(environment variables, etc.) are > > > > not described in DEVNOTES at all. > > > I have to crawl through the Team City configuration to find it > > > > out. > > > I need following environment variables to run tests: > > > BOOST_TEST_CATCH_SYSTEM_ERRORS=no > > > IGNITE_NATIVE_TEST_CLASSPATH=true > > > > > > > > > IGNITE_NATIVE_TEST_CPP_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/core-test/config > > > > > > > > > IGNITE_NATIVE_TEST_ODBC_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/odbc-test/config > > > > > > > LD_LIBRARY_PATH=$LD_LIBARY_PATH:$JAVA_HOME/jre/lib/amd64/server > > > > > > 2. Windows - > > > Microsoft provides a free virtual machine with Windows 10 and > > > > Visual Studio 2017 [2], but I can't build a project on it. > > > Our documentation [1] claim: "Visual Studio 2010 and above". > > > > > > Actually, I'm not a C++ developer, so I can miss some steps that are > > > > obvious to a C++ developer. > > > But, I think, we should make instructions for community developers as > > > > clear as possible. > > > > > > Igor, as a maintainer of CPP clients, can you, please, update > > > > development documentation > > > and provide clear steps to setup the project on Linux and Windows? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6055 > > > [2] > > > > https://developer.microsoft.com/ru-ru/windows/downloads/virtual-machines > > > > > > > > -- > > Best Regards, Vyacheslav D. > > signature.asc Description: This is a digitally signed message part
Re: Ignite CPP client local development
Nikolay, You are right, it should be much easier. Unfortunately, there are very few C++ developers in community. I'm planning on fix all these, but I'm always lack time. I'd appreciate a lot and provide any help if anyone could help me with those tasks. Best Regards, Igor On Wed, Aug 22, 2018 at 12:17 PM Vyacheslav Daradur wrote: > Hi, > > I think we should provide an article "Ignite.C++ development" in the > project's wiki. > > Such article as "Ignite.NET Development" [1] helped me to prepare the > environment step by step. > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/Ignite.NET+Development > > On Wed, Aug 22, 2018 at 12:06 PM Nikolay Izhikov > wrote: > > > > Hello, Igniters. > > > > While working on IGNITE-6055 [1] I need to do some fixes in ODBC. > > I find out that setup of local CPP development environment is a very > hard task: > > > > 1. Linux - > > - We use a very old version of boost library. > > Modern Linux distribution uses the new one. > > There are no clear instructions about how download and setup > boost for an Ignite CPP client. > > There are no clear instructions about boost version. > > > > - Many of the required steps(environment variables, etc.) are > not described in DEVNOTES at all. > > I have to crawl through the Team City configuration to find it > out. > > I need following environment variables to run tests: > > BOOST_TEST_CATCH_SYSTEM_ERRORS=no > > IGNITE_NATIVE_TEST_CLASSPATH=true > > > > IGNITE_NATIVE_TEST_CPP_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/core-test/config > > > > IGNITE_NATIVE_TEST_ODBC_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/odbc-test/config > > > LD_LIBRARY_PATH=$LD_LIBARY_PATH:$JAVA_HOME/jre/lib/amd64/server > > > > 2. Windows - > > Microsoft provides a free virtual machine with Windows 10 and > Visual Studio 2017 [2], but I can't build a project on it. > > Our documentation [1] claim: "Visual Studio 2010 and above". > > > > Actually, I'm not a C++ developer, so I can miss some steps that are > obvious to a C++ developer. > > But, I think, we should make instructions for community developers as > clear as possible. > > > > Igor, as a maintainer of CPP clients, can you, please, update > development documentation > > and provide clear steps to setup the project on Linux and Windows? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6055 > > [2] > https://developer.microsoft.com/ru-ru/windows/downloads/virtual-machines > > > > -- > Best Regards, Vyacheslav D. >
[GitHub] ignite pull request #4579: IGNITE-9327 stop the node if IgniteNeedReconnectE...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4579 ---
[GitHub] ignite pull request #4550: IGNITE-9279: support custom default SQL schema.
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4550 ---
[jira] [Created] (IGNITE-9345) Document custom SQL schemas
Vladimir Ozerov created IGNITE-9345:
---
Summary: Document custom SQL schemas
Key: IGNITE-9345
URL: https://issues.apache.org/jira/browse/IGNITE-9345
Project: Ignite
Issue Type: Task
Components: documentation, sql
Reporter: Vladimir Ozerov
Fix For: 2.7
Key highlights:
1) DDL operations \{{CREATE TABLE}} and \{{DROP TABLE}} are no longer tied to
\{{PUBLIC}} schema. They can be executed on any schema. Need to remove this
from docs (if there is such paragraph)
2) Added a property \{{IgniteConfiguration.sqlSchemas}} which creates local
schemas with provided names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ignite pull request #4586: IGNITE-9279
Github user devozerov closed the pull request at: https://github.com/apache/ignite/pull/4586 ---
Re: [MTCGA]: new failures in builds [1703433] needs to be handled
Hi Igniters, It seems it is a flaky test. We need to improve flakiness detection in the Bot. Please join separate discussion started by Dmitrii Ryabov. Sincerely, Dmitriy Pavlov ср, 22 авг. 2018 г. в 3:22, : > Hi Ignite Developer, > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. > I hope you can help. > > *New test failure in master > IgniteCacheAtomicProtocolTest.testPutReaderUpdate1 > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5377747406804372686=%3Cdefault%3E=testDetails > No changes in build > > - If your changes can led to this failure(s), please create issue > with label MakeTeamCityGreenAgain and assign it to you. > -- If you have fix, please set ticket to PA state and write to dev > list fix is ready > -- For case fix will require some time please mute test and set > label Muted_Test to issue > - If you know which change caused failure please contact change > author directly > - If you don't know which change caused failure please send > message to dev list to find out > Should you have any questions please contact dpav...@apache.org or write > to dev.list > Best Regards, > MTCGA.Bot > Notification generated at Wed Aug 22 03:22:33 MSK 2018 >
[GitHub] ignite pull request #4580: Need to change merge metadata behavior during new...
Github user zstan closed the pull request at: https://github.com/apache/ignite/pull/4580 ---
Re: Ignite CPP client local development
Hi, I think we should provide an article "Ignite.C++ development" in the project's wiki. Such article as "Ignite.NET Development" [1] helped me to prepare the environment step by step. [1] https://cwiki.apache.org/confluence/display/IGNITE/Ignite.NET+Development On Wed, Aug 22, 2018 at 12:06 PM Nikolay Izhikov wrote: > > Hello, Igniters. > > While working on IGNITE-6055 [1] I need to do some fixes in ODBC. > I find out that setup of local CPP development environment is a very hard > task: > > 1. Linux - > - We use a very old version of boost library. > Modern Linux distribution uses the new one. > There are no clear instructions about how download and setup boost > for an Ignite CPP client. > There are no clear instructions about boost version. > > - Many of the required steps(environment variables, etc.) are not > described in DEVNOTES at all. > I have to crawl through the Team City configuration to find it out. > I need following environment variables to run tests: > BOOST_TEST_CATCH_SYSTEM_ERRORS=no > IGNITE_NATIVE_TEST_CLASSPATH=true > > IGNITE_NATIVE_TEST_CPP_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/core-test/config > > IGNITE_NATIVE_TEST_ODBC_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/odbc-test/config > > LD_LIBRARY_PATH=$LD_LIBARY_PATH:$JAVA_HOME/jre/lib/amd64/server > > 2. Windows - > Microsoft provides a free virtual machine with Windows 10 and Visual > Studio 2017 [2], but I can't build a project on it. > Our documentation [1] claim: "Visual Studio 2010 and above". > > Actually, I'm not a C++ developer, so I can miss some steps that are obvious > to a C++ developer. > But, I think, we should make instructions for community developers as clear > as possible. > > Igor, as a maintainer of CPP clients, can you, please, update development > documentation > and provide clear steps to setup the project on Linux and Windows? > > [1] https://issues.apache.org/jira/browse/IGNITE-6055 > [2] https://developer.microsoft.com/ru-ru/windows/downloads/virtual-machines -- Best Regards, Vyacheslav D.
Ignite CPP client local development
Hello, Igniters. While working on IGNITE-6055 [1] I need to do some fixes in ODBC. I find out that setup of local CPP development environment is a very hard task: 1. Linux - - We use a very old version of boost library. Modern Linux distribution uses the new one. There are no clear instructions about how download and setup boost for an Ignite CPP client. There are no clear instructions about boost version. - Many of the required steps(environment variables, etc.) are not described in DEVNOTES at all. I have to crawl through the Team City configuration to find it out. I need following environment variables to run tests: BOOST_TEST_CATCH_SYSTEM_ERRORS=no IGNITE_NATIVE_TEST_CLASSPATH=true IGNITE_NATIVE_TEST_CPP_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/core-test/config IGNITE_NATIVE_TEST_ODBC_CONFIG_PATH=/home/user/src/ignite/modules/platforms/cpp/odbc-test/config LD_LIBRARY_PATH=$LD_LIBARY_PATH:$JAVA_HOME/jre/lib/amd64/server 2. Windows - Microsoft provides a free virtual machine with Windows 10 and Visual Studio 2017 [2], but I can't build a project on it. Our documentation [1] claim: "Visual Studio 2010 and above". Actually, I'm not a C++ developer, so I can miss some steps that are obvious to a C++ developer. But, I think, we should make instructions for community developers as clear as possible. Igor, as a maintainer of CPP clients, can you, please, update development documentation and provide clear steps to setup the project on Linux and Windows? [1] https://issues.apache.org/jira/browse/IGNITE-6055 [2] https://developer.microsoft.com/ru-ru/windows/downloads/virtual-machines signature.asc Description: This is a digitally signed message part
Re: [MTCGA]: new failures in builds [1701592] needs to be handled
Igniters, I've checked this failure. Can't reproduce it locally. It seems mentioned commits are not related to this test. The test has been a long flaky and we just got an unlucky day to catch fail 3 times in a row. I think, we should fix it as an ordinary MTCGA way. Issue created - [1] [1] https://issues.apache.org/jira/browse/IGNITE-9344 On Wed, 22 Aug 2018 at 01:22 wrote: > Hi Ignite Developer, > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. > I hope you can help. > > *New test failure in master > IgniteClusterActivateDeactivateTestWithPersistence.testClientReconnectClusterActivated > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-390174676041430081=%3Cdefault%3E=testDetails > Changes may led to failure were done by > - dmitrievanthony > http://ci.ignite.apache.org/viewModification.html?modId=829030=false > - zaleslaw.sin > http://ci.ignite.apache.org/viewModification.html?modId=829029=false > - michael.cherkasov > http://ci.ignite.apache.org/viewModification.html?modId=829026=false > - alexey.goncharuk > http://ci.ignite.apache.org/viewModification.html?modId=829012=false > > - If your changes can led to this failure(s), please create issue > with label MakeTeamCityGreenAgain and assign it to you. > -- If you have fix, please set ticket to PA state and write to dev > list fix is ready > -- For case fix will require some time please mute test and set > label Muted_Test to issue > - If you know which change caused failure please contact change > author directly > - If you don't know which change caused failure please send > message to dev list to find out > Should you have any questions please contact dpav...@apache.org or write > to dev.list > Best Regards, > MTCGA.Bot > Notification generated at Wed Aug 22 01:22:37 MSK 2018 > -- -- Maxim Muzafarov
[jira] [Created] (IGNITE-9344) Flaky test `testClientReconnectClusterActivated` detected
Maxim Muzafarov created IGNITE-9344:
---
Summary: Flaky test `testClientReconnectClusterActivated` detected
Key: IGNITE-9344
URL: https://issues.apache.org/jira/browse/IGNITE-9344
Project: Ignite
Issue Type: Bug
Reporter: Maxim Muzafarov
{{IgniteClusterActivateDeactivateTestWithPersistence.testClientReconnectClusterActivated}}
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-390174676041430081=%3Cdefault%3E=testDetails
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Table Names in Spark Catalog
Hello, Valentin.
> I believe we should get rid of this logic and use Ignite schema name as
> database name in Spark's catalog.
When I develop Ignite integration with Spark Data Frame I use following
abstraction described by Vladimir Ozerov:
"1) Let's consider Ignite cluster as a single database ("catalog" in ANSI
SQL'92 terms)." [1]
Am I was wrong? If yes - let's fix it.
[1]
http://apache-ignite-developers.2346864.n4.nabble.com/SQL-usability-catalogs-schemas-and-tables-td17148.html
В Ср, 22/08/2018 в 09:26 +0100, Stuart Macdonald пишет:
> Hi Val, yes that's correct. I'd be happy to make the change to have the
> database reference the schema if Nikolay agrees. (I'll first need to do a
> bit of research into how to obtain the list of all available schemata...)
>
> Thanks,
> Stuart.
>
> On Tue, Aug 21, 2018 at 9:43 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Stuart,
> >
> > Thanks for pointing this out, I was not aware that we use Spark database
> > concept this way. Actually, this confuses me a lot. As far as I understand,
> > catalog is created in the scope of a particular IgniteSparkSession, which
> > in turn is assigned to a particular IgniteContext and therefore single
> > Ignite client. If that's the case, I don't think it should be aware of
> > other Ignite clients that are connected to other clusters. This doesn't
> > look like correct behavior to me, not to mention that with this approach
> > having multiple databases would be a very rare case. I believe we should
> > get rid of this logic and use Ignite schema name as database name in
> > Spark's catalog.
> >
> > Nikolay, what do you think?
> >
> > -Val
> >
> > On Tue, Aug 21, 2018 at 8:17 AM Stuart Macdonald
> > wrote:
> >
> > > Nikolay, Val,
> > >
> > > The JDBC Spark datasource[1] -- as far as I can tell -- has no
> > > ExternalCatalog implementation, it just uses the database specified in the
> > > JDBC URL. So I don't believe there is any way to call listTables() or
> > > listDatabases() for JDBC provider.
> > >
> > > The Hive ExternalCatalog[2] makes the distinction between database and
> > > table using the actual database and table mechanisms built into the
> > > catalog, which is fine because Hive has the clear distinction and
> > > hierarchy
> > > of databases and tables.
> > >
> > > *However* Ignite already uses the "database" concept in the Ignite
> > >
> > > ExternalCatalog[3] to mean the name of an Ignite instance. So in Ignite we
> > > have instances containing schemas containing tables, and Spark only has
> > > the
> > > concept of databases and tables so it seems like either we ignore one of
> > > the three Ignite concepts or combine two of them into database or table.
> > > The current implementation in the pull request combines Ignite schema and
> > > table attributes into the Spark table attribute.
> > >
> > > Stuart.
> > >
> > > [1]
> > > https://github.com/apache/spark/blob/master/sql/core/
> > > src/main/scala/org/apache/spark/sql/execution/
> > > datasources/jdbc/JDBCRelation.scala
> > > [2]
> > > https://github.com/apache/spark/blob/master/sql/hive/
> > > src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
> > > [3]
> > > https://github.com/apache/ignite/blob/master/modules/
> > > spark/src/main/scala/org/apache/spark/sql/ignite/
> > > IgniteExternalCatalog.scala
> > >
> > > On Tue, Aug 21, 2018 at 9:31 AM, Nikolay Izhikov
> > > wrote:
> > >
> > > > Hello, Stuart.
> > > >
> > > > Can you do some research and find out how schema is handled in Data
> > >
> > > Frames
> > > > for a regular RDBMS such as Oracle, MySQL, etc?
> > > >
> > > > В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет:
> > > > > Stuart, Nikolay,
> > > > >
> > > > > I see that the 'Table' class (returned by listTables method) has a
> > > >
> > > > 'database' field. Can we use this one to report schema name?
> > > > >
> > > > > In any case, I think we should look into how this is done in data
> > >
> > > source
> > > > implementations for other databases. Any relational database has a
> > >
> > > notion
> > > > of schema, and I'm sure Spark integrations take this into account
> > >
> > > somehow.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov
> > > >
> > > > wrote:
> > > > > > Hello, Stuart.
> > > > > >
> > > > > > Personally, I think we should change current tables naming and
> > >
> > > return
> > > > table in form of `schema.table`.
> > > > > >
> > > > > > Valentin, could you share your opinion?
> > > > > >
> > > > > >
> > > > > > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет:
> > > > > > > Igniters,
> > > > > > >
> > > > > > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I
> > >
> > > are
> > > > > > > discussing whether to introduce a change which may impact
> > >
> > > backwards
> > > > > > > compatibility; Nikolay suggested we take the discussion to this
> > >
> > > list.
> > > > >
Re: Table Names in Spark Catalog
Hi Val, yes that's correct. I'd be happy to make the change to have the database reference the schema if Nikolay agrees. (I'll first need to do a bit of research into how to obtain the list of all available schemata...) Thanks, Stuart. On Tue, Aug 21, 2018 at 9:43 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Stuart, > > Thanks for pointing this out, I was not aware that we use Spark database > concept this way. Actually, this confuses me a lot. As far as I understand, > catalog is created in the scope of a particular IgniteSparkSession, which > in turn is assigned to a particular IgniteContext and therefore single > Ignite client. If that's the case, I don't think it should be aware of > other Ignite clients that are connected to other clusters. This doesn't > look like correct behavior to me, not to mention that with this approach > having multiple databases would be a very rare case. I believe we should > get rid of this logic and use Ignite schema name as database name in > Spark's catalog. > > Nikolay, what do you think? > > -Val > > On Tue, Aug 21, 2018 at 8:17 AM Stuart Macdonald > wrote: > >> Nikolay, Val, >> >> The JDBC Spark datasource[1] -- as far as I can tell -- has no >> ExternalCatalog implementation, it just uses the database specified in the >> JDBC URL. So I don't believe there is any way to call listTables() or >> listDatabases() for JDBC provider. >> >> The Hive ExternalCatalog[2] makes the distinction between database and >> table using the actual database and table mechanisms built into the >> catalog, which is fine because Hive has the clear distinction and >> hierarchy >> of databases and tables. >> >> *However* Ignite already uses the "database" concept in the Ignite >> >> ExternalCatalog[3] to mean the name of an Ignite instance. So in Ignite we >> have instances containing schemas containing tables, and Spark only has >> the >> concept of databases and tables so it seems like either we ignore one of >> the three Ignite concepts or combine two of them into database or table. >> The current implementation in the pull request combines Ignite schema and >> table attributes into the Spark table attribute. >> >> Stuart. >> >> [1] >> https://github.com/apache/spark/blob/master/sql/core/ >> src/main/scala/org/apache/spark/sql/execution/ >> datasources/jdbc/JDBCRelation.scala >> [2] >> https://github.com/apache/spark/blob/master/sql/hive/ >> src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala >> [3] >> https://github.com/apache/ignite/blob/master/modules/ >> spark/src/main/scala/org/apache/spark/sql/ignite/ >> IgniteExternalCatalog.scala >> >> On Tue, Aug 21, 2018 at 9:31 AM, Nikolay Izhikov >> wrote: >> >> > Hello, Stuart. >> > >> > Can you do some research and find out how schema is handled in Data >> Frames >> > for a regular RDBMS such as Oracle, MySQL, etc? >> > >> > В Пн, 20/08/2018 в 15:37 -0700, Valentin Kulichenko пишет: >> > > Stuart, Nikolay, >> > > >> > > I see that the 'Table' class (returned by listTables method) has a >> > 'database' field. Can we use this one to report schema name? >> > > >> > > In any case, I think we should look into how this is done in data >> source >> > implementations for other databases. Any relational database has a >> notion >> > of schema, and I'm sure Spark integrations take this into account >> somehow. >> > > >> > > -Val >> > > >> > > On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov >> > wrote: >> > > > Hello, Stuart. >> > > > >> > > > Personally, I think we should change current tables naming and >> return >> > table in form of `schema.table`. >> > > > >> > > > Valentin, could you share your opinion? >> > > > >> > > > >> > > > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет: >> > > > > Igniters, >> > > > > >> > > > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I >> are >> > > > > discussing whether to introduce a change which may impact >> backwards >> > > > > compatibility; Nikolay suggested we take the discussion to this >> list. >> > > > > >> > > > > Ignite implements a custom Spark catalog which provides an API by >> > which >> > > > > Spark users can list the tables which are available in Ignite >> which >> > can be >> > > > > queried via Spark SQL. Currently that table name list includes >> just >> > the >> > > > > names of the tables, but IGNITE-9228 is introducing a change which >> > allows >> > > > > optional prefixing of schema names to table names to disambiguate >> > multiple >> > > > > tables with the same name in different schemas. For the "list >> > tables" API >> > > > > we therefore have two options: >> > > > > >> > > > > 1. List the tables using both their table names and >> schema-qualified >> > table >> > > > > names (eg. [ "myTable", "mySchema.myTable" ]) even though they are >> > the same >> > > > > underlying table. This retains backwards compatibility with users >> who >> > > > > expect "myTable" to appear in the catalog. >> > > > > 2. List the tables
[GitHub] ignite pull request #4591: Ignite gg 13195 2.5.1 master
GitHub user macrergate opened a pull request: https://github.com/apache/ignite/pull/4591 Ignite gg 13195 2.5.1 master You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-13195-2.5.1-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4591.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4591 commit 4ab86f13cbc7fef68f5fd39354e242fe6cca285a Author: skalashnikov Date: 2018-04-18T13:37:20Z IGNITE-7512 Check for null before validateKeyAndValue in GridDhtAtomicCache.updateWithBatch - Fixes #3429. Signed-off-by: Alexey Goncharuk commit 7f318da5b357cca1a7e483a37d2556fbd56fc2a7 Author: Ilya Lantukh Date: 2018-04-18T14:56:56Z IGNITE-8017 Disable WAL during initial rebalancing commit b80fdbedc11b48aa89c6f29146363cec6f552abb Author: Ilya Lantukh Date: 2018-04-18T16:04:39Z IGNITE-8276 Fixed incorrect assertion during initialValue - Fixes #3827. Signed-off-by: Alexey Goncharuk commit 5cd32329fe5f303eaf369519771ee22f2a2cf822 Author: Pavel Kovalenko Date: 2018-04-18T16:41:44Z IGNITE-8116 Fixed historical rebalance from WAL commit 442d87d5799c27ef7258c7ac36e3e0b312676713 Author: Pavel Kovalenko Date: 2018-04-18T18:03:32Z IGNITE-8243 Fixed possible memory leak. Added latch manager to diagnostic messages. - Fixes #3850. Signed-off-by: dpavlov (cherry picked from commit 6c6f094) commit 5bc9fe9fa4e51c9af677f8120528746f345d990f Author: dpavlov Date: 2018-04-18T18:15:17Z IGNITE-8302 Reduce test IO consumption in case Direct IO is enabled, fixes flakyness of testPageRecoveryAfterFileCorruption - Fixes #3865. Signed-off-by: dpavlov (cherry picked from commit d853ebb) commit 3696c064f02567f471420ced50e2786a4f83d8c9 Author: dpavlov Date: 2018-04-18T18:16:35Z IGNITE-8302 Licenses fix for Reduce test IO consumption in case Direct IO (cherry picked from commit e7716f4) commit 061a5a82c95f4f926c7789764901b70383152df1 Author: shroman Date: 2018-04-18T10:38:52Z IGNITE-8143: Modified fetching EXTERNAL_LIBS for docker container. - Fixes #3751. Signed-off-by: shroman commit 83d488b083706e1d04200b4e603db1dc337d8fb4 Author: Dmitriy Shabalin Date: 2018-04-19T08:16:18Z IGNITE-8298 Web Console: Fixed loader under Safari. (cherry picked from commit 0897309) commit f9a0ee18f9d88b8f278d27dddc5f7cd3a52ef5c2 Author: Nikita Amelchev Date: 2018-04-19T09:57:06Z IGNITE-: Support Time data type in BinaryObjectImpl.writeFieldByOrder. This closes #2878. commit a095b98e1123f9562cbbc85d1c1c4b4221542c71 Author: Ivan Daschinskiy Date: 2018-04-19T12:25:23Z IGNITE-8021 Fixed tests - Fixes #3864. Signed-off-by: Alexey Goncharuk commit 27385e73183fe4bbc2f366f975facaefbcad2a6e Author: Alexey Goncharuk Date: 2018-04-18T07:59:46Z IGNITE-8201 Fixed NPE commit b2e5e66c21676d833b92cdb0420946b155076077 Author: Alexey Goncharuk Date: 2018-04-19T12:44:26Z IGNITE-8082 Added ZookeeperDiscoverySpi MBean - Fixes #3862. Signed-off-by: Alexey Goncharuk commit 7da1cb14ef4c06a6c4d29f04194b1f841bd3754d Author: Ilya Lantukh Date: 2018-04-19T14:53:26Z IGNITE-8327 Removed delaying communication SPI from tests commit bf3e0b5ddec5e3b1fa145e9a5eb72b1fb6569b54 Author: Ivan Rakov Date: 2018-04-19T16:04:08Z IGNITE-8325 Compressor thread may miss notification on stop - Fixes #3877. Signed-off-by: Alexey Goncharuk commit 7a4e804dbd9691f71b15bee49fe5c1858eae7b0b Author: dpavlov Date: 2018-04-19T12:17:58Z IGNITE-8017 Javadoc fix for Disable WAL during initial rebalancing (cherry picked from commit 02a0865) commit 26d78660809a84ebf959dd18ef4f8aef7e6cacbf Author: Alexey Goncharuk Date: 2018-04-20T12:28:31Z IGNITE-8258 Race in page memory in copyPageForCheckpoint, Ignite PDS 1 suite, test failed suite IgnitePdsPageReplacementTest.testPageReplacement. - Fixes #3821. Signed-off-by: dpavlov (cherry picked from commit 8caeb3d) commit e67ac4c170eaf74c4544a4408a58967fe066f371 Author: Dmitriy Govorukhin Date: 2018-04-20T13:00:00Z IGNITE-8078 Introduced additional data storage metrics - Fixes #3845. Signed-off-by: Alexey Goncharuk commit 553e0ff7c2fdecd5d4a64afd88d93e3ccffbe37a Author: Dmitriy Govorukhin Date: 2018-04-20T13:20:00Z IGNITE-8077 Introduced additional transaction metrics - Fixes #3879. Signed-off-by: Alexey Goncharuk commit d2332c4865445f372f6ce0ba70b7da1fec987a8f Author: Andrey Kuznetsov Date: 2018-04-20T16:51:59Z IGNITE-8303 Avoided failure handler invocation when exchange-worker terminates before reconnect. Signed-off-by: Andrey Gura commit
Re: Exchange stucks while node restoring state from WAL
Pavel, Thank you for so detailed introduction into the root of the problem of ticket IGNITE-7196. As you mentioned before, we can divide this ticket into two sub-tasks. So, I will file new issue for `Logical consistency recovery`. I think it's more convenient way for reviewing and merging each of these high level Steps (PRs). Currently, let's focus on `Physical(binary) consistency recovery` as the most critical part of this issue and synchronize our vision and vision of other community members on implementation details of Step 1. From this point everything what I'm saying is about only binary recovery. I think PR is almost ready (I need to check TC carefully). Can you look at my changes? Am I going the right way? PR: https://github.com/apache/ignite/pull/4520 Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-727 JIRA: https://issues.apache.org/jira/browse/IGNITE-7196 These are my milestones which I've tried to solve: 1. On first time cluster activation no changes required. We should keep this behavior. 2. `startMemoryPolicies` earlier if need. Now at the master branch data regions starts at onActivate method called. If we are trying to restore binary state on node reconnects to cluster we should start regions earlier. 3. `suspendLogging` method added to switch off handlers. Keep fast PDS cleanup when joining is not in BLT (or belongs to another BLT). Restore memory and metastorage initialization required `WALWriter` and `FileWriteHandle` to be started. They can lock directory\files and block cleanups. 4. `isBaselineNode` check before resumming logging. The same as point 3. Local node gets discovery data and determines #isLocalNodeNotInBaseline. If node not in current baseline it performs cleanup local PDS and we should reset state of previously initialized at startup metastorage instance. 5. `cacheProcessorStarted` callback added to restore memory at startup. We can restore memory only after having info about all `CacheGroupDescriptor`. To achieve this we should do restoring at the moment `GridCacheProcessor` started. 6. Move `fsync` for `MemoryRecoveryRecord` at node startup. We are fsync'ing it inside running PME. Suppose, it used for recovery actions and\or control cluster state outside Ignite project (methods `nodeStart` and `nodeStartedPointers` of this are not used at Ignite project code). I've moved them at the node startup, but may be we should dumping timestamp only at the time of node activation. Not sure. 7. `testBinaryRecoverBeforeExchange` test added to check restore before PME. Not sure that we should repeat wal reading logic as it was in `testApplyDeltaRecords` test for checking memory restoring. Suppose, minimally necessary condition is to check `lastRestored` and `checkpointHistory` states of database manager's (they should happend before node joins to cluster). I will create separate ticket and refactor these minor issues after we will finish with 7196: - simlify registerMBeans methods for `Ignite\GridCacheDatabaseSharedManager` - add @Override annotations if missed - fix abbreviations regarding IdeaAbbrPlugin - createDataRegionConfiguration method can be removed and simplified - startMemoryPolicies rename to startDataRegion - output messages format fix code style needed - fix format javadoc's - make inner classes private - missed logging for cleanupCheckpointDirectory, cleanupCheckpointDirectory, cleanupWalDirectories On Fri, 3 Aug 2018 at 18:20 Pavel Kovalenko wrote: > Maxim, > > In the addition to my answer, I propose to divide the whole task into 2 > steps: > 1) Physical consistency recovery before PME. > 2) Logical consistency recovery before PME. > > The first step can be done before the discovery manager start, as it no > needs any information about caches, it operates only with pages binary > data. On this step, we can do points a) and b) from answer 3) and fsync the > state of the page memory. > The second step is much harder to implement as it requires major changes in > Discovery SPI. Discovery node join event is triggered on coordinator after > node successfully joins to the cluster. To avoid it, we should introduce a > new state of node and fire discovery event only after logical recovery has > finished. I doubt that logical recovery consumes a lot of time of exchange, > but physical recovery does, as it performs fsync on disk after restore. > > > > > 2018-08-03 17:46 GMT+03:00 Pavel Kovalenko : > > > Hello Maxim, > > > > 1) Yes, Discovery Manager is starting after GridCacheProcessor, which > > starts GridCacheDatabaseSharedManager which invokes readMetastorage on > > start. > > > > 2) Before we complete the local join future, we create and add Exchange > > future on local node join to ExchangeManager. So, when local join future > > completes, first PME on that node should be already run or at least there > > will be first Exchange future in Exchange worker queue. > > > > 3) I don't exactly understand what do you mean saying "update obsolete
