Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Igniters, I see nothing wrong with Yury's proposal in regards full-text search API evolution as long as Yury is ready to push it forward. As for the in-memory mode only, it makes total sense for in-memory data grid deployments when Ignite caches data of an underlying DB like Postgres. As part of the changes, I would simply throw an exception (by default) if the one attempts to use text indices with the native persistence enabled. If the person is ready to live with that limitation that an explicit configuration change is needed to come around the exception. Thoughts? - Denis On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga wrote: > Hello to all again, > > Thank you for important comments and notes given below! > > Let me answer and continue the discussion. > > (I) Overall needs in Lucene indexing > > Alexei has referenced to > https://issues.apache.org/jira/browse/IGNITE-5371 where > absence of index persistence was declared as an obstacle to further > development. > > a) This ticket is already closed as not valid.b) There are definite needs > (and in our project as well) in just in-memory indexing of selected data. > We intend to use search capabilities for fetching limited amount of records > that should be used in type-ahead search / suggestions. > Not all of the data will be indexed and the are no need in Lucene index to > be persistence. Hope this is a wide pattern of text-search usage. > > (II) Necessary fixes in current implementation. > > a) Implementation of correct *limit *(*offset* seems to be not required in > text-search tasks for now) > I have investigated the data flow for distributed text queries. it was > simple test prefix query, like 'name'*='ene*'* > For now each server-node returns all response records to the client-node > and it may contain ~thousands, ~hundred thousands records. > Event if we need only first 10-100. Again, all the results are added to > queue in GridCacheQueryFutureAdapter in arbitrary order by pages. > I did not find here any means to deliver deterministic result. > So implementing limit as part of query and (GridCacheQueryRequest) will not > change the nature of response but will limit load on nodes and networking. > > Can we consider to open a ticket for this? > > (III) Further extension of Lucene API exposition to Ignite > > a) Sorting > The solution for this could be: > - Make entities comparable > - Add custom comparator to entity > - Add annotations to mark sorted fields for Lucene indexing > - Use comparators when merging responses or reducing to desired limit on > client node. > Will require full result set to be loaded into memory. Though can be used > for relatively small limits. > BR, > Yuriy Shuliha > > пт, 30 серп. 2019 о 10:37 Alexei Scherbakov > пише: > > > Yuriy, > > > > Note what one of major blockers for text queries is [1] which makes > lucene > > indexes unusable with persistence and main reason for discontinuation. > > Probably it's should be addressed first to make text queries a valid > > product feature. > > > > Distributed sorting and advanved querying is indeed not a trivial task. > > Some kind of merging must be implemented on query originating node. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5371 > > > > чт, 29 авг. 2019 г. в 23:38, Denis Magda : > > > > > Yuriy, > > > > > > If you are ready to take over the full-text search indexes then please > go > > > ahead. The primary reason why the community wants to discontinue them > > first > > > (and, probable, resurrect later) are the limitations listed by Andrey > and > > > minimal support from the community end. > > > > > > - > > > Denis > > > > > > > > > On Thu, Aug 29, 2019 at 1:29 PM Andrey Mashenkov < > > > andrey.mashen...@gmail.com> > > > wrote: > > > > > > > Hi Yuriy, > > > > > > > > Unfortunatelly, there is a plan to discontinue TextQueries in Ignite > > [1]. > > > > Motivation here is text indexes are not persistent, not transactional > > and > > > > can't be user together with SQL or inside SQL. > > > > and there is a lack of interest from community side. > > > > You are weclome to take on these issues and make TextQueries great. > > > > > > > > 1, PageSize can't be used to limit resultset. > > > > Query results return from data node to client-side cursor in > > page-by-page > > > > manner and > > > > this parameter is designed control page size. It is supposed query > > > executes > > > > lazily on server side and > > > > it is not excepted full resultset be loaded to memory on server side > at > > > > once, but by pages. > > > > Do you mean you found Lucene load entire resultset into memory before > > > first > > > > page is sent to client? > > > > > > > > I'd think a new parameter should be added to limit result. The best > > > > solution is to use query language commands for this, e.g. > > "LIMIT/OFFSET" > > > in > > > > SQL. > > > > > > > > This task doesn't look trivial. Query is distributed operation and > same > > > > user query will be executed on
[jira] [Created] (IGNITE-12181) Rebalance hangs on BLT change on cluster with in-memory regions
Maxim Muzafarov created IGNITE-12181: Summary: Rebalance hangs on BLT change on cluster with in-memory regions Key: IGNITE-12181 URL: https://issues.apache.org/jira/browse/IGNITE-12181 Project: Ignite Issue Type: Bug Reporter: Maxim Muzafarov Assignee: Maxim Muzafarov Fix For: 2.8 Attachments: RebalanceInMemoryWithPersistence.java # Configured BLT on the _node-1_ (cluster activated) # Configured in-memory region with a single cache on the _node-1_ # A new _node-2_ join to the cluster # Set new BLT based on two nodes # Rebalance hangs -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IGNITE-12180) [ML] Add support of the next Imputing Strategies: MIN, MAX
Aleksey Zinoviev created IGNITE-12180: - Summary: [ML] Add support of the next Imputing Strategies: MIN, MAX Key: IGNITE-12180 URL: https://issues.apache.org/jira/browse/IGNITE-12180 Project: Ignite Issue Type: Sub-task Components: ml Affects Versions: 2.8 Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Fix For: 2.8 Add support of the next Imputing Strategies: MIN, MAX -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IGNITE-12179) Test and javadoc fixes
Anton Kalashnikov created IGNITE-12179: -- Summary: Test and javadoc fixes Key: IGNITE-12179 URL: https://issues.apache.org/jira/browse/IGNITE-12179 Project: Ignite Issue Type: Bug Reporter: Anton Kalashnikov Assignee: Anton Kalashnikov Some javadoc package descriptions missed: * org.apache.ignite.spi.communication.tcp.internal * org.apache.ignite.spi.discovery.zk * org.apache.ignite.spi.discovery.zk.internal * org.apache.ignite.ml.structures.partition * org.gridgain.grid.persistentstore.snapshot.file.copy Unclear CLEANUP_RESTARTING_CACHES command in snapshot utility unclear error when connecting to secure cluster (SSL + Auth) Update log message to avoid confusion for an user *.testTtlNoTx flaky failed on TC TcpCommunicationSpiFreezingClientTest failed TcpCommunicationSpiFaultyClientSslTest.testNotAcceptedConnection failed testCacheIdleVerifyPrintLostPartitions failed -- This message was sent by Atlassian Jira (v8.3.2#803003)
Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Hello to all again, Thank you for important comments and notes given below! Let me answer and continue the discussion. (I) Overall needs in Lucene indexing Alexei has referenced to https://issues.apache.org/jira/browse/IGNITE-5371 where absence of index persistence was declared as an obstacle to further development. a) This ticket is already closed as not valid.b) There are definite needs (and in our project as well) in just in-memory indexing of selected data. We intend to use search capabilities for fetching limited amount of records that should be used in type-ahead search / suggestions. Not all of the data will be indexed and the are no need in Lucene index to be persistence. Hope this is a wide pattern of text-search usage. (II) Necessary fixes in current implementation. a) Implementation of correct *limit *(*offset* seems to be not required in text-search tasks for now) I have investigated the data flow for distributed text queries. it was simple test prefix query, like 'name'*='ene*'* For now each server-node returns all response records to the client-node and it may contain ~thousands, ~hundred thousands records. Event if we need only first 10-100. Again, all the results are added to queue in GridCacheQueryFutureAdapter in arbitrary order by pages. I did not find here any means to deliver deterministic result. So implementing limit as part of query and (GridCacheQueryRequest) will not change the nature of response but will limit load on nodes and networking. Can we consider to open a ticket for this? (III) Further extension of Lucene API exposition to Ignite a) Sorting The solution for this could be: - Make entities comparable - Add custom comparator to entity - Add annotations to mark sorted fields for Lucene indexing - Use comparators when merging responses or reducing to desired limit on client node. Will require full result set to be loaded into memory. Though can be used for relatively small limits. BR, Yuriy Shuliha пт, 30 серп. 2019 о 10:37 Alexei Scherbakov пише: > Yuriy, > > Note what one of major blockers for text queries is [1] which makes lucene > indexes unusable with persistence and main reason for discontinuation. > Probably it's should be addressed first to make text queries a valid > product feature. > > Distributed sorting and advanved querying is indeed not a trivial task. > Some kind of merging must be implemented on query originating node. > > [1] https://issues.apache.org/jira/browse/IGNITE-5371 > > чт, 29 авг. 2019 г. в 23:38, Denis Magda : > > > Yuriy, > > > > If you are ready to take over the full-text search indexes then please go > > ahead. The primary reason why the community wants to discontinue them > first > > (and, probable, resurrect later) are the limitations listed by Andrey and > > minimal support from the community end. > > > > - > > Denis > > > > > > On Thu, Aug 29, 2019 at 1:29 PM Andrey Mashenkov < > > andrey.mashen...@gmail.com> > > wrote: > > > > > Hi Yuriy, > > > > > > Unfortunatelly, there is a plan to discontinue TextQueries in Ignite > [1]. > > > Motivation here is text indexes are not persistent, not transactional > and > > > can't be user together with SQL or inside SQL. > > > and there is a lack of interest from community side. > > > You are weclome to take on these issues and make TextQueries great. > > > > > > 1, PageSize can't be used to limit resultset. > > > Query results return from data node to client-side cursor in > page-by-page > > > manner and > > > this parameter is designed control page size. It is supposed query > > executes > > > lazily on server side and > > > it is not excepted full resultset be loaded to memory on server side at > > > once, but by pages. > > > Do you mean you found Lucene load entire resultset into memory before > > first > > > page is sent to client? > > > > > > I'd think a new parameter should be added to limit result. The best > > > solution is to use query language commands for this, e.g. > "LIMIT/OFFSET" > > in > > > SQL. > > > > > > This task doesn't look trivial. Query is distributed operation and same > > > user query will be executed on data nodes > > > and then results from all nodes should be correcly merged before being > > > returned via client-cursor. > > > So, LIMIT should be applied on every node and then on merge phase. > > > > > > Also, this may be non-obviuos, limiting results make no sence without > > > sorting, > > > as there is no guarantee every next query run will return same data > > because > > > of page reordeing. > > > Basically, merge phase receive results from data nodes asynchronously > and > > > messages from different nodes can't be ordered. > > > > > > 2. > > > a. "tokenize" param name (for @QueryTextFiled) looks more verbose, > isn't > > > it. > > > b,c. What about distributed query? How partial results from nodes will > be > > > merged? > > > Does Lucene allows to configure comparator for data sorting? > > > What comparator Ignite should choose to sort result on
[jira] [Created] (IGNITE-12178) DEBUG logging may throw exception from toString of tx objects
Ilya Kasnacheev created IGNITE-12178:
Summary: DEBUG logging may throw exception from toString of tx
objects
Key: IGNITE-12178
URL: https://issues.apache.org/jira/browse/IGNITE-12178
Project: Ignite
Issue Type: Improvement
Components: general
Reporter: Ilya Kasnacheev
See
https://stackoverflow.com/questions/57727242/apache-ignite-failed-to-create-string-representation-of-binary-object
{code:java}
0
I am getting below exception and not able to figure out what is wrong with the
code.
I have simplified my pojo classes which are to be persisted to ignite cache,
but still the complexity remains.
All my pojos are serializable but few of them have business logic code, dao,
application context object. These objects can't be removed. Removing these
things from the code will require whole code refractoring.
class org.apache.ignite.IgniteException: Failed to create string representation
of binary object.
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1022)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:864)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearSingleGetResponse.toString(GridNearSingleGetResponse.java:317)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1162)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1209)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$6.apply(GridDhtCacheAdapter.java:1003)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter$6.apply(GridDhtCacheAdapter.java:938)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetRequest(GridDhtCacheAdapter.java:938)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$300(GridDhtAtomicCache.java:135)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$4.apply(GridDhtAtomicCache.java:257)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$4.apply(GridDhtAtomicCache.java:252)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteException: Failed to create string
representation of binary object.
at
org.apache.ignite.internal.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:189)
at
org.apache.ignite.internal.binary.BinaryObjectImpl.toString(BinaryObjectImpl.java:920)
at java.lang.String.valueOf(String.java:2994)
at
org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
at
org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
... 27 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read
field: currentContacts
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.wrapFieldException(BinaryReaderExImpl.java:446)
at
[jira] [Created] (IGNITE-12177) CPP: Add Java compute support
Alexandr Shapkin created IGNITE-12177: - Summary: CPP: Add Java compute support Key: IGNITE-12177 URL: https://issues.apache.org/jira/browse/IGNITE-12177 Project: Ignite Issue Type: Improvement Components: platforms Affects Versions: 2.7.5 Reporter: Alexandr Shapkin Currently compute.h contains only platform-specific API declaration (C++). But Ignite does allow a native Java task execution. This feature already has been implemented for .NET platform. Lets port the same changes to C++ client (use PlatformCompute#OP_EXEC_NATIVE). This would help us to create a workaround for a mixed platform cluster that requires a compute calls. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IGNITE-12175) ODBC driver does not support WVARCHAR (typeId=-9) which breaks pyodbc
Ilya Kasnacheev created IGNITE-12175:
Summary: ODBC driver does not support WVARCHAR (typeId=-9) which
breaks pyodbc
Key: IGNITE-12175
URL: https://issues.apache.org/jira/browse/IGNITE-12175
Project: Ignite
Issue Type: Improvement
Components: odbc
Affects Versions: 2.7.5
Reporter: Ilya Kasnacheev
Subj:
pyodbc.Error: ('HYC00', '[HYC00] Data type is not supported. [typeId=-9] (0)
(SQLBindParameter)')
https://stackoverflow.com/questions/57971252/python-string-to-varchar-mapping-using-odbc-with-apache-ignite-not-supported/57973839#57973839
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (IGNITE-12176) .NET: Apache.Ignite.exe fails when config has custom logger or plugins
Pavel Tupitsyn created IGNITE-12176:
---
Summary: .NET: Apache.Ignite.exe fails when config has custom
logger or plugins
Key: IGNITE-12176
URL: https://issues.apache.org/jira/browse/IGNITE-12176
Project: Ignite
Issue Type: Bug
Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
Steps to reproduce:
* Implement Apache.Ignite.Core.Log.ILogger interface, compiled into assembly
* Create a config file with that logger:
{code}
http://ignite.apache.org/schema/dotnet/IgniteConfigurationSection;>
{code}
* Run Apache.Ignite.exe with that config file while CustomAsm is in a separate
dir:
{code}
Apache.Ignite.exe -ConfigFileName=myconfig.xml -assembly=c:\w\CustomAsm.dll
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
Re: Nabble message wrapping
I've created an INFRA ticket: https://issues.apache.org/jira/browse/INFRA-19042 Denis On 12 Sep 2019, 00:46 +0300, Denis Magda , wrote: > Thanks for sharing details. > > 2. Change the ezmlm configuration to match the one of the users list. We > > need to find a person who has access to the Apache infra in order to do this > > > We can open a ticket for ASF INFRA. Could you please do that and see what > they say? > > - > Denis > > > On Wed, Sep 11, 2019 at 1:36 AM Denis Mekhanikov > wrote: > > > Denis, > > > > I did some investigation on this and started a Nabble support thread: > > http://support.nabble.com/Line-wrapping-in-quotes-td7604136.html > > It turned out, that Nabble has a little to do with it. It’s the mail list > > itself. > > > > When you send a plain text message to the dev list either from Nabble or > > in any other way, it’s getting wrapped to be 80 characters wide. > > HTML messages don’t have this issue. Nabble uses plain text by default. > > User list doesn’t wrap quotes though, even if you use plain text format. > > You can see that my reply in the following thread is wrapped, but not the > > quote: > > http://apache-ignite-users.70518.x6.nabble.com/Ignition-Start-Timeout-if-connection-is-unsuccessful-td29289.html > > > > I see two possibilities of solving this issue: > > > > 1. Make Nabble always send messages in HTML format somehow. The issue will > > still occur for other mail clients though. > > 2. Change the ezmlm configuration to match the one of the users list. We > > need to find a person who has access to the Apache infra in order to do > > this. > >
Re: [IEP-35] Monitoring & Profiling. Phase 2
Folks, I honestly tried to follow the discussion, but I think that I lost the point of the debate. Should we try to exploit the newly introduced slack to discuss the change and then send a follow-up here? --AG
Re: [IGNITE-9836] Invalid check of ea java versions
I can’t take any credit for the patch but if the original author has lost interest I’m happy to help push it through. Regards, Stephen > On 16 Sep 2019, at 19:31, Denis Magda wrote: > > Stephen, > > Thanks for sending the patch! Seems that Igniters are already actively > reviewing it in JIRA. > > - > Denis > > > On Mon, Sep 16, 2019 at 7:03 AM Stephen Darlington < > stephen.darling...@gridgain.com> wrote: > >> Hi, >> >> Would someone mind taking a quick look at this ticket? Basically, a clean >> download of Ignite won’t start if the version of Java you’re using has a >> number like “java version "1.8.0_202-ea””. (This is the default if you get >> your JDK using Homebrew on a Mac.) >> >>> https://issues.apache.org/jira/browse/IGNITE-9836 < >> https://issues.apache.org/jira/browse/IGNITE-9836> >> >> This has been bugging me for ages and now that I look at it I find that >> there’s already a tiny, working patch available. >> >> Regards, >> Stephen
