Re: New SQL execution engine
Dear all, would it be possible to also have then // execution of sql queries on single node with that approach? My understanding is that, for the moment, the SQL queries a re single-threaded for a given node if there is no affinity. Best Regards -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: New SQL execution engine
Hi Steve, it is possible to execute queries in parallel even in the current engine, see docs here [1]. And of course this feature should also be available in the new engine, though it's architecture may be changed. [1] https://apacheignite.readme.io/v2.0/docs/sql-performance-and-debugging#query-parallelism -- Kind Regards Roman Kondakov On 15.11.2019 12:53, steve.hostett...@gmail.com wrote: Dear all, would it be possible to also have then // execution of sql queries on single node with that approach? My understanding is that, for the moment, the SQL queries a re single-threaded for a given node if there is no affinity. Best Regards -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[jira] [Created] (IGNITE-12375) Inconsistent persistent cache behaviour: containsKey returns false on a key returned by iterator
Matija Polajnar created IGNITE-12375:
Summary: Inconsistent persistent cache behaviour: containsKey
returns false on a key returned by iterator
Key: IGNITE-12375
URL: https://issues.apache.org/jira/browse/IGNITE-12375
Project: Ignite
Issue Type: Bug
Affects Versions: 2.7.6, 2.7
Reporter: Matija Polajnar
Attachments: ignite-bug.zip
On a fairly complex spring boot application using embedded Ignite persistent
storage, we've managed (multiple times) to get into a situation where some
persistent caches start behaving weirdly. The symptoms are such: caches'
{{iterator()}} method returns the elements we previously put into caches as
expected. {{size()}} also returns the expected value. But {{containsKey(x)}}
and {{get(x)}} return {{false}} and {{null}} respectively for some (or all)
keys that are expected to be in the cache and are even returned by the
{{iterator()}}.
The problem never starts occurring mid-run, but always after cluster restarts;
not at all always, and we suspect a necessary precondition is that cache
configurations are slightly changed, like having modified QueryEntities and
such. We also suspect this only happens on single-node clusters, so it might be
related to IGNITE-12297, but the workaround that works for that problem does
not fix the problem described here.
The caches in question then cannot be repaired short of destroying and
re-creating them and re-importing data.
We tried and failed to reproduce the problem from scratch in a small demo
application. We managed, however, to grab a {{work}} directory from our
application after corruption and then create a demo application with a minimal
set of classes needed to demonstrate the issue on reading (after corruption is
already present).
I'm attaching a zip file with the code (along with a maven pom.xml) and the
corrupted work directory. You can directly execute the demo by issuing {{mvn
compile exec:java}}, which will execute the
{{care.better.demo.ignitebug.BugApp}} class. In this class there's this method:
{code:java}
private static void replicateProblem(IgniteCache cache) {
int seen = 0;
Iterator> entryIterator = cache.iterator();
while (entryIterator.hasNext()) {
Object key = entryIterator.next().getKey();
if (!cache.containsKey(key) || cache.get(key) == null) {
LOG.error("UNSEEN KEY: {}", key);
} else {
seen++;
}
}
LOG.info("Size {}, seen {}.", cache.size(), seen);
}
{code}
After execution you will note log records like this one: ERROR
care.better.demo.ignitebug.BugApp.replicateProblem - UNSEEN KEY:
QueueKey{affinityKey=PartyIdArg{namespace='ЭМИАС Медработники', id='222'},
entryId=c059b587-78d3-4c75-b64f-8575ae3d2318}
We had no success in trying to find any lead while debugging through Ignite
source code so we kindly ask your assistance in hunting down this bug and,
until it is fixed, suggesting any possible work-around should this occur in a
production environment (it has not so far) where it is not practical to dump
all data from some cache into a file to be able to destroy, re-create and
re-import it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12376) Change level of "TCP client created ..." log messages to WARN
Ruslan Kuzmin created IGNITE-12376: -- Summary: Change level of "TCP client created ..." log messages to WARN Key: IGNITE-12376 URL: https://issues.apache.org/jira/browse/IGNITE-12376 Project: Ignite Issue Type: Improvement Reporter: Ruslan Kuzmin Log messages generated when createTcpClient takes more than CONNECTION_ESTABLISH_THRESHOLD_MS (100 by default) has INFO level. It seems it should be changed to WARN level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12377) .NET: Add IBinaryObjectBuilder.SetField(name, val, type) overload
Pavel Tupitsyn created IGNITE-12377: --- Summary: .NET: Add IBinaryObjectBuilder.SetField(name, val, type) overload Key: IGNITE-12377 URL: https://issues.apache.org/jira/browse/IGNITE-12377 Project: Ignite Issue Type: Improvement Components: platforms Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 2.8 -- This message was sent by Atlassian Jira (v8.3.4#803005)
RE: New SQL execution engine
Hi Roman, Actually it does not work as I expect it. Please see https://github.com/hostettler/igniteParallelQueries Do mvn clean install and then java -jar target/ignite-parallel-query-1.0.0-SNAPSHOT-jar-with-dependencies.jar This demonstrates that with or without the flag the query does not return the same result. I understand that it probably because I did not set an affinity but it is very counter-intuitive. Am I missing something? -Original Message- From: Roman Kondakov Sent: Friday, November 15, 2019 11:46 AM To: dev@ignite.apache.org Subject: Re: New SQL execution engine Hi Steve, it is possible to execute queries in parallel even in the current engine, see docs here [1]. And of course this feature should also be available in the new engine, though it's architecture may be changed. [1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite.readme.io%2Fv2.0%2Fdocs%2Fsql-performance-and-debugging%23query-parallelism&data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C2b752425baeb422af60408d769b9159d%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637094115967087030&sdata=eN7b2RCJegg8J9KQVK6TIFhcS6NG7j5pWKFxX9GWyYk%3D&reserved=0 -- Kind Regards Roman Kondakov On 15.11.2019 12:53, steve.hostett...@gmail.com wrote: > Dear all, > > would it be possible to also have then // execution of sql queries on single > node with that approach? > My understanding is that, for the moment, the SQL queries a re > single-threaded for a given node if there is no affinity. > > Best Regards > > > > -- > Sent from: > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-ignite-developers.2346864.n4.nabble.com%2F&data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C2b752425baeb422af60408d769b9159d%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637094115967087030&sdata=jXtLMt2dWYqM4KcRFkw4lby6K0o8glKnrLFgxZ96LbQ%3D&reserved=0
Re: IGNITE-12356 Migrate Flink module to ignite-extensions
Hi Saikat, As I guess the implementation stayed the same and we can skip a review of the source code. Were you able to integrate this into the TeamCity and with the release scripts? - Denis On Sun, Nov 10, 2019 at 2:55 PM Saikat Maitra wrote: > Hi > > I have raised a PR to migrate Flink module to ignite-extensions. > > PR https://github.com/apache/ignite-extensions/pull/1 > > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 > > Please review and share feedback. > > I am observing error with initialization of GridAbstractTest as > GridTestProperties is unable to access " > modules/core/src/test/config/test.properties" > > I am looking into it further. > > Regards, > Saikat >
Re: Collocated/replicatedOnly flags for Thin JDBC driver
Ignite SQL experts, Could you remind all of us what was the primary reason for adding the flags? It was about corner cases if I'm not mistaken and it makes sense to review those usage scenarios/patterns again. The flags are used for both JDBC and ODBC drivers and I would remove them only if they became irrelevant. - Denis On Thu, Nov 14, 2019 at 3:24 AM Sergey Kozlov wrote: > Hi, Igniters > > During testing Thin JDBC Driver I found some interesting things that would > be good to discuss: > > collocated flag > == > > The flag helps to optimize query against collocated data in advance. We've > following: > 1. Simple queries against single tables return weird results [3]. It means > that either a connection has reset every switch collocated/non-collocated > requests or make two connections for collocated and collocated > requests respectively. > 2. *distrubitedJoins *flag mostly covers the cases: > - if it is true we don't concern how the data distributed over cluster > - if not we always operate local data no matter it's collocated or not > 3. There was an activity to remove it but it was cancelled [2]. > > > replicatedOnly flag > === > > The flag helps to optimize query against replicated tables (caches) and > query always operates local data. > 1. But why can't we find at preparing the request for execution? > 2. For PRIMARY_SYNC cache mode using local node data may lead to > inconsistent results. Thus it may be implemented as an explicit hint for a > query if user want to do that and accept the possible risks or just re-use > *distrubitedJoins=false*. > 3. Same concern that it used per JDBC connection and required reconnect for > change the flag. > > I guess both flags should be deprecated and removed (2.8?). > > Ideally *distrubitedJoins *should be removed as well and replaced by query > hints (default *distrubitedJoins=true*) > > 1. https://apacheignite-sql.readme.io/docs/jdbc-driver > 2. https://issues.apache.org/jira/browse/IGNITE-6296 > 3. https://issues.apache.org/jira/browse/IGNITE-12372 > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com >
Re: [DISCUSS] Proposal for Ignite Extensions as a separate Bahir module or Incubator project
Hi Denis, Yes, sure I will start moving each extension one at a time and share updates. I am currently working on Flink extension. Regards, Saikat On Tue, Nov 12, 2019 at 1:55 PM Denis Magda wrote: > Saikat, > > Thanks for decoupling the task. Please ping the community once there is > anything ready for a review or a further discussion. > > - > Denis > > > On Sat, Nov 9, 2019 at 3:58 PM Saikat Maitra > wrote: > > > Hello, > > > > As discussed I have created the following jira issues for tracking the > > migration work for ignite-extensions. > > > > Main issue : IEP-36: Modularization > > https://issues.apache.org/jira/browse/IGNITE-12355 > > > > Sub-Tasks > > > > Migrate flink module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12356 > > Migrate Twitter module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12357 > > Migrate ZeroMQ module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12358 > > Migrate RocketMQ module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12359 > > Migrate Storm module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12360 > > Migrate Flume module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12361 > > Migrate MQTT module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12362 > > Migrate Camel module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12363 > > Migrate JMS module to ignite-extensions > > https://issues.apache.org/jira/browse/IGNITE-12364 > > > > Regards, > > Saikat > > > > On Fri, Nov 1, 2019 at 9:23 PM Saikat Maitra > > wrote: > > > > > Ivan, Thank you for initialising the master branch, much appreciate it. > > > > > > Denis, Emmanouil, > > > > > > We use both local server and mockito to write the tests for flink. > Incase > > > of sink it is easier to use a local server mode and publish the data > into > > > Ignite Sink where as in case of Ignite used as data source we use > mockito > > > to mock Ignite events and Ignite clusters. > > > > > > Regards, > > > Saikat > > > > > > > > > > > > On Fri, Nov 1, 2019 at 5:28 PM Emmanouil Gkatziouras < > > gkatzio...@gmail.com> > > > wrote: > > > > > >> Hi, > > >> > > >> I just took another look at the Kafka Implementation and I realised > that > > >> it > > >> is a local implementation that is used on the tests (I thought it was > a > > >> server already existing somewhere). > > >> It seems I might be able to spin up the Pub/Sub server locally (I > found > > an > > >> official tool which might work), in the same way it is done for Kafka. > > >> I will try to do so and come back with a pull request. > > >> Kind regards > > >> > > >> *Emmanouil Gkatziouras* > > >> https://egkatzioura.com/ | > > >> https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> https://github.com/gkatzioura > > >> > > >> > > >> On Fri, 1 Nov 2019 at 21:31, Denis Magda wrote: > > >> > > >> > Saikat, Roman, > > >> > > > >> > What's your thinking? I'm not sure we set any infrastructure to > > validate > > >> > our Kafka or Flink integrations. For instance, for Flink we use > > Mockito > > >> > just to ensure that basic tests pass: > > >> > https://en.wikipedia.org/wiki/Mockito > > >> > > > >> > Our IP finders for AWS, GCE or K8 don't start up any environment for > > >> > testing. Instead, we create some smoke tests ensuring that the APIs > > are > > >> not > > >> > broken. > > >> > > > >> > - > > >> > Denis > > >> > > > >> > > > >> > On Fri, Nov 1, 2019 at 1:32 PM Emmanouil Gkatziouras < > > >> gkatzio...@gmail.com > > >> > > > > >> > wrote: > > >> > > > >> > > Hi all, > > >> > > > > >> > > That's great news! Yes I am ready to do a pull request on the new > > >> repo. > > >> > > I believe it is ok to create a ticket for the Pub/Sub proposal > [1]. > > >> > > My next question has to do with the CI and the integration tests. > In > > >> > order > > >> > > for my tests to work, a local Pub/Sub server needs to be up and > > >> running. > > >> > > I tried to find any deployment scripts that set the infrastructure > > >> needed > > >> > > up and running (for example Kafka) but I was not successful on > this > > >> one. > > >> > > How should I proceed on that? > > >> > > > > >> > > Kind regards > > >> > > > > >> > > [1] > > >> > > > > >> > > > > >> > > > >> > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Pub-Sub-Streamer-Implementation-tc43944.html > > >> > > > > >> > > *Emmanouil Gkatziouras* > > >> > > https://egkatzioura.com/ | > > >> > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > >> > > https://github.com/gkatzioura > > >> > > > > >> > > > > >> > > On Fri, 1 Nov 2019 at 06:52, Ivan Pavlukhin > > >> wrote: > > >> > > > > >> > > > Saikat, > > >> > > > > > >> > > > I initialized master branch with empty README file [1]. I > suppose > > >> all > > >> > > > ignite committers have rights for push. > > >> > > > > > >> > > > [1] https://github.co
Re: IGNITE-12356 Migrate Flink module to ignite-extensions
Hi Denis, I am working on fixing the tests, an issue I am facing locally is since GridTestProperties is unable to find the test.properties file in local ignite-extensions repo, the tests are failing. I do have core module as a dependency for the project which has this test.properties file in config folder. I am debugging it further. I have not yet integrated with TeamCity or moved the release scripts. I will take them up as well in this PR. Regards, Saikat On Fri, Nov 15, 2019 at 3:52 PM Denis Magda wrote: > Hi Saikat, > > As I guess the implementation stayed the same and we can skip a review of > the source code. Were you able to integrate this into the TeamCity and with > the release scripts? > > - > Denis > > > On Sun, Nov 10, 2019 at 2:55 PM Saikat Maitra > wrote: > > > Hi > > > > I have raised a PR to migrate Flink module to ignite-extensions. > > > > PR https://github.com/apache/ignite-extensions/pull/1 > > > > Jira : https://issues.apache.org/jira/browse/IGNITE-12356 > > > > Please review and share feedback. > > > > I am observing error with initialization of GridAbstractTest as > > GridTestProperties is unable to access " > > modules/core/src/test/config/test.properties" > > > > I am looking into it further. > > > > Regards, > > Saikat > > >
