GridDhtPartitionsExchangeFuture.waitPartitionRelease issue - Unable to await partitions ?

[userx]

Hi,

I entered into the following scenario.

3 Ignite data grid servers started in PersistentMode say S1,S2,S3 in a
cluster.

On S1, there is a rebalancing started for a previous topology version say
24. 
The following log mentions that
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander
- Starting rebalancing [grp=PCache, mode=ASYNC,
fromNode=a8b0f10f-8ad9-45a4-aab3-a0562fd0d202, partitionsCount=311,
topology=AffinityTopologyVersion [topVer=24, minorTopVer=0], rebalanceId=65]

During the process, S1 got an exception while archiving a WAL segment and
threw an exception
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to archive
WAL segment
-> No space left on device

Since S2 and S3 are in the same cluster the corresponding threads are going
in a while loop with a message

"Unable to await partitions release latch within timeout: ServerLatch
[permits=1, pendingAcks=[327b3756-da19-4dd6-90da-7e0797f269d7],
super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion
[topVer=24, minorTopVer=1]]]"

The pending acks mentions the node id of S1.

I looked into the source and it is revolving in the following loop

if (!localJoinExchange()) {
try {
while (true) {
try {
releaseLatch.await(waitTimeout,
TimeUnit.MILLISECONDS);

if (log.isInfoEnabled())
log.info("Finished waiting for partitions
release latch: " + releaseLatch);

break;
}
catch (IgniteFutureTimeoutCheckedException ignored) {
U.warn(log, "Unable to await partitions release
latch within timeout: " + releaseLatch);

// Try to resend ack.
releaseLatch.countDown();
}
}
}
catch (IgniteCheckedException e) {
U.warn(log, "Stop waiting for partitions release latch: " +
e.getMessage());
}
}

Although the cluster is able to add a new node and reflect that in topology
but the re-balancing of previous topVersion 24 never finishes.

Is there a timeout to come out of this loop ? or any other configuration so
that the cluster can move forwards with S2 and S3 ?













 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Securing Connection Between Nodes

[Shesha Nanda]

Hi,

I have enabled SSL security for the ignite REST API's by adding below
configurations in the jetty.xml file.


/etc/keyStorePath


/etc/trustStorePath 



I am able to access access iginte REST API using HTTPS.


Now am trying to configure *Secure Connection Between Ignite Nodes.*
To enable Secure connection between Nodes, can we use the same keyStore and
trustStore files that's used for the Ignite REST API's?

-- 
*Regards*
*Sheshananda*

Securing Connection Between Nodes

[sheshananda]

Hi,

I have enabled SSL security for the ignite REST API's by adding below
configurations in the jetty.xml file.


/etc/keyStorePath


/etc/trustStorePath 



I am able to access access iginte REST API using HTTPS.


Now am trying to configure Secure Connection Between Ignite Nodes.
To enable Secure connection between Nodes, can we use the same keyStore and
trustStore files that's used for the Ignite REST API's?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

SQL Index Payload size?

[joseheitor]

Ho do I find the payload size of an index?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IGNITE_MAX_INDEX_PAYLOAD_SIZE

[joseheitor]

Thanks, Ilya.

How can find the payload size of an existing index?

Jose



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Limiting the number of files in WalArchive folder

[aMark]

Hi,

We are using Ignite 2.6 as persistent store in Partitioned Mode having 6
cluster 
node running, each node is running on different machine. 

Our cluster is having ~50G of data in IgnitePersistentStore but if I check
WalArchive folder then it contains lots of files and size is around ~200G. 
This uncontrollable growth has created space issue on drive and eventually
results in hang state for cluster.

I understand that WalArchive is needed to recover data in case of node
crash. So we dont want to disable the WalArchive altogether.

Is there a way to limit the number of files in WalArchive so that disc does
not run out of space ? 







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [RESOLVED] JDBC Streaming

[joseheitor]

Awesome - it worked!

Thanks, Ilya.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Changing Existing Model Class Name Not Accepting

[Ken Cheng]

Hi  Ilya Kasnacheev,

I ran into the same issue.

I just want to know is there any impact to delete the whole ignite workdir?
(is it safe to delete the workdir)

by the way, if it's a safe action, can I delete all cluster nodes' workdir
without stopping the node?


Thanks,
Ken Cheng


On Fri, Dec 7, 2018 at 10:58 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I don't think you can do that without removing your whole Ignite workdir.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 7 дек. 2018 г. в 17:22, siva :
>
>> Hi,
>>
>> I have changed existing Model class from "Entity" to "entity" and deployed
>> .when trying to insert records throwing exception duplicate type (Using
>> Native Persistence)
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: [RESOLVED] JDBC Streaming

[joseheitor]

Thanks Ilya,

Will try your suggestions...

Where in the XML config do I set the 'failure detection timeout'?

Thanks,
Jose



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: [RESOLVED] JDBC Streaming

[Ilya Kasnacheev]

Hello!

Can you try to increase 'failure detection timeout' property of your
DiscoverySpi to something like 5 minutes (30)?
This is failure detection handler which is supposed to catch deadlocks but
instead it catches things like WAL copying.

Also consider turning off WAL while ingesting data, turning it back on when
data load is finished.

this is by ALTER TABLE public.tableName NOLOGGING;
and then back ALTER TABLE public.tableName LOGGING;

Regards,
-- 
Ilya Kasnacheev


пт, 7 дек. 2018 г. в 19:29, joseheitor :

> Hi Ilya,
>
> Your recommendation works - thanks.
>
> When loading a few million records, I always seem to experience a node
> failure on one of my two nodes.
>
> I have attached the log from a recent run on version 2.7.0, failure
> occurred
> after about 10 minutes...
>
> Please let me know if you see anything I must address to prevent this?
>
> Thanks,
> Jose
>
> ignite-66c8e66a.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1652/ignite-66c8e66a.gz>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: IGNITE_MAX_INDEX_PAYLOAD_SIZE

[Ilya Kasnacheev]

Hello!

Yes, you should put a number there, instead of "true". My bad.

As for your problem, I have filed a ticket:
https://issues.apache.org/jira/browse/IGNITE-10602
You can work around this problem by invoking every CREATE INDEX in its own
statement.

Regards,
-- 
Ilya Kasnacheev


пт, 7 дек. 2018 г. в 18:53, joseheitor :

> I have found that issuing the CREATE INDEX commands separately (not as part
> of a multiple-statement command) works fine.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: [RESOLVED] JDBC Streaming

[joseheitor]

Hi Ilya,

Your recommendation works - thanks.

When loading a few million records, I always seem to experience a node
failure on one of my two nodes.

I have attached the log from a recent run on version 2.7.0, failure occurred
after about 10 minutes...

Please let me know if you see anything I must address to prevent this?

Thanks,
Jose

ignite-66c8e66a.gz
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IGNITE_MAX_INDEX_PAYLOAD_SIZE

[joseheitor]

I have found that issuing the CREATE INDEX commands separately (not as part
of a multiple-statement command) works fine.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IGNITE_MAX_INDEX_PAYLOAD_SIZE

[joseheitor]

Thanks, Ilya,

Isn't the IGNITE_MAX_INDEX_PAYLOAD_SIZE setting expecting a number
(byte-count)?

The SQL command(s) I run is as follows:

String cmd = "DROP TABLE IF EXISTS public.transactions;" +
 "DROP INDEX IF EXISTS transactions_id_k_v;" +
 "DROP INDEX IF EXISTS transactions_k_v_id;" +
 "DROP INDEX IF EXISTS transactions_pk_id;" +
 "CREATE TABLE public.transactions (pk INT, id INT, k
VARCHAR, v VARCHAR, PRIMARY KEY (pk, id))" +
 " WITH \"TEMPLATE=REPLICATED, BACKUPS=0,
ATOMICITY=TRANSACTIONAL_SNAPSHOT, WRITE_SYNCHRONIZATION_MODE=FULL_SYNC,
AFFINITY_KEY=id\";" +
 "CREATE INDEX transactions_id_k_v ON
public.transactions (id, k, v) INLINE_SIZE 150;" +
 "CREATE INDEX transactions_k_v_id ON
public.transactions (k, v, id) INLINE_SIZE 150;" +
 "CREATE INDEX transactions_pk_id ON public.transactions
(pk, id) INLINE_SIZE 20;";

-->

The error I get is:

java.sql.SQLException: Failed to parse query. Syntax error in SQL statement
"CREATE INDEX TRANSACTIONS_ID_K_V ON PUBLIC.TRANSACTIONS (ID, K, V)
INLINE_SIZE[*] 150;CREATE INDEX TRANSACTIONS_K_V_ID ON PUBLIC.TRANSACTIONS
(K, V, ID) INLINE_SIZE 150;CREATE INDEX TRANSACTIONS_PK_ID ON
PUBLIC.TRANSACTIONS (PK, ID) INLINE_SIZE 20; "; SQL statement:
CREATE INDEX transactions_id_k_v ON public.transactions (id, k, v)
INLINE_SIZE 150;CREATE INDEX transactions_k_v_id ON public.transactions (k,
v, id) INLINE_SIZE 150;CREATE INDEX transactions_pk_id ON
public.transactions (pk, id) INLINE_SIZE 20; [42000-197]
at
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:750)
at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:212)
at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:475)
at
com.example.ignite.IgniteApplication.initSQLDatabase(IgniteApplication.java:91)
at com.example.ignite.IgniteApplication.main(IgniteApplication.java:52)

--->

What am I doing wrong?

Jose



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: "Unable to await partitions release latch within timeout: ServerLatch" exception causing cluster freeze

[userx]

Hi Pavel

I am encountering the same issue in which it seems like the Server has
entered into an infinite loop and every 10 seconds i am seeing the following
message.


2018-12-06 15:49:23,188 WARN
[exchange-worker-#122%5b9b0820-ec94-493c-ae58-bc31aac873c6%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Unable to await partitions release latch within timeout: ClientLatch
[coordinator=TcpDiscoveryNode [id=a8b0f10f-8ad9-45a4-aab3-a0562fd0d202,
addrs=[10.62.21.54, 10.62.44.22, 10.63.216.22, 127.0.0.1],
sockAddrs=[ueu-ip-lapp0002.coresit.msci.org/10.63.216.22:47500,
ueu-ip-lapp0002.mgmt.msci.org/10.62.44.22:47500, /10.62.21.54:47500,
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1544102518093, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=24, minorTopVer=1]]]


2018-12-06 15:49:33,189 WARN
[exchange-worker-#122%5b9b0820-ec94-493c-ae58-bc31aac873c6%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Unable to await partitions release latch within timeout: ClientLatch
[coordinator=TcpDiscoveryNode [id=a8b0f10f-8ad9-45a4-aab3-a0562fd0d202,
addrs=[10.62.21.54, 10.62.44.22, 10.63.216.22, 127.0.0.1],
sockAddrs=[ueu-ip-lapp0002.coresit.msci.org/10.63.216.22:47500,
ueu-ip-lapp0002.mgmt.msci.org/10.62.44.22:47500, /10.62.21.54:47500,
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1544102518093, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=24, minorTopVer=1]]]


The code in which it seems to be in an infinite loop is 


if (!localJoinExchange()) {
try {
while (true) {
try {
releaseLatch.await(waitTimeout,
TimeUnit.MILLISECONDS);

if (log.isInfoEnabled())
log.info("Finished waiting for partitions
release latch: " + releaseLatch);

break;
}
catch (IgniteFutureTimeoutCheckedException ignored) {
U.warn(log, "Unable to await partitions release
latch within timeout: " + releaseLatch);

// Try to resend ack.
releaseLatch.countDown();
}
}
}
catch (IgniteCheckedException e) {
U.warn(log, "Stop waiting for partitions release latch: " +
e.getMessage());
}
}

Interestingly the PME is only seeming to be stuck with topology version 24.
Although the last topology version was 83 because of a new node added.

Where as for topology version 23, there was no such iteration of 10 seconds
so the PME would have completed just fine.

Can you help me out here ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Changing Existing Model Class Name Not Accepting

[Ilya Kasnacheev]

Hello!

I don't think you can do that without removing your whole Ignite workdir.

Regards,
-- 
Ilya Kasnacheev


пт, 7 дек. 2018 г. в 17:22, siva :

> Hi,
>
> I have changed existing Model class from "Entity" to "entity" and deployed
> .when trying to insert records throwing exception duplicate type (Using
> Native Persistence)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Changing Existing Model Class Name Not Accepting

[siva]

Hi,

I have changed existing Model class from "Entity" to "entity" and deployed
.when trying to insert records throwing exception duplicate type (Using
Native Persistence)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[ANNOUNCE] Apache Ignite 2.7.0 Vulnerable Dependecies Updates

[Dmitriy Pavlov]

The Apache Ignite Community is pleased to announce that recently released
Apache Ignite 2.7.0 replaces some vulnerable dependencies to versions with
fixes.



Apache Ignite https://ignite.apache.org/  is a memory-centric distributed
database, caching, and processing platform for transactional, analytical,
and streaming workloads delivering in-memory speeds at petabyte scale.



Apache Ignite 2.7 replaced following dependencies in to avoid usage of
vulnerable 3rd party software by end users:



Apache Log4j
https://nvd.nist.gov/vuln/detail/CVE-2017-5645

FasterXML jackson-databind
https://nvd.nist.gov/vuln/detail/CVE-2017-15095 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-17485 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-7525 ,

https://nvd.nist.gov/vuln/detail/CVE-2018-5968 ,

https://nvd.nist.gov/vuln/detail/CVE-2018-7489



Scala
https://nvd.nist.gov/vuln/detail/CVE-2017-15288

Apache Commons
https://nvd.nist.gov/vuln/detail/CVE-2015-6420 ,

https://nvd.nist.gov/vuln/detail/CVE-2015-7501 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-15708



Netty Project

https://nvd.nist.gov/vuln/detail/CVE-2016-4970

JCraft

https://nvd.nist.gov/vuln/detail/CVE-2016-5725



Apache Tomcat
https://nvd.nist.gov/vuln/detail/CVE-2016-3092 ,

https://nvd.nist.gov/vuln/detail/CVE-2016-8735 ,

https://nvd.nist.gov/vuln/detail/CVE-2018-8014


Guava
https://nvd.nist.gov/vuln/detail/CVE-2018-10237

Apache Camel
https://nvd.nist.gov/vuln/detail/CVE-2015-5344 ,

https://nvd.nist.gov/vuln/detail/CVE-2015-5348 ,

https://nvd.nist.gov/vuln/detail/CVE-2016-8749 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-12633 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-12634  ,

https://nvd.nist.gov/vuln/detail/CVE-2017-3159 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-5643

Spring Framework

https://nvd.nist.gov/vuln/detail/CVE-2018-1257 ,

https://nvd.nist.gov/vuln/detail/CVE-2018-1258



Spring Data Commons

https://nvd.nist.gov/vuln/detail/CVE-2018-1259 ,

https://nvd.nist.gov/vuln/detail/CVE-2018-1273



Jetty

https://nvd.nist.gov/vuln/detail/CVE-2016-4800 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-9735 ,

https://nvd.nist.gov/vuln/detail/CVE-2016-4800 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-9735 ,

https://nvd.nist.gov/vuln/detail/CVE-2016-4800 ,

https://nvd.nist.gov/vuln/detail/CVE-2017-7658



Lucene
https://nvd.nist.gov/vuln/detail/CVE-2017-12629

Mitigation:
•Upgrade to Apache Ignite 2.7 or later version



Credit:
Segu Riluvan discovered the usage of vulnerable modules in dependencies of
Apache Ignite.


Thanks for everyone who was involved into dependencies migration.

Best Regards,

Dmitriy Pavlov on behalf of Apache Ignite community

Re: IGNITE_MAX_INDEX_PAYLOAD_SIZE

[Ilya Kasnacheev]

Hello!

Just start your JVM with -DIGNITE_MAX_INDEX_PAYLOAD_SIZE=true or set
IGNITE_MAX_INDEX_PAYLOAD_SIZE environment variable to "true" before
starting JVM.

INLINE_SIZE worked the last time I have checked. Can you show the exact
statement and error message?

Regards,
-- 
Ilya Kasnacheev


пт, 7 дек. 2018 г. в 15:37, joseheitor :

> Hi,
>
> How do we set the global default IGNITE_MAX_INDEX_PAYLOAD_SIZE?
>
> (Note: Adding INLINE_SIZE xx to a CREATE INDEX command, as per the docs is
> throwing an SqlException!)
>
> Thanks,
> Jose
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: SQL Query plan confusion

[joseheitor]

My apologies, Vladimir.

The query-plans are swapped around (oops!).

The query-plan for the efficient query is in the 'bad' file. And
vice-versa...

Thanks,
Jose



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

IGNITE_MAX_INDEX_PAYLOAD_SIZE

[joseheitor]

Hi,

How do we set the global default IGNITE_MAX_INDEX_PAYLOAD_SIZE?

(Note: Adding INLINE_SIZE xx to a CREATE INDEX command, as per the docs is
throwing an SqlException!)

Thanks,
Jose



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite node stops on read/insert/update operations from ycsb

[Ilya Kasnacheev]

Hello!

You have to try different values of queue length, see what works best for
you.

Ignite will not listen 2 different IPs. I'm pretty sure there are ways to
turn 2 different physical interfaces into one logical one and bind on that.
By setting local address you will specify which one you want to bind on.

Regards,
-- 
Ilya Kasnacheev


чт, 6 дек. 2018 г. в 22:24, summasumma :

> Thanks for the details, Ilya.
>
> Even with connpair enabled, i got same issue when i pump in more data from
> ycsb.
> Then enabled MessageQueue=1024 as you have mentioned and now the issue is
> gone.
>
> Not sure how to arrive at correct number for messagequeue though? can u pls
> provide some inputs here since CPU also not utilized much.?
>
> One more clarification:
> I see the network bandwidth is the limit getting hit in ignite. so wanted
> to
> scale horizontally and added one more interface with 1Gbps. But when i
> added
> following property under tcpcommunucationspi its throwing error and ignite
> is not starting:
> >
>
> Is it not possible to use Ignite node to start listening on 2 IP addresses
> on 2 different interface in a VM?
> Actually i have 3 interfaces in my VM, but the 1st interface is mgmt-port.
> So wanted to bind on remaining 2 interfaces. Is it possible?
> Note: I do not want to start another node with localaddress property set to
> another IP since it will again occupy same amount of cache (in-memory)
> which
> will warrant to increase ram again.
>
> Please clarify.
>
> Thanks,
> ...summa
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: HELLO WORLD GA EXAMPLE

[matt egler]

I guess but why isnt it showing up in zeppelin?

On Dec 7, 2018 2:52 AM, "Ilya Kasnacheev"  wrote:

Hello!

I don't see any errors here, looks like this example has finished
successfully.

Regards,

-- 
Ilya Kasnacheev


пт, 7 дек. 2018 г. в 07:07, AlphaMufasaOmega :

> Dear @Zaleslaw could you please provide working links to those examples?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Failed to fetch SQL query result

[Ilya Kasnacheev]

Hello!

After more investigation we have opened a new ticket:
https://issues.apache.org/jira/browse/IGNITE-10580
And the workaround is different: you need to make query non-local, for
example by turning cache partitioned instead of replicated.
If you execute two identical local queries (ones that query planner
considers such) you will get into such problems.

You can also drop lazy=false for now.

Regards,
-- 
Ilya Kasnacheev


пт, 7 дек. 2018 г. в 06:56, yangjiajun <1371549...@qq.com>:

> Hello!
>
> Please fell free to deal with the code.
>
> I try to add where clause to my query,but it does not save me.The test code
> still throws the exception after a while.My ignite's version is 2.6.Does
> the
> workaround work in latest version?Or does this workaround only reduce the
> probability of the issue?
>
>
> ilya.kasnacheev wrote
> > Hello!
> >
> > I have confirmed this problem and we are working on it in
> > https://issues.apache.org/jira/browse/IGNITE-9171
> > I have converted your reproducer to Ignite test, hope you don't mind.
> >
> > There is a workaround in that you could add a parameter to your query,
> > such
> > as WHERE id > ?, and pass something like -1 to bind to this parameter,
> > which should prevent the issue from occurring.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 5 дек. 2018 г. в 16:21, yangjiajun <
>
> > 1371549332@
>
> >>:
> >
> >> Hello.
> >>
> >> I test setQueryTimeout method  to see whether it works or not because I
> >> try
> >> to prevent long time queries from blocking other operations.According to
> >> my
> >> test,this method has no effect.You can also see that I have already
> >> removed
> >> it in my latest test code.The code is simple and reproduce my issue.
> >>
> >>
> >> Hello!
> >>
> >> Why do you have setQueryTimeout here? What happens if you take it off?
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> ср, 5 дек. 2018 г. в 09:17, yangjiajun <1371549332@>:
> >>
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: HELLO WORLD GA EXAMPLE

[Ilya Kasnacheev]

Hello!

I don't see any errors here, looks like this example has finished
successfully.

Regards,
-- 
Ilya Kasnacheev


пт, 7 дек. 2018 г. в 07:07, AlphaMufasaOmega :

> Dear @Zaleslaw could you please provide working links to those examples?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: SQL Query plan confusion

[Andrey Mashenkov]

Hi,

The difference in query plans is a 'bad' one use  PUBLIC.TEST_DATA_K_V_ID
index instead of PUBLIC.TEST_DATA_ID_K_V.
Have you tries to remove other indices and left TEST_DATA_ID_K_V only?

It looks weird for me, the 'good' query plan shows unicast requests as ID
is affinity field, but 'bad' says nothing about broadcast.
Are you run query with same flags? Do you have setDistributedJoins=true?

Another possible reason: TEST_DATA_K_V_ID can't be used efficiently as
default inline size is 10, that means only 7 first chars of composite index
will be inlined. You have at least 2 fields with 'trans.c' prefix, so
Ignite may scan a huge number of rows with using this index.
Try to remove it of use Hints to force Ignite use 'efficient' one.

In case you are really need to have 'k' as a first column in composite
index, you can try to increase inline size.
FYI: For now, fixed length columns will use 1 addition byte to inline (for
a column type prefix) and var-len column requires 3 additional bytes for
inline (1 byte for type, 2 bytes for size)


On Fri, Dec 7, 2018 at 11:19 AM Vladimir Ozerov 
wrote:

> Hi Jose,
>
> I am a bit lost in two provided explains. Both "godd" and "bad" contain "k
> = 'trans.cust.first_name'" condition. Could you please confirm that they
> are correct? Specifically, I cannot understand why this condition is
> present in "good" explain.
>
> On Tue, Nov 27, 2018 at 12:06 PM joseheitor 
> wrote:
>
>> 1. - I have a nested join query on a table of 8,000,000 records which
>> performs similar or better than PostreSQL (~10ms) on my small test setup
>> (2x
>> nodes, 8GB, 2CPU):
>>
>> SELECT
>> mainTable.pk, mainTable.id, mainTable.k, mainTable.v
>> FROM
>> public.test_data AS mainTable
>> INNER JOIN (
>> SELECT
>> lastName.id
>> FROM
>> (SELECT id FROM public.test_data WHERE k
>> = 'trans.cust.last_name' AND v
>> = 'Smythe-Veall') AS lastName
>> INNER JOIN
>> (SELECT id FROM
>> public.test_data WHERE k = 'trans.date' AND v =
>> '2017-12-21') AS transDate ON transDate.id = lastName.id
>> INNER JOIN
>> (SELECT id FROM
>> public.test_data WHERE k = 'trans.amount' AND cast(v
>> AS integer) > 9) AS transAmount ON transAmount.id = lastName.id
>> ) AS subTable ON mainTable.id = subTable.id
>> ORDER BY 1, 2
>>
>>
>> 2. - By simply adding a WHERE clause at the end, the performance becomes
>> catastrophic on Ignite (~10s for subsequent queries - first query takes
>> many
>> minutes). On PostgreSQL performance does not change...
>>
>> SELECT
>> mainTable.pk, mainTable.id, mainTable.k, mainTable.v
>> FROM
>> public.test_data AS mainTable
>> INNER JOIN (
>> SELECT
>> lastName.id
>> FROM
>> (SELECT id FROM public.test_data WHERE k
>> = 'trans.cust.last_name' AND v
>> = 'Smythe-Veall') AS lastName
>> INNER JOIN
>> (SELECT id FROM
>> public.test_data WHERE k = 'trans.date' AND v =
>> '2017-12-21') AS transDate ON transDate.id = lastName.id
>> INNER JOIN
>> (SELECT id FROM
>> public.test_data WHERE k = 'trans.amount' AND cast(v
>> AS integer) > 9) AS transAmount ON transAmount.id = lastName.id
>> ) AS subTable ON mainTable.id = subTable.id
>> *WHERE
>> mainTable.k = 'trans.cust.first_name'*
>> ORDER BY 1, 2
>>
>> What can I do to optimise this query for Ignite???
>>
>> (Table structure and query plans attached for reference)
>>
>> Thanks,
>> Jose
>> table.sql
>> 
>> good-join-query.txt
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/t1652/good-join-query.txt>
>>
>> bad-join-query.txt
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/file/t1652/bad-join-query.txt>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

-- 
Best regards,
Andrey V. Mashenkov

Re: SQL Query plan confusion

[Vladimir Ozerov]

Hi Jose,

I am a bit lost in two provided explains. Both "godd" and "bad" contain "k
= 'trans.cust.first_name'" condition. Could you please confirm that they
are correct? Specifically, I cannot understand why this condition is
present in "good" explain.

On Tue, Nov 27, 2018 at 12:06 PM joseheitor  wrote:

> 1. - I have a nested join query on a table of 8,000,000 records which
> performs similar or better than PostreSQL (~10ms) on my small test setup
> (2x
> nodes, 8GB, 2CPU):
>
> SELECT
> mainTable.pk, mainTable.id, mainTable.k, mainTable.v
> FROM
> public.test_data AS mainTable
> INNER JOIN (
> SELECT
> lastName.id
> FROM
> (SELECT id FROM public.test_data WHERE k =
> 'trans.cust.last_name' AND v
> = 'Smythe-Veall') AS lastName
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.date' AND v =
> '2017-12-21') AS transDate ON transDate.id = lastName.id
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.amount' AND cast(v
> AS integer) > 9) AS transAmount ON transAmount.id = lastName.id
> ) AS subTable ON mainTable.id = subTable.id
> ORDER BY 1, 2
>
>
> 2. - By simply adding a WHERE clause at the end, the performance becomes
> catastrophic on Ignite (~10s for subsequent queries - first query takes
> many
> minutes). On PostgreSQL performance does not change...
>
> SELECT
> mainTable.pk, mainTable.id, mainTable.k, mainTable.v
> FROM
> public.test_data AS mainTable
> INNER JOIN (
> SELECT
> lastName.id
> FROM
> (SELECT id FROM public.test_data WHERE k =
> 'trans.cust.last_name' AND v
> = 'Smythe-Veall') AS lastName
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.date' AND v =
> '2017-12-21') AS transDate ON transDate.id = lastName.id
> INNER JOIN
> (SELECT id FROM
> public.test_data WHERE k = 'trans.amount' AND cast(v
> AS integer) > 9) AS transAmount ON transAmount.id = lastName.id
> ) AS subTable ON mainTable.id = subTable.id
> *WHERE
> mainTable.k = 'trans.cust.first_name'*
> ORDER BY 1, 2
>
> What can I do to optimise this query for Ignite???
>
> (Table structure and query plans attached for reference)
>
> Thanks,
> Jose
> table.sql
> 
> good-join-query.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1652/good-join-query.txt>
>
> bad-join-query.txt
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1652/bad-join-query.txt>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>