Re: sizing

[Denis Mekhanikov]

Clay,

If you want to store plain strings without any schema or markup, then use
varchar.
But if you plan to store POJOs, then binary objects should certainly be
used instead of varchar. Binary types contain meta information, improving
type safety of stored data.

Binary objects don't apply any compression. This improvement is planned to
be implemented on data storage level, not the objects format.
IEP related to data compression:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite

Denis

ср, 1 мая 2019 г. в 19:33, Evgenii Zhuravlev :

> Hi,
>
> What do you mean here? Do you mean binary fields VS varchars? Could you
> please give an example of the object?
>
> Thank you,
> Evgenii
>
> вт, 30 апр. 2019 г. в 12:27, Clay Teahouse :
>
>> Hi All
>> If I choose binary data type for my objects, as opposed to varchar, will
>> it result in any saving, and yes, how much?
>> I know that binary type would be faster to read/write but wanted to see
>> if there will be any saving in storage.
>>
>> thanks
>> Clay
>>
>

Re: sizing

[Evgenii Zhuravlev]

Hi,

What do you mean here? Do you mean binary fields VS varchars? Could you
please give an example of the object?

Thank you,
Evgenii

вт, 30 апр. 2019 г. в 12:27, Clay Teahouse :

> Hi All
> If I choose binary data type for my objects, as opposed to varchar, will
> it result in any saving, and yes, how much?
> I know that binary type would be faster to read/write but wanted to see if
> there will be any saving in storage.
>
> thanks
> Clay
>

sizing

[Clay Teahouse]

Hi All
If I choose binary data type for my objects, as opposed to varchar, will it
result in any saving, and yes, how much?
I know that binary type would be faster to read/write but wanted to see if
there will be any saving in storage.

thanks
Clay

Re: SELECT Statement cancellation & memory sizing

[Dave Harvey]

Just saw 2.4 release notes: Improved COUNT(*) performance 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: SELECT Statement cancellation & memory sizing

[lawrencefinn]

Also AFAIK, visor is very inaccurate on offheap memory measurements.  at
least it used to be



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: SELECT Statement cancellation & memory sizing

[lawrencefinn]

ignite is not optimized for count queries.  it visits each object to do the
count and doesn't do the count off of an index (or some cached store). 
seems kind of silly, especially if you have a count on indexed fields.

i think query cancellation only works within the API.  I think you can
either set a timeout for the query or you can cancel it with the handle to
the cursor.  i doubt cancelling through JDBC works.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

SELECT Statement cancellation & memory sizing

[David Harvey]

I have a question about SQL query via JDBC cancellation, as well as a
performance question.

I've connected Ignite from SQL/Workbench using the Thin driver, however via
ssh -L proxy and I issued
"select count(*) from table1 where indexedField = x;" and that took 1:40
(m:ss)

"select count(*) from table1;"   seemed to take 1:20:00, based on when the
cluster went idle again.

This is using Ignite Persistence on 2.3.   The cluster is 8 x i3.8xlarge,
with the 4 local SSDs configured as a RAID 0.  These have 244MB memory
each, and 32 CPUs. The table perhaps has 500M rows at an average of 1-2KB
each.QueryParallelism is at the default of 1.  Each node has about
220GB of SSB space allocated.

Repeating this when everything was idle,I issue the select statement,
and then cancelled it using the SQL/Workbench button (which works for
MySQL).   Nothing happened, and I can see that the cluster is reading about
50MB/s from each node.   I then killed SQL/Workbench  to break the
connection, and then the ssh proxy, but the connection loss did not
terminate the query either.   What is the expected behavior when a Query is
killed via JDBC.

On the performance side, it appears that there are no optimizations around
"count(*)", and it is reading all of the data on the SSDs (50MB/s gets
close to 220GB in 1:20:00).   I could believe this is just a standard scan
problem when the data size is a bit larger than the cache, and we could
improve this by increasing query parallelism, or adding nodes, if this use
case was important.

However, node stats from visor, if I'm reading them correctly,  indicate it
is not using the memory, but I believe I enabled 160GB of off-heap space.
 How do I reconcile the visor stats with the memory settings?   All of the
nodes have been up since the data was loaded.

 Current CPU load %  | 0.03%
|

| Average CPU load %  | 0.16%
  |

| Heap memory initialized | 30gb
  |

| Heap memory used| 15gb
  |

| Heap memory committed   | 30gb
  |

| Heap memory maximum | 30gb
  |

| Non-heap memory initialized | 2mb
  |

| Non-heap memory used| 125mb
  |

| Non-heap memory committed   | 132mb
  |

| Non-heap memory maximum | 1gb




101 

102 

103

104 

105 

106 

107 

108

109 

110 

111

112 

113

114

115

116

117

118

119

120

121

122

123  

124  

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Sizing in Ignite

[vkulichenko]

What you quote is just an example emphasizing that Ignite and relational DB
are completely different storages and that mapping memory estimates one by
one are wrong. You should use the actual guide for calculations that doesn't
mention any multipliers like this.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Sizing-in-Ignite-tp12600p12895.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Sizing in Ignite

[Guillermo Ortiz]

"*- I have 300GB of data in DB. Will this be the same in Ignite?* No, data
size on disk is not a direct 1-to-1 mapping in memory. As a very rough
estimate, it can be about 2.5/3 times size on disk excluding indexes and
any other overhead."

I am getting aprox. these numbers.
1.3 = 30% because of indices


2017-05-11 0:59 GMT+02:00 Denis Magda :

> Hi,
>
> Honestly, it’s unclear why you use multipliers like 2.5 and 1.3 in the
> formula.
>
> Please refer to this capacity guide to make up a rough estimation:
> https://apacheignite.readme.io/docs/capacity-planning
>
> —
> Denis
>
> On May 10, 2017, at 8:35 AM, Guillermo Ortiz  wrote:
>
> What's the reason because it needs so many space Ignite to store data in
> memory?.
> For example, if your dataset it's about 4TB and you are going to use
> backup (1 replica) the final size it's about
>
> 5TBx2.5x1.3(indices) x backcup = 26TB more or less... So, it could be 28TB
> with the memory of the system to work.
>
> It seems insane to have that memory in many cases. How does Ignite store
> data to spend 3 times the size on disk? is it possible to reduce this?
>
>
>

Re: Sizing in Ignite

[Denis Magda]

Hi,

Honestly, it’s unclear why you use multipliers like 2.5 and 1.3 in the formula. 

Please refer to this capacity guide to make up a rough estimation:
https://apacheignite.readme.io/docs/capacity-planning 


—
Denis

> On May 10, 2017, at 8:35 AM, Guillermo Ortiz  wrote:
> 
> What's the reason because it needs so many space Ignite to store data in 
> memory?. 
> For example, if your dataset it's about 4TB and you are going to use backup 
> (1 replica) the final size it's about
> 
> 5TBx2.5x1.3(indices) x backcup = 26TB more or less... So, it could be 28TB 
> with the memory of the system to work. 
> 
> It seems insane to have that memory in many cases. How does Ignite store data 
> to spend 3 times the size on disk? is it possible to reduce this?

Re: Sizing Guide

[dkarachentsev]

Hi,

I'm not sure I understand your question, but if you need to calculate memory
consumption, this might be helpful [1].

[1] https://apacheignite.readme.io/v1.9/docs/capacity-planning

-Dmitry



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Sizing-Guide-tp12477p12538.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.