Hi, Dren!
Unfortunatelly, pyignite doesn't have an efficient native serialization
library, whereas psycopg2 has (it is a thin wrapper around libpq).
I would suggest two options:
1. Reduce a default batch size like this : `client.sql("SELECT * FROM
TABLE", page_size=10)`. Default 1024 seems too big and parsing of such a
big response seems to be really slow.
2. Use ignite odbc driver and pyodbc over it. Both of them work pretty well.
вт, 14 мар. 2023 г. в 14:10, Dren Butković <dren.butko...@gmail.com>:
>
> Ignite and py client versions:
>
> - Apache Ignite 2.13.0
> - pyignite 0.5.2
>
> On Tue, Mar 14, 2023 at 11:46 AM Zhenya Stanilovsky via user <
> user@ignite.apache.org> wrote:
>
>> Hi, plz append ignite and py client versions.
>>
>>
>> Hi,
>>
>> I made a speed comparison of retrieving data from Apache Ignite using
>> several methods. All records are in one table, I did not use any WHERE
>> condition, only a SELECT * FROM TABLE XYZ LIMIT 20000.
>>
>> Test results are:
>> Apache Ignite
>>
>> - Apache Ignite REST API - 0.52 seconds
>> - JDBC - 4 seconds
>> - Python pyignite - 40 seconds !!!
>>
>> pseudocode in Python using pyignite:
>>
>> client = Client(username="ignite", password="pass", use_ssl=False)
>> client.connect('localhost', 10800)
>>
>> cursor=client.sql('SELECT * FROM TABLE_XYZ LIMIT 20000')for row in cursor:
>> pass
>>
>> After that I made a speed comparison of retrieving data from PostgreSQL
>> using JDBC and psycopg2 Python package. SQL select is same, SELECT * FROM
>> TABLE XYZ LIMIT 20000
>> PostgreSQL
>>
>> - JDBC - 3 seconds
>> - Python psycopg2 using fetchall - 3 seconds
>> - Python psycopg2 using fetchone - 4 seconds
>>
>> pseudocode in Python using psycopg2:
>>
>> import psycopg2
>>
>> conn = psycopg2.connect(database=DB_NAME,
>> user=DB_USER,
>> password=DB_PASS,
>> host=DB_HOST,
>> port=DB_PORT)
>>
>> cur = conn.cursor()
>> cur.execute("SELECT * FROM TABLE_XYZ LIMIT 20000")
>> rows = cur.fetchall()for data in rows:
>> pass
>>
>> I can conclude that the pyignite implementation has much worse
>> performance compared to psycopg2 tests. The performance difference on
>> PostgreSQL between Java JDBC and Python psycopg2 is negligible.
>>
>> The performance difference on Apache Ignite between Java JDBC and Python
>> pyignite is very big.
>>
>> Please if someone can comment on the tests, did I do something wrong or
>> are these results expected? How can such large differences in execution
>> times be explained? Do you have any suggestions to get better results using
>> pyignite?
>>
>> Thank you
>>
>>
>>
>>
>>
>>
>
--
Sincerely yours, Ivan Daschinskiy
