On Friday, November 2, 2018 at 7:48:22 PM UTC-4, Ruben Di Battista wrote:
>
> ...
>
I was asked to furtherly optimise that query and the solution that was
> found was to build up the query textually. I really hate it for a multitude
> of reasons, but well… I’m not the one making the decisions!
Hi Jonhatan,
That query inserts around 1 million of rows nowadays (in more or less 8 minutes
on remote DB — while the profiling data in this thread are on localhost — ) 2/3
times a day. This is expected to increase of a factor around 10x in next
months/year.
I'm personally not targeting any
can you elaborate on how much data is being loaded and what performance
you're targeting ?
if you're concerned with loading many MB of data as periodic batches, the
best performance by far is going to be generating a text file in one of the
formats your database server natively supports, and us
Thank you Mike.
I actually tried with the limit raised up, and the performances became actually
slightly worse even.
So the problem resides in the prepared statements logic of `mysqlclient`, that
does some additional escaping on parameters.
I will try to see if there’s a workaround on that… T
On Fri, Nov 2, 2018 at 1:08 PM Ruben Di Battista
wrote:
>
> Thanks Mike as always,
>
> I'm diving a bit more in the problem. The solution they decided to apply is
> to modify the code from above, like this:
>
> header = 'INSERT INTO `passageData` (`time`, `azimuth`, ' \
>
Thanks Mike as always,
I'm diving a bit more in the problem. The solution they decided to apply is
to modify the code from above, like this:
header = 'INSERT INTO `passageData` (`time`, `azimuth`, ' \
'`elevation`, `doppler`, `slant`, `passageID`) VALUES '
with
On Fri, Nov 2, 2018 at 11:17 AM Ruben Di Battista
wrote:
>
> Hello,
>
> I have a huge insert of the type:
>
> ```
> session.execute(
> insert_query,
> [
> {
> 'time': times[i],
> 'elevation': elevation[i],
>
Hello,
I have a huge insert of the type:
```
session.execute(
insert_query,
[
{
'time': times[i],
'elevation': elevation[i],
'azimuth': azimuth[i],
'doppler': doppler[i],
