Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
E.Boer, nice to know you've been able to get such thoughput, which seems much more reasonable. > You don't happen to have a version of your script that could work with > oerp6.1->tryton4.4? > We are migrating from openerp v5 to tryton 4.0. Yes we are working on it for a long time, but we want to do it right. After migration we are going to update to Tryton 4.4. I can send you parts of the script so you can have an idea how we did it, but just to mention that it's very specific for our needs. The main thing is how we connect to the openerp database and connect to Tryton. So please let me know if you want parts of the script. -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/108ecc08-7e6c-401a-a00a-398bf91e6a92%40googlegroups.com.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
Le 21/07/2017 à 15:59, Sergi Almacellas Abellana a écrit : If you are using proteus 4.4, it's already doing one single call to the server using the create and the list of all lines. IIUC, you are creating a move with 15k lines. So I'm wondering if increasing the record cache size [1] to a number higher than 15k for the migration will speedup things a little bit. Of course, this will increase the memory usage, so just take care you don't get out of memory. Hope it helps. [1] http://doc.tryton.org/4.4/trytond/doc/topics/configuration.html?highlight=cache#record This was a nice tip, and it allowed me to get much much further. Thanks. In the end, once I created a tryton move per opererp move, it seemed I could live without modifying the cache and get 5 1/2 years in under a couple of hours (fiscal year by fiscal year). -- Richard PALO "Ne croyez rien, peu importe où vous l'avez lu, ou qui l'a dit, à moins que cela ne s'accorde avec votre raison." -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/381c0db4-21a7-fc36-76a8-43ce90b6037f%40free.fr.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
Le 21/07/2017 à 00:42, Cédric Krier a écrit : On 2017-07-20 14:00, E. Boer wrote: Like Cedric already said, Proteus is not ment to be used for a huge amount of data. You can create a script like explained in the (very) old wiki: http://hg.tryton.org/deprecated/tryton.wiki/file/tip/RemoteCalls.wiki Indeed such call is doable with proteus by using the raw function of the Model like Model.create or Model.write. E.Boer, nice to know you've been able to get such thoughput, which seems much more reasonable. You don't happen to have a version of your script that could work with oerp6.1->tryton4.4? Cedric, since I'm already in proteus, the Model.write() seems quite worthwhile to try out first. Would it be possible to have a very simple example snippet with a move record including a number of move lines such that I can better understand the syntax of what is expected? as to .save(), it appears to seriously suffer, even exponentially, with large numbers of move lines. for info, I placed the move.save() inside the loop to see via the database how things advanced with 15K lines (having already split by fiscal year), and it would do a couple hundred relatively fast, but a thousand was slowing down fast, and many hours later only a total of roughly 2500 move lines were saved so extrapolating 15K is probably close to infinity,timewise. *I presume* multiple moves would help already greatly (but I haven't had a chance yet to try it). cheers -- Richard PALO -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/75002dd3-50bf-1045-c4b7-3d10bd5ccbf9%40free.fr.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
On 2017-07-20 14:00, E. Boer wrote: > Like Cedric already said, Proteus is not ment to be used for a huge amount > of data. You can create a script like explained in the (very) old wiki: > http://hg.tryton.org/deprecated/tryton.wiki/file/tip/RemoteCalls.wiki Indeed such call is doable with proteus by using the raw function of the Model like Model.create or Model.write. -- Cédric Krier - B2CK SPRL Email/Jabber: cedric.kr...@b2ck.com Tel: +32 472 54 46 59 Website: http://www.b2ck.com/ -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/20170720224252.GH30717%40kei.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
On Thursday, 20 July 2017 17:32:00 UTC+2, Richard PALO wrote: > > A bit more info now, seems the French Account plan has some issues where > some > accounts have "party_required == true" (such as 421, 4387, 4687, 486) > but these don't necessarily have parties identified. > You are running into the same problems as we had because OpenERP is very forgiving when you put garbage in. Tryton is very strict which we realy like because it keeps the data"clean". I think you will also run into trouble when migration sales and invoices because maybe you added the wrong taxes or a wrong account to the invoice or sale order. > Can't directly explain why it takes so long before throwing this error, but > oh well, > it may be because there are between 15K and 25K move lines per fiscal > year. So far I split > by year, but might as well go for year/period/journal/move why I'm at it > (for a full, deep migration). > Like Cedric already said, Proteus is not ment to be used for a huge amount of data. You can create a script like explained in the (very) old wiki: http://hg.tryton.org/deprecated/tryton.wiki/file/tip/RemoteCalls.wiki -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/6ea44d65-faba-484b-8103-bda8378cd9ff%40googlegroups.com.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
> > > >> when I say long, it is already over 5 hours! > > > > Yep, proteus is not specially designed for performance. > > > yikes, guess I'll need to try breaking things down a bit differently, > that is instead of putting all move lines in a monster 'migration' move. > We ran into the same problem and created a client script which talks over JSON-RPC with the tryton-server. We are now migrating 300.000 account_move_lines within 1:45 hour. It's a bit more work to get it running. Another option is to create a migration_module which does the job. -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/3f61059d-ba38-49b6-a178-5f2e275d7b83%40googlegroups.com.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
Le 18/07/2017 à 13:27, Cédric Krier a écrit : On 2017-07-18 13:01, Richard PALO wrote: I'm looking for hints as to how to prod move.save() to complete or at least to give some info as to why it isn't. It appears to be cpu bound (indeed, the openerp base is quite large). move.save is at the end of the following, from line 1065..1070 in migrate_account_balance move.lines.append(new_line) new_line.account = get_account(line.code, code2account) if new_line.account.party_required: new_line.party = code2party[str(line.partner_id)] move.save() How do you know it is the problem? I added print statements to trace ensuring that the 'for line in cur' loop was completed. ... I do not think it could be a problem, I guess it just show you the last query run by the session. yeah, seems probably the case. when I say long, it is already over 5 hours! Yep, proteus is not specially designed for performance. yikes, guess I'll need to try breaking things down a bit differently, that is instead of putting all move lines in a monster 'migration' move. cheers, -- Richard PALO -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/e4ac7cf6-ad50-7def-8c43-1f520d28bcb9%40free.fr.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
On 2017-07-18 13:27, Cédric Krier wrote: > On 2017-07-18 13:01, Richard PALO wrote: > > I'm looking for hints as to how to prod move.save() > > to complete or at least to give some info as to why it isn't. > > > > It appears to be cpu bound (indeed, the openerp base is quite large). It will be good to have an estimation of the accounts, partner. Because the query for the balance could return a huge number of lines and it may not fit well in the memory of the host. In this case, it may be good to create the move without lines and create line per small bunch. -- Cédric Krier - B2CK SPRL Email/Jabber: cedric.kr...@b2ck.com Tel: +32 472 54 46 59 Website: http://www.b2ck.com/ -- You received this message because you are subscribed to the Google Groups "tryton" group. To view this discussion on the web visit https://groups.google.com/d/msgid/tryton/20170718113041.GR30717%40kei.
Re: [tryton] openerp2tryton : painfully long in migrate_account_balance
On 2017-07-18 13:01, Richard PALO wrote:
> I'm looking for hints as to how to prod move.save()
> to complete or at least to give some info as to why it isn't.
>
> It appears to be cpu bound (indeed, the openerp base is quite large).
>
> move.save is at the end of the following, from line 1065..1070 in
> migrate_account_balance
> > move.lines.append(new_line)
> > new_line.account = get_account(line.code, code2account)
> > if new_line.account.party_required:
> > new_line.party = code2party[str(line.partner_id)]
> >
> > move.save()
How do you know it is the problem?
> phppgadmin seems to believe that the following is cycling:
> > SELECT "a"."rounding" AS "rounding", "a"."active" AS "active", "a"."code"
> > AS "code", "a"."create_date" AS "create_date", "a"."create_uid" AS
> > "create_uid", "a"."digits" AS "digits", "a"."id" AS "id",
> > "a"."mon_decimal_point" AS "mon_decimal_point", "a"."mon_grouping" AS
> > "mon_grouping", "a"."mon_thousands_sep" AS "mon_thousands_sep",
> > "a"."n_cs_precedes" AS "n_cs_precedes", "a"."n_sep_by_space" AS
> > "n_sep_by_space", "a"."n_sign_posn" AS "n_sign_posn", "a"."negative_sign"
> > AS "negative_sign", "a"."numeric_code" AS "numeric_code",
> > "a"."p_cs_precedes" AS "p_cs_precedes", "a"."p_sep_by_space" AS
> > "p_sep_by_space", "a"."p_sign_posn" AS "p_sign_posn", "a"."positive_sign"
> > AS "positive_sign", "a"."symbol" AS "symbol", "a"."write_date" AS
> > "write_date", "a"."write_uid" AS "write_uid" FROM "currency_currency" AS
> > "a" WHERE (("a"."id" IN (44)))
I do not think it could be a problem, I guess it just show you the last
query run by the session.
> when I say long, it is already over 5 hours!
Yep, proteus is not specially designed for performance.
--
Cédric Krier - B2CK SPRL
Email/Jabber: cedric.kr...@b2ck.com
Tel: +32 472 54 46 59
Website: http://www.b2ck.com/
--
You received this message because you are subscribed to the Google Groups
"tryton" group.
To view this discussion on the web visit
https://groups.google.com/d/msgid/tryton/20170718112702.GQ30717%40kei.
[tryton] openerp2tryton : painfully long in migrate_account_balance
I'm looking for hints as to how to prod move.save()
to complete or at least to give some info as to why it isn't.
It appears to be cpu bound (indeed, the openerp base is quite large).
move.save is at the end of the following, from line 1065..1070 in
migrate_account_balance
move.lines.append(new_line)
new_line.account = get_account(line.code, code2account)
if new_line.account.party_required:
new_line.party = code2party[str(line.partner_id)]
move.save()
phppgadmin seems to believe that the following is cycling:
SELECT "a"."rounding" AS "rounding", "a"."active" AS "active", "a"."code" AS "code", "a"."create_date" AS "create_date", "a"."create_uid" AS "create_uid", "a"."digits" AS "digits", "a"."id" AS "id", "a"."mon_decimal_point" AS "mon_decimal_point", "a"."mon_grouping" AS "mon_grouping", "a"."mon_thousands_sep" AS "mon_thousands_sep", "a"."n_cs_precedes" AS "n_cs_precedes", "a"."n_sep_by_space" AS "n_sep_by_space",
"a"."n_sign_posn" AS "n_sign_posn", "a"."negative_sign" AS "negative_sign", "a"."numeric_code" AS "numeric_code", "a"."p_cs_precedes" AS "p_cs_precedes", "a"."p_sep_by_space" AS "p_sep_by_space", "a"."p_sign_posn" AS "p_sign_posn", "a"."positive_sign" AS "positive_sign", "a"."symbol" AS "symbol", "a"."write_date" AS "write_date", "a"."write_uid" AS "write_uid" FROM "currency_currency" AS "a" WHERE (("a"."id" IN (44)))
when I say long, it is already over 5 hours!
--
Richard PALO
--
You received this message because you are subscribed to the Google Groups
"tryton" group.
To view this discussion on the web visit
https://groups.google.com/d/msgid/tryton/df654e4b-8312-5929-0123-7939fe5cd06f%40free.fr.
