This is a classic queue pattern: Make a task list of the items to be
processed (and perhaps breaking up multiple tasks for each item), and run
multiple workers against the list, updating the status as they go. As
Christof says, ultimately you have to LOCK, UPDATE, UNLOCK and think about
REFRESH.
64,000 records should take seconds, except for the slow parts. Focus on
those: is it a slow API? Is it setup and takedown that you could cache?
Think about the idea of separating the slow parts from the fast parts, so
your queue might have a slow part 1 and fast part 2 to be completed by
different executables. Then you could launch a dozen EXEs to process the
slow parts, improving the throughput, and a few EXEs to do the fast parts.
On Sat, Aug 1, 2020 at 1:07 PM MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:
> I've got a regular process that runs, basically using key information to
> grab data from an API and then update the local VFP database. There are
> maybe 64000 records to process, and each record to update through this
> process takes about a second, so to process this group would take over
> 17 hours. Each record could be processed on its own; there are no
> relationships between each.
>
> I don't want to start it and run the 64000 in a row for 17+ hours. I
> would like to design the app to use the table, RLOCK the row I'm
> processing, and the UNLOCK the row when I'm done. I figure with this
> design, I could run multiple instances of the MyProgram.exe (similar to
> how WestWind Web Connection allows you to run multiple instances) to
> process the batch maybe 4x faster (if I launched 4 instances of
> MyProgram.exe). The basic construct would be as follows:
>
> USE ListOfRecsToProcess IN 0 SHARED Alias MyList && record is PK (to
> process) i, tProcessed t, tError t, cSession c(10)
> SCAN FOR EMPTY(cSession) AND RLOCK('MyList')
> IF ProcessRecord(MyList.ID) THEN
> REPLACE tProcessed WITH DATETIME(), cSession WITH this.cSession
> IN MyList
> ELSE
> REPLACE tError WITH DATETIME(), cSession WITH this.cSession IN
> MyList
> ENDIF
> ENDSCAN
>
>
> Does anybody see any problems with that general design? The
> ProcessRecord method calls an API to get values and then updates the
> local VFP record accordingly.
>
> tia,
> --Mike
>
>
>
> --
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
[excessive quoting removed by server]
_______________________________________________
Post Messages to: ProFox@leafe.com
Subscription Maintenance: https://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: https://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: https://leafe.com/archives
This message:
https://leafe.com/archives/byMID/cacw6n4tns0vjmhw8idlkrld7p0mkv8e5h0t2wrwypgb3vh1...@mail.gmail.com
** All postings, unless explicitly stated otherwise, are the opinions of the
author, and do not constitute legal or medical advice. This statement is added
to the messages for those lawyers who are too stupid to see the obvious.
