On Tue, Dec 6, 2016 at 4:54 PM Ole Tange <[email protected]> wrote:
> On Tue, Dec 6, 2016 at 6:10 PM, Andy Loftus <[email protected]> wrote:
>
> > Currently, sqlmaster appears to populate a table, first dropping any
> > existing table, and waits for jobs to complete. Wondering if there is a
> way
> > to:
> >
> > 1. populate the database then exit immediately without waiting?
> > My particular use case is parallelizing backup tasks that are expected to
> > run a long time (several hours on average).
>
> Hmmm... maybe we should change, so you need to add '--wait' if you
> want '--sqlmaster' to wait. Seems like a reasonable change.
Even better! That was my first assumption about how it worked. Took me a
little bit of time to figure out why --sqlmaster never exited, haha :)
>
> On a related note, why does the sqlworker command require the exact same
> > input as the sqlmaster? Shouldn't it be sufficient that all necessary
> > information is stored in the database and then the sqlworker can just
> pull
> > tasks from the database?
>
> --sqlmaster only inserts the values - not the command. The problem
> comes with replacement strings like {%}. You will never know in
> advance which job slot a job will be run as:
>
> parallel echo {%} ::: {a..j}
>
> So --sqlmaster cannot store the actual command to run.
>
> It _could_ be changed so that --sqlmaster stores the template command
> into the command column, and --sqlworker fetches it form here, and
> replaces the command column with the actual command run when done.
>
> It would, however, be a fairly big change of GNU Parallel: The
> assumption has always been that the template command remains the same
> for the whole run, and quite a bit of optimization depends on this.
>
> On a similar note: What would you expect the table should look like
> when you run:
>
> # These do not work - but what would you expect them to do to the table?
> parallel --sqlandworker $DBURL -X echo {%}: {} ::: {1..10}
> parallel --sqlandworker $DBURL -N3 echo {%}: {} ::: {1..10}
> parallel --sqlandworker $DBURL echo {%} '{= $_=total_jobs() =}' ::: {1..10}
>
In general, I think the best approach is "keep it simple" and try to retain
as much flexibility as possible by putting the template in the command
column. Some things could be replaced, such as {#}, but contexts that
don't make sense until runtime, like {%}, would have to be inserted
verbatim. Regarding the dynamic calculation of "total_jobs", several
options come to mind. First is to leave it dynamic and each job run by a
sqlworker will calculate it at runtime, perhaps by looking up the number of
rows in the table. If appending new jobs is allowed, the number will
change based on the time each job was run relative to when appends were
made to the table. I think that's okay, if it's documented that it works
that way. The other approach would be to auto-fill dynamic content at
sqlmaster insert time; then "total_jobs" would be fixed at the number of
jobs that were added by sqlmaster. I like the first approach better (leave
as much stuff figured at runtime as makes sense.) In the end, I think
sqlmaster and sqlworker are very well suited for some use cases but not for
others, so if it works for the majority of cases where it makes sense,
that's a good place to start. As your examples point out, they don't work,
but do they need to? It's hard to tell without a good use case, but it
seems to me that these examples (especially #2 and #3) are less well suited
to usage by sqlmaster and sqlworker. Here's my thoughts of what the
example commands might look like in the database after sqlmaster put them
there but before a sqlworker executed anything:
# These do not work - but what would you expect them to do to the table?
parallel --sqlandworker $DBURL -X echo {%}: {} ::: {1..10}
Seq|...|Exitval|...|Command |V1|...
1| | -1000| |echo {%}: {}| 1|
2| | -1000| |echo {%}: {}| 2|
3| | -1000| |echo {%}: {}| 3|
...etc
10| | -1000| |echo {%}: {}|10|
parallel --sqlandworker $DBURL -N3 echo {%}: {} ::: {1..10}
Seq|...|Exitval|...|Command |V1|V2|V3|...
1| | -1000| |echo {%}: {}| 1| 2| 3|
2| | -1000| |echo {%}: {}| 4| 5| 6|
3| | -1000| |echo {%}: {}| 7| 8| 9|
4| | -1000| |echo {%}: {}|10| | |
OR
4| | -1000| |echo {%}: {}|10| NULL | NULL |
OR
Seq|...|Exitval|...|Command | V1|...
1| | -1000| |echo {%}: {}| 1 2 3|
2| | -1000| |echo {%}: {}| 4 5 6|
3| | -1000| |echo {%}: {}| 7 8 9|
4| | -1000| |echo {%}: {}|10 |
parallel --sqlandworker $DBURL echo {%} '{= $_=total_jobs() =}' ::: {1..10}
Seq|...|Exitval|...|Command |V1|...
1| | -1000| |echo {%} '{= $_=total_jobs() =} | 1|
2| | -1000| |echo {%} '{= $_=total_jobs() =} | 2|
3| | -1000| |echo {%} '{= $_=total_jobs() =} | 3|
...etc
10| | -1000| |echo {%} '{= $_=total_jobs() =} |10|
As mentioned earlier, let the sqlworker calculate the "total_jobs" at
runtime. It may not make sense, but if it works consistently and is
documented, that should be sufficient.
> > 2. append new tasks to an existing database?
> > I think this is more likely a feature request since the man page
> > specifically says table will be clobbered. As I understand, the reason is
> > that the table schema must/should match, especially the V* columns. But,
> > really, isn't that ultimately a burden for the user (as opposed to the
> > developer)?
>
> I struggled with this decision, too. My reasoning was, that if you run:
>
> parallel --sqlandmaster $DBURL echo ::: {1..3}
>
> but really meant:
>
> parallel --sqlandmaster $DBURL echo ::: {1..3} ::: {4..6}
>
> then you would have to find a way to clean the database first.
>
That sounds entirely like user error to me and the user can then decide how
best to proceed (ie: either sanitize the DB table or drop and start over.)
As a matter of fact, that's actually more flexible, and what I,
personally, would expect anyway. I would expect the tool to fail if the DB
table already exists and/or has a mis-matched schema unless I specified
that it's okay to clobber an existing table.
>
> > Perhaps a specific flag allowing "append" operation so user can
> > be duly warned and could still check that the number of V* columns
> matches.
>
> Like having the DBURL start with '+': +pg://tange:mypass/tange/TBL8007
>
> I believe this part is relatively simple to do - especially if we
> allow it to die horribly if the columns do not match.
>
> It should:
>
> * Not drop table
> * Find the max seq-number, and continue from there
>
> > A sample use case is executing a bunch of generated bash scripts (per
> above,
> > this is how the parallel backups are handled), so the V1 column is the
> > absolute path to the script and the command is simply "bash".
>
> I can definitely see it being useful, and I believe the easy changes
> would accommodate this:
>
> # Append backup-script* to the queue in $DBURL. Exit immediately (no
> --wait).
> parallel --sqlmaster +$DBURL bash ::: backup-script*
> # or even:
> chmod 755 backup-script*
> parallel --sqlmaster +$DBURL ::: backup-script*
>
> # Do the work by grapping arguments from $DBURL
> parallel --sqlworker $DBURL bash
> # or even:
> PATH=.:$PATH
> parallel --sqlworker $DBURL
>
>
> /Ole
>
