On Wed, Feb 8, 2017 at 2:26 PM, Kamil Paral <kpa...@redhat.com> wrote:
> This is what I meant - keeping item as is, but being able to pass another
> structure to the formula, which can then be used from it. I'd still like to
> keep the item to a single string, so it can be queried easily in the
> resultsdb. The item should still represent what was tested. It's just that
> I want to be able to pass arbitrary data to the formulae, without the need
> for ugly hacks like we have seen with the git commits lately.
>
>
> So, the question is now how much we want the `item` to uniquely identify
> the item under test. Currently we mostly do (rpmlint, rpmgrill) and
> sometimes don't (depcheck, because item is NVR, but the full ID is NEVRA,
> and we store arch in the results extradata section).
>
>
I still kind of believe that the `item` should be chosen with great respect
to what actually is the item under test, but it also really depends on what
you want to do with it later on. Note that the `item` is actually a
convention (yay, more water to adamw's "if we only had some awesome new
project" mill), and is not enforced in any way. I believe that there should
be firm rules (once again - conventions) on what the item is for each "well
known" item type, so you can kind-of assume that if you query for
`item=foo&type=koji_build` you are getting the results related to that
build.
As we were discussing privately with the item types (I'm not going to go
into much detail here, but for the rest of you guys - I'm contemplating
making the types more general, and using more of the 'metadata' to store
additional spefics - like replacing `type=koji_build` with `type=build,
source=koji`, or `type=build, source=brew` - on the high level, you know
that a package/build was tested, and you don't really care where it came
from, but you sometimes might care, and so there is the additional metadata
stored. We could even have more types stored for one results, or I don't
know... It's complicated), the idea behind item is that it should be a
reasonable value, that carries the "what was tested" information, and you
will use the other "extra-data" fields to provide more details (like we
kind-of want to do with arch, but we don't really..). The reason for it to
be 'reasonable value" and not "superset of all values that we have" is to
make the general querying a bit more straightforward.
> If we have structured input data, what happens to `item` for
> check_modulemd? Currently it is "namespace/module#commithash". Will it stay
> the same, and they'll just avoid parsing it because we'll also provide
> ${data.namespace}, ${data.module} and ${data.hash}? Or will the `item` be
> perhaps just "module" (and the rest will be stored as extradata)? What
> happens when we have a generic git_commit type, and the source can be an
> arbitrary service? Will have some convention to use item as
> "giturl#commithash"?
>
>
Once again - whatever makes sense as the item. For me that would be the
Repo/SHA combo, with server, repo, branch, and commit in extradata.
And it comes to "storing as much relevant metadata as possible" once again.
The thing is, that as long as stuff is predictable, it almost does not
matter what it is, and it once again points out how good of an idea is the
conventions stuff. I believe that we are now storing much less metadata in
resultsdb than we should, and it is caused mostly by the fact that
- we did not really need to use the results much so far
- it is pretty hard to pass data into libtaskotron, and querying all the
services all the time, to get the metadata, is/was deemed a bad idea - why
do it ourselves, if the consumer can get it himself. They know that it is
koji_build, so they can query koji.
There is a fine balance to be struck, IMO, so we don't end up storing "all
the data" in resultsdb. But I believe that the stuff relevant for the
result consumption should be there.
Because the ideal way would be to store the whole item+data structure as
> item in resultsdb. But that's hard to query for humans, so we want a simple
> string as an identifier.
>
This, for me, is once again about being predictable. As I said above, I
still think that `item` should be a reasonable identifier, but not
necessary a superset of all the info. That is what the extra data is for.
Talking about...
> But sometimes there can be a lot of data points which uniquely identify
> the thing under test only when you specify it all (for example what Dan
> wrote, sometimes the ID is the old NVR *plus* the new NVR). Will we want to
> somehow combine them into a single item value? We should give some
> directions how people should construct their items.
>
>
My gut feeling here would be storing the "new NVR" (the thing that actually
caused the test to be executed) as item, and adding 'old nvr' to extra
data. But I'm not that familiar with the specific usecase. To me, this
would make sense, because when you query for "this NVR related results"
you'd get the results too. If you wanted to have the specifics, then you
could narrow that search by using the relevant extra-data key-vals.
> Once, for each item :) If a task developer wants to execute his task on a
> NVR, he'll need to prepare the structured input data for each NVR he wants
> to test. That might not be difficult, but it's more work than it is
> currently, so we should know whether we're fine with that. I guess we are,
> but there can be some gotchas, e.g. sometimes it might not be obvious how
> to get some extra data. For example:
>
> nvr = htop-1.0-2.fc25
> inputdata = {name: htop; epoch: 0; build_id: 482735}
>
>
I think that you are, once again, being overly defensive. The task does not
_need_ to use/access all the data that the 'item'(type) can possible have
associated with it. And the values the task needs, the task-developer
should be able to provide. I mean - the developer knows, what he wants to
test, right? And if not, is it a good idea for him to put the test together
in the first place?
> We'll probably end up having a mix of necessary and convenience values in
> the inputdata. "name" is probably a convenience value here, so that tasks
> don't have to parse if they need to use it in a certain directive. "epoch"
> might be an important value for some test cases, and let's say we learn the
> value in trigger during scheduling investigation, so we decide to pass it
> down. But that information is not that easy to get manually. If you know
> what to do, you'll open up a particular koji page and see it. But you can
> also be clueless about how to figure it out. The same goes for build_id,
> again can be important, but also can be retrieved later, so more of a
> convenience data (saving you from writing a koji query). This is just an
> example for illustration, might not match real-world use cases.
>
>
If you are clueless about getting information necessary for your test to
run, you (IMO) should not be writing the test. Because that most probably
mean, that you do not have the necessary expertise to design the test in
the first place.
>
> Makes sense?
>
>
> I'm a bit torn between providing as much useful data as we can when
> scheduling (because a) yaml formulas are very limited and you can't do
> stuff like string parsing/splitting b) might save you a lot of work/code to
> have this data presented to you right from the start), and the easy manual
> execution (when you need to gather and provide all that data manually).
> It's probably about finding the right balance. We can't avoid having
> structured multi-data input, I don't think.
>
>
As I said already, I firmly believe that we should be providing as much
reasonable, useful data as possible. Most of the tests will not need it
all, and if they do, the maintainers/developers should know what the
possible values are anyway.
It sure can be a minor pita to put together the first "testing" datafiles,
but you only need to do it once.
Joza
_______________________________________________
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
