TAP conventions for aurweb’s test suites

[Frédéric Mangano-Tarumi]

Lukas Fleischer [2020-02-22 09:58:15 +0100]
> The current tests mostly execute commands and compare their outputs, and
> I don't see much benefit in rewriting them in Python. That being said, I
> think we could benefit from additional unit tests.

Exactly. Using .t instead of .sh is about opening doors. Rewriting
existing tests would have no benefit per se. The beauty of TAP’s
language-agnosticness is the liberty to use to right tool for the right
job. I’ll probably bring up testing in Python specifically soon.

> I like the idea of being able to use prove but I'd prefer still having a
> way to run tests without having prove installed. How about adding a
> "prove" target to the Makefile instead of changing `make check`?

In Arch Linux, prove is part of the perl package, which is a direct
dependency of git and openssl. Even though Perl is being less and less
used these days, I don’t think you’ll ever find a system without Perl
installed. Therefore, I personally don’t think it’s worth checking
whether prove is installed or not.

If we do want to support systems without prove anyway, I suggest we find
a way to let make detect if prove is installed, and depending on the
result change the behavior of make check. `make prove` has no utility
as someone who knows about prove would call it directly instead.

> > 4. Delete the `make t1234-xxx.sh` rules.
> > 5. Delete the whole Makefile altogether.
> 
> Why? Those are small, super low maintenance, and provide the benefit of
> being able to easily run the test suite without prove.

`make something.sh` should generate or update something.sh, and not run
it. This is counter-intuitive to the point of being harmful, because it
teaches bad practice.

Usually, when a project uses make, it provides a Makefile at the root of
the project, with many rules including check, and is run from the root
of the project. test/Makefile is like hidden and lonely. I wouldn’t mind
keeping make check, but I think most developers would rather know they
can type “prove a.t b.t” to run specific tests and have nice reports.
`make check` is simpler for users, but prove is better for developers,
and given the nature of aurweb we should target the latter.

aurweb could benefit from a Makefile if it had rules like “make
test-env” to automatically execute the steps in TESTING, “make run” to
start the PHP test server, etc. To exaggerate a bit, having the
developer manually set up their database and then come up with “hey we
have a hidden make check rule to save you from typing prove” somewhat
feels weird.

> I agree with sticking to TAP conventions but it should be easy to keep
> almost everything working as-is at the same time without any extra work.

Don’t worry! I’ll send patches progressively and you’ll tell me if
something feels too disruptive.