140 testing mechanism #188
Conversation
This overwrites input fortran files which can be useful for - testing idempotency (by running tests twice) - tracking formatting changes when tests are failing (by running tests twice with different fprettify versions) Dump original Fortran files into timestamped backup directory
this bug caused 2 test cases to be indeterministic: - RosettaCodeData/Task/Cartesian-product-of-two-or-more-lists/Fortran/cartesian-product-of-two-or-more-lists.f - RosettaCodeData/Task/Evolutionary-algorithm/Fortran/evolutionary-algorithm-1.f
and remove lengthy tests from regular testing
Pull Request Test Coverage Report for Build 19276268481Details
💛 - Coveralls |
run_tests.py
Outdated
| parser.add_argument("-n", "--name", type=str, help="select tests by name (sections in testsuites.config).") | ||
|
|
||
| suite_default = ["unittests", "builtin"] | ||
| parser.add_argument("-s", "--suite", type=str, default=suite_default, help="select suite.", choices=["unittests", "builtin", "regular", "cron", "custom"], action='append') | ||
|
|
||
| args = parser.parse_args() | ||
|
|
||
| suite = unittest.TestLoader().loadTestsFromTestCase(FPrettifyTestCase) | ||
| unittest.TextTestRunner(verbosity=2).run(suite) | ||
| # remove defaults if user has specified anything | ||
| if args.suite[:2] == suite_default and len(args.suite) > 2: | ||
| args.suite = args.suite[2:] |
There was a problem hiding this comment.
I am not sure I completely understand what you are trying to achieve here, but it seems a bit like a hack, wouldn't this be equivalent?
parser.add_argument(
"-s", "--suite",
nargs="+",
default=["unittests", "builtin"],
choices=["unittests", "builtin", "regular", "cron", "custom"],
help="select suite."
)There was a problem hiding this comment.
You're right, that's more direct, I'll implement your suggestion.
README.md
Outdated
| 1. one or more unit tests are added which test formatting of small Fortran code | ||
| snippets, covering all relevant aspects of the added features. | ||
| 2. if the changes lead to failures of existing tests, these test failures | ||
| should be carefully examinated. Only if the test failures are due to |
There was a problem hiding this comment.
'examinated' -> 'examined'
README.md
Outdated
| in = "Some Fortran code" | ||
| out = "Same Fortran code after fprettify formatting" | ||
|
|
||
| # seleced fprettify command line arguments, as documented in "fprettify.py -h": |
| else | ||
| l = 1 | ||
| endif | ||
| end if |
There was a problem hiding this comment.
I didn't check the actual output, but should more than the case change? There are more below.
There was a problem hiding this comment.
this test case uses default formatting, and on top of that it lowercases keywords, as specified by a special annotation at the top of the file:
)This is in line with how all unit tests are written - all options are tested on top of default options.
FYI: this file has changed because only with the changes in this pr, we are able to test files with custom options, see also my comment #49 (comment)
There was a problem hiding this comment.
That's what I was thinking has happened. Sorry, I should have added that I just wanted to double check that's the reason. I was surprised the splitting of the endif/end if would only show up there.
max-models
left a comment
max-models
left a comment
There was a problem hiding this comment.
I couldn't find any breaking errors. Here and there I saw some points on the python code style which I would have done differently, but I don't think it's worth quibbling over. The PR looks good to me now :)
| output. | ||
| """ | ||
| if cls.n_parsefail + cls.n_internalfail > 0: | ||
| format = "{:<20}{:<6}" |
There was a problem hiding this comment.
You are overwriting the built-in function format() with a local variable named format, maybe it would be better if you renamed the variable to fmt or format_string or so.
4 participants
I finally found time to wrap up refactoring of the testing mechanism, so that it's better maintainable and so that results are more transparent. The new mechanism is documented in
README.md. This fixes #44, fixes #140 and fixes #134.