aboutsummaryrefslogtreecommitdiff
path: root/src/test.zig
AgeCommit message (Collapse)Author
2022-09-09Merge pull request #12788 from ziglang/detect-native-glibcAndrew Kelley
std.zig.system.NativeTargetInfo: improve glibc version and dynamic linker detection
2022-09-08std.zig.system.NativeTargetInfo: detection ignores self exeAndrew Kelley
Before, native glibc and dynamic linker detection attempted to use the executable's own binary if it was dynamically linked to answer both the C ABI question and the dynamic linker question. However, this could be problematic on a system that uses a RUNPATH for the compiler binary, locking it to an older glibc version, while system binaries such as /usr/bin/env use a newer glibc version. The problem is that libc.so.6 glibc version will match that of the system while the dynamic linker will match that of the compiler binary. Executables with these versions mismatching will fail to run. Therefore, this commit changes the logic to be the same regardless of whether the compiler binary is dynamically or statically linked. It inspects `/usr/bin/env` as an ELF file to find the answer to these questions, or if there is a shebang line, then it chases the referenced file recursively. If that does not provide the answer, then the function falls back to defaults. This commit also solves a TODO to remove an Allocator parameter to the detect() function.
2022-09-07tests: force LLD off for stage2 backends until auto-select deems worthyJakub Konka
2022-09-07test-cases: enable stage2 tests on WindowsJakub Konka
2022-08-19make self-hosted the default compilerAndrew Kelley
stage1 is available behind the -fstage1 flag. closes #89
2022-08-18test harness: fix handling of object formatAndrew Kelley
Follow-up to b975f7a56fec9e0e7aca9832282bc772c743d731.
2022-08-11test-cases: remove multi-threadingAndrew Kelley
This effectively reverts 22690efcc2378222503cb8aaad26a6f4a539f5aa, re-opening #11818. This had the following problems: * Buggy on some targets (macOS, Windows) * Messy output to the terminal These problems need to be solved before moving forward with this.
2022-07-27test-cases harness: annotate an optional typeAndrew Kelley
Not sure why this is needed by the CI; it's not needed locally. This is a mystery that will have to wait for another day.
2022-07-26test-cases harness: refresh just before update()Andrew Kelley
This makes it so that in a -Dsingle-threaded build of test-cases, if a crash happens, the test case name will be printed just before the stderr of the crash.
2022-07-26test-cases harness: improve stage2 compatibilityAndrew Kelley
* proper skip_stage1 mechanism that doesn't get side-stepped with manually added test cases. * avoid runtime-known function pointers. * check for type equality more simply without checking the type name.
2022-07-26test-cases harness: test all updatesAndrew Kelley
even if some are "run" on foreign hosts. closes #12193
2022-07-25std.mem: add `first` method to `SplitIterator` and `SplitBackwardsIterator`r00ster
2022-07-15std.fs: remove `OpenDirOptions.iterate`Veikka Tuominen
2022-07-15std.fs: split `Dir` into `IterableDir`Veikka Tuominen
Also adds safety check for attempting to iterate directory not opened with `iterate = true`.
2022-07-12Compilation: indent multiline error messages properlyr00ster91
Co-authored-by: Veikka Tuominen <git@vexu.eu>
2022-06-28test: return error on unknown config valueJakub Konka
2022-06-14test-cases: avoid using realpath since it is not portableAndrew Kelley
For example FreeBSD does not support this syscall.
2022-06-14test-cases: fix race with `zig run` on C backend testsAndrew Kelley
Also avoid redundantly doing compile-error checks on multiple targets for test cases where that is not helpful.
2022-06-14multi-thread `zig build test-cases`Andrew Kelley
Instead of always using std.testing.allocator, the test harness now follows the same logic as self-hosted for choosing an allocator - that is - it uses C allocator when linking libc, std.testing.allocator otherwise, and respects `-Dforce-gpa` to override the decision. I did this because I found GeneralPurposeAllocator to be prohibitively slow when doing multi-threading, even in the context of a debug build. There is now a second thread pool which is used to spawn each test case. The stage2 tests are passed the first thread pool. If it were only multi-threading the stage1 tests then we could use the same thread pool for everything. However, the problem with this strategy with stage2 is that stage2 wants to spawn tasks and then call wait() on the main thread. If we use the same thread pool for everything, we get a deadlock because all the threads end up all hanging at wait() and nothing is getting done. So we use our second thread pool to simulate a "process pool" of sorts. I spent most of the time working on this commit scratching my head trying to figure out why I was getting ETXTBSY when spawning the test cases. Turns out it's a fundamental Unix design flaw, already a known, unsolved issue by Go and Java maintainers: https://github.com/golang/go/issues/22315 https://bugs.openjdk.org/browse/JDK-8068370 With this change, the following command, executed on my laptop, went from 6m24s to 1m44s: ``` stage1/bin/zig build test-cases -fqemu -fwasmtime -Denable-llvm ``` closes #11818
2022-06-09test harness: fix handling of countsAndrew Kelley
I'm not really happy with parsing compile errors; I think we should just be checking that the expected compile error matches the actual rendered version. I will save that change for a later date however.
2022-06-08test harness: fix sort comparatorAndrew Kelley
It was returning "true" for lessThan() when the objects were in fact equal.
2022-05-26test: correctly track identical error msgs in handled errors listJakub Konka
Prior to this change, for an example compiler error test case with multiple identical errors messages such as ``` :1:2: error: foo :1:2: error: foo ``` the test harness would never increment the error index thus only marking the very first error message as handled yielding a false positive. Additionally, while here, regress `dereference_anyopaque` test case as not passing on `wasm32-wasi` target.
2022-05-15test harness: Set filename on error returnLuuk de Gram
While calling `next` an error can occur while parsing the file. However, we don't set the filename that is currently being processed, until `next` completed successfully. This means that for invalid test names, the wrong filename was being displayed in the panic message. The fix is to retrieve the correct filename when an error occurs and then setting the filename appropriately.
2022-05-13test harness: actually run the stage1 "run" testsAndrew Kelley
2022-05-13test-cases: honor -Dtest-filter argument from zig buildAndrew Kelley
2022-05-13test harness: dump stderr when compiler crashesAndrew Kelley
2022-05-13migrate runtime safety tests to the new test harnessAndrew Kelley
* migrate runtime safety tests to the new test harness - this required adding compare output / execution support for stage1 to the test harness. * rename `zig build test-stage2` to `zig build test-cases` since it now does quite a bit of stage1 testing actually. I named it this way since the main directory in the source tree associated with these tests is "test/cases/". * add some documentation for the test manifest format.
2022-05-04test: move compile errors and incremental tests into common dirJakub Konka
2022-05-04test: improve test batch/sequence iteratorJakub Konka
With this improved iterator, type of test is now inferred from the filename, enabling us to put all cases in one common parent directory, and iterate over that, thus automating a lot of tasks.
2022-05-02test: regression fix: skip stage1 if not enabledJakub Konka
2022-04-28test: remove redundant codepaths from test harnessJakub Konka
2022-04-28test: migrate stage1 compile error tests to updated test manifestJakub Konka
2022-04-28test: migrate stage2 independent compile errors to test manifest parserJakub Konka
2022-04-28test: correctly handle multiple backendsJakub Konka
To correctly handle multiple backends crossed with multiple targets, we need to push all elements in separate allocated arrays rather than operate on raw iterators. Hence, introduce `getConfigForKeyAlloc`.
2022-04-28test: enable wasm32-wasi incremental testsJakub Konka
2022-04-28test: recursively walk dir with testsJakub Konka
Prune incremental tests by moving non-incremental behavior tests to behavior test suite instead.
2022-04-28test: fix incorrect default target spec; port all incremental testsJakub Konka
2022-04-28test: leave a todo note to explicitly specify ABI for targets in the futureJakub Konka
2022-04-28test: fix pointer invalidation bug in the harnessJakub Konka
2022-04-28test: set case name from initial filename for a sequenceJakub Konka
Port more incremental tests.
2022-04-28test: unroll into multiple cases, provide default parsersJakub Konka
Provide default parsers for obvious config options such as `CrossTarget` or `Backend` (or any enum for that matter). Unroll iterator loops into multiple cases - we need to create a Cartesian product for all possibilities specified in the test manifest.
2022-04-28test: add comptime struct holding default config valuesJakub Konka
2022-04-28test: pass Strategy per directory of testsJakub Konka
2022-04-28test: parse all of manifest in TestManifest abstractionJakub Konka
2022-04-28test: abstract away test manifest parser into separate structJakub Konka
2022-04-13Use 0-indexing for incremental compile error testsCody Tapscott
2022-04-11Add file support for incremental error testsCody Tapscott
Compile error test cases can now be given as a sequence of files: - "foo.1.zig" - "foo.2.zig" - "foo.3.zig" - etc. This sequence of files is tested as incremental compilation updates to a single "foo.zig" source file. To help avoid mistakes, we enforce strict ordering for these files. "foo.zig" cannot co-exist with "foo.X.zig", the sequence must include "foo.1.zig", and no numbers may be skipped.
2022-04-11Improve whitespace-handling in (compile-error) test manifest parsingCody Tapscott
2022-03-31test harness improvementsAndrew Kelley
* `-Dskip-compile-errors` is removed; `-Dskip-stage1` is added. * Use `std.testing.allocator` instead of a new instance of GPA. - Fix the memory leaks this revealed. * Show the file name when it is not parsed correctly such as when the manifest is missing. - Better error messages when test files are not parsed correctly. * Ignore unknown files such as swap files. * Move logic from declarative file to the test harness implementation. * Move stage1 tests to stage2 tests where appropriate.
2022-03-29test harness: fix not honoring one_test_case_per_fileAndrew Kelley
I regressed this in 9aa431cba34699ae35f7905398a0c8263b2ad453. thanks @topolarity for pointing out the issue