v8_unittests pays process creation, ~6,900 static GTest registrations,
ICU, snapshot and flag parsing for every single test, because the
testrunner spawns a fresh process per test.
Add a --fork-server mode on POSIX (excluding TSan, Fuchsia, iOS) that
performs this initialization once and then forks a child per
--gtest_filter request read from stdin. Each test still runs in its own
OS process. The testrunner keeps one server per worker and variant,
falls back to direct execution on any failure, and skips the server for
Clang coverage, num_fuzzer, per-subtest seeds and --command-prefix.
Opt out with --no-fork-server or V8_NO_FORK_SERVER=1.
Full-suite wall-clock time in x64.release drops from 12.1s to 5.8s.
TAG=agy
CONV=27a1690b-7f9b-495b-8ad4-dd989d7f50bd
Change-Id: I22a7985cde2713c146360030e6f4d16425ebb8bc
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/8461671
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Hannes Payer <hpayer@chromium.org>
Cr-Commit-Position: refs/heads/main@{#110460}
Automatically skip tests that require "--foo" in configurations that
define "--no-foo", or vice versa (skip a test requiring "--no-bar" in
configurations that define "--bar").
Remove some test skippage and flag conflict declarations that become
redundant by this new logic. Forbid a flag name to start from "no", to
avoid confusing the negation stripping logic.
Note: there's some slowdown incurred by this CL, since the conflicting tests are not skipped; instead, the process is started, then the flag check fails, and the process aborts. We expect the global impact to be less than 0.3% though, because the CL only touches a small fraction of all tests.
Change-Id: I55a6eae0e936a4d8c51ed20db7787437f0c4f909
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/8146719
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Maksim Ivanov <emaxx@google.com>
Cr-Commit-Position: refs/heads/main@{#109243}
Turns out that https://crrev.com/c/7803695 was not enough: We need to
set `v8_flags.fuzzing` early (during flag parsing already) in order to
disable some checks on following flags.
This also updates the test runner to treat --fuzzing as a global
override for flag contradictions, matching the new d8 behavior.
Setting the fuzzing flag early causes d8 to allow contradictory flags
during parsing. To avoid infinite loops in the implication engine (where
one flag implies another, which in turn implies the negation of the
first), this also adds several new contradiction rules to
ResolveContradictionsWhenFuzzing. These rules explicitly break such
cycles by resetting one of the involved flags.
R=mliedtke@chromium.org
Bug: 507307609
Change-Id: I7ecf49922b519d77e8d90b67a569412cf04ee95b
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/7807588
Reviewed-by: Samuel Groß <saelo@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Reviewed-by: Matthias Liedtke <mliedtke@chromium.org>
Cr-Commit-Position: refs/heads/main@{#107017}
As we can now use --flag-processing-mode=ignore-contradictions and
because the old flag can be misused by bug finding systems that set
custom and/or random flags to cause false-positive reports.
Bug: 503869331
Change-Id: Ib0b83111aa29035a923514e4520f1d9e78068c1f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/7775553
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Samuel Groß <saelo@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#106618}
This new flag consolidates a number of existing flags that each
influenced the behavior of the flag processing logic in some way. With
the new flag, it becomes more explicit how errors should be handled
during flag processing.
Bug: 500181840, 457654443
Change-Id: I6f067697d614d3f1cf1aa7e935c749345cfe5f0e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/7761985
Commit-Queue: Samuel Groß <saelo@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#106511}
So far we always needed to specify exactly which benchmark substories
we want to report on the perf waterfall. This does not scale if we
have quickly evolving benchmarks.
Drive-by-fix:
- Add a bit more type annotations and fix a few minor type issues
- Use python3 super(). calls
Change-Id: I0c94a1d52796f3e00b8d5b414ee979f23aee6c4a
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/7722342
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Cr-Commit-Position: refs/heads/main@{#106490}
This adds the pattern to identify test dependencies coming from
import.defer calls.
Bug: 487197681
Change-Id: I37b45123fef1b370bc093dca4a00068043d77017
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_p7_rel
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/7603075
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Olivier Flückiger <olivf@chromium.org>
Commit-Queue: Caio Lima <caiolima@igalia.com>
Cr-Commit-Position: refs/heads/main@{#105420}
This is a reland of commit 0b1d664b6d.
The android file upload was fixed in PS1..PS8.
Original change's description:
> [testrunner] Use relative paths consistently
>
> Instead of making paths relative when printing (in the progress
> indicator / on failures), use relative paths already for constructing
> the command line.
> This ensures that the command line is consistent between
> - what is printed by `--verbose`,
> - what is printed by progress indicators, and
> - what is actually being executed.
>
> R=machenbach@chromium.org
>
> Change-Id: I5f2fe5c727db8cc2013673e816ce3e18e726559c
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/6781191
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Clemens Backes <clemensb@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#101685}
Change-Id: I5787e23fe5528deaa720a822c149ba54d245d369
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_p7_rel
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/6805698
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/main@{#101847}
Instead of making paths relative when printing (in the progress
indicator / on failures), use relative paths already for constructing
the command line.
This ensures that the command line is consistent between
- what is printed by `--verbose`,
- what is printed by progress indicators, and
- what is actually being executed.
R=machenbach@chromium.org
Change-Id: I5f2fe5c727db8cc2013673e816ce3e18e726559c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/6781191
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Clemens Backes <clemensb@chromium.org>
Cr-Commit-Position: refs/heads/main@{#101685}
The script that pushes test resources to android devices is not happy
when a resource turns out to be a directory. This may happen if, e.g.,
a JavaScript file contains `import('.')`; this is probably meaningless
in JavaScript but should still not fail with an exception before the
test even starts.
This CL fixes the issue by ignoring directory resources. It also
re-enables a test that had been skipped on android, for this reason.
Bug: 424836763
Bug: 422706696
Change-Id: I419513b42fc874c95b202812a0cc8d22727ba3df
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/6650906
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Nikolaos Papaspyrou <nikolaos@chromium.org>
Cr-Commit-Position: refs/heads/main@{#100883}
Some tests are susceptible to run really slow with some of the flags
passed on num-fuzz. This currently burns a lot of compute time and
prevents other, faster tests to execute more often.
This change adds a new status file state called "FUZZ_RARE" with
which such tests can be marked. On num-fuzz we add a new kind of
test filter that skips such tests with a certain probability. The
probability can be changed on the command line. As a result, such
tests will still be fuzzed, but less often. The non-determinism
introduced by this should be ok on num-fuzz, which already tests
non-deterministically (e.g. choosing random flags or flag values).
Bug: 417413668
Change-Id: Id68ab695a08190c224010241c80a47a8f62989d6
Cq-Include-Trybots: luci.v8.try:v8_numfuzz_dbg,v8_numfuzz_tsan_rel
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/6565342
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/main@{#100403}
This is a reland of commit 7c885b0383
Changed in the reland: reverted the cached_property change since
that messes with the numfuzz runner.
Original change's description:
> Fix run-tests.py behavior wrt flag contradictions
>
> Test runner behavior should be equivalent to what v8 does in
> ShouldCheckFlagContradictions. Specifically:
>
> - All contradictions after the flags --fuzzing and
> --no-abort-on-contradictory-flags are ignored.
> - A contradiction in the flag after --allow-overwriting-for-next-flag
> is ignored.
>
> Also, ensure that the test runner runs contradiction logic on all flags
> that are passed to a test. Prior to this CL, we unintentionally skipped
> a few, which caused us to miss e.g. --verify-heap contradictions.
>
> Fixed: 374991968
> Change-Id: Ia77c08ecfedfaedcafa531d5e4113e96ad62946f
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5952541
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Jakob Linke <jgruber@chromium.org>
> Auto-Submit: Jakob Linke <jgruber@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#96892}
Fixed: 374991968
Cq-Include-Trybots: luci.v8.try:v8_numfuzz_rel
Change-Id: I9a203d71b796bfa765d05a380c0a70e762098901
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5976627
Auto-Submit: Jakob Linke <jgruber@chromium.org>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Commit-Queue: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/main@{#96913}
This reverts commit 7c885b0383.
Reason for revert: Breaks numfuzz:
https://ci.chromium.org/ui/p/v8/builders/ci/V8%20NumFuzz%20-%20debug/27376/overview
Original change's description:
> Fix run-tests.py behavior wrt flag contradictions
>
> Test runner behavior should be equivalent to what v8 does in
> ShouldCheckFlagContradictions. Specifically:
>
> - All contradictions after the flags --fuzzing and
> --no-abort-on-contradictory-flags are ignored.
> - A contradiction in the flag after --allow-overwriting-for-next-flag
> is ignored.
>
> Also, ensure that the test runner runs contradiction logic on all flags
> that are passed to a test. Prior to this CL, we unintentionally skipped
> a few, which caused us to miss e.g. --verify-heap contradictions.
>
> Fixed: 374991968
> Change-Id: Ia77c08ecfedfaedcafa531d5e4113e96ad62946f
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5952541
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Jakob Linke <jgruber@chromium.org>
> Auto-Submit: Jakob Linke <jgruber@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#96892}
Change-Id: I9f155293d763fea21780996fb9b215de01760310
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5972639
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Cr-Commit-Position: refs/heads/main@{#96895}
Test runner behavior should be equivalent to what v8 does in
ShouldCheckFlagContradictions. Specifically:
- All contradictions after the flags --fuzzing and
--no-abort-on-contradictory-flags are ignored.
- A contradiction in the flag after --allow-overwriting-for-next-flag
is ignored.
Also, ensure that the test runner runs contradiction logic on all flags
that are passed to a test. Prior to this CL, we unintentionally skipped
a few, which caused us to miss e.g. --verify-heap contradictions.
Fixed: 374991968
Change-Id: Ia77c08ecfedfaedcafa531d5e4113e96ad62946f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5952541
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Jakob Linke <jgruber@chromium.org>
Auto-Submit: Jakob Linke <jgruber@chromium.org>
Cr-Commit-Position: refs/heads/main@{#96892}
import.source() dynamically imports other files, so the testing
infrastructure need to look for these calls to gather files to push to
e.g. test devices.
Bug: 365792900, 42204365
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel
Change-Id: I67fd9411ac4ac071cbad89914021b1efffd23ab1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5861022
Commit-Queue: Chengzhong Wu (legendecas) <legendecas@gmail.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#96100}
crrev.com/c/5537448 changed flag implication applications such that it
happens once after all flags are set. Unfortunately the timing of
applying implications was too late and some implications were already
needed before (specifically v8_flags.soft_abort in [1]).
We fix this issue by moving the implication application up to before any
of the flags are read.
V8::Initialize also included some hard-coded implications. To avoid
conflicts between these and regular implications, all the hard-coded
implications are replaced equivalent regular flag implications.
This CL also allows non-bool flags to appear multiple times as long
as they have the same value. The variants scripts are also updated to
not expect that such cases crash.
[1] https://source.chromium.org/chromium/chromium/src/+/main:v8/src/init/v8.cc;drc=a678f03a05eb69165d7c3d7eaea9b4358508eef0;l=242
Change-Id: I4e535cc0c1ce9a9195686d195b15e2b52eb3616f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5557771
Reviewed-by: Alexander Schulze <alexschulze@chromium.org>
Commit-Queue: Omer Katz <omerkatz@chromium.org>
Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/main@{#94142}
D8 has some specific flags that aren't available in other V8
executables, e.g. in the unit tests. This prepares using unit
tests with asan and debug, which would fail on using the d8
flag --invoke-weak-callbacks.
Bug: chromium:1500944
Change-Id: I857b07972562b69fdeba53856fb25278d8ece568
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5192466
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@google.com>
Cr-Commit-Position: refs/heads/main@{#91806}
Previously a HEAVY test was also treated like a SLOW test in all
regards. But since we can have have memory-hungry, but fast tests,
we now lift this implication.
Additionally, we ensure that heavy tests have the highest priority
in the test ordering, since they create the biggest bottleneck,
being executed in sequence. The slow tests come second, followed
by the remaining tests.
In practice, the implementation of the sequence proc just buffers
the other heavy tests, as soon as one is in the pipeline. Hence,
independent of the number of tests in each group, we should get
an overall ordering like (H: heavy, S: slow, R: remaining):
Worker 1: H, H, H, H, H, R, R
Worker 2: S, S, S, R, R, R, R,
Worker 3: S, S, R, R, R, R, R
Before this change, heavy and slow tests where treated with the
same priority, which could lead to heavy tests starting much later,
creating a bottleneck.
After this change, it's possible that some slow tests, that are also
heavy, time out. We'll just add back separate SLOW status entries
for such tests, if there are any.
Bug: v8:14373
Change-Id: I789f8b00e3d73ab975c9e4655e02a4bed24b8bd5
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/5002174
Reviewed-by: Liviu Rau <liviurau@google.com>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#90731}
This adds a probing mechanism for tests executed on desktop platforms.
The probe tracks memory statistics with psutil. The results are
added to the resultsdb entry and the json test results.
For the json test results, only the top 20 tests are tracked,
similar to duration.
When running locally or when psutil is not available, the probe will
just stay inactive. There is no significant overhead when probing
every 0.2 seconds. Very fast tests will show 0 results.
Similar to test duration, we can use the information from the json
to list the top memory-hungry tests, which are not yet marked
as HEAVY in the status file.
Such a listing will be implemented as a follow up on the infra side.
Bug: v8:14373
Change-Id: Ief162d5ef88bdd7792a953505feeb5a8f1e4fe94
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4946490
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@google.com>
Cr-Commit-Position: refs/heads/main@{#90492}
This is a reland of commit 1cfed53b01
Patchset 2 fixes a bug in the perf runner on Android, which
previously set the duration directly - now it sets start and
end time.
Original change's description:
> [test] Stream full test schedule alongside json results
>
> This writes a file in a default location alongside the json test
> results, containing all start and end timestamps of all executed
> tests.
>
> Bug: v8:14373
> Change-Id: Idb55b6cafc022814a4a6aec9590bf0cae2f111c8
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4931699
> Reviewed-by: Liviu Rau <liviurau@google.com>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#90377}
Bug: v8:14373
Change-Id: I21c042a22a52351bc084bb2ec0bd83059f04740b
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4933861
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Liviu Rau <liviurau@google.com>
Auto-Submit: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#90380}
This writes a file in a default location alongside the json test
results, containing all start and end timestamps of all executed
tests.
Bug: v8:14373
Change-Id: Idb55b6cafc022814a4a6aec9590bf0cae2f111c8
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4931699
Reviewed-by: Liviu Rau <liviurau@google.com>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#90377}
TestConfig's resolve_shell method (introduced in
https://crrev.com/c/4538761) lacks the ability to return shell path
directory in contexts such as iOS (and probably Android, among others
also). This fixes that by effectively removing resolve_shell and using
Context's platform_shell method instead. In practice, this CL brings
back iOS support for those test runners.
Bug: chromium:1132088
Change-Id: I30b1d48d834caa578ba0a34837d63eb01af89ef6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4652955
Auto-Submit: Tiago Vignatti <tvignatti@igalia.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Tiago Vignatti <tvignatti@igalia.com>
Cr-Commit-Position: refs/heads/main@{#88556}
This turns all paths in the test frameworks and all its suite
implementations from strings into pathlib paths. This makes the
framework more future-proof, simplifies a lot of code and
enhances readability. It comes with the risk of introducing new
bugs. This is lowered using an experimental test comparison
script: https://crrev.com/c/4648372.
Some detailed changes:
- The "path" of a test case is now a pathlib path (also in cases
where the path doesn't correspond to a location on the file
system). This path is always relative to the test root.
- The "name" of a test case is the posix string of that path
used in status files and at the command line.
- All other paths are absolute.
- We use some more wrapping with Path() than absolutely
necessary. Reducing such a wrapping where it's not needed is a
micro-optimization and the potential by introducing a bug by
forgetting one is worse.
- Some modules are not happy with pathlib objects, hence a few
explicit str() conversions are added.
Thoughts on risk:
- The highest risk is that due to the newly introduced logic, some
test cases are silently filtered and not run now.
We'll manually compare runs before and after this change.
- Each test attempted to run will either pass or fail,
failures won't be silent. E.g. a wrong path passed to the cctest
executable is flagged as a failure.
- Wrongly constructed paths to JS test cases lead to failures
(tested several manually).
Other failure types:
- Some string path not turned into pathlib: Will typically fail
when calling any pathlib function.
- Some pathlib object treated as a string: Will typically raise
an exception, e.g. using in-operator.
- Logic bugs in pathlib functions behaving differently than
anticipated, e.g. with_suffix(): Will typically result in wrong
paths, resulting in errors reading a test case. Though worst case
could be silent ignoring.
Cq-Include-Trybots: luci.v8.try:v8_numfuzz_dbg
Cq-Include-Trybots: luci.v8.try:v8_numfuzz_rel
Cq-Include-Trybots: luci.v8.try:v8_numfuzz_tsan_rel
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel
Cq-Include-Trybots: luci.v8.try:v8_linux64_msan_rel
Cq-Include-Trybots: luci.v8.try:v8_linux64_ubsan_rel
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel
Bug: chromium:1132088
Change-Id: Ifc94feb292174935db969dcf5d174d2eedcf3dfd
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4538761
Reviewed-by: Liviu Rau <liviurau@google.com>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#88502}
This CL adds the ability to test runners on iOS for mjsunit (using d8).
Rather than having to use an actual device (iPhone, iPad, etc), we use
the iOS Simulator for the test runners and in order to ease the job of
builders and testers that won't require a physical device.
At the moment Chromium's iossim tool is being used, which is a wrapper
around 'simctl' macOS command utility.
I've tested with variations of the following command line:
./tools/run-tests.py --progress=verbose --outdir ../out/Debug-iphonesimulator "mjsunit/delete*"
Bug: v8:13972
Change-Id: Ibe18b0da75cf3b452c507d1a05c88b8aa473ed54
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4553916
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Reviewed-by: Liviu Rau <liviurau@google.com>
Auto-Submit: Tiago Vignatti <tvignatti@igalia.com>
Cr-Commit-Position: refs/heads/main@{#88337}
Drive-by refactoring: Make it evident that currently we upload additional files only for Android platform.
Bug: v8:13686
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel
Change-Id: I8081c1185d6a92dfdcef82e697e301f3e7838dc1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4205916
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@google.com>
Cr-Commit-Position: refs/heads/main@{#85592}
This will allow on the infra side to easier link to the respective
shard on a test failure. Without that it's a hassle to find out on
which shard the failing test ran.
This also simplifies how the global test_config stores information.
Some information was duplicated, but is now rather shared through
properties if the owning object is already present.
Bug: v8:13681
Change-Id: I52f01a4fac74627575d80f25923faba99eb6a1fb
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4181030
Reviewed-by: Liviu Rau <liviurau@google.com>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#85429}
.. since many of these rules are actually relevant when some build
variable is *not* set. Instead of defining an artificial "no_foo"
variable in addition to "foo", allow definition of rules on a negative
build variable condition, e.g.:
"!is_debug": [...]
This new syntax will be used extensively in a followup CL.
Bug: v8:13629,v8:10577
Change-Id: I5ad432e71249b50d15047930e3f9143e872716d8
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4134247
Reviewed-by: Tobias Tebbi <tebbi@chromium.org>
Auto-Submit: Jakob Linke <jgruber@chromium.org>
Commit-Queue: Tobias Tebbi <tebbi@chromium.org>
Cr-Commit-Position: refs/heads/main@{#85090}
Test ids would get an uniform format between different kinds of tests:
prefix//test_id//suffix
- prefix:
- empty for regular tests
- or 'special test' token, as in 'numfuzz'
- test_id is the full name of the test as generated by test runner:
- suite_name/path/to/actual/test_name
- suffix is anything a test runner processor might want to add to the name:
- numfuzz processor will add 'analysis' of a numeric value
- variant processor will add the variant name
Bug: v8:13316
Change-Id: Ied8f958173f82d8e26c62e39ccc21167ca2928ab
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4084763
Commit-Queue: Liviu Rau <liviurau@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#84736}
Data added:
- subtest_id as it is generated by Processors
- processor_name to trace back the processors that generate subtests
- path of the test
- test_id suffixes introduced by processors:
- numfuzz will have
- an `analysis` suffix for analysis phase
- a number suffix for the variants generated after analysis
- variant processor adds a suffix for every variant
- subtests will inherit suffixes from the parent tests (origin)
V8-Recipe-Flags: resultdb
Cq-Include-Trybots: luci.v8.try:v8_numfuzz_dbg,v8_numfuzz_rel,v8_numfuzz_tsan_compile_rel,v8_numfuzz_tsan_rel
Bug: v8:13316
Change-Id: I67d8b92b575c31b201238cfbcfc76cd076a2f7af
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4081127
Commit-Queue: Liviu Rau <liviurau@google.com>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/main@{#84720}
Drive by: refactor framework_name propagation. The property was already injected in the TestSuite objects. Since it finally got attached to the result record it was natural to have it attached on the TestCase object at creation time. This eliminates the need to inject it through progress objects.
Change-Id: Ic4028d24589a241fb6225dc53ccef2215728d569
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4079228
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@google.com>
Cr-Commit-Position: refs/heads/main@{#84670}
This is a reland of commit 237de893e1
We now guard against requests Python module not being available when running the testrunner. If preconditions (modules & luci context) are not met we no longer add ResultDBIndicator to the chain.
Original change's description:
> [resultdb] Add ResultDB indicator
>
> Adds a new indicator that will send every result to ResultDB (and ultimately in a bq table; to be configured later).
>
> If we are not running in a ResultDB context we introduce only a minimal overhead by exiting early from indicator.
>
> To test these changes in a luci context with ResultDB we activated resultdb feature flag via V8-Recipe-Flags. This feature got implemented in https://crrev.com/c/3925576 .
>
>
> V8-Recipe-Flags: resultdb
> Bug: v8:13316
> Change-Id: I5d98e8f27531b536686a8d63b993313b9d6f62c5
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3905385
> Commit-Queue: Liviu Rau <liviurau@google.com>
> Reviewed-by: Alexander Schulze <alexschulze@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#83672}
V8-Recipe-Flags: resultdb
Bug: v8:13316
Change-Id: I0bdfae13cc7f250c41a18f2d3a513a3bfc580f6d
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3955263
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@google.com>
Cr-Commit-Position: refs/heads/main@{#83711}
Adds a new indicator that will send every result to ResultDB (and ultimately in a bq table; to be configured later).
If we are not running in a ResultDB context we introduce only a minimal overhead by exiting early from indicator.
To test these changes in a luci context with ResultDB we activated resultdb feature flag via V8-Recipe-Flags. This feature got implemented in https://crrev.com/c/3925576 .
V8-Recipe-Flags: resultdb
Bug: v8:13316
Change-Id: I5d98e8f27531b536686a8d63b993313b9d6f62c5
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3905385
Commit-Queue: Liviu Rau <liviurau@google.com>
Reviewed-by: Alexander Schulze <alexschulze@chromium.org>
Cr-Commit-Position: refs/heads/main@{#83672}
--stress-opt never did what we wanted it to; it ran its runs in
different contexts (therefore not able to share feedback across runs),
and even if it didn't, each run would create new closures for any
defined closures, so we'd still more than likely end up poly- or
mega-morphic.
Fuzzers cover this use case better than --stress-opt ever did, so now
it's just using precious bot time. We can get rid of it.
Bug: v8:10386
Change-Id: Ibbb9207d887b4b1dc4ec9093858d477c0f95eb37
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3803228
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Commit-Queue: Leszek Swirski <leszeks@chromium.org>
Auto-Submit: Leszek Swirski <leszeks@chromium.org>
Cr-Commit-Position: refs/heads/main@{#82722}
- Removed duplication and unnecessary indirection from all suites testcfgs.
- Introduced a more comprehensive context to cover both command context and other platform specific concerns.
- Propagated above context to TestLoader to allow for test counting command execution on all platforms.
- Wrapped original pool with another class to give it a new interface and allow injecting different implementations in the future.
- Consolidated progress indicators under a single processor in the pipeline.
- Consolidated result retention requirements calculation outside of pipeline chain.
- Refactored LoaderProc and got it under tests.
- Added some more tests for the standard runner.
- Extracted BuildConfig class.
Bug: v8:12785
Change-Id: I87be040e91f792a983662bb5a10d55b36a14ea7f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3701595
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Liviu Rau <liviurau@chromium.org>
Cr-Commit-Position: refs/heads/main@{#81770}
NumFuzz passes various flags to V8 testing randomly, which can lead to
various flag contradictions with existing flags. Up to now the system
ignored the check for contradictions and kept running the test cases,
leading to false positives.
This change adds a new v8 flag --exit-on-contradictory-flags that
exists gracefully when a contradiction is detected. On the numfuzz
side we now filter simple contradictions beforehand.
Measurements showed that ~2% of all numfuzz tests ran into
contradictions. Around half of them are simple contradictions
(repetitions and inversions), which are now filtered beforehand.
The remaining ones (redundant or contradictory implications) are
now ignored.
Bug: v8:11826
Change-Id: I9942e203ba9668a097fabe1343dd1365c9da94c1
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3650746
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Tobias Tebbi <tebbi@chromium.org>
Reviewed-by: Almothana Athamneh <almuthanna@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80589}
ShadowRealm.prototype.importValue dynamically imports other files, so
the testing infrastructure need to look for these calls to gather files
to push to e.g. test devices.
The reason to do this over explicit Resources: comment lines is to also
cover test262.
Bug: v8:12829
Cq-Include-Trybots: luci.v8.try:v8_android_arm64_n5x_rel_ng
Change-Id: I6a06933d5da849157b2c7d5fa6b7b98d39f7d39f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3606391
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Shu-yu Guo <syg@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80189}
Hannes Payer
Maksim Ivanov
Clemens Backes
Samuel Groß
Camillo Bruni
Caio Lima
Leszek Swirski
Michael Achenbach
Nikolaos Papaspyrou
Jakob Linke
Chengzhong Wu
Omer Katz
Tiago Vignatti
Liviu Rau
Liviu Rau
Shu-yu Guo
Victor Gomes
Al Muthanna Athamina