I want to use this for a few other longstanding known failures.
This patch:
- Changes the variable names from disable demo to run xfail, for the
sake of being general and avoiding double negatives everywhere.
- Changes the way the check is made to be consistent with SHOW_BROWSER
- Documents the variable in tests/README.md
Use the old wire protocol with ChromeDriver (set w3c to false). This fixes
some Chrome misbehaviors, including failing to switch browser windows. Tested
with Chromium 80.
I expect this to change the failure mode for `tests/account-settings` from
failing at line 24 of uploadMeteorTestApp.js to possibly failing somewhere else
(later in the test flow) instead.
...by implementing the proper fix for the race condition described in
As expected, this shaves about 5 seconds off of each of these tests, or
about 15 seconds off of the test suite as a whole.
Note that there are two other calls to pause() in there; these are a bit
harder to eliminate, since the events they're waiting for are triggered
by things outside of any http request, so we can't easily create an
element on the page and do waitForVisible or the like, and as far as I
can tell nightwatch doesn't give us an easy way to say "wait until this
element contains this text," which is what we really want here.
When navigating to /apps, it's possible that the introjs overlay will make the
.app-list element not "visible" (because the overlay is blocking the entirety
of the page).
It's also possible that we'll win that race, but then once we navigate to the
grain list page, we'll lose a race against a different introjs overlay.
Additionally, the introjs overlay does not appear to get reactively removed
when we mark that it's no longer needed in the Session. So we need to catch
things early.
Making the call to `disableGuidedTour()` *before* navigating to any of the
pages that feature introjs helpers should help ensure that they never render
and never cause test failures.
It's possible we should rethink how we manage introjs and testing -- perhaps we
should disable the overlays earlier for testing purpose, like in user login.
There's only two tests that actually bother interacting with the introjs
overlays, and perhaps they could use a different login method (or pass a
parameter to decline disabling all introjs overlays).
I believe this patch may fix the flakiness around `apps/collections` that we're
seeing in the checks for #3293.
@zarvox pointed out that there's a race condition where this happens
before the job is actually inserted into the database.
Strictly speaking, inserting a .pause here doesn't actually fix the race
condition, but (a) the test is using this technique in several other
places, and (b) If it takes 5 seconds for the job to hit the DB
something strange is going on.
It would be nice to re-work the test to avoid these pauses, if only so
that it will run faster, but if the race condition is the cause of the
issues we're seeing, this should at least get things passing again.
The tests that use this are broken anyway (#3188) and fixing them
probably means moving to a different testing framework. Meanwhile,
spacejam pulls in phantomjs, which is unmaintained and seems to be
broken on node 12, so this breaks the whole CI system.
Hopefully fixes#3277
Same issue as #3227. This isn't actually broken currently, since there's
only one test in the file, but presumably we would hit the same problem
if we added another test.
Fails currently; for some reason the grain log never opens. Need to
investigate, maybe the "you've created your first grain" modal is
blocking the click?
There's no real need to actually mock Date.now(); we're already passing
the date in most of the way. This is simpler and it also solves the
problem we were having with sandstorm rejecting the one-shot jobs
because they were "in the past."
The one hitch is that the test for one-shot jobs only works if
the server has been restarted since the last time
advanceTimeMillis() was called. Otherwise, sandstorm rejects the app's
requested date because it is in the past relative to the mocked time,
since the C++ test app isn't using our mock. That still needs fixing.
Maybe we should just be more lenient? Without the check if an app picks
a time slightly in the past, it will just get run as soon as possible,
which seems like reasonable behavior.
The ancient version of Selenium we were using no longer works with the version of Firefox ESR on Debian Sid. Newer Seleniums no longer build-in Firefox support but rather require separately-installed drivers. Nightwatch also needed to be updated to work with newer Selenium. I switched to installing Selenium via npm as this seems more convenient.
For Firefox, the driver is `geckodriver`, but Nightwatch has a bug where its switchWindow() function doesn't work with `geckodriver`. The latest beta version of Nightwatch fixes this bug but seems to intorduce some other bugs. Instead, I switched to running the tests in Chrome.
Various tests needed updates to work correctly on the new version.
In some cases, like old-sharing-model grains, we use the identicon ID as the user's identity ID. But identity IDs must be 32 bytes / 64 hex digits, even though identicons themselves only use the first 16 bytes / 32 digits to generate the icon.
Ian Denhardt
Kenton Varda
Jim Garrison
Jacob Weisz
Troy J. Farrell
Drew Fisher
Kenton Varda