This fixes the original Python wrapper. So basically, the Python developers
have a poor grasp of how Windows handles anything outside of UTF-16 and
the W APIs, and barely understand ANSI codepages, let alone console codepages.
Both MSYS1 and Python 3 seem to treat Windows NT like Windows 9x way too often.
Not the kind of 1990s nostalgia trip we had in mind, but it's the one we got.
Should be safe because this code path will only fire on Linux and Unix-likes.
Committing directly to master due to it being a one-line change similar to all
the other Python 3 changes.
None of these resulted in build bustage using our default configuration,
but apparently ccache builds were broken due to a bytes/text issue. Wish
the original reporter had mentioned he was using ccache, but he saved me
a lot of busywork. Changes appear safe across whole range of Python
versions we support.
Forgot to test packaging on Linux this whole time.
Also, kept forgetting to commit these random changes to mozbuild
files that just refer to GNU/kFreeBSD consistently as kFreeBSD.
Not because it's the right thing to call it, but because it's
what we have in the enums that works consistently right now.
And ripping out support entirely would be out of scope.
I felt like this was a cheat when I did it in my hacky build, and
seriously thought ChatGPT was just suggesting this because it was too
lazy to figure out how to fix virtualenv. I learned several things:
- The approach virtualenv 15.x uses to simulate a Python environment
does not work after Python 3.10. So we'd need a newer virtualenv
anyway, which has different quirks. I got it to "work" on Python 3.11...
but only by copying (almost) the whole stdlib and a bunch of DLL files to
bootstrap it. Which is no longer a virtualenv, it's... just an env.
- VirtualenvManager is Mozilla's code, not a part of virtualenv itself.
Which means everything we need is now contained within our own code,
no third-party module like virtualenv required.
- Our build system doesn't depend on virtualenv quirks at all. In
a pinch, if you copy all the .pth files it generates into the
system's site-packages directory (after turning them all into
absolute paths), it would even run under system Python. It really
only "needs" virtualenv to find its own modules because of how it
was designed.
- I've tested venv extensively across a wide range of Python 3 versions.
not a single one has any issues or behaves in a noticably different
fashion from a build based on a repaired virtualenv extended up to
Python 3.10, which means no regressions were introduced by the switch.
- Python's devs want the venv route over the way virtualenv did things.
PEP 405 was very critical of virtualenv's design, and PEP 587 created a
a new configuration mechanism precisely so no application would have to
do what virtualenv did to get isolation without copying
the whole interpreter into another directory.
- Mozilla also did something similar in BZ 1766497, so they clearly
thought venv had enough features for their build system.
This led to a lot of false positives while trying to resolve the other
issue. The changes don't seem to cause harm, so I'd like to keep them...
they make debugging easier.
Sometimes Python 3.11+ keeps thinking the backend has changed due to
excessively precise timestamps or something. This workaround makes sure
that make doesn't regenerate the backend more than once per run.
pip was apparently breaking Linux builds and I didn't even notice. Turns out
virtualenv doesn't even need pip, and Python ships its own pip so I don't know
why this was even needed...
imp was actually still available up until Python 3.9 or so. Ripping
it out made me go through the headache of debugging our import logic.
Granted, it was deprecated long ago and there was plenty of warning, but
since we are doing the migration now it's all hitting at once if we want
to support any semi-modern Python.
Python no longer provides any version-related utilities in stdlib.
Did a thorough grep and made absolutely sure to replace every last
use. This was the work I was avoiding during my first port...
Ref: PEP 632, BZ 1777414, BZ 1858065, BZ 1766085
Got everything working under older Python 3, to deal with fewer issues at once.
Now the following had to be done:
- time.clock had to be replaced with time.process_time
- Drive letters now need explicit normalization because they are uppercase
by default (was the reason why I had to turn everything into a relpath before).
- Virtualenv works, but needs a couple extra modules because of new includes
in newer Python 3 versions.
Most notably, these tweaks were needed to fix ./mach build faster.
Also chose to set NSDISTMODE=copy by default for Windows, since
wildcards will be symlinks otherwise. This will always fail
because our binaries have the symlinked MSVC DLLs in their import
tables, and Windows loader silently fails in this situation.
For strings/bytes issue, used a similar strategy to what I saw in our old
layout code, having a flag set to be false by default as it's passed from
function to function, eventually being activated and doing something at
a specific point in the loop if it's ever true. The datetime stuff is
simply because in Python 3, integer comparisons were failing so I had
to make both sides be an integer Unix timestamp.
More small tweaks mostly related to decoding/encoding, some not even related to
failures but are more anxiously done in anticipation of something going wrong.
Even though these are called commits, I often revise little things several
times because I am not sure what the most robust approach would be.
The issue was essentially that the fourth field was being stripped due to the
way Python 3 handles text. I solved the problem originally with a "pad" hack to
add the extra space back in, since that actually gets interpreted as a reference
to the current directory. The eventual solution mostly involved using . instead of
a blank space to refer to the current directory. Don't know if the other stuff
was needed, may remove later if it causes any issues.
Most notable changes are the ugly stuff I had to do in FileAvoidWrite and
everything that interacts with it, which is where I expect a lot future pain
and revision to enter the picture from, if I have to revisit this part of the
work again.
Dropping this in actually just works for NSS (no promises this will hold true
with Basilisk), because gyp's design is a lot more conservative and consistent
compared with upstream Mozilla code. For some reason, the new gyp couldn't find
the six library it relies on that's already in our tree, and the library also
needed a small update. Also incorporates a few simple Python 3 tweaks to our
gyp_reader code.
Not really sure how to categorize these, except they are either
language-related cleanups, small bug fixes, or are careful to decode
things using the preferred OS encoding where reasonable. A lot of UTF-8 only
worked by luck because it only tried to decode ASCII bytes, or had good
replacements for bad chars due to explicit error handling.
I'm basically just commiting the last state before I update gyp, because
I can't really proceed without updating it, and that's a big enough step I want
it to have its own commit... and also don't want it mixed in with other stuff.
indented_repr and encode were introducing corruption into config.status
blocking the build. TypedTuple's logic needed to change a bit due to Python 3.
encode can just be removed because it does more harm than good now.
indented_repr was fixed. Note it was always limited to ASCII range because
that's the "safe zone" CP1252 and UTF-8 have in common. Investigate replacing
this with pprint and repr if we ever need Unicode in config.status...
Wasn't as hard as I expected to get this working. Mostly just missing imports,
and they are only missing because Mozilla added a couple to the base import
that caused no issues on Python 2, but require an additional slew of imports
on Python 3.
In decorators.py, the original logic fails because cls.__init__ is now a
function instead of a method in Python 3.
In string_version.py, the issue is that str.__init__(self, args) is no longer a
valid way of referencing the parent class of the function, the no-argument
super() can do that in Python 3 properly by using the __class__ cell variable.
Ref: PEP 3135
https://peps.python.org/pep-3135/
The bytestrings seem to break the build system. Also, get_hex is no longer a
thing, those two basestrings can become str (had to check), and
also noticed load_settings was ignoring its own argument.
Ref: BZ 1473498, BZ 1739556
https://hg-edge.mozilla.org/mozilla-central/rev/fa5e23975029
It seems to have moved out of the body of the function,
into the class definition. Hit this a couple times...
2to3 . --fix metaclass -wpn
Ref: PEP 3115
https://peps.python.org/pep-3115/
Basilisk-Dev
Moonchild
Jeremy Andrews