I looked at the Python wrapper so many times and felt like I was
writing C by proxy to clean up Python assumptions, and started
wondering if C++ was the right tool for this job instead of Python.
Here's my attempt at a C++ version of this wrapper. I'm offering both
options, you can revert this commit if you prefer the Python version.
This fixes the original Python wrapper. So basically, the Python developers
have a poor grasp of how Windows handles anything outside of UTF-16 and
the W APIs, and barely understand ANSI codepages, let alone console codepages.
Both MSYS1 and Python 3 seem to treat Windows NT like Windows 9x way too often.
Not the kind of 1990s nostalgia trip we had in mind, but it's the one we got.
This only happened because Python 3 no longer has the limitation of requiring VS 2009 to build binary extensions, and Python uses powershell to build those on Windows for some reason. Was an optional module anyway, not worth the annoyance it caused some users.
Should be safe because this code path will only fire on Linux and Unix-likes.
Committing directly to master due to it being a one-line change similar to all
the other Python 3 changes.
Probably not strictly related to Python 3 changes, but small enough
that expanding the scope to general UXP build issues on Debian
won't be too bad an idea, since the issue was reported there anyway.
Likely related to binutils updates reporter mentioned.
Reviewed-on: https://repo.palemoon.org/MoonchildProductions/UXP/pulls/3039
I reviewed and tested this PR on Fedora 43 two days ago. Gave it an r+. Was admittedly hoping Travis (our main Linux guy) would confirm my results, but Basilisk-Dev already tested Oracle Linux and I already tested Fedora, so that is likely good enough. Looked at the patch visually and see nothing that could go wrong on Windows or any other platform, since all the changes appear confined to Linux and WebRTC code paths that we don't use for Windows or Mac. Looks correct, and should be low-risk even if there are issues.
**Justification for merge into master on own initiative:**
1. I have enough Linux experience to confirm the patch shouldn't break our official Linux builds.
2. Patch mostly touches WebRTC related stuff in Basilisk-Dev's area of concern, so he is the most qualified to sign off on such changes.
3. The patch has been tested twice.
4. MC is a bit preoccupied with the server move.
5. Travis also appears to have been busy.
6. 10 days have passed since PR was first submitted.
If master breaks or anything unusual happens as a result of this merge, I am prepared to accept the blame and consequences.
Sincerely,
Jeremy Andrews (athenian200)
None of these resulted in build bustage using our default configuration,
but apparently ccache builds were broken due to a bytes/text issue. Wish
the original reporter had mentioned he was using ccache, but he saved me
a lot of busywork. Changes appear safe across whole range of Python
versions we support.
Apparently on 32-bit builds, bytes are passed into the build system in exactly
one more place than on 64-bit builds. The fix is trivial and one-line...
hopefully no one minds me doing this as a quick follow-up to the original
issue, it was very small and missed during testing.
Normally we'd want to do this as part of clean-up for Android-related
enums we don't use or making mozbuild syntax more consistent, but
Python 3's strict enforcement of enums kinda forced my hand here.
In principle, it's a little out of scope, but in practice the "scope"
is practically every Python file in the build system at this
point, unfortunately including mozbuild.
This was unneeded on my system because I made sure the build system couldn't
find Python 2 on the PATH, forcing it to use Python 3. Many developers may
be in environments with both available, so this should be changed.
Forgot to test packaging on Linux this whole time.
Also, kept forgetting to commit these random changes to mozbuild
files that just refer to GNU/kFreeBSD consistently as kFreeBSD.
Not because it's the right thing to call it, but because it's
what we have in the enums that works consistently right now.
And ripping out support entirely would be out of scope.
I felt like this was a cheat when I did it in my hacky build, and
seriously thought ChatGPT was just suggesting this because it was too
lazy to figure out how to fix virtualenv. I learned several things:
- The approach virtualenv 15.x uses to simulate a Python environment
does not work after Python 3.10. So we'd need a newer virtualenv
anyway, which has different quirks. I got it to "work" on Python 3.11...
but only by copying (almost) the whole stdlib and a bunch of DLL files to
bootstrap it. Which is no longer a virtualenv, it's... just an env.
- VirtualenvManager is Mozilla's code, not a part of virtualenv itself.
Which means everything we need is now contained within our own code,
no third-party module like virtualenv required.
- Our build system doesn't depend on virtualenv quirks at all. In
a pinch, if you copy all the .pth files it generates into the
system's site-packages directory (after turning them all into
absolute paths), it would even run under system Python. It really
only "needs" virtualenv to find its own modules because of how it
was designed.
- I've tested venv extensively across a wide range of Python 3 versions.
not a single one has any issues or behaves in a noticably different
fashion from a build based on a repaired virtualenv extended up to
Python 3.10, which means no regressions were introduced by the switch.
- Python's devs want the venv route over the way virtualenv did things.
PEP 405 was very critical of virtualenv's design, and PEP 587 created a
a new configuration mechanism precisely so no application would have to
do what virtualenv did to get isolation without copying
the whole interpreter into another directory.
- Mozilla also did something similar in BZ 1766497, so they clearly
thought venv had enough features for their build system.
The cgi module was removed from Python 3.14. html module has almost
the exact same thing, and this was apparently used to format html
rather than cgi anyway, so it's probably more correct.
Ref: PEP 594
https://peps.python.org/pep-0594/#cgi
This led to a lot of false positives while trying to resolve the other
issue. The changes don't seem to cause harm, so I'd like to keep them...
they make debugging easier.
Sometimes Python 3.11+ keeps thinking the backend has changed due to
excessively precise timestamps or something. This workaround makes sure
that make doesn't regenerate the backend more than once per run.
pip was apparently breaking Linux builds and I didn't even notice. Turns out
virtualenv doesn't even need pip, and Python ships its own pip so I don't know
why this was even needed...
imp was actually still available up until Python 3.9 or so. Ripping
it out made me go through the headache of debugging our import logic.
Granted, it was deprecated long ago and there was plenty of warning, but
since we are doing the migration now it's all hitting at once if we want
to support any semi-modern Python.
Python no longer provides any version-related utilities in stdlib.
Did a thorough grep and made absolutely sure to replace every last
use. This was the work I was avoiding during my first port...
Ref: PEP 632, BZ 1777414, BZ 1858065, BZ 1766085
Got everything working under older Python 3, to deal with fewer issues at once.
Now the following had to be done:
- time.clock had to be replaced with time.process_time
- Drive letters now need explicit normalization because they are uppercase
by default (was the reason why I had to turn everything into a relpath before).
- Virtualenv works, but needs a couple extra modules because of new includes
in newer Python 3 versions.
Most notably, these tweaks were needed to fix ./mach build faster.
Also chose to set NSDISTMODE=copy by default for Windows, since
wildcards will be symlinks otherwise. This will always fail
because our binaries have the symlinked MSVC DLLs in their import
tables, and Windows loader silently fails in this situation.
I also wound up having it point out files that can't be decoded as UTF-8.
Not sure if it was the right move, can remove that feature if it's not wanted.
For strings/bytes issue, used a similar strategy to what I saw in our old
layout code, having a flag set to be false by default as it's passed from
function to function, eventually being activated and doing something at
a specific point in the loop if it's ever true. The datetime stuff is
simply because in Python 3, integer comparisons were failing so I had
to make both sides be an integer Unix timestamp.
More small tweaks mostly related to decoding/encoding, some not even related to
failures but are more anxiously done in anticipation of something going wrong.
Even though these are called commits, I often revise little things several
times because I am not sure what the most robust approach would be.
This is fewer issues than I remember encountering last time... 2to3 must have
helped. Though I won't hold my breath that newer Python 3 will be this
forgiving when I get around to that near the end.
The issue was essentially that the fourth field was being stripped due to the
way Python 3 handles text. I solved the problem originally with a "pad" hack to
add the extra space back in, since that actually gets interpreted as a reference
to the current directory. The eventual solution mostly involved using . instead of
a blank space to refer to the current directory. Don't know if the other stuff
was needed, may remove later if it causes any issues.
Most of this is taken from my original port, but I did revisit xpt.py
and rework the way cmp was translated to Python 3 by using a
ComparableMixin class.
Most notable changes are the ugly stuff I had to do in FileAvoidWrite and
everything that interacts with it, which is where I expect a lot future pain
and revision to enter the picture from, if I have to revisit this part of the
work again.
Dropping this in actually just works for NSS (no promises this will hold true
with Basilisk), because gyp's design is a lot more conservative and consistent
compared with upstream Mozilla code. For some reason, the new gyp couldn't find
the six library it relies on that's already in our tree, and the library also
needed a small update. Also incorporates a few simple Python 3 tweaks to our
gyp_reader code.
Not really sure how to categorize these, except they are either
language-related cleanups, small bug fixes, or are careful to decode
things using the preferred OS encoding where reasonable. A lot of UTF-8 only
worked by luck because it only tried to decode ASCII bytes, or had good
replacements for bad chars due to explicit error handling.
I'm basically just commiting the last state before I update gyp, because
I can't really proceed without updating it, and that's a big enough step I want
it to have its own commit... and also don't want it mixed in with other stuff.
indented_repr and encode were introducing corruption into config.status
blocking the build. TypedTuple's logic needed to change a bit due to Python 3.
encode can just be removed because it does more harm than good now.
indented_repr was fixed. Note it was always limited to ASCII range because
that's the "safe zone" CP1252 and UTF-8 have in common. Investigate replacing
this with pprint and repr if we ever need Unicode in config.status...
Wasn't as hard as I expected to get this working. Mostly just missing imports,
and they are only missing because Mozilla added a couple to the base import
that caused no issues on Python 2, but require an additional slew of imports
on Python 3.
This is apparently the one place we can't use UTF-8 on Windows, we absolutely
need CP1252 here. Mozilla originally avoided a lot of these encoding issues by
treating everything as bytes. That trick won't fly in Python 3, so we now
have to deal with text streams and face Windows and Unicode issues.
Ref: BZ 1585702
In decorators.py, the original logic fails because cls.__init__ is now a
function instead of a method in Python 3.
In string_version.py, the issue is that str.__init__(self, args) is no longer a
valid way of referencing the parent class of the function, the no-argument
super() can do that in Python 3 properly by using the __class__ cell variable.
Ref: PEP 3135
https://peps.python.org/pep-3135/
The bytestrings seem to break the build system. Also, get_hex is no longer a
thing, those two basestrings can become str (had to check), and
also noticed load_settings was ignoring its own argument.
Ref: BZ 1473498, BZ 1739556
https://hg-edge.mozilla.org/mozilla-central/rev/fa5e23975029
The issue requiring this only affected older versions of Python 3.
Newer Python 3 no longer requires this. Mozilla also did this in their
initial Python 3 support, after moving the hack itself from mach to
somewhere in mozbuild. This commit also unblocks newer Python versions.
Ref: BZ 1473498
https://hg-edge.mozilla.org/mozilla-central/rev/fa5e23975029
It seems to have moved out of the body of the function,
into the class definition. Hit this a couple times...
2to3 . --fix metaclass -wpn
Ref: PEP 3115
https://peps.python.org/pep-3115/
2to3 . --fix tuple_params -wpn
They actually had a fixer for this?! Anyway... yeah,
Py2 allowed tuple parameter unpacking. Py3 doesn't.
However, the adaptations are fairly mechanical.
Ref: PEP 3113
https://peps.python.org/pep-3113/
2to3 . --fix input -wpn
2to3 . --fix raw_input -wpn
Py2 input is the new Py3 input wrapped in eval tags.
The new input is the old raw_input.
This lets you have the old behavior. Mechanical.
Ref: PEP 3111
https://peps.python.org/pep-3111/
2to3 . --fix except -wpn
Python 3 exception syntax is different. Comma-based syntax is gone,
much like print syntax is changed.
This lets you have the old behavior. Mechanical.
Ref: PEP 3110
https://peps.python.org/pep-3110/
2to3 . --fix raise -wpn
Python 3 raise syntax is different. Three-argument raise is
deprecated, much like print syntax is changed.
This lets you have the old behavior. Mechanical.
Ref: PEP 3109
https://peps.python.org/pep-3109/
2to3 . --fix dict -wpn
Py2 dict was a list by default, the iterator versions
were switched around to be the default.
This lets you have the old behavior. Mechanical.
Ref: PEP 3106
https://peps.python.org/pep-3106/
This should not be kept around because AIX is not a valid OS_TARGET and I've seen mach fail at least once because of these. Meant to clean them up years ago.
Adds --disable-npapi as an additional build option. The default is still to build in NPAPI compatibility, but with this flag, it's finally possible to build without pulling in GTK2 on systems that don't have the package.
This essentially ifdefs everything associated with NPAPI, following various Mozilla removal bugs.
Ref: BZ 1682030, BZ 1251202, BZ 1217665, BZ 1687239
This was already in the codebase, but not exposed by default. Mozilla exposed
the pref to fulfill user requests in Firefox 59. Forum users requested the same feaure of pseudo-fullscreen windows. Seems minor enough not to be worth
creating an issue for, since users can already access this by creating a bool
pref.
Ref: BZ 1422535
I've been thinking about this issue, and ultimately it's just correcting a
mistake Mozilla made that no consumer of UXP's MailNews ever complained about
before, so I think it's reasonable to fix the default behavior first, and then
let others complain before I start worrying about prefs or ifdefs. One thing
at a time.
It's really easy to overlook configure options in moz.configure,
since most of what we work with is old-configure. Might as well move it
back in here so we can keep track more easily.
Jeremy Andrews