Expand float and complex strict mode to allow ints and ints/float (for PEP 484 compatibility). #5879

InvincibleRMC · 2025-10-23T16:03:05Z

Description

Breaking change to allow Python ints into float when under strict typing (noconvert), to better match PEP 484 numeric tower rules.

This change was also done to complex to allow floats and ints. Now that Python ints can be passed into floats it changes behavior for overload resolution. A method that takes float that is registered before a method that takes an int will now get executed when a Python int is passed in. This overload resolution also affects methods with std::complex.

Resolves #5878
Provides a work around for #5767

NOTE: This PR was reviewed extensively and is approved. However, because of the breaking change, it will be merged only after the next patch release.

Suggested changelog entry:

Breaking change for PEP 484 compatibility: Expand float and complex strict mode to allow ints and ints/float.

📚 Documentation preview 📚: https://pybind11--5879.org.readthedocs.build/

Signed-off-by: Michael Carlstrom <rmc@carlstrom.com>

gentlegiantJGC · 2025-10-23T16:27:22Z

I have started working on this too.
Your code change is more complex than required.
I will finish up the documentation changes and commit what I have and you can take it from there.

This is known at compile time so it can be constexpr

gentlegiantJGC · 2025-10-23T16:49:05Z

Here is my implementation and documentation changes.
I have modified the behaviour of complex as well.
https://github.com/gentlegiantJGC/pybind11/tree/fix-pep-484
master...gentlegiantJGC:pybind11:fix-pep-484

It needs tests but I am going to stop for today.
You are welcome to use my commits.

…trict

gentlegiantJGC · 2025-10-23T17:57:43Z

You will need to remove the constexpr check. I didn't realise it was a newer C++ feature

Signed-off-by: Michael Carlstrom <rmc@carlstrom.com>

InvincibleRMC · 2025-10-23T19:57:40Z

TODO: figure out why it doesn't work in C++11.

Signed-off-by: Michael Carlstrom <rmc@carlstrom.com>

gentlegiantJGC · 2025-10-24T07:41:04Z

I have just found that a python float passed to an int argument will raise a TypeError.
It will happily except anything that implements __int__ (eg a numpy float) but explicity not a python float.

This is probably something we should resolve here because it effects these tests.

m.def("func", [](int){});

>>> func.__doc__
  func(arg: typing.SupportsInt) -> None
>>> func(5.5)
  TypeError
>>> func(numpy.float32(5.5))  # This works fine

gentlegiantJGC · 2025-10-24T07:56:18Z

Here is the commit that implemented that.
3f200fa

@rwgk Can I get your input on this?
The int type caster supports anything that implements __int__ with the explicit exception of the python float.

This conflicts with the typing.SupportsInt type hint.

The int type caster allows anything that implements __int__ with explicit exception of the python float. I can't see any reason for this. This modifies the int casting behaviour to accept a float. If the argument is marked as noconvert() it will only accept int.

gentlegiantJGC · 2025-10-24T08:20:39Z

Here is my proposed change. You will need to update the tests.
gentlegiantJGC@d42c8e8
https://github.com/gentlegiantJGC/pybind11/tree/pr-5879-jgc

Here are my unittests
test_cast.py
cast.cpp

InvincibleRMC · 2025-10-24T19:14:19Z

I noticed something else to have address. According to the documentation all of int,float,complex support falling back on __index__ when converting so updated the type hint from typing.SupportsInt -> typing.SupportsInt | typing.SupportsIndex. complex also supports a __float__ fall back so its type is now typing.SupportsComplex | typing.SupportsFloat | typing.SupportsIndex

rwgk · 2025-11-11T05:02:48Z

That's much easier to review now!

The production code changes are actually very easy to review now.

I'll comb through the test changes carefully tomorrow, for a final confirmation that everything is OK.

I want to hold back merging though. The plan I have in mind:

Review ~~and merge~~ Improve performance of enum_ operators by going back to specific implementation #5887
Make a pybind11 v3.0.2 release.
Merge this PR, bumping the version to v3.1.0a0
Let this sit on master for a few weeks to see what feedback we get.

@henryiii @swolchok for visibility

rwgk · 2025-11-11T22:01:45Z

xref regarding merge & release planning: #5887 (comment)

Replace cryptic 'So: PYBIND11_INDEX_CHECK(src.ptr())' comment with clearer explanation of the logic: - Explains that we need to call PyNumber_Index explicitly on PyPy for non-PyLong objects - Clarifies the relationship to the outer condition: when convert is false, we only reach this point if PYBIND11_INDEX_CHECK passed above This makes the code more maintainable and easier to understand during review.

During merge, HEAD's regex pattern was kept, but master's version is preferred. The order of ` ` and `\|` in the character class is arbitrary. Keep master's order (already fixed in PR pybind#5891; sorry I missed looking back here when working on 5891).

…1)` call and clearly explain the behavior change.

Enabling implicit float → int conversion in convert mode causes silent truncation (e.g., 1.9 → 1). This is dangerous because: 1. It's implicit - users don't expect truncation when calling functions 2. It's silent - no warning or error 3. It can hide bugs - precision loss is hard to detect This change restores the explicit rejection of PyFloat_Check for integer casters, even in convert mode. This is more in line with Python's behavior where int(1.9) must be explicit. Note that the int → float conversion in noconvert mode is preserved, as that's a safe widening conversion.

This reverts all test modifications that were made to accommodate implicit float→int conversion in convert mode. With the production code change that explicitly rejects float→int conversion even in convert mode, these test workarounds are no longer needed. Changes reverted: - test_builtin_casters.py: Restored cant_convert(3.14159) and np.float32 conversion with deprecated_call wrapper - test_custom_type_casters.py: Restored TypeError expectation for m.ints_preferred(4.0) - test_methods_and_attributes.py: Restored TypeError expectation for m.overload_order(1.1) - test_stl.py: Restored float literals (2.0) that were replaced with strings to avoid conversion - test_factory_constructors.py: Restored original constructor calls that were modified to avoid float→int conversion Also removes the unused avoid_PyLong_AsLong_deprecation fixture and related TypeVar imports, as all uses were removed.

The env.deprecated_call() function was removed, but two test cases still reference it. Replace with pytest.deprecated_call(), which is the standard pytest context manager for handling deprecation warnings. Since we already require pytest>=6 (see tests/requirements.txt), the compatibility function is obsolete and pytest.deprecated_call() is available.

PR 5879 swapped the order of NoisyAlloc constructor overloads: - (int i, double) is now placement new (comes first) - (double d, double) is now factory pointer (comes second) This swap is necessary because pybind11 tries overloads in order until one matches. With int → float conversion now allowed: - create_and_destroy(4, 0.5): Without the swap, (double d, double) would match first (since int → double conversion is allowed), bypassing the more specific (int i, double) overload. With the swap, (int i, double) matches first (exact match), which is correct. - create_and_destroy(3.5, 4.5): (int i, double) fails (float → int is rejected), then (double d, double) matches, which is correct. The swap ensures exact int matches are preferred over double matches when an int is provided, which is the expected overload resolution behavior. Update the test expectations to match the new overload resolution order.

rwgk · 2025-11-12T01:40:41Z

I've been working through the PR to understand the behavior changes and ensure everything is correct. Sorry it took me way longer than I'd want to admit to realize that this PR was enabling implicit, narrowing Python float to C++ int conversions. Here is where I realized:

c7e959e

After the penny dropped, I added this commit:

e4023cd - Reject float → int conversion even in convert mode

Please see the comprehensive commit message for the rationale.

These are follow-on commits to make the tests pass again:

cc981a8 — Revert test changes that sidestepped implicit float→int conversion
cc411c6 — Replace env.deprecated_call() with pytest.deprecated_call()
ac957e3 — Update test expectations for swapped NoisyAlloc overloads

Before that I added a couple minor commits:

0a00147 - Undo inconsequential change to regex in test_enum.py
41ff876 - Improve comment clarity for PyPy __index__ handling

There are still things I want to look into [EDIT: DONE], to make sure we're not creating weird oddities (beyond already existing oddities):

~~What is the conversion behavior for NumPy scalars (e.g., np.float32, np.float64)? Should we add corresponding tests?~~
What is the PyNumber_Index behavior with floats?
Should we add a validation to ensure that __index__ actually returns int (when converting to C++ int)?

I probably won't have a block of time to focus on this PR again before next weekend, but I'll read all comments.

Note: I've been using Cursor a lot, in case you wonder about all the long commit messages.

Since we already require pytest>=6 (see tests/requirements.txt), the old compatibility function is obsolete and pytest.deprecated_call() can be used directly. Extracted from PR pybind#5879 Co-authored-by: Michael Carlstrom <rmc@carlstrom.com> Co-authored-by: gentlegiantJGC <gentlegiantJGC@users.noreply.github.com>

rwgk · 2025-11-12T02:07:06Z

FYI — While I was at it, I isolated out this change, to be merged asap:

rwgk@89917e9

I'll mark you a co-authors there. (I should have done that yesterday for the other PR, too, but sorry it missed the opportunity.)

The goal is to make this critical PR as clean and lean as possible. I expect that it will be looked back at a lot, b/o the behavior change.

I'll convert the branch to a PR when the CI here is done. (We only have the free GHA quota now, unfortunately.)

EDIT: That's PR #5893 now.

Since we already require pytest>=6 (see tests/requirements.txt), the old compatibility function is obsolete and pytest.deprecated_call() can be used directly. Extracted from PR #5879 Co-authored-by: Michael Carlstrom <rmc@carlstrom.com> Co-authored-by: gentlegiantJGC <gentlegiantJGC@users.noreply.github.com>

/__w/pybind11/pybind11/include/pybind11/cast.h:253:46: error: repeated branch body in conditional chain [bugprone-branch-clone,-warnings-as-errors] 253 | } else if (PyFloat_Check(src.ptr())) { | ^ /__w/pybind11/pybind11/include/pybind11/cast.h:258:10: note: end of the original 258 | } else if (convert || PYBIND11_LONG_CHECK(src.ptr()) || PYBIND11_INDEX_CHECK(src.ptr())) { | ^ /__w/pybind11/pybind11/include/pybind11/cast.h:283:16: note: clone 1 starts here 283 | } else { | ^

gentlegiantJGC

I can't see any obvious issues

gentlegiantJGC · 2025-11-12T07:09:39Z

I will open a new issue about the custom float -> int conversion

…eturning float These tests ensure that: - Invalid return types (floats) are properly rejected - The fallback from __index__ to __int__ works correctly in convert mode - noconvert mode correctly prevents fallback when __index__ fails

rwgk · 2025-11-12T09:48:48Z

The extra tests (commit f1d8158) are:

To be certain about the exact behavior in pathological cases, so we're sure we don't have silent float to int conversions.
To find out if the PyPy code path in cast.h behaves identically.

This is a bit on the paranoid side, but I wanted to reassure myself that we're not putting the world through a behavior change without having looked at every dark corner.

rwgk · 2025-11-12T10:03:11Z

BTW, having looked at this so intensely ...

        bool py_err = py_value == (py_type) -1 && PyErr_Occurred();

this will be OK if Python is healthy, but if something goes wrong (e.g. UB, or bug in Python beta testing) and py_value is not -1 even though PyErr_Occurred() is true, the error will not be cleared and could get reported in the wrong context.

So that condition basically errs on the unsafe side, it prioritizes the return value check over error detection.

I think it'd be better to write:

        bool py_err = PyErr_Occurred();
        if (py_err) assert(py_value == (py_type) -1);

This prioritizes error detection over return value validation.

WDYT?

…ere (e.g. UB, or bug in Python beta testing). See also: pybind#5879 (comment)

rwgk · 2025-11-12T16:49:19Z

I can't see any obvious issues

Same. I looked through everything several times now. Thanks!

init

515a3ab

Signed-off-by: Michael Carlstrom <rmc@carlstrom.com>

InvincibleRMC mentioned this pull request Oct 23, 2025

[BUG]: SupportsInt and SupportsFloat changes cause type checking failures on python classes that override these functions #5767

Open

3 tasks

gentlegiantJGC added 4 commits October 23, 2025 17:36

Add constexpr to is_floating_point check

aa2693f

This is known at compile time so it can be constexpr

Allow noconvert float to accept int

7948b27

Update noconvert documentation

47746af

Allow noconvert complex to accept int and float

507d31c

InvincibleRMC and others added 5 commits October 23, 2025 10:02

Merge remote-tracking branch 'collab/fix-pep-484' into expand-float-s…

f4115fe

…trict

Add complex strict test

4ea8bcb

style: pre-commit fixes

81e49f6

Update unit tests so int, becomes double.

0b94f21

style: pre-commit fixes

e9bf4e5

InvincibleRMC changed the title ~~Expand float strict to take int~~ Expand float strict to take int and complex strict to take float/int Oct 23, 2025

InvincibleRMC added 3 commits October 23, 2025 11:00

remove if (constexpr)

21f8447

Signed-off-by: Michael Carlstrom <rmc@carlstrom.com>

fix spelling error

5806318

bump order in #else

a0fb6dc

Switch order in c++11 only section

2692820

Signed-off-by: Michael Carlstrom <rmc@carlstrom.com>

InvincibleRMC marked this pull request as ready for review October 24, 2025 00:59

InvincibleRMC added 2 commits October 23, 2025 18:11

ci: trigger build

b12f5a8

ci: trigger build

2187a65

InvincibleRMC added 2 commits October 24, 2025 11:29

tests for py::float into int

16bdff3

Update complex_cast tests

358266f

rwgk mentioned this pull request Nov 11, 2025

Improve performance of enum_ operators by going back to specific implementation #5887

Open

rwgk added 7 commits November 11, 2025 14:18

test_methods_and_attributes.py: Restore existing `m.overload_order(1.…

c7e959e

…1)` call and clearly explain the behavior change.

rwgk mentioned this pull request Nov 12, 2025

Replace env.deprecated_call() with pytest.deprecated_call() #5893

Merged

rwgk changed the title ~~Expand float and complex strict mode to allow ints and ints/float. Updates type hints to match better with typing.SupportsIndex~~ Expand float and complex strict mode to allow ints and ints/float. Nov 12, 2025

rwgk added 2 commits November 11, 2025 19:35

Merge branch 'master' into InvincibleRMC→expand-float-strict

2a32770

gentlegiantJGC approved these changes Nov 12, 2025

View reviewed changes

Minor comment-only changes: add PR number, for easy future reference

0f1f8ea

rwgk changed the title ~~Expand float and complex strict mode to allow ints and ints/float.~~ Expand float and complex strict mode to allow ints and ints/float (for PEP 484 compatibility). Nov 12, 2025

Ensure we are not leaking a Python error is something is wrong elsewh…

816298a

…ere (e.g. UB, or bug in Python beta testing). See also: pybind#5879 (comment)

rwgk approved these changes Nov 12, 2025

View reviewed changes

Merge branch 'master' into expand-float-strict

cc3209c

Expand float and complex strict mode to allow ints and ints/float (for PEP 484 compatibility). #5879

Are you sure you want to change the base?

Expand float and complex strict mode to allow ints and ints/float (for PEP 484 compatibility). #5879

Conversation

InvincibleRMC commented Oct 23, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Suggested changelog entry:

Uh oh!

gentlegiantJGC commented Oct 23, 2025

Uh oh!

gentlegiantJGC commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gentlegiantJGC commented Oct 23, 2025

Uh oh!

InvincibleRMC commented Oct 23, 2025

Uh oh!

gentlegiantJGC commented Oct 24, 2025

Uh oh!

gentlegiantJGC commented Oct 24, 2025

Uh oh!

gentlegiantJGC commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

InvincibleRMC commented Oct 24, 2025

Uh oh!

rwgk commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rwgk commented Nov 11, 2025

Uh oh!

rwgk commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rwgk commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gentlegiantJGC left a comment

Choose a reason for hiding this comment

Uh oh!

gentlegiantJGC commented Nov 12, 2025

Uh oh!

rwgk commented Nov 12, 2025

Uh oh!

rwgk commented Nov 12, 2025

Uh oh!

rwgk commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

InvincibleRMC commented Oct 23, 2025 •

edited by github-actions bot

Loading

gentlegiantJGC commented Oct 23, 2025 •

edited

Loading

gentlegiantJGC commented Oct 24, 2025 •

edited

Loading

rwgk commented Nov 11, 2025 •

edited

Loading

rwgk commented Nov 12, 2025 •

edited

Loading

rwgk commented Nov 12, 2025 •

edited

Loading