Merge remote-tracking branch 'upstream/main' into string-dtype-categorical-repr

Author: jorisvandenbossche

.github/workflows/docbuild-and-upload.yml

@@ -57,8 +57,6 @@ jobs:
       run: python web/pandas_web.py web/pandas --target-path=web/build
 
     - name: Build documentation
-      # TEMP don't let errors fail the build until all string dtype changes are fixed
-      continue-on-error: true
       run: doc/make.py --warnings-are-errors
 
     - name: Build the interactive terminal

.github/workflows/unit-tests.yml

@@ -91,7 +91,7 @@ jobs:
       LANG: ${{ matrix.lang || 'C.UTF-8' }}
       LC_ALL: ${{ matrix.lc_all || '' }}
       PANDAS_CI: '1'
-      PANDAS_FUTURE_INFER_STRING: ${{ matrix.pandas_future_infer_string || '0' }}
+      PANDAS_FUTURE_INFER_STRING: ${{ matrix.pandas_future_infer_string || '1' }}
       TEST_ARGS: ${{ matrix.test_args || '' }}
       PYTEST_WORKERS: 'auto'
       PYTEST_TARGET: ${{ matrix.pytest_target || 'pandas' }}

ci/code_checks.sh

@@ -74,6 +74,7 @@ if [[ -z "$CHECK" || "$CHECK" == "docstrings" ]]; then
         -i "pandas.Series.dt PR01" `# Accessors are implemented as classes, but we do not document the Parameters section` \
         -i "pandas.Period.freq GL08" \
         -i "pandas.Period.ordinal GL08" \
+        -i "pandas.errors.IncompatibleFrequency SA01,SS06,EX01" \
         -i "pandas.core.groupby.DataFrameGroupBy.plot PR02" \
         -i "pandas.core.groupby.SeriesGroupBy.plot PR02" \
         -i "pandas.core.resample.Resampler.quantile PR01,PR07" \

doc/source/development/maintaining.rst

@@ -388,8 +388,11 @@ Pre-release
 
 3. Make sure the CI is green for the last commit of the branch being released.
 
-4. If not a release candidate, make sure all backporting pull requests to the branch
-   being released are merged.
+4. If not a release candidate, make sure all backporting pull requests to the
+   branch being released are merged, and no merged pull requests are missing a
+   backport (check the
+   ["Still Needs Manual Backport"](https://github.com/pandas-dev/pandas/labels/Still%20Needs%20Manual%20Backport)
+   label for this).
 
 5. Create a new issue and milestone for the version after the one being released.
    If the release was a release candidate, we would usually want to create issues and
@@ -435,6 +438,9 @@ which will be triggered when the tag is pushed.
 
     scripts/download_wheels.sh <VERSION>
 
+   ATTENTION: this is currently not downloading *all* wheels, and you have to
+   manually download the remainings wheels and sdist!
+
 4. Create a `new GitHub release <https://github.com/pandas-dev/pandas/releases/new>`_:
 
    - Tag: ``<version>``
@@ -462,15 +468,22 @@ Post-Release
 ````````````
 
 1. Update symlinks to stable documentation by logging in to our web server, and
-   editing ``/var/www/html/pandas-docs/stable`` to point to ``version/<latest-version>``
-   for major and minor releases, or ``version/<minor>`` to ``version/<patch>`` for
+   editing ``/var/www/html/pandas-docs/stable`` to point to ``version/<X.Y>``
+   for major and minor releases, or ``version/<X.Y>`` to ``version/<patch>`` for
    patch releases. The exact instructions are (replace the example version numbers by
    the appropriate ones for the version you are releasing):
 
    - Log in to the server and use the correct user.
    - ``cd /var/www/html/pandas-docs/``
-   - ``ln -sfn version/2.1 stable`` (for a major or minor release)
-   - ``ln -sfn version/2.0.3 version/2.0`` (for a patch release)
+   - For a major or minor release (assuming the ``/version/2.1.0/`` docs have been uploaded to the server):
+
+     -  Create a new X.Y symlink to X.Y.Z: ``cd version; ln -sfn 2.1.0 2.1``
+     -  Update stable symlink to point to X.Y: ``ln -sfn version/2.1 stable``
+
+   - For a patch release (assuming the ``/version/2.1.3/`` docs have been uploaded to the server):
+
+     - Update the X.Y symlink to the new X.Y.Z patch version: ``cd version; ln -sfn 2.1.3 2.1``
+     - (the stable symlink should already be pointing to the correct X.Y version)
 
 2. If releasing a major or minor release, open a PR in our source code to update
    ``web/pandas/versions.json``, to have the desired versions in the documentation

doc/source/reference/testing.rst

@@ -36,6 +36,7 @@ Exceptions and warnings
    errors.DuplicateLabelError
    errors.EmptyDataError
    errors.IncompatibilityWarning
+   errors.IncompatibleFrequency
    errors.IndexingError
    errors.InvalidColumnName
    errors.InvalidComparison

doc/source/user_guide/basics.rst

@@ -590,7 +590,7 @@ arguments. The special value ``all`` can also be used:
 
 .. ipython:: python
 
-    frame.describe(include=["object"])
+    frame.describe(include=["str"])
     frame.describe(include=["number"])
     frame.describe(include="all")
 

doc/source/whatsnew/v0.13.0.rst

@@ -184,7 +184,7 @@ API changes
   .. ipython:: python
      :okwarning:
 
-     dfc.loc[0]['A'] = 1111
+     dfc.loc[0]['B'] = 1111
 
   ::
 
@@ -198,7 +198,7 @@ API changes
 
   .. ipython:: python
 
-     dfc.loc[0, 'A'] = 11
+     dfc.loc[0, 'B'] = 1111
      dfc
 
 - ``Panel.reindex`` has the following call signature ``Panel.reindex(items=None, major_axis=None, minor_axis=None, **kwargs)``

doc/source/whatsnew/v0.15.0.rst

@@ -1025,20 +1025,49 @@ Other:
 - :func:`describe` on mixed-types DataFrames is more flexible. Type-based column filtering is now possible via the ``include``/``exclude`` arguments.
   See the :ref:`docs <basics.describe>` (:issue:`8164`).
 
-  .. ipython:: python
+  .. code-block:: python
 
-    df = pd.DataFrame({'catA': ['foo', 'foo', 'bar'] * 8,
-                       'catB': ['a', 'b', 'c', 'd'] * 6,
-                       'numC': np.arange(24),
-                       'numD': np.arange(24.) + .5})
-    df.describe(include=["object"])
-    df.describe(include=["number", "object"], exclude=["float"])
+    >>> df = pd.DataFrame({'catA': ['foo', 'foo', 'bar'] * 8,
+    ...                    'catB': ['a', 'b', 'c', 'd'] * 6,
+    ...                    'numC': np.arange(24),
+    ...                    'numD': np.arange(24.) + .5})
+    >>> df.describe(include=["object"])
+           catA catB
+    count    24   24
+    unique    2    4
+    top     foo    a
+    freq     16    6
+    >>> df.describe(include=["number", "object"], exclude=["float"])
+           catA catB       numC
+    count    24   24  24.000000
+    unique    2    4        NaN
+    top     foo    a        NaN
+    freq     16    6        NaN
+    mean    NaN  NaN  11.500000
+    std     NaN  NaN   7.071068
+    min     NaN  NaN   0.000000
+    25%     NaN  NaN   5.750000
+    50%     NaN  NaN  11.500000
+    75%     NaN  NaN  17.250000
+    max     NaN  NaN  23.000000
 
   Requesting all columns is possible with the shorthand 'all'
 
-  .. ipython:: python
+  .. code-block:: python
 
-    df.describe(include='all')
+    >>> df.describe(include='all')
+           catA catB       numC       numD
+    count    24   24  24.000000  24.000000
+    unique    2    4        NaN        NaN
+    top     foo    a        NaN        NaN
+    freq     16    6        NaN        NaN
+    mean    NaN  NaN  11.500000  12.000000
+    std     NaN  NaN   7.071068   7.071068
+    min     NaN  NaN   0.000000   0.500000
+    25%     NaN  NaN   5.750000   6.250000
+    50%     NaN  NaN  11.500000  12.000000
+    75%     NaN  NaN  17.250000  17.750000
+    max     NaN  NaN  23.000000  23.500000
 
   Without those arguments, ``describe`` will behave as before, including only numerical columns or, if none are, only categorical columns. See also the :ref:`docs <basics.describe>`
 

doc/source/whatsnew/v3.0.0.rst

@@ -1,6 +1,6 @@
 .. _whatsnew_300:
 
-What's new in 3.0.0 (Month XX, 2024)
+What's new in 3.0.0 (Month XX, 2025)
 ------------------------------------
 
 These are the changes in pandas 3.0.0. See :ref:`release` for a full changelog
@@ -414,6 +414,7 @@ Other API changes
 - Index set operations (like union or intersection) will now ignore the dtype of
   an empty ``RangeIndex`` or empty ``Index`` with object dtype when determining
   the dtype of the resulting Index (:issue:`60797`)
+- :class:`IncompatibleFrequency` now subclasses ``TypeError`` instead of ``ValueError``. As a result, joins with mismatched frequencies now cast to object like other non-comparable joins, and arithmetic with indexes with mismatched frequencies align (:issue:`55782`)
 - Comparison operations between :class:`Index` and :class:`Series` now consistently return :class:`Series` regardless of which object is on the left or right (:issue:`36759`)
 - Numpy functions like ``np.isinf`` that return a bool dtype when called on a :class:`Index` object now return a bool-dtype :class:`Index` instead of ``np.ndarray`` (:issue:`52676`)
 
@@ -456,6 +457,7 @@ Other Deprecations
 - Deprecated :meth:`.DataFrameGroupby.corrwith` (:issue:`57158`)
 - Deprecated :meth:`Timestamp.utcfromtimestamp`, use ``Timestamp.fromtimestamp(ts, "UTC")`` instead (:issue:`56680`)
 - Deprecated :meth:`Timestamp.utcnow`, use ``Timestamp.now("UTC")`` instead (:issue:`56680`)
+- Deprecated ``pd.core.internals.api.maybe_infer_ndim`` (:issue:`40226`)
 - Deprecated allowing non-keyword arguments in :meth:`DataFrame.all`, :meth:`DataFrame.min`, :meth:`DataFrame.max`, :meth:`DataFrame.sum`, :meth:`DataFrame.prod`, :meth:`DataFrame.mean`, :meth:`DataFrame.median`, :meth:`DataFrame.sem`, :meth:`DataFrame.var`, :meth:`DataFrame.std`, :meth:`DataFrame.skew`, :meth:`DataFrame.kurt`, :meth:`Series.all`,  :meth:`Series.min`, :meth:`Series.max`, :meth:`Series.sum`, :meth:`Series.prod`, :meth:`Series.mean`, :meth:`Series.median`, :meth:`Series.sem`, :meth:`Series.var`, :meth:`Series.std`, :meth:`Series.skew`, and :meth:`Series.kurt`. (:issue:`57087`)
 - Deprecated allowing non-keyword arguments in :meth:`Series.to_markdown` except ``buf``. (:issue:`57280`)
 - Deprecated allowing non-keyword arguments in :meth:`Series.to_string` except ``buf``. (:issue:`57280`)
@@ -694,6 +696,7 @@ Datetimelike
 - Bug in :attr:`is_year_start` where a DateTimeIndex constructed via a date_range with frequency 'MS' wouldn't have the correct year or quarter start attributes (:issue:`57377`)
 - Bug in :class:`DataFrame` raising ``ValueError`` when ``dtype`` is ``timedelta64`` and ``data`` is a list containing ``None`` (:issue:`60064`)
 - Bug in :class:`Timestamp` constructor failing to raise when ``tz=None`` is explicitly specified in conjunction with timezone-aware ``tzinfo`` or data (:issue:`48688`)
+- Bug in :class:`Timestamp` constructor failing to raise when given a ``np.datetime64`` object with non-standard unit (:issue:`25611`)
 - Bug in :func:`date_range` where the last valid timestamp would sometimes not be produced (:issue:`56134`)
 - Bug in :func:`date_range` where using a negative frequency value would not include all points between the start and end values (:issue:`56147`)
 - Bug in :func:`tseries.api.guess_datetime_format` would fail to infer time format when "%Y" == "%H%M" (:issue:`57452`)

pandas/_libs/groupby.pyi

@@ -67,6 +67,7 @@ def group_sum(
     result_mask: np.ndarray | None = ...,
     min_count: int = ...,
     is_datetimelike: bool = ...,
+    initial: object = ...,
     skipna: bool = ...,
 ) -> None: ...
 def group_prod(