QST: Is it intended that empty dictionary aggregation raises exception since 1.2.0?

[dchigarev]

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

---

Question about pandas

import pandas

df = pandas.DataFrame({"a": [1, 2], "b": [3, 4]})
print(df.agg({})) # ValueError: No objects to concatenate

Traceback

Traceback (most recent call last):
  File "../exp.py", line 4, in <module>
    print(df.agg({}))
  File "/localdisk/dchigare/repos/pandas/pandas/core/frame.py", line 7689, in aggregate
    result, how = self._aggregate(func, axis, *args, **kwargs)
  File "/localdisk/dchigare/repos/pandas/pandas/core/frame.py", line 7723, in _aggregate
    result, how = op.agg()
  File "/localdisk/dchigare/repos/pandas/pandas/core/apply.py", line 177, in agg
    return self.agg_dict_like(_axis), True
  File "/localdisk/dchigare/repos/pandas/pandas/core/apply.py", line 374, in agg_dict_like
    result = concat({k: results[k] for k in keys_to_use}, axis=axis)
  File "/localdisk/dchigare/repos/pandas/pandas/core/reshape/concat.py", line 288, in concat
    op = _Concatenator(
  File "/localdisk/dchigare/repos/pandas/pandas/core/reshape/concat.py", line 345, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate

Since #37227, aggregation with empty dictionary started to raise ValueError (previously it return an empty frame) which comes from .concat when trying to concat results:

pandas/pandas/core/apply.py

Line 374 in cb1486c

result = concat({k: results[k] for k in keys_to_use}, axis=axis)

Since results are empty, it's logical that it raise exception. But the logic that was touched by #37277 doesn't touch that .concat part, so the behavior should stay the same - return an empty frame.

Previously it goes to the concat branch if any(cond1(x) for x in agg_dict) and now it's all(cond2(x) for x in agg_dict). cond1 and cond2 was written in that way so the condition with any and all are technically the same. But the difference between any and all is that they give the different results on an empty sequences:

>>> all(x for x in [])
True
>>> any(x for x in [])
False

So after any was changed to all it gets to the concat branch even if results are empty and gets an error on .concat.

So the question is, is it desired behavior that now aggregation raises exception on empty dict? And if it is, should it catch these cases in agg_dict_line and give a nicer error?

[rudolfbyker]

If we follow semantic versioning strictly, then this is a regression. A minor version increase should not break any existing API-contract.

Example code:

import pandas as pd

if __name__ == '__main__':
    pd.show_versions()
    print()

    df = pd.DataFrame(data=[{'a': 1}])
    g = df.groupby(by='a')
    a = g.aggregate({})
    print(a)

Output in version 1.1.3:

INSTALLED VERSIONS
------------------
commit           : db08276bc116c438d3fdee492026f8223584c477
python           : 3.8.10.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.19042
machine          : AMD64
processor        : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : English_South Africa.1252

pandas           : 1.1.3
numpy            : 1.20.3
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.1.1
setuptools       : 52.0.0.post20210125
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

Empty DataFrame
Columns: []
Index: []

Output in version 1.2.4:

INSTALLED VERSIONS
------------------
commit           : 2cb96529396d93b46abab7bbc73a208e708c642e
python           : 3.8.10.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.19042
machine          : AMD64
processor        : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : English_South Africa.1252

pandas           : 1.2.4
numpy            : 1.20.3
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.1.1
setuptools       : 52.0.0.post20210125
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

Traceback (most recent call last):
  File "...", line 9, in <module>
    a = g.aggregate({})
  File "...\site-packages\pandas\core\groupby\generic.py", line 945, in aggregate
    result, how = aggregate(self, func, *args, **kwargs)
  File "...\site-packages\pandas\core\aggregation.py", line 582, in aggregate
    return agg_dict_like(obj, arg, _axis), True
  File "...\site-packages\pandas\core\aggregation.py", line 782, in agg_dict_like
    result = concat({k: results[k] for k in keys_to_use}, axis=axis)
  File "...\site-packages\pandas\core\reshape\concat.py", line 285, in concat
    op = _Concatenator(
  File "...\site-packages\pandas\core\reshape\concat.py", line 342, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate

[jbrockmendel]

@rhshadrach i think this can be answered with a "yes" or "no" from you

[rhshadrach]

I am seeing DataFrameGroupBy.agg with [] not raise, where as all other ops below currently raise. It does seem to me like we should make this consistent, and it seems to me we can handle this edge case in a straightforward way. I'm favorable of not raising here.

Below, when an index/columns is empty, I think it should have object dtype for now (and in the future, void dtype).

.agg([]):

• Series.agg - A Series with empty index, name the same as the input.
• DataFrame.agg - a DataFrame with empty index, same columns as the input.
• SeriesGroupBy.agg - a DataFrame with index given by result_index and empty columns.
• DataFrameGroupBy.agg - a DataFrame with index given by result_index and columns an empty MultiIndex.

.agg({}):

• Series.agg - A Series with empty index, name the same as the input.
• DataFrame.agg - a DataFrame with empty index and empty rows.
• SeriesGroupBy.agg - Raises pandas.errors.SpecificationError: nested renamer is not supported
• DataFrameGroupBy.agg - a DataFrame with index given by result_index and empty columns.