DEPR: DataFrame.combine propagates nan values

[aktiur]

The documentation for DataFrame.combine claims that the method "do[es] not propagate NaN values, so if for a (column, time) one frame is missing a value, it will default to the other frame’s value". However, this does not seem to correspond to the actual behaviour of DataFrame.combine.

Sample code:

>>> import pandas as pd
>>> from operator import add
>>> a = pd.DataFrame({
...        'a': pd.Series([1, 3], index=[0, 1]),
...        'b': pd.Series([2, 3], index=[1, 2]),
...    })
>>> b = pd.DataFrame({
...         'a': pd.Series([3, 5], index=[1, 2]),
...         'c': pd.Series([1, 2, 3], index=[2, 3, 4])
...     })
>>> a.combine(b, add)
    a   b   c
0 NaN NaN NaN
1   6 NaN NaN
2 NaN NaN NaN
3 NaN NaN NaN
4 NaN NaN NaN

In many cases (as in this one), it may be remedied by using the fill_value parameter of DataFrame.combine. However, it might be a problem when there is no acceptable neutral element for the given combining function.

>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: Windows
OS-release: 8
machine: AMD64
processor: Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.16.2
nose: 1.3.7
Cython: 0.22
numpy: 1.9.2
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.1.0
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.4
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.6.7
lxml: 3.4.2
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.6
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)

[jreback]

you can simply do this:

In [13]: a.add(b)
Out[13]: 
    a   b   c
0 NaN NaN NaN
1   6 NaN NaN
2 NaN NaN NaN
3 NaN NaN NaN
4 NaN NaN NaN

or

In [31]: a.add(b,fill_value=0)
Out[31]: 
    a   b   c
0   1 NaN NaN
1   6   2 NaN
2   5   3   1
3 NaN NaN   2
4 NaN NaN   3

maybe explain what the issue you are seeing. Essentially you are doing this:

In [32]: a1, b1 = a.align(b)

In [33]: a1
Out[33]: 
    a   b   c
0   1 NaN NaN
1   3   2 NaN
2 NaN   3 NaN
3 NaN NaN NaN
4 NaN NaN NaN

In [34]: b1
Out[34]: 
    a   b   c
0 NaN NaN NaN
1   3 NaN NaN
2   5 NaN   1
3 NaN NaN   2
4 NaN NaN   3

so in this example is IS working as adverisied. The NaN's are coming from the addition.

[jorisvandenbossche]

@jreback you are fully correct that those (a.add(b) or a.add(b, fill_value=0)) are the logical functions to use instead of combine.

But, still, it is not really clear what the docs of combine even mean with:

Add two DataFrame objects and do not propagate NaN values, so if for a (column, time) one frame is missing a value, it will default to the other frame’s value (which might be NaN as well)

What does 'do not propagate NaN values' mean? What if the result of the function at a certain place in the frame is NaN, what should it do? Not propagate? (but this is what it does) Default to the value in other? So this would mean that in a11 + b11 it gives you b11 if a11 is NaN, but NaN if b11 is NaN and a11 not .. (seems like a very strange operation).

In any case, docstring can use an update, as it does not seem correct.

[jreback]

yeh doc-string prob not clear. .combine is not really used much (except by .combine_first). Its the logical equivalent of .apply but for binary functions.

Maybe we should just deprecate this.

[jsevo]

+1 to deprecate. This is pretty confusing. It also has no explanation what func ought to be. Is combine with overwrite=True equivalent to update? Not a very clear situation for what must be pretty common operations.

[auvipy]

I want to depricate this. how to do so? never contributed to pandas before!

[jreback]

look at how .combineAdd is done. note that we can't do this until 0.19.0.