Whitespace incorrectly treated as tangible part of macro argument preprocessing tokens

[notfoundry]

While playing around with some macros in messer (great work by the way! I love the feel of it), I noticed that whitespace characters seem to be considered part of the preprocessing tokens used as macro arguments. The result of preprocessing the following source file loses whitespace information when preprocessed on GCC, Clang, MSVC, and ICC (see https://godbolt.org/z/P3x53cGjc), but retains that information in Messer v0.3.0.

#define IDENTITY(x) x

IDENTITY(a     b     c)

This would seem to contradict section 19.3.11 of the C++17 standard, which states that before the arguments to a function-like macro are substituted, "the sequence of preprocessing tokens bounded by the outside-most matching parentheses forms the list of arguments for the function-like macro", where "preprocessing tokens" are defined in annex A.2 of the standard as explicitly not including whitespace. Boost.Wave handles this distinction by storing the arguments to a function-like macro invocation as a List of Lists of Tokens, where each inner list corresponds to the non-whitespace tokens of each respective argument.

A consequence of whitespace being significant is that token-pasting macros can become subtly broken when a macro argument contains any whitespace. When the following source file is preprocessed on the same four compilers as before, the result of expanding ADJACENT_CAT(FOO,,BAR) is PASS (see https://godbolt.org/z/Exqha6dP1). Messer will produce FOO BAR as the expansion result for the same source file, however.

#define CAT(a,b) a##b
#define ADJACENT_CAT(a,b,c) CAT(a b,c)

#define FOOBAR PASS
ADJACENT_CAT(FOO, ,BAR)

I'm not sure how labor-intensive a fix for this sort of thing might be, but considering how the README explicitly lists "conforming to C++17 standard" as a selling point, I feel that it's worth pointing this out.

[wx257osn2]

Hi, @notfoundry

Thank you for trying Messer, and pointed out the issue!

The result of preprocessing the following source file loses whitespace information when preprocessed on GCC, Clang, MSVC, and ICC (see https://godbolt.org/z/P3x53cGjc), but retains that information in Messer v0.3.0.
This would seem to contradict section 19.3.11

The behavior of messer which retains whitespaces information has been one of design decisions (and we plan to add the option to change the behavior to replace whitespaces by one space character), but in the context of function-like macro invocation, replacing the whitespaces is the mandatory process.
This is the bug, so we will fix this.

A consequence of whitespace being significant is that token-pasting macros can become subtly broken when a macro argument contains any whitespace.

Actually, the half of the problem is due to the handling of whitespaces, but the other half should be due to the implementation of the operator ## .
Anyway, we think that this is more critical bug rather than above one, and we will also fix this.

As you said, we advocate that messer conforms to C++17 standard, so we will fix these problem to avoid untrue.

We will mention to you here when we will fix the problem.

Best regards.

[wx257osn2]

@notfoundry The second problem that messer has expanded ADJACENT_CAT(FOO, ,BAR) as FOO BAR has been fixed! You can try the fixed one to build from source on master branch.

We also working continuously about the first problem, current messer retains whitespaces information.

Best regards.