Simplified Grammar: Why Less Complexity is Better

[lalvarezt]

Simplified Grammar: Why Less Complexity is Better

The refactor made in branch simplified-grammar simplifies the template parser by removing expensive lookahead operations and consolidating multiple context-specific argument parsers into a single, unified approach.

The Problem

The previous grammar implementation attempted to be "smart" by using context-specific parsing rules:

• simple_arg for basic operations (append, prepend, join)
• regex_arg for regex patterns
• split_arg for split separators
• map_regex_arg for regex within map operations

Each parser had different lookahead logic to determine when special characters (|, :, {, }) should be treated as literals versus syntax. This required extensive operation_keyword lookahead and complex negative assertions like:

split_content = { !(":" ~ (number | range_part)) ~ !("|" ~ operation_keyword) ~ !("}" ~ EOI) ~ ANY }

The Solution

The refactor introduces a single unified argument parser:

argument     = { (escaped_char | normal_char)* }
normal_char  = { !("|" | "}" | "{" | ":" | "\\") ~ ANY }
escaped_char = { "\\" ~ ANY }

All operations now use the same escaping rules. Special characters require explicit escaping - no exceptions, no context-dependent behavior.

Why This is Better

Performance: Lookahead operations are expensive. Removing them provides immediate parsing performance improvements, especially for complex templates.

Maintainability: One set of rules to understand, test, and maintain instead of four context-specific parsers.

Predictability: Users now have clear, consistent rules. If a character is special (|, :, {, }, \), escape it. No need to understand parser internals or memorize context-specific exceptions.

Explicitness: Templates become more intentional. Compare:

{split:|:0}           # old: relies on smart parsing
{split:\|:0}          # new: explicit intent

The second version clearly shows what's a separator and what's syntax.

Robustness: Fewer edge cases means fewer bugs. Complex lookahead logic often fails in corner cases that are hard to predict.

The Trade-off

This is a breaking change. Templates that previously relied on smart escaping need updates:

• Pipes in split separators: {split:|:0} → {split:\|:0}
• Colons in regex patterns: {regex_extract:Version: (\d+):1} → {regex_extract:Version\: (\d+):1}

However, the migration path is straightforward, and the long-term benefits of a simpler, faster, more predictable parser far outweigh the one-time update cost.

[lalvarezt]

@alexpasmantier any thoughts?