Tokenize spaces when explicitly requested and reduce iterations in occurrence counting loop#6

Open

colin-brown wants to merge 1 commit intoandreekeberg:masterfrom

colin-brown:tokenizer-improvements

Commits on Aug 4, 2021

Added test to ensure character arrays that include spaces are tokenized as expected. Improved tokenizer to 1. create tokens for spaces when supplied array includes them 2. optimize occurance counti…
colin-brown
committed