split_hyphen_unanalyzable

scripts.fix_tokenization.split_hyphen_unanalyzable(i, terminals, preterminals, preterminal_parents, state)[source]

split token with hyphens to unanalyzable tokens. xxx-xxx-xx -> xxx - xxx - xx