Lex failed to parse last unterminated directive

[htv2012]

Describe the bug
If the last directive is not terminated by a semicolon ( ; ), then lex drops the last token.

To Reproduce
Steps to reproduce the behavior:

1. Create an nginx config file with 'user www www;\nworker_processes 2'
2. Run crossplane lex on nginx config

Output:

["user","www","www",";","worker_processes"]

Expected behavior
I expect to see the "2" token.

Your environment

• macOS Sequoia 15.3
• crossplane v0.5.8

Additional context
If I drop the last "2" from the config file, the output becomes:

["user","www","www",";"]

I interpreted this behavior as the lexer always dropping the last token if the last directive is not terminated with a semicolon.

Going down stream to parse: When I feed the above faulty configuration file to parse, here is what I got:

$ crossplane parse samples/faulty.conf | jq .                                                                [nginx-conf-tool:add-tree:]
{
  "status": "failed",
  "errors": [
    {
      "file": "samples/faulty.conf",
      "error": "",
      "line": null
    }
  ],
  "config": [
    {
      "file": "samples/faulty.conf",
      "status": "failed",
      "errors": [
        {
          "error": "",
          "line": null
        }
      ],
      "parsed": []
    }
  ]
}

Notice the empty error. I expect to see better error messages. I also expect to see line number where the error happened.