Skip to content

fix(md-lex): ignore lang tag after opening code block#26

Open
ardnew wants to merge 2 commits into
goccmack:masterfrom
ardnew:fix/md-lex-lang-tag
Open

fix(md-lex): ignore lang tag after opening code block#26
ardnew wants to merge 2 commits into
goccmack:masterfrom
ardnew:fix/md-lex-lang-tag

Conversation

@ardnew

@ardnew ardnew commented Dec 31, 2025

Copy link
Copy Markdown

When parsing rules from Markdown files, the language tag following the opening fence should not be included as part of the grammar definition (resulting in a syntax error). More on fenced code blocks.

For example, the "gogll" at the top:

    ```gogll

    package "foo"

    ident:
       letter { letter | number | '_' } ;
    
    ```

Previously, this example would fail with an error like:

$ gogll -v grammar.md
Parse Errors:
Parse Error: GoGLL : ∙Package Rules  I[0]=tokid (156,161) gogll at line 7 col 4
Expected one of: [package]

With this PR, everything on the same line following the opening backticks is ignored (up until the first \n). This means whitespace can be part of a language tag — which is apparently allowed.

Copilot AI review requested due to automatic review settings December 31, 2025 21:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug in the markdown lexer where language tags following opening code fence backticks were incorrectly included as part of the grammar definition, causing parse errors. The fix ensures that everything on the same line after the opening ``` is ignored (replaced with spaces) until the first newline character.

Key Changes:

  • Added logic to skip language tags after opening code block fences in markdown files
  • The fix allows markdown files to use standard fenced code blocks with language identifiers (e.g., ```gogll)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
lexer/lexer.go Added logic in loadMd function to skip content after opening backticks until newline, preventing language tags from being parsed as grammar
gen/golang/lexer/lexer.go Applied identical fix to the generated lexer template, ensuring consistency across generated code

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread gen/golang/lexer/lexer.go
Comment thread lexer/lexer.go
@ardnew

ardnew commented Jan 19, 2026

Copy link
Copy Markdown
Author

Resolved all review comments with test cases implemented in 5bce663

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants