Building Your Own Language in Monaco Editor: Tokenizers, Embedded Cells and Semantic Tokens

By /

Monaco — the editor that powers VS Code — ships with grammars for the usual suspects. But if your product has its own dialect, you end up needing a Monaco Editor custom language: your own tokenizer, your own autocomplete, your own squiggly underlines.

We did exactly this for FoxSchema's SQL Editor. Our language, FoxScript, is SQL-first but lets you drop into TypeScript or JavaScript mid-file. Here's how the pieces fit together, and the parts that were less obvious than the docs suggest.

1. Register the language and give it a tokenizer

Everything starts with an id. Register it once, then attach a Monarch tokenizer — a declarative state machine that turns text into token types.

monaco.languages.register({ id: 'foxscript', aliases: ['FoxScript'] });

monaco.languages.setMonarchTokensProvider('foxscript', {
  keywords: ['SELECT', 'FROM', 'WHERE', 'JOIN'],
  tokenizer: {
    root: [
      [/--.*$/, 'comment'],
      [/'[^']*'/, 'string'],
      [/\b\d+\b/, 'number'],
      [/[a-zA-Z_]\w*/, { cases: { '@keywords': 'keyword', '@default': 'identifier' } }],
    ],
  },
});

A tip that saved us a lot of work: you don't have to write a grammar from scratch. Monaco already knows SQL, so we extend the built-in definition — spread its tokenizer, then prepend our own rules to root. You inherit every keyword and string rule for free and only maintain the delta.

2. Language configuration — the small stuff users notice

The tokenizer only colours text. Bracket matching, comment toggling and auto-closing quotes come from setLanguageConfiguration. It is a handful of lines and it is the difference between "a textarea with colours" and "an editor".

monaco.languages.setLanguageConfiguration('foxscript', {
  comments: { lineComment: '--', blockComment: ['/*', '*/'] },
  brackets: [['(', ')'], ['[', ']']],
  autoClosingPairs: [
    { open: '(', close: ')' },
    { open: "'", close: "'", notIn: ['string'] },
  ],
});

3. Embedding another language inside yours

This was the interesting problem. We wanted a SQL file that can contain real TypeScript cells:

SELECT * FROM customers;

-- @@typescript
const top = last.rows.slice(0, 10);
console.log(top);
-- @@end

Monarch supports this directly through nextEmbedded. When a rule matches your opening fence, you tell Monaco to switch to another language's tokenizer, and @pop it back at the closing fence:

tokenizer: {
  root: [
    [/^[ \t]*--[ \t]*@@(?:typescript|ts)[ \t]*$/,
      { token: 'comment.fence', next: '@tsEmbedded', nextEmbedded: 'text/typescript' }],
    ...sqlRules,
  ],
  tsEmbedded: [
    [/^[ \t]*--[ \t]*@@end[ \t]*$/,
      { token: 'comment.fence', next: '@pop', nextEmbedded: '@pop' }],
    [/.*$/, ''],
  ],
}

Two gotchas cost us time. First, nextEmbedded takes a MIME type (text/typescript), not the language id. Second, if you write the pattern as a JavaScript regex literal, Monarch treats a bare @ as an attribute reference and your rule silently never fires — use the string form and escape it as @@.

4. Real IntelliSense inside the embedded cells

Embedding gets you syntax colours, but not completion — Monaco's TypeScript service knows nothing about a fenced region inside a SQL file.

The trick is virtual documents. For each fence, create an in-memory model containing just that cell's body, and prepend a typed prelude that declares the variables your runtime injects:

const uri = monaco.Uri.parse(`inmemory://foxscript/cell-${index}.ts`);
const content = PRELUDE + '\n' + cellBody;
const model = monaco.editor.getModel(uri) ?? monaco.editor.createModel(content, 'typescript', uri);

Your hover and completion providers then project the cursor position from the real document into the virtual one and ask Monaco's TypeScript service. Users get typed completions on your host objects without you writing a JavaScript parser.

One honest limitation: full TS diagnostics inside cells need the TypeScript worker loaded. If you ship a slimmed Monaco bundle (we ship SQL-only to keep it lean), you get completion and hover but not full type-checking — so we cover fence structure with our own markers instead.

5. Completions that know your data

Generic keyword completion is table stakes. The useful version is context-aware: after FROM, suggest tables; after alias., suggest that table's columns. Because we already introspect the connected database for schema comparison, the editor completes real table and column names.

monaco.languages.registerCompletionItemProvider('foxscript', {
  triggerCharacters: ['.', ' '],
  provideCompletionItems(model, position) {
    const word = model.getWordUntilPosition(position);
    const range = { startLineNumber: position.lineNumber, endLineNumber: position.lineNumber,
                    startColumn: word.startColumn, endColumn: word.endColumn };
    return {
      suggestions: tables.map((t) => ({
        label: t.name,
        kind: monaco.languages.CompletionItemKind.Struct,
        insertText: t.name,
        range,
      })),
    };
  },
});

Always return an explicit range. Omit it and Monaco guesses the replacement span, which produces duplicated text the moment a user completes mid-identifier.

6. Semantic tokens: colour what actually exists

Monarch is regex-based, so it cannot tell a real table name from any other identifier. A semantic tokens provider runs after tokenizing and can colour identifiers using knowledge the regex doesn't have — in our case, the live schema cache. Names that genuinely exist in the connected database get highlighted; typos stay plain, which is a quiet form of validation.

monaco.languages.registerDocumentSemanticTokensProvider('foxscript', {
  getLegend: () => ({ tokenTypes: ['type', 'variable'], tokenModifiers: [] }),
  provideDocumentSemanticTokens(model) {
    return { data: buildTokenData(model, schemaCache) };
  },
  releaseDocumentSemanticTokens: () => undefined,
});

The encoding is the fiddly bit: a flat Uint32Array of five numbers per token — line delta, start delta, length, type index, modifiers — all relative to the previous token.

7. Diagnostics

Errors are just markers on the model. Parse, then publish:

monaco.editor.setModelMarkers(model, 'foxscript', [{
  severity: monaco.MarkerSeverity.Error,
  message: 'Unterminated code fence — expected `-- @@end`',
  startLineNumber: line, endLineNumber: line, startColumn: 1, endColumn: 80,
}]);

Use a consistent owner string ("foxscript" here). Monaco replaces all markers for that owner on each call, so you never have to clear stale ones yourself.

What we'd tell you before you start

  • Extend, don't rewrite. Inheriting Monaco's SQL grammar removed most of the work.
  • Ship a slim bundle deliberately. Monaco is large; loading every language costs real download. Know which features you lose.
  • Semantic tokens are worth it. Highlighting only names that exist catches typos before running anything.
  • Test the tokenizer. Monarch fails silently — a bad rule doesn't throw, it just never matches.

The result is a SQL editor that autocompletes your actual schema and lets you drop into TypeScript when SQL alone isn't enough. You can try it in FoxSchema — see the user guide for a walkthrough of the SQL Editor, or install it and open the editor against your own database. The whole implementation is open source on GitHub.

Scroll to Top