Preprocessing with YAML Macros

Syntax definitions can have a lot of repetitive elements to them. Sometimes, these elements are simple enough that you can simple include a utility context:

contexts:
  else-pop:
    - match: (?=\S)
      pop: true

  order-expression:
    - match: (?i)\b(?:ASC|DESC|NULLS|FIRST|LAST)\b
      scope: keyword.other.sql
    - include: else-pop

But this example contains another common SQL idiom: keywords are always case-insensitive and surrounded by word breaks, so the syntax will contain many repetitions of the (?i)\b(?:...)\b pattern. A typo could easily slip into one such repetition. So why write it yourself each time? Instead, use YAML tags and a preprocessor:

  order-expression:
    - match: !word ASC|DESC|NULLS|FIRST|LAST
      scope: keyword.other.sql
    - include: else-pop

The macro:

# macros.py

def word(match):
    return r'(?i)\b(?:%s)\b' % match

The engine:

# build.py

import yaml, sys
from os import path

filename = sys.argv[1]

output_path, extension = path.splitext(path.basename(filename))

if extension != '.source': raise "Not a .source file!"

input_file = open(filename, 'r')
output_file = open(output_path, 'w')

import macros

PREAMBLE = '%YAML 1.2\n---\n'

def apply_transform(loader, node, transform):
    try:
        if isinstance(node, yaml.ScalarNode):
            return transform(loader.construct_scalar(node))
        elif isinstance(node, yaml.SequenceNode):
            return transform(*loader.construct_sequence(node))
        elif isinstance(node, yaml.MappingNode):
            return transform(**loader.construct_mapping(node))
    except TypeError as e:
        print('Failed to transform node: {}\n{}'.format(str(e), node))

def get_constructor(transform):
    return lambda loader, node: apply_transform(loader, node, transform)

for name, transform in macros.__dict__.items():
    if callable(transform):
        yaml.add_constructor(
            '!'+name.lstrip('_'),
            get_constructor(transform)
        )

syntax = yaml.load(input_file)

output_file.write(PREAMBLE)
yaml.dump(syntax, output_file)

Another example:

def meta(name):
    return [
        { 'meta_scope': 'meta.%s.sql' % name, },
        { 'match': '', 'pop': True },
    ]

block:
  - match: !word BEGIN
    scope: keyword.control.sql
    push:
      - !meta block
      - statements

We can go further. In Oracle SQL, any identifier may be wrapped in optional double quotes:

create table mytable ... ;
create table "mytable" ... ;

Implementing both versions is possible:

  expect-table-name:
    - match: \b{{ident}}\b
      scope: entity.name.table.sql
      pop: true
    - match: (")([^"]+)(")
      captures:
        1: punctuation.definition.string.begin.sql
        2: entity.name.table.sql
        3: punctuation.definition.string.end.sql
      pop: true
    - include: else-pop

But then we have to do this for every single type of identifier -- procedure names, aliases, variables, etc. So we write a macro for it:

def expect_identifier(scope):
    return [
        { 'match': r'\b{{ident}}\b', 'scope': scope, 'pop': True },
        {
            'match': r'(")([^"]+)(")',
            'scope': 'string.quoted.double.sql',
            'captures': {
                '1': 'punctuation.definition.string.begin.sql',
                '2': scope,
                '3': 'punctuation.definition.string.end.sql',
            },
            'pop': True,
        },

        { 'match': r'(?=\S)', 'pop': True },
    ]

Define all of these scopes the same way:

  expect-table-name: !expect_identifier entity.name.table.sql
  expect-alias: !expect_identifier entity.name.alias.sql

Or just use the macro inline:

declarations:
  ...
  - match: !word TYPE
    scope: storage.type.sql
    push:
      - !meta declaration.type
      - type-definition-value
      - !expect_keyword IS
      - !expect_identifier entity.name.type.sql
  ...

Using macros, you can define very complicated constructs in a compact fashion that is easy to understand and reasonably robust against invalid input:

- match: !word FOR
  scope: keyword.control.sql
  push:
    - !meta control.for
    - !expect_keyword LOOP
    - !expect_keyword END
    - statements
    - !expect_keyword LOOP
    - expression
    - !expect [ \.\., keyword.operator.other ]
    - expression
    - !expect_keyword REVERSE
    - !expect_keyword IN
    - !expect_identifier variable.other.sql

Syntax Development Tips/Advice #757

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions