The Template Language

This blog is a part 2 of my bluebird blog. See part 1

For my twitter clone without dependencies, I've decided to design my route handlers in a way that they'll return variables and a template name. This allows for easy testing, as my tests can just do assertions on the template name and variables rather than inspecting a HTML document.

// request
{
  method: "GET",
  path: "/profile/1234",
  cookies: { "user-id": 54 },
}

// response
{
  status: 200,
  template: "public-profile-show",
  variables: {
    user: {
      id: 54,
      name: "John Doe",
    },
    posts: [
      { id: 55412, message: "Have you seen the new iThing?",
        createdAt: 1699788972 }
    ]
  }
}

In this blog, I'm going to implement this template language.

Designing the template language

The template language I need, needs to output HTML documents with only a set of variables as input. I want a template to be compiled to a JS function. For example, I want Hello <%= name %> to compile to something like this:

({ name }) => `Hello ${escapeHtml(name)}`;

I'll go with a classic <%= %> syntax, because it's incredibly common and well known. Most developers that see this syntax will intuitively know that you can just write regular code in there and the output of the code will be added to the output.

It must support variables and auto-escape HTML entities. Loops, if/else statements and including other templates must also be supported. It would be nice if we could invoke arbitrary functions and do some basic math;

So basically, I want it to be able to execute arbitrary javascript.

The implementation

I guess I just get started writing code and see where I end up. First, a test.

it("simple template", () => {
  const fn = Template.parse("Hello, <%= name %>");
  assert.equal(fn({ name: "world" }), "Hello, world");
});

The simplest implementation I can think of is to use a regular expression. All content outside <%= %> will just be added to the output, and the content between will be executed as JS.

The regular expression used is /(.*?)<%(.*?)%>/sg. This regular expression captures any text until the first <% it finds using (.*?)<%. Then it captures any text until %> using (.*?)%>. The s modifier allows the . (dot) to match newlines. The g modifier allows multiple matches.

Javascript's replace function on string allows executing code for every match while also returning a replacement value, "" in my code. Because every match is replaced by an empty string, only the text after the last %> is returned by the replace-function, which I call the tail.

I use JSON.stringify to create a string literal.

const Template = {
  parse(template) {
    let body = [
      "eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);",
      `out = [];`
    ];
    const tail = template.replace(/(.*?)<%(.*?)%>/sg, (_, content, code) => {
      body.push(`out.push(${JSON.stringify(content)});`);

      if (code.startsWith("="))
        body.push(`out.push(${code.substr(1)});`);

      return "";
    });

    body.push(`out.push(${JSON.stringify(tail)});`);
    body.push(`return out.join("");`);

    return new Function('vars', body.join("\n"))
  }
};

For the template in the test, this function returns a function like this:

function(vars) {
  eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);
  out = [];
  out.push("Hello, ");
  out.push(name);
  out.push("");
  return out.join("");
}

Another notable part of this code is the eval statement. To allow the template to refer to any variable in vars (name in this example), I need to make the properties in vars available as local variables in the function.

There is no easy way to determine the possible variables while compiling, so I generate them at runtime. The only way I know to assign arbitrary local variables at runtime, is to use eval.

// request
{
  method: "GET",
  path: "/profile/1234",
  cookies: { "user-id": 54 },
}

// response
{
  status: 200,
  template: "public-profile-show",
  variables: {
    user: {
      id: 54,
      name: "John Doe",
    },
    posts: [
      { id: 55412, message: "Have you seen the new iThing?",
        createdAt: 1699788972 }
    ]
  }
}

Another method is to use the with-statement, which is discouraged. Let's try it anyway.

({ name }) => `Hello ${escapeHtml(name)}`;

The generated function works perfectly. Too bad the feature is discouraged, legacy or deprecated, depending on who you ask. So far my options are evil eval or deprecated with. Ideally, I want to determine the variables used at compile-time, but this requires compiling the Javascript code to determine the variables used.

There is no easy way to get an abstract syntax tree of some piece of Javascript using plain NodeJS.

Now to escape HTML entities, support if/else statements and add some minor fixes.

it("simple template", () => {
  const fn = Template.parse("Hello, <%= name %>");
  assert.equal(fn({ name: "world" }), "Hello, world");
});

I also added some more tests.

const Template = {
  parse(template) {
    let body = [
      "eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);",
      `out = [];`
    ];
    const tail = template.replace(/(.*?)<%(.*?)%>/sg, (_, content, code) => {
      body.push(`out.push(${JSON.stringify(content)});`);

      if (code.startsWith("="))
        body.push(`out.push(${code.substr(1)});`);

      return "";
    });

    body.push(`out.push(${JSON.stringify(tail)});`);
    body.push(`return out.join("");`);

    return new Function('vars', body.join("\n"))
  }
};

To allow includes, I'm going to add a function to parse all template files in a directory. This function will keep a dictionary with template names as keys and their parsed template functions as values.

src/template.mjs

function(vars) {
  eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);
  out = [];
  out.push("Hello, ");
  out.push(name);
  out.push("");
  return out.join("");
}

test/templates/**.ejs

var { foo } = { foo: 1 };
// foo = 1
eval('var { bar } = { bar: 2 }');
// bar = 2

test/template.test.mjs

function(vars) {
  with (vars) {
    out = [];
    out.push("Hello, ");
    out.push(name);
    out.push("");
    return out.join("");
  }
}

Now to integrate this template engine in the main.mjs file to render the template using the .ejs templates.

templates/home.ejs

// Template.parse
let body = [
  "eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);",
  `out = [];`
];
const tail = template.replace(/(.*?)<%(.*?)%>/sg, (_, content, code) => {
  if (content)
    body.push(`out.push(${JSON.stringify(content)});`);

  if (code.startsWith("="))
    body.push(`out.push(escapeHtml(${code.substr(1)}));`);
  else if (code.startsWith("-"))
    body.push(`out.push(${code.substr(1)});`);
  else
    body.push(code);

  return "";
});

if (tail.length > 0) body.push(`out.push(${JSON.stringify(tail)});`);

body.push(`return out.join("");`);
body = body.join("\n");

const fn = new Function('vars', body);
return (vars) => fn({ ...vars, ...Template.locals });

// Template.locals
locals: {
  escapeHtml: (str) => `${str}`.replace(/[<>&"']/g, s =>
    ({ "<": "<", ">": ">", "&": "&amp;", '"': "&quot;", "'": "&#39;" })[s])
}

src/main.mjs

describe("Template.parse", () => {
  it("simple template", () => {
    const fn = Template.parse("Hello, <%= name %>");
    assert.equal(fn({ name: "world" }), "Hello, world");
  });

  it("math template", () => {
    const fn = Template.parse("1 + 1 = <%= 1 + 1 %>");
    assert.equal(fn({}), "1 + 1 = 2");
  });

  it("function template", () => {
    const fn = Template.parse("Hello <%= foo() %>");
    assert.equal(fn({ foo: () => "world" }), "Hello world");
  });

  it("if-else template", () => {
    const fn = Template.parse(`Answer: <% if (answer) { %>Yes<% } else { %>No<% } %>`);
    assert.deepEqual(
      [fn({ answer: true }), fn({ answer: false })],
      ["Answer: Yes", "Answer: No"]);
  });

  it("multiline template", () => {
    const fn = Template.parse(`
    Answer:
    <% if (answer) { %>
      Yes
    <% } else { %>
      No
    <% } %>
  `);
    assert.deepEqual(
      [delws(fn({ answer: true })), delws(fn({ answer: false }))],
      ["Answer: Yes", "Answer: No"]);
  });

  it("escape html", () => {
    const fn = Template.parse("Hello, <%= name %>");
    assert.equal(fn({ name: "<script> & \" '" }), "Hello, <script> &amp; &quot; &#39;");
  });
});

function delws(str) {
  return str.replace(/^\s+|\s+$/g, "").replace(/\s+/g, " ");
}

export const Template = {
  /**
   *
   * @param {string} path Directory containing one or more template files
   * @returns {Promise<(template: string, vars: Record<string, any>) => string>}
   */
  async parseDirectory(path) {
    /** @type {Map<string, function>} */
    const templates = new Map();

    const include = (templateName, vars) => {
      const template = templates.get(templateName);
      if (!template) throw new Error(`Template ${path}/${templateName} does not exist, `
        + `templates found: ${Array.from(templates.keys()).join(", ")}`);
      return template({ ...vars, include });
    };

    const readDir = async (prefix) => {
      const innerPath = join(path, prefix);
      const fileNames = await promises.readdir(join(innerPath));

      for (const fileName of fileNames) {
        const templateName = join(prefix, fileName);
        const filePath = join(innerPath, fileName);
        if (fileName.endsWith(".ejs")) {
          const body = await promises.readFile(filePath, { encoding: "utf-8" });
          templates.set(templateName, Template.parse(body, { filePath }));
        } else if ((await promises.stat(filePath)).isDirectory) {
          await readDir(join(prefix, fileName));
        }
      }
    };

    await readDir("");

    return include;
  }
}

Now we're ready to start writing our application, which will continue in the next blog

The above is the detailed content of Template language without dependencies. For more information, please follow other related articles on the PHP Chinese website!