import/export in browser AND node

I had a parser project with three main files that I wanted to turn into an html repl. This was initially a nodejs project and so I needed to make things work in the browser now, too. I decided to try experimentally supported es-modules which uses the import/export syntax to see how far I could get this way. It would allow me to keep my sources fairly as-is and have the same source in nodejs and in the browser. Sort of.

The premis

The "es modules" stem from the ES6 era, released in 2015, which introduced a syntactical way of importing and exporting files as modules. If you're reading this you're probably well aware of the details so I'll skip the details on this.

Despite being three years old, environments are still catching up now. And even though caniuse looks pretty optimistic there still a lot of caveats to actually using them.

Conversion

The project itself used requires ("commonjs"). This doesn't work in the browser so you can either strip the requires and include them with a script tag, or fake the require function with some boilerplate to get things to work.

I didn't see a nice way around keeping the sources as is while using them both in node and the browser without an explicit build step. So that's when I looked into using es modules. I knew nodejs started adding experimental support for them and that browsers had done the same thing. How bad could it be?

Well.

Layout

This project has three main js source files. The repl has an html and js file. There are also a ton of js test files, but that shouldn't be relevant for the repl.

I run a server through python's SimpleHttpServer, which is one of the webserver one liners that'll just work without any further installs. Primitive but it gets the job done. Most of the time.

Conversion

The first thing I did was convert the three main files. The require would become import and the module.exports would become exports. This was fairly easy.

I setup the html and added the script tag with type="module" and low and beho... nothing. Chrome showed an inifini loading spinner in the browser tab, no further feedback. The python server seemed to be happy. What the hell. Firefox didn't work either.

Turns out that there's an issue with the simple webserver where multiple requests were causing a deadlock without feedback. You can unlock them by pressing ctrl+c in the terminal, after which the page loads file (so what's cancelled then?).

This initially led to a pretty quick proof of concept that worked. Nice!

Back to node

Unfortunately that's when I tried running my test runner in nodejs. Everything was broken because node doesn't really know about the module goal.

The feature is tucked away behind an experimental flag. You have to run it like this:

node --experimental-modules foo.js

And not just that, you also have to rename the files to use the .mjs extension.

Once you do this for the main entry point you quickly discover this process kind of "taints" the other files. You import one module and it imports another and it all wants to use the import syntax. Apparently there's supposed to be this cross esm/commonsjs support but I don't think that worked for me. So I had to port all the files in the project, including tests, to use import/export. Sigh.

Furthermore, since node currently requires .mjs for es modules at all, all the files had to be renamed. Sigh. Ok, fine.

Implicit globals

The next problem turned up; things like require and dirname were no longer auto-defined. Gah.

So __dirname is at least an easy fix;

let filePath = import.meta.url.replace(/^file:\/\//,'');
let dirname = path.dirname(filePath);

This seems to work in node v10.11.0 and I suspect that'll be fairly solid.

The require is a little trickier because it does something that import declarations currently cannot do: conditional inclusion. Or even detection.

So I would do lazy loading of helper functions when they were needed, like require('util').inspect(root, false, null) for logging. That wouldn't work anymore and I now needed to ensure this was loadded at the toplevel.

Another trick I did was conditionally load libraries in the test runner and gracefully accept their absence. That's also something you can't really do with es modules since they are declarative. So you can't catch them and there is currently no way to ignore a failed import and have it default to undefined.

import()

The only way within the confines of es modules that I can think of that would allow me to do the same as before was to use the dynamic import() function. However, unlike require, this is an async operation. Meh.

So for the test runner I now have a Promise.all that includes all the test files (which export test cases) and it'll wait for the optional libraries to resolve somehow.

I think this is a good candidate for toplevel await since now I have to wrap at least part of the global code in a function for the sake of awaiting it;

const start = async () => {
await Promise.all([
(async () => { try { ({format: prettierFormat} = (await import('prettier')).default); } catch(e) {} })(),
...
}

start().then(() => console.log('done')).catch(e => console.log('crash:', e));

I think it would be nicer if I could just do a toplevel await on the promise;

await Promise.all([...]);

I don't know how generic that kind of case is but at least it seems to me like a good case.

Note that at the time of writing (according to caniuse) Chrome supports import() while Firefox does not. Safari allegedly too but I can't test this.

Mimetypes

The es module spec requires modules to be served with a text/javascript mimetype. I'm not sure why this is so important but you bet browsers care.

So after I fixed all the node issues and I reloaded the repl in the browser it stopped working because it wasn't serving the correct mimetype. Grrr.

What's worse is I couldn't find a simple oneliner webserver that would allow me to set the mimetypes from the cli. That's really annoying.

Luckily I found a quick fix for this: patching /etc/mime.types with the .mjs mimetype.

sudo nano /etc/mime.types
// add a line like This
text/javascript mjs

After saving the SimpleHttpServer would serve the .mjs with the specified mimetype and the page would load (after the mandatory ctrl+c). Huuraah!

Actual changes

The commit to convert everything to es modules is here: https://github.com/pvdz/zeparser3/commit/945f405c5fafac8cab1976919e4bb7a3a95e13bc

In most cases it's a simple replacement like most of the test files. I did have to fix one case of octal literals (copy paste default for terminal ansi colors) and remove all the requires.

"use module"

So in hindsight there's quite some overhead to getting sources parsed with the module goal. Browsers require the type="module" addition, nodejs requires an experimental flag AND the .mjs file extension, Babel requires the sourceType option to be set.

All in all this could have been made much easier with a toplevel "use module" directive, as proposed. Unfortunately the proposal didn't move and so we still can't use it today. Instead we have to pull crazy stunts to get es modules to work in both environments.

I didn't follow any of that discussion so I'm curious now why that didn't go anywhere. Seems to me far less intrusive than the current requirements. No changes to the file name, just an otherwise dead statement. If an environment doesn't support it then it will ignore the "module" directive and still crash on the import/export keywords.

Optional

Another thing that I think is a big miss is not allowing a module to be optionally included. So what if the module doesn't exist? Just initialize all bindings to undefined and call it a day. Alternatively I could see two syntax additions to aid;

import x from "foo" optionally;

import x from "foo" catch(e) { ... };

One could even go as far as to allow an import inside a toplevel try block, but I think that has its own drawbacks.

You can work around this with dynamic import() but that is currently still a stage 3 proposal ("almost part of the spec but not quite") so technically we can't even use it yet.

And even if you can use import(), you'll still have to update your code to work async for the bits that need conditional stuff. That's just not ideal.

Conclusion

While it's definitely possible to use import/export right now both in nodejs and in the browser, it'll be a rough ride for now. It does enable you to use the same sources in both environments without any kind of preprocessor or build step and I think that's got to count for something.

Here's my live example: https://pvdz.github.io/zeparser3/tests/web/repl.html

Hope my rambling helps you :)