I had a parser project with three main files that I wanted to turn into an html repl. This was initially a nodejs project and so I needed to make things work in the browser now, too. I decided to try experimentally supported es-modules which uses the
import
/
export
syntax to see how far I could get this way. It would allow me to keep my sources fairly as-is and have the same source in nodejs and in the browser. Sort of.
The premis
The "es modules" stem from the ES6 era, released in 2015, which introduced a syntactical way of importing and exporting files as modules. If you're reading this you're probably well aware of the details so I'll skip the details on this.
Despite being three years old, environments are still catching up now. And even though
caniuse looks pretty optimistic there still a lot of caveats to actually using them.
Conversion
The project itself used requires ("commonjs"). This doesn't work in the browser so you can either strip the requires and include them with a script tag, or fake the
require
function with some boilerplate to get things to work.
I didn't see a nice way around keeping the sources as is while using them both in node and the browser without an explicit build step. So that's when I looked into using es modules. I knew nodejs started adding experimental support for them and that browsers had done the same thing. How bad could it be?
Well.
Layout
This project has three main js source files. The repl has an html and js file. There are also a ton of js test files, but that shouldn't be relevant for the repl.
I run a server through python's SimpleHttpServer, which is
one of the webserver one liners that'll just work without any further installs. Primitive but it gets the job done. Most of the time.
Conversion
The first thing I did was convert the three main files. The
require
would become
import
and the
module.exports
would become
exports
. This was fairly easy.
I setup the html and added the script tag with
type="module"
and low and beho... nothing. Chrome showed an inifini loading spinner in the browser tab, no further feedback. The python server seemed to be happy. What the hell. Firefox didn't work either.
Turns out that there's
an issue with the simple webserver where multiple requests were causing a deadlock without feedback. You can unlock them by pressing ctrl+c in the terminal, after which the page loads file (so what's cancelled then?).
This initially led to a pretty quick proof of concept that worked. Nice!
Back to node
Unfortunately that's when I tried running my test runner in nodejs. Everything was broken because node doesn't really know about the module goal.
The feature is tucked away behind an experimental flag. You have to run it like this:
node --experimental-modules foo.js
And not just that, you also have to rename the files to use the
.mjs
extension.
Once you do this for the main entry point you quickly discover this process kind of "taints" the other files. You import one module and it imports another and it all wants to use the import syntax. Apparently there's supposed to be this cross esm/commonsjs support but I don't think that worked for me. So I had to port all the files in the project, including tests, to use import/export. Sigh.
Furthermore, since node currently requires
.mjs
for es modules at all, all the files had to be renamed. Sigh. Ok, fine.
Implicit globals
The next problem turned up; things like
require
and
dirname
were no longer auto-defined. Gah.
So
__dirname
is at least an easy fix;
let filePath = import.meta.url.replace(/^file:\/\//,'');
let dirname = path.dirname(filePath);
This seems to work in node v10.11.0 and I suspect that'll be fairly solid.
The
require
is a little trickier because it does something that
import
declarations currently cannot do: conditional inclusion. Or even detection.
So I would do lazy loading of helper functions when they were needed, like
require('util').inspect(root, false, null)
for logging. That wouldn't work anymore and I now needed to ensure this was loadded at the toplevel.
Another trick I did was conditionally load libraries in the test runner and gracefully accept their absence. That's also something you can't really do with es modules since they are declarative. So you can't catch them and there is currently no way to ignore a failed import and have it default to
undefined
.
import()
The only way within the confines of es modules that I can think of that would allow me to do the same as before was to use the dynamic
import()
function. However, unlike
require
, this is an async operation. Meh.
So for the test runner I now have a
Promise.all
that includes all the test files (which export test cases) and it'll wait for the optional libraries to resolve somehow.
I think this is a good candidate for toplevel
await
since now I have to wrap at least part of the global code in a function for the sake of awaiting it;
const start = async () => {
await Promise.all([
(async () => { try { ({format: prettierFormat} = (await import('prettier')).default); } catch(e) {} })(),
...
}
start().then(() => console.log('done')).catch(e => console.log('crash:', e));
I think it would be nicer if I could just do a toplevel
await
on the promise;
await Promise.all([...]);
I don't know how generic that kind of case is but at least it seems to me like a good case.
Note that at the time of writing (
according to caniuse) Chrome supports
import()
while Firefox does not. Safari allegedly too but I can't test this.
Mimetypes
The es module spec requires modules to be served with a
text/javascript
mimetype. I'm not sure why this is so important but you bet browsers care.
So after I fixed all the node issues and I reloaded the repl in the browser it stopped working because it wasn't serving the correct mimetype. Grrr.
What's worse is I couldn't find a simple oneliner webserver that would allow me to set the mimetypes from the cli. That's really annoying.
Luckily I found
a quick fix for this: patching
/etc/mime.types
with the
.mjs
mimetype.
sudo nano /etc/mime.types
// add a line like This
text/javascript mjs
After saving the SimpleHttpServer would serve the
.mjs
with the specified mimetype and the page would load (after the mandatory ctrl+c). Huuraah!
Actual changes
The commit to convert everything to es modules is here:
https://github.com/pvdz/zeparser3/commit/945f405c5fafac8cab1976919e4bb7a3a95e13bcIn most cases it's a simple replacement like most of the test files. I did have to fix one case of octal literals (copy paste default for terminal ansi colors) and remove all the requires.
"use module"
So in hindsight there's quite some overhead to getting sources parsed with the module goal. Browsers require the
type="module"
addition, nodejs requires an experimental flag AND the
.mjs
file extension, Babel requires the
sourceType
option to be set.
All in all this could have been made much easier with a toplevel
"use module"
directive,
as proposed. Unfortunately the proposal didn't move and so we still can't use it today. Instead we have to pull crazy stunts to get es modules to work in both environments.
I didn't follow any of that discussion so I'm curious now why that didn't go anywhere. Seems to me far less intrusive than the current requirements. No changes to the file name, just an otherwise dead statement. If an environment doesn't support it then it will ignore the "module" directive and still crash on the
import
/
export
keywords.
Optional
Another thing that I think is a big miss is not allowing a module to be optionally included. So what if the module doesn't exist? Just initialize all bindings to
undefined
and call it a day. Alternatively I could see two syntax additions to aid;
import x from "foo" optionally;
import x from "foo" catch(e) { ... };
One could even go as far as to allow an
import
inside a toplevel
try
block, but I think that has its own drawbacks.
You can work around this with dynamic
import()
but that is currently still
a stage 3 proposal ("almost part of the spec but not quite") so technically we can't even use it yet.
And even if you can use
import()
, you'll still have to update your code to work async for the bits that need conditional stuff. That's just not ideal.
Conclusion
While it's definitely possible to use import/export right now both in nodejs and in the browser, it'll be a rough ride for now. It does enable you to use the same sources in both environments without any kind of preprocessor or build step and I think that's got to count for something.
Here's my live example:
https://pvdz.github.io/zeparser3/tests/web/repl.htmlHope my rambling helps you :)