Migration to ES Modules

[Josh-Cena]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Motivation

I did a very naïve attempt to migrate to ESM in #5816, but I far underestimated the difficulty in pushing it through. After some pondering, I think this should be rolled out progressively.

Status of ESM

ESM is a new type of Node modules system, replacing the old common JS system (require + module.exports). For the engine, the parsing goal is different ("module" or "script"), so a file needs to be determined as ESM or CJS before executing it, either through the .mjs extension or through the "type": "module" entry in the nearest package.json.

If a file is ESM, it can import from other ES modules. However, it may not be able to import all CJS modules, depending on how the CJS module is structured, because exported symbols, per the ES spec, need to be lexically determined, while CJS can be far more dynamic.

If a file is CJS, it can't import ESM modules, unless the await import() dynamic import is used. However, this is against the norm of how most modules are imported. This effectively means as soon as a considerable number of the dependencies are ESM, we have to migrate ourselves.

Many packages on NPM are now ESM-only, most notably the packages by Sindre Sorhus, and MDX v2, which is the pillarpost of our architecture.

Benefits

1. Unlock future dependency upgrades. Many popular libraries are seeking to upgrade to ESM; if we can be ESM, we can interoperate with them. For example, MDX v2, chalk...
2. A similar transpilation target for client and server code; no need for multiple tsconfigs
3. Permit top-level awaits

Blockers

1. TypeScript has very lame support of ESM transpilation so far. It seems their deferred ESM support still won't land in 4.6, so in the meantime we may have to run all our Node code with --experimental-specifier-resolution=node and keep the old resolution
2. Jest doesn't seem to like importing ESM dependencies
3. We use import-fresh to bypass cache and always import fresh modules for hot reloading, etc. But ES Modules don't expose caching manipulation yet
4. Community dividing: although the ESM migration will surely be a major version, it means in the foreseeable future after that, some plugins will not be compatible with the new version (especially those that import utils and logger)

Actions needed

1. Removing __filename and __dirname. These globals don't exist in the ESM scope, replaced by the import.meta.url
2. Setting target: 'nodenext' in the tsconfig.
3. Changing our import paths. ESM requires the index.js name and the .js extension to be explicit.
4. Setting "type": "module" in our package.json.
5. Tweaking the configuration for related tools?

Plan

1. Config files: JS config files like docusaurus.config.js and sidebars.js can be allowed in ESM once we figure out how to bypass cache and fresh-import ESM
2. Utils: this includes utils, utils-validation, logger. They should always be distributed as dual-package because plugin authors are likely to import them as CJS.
3. Core: the biggest blocker is still the extensive use of import-fresh. Migration to ESM means we can import both CJS and ESM plugin modules with await import. However, because users only interact with the core through its own CLI, we don't have to care about others importing this.
4. Plugins: migration of plugins can only happen after migration of core (or at least solving import-fresh in the core), because they have to be imported as ESM.

Related issues/PRs

If an issue or PR is related to the ESM migration process, please link to this meta-issue and we will add it to the list below for tracking purposes.

• Support ESM config file #5379
• misc: migrate all packages to ESM-only #5816
• misc: convert all internal scripts to ESM #6286
• refactor: mark all functions that import external modules as async #6521
• refactor: convert CLI entry points to ESM; migrate create-docusaurus to ESM #6661
• refactor: ensure lodash is default-imported #6716
• refactor: import jest as global; unify import style of some modules #6898
• refactor(theme-common): mark package as ESM #6899
• feat(core): allow plugin lifecycles to return relative paths #6921
• feat(core): support docusaurus.config.cjs as default file name #7371
• fix(core): make generated files have unambiguous module types #7379

[iPurpl3x]

Yes !!!! +1 !
Finally I stumbled on this issue, I had updated @mdx-js/react and had errors and it took me a while to understand what happened...

Error was something like

_mdx_js_react__WEBPACK_IMPORTED_MODULE_2__.mdx is not a function

and

export 'mdx' (imported as 'mdx') was not found in '@mdx-js/react' (possible exports: MDXContext, MDXProvider, useMDXComponents, withMDXComponents)

[Josh-Cena]

@iPurpl3x Your problem is probably not because of ESM, because @mdx-js/react is used on client-side, where Webpack can take care of ESM syntax. Instead, it's because of the API changes in v2. MDX will be compiled to a JSX file containing a line import { mdx } from '@mdx-js/react'; which doesn't exist in the latest MDX version.