Clean up process to modify design tokens (#1183)

# Pull Request

## 🤨 Rationale

This started as [a joke to modify some token
values](https://60e89457a987cf003efc0a5b-fftmbqhrah.chromatic.com/?path=/story/tests-button--button-theme-matrix)
but turned into refactoring how we source our design tokens when I
wrapped my head around the problems with our current approach:

1. our documentation describes importing tokens from Adobe XD and using
the Design System Provider but those processes no longer work
2. our Style Dictionary configuration and build scripts were in the
`dist` directory when they should be source
3. the process to generate Style Dictionary output (the platform
specific token files) was manual and the build output was checked into
source

The new approach is to remove DSP, make Style Dictionary JSON files in
this repo our source of truth for design token values, and generate
token output as part of the build.

The changes I'm making here should make #1175 easier too. (The Figma
plugin linked in that issue can output Style Dictionary JSON files).

## 👩‍💻 Implementation

1. Remove everything about storing tokens in XD and Design System
Provider. The source of truth is now Style Dictionary JSON files.
2. Move Style Dictionary build configuration to the `source` directory.
This required some changes to get lint to pass since some of these files
weren't previously linted. (See comments in PR)
3. Generate Style Dictionary output as part of `npm run build` instead
of checking it in to source
4. Update CONTRIBUTING docs to describe the current process
5. Pull token values for base tokens story from JS build output instead
of JSON source. This changed the order of the tokens in the story.

## 🧪 Testing

Made token changes and verified that the built components uptake them.

## ✅ Checklist

<!--- Review the list and put an x in the boxes that apply or ~~strike
through~~ around items that don't (along with an explanation). -->

- [x] I have updated the project documentation to reflect my changes or
determined no changes are needed.

---------

Co-authored-by: rajsite <[email protected]>

Author: jattasNI

change/@ni-nimble-components-0c56aa86-23c7-4259-abc3-aaccccf9e8c6.json

@@ -0,0 +1,7 @@
+{
+  "type": "patch",
+  "comment": "Storybook updates for design token build changes",
+  "packageName": "@ni/nimble-components",
+  "email": "[email protected]",
+  "dependentChangeType": "patch"
+}

change/@ni-nimble-tokens-09048a45-9290-4f9a-a5d8-940ddd6007a4.json

@@ -0,0 +1,7 @@
+{
+  "type": "major",
+  "comment": "Refactor token build output. This removes some source files from the built package.",
+  "packageName": "@ni/nimble-tokens",
+  "email": "[email protected]",
+  "dependentChangeType": "patch"
+}

packages/nimble-components/src/theme-provider/tests/color.stories.ts

@@ -1,6 +1,6 @@
 import type { Meta, StoryObj } from '@storybook/html';
 import { html, repeat } from '@microsoft/fast-element';
-import nimbleColorsMapJson from '@ni/nimble-tokens/dist/styledictionary/properties/colors.json';
+import * as tokens from '@ni/nimble-tokens/dist/styledictionary/js/tokens';
 import { createUserSelectedThemeStory } from '../../utilities/tests/storybook';
 import { controlLabelFont, controlLabelFontColor } from '../design-tokens';
 
@@ -9,12 +9,13 @@ interface NimbleColor {
     data: string;
 }
 
-const nimbleBaseColors: NimbleColor[] = Object.entries(
-    nimbleColorsMapJson.color
-).map(([key, valueObj]) => ({
-    name: key,
-    data: valueObj.value
-}));
+const colorRegExp = /^#([0-9a-zA-Z]{6})$/;
+const nimbleBaseColors: NimbleColor[] = Object.entries(tokens)
+    .filter(([_key, value]) => colorRegExp.test(value))
+    .map(([key, value]) => ({
+        name: key,
+        data: value
+    }));
 
 const styleMarkup = html`
     <style>

packages/nimble-tokens/.eslintignore

@@ -1,4 +1,2 @@
 dist/*
-!dist/styledictionary
 dist/styledictionary/*
-!dist/styledictionary/nimble-extensions.js

packages/nimble-tokens/.eslintrc.js

@@ -5,8 +5,11 @@ module.exports = {
     ],
     overrides: [
         {
+            parserOptions: {
+                ecmaVersion: '2020'
+            },
             files: [
-                'dist/styledictionary/nimble-extensions.js'
+                'source/styledictionary/*.js',
             ],
             rules: {
                 // Build scripts will not be in published package and are allowed to use devDependencies

packages/nimble-tokens/.gitignore

@@ -1,12 +1,10 @@
-# remove the dsp assets as we don't currently
-# leverage the dsp component definitions
-/assets
-
 dist/icons/*
 !dist/icons/svg
 
 dist/fonts/scss
 
+dist/styledictionary
+
 NimbleTokens/bin
 NimbleTokens/obj
 NimbleTokens/.vs

packages/nimble-tokens/CONTRIBUTING.md

@@ -20,25 +20,11 @@
 
 ## Editing Tokens
 
-New tokens are added to this package with the [Adobe XD Visual Studio extension](https://marketplace.visualstudio.com/items?itemName=Adobe.xd&ssr=false#overview).
+Tokens are generated using the [Style Dictionary](https://amzn.github.io/style-dictionary/#/) build system, which transforms platform-agnostic token definition files into platform-specific output. These JSON definition files are the source of truth for the colors, fonts, and component design tokens in this repository. To modify the generated tokens, complete these steps:
 
-The source of truth for the colors, fonts, and component design tokens in this repository is an Adobe XD [Design Library file](https://xd.adobe.com/view/33ffad4a-eb2c-4241-b8c5-ebfff1faf6f6-66ac/) maintained by the Visual Design team.
-
-For changes to any token values, work with the Visual Design team to update the library, and then perform the following steps:
-
-1. Open the XD extension (invoke [Show All Commands](https://code.visualstudio.com/docs/getstarted/keybindings#_navigation) and execute command `XD: Toggle Adobe XD Panel`) and load the nimble-tokens-dsp package by selecting the `nimble-tokens` folder.
-2. Click the **DSP Setting** button (bottom left). If the button is not available, open the XD extension settings to confirm that the `XD: Global Editor` setting is **checked** for both user and workspace.
-3. Scroll to the bottom of the DSP Setting page and click the **Re-import** button to update the CC LIBRARY LINK.
-4. Save the settings change.
-5. Click the **Start Editing** button and then the **Stop Editing** button to trigger the StyleDictionary token build.
-6. Commit these changes to the repo.
-
-For any token metadata changes (e.g. documentation, code snippets, etc.):
-
-1. Follow step 1 above.
-2. Click the **Start Editing** button and make your changes.
-3. Click the **Stop Editing** button to trigger a token build.
-4. Commit these changes to the repo.
+1. Edit the JSON files in `source/styledictionary/properties`. Long term these tokens will be sourced from a Figma design spec but for now it's OK to make manual edits.
+2. Rebuild the generated token files by running the repository's build command, `npm run build`.
+3. Test your changes locally and create a PR using the normal process.
 
 ## Updating icons