docs: multimodal

docs/evaluation/how_to_guides/evaluate_with_attachments.mdx

@@ -4,18 +4,22 @@ import {
   PythonBlock,
   TypeScriptBlock,
 } from "@site/src/components/InstructionsWithCode";
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
 
-# Run an evaluation with large file inputs
+# Run an evaluation with multimodal content
 
 LangSmith supports creating dataset examples with file attachments, which you can consume when running evals over that dataset.
 
-Attachments are most useful when working with LLM applications that require multimodal inputs or produce multimodal outputs.
-While multimodal data can be base64 encoded and uploaded as part of an example's inputs/outputs, base64 encodings are fairly space inefficient relative to the underlying binary data making them slower to upload/download to/from LangSmith.
-By using attachments you can speed up uploads/downloads and get nicer renderings of different file types in the LangSmith UI.
+Attachments are particularly valuable for LLM applications that work with multimodal inputs or outputs, such as images, audio, or documents. While you can include multimodal data in your examples by base64 encoding it, this approach is inefficient - the encoded data takes up more space than the original binary files, resulting in slower transfers to and from LangSmith. Using attachments instead provides two key benefits:
 
-## Create examples with attachments
+1. Faster upload and download speeds due to more efficient binary file transfers
+2. Enhanced visualization of different file types in the LangSmith UI
 
-### Using the SDK
+<Tabs className="interface-tabs" queryString="mode">
+  <TabItem value="sdk" label="SDK">
+
+## 1. Create examples with attachments
 
 To upload examples with attachments using the SDK, use the [create_examples](https://docs.smith.langchain.com/reference/python/client/langsmith.client.Client#langsmith.client.Client.create_examples) / [update_examples](https://docs.smith.langchain.com/reference/python/client/langsmith.client.Client#langsmith.client.Client.update_examples) Python methods or the [uploadExamplesMultipart](https://docs.smith.langchain.com/reference/js/classes/client.Client#uploadexamplesmultipart) / [updateExamplesMultipart](https://docs.smith.langchain.com/reference/js/classes/client.Client#updateexamplesmultipart) TypeScript methods.
 
@@ -147,7 +151,7 @@ Note that you cannot directly pass in a file path in the TypeScript SDK, as acce
   ]}
   groupId="client-language"
   />
-  
+
 :::info Uploading from filesystem
 
 Along with being passed in as bytes, attachments can be specified as paths to local files.
@@ -159,29 +163,7 @@ client.create_examples(..., dangerously_allow_filesystem=True)
 
 :::
 
-Once you upload examples with attachments, you can view them in the LangSmith UI. Each attachment will be rendered as a file with a preview, making it easy to inspect the contents.
-![](./static/attachments_with_examples.png)
-
-
-### From the UI
-
-#### From existing runs
-
-When adding runs to a LangSmith dataset, attachments can be selectively propagated from the source run to the destination example.
-To learn more, please see [this guide](./manage_datasets_in_application#add-runs-from-the-tracing-project-ui).
-
-![](./static/add_trace_with_attachments_to_dataset.png)
-
-#### From scratch
-
-You can also upload examples with attachments directly from the LangSmith UI. You can do so by clicking the `+ Example` button in the `Examples` tab of the dataset UI.
-You can then upload the attachments that you want by using the "Upload Files" button:
-
-![](./static/create_example_with_attachments.png)
-
-## Run evaluations
-
-Once you have a dataset that contains examples with file attachments, you can run evaluations that process these attachments.
+## 2. Run evaluations
 
 ### Define a target function 
 
@@ -334,7 +316,6 @@ groupId="client-language"
 
 ### Define custom evaluators
 
-In addition to using attachments inside of your target function, you can also use them inside of your evaluators as follows.
 The exact same rules apply as above to determine whether the evaluator should receive attachments.
 
 The evaluator below uses an LLM to judge if the reasoning and the answer are consistent.
@@ -432,9 +413,7 @@ groupId="client-language"
 
 ## Update examples with attachments
 
-### Using the SDK
-
-In the code [above](#create-examples-with-attachments), we showed how to add examples with attachments to a dataset.
+In the code above, we showed how to add examples with attachments to a dataset.
 It is also possible to update these same examples using the SDK.
 
 As with existing examples, datasets are versioned when you update them with attachments. Therefore, you can navigate to the dataset version history to see the changes made to each example.
@@ -507,14 +486,67 @@ dataset.id,
 groupId="client-language"
 />
 
-### From the UI
+  </TabItem>
+  <TabItem value="ui" label="UI">
+## 1. Create examples with attachments
+
+You can add examples with attachments to a dataset in a few different ways.
+
+### From existing runs
+
+When adding runs to a LangSmith dataset, attachments can be selectively propagated from the source run to the destination example.
+To learn more, please see [this guide](./manage_datasets_in_application#add-runs-from-the-tracing-project-ui).
+
+![](./static/add_trace_with_attachments_to_dataset.png)
+
+### From scratch
+
+You can create examples with attachments directly from the LangSmith UI. Click the `+ Example` button in the `Examples` tab of the dataset UI.
+Then upload attachments using the "Upload Files" button:
+
+![](./static/create_example_with_attachments.png)
+
+Once uploaded, you can view examples with attachments in the LangSmith UI. Each attachment will be rendered with a preview for easy inspection.
+![](./static/attachments_with_examples.png)
+
+## 2. Create a multimodal prompt
+The LangSmith UI allows you to include attachments in your prompts when evaluating multimodal models:
+
+First, click the file icon in the message where you want to add multimodal content.
+Next, add a template variable for the attachment(s) you want to include for each example.
+- For a single attachment type: Use the suggested variable name. Note: all examples must have an attachment with this name.
+- For multiple attachments or if your attachments have varying names from one example to another: Use the `All attachments` variable to include all available attachments for each example.
+
+![](./static/adding_multimodal_variable.gif)
+
+## Define custom evaluators
+
+:::note Multimodal evaluators
+The LangSmith playground does not currently support pulling multimodal content into evaluators. If this would be helpful for your use case, please let us know in the [LangChain's community Slack](https://langchaincommunity.slack.com/) (sign up [here](https://www.langchain.com/join-community) if you're not already a member)!
+:::
+
+You can evaluate a model's text output by adding an evaluator that takes in the example's inputs and outputs.
+Even without multimodal support in your evaluators, you can still run text-only evaluations. For example:
+
+- OCR → text correction: Use a vision model to extract text from a document, then evaluate the accuracy of the extracted output.
+- Speech-to-text → transcription quality: Use a voice model to transcribe audio to text, then evaluate the transcription against your reference.
+
+For more information on defining custom evaluators, see the [LLM as Judge](../how_to_guides/llm_as_judge) guide.
+
+## Update examples with attachments
 
 :::note Attachment Size Limit
 Attachments are limited to 20MB in size in the UI.
 :::
 
-When editing an example in the UI, you can upload new attachments, rename and delete attachemnts,
-and there is also a quick reset button to restore the attachments to what previously existed on the example.
-No changes will be saved until you click submit.
+When editing an example in the UI, you can:
+- Upload new attachments
+- Rename and delete attachments  
+- Reset attachments to their previous state using the quick reset button
+
+Changes are not saved until you click submit.
 
 ![](./static/attachment_editing.gif)
+
+  </TabItem>
+</Tabs>

docs/evaluation/how_to_guides/index.md

@@ -29,7 +29,7 @@ Evaluate and improve your application before deploying it.
 - [Evaluate an existing experiment (Python only)](./how_to_guides/evaluate_existing_experiment)
 - [Run an evaluation from the UI](./how_to_guides/run_evaluation_from_prompt_playground)
 - [Run an evaluation via the REST API](./how_to_guides/run_evals_api_only)
-- [Run an evaluation with large file inputs](./how_to_guides/evaluate_with_attachments)
+- [Run an evaluation with multimodal content](./how_to_guides/evaluate_with_attachments)
 - [Set up a multi-turn evaluation](./how_to_guides/multiturn_evaluation)
 
 ### Define an evaluator

docs/prompt_engineering/how_to_guides/index.md

@@ -17,6 +17,7 @@ Organize and manage prompts in LangSmith to streamline your LLM development work
 Quickly iterate on prompts and models in the LangSmith Playground.
 
 - [Run an evaluation in the Playground](/evaluation?mode=ui)
+- [Include multimodal content in a prompt](./how_to_guides/multimodal_content)
 - [Manage prompt settings](./how_to_guides/managing_model_configurations)
 - [Iterate on your prompts with the Prompt Canvas](./how_to_guides/prompt_canvas)
 - [Run the playground against a custom LangServe model server](./how_to_guides/custom_endpoint)

docs/prompt_engineering/how_to_guides/multimodal_content.mdx

@@ -0,0 +1,42 @@
+---
+sidebar_position: 7
+table_of_contents: false
+hide_table_of_contents: true
+---
+
+# Include multimodal content
+
+Some applications are based around multimodal content, like a chatbot that can answer questions about a PDF or image.
+In these cases, you'll want to include multimodal content in your prompt and test the model's ability to answer questions about the content.
+
+The LangSmith Playground supports two methods for incorporating multimodal content in your prompts:
+
+1. Inline content: Embed static files (images, PDFs, audio) directly in your prompt. This is ideal when you want to consistently include the same multimodal content across all uses of the prompt. For example, you might include a reference image that helps ground the model's responses.
+2. Template variables: Create dynamic placeholders for attachments that can be populated with different content each time. This approach offers more flexibility, allowing you to:
+   - Test how the model handles different inputs
+   - Create reusable prompts that work with varying content
+
+## Inline content
+
+Click the file icon in the message where you want to add multimodal content.
+Under the `Upload content` tab, you can upload a file and include it inline in the prompt.
+
+![](./static/upload_inline_multimodal_content.png)
+
+## Template variables
+
+Click the file icon in the message where you want to add multimodal content.
+Under the `Template variables` tab, you can create a template variable for a specific attachment type.
+Currently, only images, PDFs, and audio files (.wav, .mp3) are supported.
+
+![](./static/template_variable_multimodal_content.png)
+
+## Populate the template variable
+
+Once you've added a template variable, you can provide content for it using the panel on the right side of the screen. Simply click the `+` button to upload or select content that will be used to populate the template variable.
+
+![](./static/manual_prompt_multimodal.png)
+
+## Run an evaluation
+
+After testing out your prompt manually, you can [run an evaluation](../../evaluation/how_to_guides/evaluate_with_attachments?mode=ui) to see how the prompt performs over a golden dataset of examples.

src/css/custom.css

@@ -267,7 +267,7 @@ html[data-theme="dark"] {
 }
 
 /* Dark mode specific styling */
-[data-theme='dark'] .fold-markers {
+[data-theme="dark"] .fold-markers {
   /* background: var(--prism-background-color) !important; */
   background: #1e1e1e;
 }
@@ -322,7 +322,7 @@ html[data-theme="dark"] {
 }
 
 .code-block-with-gutter {
-  padding-left: 12px
+  padding-left: 12px;
 }
 
 /* Remove theme border */
@@ -359,7 +359,7 @@ html[data-theme="dark"] {
   .tabs-container > .code-tabs {
     background: #1e1e1e;
   }
-  
+
   .tabs-container .code-caption {
     background: #1e1e1e;
   }
@@ -422,7 +422,7 @@ hr {
   background-color: var(--joy-palette-background-level2) !important;
 }
 
-div[class^='announcementBar_'] {
+div[class^="announcementBar_"] {
   font-size: 20px;
   background-color: #d0c9feff;
   font-weight: bold;
@@ -439,9 +439,9 @@ div[class^='announcementBar_'] {
 .interface-tabs {
   padding: 10px;
   border: none;
-  border-bottom: 1.5px solid #cbbbbb; 
+  border-bottom: 1.5px solid #cbbbbb;
 }
 
 .interface-tabs .tabs__item {
   font-size: 1.25rem;
-}
+}