You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
LangSmith lets you create dataset examples with file attachments—like images, audio files, or documents—so you can reference them when evaluating an application that uses multimodal inputs or outputs.
9
18
10
-
LangSmith supports creating dataset examples with file attachments, which you can consume when running evals over that dataset.
19
+
While you can include multimodal data in your examples by base64 encoding it, this approach is inefficient - the encoded data takes up more space than the original binary files, resulting in slower transfers to and from LangSmith. Using attachments instead provides two key benefits:
11
20
12
-
Attachments are most useful when working with LLM applications that require multimodal inputs or produce multimodal outputs.
13
-
While multimodal data can be base64 encoded and uploaded as part of an example's inputs/outputs, base64 encodings are fairly space inefficient relative to the underlying binary data making them slower to upload/download to/from LangSmith.
14
-
By using attachments you can speed up uploads/downloads and get nicer renderings of different file types in the LangSmith UI.
21
+
1. Faster upload and download speeds due to more efficient binary file transfers
22
+
2. Enhanced visualization of different file types in the LangSmith UI
To upload examples with attachments using the SDK, use the [create_examples](https://docs.smith.langchain.com/reference/python/client/langsmith.client.Client#langsmith.client.Client.create_examples) / [update_examples](https://docs.smith.langchain.com/reference/python/client/langsmith.client.Client#langsmith.client.Client.update_examples) Python methods or the [uploadExamplesMultipart](https://docs.smith.langchain.com/reference/js/classes/client.Client#uploadexamplesmultipart) / [updateExamplesMultipart](https://docs.smith.langchain.com/reference/js/classes/client.Client#updateexamplesmultipart) TypeScript methods.
21
30
@@ -147,7 +156,7 @@ Note that you cannot directly pass in a file path in the TypeScript SDK, as acce
147
156
]}
148
157
groupId="client-language"
149
158
/>
150
-
159
+
151
160
:::info Uploading from filesystem
152
161
153
162
Along with being passed in as bytes, attachments can be specified as paths to local files.
Once you upload examples with attachments, you can view them in the LangSmith UI. Each attachment will be rendered as a file with a preview, making it easy to inspect the contents.
163
-
164
-
165
-
166
-
### From the UI
167
-
168
-
#### From existing runs
169
-
170
-
When adding runs to a LangSmith dataset, attachments can be selectively propagated from the source run to the destination example.
171
-
To learn more, please see [this guide](./manage_datasets_in_application#add-runs-from-the-tracing-project-ui).
You can also upload examples with attachments directly from the LangSmith UI. You can do so by clicking the `+ Example` button in the `Examples` tab of the dataset UI.
178
-
You can then upload the attachments that you want by using the "Upload Files" button:
179
-
180
-
181
-
182
-
## Run evaluations
183
-
184
-
Once you have a dataset that contains examples with file attachments, you can run evaluations that process these attachments.
171
+
## 2. Run evaluations
185
172
186
173
### Define a target function
187
174
@@ -334,7 +321,6 @@ groupId="client-language"
334
321
335
322
### Define custom evaluators
336
323
337
-
In addition to using attachments inside of your target function, you can also use them inside of your evaluators as follows.
338
324
The exact same rules apply as above to determine whether the evaluator should receive attachments.
339
325
340
326
The evaluator below uses an LLM to judge if the reasoning and the answer are consistent.
@@ -432,9 +418,7 @@ groupId="client-language"
432
418
433
419
## Update examples with attachments
434
420
435
-
### Using the SDK
436
-
437
-
In the code [above](#create-examples-with-attachments), we showed how to add examples with attachments to a dataset.
421
+
In the code above, we showed how to add examples with attachments to a dataset.
438
422
It is also possible to update these same examples using the SDK.
439
423
440
424
As with existing examples, datasets are versioned when you update them with attachments. Therefore, you can navigate to the dataset version history to see the changes made to each example.
@@ -507,14 +491,67 @@ dataset.id,
507
491
groupId="client-language"
508
492
/>
509
493
510
-
### From the UI
494
+
</TabItem>
495
+
<TabItemvalue="ui"label="UI">
496
+
## 1. Create examples with attachments
497
+
498
+
You can add examples with attachments to a dataset in a few different ways.
499
+
500
+
### From existing runs
501
+
502
+
When adding runs to a LangSmith dataset, attachments can be selectively propagated from the source run to the destination example.
503
+
To learn more, please see [this guide](./manage_datasets_in_application#add-runs-from-the-tracing-project-ui).
You can create examples with attachments directly from the LangSmith UI. Click the `+ Example` button in the `Examples` tab of the dataset UI.
510
+
Then upload attachments using the "Upload Files" button:
511
+
512
+
513
+
514
+
Once uploaded, you can view examples with attachments in the LangSmith UI. Each attachment will be rendered with a preview for easy inspection.
515
+
516
+
517
+
## 2. Create a multimodal prompt
518
+
The LangSmith UI allows you to include attachments in your prompts when evaluating multimodal models:
519
+
520
+
First, click the file icon in the message where you want to add multimodal content.
521
+
Next, add a template variable for the attachment(s) you want to include for each example.
522
+
- For a single attachment type: Use the suggested variable name. Note: all examples must have an attachment with this name.
523
+
- For multiple attachments or if your attachments have varying names from one example to another: Use the `All attachments` variable to include all available attachments for each example.
524
+
525
+
526
+
527
+
## Define custom evaluators
528
+
529
+
:::note Multimodal evaluators
530
+
The LangSmith playground does not currently support pulling multimodal content into evaluators. If this would be helpful for your use case, please let us know in the [LangChain's community Slack](https://langchaincommunity.slack.com/) (sign up [here](https://www.langchain.com/join-community) if you're not already a member)!
531
+
:::
532
+
533
+
You can evaluate a model's text output by adding an evaluator that takes in the example's inputs and outputs.
534
+
Even without multimodal support in your evaluators, you can still run text-only evaluations. For example:
535
+
536
+
- OCR → text correction: Use a vision model to extract text from a document, then evaluate the accuracy of the extracted output.
537
+
- Speech-to-text → transcription quality: Use a voice model to transcribe audio to text, then evaluate the transcription against your reference.
538
+
539
+
For more information on defining custom evaluators, see the [LLM as Judge](../how_to_guides/llm_as_judge) guide.
540
+
541
+
## Update examples with attachments
511
542
512
543
:::note Attachment Size Limit
513
544
Attachments are limited to 20MB in size in the UI.
514
545
:::
515
546
516
-
When editing an example in the UI, you can upload new attachments, rename and delete attachemnts,
517
-
and there is also a quick reset button to restore the attachments to what previously existed on the example.
518
-
No changes will be saved until you click submit.
547
+
When editing an example in the UI, you can:
548
+
- Upload new attachments
549
+
- Rename and delete attachments
550
+
- Reset attachments to their previous state using the quick reset button
Some applications are based around multimodal content, like a chatbot that can answer questions about a PDF or image.
8
+
In these cases, you'll want to include multimodal content in your prompt and test the model's ability to answer questions about the content.
9
+
10
+
The LangSmith Playground supports two methods for incorporating multimodal content in your prompts:
11
+
12
+
1. Inline content: Embed static files (images, PDFs, audio) directly in your prompt. This is ideal when you want to consistently include the same multimodal content across all uses of the prompt. For example, you might include a reference image that helps ground the model's responses.
13
+
2. Template variables: Create dynamic placeholders for attachments that can be populated with different content each time. This approach offers more flexibility, allowing you to:
14
+
- Test how the model handles different inputs
15
+
- Create reusable prompts that work with varying content
16
+
17
+
:::note
18
+
Not all models support multimodal content. Before using multimodal features in the playground, make sure your selected model supports the file types you want to use.
19
+
:::
20
+
21
+
## Inline content
22
+
23
+
Click the file icon in the message where you want to add multimodal content.
24
+
Under the `Upload content` tab, you can upload a file and include it inline in the prompt.
Once you've added a template variable, you can provide content for it using the panel on the right side of the screen. Simply click the `+` button to upload or select content that will be used to populate the template variable.
39
+
40
+
41
+
42
+
## Run an evaluation
43
+
44
+
After testing out your prompt manually, you can [run an evaluation](../../evaluation/how_to_guides/evaluate_with_attachments?mode=ui) to see how the prompt performs over a golden dataset of examples.
0 commit comments