Skip to content

docs: Move multiple playground messages guide #779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 9, 2025
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/evaluation/how_to_guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ Evaluate and improve your application before deploying it.
- [Run an evaluation from the UI](./how_to_guides/run_evaluation_from_prompt_playground)
- [Run an evaluation via the REST API](./how_to_guides/run_evals_api_only)
- [Run an evaluation with multimodal content](./how_to_guides/evaluate_with_attachments)
- [Set up a multi-turn evaluation](./how_to_guides/multiturn_evaluation)

### Define an evaluator

Expand Down
3 changes: 2 additions & 1 deletion docs/prompt_engineering/how_to_guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,13 @@ Organize and manage prompts in LangSmith to streamline your LLM development work

Quickly iterate on prompts and models in the LangSmith Playground.

- [Run an evaluation in the Playground](/evaluation?mode=ui)
- [Run an evaluation in the playground](/evaluation?mode=ui)
- [Include multimodal content in a prompt](./how_to_guides/multimodal_content)
- [Manage prompt settings](./how_to_guides/managing_model_configurations)
- [Iterate on your prompts with the Prompt Canvas](./how_to_guides/prompt_canvas)
- [Run the playground against a custom LangServe model server](./how_to_guides/custom_endpoint)
- [Run the playground against an OpenAI-compliant model provider/proxy](./how_to_guides/custom_openai_compliant_model)
- [Use multiple messages in the playground](./how_to_guides/multiturn_evaluation)

## Few shot prompting

Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
# How to set up a multi-turn evaluation
# How to use multiple messages in the playground

LangSmith makes it easy to evaluate multi-turn conversations in the playground. This allows you to evaluate how changing your system prompt,
the tools available to the model, or the output schema affects a conversation with multiple messages.

This how-to guide walks you through the various ways you can set up the playground for multi-turn evaluation, which will allow you to
test different tool configurations and system prompts to see how they impact your system.
This how-to guide walks you through the various ways you can set up the playground for multi-turn conversations, which will allow you to
test different tool configurations and system prompts against longer threads of messages.

![](./static/multiturn_diagram.png)

Expand All @@ -22,7 +19,7 @@ You can then edit the system prompt, tweak the tools and/or output schema and ob
Before starting, make sure you have [set up your dataset](../../evaluation/how_to_guides/manage_datasets_in_application). Since you want to evaluate
multi-turn conversations, make sure there is a key in your inputs that contains a list of messages.

Once you have created your dataset, head to the playground and [load your dataset](/evaluation/how_to_guides/manage_datasets_in_application#from-the-prompt-playground) to evaluate.
Once you have created your dataset, head to the playground and [load your dataset](../../evaluation/how_to_guides/manage_datasets_in_application#from-the-prompt-playground) to evaluate.

Then, add a messages list variable to your prompt, making sure to name it the same as the key in your inputs that contains the list of messages:

Expand All @@ -48,6 +45,6 @@ of the `Messages List` variable, allowing you to reuse this prompt across variou
## Next Steps

Now that you know how to set up your multi-turn evaluation, you can either manually
inspect and judge the outputs, or you can [add evaluators](../how_to_guides#define-an-evaluator) to get repeatable, quantitative results.
inspect and judge the outputs, or you can [add evaluators](../../evaluation/how_to_guides#define-an-evaluator) to classify results.

You can also read [these how-to guides](../../prompt_engineering/how_to_guides#playground) to learn more about how to use the playground to run evaluations.
4 changes: 4 additions & 0 deletions vercel.json
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,10 @@
"source": "/evaluation/how_to_guides/use_langchain_off_the_shelf_evaluators",
"destination": "/evaluation/how_to_guides/prebuilt_evaluators"
},
{
"source": "/evaluation/how_to_guides/multiturn_evaluation",
"destination": "/prompt_engineering/how_to_guides/multiple_messages"
},
{
"source": "/old/tracing/faq/querying_traces",
"destination": "/observability/how_to_guides/export_traces"
Expand Down
Loading