Skip to content

DOC: Multimodal Message Schema for OpenAI Models - Docs + Error Improvements: include metadata property #8227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
2 tasks done
starmorph opened this issue May 22, 2025 · 1 comment
Labels
auto:documentation Changes to documentation and examples, like .md, .rst, .ipynb files. Changes to the docs/ folder auto:nit Small modifications/deletions, fixes, deps or improvements to existing code or docs

Comments

@starmorph
Copy link

starmorph commented May 22, 2025

Checklist

  • I added a very descriptive title to this issue.
  • I included a link to the documentation page I am referring to (if applicable).

Issue with current documentation:

When creating a Multimodal message (PDF/Image) to an OpenAI chat model in the format below, our docs and error throwing is not clear that the metadata property is required. Currently, If the metadata property is not set correctly in a multimodal Base64ContentBlock message, we throw an unintuitive OpenAI error.

Improvement Suggestion

  1. Update the following documentation pages to include the metadata property. Side note, the python version has a more comprehensive schema documented than JS.
    LangGraphJS MultiModal Inputs
    Langgraph PY MultiModal Inputs
  2. Update the error when missing the metadata property in a Base64ContentBlock from @langchain/core/messages to say "missing required metadata property" instead of OpenAI error.

Base64ContentBlock

  // PDF Example Base64ContentBlock Message
  {
    type: "file",
    source_type: "base64",
    mime_type: "application/pdf",
    data,
    metadata: { filename: file.name },
  };
}
// Image Example Base64ContentBlock Message
{
      type: "image",
      source_type: "base64",
      mime_type: file.type,
      data,
      metadata: { name: file.name },
    };

Idea or request for content:

No response

@starmorph starmorph changed the title DOC: MultiModal Message Schema for OpenAI Models Error Improving DOC: MultiModal Message Schema for OpenAI Models - Docs error Improving to Include metadata property May 22, 2025
@starmorph starmorph changed the title DOC: MultiModal Message Schema for OpenAI Models - Docs error Improving to Include metadata property DOC: MultiModal Message Schema for OpenAI Models - Docs + Error Improvements: include metadata property May 22, 2025
@starmorph starmorph changed the title DOC: MultiModal Message Schema for OpenAI Models - Docs + Error Improvements: include metadata property DOC: Multimodal Message Schema for OpenAI Models - Docs + Error Improvements: include metadata property May 22, 2025
@dosubot dosubot bot added auto:documentation Changes to documentation and examples, like .md, .rst, .ipynb files. Changes to the docs/ folder auto:nit Small modifications/deletions, fixes, deps or improvements to existing code or docs labels May 22, 2025
@byteakp
Copy link

byteakp commented May 23, 2025

Hi 👋 Just submitted a PR that addresses this issue: #8244 .

The PR improves the documentation and examples by clearly showing that the metadata property is required when sending Base64-encoded content blocks (e.g., PDFs or images) to OpenAI chat models. It also adds caution notes and clarifies code snippets for better developer understanding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto:documentation Changes to documentation and examples, like .md, .rst, .ipynb files. Changes to the docs/ folder auto:nit Small modifications/deletions, fixes, deps or improvements to existing code or docs
Projects
None yet
Development

No branches or pull requests

2 participants