[Prisma] Is there a reason why metadata includes pageContent, in the Prisma vectorstore?

[phof]

Is there a reason why metadata includes contentColumn (pageContent), in the Prisma vectorstore?

langchainjs/langchain/src/vectorstores/prisma.ts

Lines 394 to 397 in 4dbe79f

	new Document({
	pageContent: article[this.contentColumn] as string,
	metadata: article,
	}),

[dosubot[bot]]

🤖

The metadata in the Prisma vectorstore includes contentColumn (pageContent) to store the actual content of the document or text that is being processed. This is crucial because the language model needs to know the actual content of the text to generate embeddings and perform operations such as similarity searches.

This is evident in the addModels method where the contentColumn is used to extract the content of the document:

async addModels(models: TModel[]) {
  return this.addDocuments(
    models.map((metadata) => {
      const pageContent = metadata[this.contentColumn];
      if (typeof pageContent !== "string")
        throw new Error("Content column must be a string");
      return new Document({ pageContent, metadata });
    })
  );
}

And also in the similaritySearchVectorWithScore method where the contentColumn is used to create a new Document:

const results: [Document<SimilarityModel<TModel, TSelectModel>>, number][] =
  [];
for (const article of articles) {
  if (article._distance != null && article[this.contentColumn] != null) {
    results.push([
      new Document({
        pageContent: article[this.contentColumn] as string,
        metadata: article,
      }),
      article._distance,
    ]);
  }
}

In both cases, the contentColumn is used to access the actual content of the text or document that is being processed.

The metadata object in the Prisma vectorstore within the langchainjs codebase is used to store additional information about the documents that are being added to the vectorstore. This metadata is associated with each document and can be used to store any kind of additional information that might be useful for the application.

This was also discussed in a similar issue [Prisma] Is there a reason why metadata includes pageContent? in the langchainjs repository.

I hope this answers your question. If you have any more questions or need further clarification, feel free to ask.

Sources

• [Prisma] Is there a reason why metadata includes pageContent?
• langchain/src/vectorstores/prisma.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[phof]

The metadata object in the Prisma vectorstore within the langchainjs codebase is used to store additional information about the documents that are being added to the vectorstore. This metadata is associated with each document and can be used to store any kind of additional information that might be useful for the application.

That is what I expected, however in this case pageContent is also contained in the metadata object.

[phof]

cc/ @nfcampos @dqbd who seem to have worked on this previously.

[dqbd]

Hey @phof! The reasoning behind this decision was to expose the entire Prisma entity, but agreed that it's a duplication. Does this cause any issues for you?

[phof]

Hi @dqbd, thank you for the quick response! Not quite an issue, but I was curious as to why the data was duplicated.

The reasoning behind this decision was to expose the entire Prisma entity

Can you kindly expand on this? It doesn't look like other columns are returned when e.g. doing a similaritySearch.

[dqbd]

I believe it should return all the columns specified in the columns property when constructing the Prisma vectorstore.

The implementation boils down to subjective usage preference, as one could directly pick the entity type when consuming metadata instead of passing the wrapper Document type.

[phof]

I see, thanks, it sure returns the columns defined in the vectorstore constructor.

I'm still a little confused as to how to use this particular vectorstore, it doesn't seem I can simply replace pgvector with it:

• it doesn't create records in the database when using addDocuments, it updates them
• it filters records when using similaritySearchWithVectorScore but not when using asRetriever
• are there more examples of how to use the retrievalQAChain with it?

[phof]

Filtering asRetriever appears in a previous conversation, #2757