Persistent directory storage for docstore in MultiVectorRetriever

[sumedha73]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import uuid

from langchain.retrievers.multi_vector import MultiVectorRetriever
from langchain.storage import InMemoryStore, LocalFileStore
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document

def create_multi_vector_retriever(store, vectorstore, id_key, text_summaries=None, texts=None):
  """
  Create a multi-vector retriever that indexes summaries and stores embeddings in a folder.

  Args:
      folder_path: Path to the folder for storing embeddings.
      text_summaries: List of text summaries for new documents (optional).
      texts: List of corresponding texts/images/etc. for new documents ek(optional).

  Returns:
      The multi-vector retriever instance.
  """

  # Create the multi-vector retriever
  retriever = MultiVectorRetriever(
      vectorstore=vectorstore,
      docstore=store,
      id_key=id_key,
      search_kwargs={"k": 5},
  )

  def add_documents(retriever, doc_summaries, doc_contents):
      doc_ids = [str(uuid.uuid4()) for _ in doc_contents]
      summary_docs = [
          Document(page_content=s, metadata={id_key: doc_ids[i]})
          for i,s, in enumerate(doc_summaries)
      ]
      retriever.vectorstore.add_documents(summary_docs)
      retriever.docstore.mset(list(zip(doc_ids, doc_contents)))
      print("added")

  # Filter out empty text_summaries
  non_empty_text_summaries = [summary for summary in text_summaries if summary.strip()]

  # Add texts, tables, and images if summaries are not empty
  if non_empty_text_summaries:
      add_documents(retriever, non_empty_text_summaries, texts)

  return retriever


vectorstore = Chroma(
    collection_name = "second",
    persist_directory = "/content/embeddings12",
    embedding_function=VertexAIEmbeddings(model_name="textembedding-gecko@latest"),
)

docstore = InMemoryStore()
id_key="doc_id"

# Create Retriever
retriever_multi_vector_img = create_multi_vector_retriever(
    docstore,
    vectorstore,
    id_key,
    doc_summaries, #description of the images
    doc_img_base64_list #the actual images
)

def multi_modal_rag_chain(retriever):
  """
  Multi-modal RAG chain
  """

  # Multi-modal LLM
  model = ChatVertexAI(
      temperature = 0, model_name = "gemini-pro-vision", max_output_tokens=2048, safety_settings = safety_settings
  )


  # RAG Pipeline
  chain = (
      {
          "context": retriever | RunnableLambda(split_image_text_types),
          "question": RunnablePassthrough()
      }
      | RunnableLambda(img_prompt_func)
      | model
      | StrOutputParser()
  )
  return chain


# Create RAG chain
chain_multimodal_rag = multi_modal_rag_chain(retriever_multi_vector_img)

Description

i have been using the multi vector retriever from langchain for storing images and their description to my embeddings db. i am embedding the descriptions and storing the images as it is for retrieval. the documents are labelled by doc_id which i am storing in a InMemoryStore but i would want this to be in a persistent directory or variable (which i can store in a directory).
i have tried using the LocalFileStore but it stores byte-like object and the docstore needs to be in str format (source) so this approach threw TypeError.

is there anyway to implement this functionality? please help me, i am just a beginner with langchain and llms.

I have attached my code for the retriever and the llm chain.

Thanks!!

System Info

I am running my python program on google colab

[dosubot[bot]]

Hey @sumedha73! 👋 I'm Dosu, your friendly neighborhood bot here to lend a hand with bugs, questions, and helping you get in the groove with contributing while we wait for a human maintainer to chime in. Just thought I'd pop in and say I'm diving into your issue and will get back to you with some insights shortly. Hang tight! 🤖🌟

[sumedha73]

asked in the js repo instead of the python repo

[dosubot[bot]]

It sounds like you're looking to use a persistent storage solution for your document store in a Python environment, specifically for use with the MultiVectorRetriever from LangChain, but encountered issues with LocalFileStore due to its byte-like object requirement. Since you're working with images and their descriptions, and you need to store doc_id persistently, you might consider serializing your data to a string format before using LocalFileStore or any file-based storage.

One approach is to convert your images and their descriptions into a JSON format, where each image and description pair is stored as a string. JSON is text-based and thus compatible with the requirement for storing data in a string format. Here's a simplified example of how you might serialize and deserialize your data for use with LocalFileStore:

import json
from langchain.storage import LocalFileStore

# Example function to serialize data
def serialize_data(doc_id, image_base64, description):
    data = {
        "doc_id": doc_id,
        "image": image_base64,
        "description": description
    }
    return json.dumps(data)

# Example function to deserialize data
def deserialize_data(data_str):
    data = json.loads(data_str)
    return data['doc_id'], data['image'], data['description']

# Assuming you have a LocalFileStore instance
local_file_store = LocalFileStore(directory_path="/path/to/your/storage")

# To store data
data_str = serialize_data("your_doc_id", "your_image_base64", "your_description")
local_file_store.set("your_doc_id", data_str)

# To retrieve data
retrieved_data_str = local_file_store.get("your_doc_id")
doc_id, image_base64, description = deserialize_data(retrieved_data_str)

This approach allows you to work around the type limitation by ensuring that all data stored in LocalFileStore is in a string format, making it suitable for persistent storage. Remember to handle errors and edge cases, such as what happens if the data cannot be serialized or deserialized correctly.

For a more robust solution tailored to your specific use case, you might need to adapt the serialization and deserialization functions to handle your data's unique structure and requirements.

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[sumedha73]

How can data stored in this way be sent to the retriever in case i lose the current instance of retriever and want to build retriever again from stored data and not redo-ing the embedding and doc_id all over again?