Fine-tune existing model for an image+text to image task

[tteguayco]

Hello,

I'm asking this question as a total beginner to the world of LLMs.

There seems to be a good bunch of tutorials and notebooks out there to fine-tune an existing Stable Diffusion model but I'm having a hard time trying to find an equivalent to fine-tune a pre-trained model when the task consists in providing an input image along with a text or prompt to generate an output image.

Could someone point me to the right direction here?

Thanks in advance!

[asomoza]

Hi, there are some variations on what you're asking, for example, to use the input image as a source and ask the model to edit something in it? or to use it as a reference?

For the first use case you can study instruct-pix2pix and for the later ones you can study IP Adapters, Tile Controlnets or Flux Redux.

Not all of them have training scripts but you can learn the theory behind them.

[tteguayco]

Super! Thanks a lot @asomoza

[tteguayco]

@asomoza I've been checking Omni Gen and it performs surprisingly well. However, I'm unable to find the cost for deploying it to Inference Endpoints and be able to use for production-ready apps. Any idea on where I can find this information?

[asomoza]

since is not added as a one-click inference, you will have to use a custom_handler, also by what I see of the model you'll need a 24 GB VRAM GPU so you'll probably need a 1x Nvidia L4

[tteguayco]

@asomoza Where do you check it will need 24GB VRAM? I would like to know for Instruct-pix2pix too, but I'm unable to find this info. Not even how big these models are in GBs.

[asomoza]

I just did a guess with the model size, some good repositories/models do publish how much VRAM do you need, for the others you will need to try to guess or do a test run.

The original instruct-pix2pix was finetuned from SD 1 so it uses the same VRAM as those, which I remember is like 2-3GB for half precision and 4-5GB for full precision. Of course you can train with other models but I remember that the dataset was low resolution so you have that limitation unless you want to use your own.