Vision support in o3 models

[RESEARCHEKASA]

The new o3-mini model seems to support Vision calls. Using ImageURL (as with chat models) doesn't work. It seems that reasoning models only accept encoded images, not URLs.

Any suggestion?

[StephenHodgson]

Is the API returning the error or the plugin?

Can you give me an example to try out?

[RESEARCHEKASA]

Sure, sorry!

It seems that the API is returning the error:

GetCompletionAsync Failed! HTTP status code: BadRequest | Response body: { "error": { "message": "Invalid image URL: 'messages[2].content[1].image_url.url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got a value without the 'data:' prefix.", "type": "invalid_request_error", "param": "messages[2].content[1].image_url.url", "code": "invalid_value" } }

[StephenHodgson]

Yeah this is from the server. Best bet is to download your image into a local folder, then encode it to base64 string and convert that to bytes when you construct your request.

Here's how I handled native images for the Unity game engine for example:

var content = new (ContentType.ImageUrl, $"data:image/png;base64,{Convert.ToBase64String(texture.EncodeToPNG())}");

I didn't implement anything like this in my vanilla dotnet plugin to give people maximum flexibility.