Implement Azure DALL-E 2 CreateImage Support (#252)

[coggsfl]

No description provided.

[codecov]

Codecov Report

Merging #391 (1316a6b) into master (b095938) will increase coverage by 0.31%.
The diff coverage is 95.31%.

@@            Coverage Diff             @@
##           master     #391      +/-   ##
==========================================
+ Coverage   95.26%   95.58%   +0.31%     
==========================================
  Files          17       17              
  Lines         676      725      +49     
==========================================
+ Hits          644      693      +49     
  Misses         22       22              
  Partials       10       10              

Impacted Files	Coverage Δ
client.go	96.25% <95.31%> (+1.65%)	⬆️

[coggsfl]

@sashabaranov please review, thanks!

[vvatanabe]

@coggsfl
Adding Azure OpenAI support would be very useful. Thanks! However, I want us to pause and consider: is it necessary to support high-level abstraction functions in go-openai? We have had similar discussions in the past.

@sashabaranov @jmacwhyte
Do you think this support for Azure OpenAI is acceptable? Also, do you intend for go-openai to continue supporting Azure OpenAI in the future? There is likely to continue to be some differences between OpenAI and Azure OpenAI going forward.

[jmacwhyte]

Thanks for working on this!

Since Microsoft is a different company from OpenAI, I think Azure support should be its own repo. It's always possible that Microsoft could, for example, roll out their own AI models to power their Azure services, and try to swap out OpenAI over time. Then we could end up in a situation where many developers are relying on the OpenAI library to use Microsoft services, and things would get messy if the Microsoft API then started to evolve independently.

I think someone should create a brand new repo (not a fork) that imports go-openai and creates bindings that are compatible for Azure services. Then if OpenAI support ever needs to be swapped out, it will be easy to do without complicating things for developers who are relying on the library.

[coggsfl]

Good conversation, I think it's important to discuss too.
I agree with the concerns raised and really don't know why MSFT went down this path with DALL-E 2, other than perhaps because they have a separate team working on image processing for their Designer app.
However, all evidence I see right now and from conversations with their AI product team suggests that their strategy is to make OpenAI features available on Azure within a week or two of OpenAI releasing them which would limit their ability to diverge much more than they have already (DALL-E 2 being the exception).
If we see further divergence then I would certainly consider kicking off a new repo to support Azure OpenAI, but I wonder if it would be premature to do so now?

[coggsfl]

@sashabaranov what are your thoughts?

[jmacwhyte]

their strategy is to make OpenAI features available on Azure within a week or two of OpenAI releasing them which would limit their ability to diverge much more than they have already

I'm not sure why this would be true. I don't see any reason they couldn't randomly decide to roll out features using an API that differs from the OpenAI API, especially if they want to combine their AI offering with other features.

If we split the repos and the MSFT API stays close to the OpenAI one, the second repo should be very easy to maintain. If we don't split the repos and MSFT starts to diverge after a lot of large projects are relying on this repo, it could get messy.

My vote is to think of MSFT as another consumer of the OpenAI service (just like everyone else using this repo!) and not a service provider (unless they offer an API that is exactly the same as the OpenAI one, in which case no modification is needed anyway). But I'd be happy to hear everyone else's thoughts on it.

[coggsfl]

I'm not sure why this would be true. I don't see any reason they couldn't randomly decide to roll out features using an API that differs from the OpenAI API, especially if they want to combine their AI offering with other features.

Of course they're free agents but I would offer two points to support my argument. 1) MSFT's deal with OpenAI was a compute for licensing deal. They're investing compute to get licensing rights to OpenAI's cool stuff. Clearly modifications were required make OpenAI available on Azure, but it would be highly inefficient to deviate significantly and require pointless refactoring for each new capability that OpenAI releases. 2) I understand that MSFT will release Functions within the next week or so. If this is true, then it demonstrates that MSFT is actively keeping in lock-step with the OpenAI release train.

If point #2 turns out to be wrong and Microsoft deviate from the OpenAI API then I would concede that you're absolutely right and a separate repo is required.

Perhaps we wait a few days and see what happens with functions?

[jmacwhyte]

@coggsfl Thank you for providing this background information and explaining your thinking. I see now where we differ and maybe I didn't explain myself well enough.

I've been thinking about this from a computer science point of view--on an academic level, what is the correct way to organize software based on the computing systems it describes and interacts with. This way of thinking ignores contracts, business arrangements, and industry personalities, because those don't impact how computers interpret code and is information new developers won't have when they are reading through a library to figure out the structure it describes.

All of my previous comments were meant to give examples of why I prefer this line of thinking and why, in my opinion, the business arragements shouldn't have any impact (i.e. anything could happen on the business side, but computers will always run code the same way).

In the same way that OpenAI's service is being imported by MSFT and provided to their users in a slightly different format, this library should be imported into an MSFT repo and provided to those users in a way that matches the MSFT system.

I hope this helps!

[coggsfl]

@jmacwhyte I see where you're coming from.

May I propose a compromise - #403 is a draft to isolate the Azure components in the project. It isolates complexity due to bifurcated APIs in the short-term, and provides a path to creating complete separation in the longer term if necessitated by further Microsoft API divergence.

@vvatanabe @sashabaranov @jmacwhyte is there a compromise available here?

[sashabaranov]

@vvatanabe @jmacwhyte @coggsfl @liushuangls

Folks, thank you for being so thoughtful about this stuff! Here are my 2 cents:

There is indeed no guarantee that Azure API would not diverge from OpenAI. They have OpenAPI spec available, I wonder if we can utilize it.

More generally — I think we need to start looking at it from the frontend/backend perspective. Azure/OpenAI/Cohere/Anthropic/Google are all LLM backend providers with ever-changing APIs.

And we can create a nice Go frontend for the general use of LLMs. I've created a separate project https://github.com/go-llm/llmlite to achieve just that. I invite you all to collaborate on it! It's still early days, and we'll need to figure out vision/goals/non-goals/API structure and … well … code it! :D

[coggsfl]

Today Microsoft released DALL-3 and realigned the Azure OpenAI image API schema with OpenAI.
I plan on transitioning to DALL-3 and dropping support for DALL-2.