Skip to content

Commit 050798e

Browse files
committed
Document asyncio mode for use()
1 parent c982d53 commit 050798e

File tree

1 file changed

+44
-5
lines changed

1 file changed

+44
-5
lines changed

README.md

Lines changed: 44 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -628,6 +628,46 @@ for output in outputs:
628628
print(get_url_path(output)) # "https://replicate.delivery/xyz"
629629
```
630630

631+
### Async Mode
632+
633+
By default `use()` will return a function instance with a sync interface. You can pass `use_async=True` to have it return an `AsyncFunction` that provides an async interface.
634+
635+
```py
636+
import asyncio
637+
import replicate
638+
639+
async def main():
640+
flux_dev = replicate.use("black-forest-labs/flux-dev", use_async=True)
641+
outputs = await flux_dev(prompt="a cat wearing an amusing hat")
642+
643+
for output in outputs:
644+
print(Path(output))
645+
646+
asyncio.run(main())
647+
```
648+
649+
If the model returns an iterator an `AsyncIterator` implementation will be used:
650+
651+
```py
652+
import asyncio
653+
import replicate
654+
655+
async def main():
656+
claude = replicate.use("anthropic/claude-3.5-haiku", use_async=True)
657+
output = await claude(prompt="say hello")
658+
659+
# Stream the response as it comes in.
660+
async for token in output:
661+
print(token)
662+
663+
# Wait until model has completed. This will return either a `list` or a `str` depending
664+
# on whether the model uses AsyncIterator or ConcatenateAsyncIterator. You can check this
665+
# on the model schema by looking for `x-cog-display: concatenate`.
666+
print(await output)
667+
668+
asyncio.run(main())
669+
```
670+
631671
### Typing
632672

633673
By default `use()` knows nothing about the interface of the model. To provide a better developer experience we provide two methods to add type annotations to the function returned by the `use()` helper.
@@ -666,11 +706,10 @@ In future we hope to provide tooling to generate and provide these models as pac
666706

667707
There are several key things still outstanding:
668708

669-
1. Support for asyncio.
670-
2. Support for streaming text when available (rather than polling)
671-
3. Support for streaming files when available (rather than polling)
672-
4. Support for cleaning up downloaded files.
673-
5. Support for streaming logs using `OutputIterator`.
709+
1. Support for streaming text when available (rather than polling)
710+
2. Support for streaming files when available (rather than polling)
711+
3. Support for cleaning up downloaded files.
712+
4. Support for streaming logs using `OutputIterator`.
674713

675714
## Development
676715

0 commit comments

Comments
 (0)