Skip to main content
Inference.net provides a distributed inference network for running AI models at scale.
Even though Inference.net is a built-in provider, it still requires you to bring your own key because ngrok.ai inference is not available for it at this time.

​
Setup

1

Create an access key

Follow the quickstart to create an access key in app.ngrok.ai.
2

Store your Inference.net API key

Sign up at inference.net and generate an API key. Then add a provider key.
3

Create a configuration

Create an access key configuration with an inference-net routing rule and allow Inference.net in the access scope.
4

Assign and send requests

Assign the configuration to your access key and send requests with your access key:
from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.ngrok.ai/v1",
    api_key="ng-xxxxx-g1-xxxxx",
)

response = client.chat.completions.create(
    model="inference-net:meta-llama/llama-3.1-8b-instruct/fp-8",
    messages=[{"role": "user", "content": "Hello!"}]
)

​
Available models

See the model catalog.

​
Next steps