Run Aya 35B 23 · F16 (aya:35b-23-q8_0) on your own GPU

Compare GPU memory for Aya 35B 23 · F16 (aya:35b-23-q8_0)

Catalog estimate at F16, not a measured benchmark. Chat estimates assume a short context; embedding memory depends on input and batch size. The comparison includes a 5% screening margin; it does not establish runtime performance. More in how much VRAM an LLM needs.

Check this estimate against your GPU with the VRAM calculator

• RTX 4060 Ti 8 GBMore headroom needed
• RTX 4070 Ti Super 16 GBMore headroom needed
• RTX 4000 Ada 20 GBMore headroom needed
• RTX 3090 / 4090 24 GBMore headroom needed
• RTX PRO 4000 Blackwell 24 GB (Hetzner GEX45)More headroom needed
• NVIDIA L4 24 GBMore headroom needed
• RTX 5090 32 GBMore headroom needed
• A100 / H100 80 GBEstimate fits
• RTX PRO 6000 Blackwell 96 GB (Hetzner GEX131)Estimate fits

Call Aya 35B 23 · F16 (aya:35b-23-q8_0) with the OpenAI SDK

Use the supported chat completions endpoint with your gpuos base URL, API key and deployed model id. Validate any model-specific features your app needs. Setup for LangChain, Continue, Open WebUI and more is in integrations.

from openai import OpenAI

client = OpenAI(base_url="https://gpuos.si/v1", api_key="gpuos_key_…")
stream = client.chat.completions.create(
    model="aya-35b-23-q8-0",
    messages=[{"role": "user", "content": "Summarize our refund policy."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

import OpenAI from "openai"

const client = new OpenAI({ baseURL: "https://gpuos.si/v1", apiKey: process.env.GPUOS_API_KEY })
const reply = await client.chat.completions.create({
  model: "aya-35b-23-q8-0",
  messages: [{ role: "user", content: "Summarize our refund policy." }],
})
console.log(reply.choices[0].message.content)

curl https://gpuos.si/v1/chat/completions \
  -H "Authorization: Bearer $GPUOS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "aya-35b-23-q8-0", "messages": [{"role": "user", "content": "Hello"}]}'

Questions

How much VRAM does Aya 35B 23 · F16 (aya:35b-23-q8_0) need?

The catalog estimate is 40.5 GB at F16 for a short context (4–8K tokens). The A100 / H100 80 GB clears this estimate with screening headroom in the comparison list. This is a planning estimate, not a tested hardware requirement. Validate context length, batch size and concurrent requests on your node.

Can I use Aya 35B 23 · F16 (aya:35b-23-q8_0) commercially?

Aya 35B 23 · F16 (aya:35b-23-q8_0) is released under the Noncommercial terms. This model has noncommercial restrictions. Review the publisher's terms.

Is Aya 35B 23 · F16 (aya:35b-23-q8_0) compatible with the OpenAI API?

Through gpuos, Aya 35B 23 · F16 (aya:35b-23-q8_0) uses the supported /v1/chat/completions endpoint with the model id "aya-35b-23-q8-0". Configure your OpenAI-compatible client with the gpuos base URL and API key. Check model-specific features and any other API operations your app needs before switching.

Which gpuos plan includes Aya 35B 23 · F16 (aya:35b-23-q8_0)?

Aya 35B 23 · F16 (aya:35b-23-q8_0) is part of the full catalog on the Pro plan, $29 per GPU per month.

Related models

• Qwen3 32BChat · ≈ 22 GB · Apache 2.0
• GLM-4 32B 0414Chat · ≈ 20.5 GB · MIT
• Qwen3 30B A3BFast MoE · ≈ 20 GB · Apache 2.0
• DeepSeek R1 Distill Qwen 32BReasoning · ≈ 22 GB · MIT