One GPU in my house, serving an open model to a few apps and friends. Below: what it's doing, updated every 10 minutes. How it was built is on YouTube.
| Day | Requests | Tokens | GPU hours | Median time | Answered | Tokens per day |
|---|---|---|---|---|---|---|
| Loading… | ||||||
| Model | Requests | Tokens | Median time | Answered |
|---|---|---|---|---|
| Loading… | ||||
An RTX 3090 in a home server runs the model with the Strata inference engine.
LiteLLM gives each person their own key with its own limits. The API is api.aimemelord.com.
Traffic comes in through a Cloudflare Tunnel, and only the chat routes are let through.