Ollaya – Ollama for open-source, Jev-style decision models

Hacker News by 3 min read 38x views
Ollaya – Ollama for open-source, Jev-style decision models

Share Post

Ask typed questions concerning any content or JSON and get calibrated answers in milliseconds. Private, open source, on your own hardware.

Real output: routed to laya:en, answered in 8.9 ms on an RTX 4090.

Fast

Decisions in milliseconds.

A decision example answers in a sole onward pass, alongside no token-by-token generation. On your own GPU, a five-question petition to Laya takes concerning 10 ms, end to end through the HTTP API.

Every model, one measure · median latency, lesser is better
  • laya:multilingual8.1 ms
  • laya:en9.6 ms
  • gliclass14.7 ms
  • nli20.4 ms
  • decider:0.8b155 ms
  • decider:2b190 ms
  • TypeSafe Jevhosted API236–276 ms

Ollaya: median of a five-question petition through the HTTP API on an NVIDIA RTX 4090 (laya in fp16, the others in fp32). Jev: median petition latency of the hosted API in third-party benchmarks (AbdelStark/jev-benchmarks, nibzard/decision-model-benchmark), which includes the network. Setups differ, so peruse it as an order-of-magnitude comparison.

Drop-in compatible

Speaks TypeSafe's API.

Ollaya serves /v1/systemone and /v1/models alongside TypeSafe's petition and reply shapes. The authoritative TypeSafe Python SDK 0.7.1 plant unchanged against a local server.

Request

 export TYPESAFE_BASE_URL=http://localhost:11435 export TYPESAFE_API_KEY=local export TYPESAFE_DEFAULT_MODEL=laya curl http://localhost:11435/v1/systemone -d '{ "model": "laya", "state": "Can I get an invoice for final month?", "questions": { "intent": { "type": "choice", "instructions": "What does the client want?", "criteria": { "invoice": "Needs an invoice or receipt", "refund": "Wants prosperity back", "other": "Anything else" } } } }'

Response

{ "model": "laya:en", "answers": { "intent": { "type": "choice", "choice": "invoice", "confidence": 0.9547, "probabilities": { "invoice": 0.9698, "refund": 0.0172, "other": 0.013 } } }, "usage": { "input_tokens": 43, "output_tokens": 0 } }

TypeSafe compatibility guide

Open models

Open weights, prepared to pull.

Start alongside Laya from Convai Innovations: an English model, a 100+ tongue model, a example fine-tuned for typed decisions, and a router that picks for you.

Your data stays yours

Private by default.

Tickets, emails and person messages are frequently the most delicate data you have. With Ollaya they are scored anywhere they already live.

Platforms

Runs anywhere you work.

A desktop app and a command row for macOS, Windows and Linux, and a Docker depiction for servers. Every example runs on the CPU; an NVIDIA GPU on Linux, in WSL 2 or in Docker takes a petition downward to milliseconds.

Install for your phase

NVIDIA GPUs need controller R580 or newer; the installers fetch the CUDA libraries lone whenever they discover one. On Apple, AMD and Intel GPUs, models run on the CPU.

Get up and operating in minutes.

One binary, one command: ollaya run laya.

macOS, Windows, Linux and Docker · Apache-2.0 · GitHub

Other Article Hacker News
↑
Close Right Ads
Close Left Ads