INFERENCE RUNTIME FOR RETRIEVAL SYSTEMS

Piramid is an inference engine for retrieval systems on one GPU. It holds the documents, the model weights and the KV cache in a single process, so retrieval can run during generation instead of once before it.

from piramid import Piramid

db = Piramid("http://localhost:6333")
db.embed("notes", ["Compaction rewrites the store without deleted documents."])

answer = db.generate(
    messages=[{"role": "user", "content": "What happens to deleted documents?"}],
    collection="notes",
    k=2,
)
print(answer["text"])

In active research. Results and write-ups will be published here.