Registry Grid
AI Grid Registry DBs
| Element | Description |
|---|---|
| Assets DB | Database backend for assets at runtime (files, datasets, models). |
| Blocks DB | Tracks deployed blocks (AIOS building units) and their runtime states. |
| Clusters DB | Manages compute clusters, nodes, GPUs, and scheduling metadata. |
| Components Registry | Registry of modular system components (APIs, controllers, services). |
| Container Registries DB | Stores references to container images and registries for deployments. |
| Networks Registry | Stores definitions of virtual networks, overlays, and routing rules. |
| Policies DB | Manages runtime policies (quota, quality, membership, security, etc.). |
| Spec Store | Repository for specifications (schemas, protobufs, templates). |
| Template Store | Stores reusable templates for deployments, jobs, and registry configs. |
| vDAGs DB | Manages virtual Directed Acyclic Graphs (vDAGs) for workflow orchestration. |
| Adhoc Servers DB | Temporary server registry for dynamic or on-demand compute endpoints. |
| Splits Runners Registry | Tracks runtime workers handling split models and distributed execution. |
AGIGrid Registries
| Element | Description |
|---|---|
| Assets Registry | Manages digital and physical asset metadata, lifecycle, and ownership. |
| Contracts Registry | Stores agreements, licenses, and governance rules linked to assets/orgs. |
| DSL Registry | Hosts Domain-Specific Languages for workflows, policies, and task graphs. |
| Exchanges Registry | Tracks exchanges of data, compute, and services between agents/orgs. |
| Functions Registry | Catalog of reusable serverless-style functions and callable units. |
| Orgs Registry | Stores organization metadata, hierarchies, and federation mappings. |
| Org Tasks Registry | Manages organization-level tasks, assignments, and workflows. |
| Registry of Registries | The discovery hub - maps and resolves all available registries in the Grid. |
| Subjects Registry | Handles known subjects (humans, agents, systems) and their identifiers. |
| Tools Registry | Repository of pluggable tools and executors accessible to agents. |
Core Registry SDK Layer
| Element | Description |
|---|---|
| Registry SDK | Client library to create, read, update, delete, query, sign, and validate records. Handles connection negotiation, retries, streaming results, and typed errors. |
| Schema & Types Module | Strongly typed entities, relationships, and constraints with versioned schemas. Ships validators, and migration helpers for forward and backward compatibility. |
| Event & Changefeed API | Subscribe to inserts, updates, deletes with at-least-once delivery semantics. Emits typed events suitable for webhooks, MQ, or mesh relays. |
| Audit Ledger | Append only records for create, update, delete, and access. Actor, reason, checksum, and before after snapshots. |
| Checksums and Signatures | MD5 or SHA families per part and whole asset. Optional detached signatures with KMS backed keys. |
| Quota and Billing | Track storage, egress, requests, and job time per tenant. Hard and soft limits with notifications. |
| Metrics and Tracing | Collects numbers on requests, errors, and response times. Traces requests across services to spot slow or failing parts. |
| Health and Alerting | Checks if services are up and working correctly. Triggers alerts or auto-fix actions when issues appear. |
API Layer
| Element | Description |
|---|---|
| REST Gateway | CRUD for assets, parts, manifests, and presigned URLs. Request shaping, rate limits, pagination, and deterministic errors. |
| GraphQL Gateway | Unified graph across assets, metadata, and jobs. Typed inputs, field resolvers with batching, and query cost hints. |
| Request Router | Routes to processing, storage, or streaming services. Supports canary routing, blue green, and version negotiation. |
| Validation Filters | JSONSchema and business rule validation at ingress. Early reject with rich error payloads and trace IDs. |
| Background Job API | Submit long running tasks and poll job status. Idempotency keys, retries, and dead letter reporting. |
Processing Layer
| Element | Description |
|---|---|
| ZIP Parser Service | Stream parse central directory and file entries. Extract manifests, checksums, and logical structure. |
| Multipart Upload Coordinator | Breaks large files into parts and orchestrates S3 compatible uploads. Tracks, resumes uploads, and finalizes commits. |
| Metadata Extractor | Reads files to pull out basic metadata details & turns all that information into a clean, standard format so it’s easy to search and use later. |
| Async Workers | Thread or process pool for CPU and I/O heavy tasks. Bounded queues, backpressure, and visibility timeouts. |
| Policy Scanner | Content inspection with pluggable engines. Pre write and pre download checks with audit trails. Enforces restrictions, constraints, license checks, and size caps. |
Service Layer
| Element | Description |
|---|---|
| Service Gateway | Single endpoint for invoking services (HTTP/gRPC). Routes by service name/version; handles auth, rate limits, and quotas. |
| Predefined Service Catalog | Registry of available services. Supports discovery, docs, and per-tenant enable/disable. |
| Microservice Runtime | Runs predefined services as containerized apps. Scales horizontally; supports request/response, batch, and cron. |
| Function Runtime (Serverless) | Runs small, user-supplied functions in WebAssembly or sandboxed VMs. Fast cold starts, strict CPU/memory/time limits. |
| Workflow Engine | Orchestrates multi-step jobs across services (DAG). |
| Event Triggers | Run services on registry events: insert/update/delete, catalog publish. Supports filters (entity type, tag, tenant) and batching windows. |
| Schedulers | Cron/timer triggers for crawl, refresh, or reconciliation jobs. Per-tenant schedules with conflict checks. |
| Sandboxing | User services run in isolated sandboxes. No raw network or disk unless explicitly granted. |
| Custom service | User defined services or workflows |
Storage Layer
| Element | Description |
|---|---|
| MongoDB Metadata Store | Metadata collections for assets, parts, jobs, and manifests. Secondary indexes for lookups and unique keys for dedupe. |
| Object Store Adapter | Stores large assets in external object storage such as S3 and Ceph Server side encryption, checksum verification, and lifecycle rules. |
| Redis Progress Tracker | Tracks job states, upload cursors, and locks. TTL based progress keys and heartbeat renewal. |
| Retention and Deletion | Time based retention, legal hold, and purge workflows. Verified delete across Mongo, object store, and caches. |
Streaming Layer
| Element | Description |
|---|---|
| In Memory Streams | In memory File like objects for zero copy handoff between services. Backpressure signals and bounded memory pools. |
| ZIP Reconstructor | Rebuilds ZIPs on the fly from multiple stored parts and URLs. |
| Range Read Service | Byte range fetches from object storage for partial downloads. Supports HTTP range requests and sparse reads. |
| Checksum Streamer | Inline hashing for integrity without extra passes. Emits digests for audit and client verification. |
Upload and Download Layer
| Element | Description |
|---|---|
| Upload Session Manager | Creates sessions, presigns part URLs, and enforces quotas. Supports pause, resume, and cancel with idempotent finalize. |
| Upload Ingest API | Accepts direct and 3rd party client based uploads. Validates content type, size, and client side checksums. |
| Download Composer | Serves single files, reconstructed ZIPs, or curated bundles. |
| Presigned URL Service | Time bound URLs for read and write with least privilege. Policy scoped paths, content length limits, and IP allowlists. |
| CDN Edge Integration | Edge caching for hot downloads and range support. |
Security and Identity
| Element | Description |
|---|---|
| AuthN Broker | OIDC, API keys, and workload identities. Session rotation and token introspection. |
| RBAC and ABAC Engine | Role and attribute based rules at field granularity. Tenancy isolation and jurisdiction aware gates. |
| Secrets Manager | KMS or Vault backed encryption keys and credentials. Envelope encryption and automatic key rotation. |
Search & Discovery Layer
| Element | Description |
|---|---|
| Index Builder | Processes registry metadata and builds search-friendly indexes. |
| Catalog Indexer | Continuously ingests product/service catalogs from registries. Normalizes fields like name, description, category, and price for discovery. |
| Query & filtering Service | Central API for structured queries & attribute-based filtering across metadata and catalogs. Handles filters, ranges, facets, sorting, and deterministic pagination. |
| Full-Text Search Engine | Enables keyword and phrase searches over asset descriptions. |
| Ranking & Scoring Engine | Calculates relevance using keyword matches, freshness, and trust signals. Supports boosting (e.g., verified entities, local data) and tunable weights. |
| Semantic Search | Uses embeddings for similarity-based discovery (vector search). Helps find related products or services beyond exact keyword matches. |
| Federated Discovery Connector | Routes search queries to external registries or networks. Merges responses into a single, normalized result set. |
| Saved Views & Materialized Queries | Lets users define reusable queries and store their results. Supports auto-refreshing views for common discovery needs. |
Federation, Sync, and Replication
| Element | Description |
|---|---|
| Federation Registry | Directory of verified participants (IDs, roles, capabilities, endpoints, keys). Exposes lookup APIs and caches public keys for signature checks. |
| Policy Guard (Inter-Network) | Enforces who can call what, where, and how often. Checks consent, purpose, jurisdiction, and payload size/type. |
| Changefeed Publisher | Continuously streams every insert, update, and delete from the registry. Packages changes into ordered events so other systems can subscribe, replay, or sync data in near real time. |
| Changefeed Subscriber | Listens to change events published by other registries or peers. Validates and applies those changes locally to keep data consistent and up to date. |
| Replication Agent (Region/Mirror) | Mirrors data to regional clusters for latency and compliance. At-least-once with idempotent writes and point-in-time replays. |
| Dispute & Conflict Resolver | Tracks mismatches, automates evidence collection and resolution workflows, Policy guided merge. Human friendly diffs and automated reconciliations. |