AI Operating System

The positioning, risks, and development of AI OS.

An overview of the AIOS architecture where responsibilities are isolated across different layers.

Challenges of LLM-based Intelligent Agents

Current LLM-based intelligent agents face significant challenges in resource management, particularly with unrestricted access to LLMs or tool resources, which can lead to inefficient resource utilization and even system failures. The lack of proper scheduling and resource management mechanisms restricts parallel processing and affects overall system efficiency. As agent systems become more diverse and complex, effectively managing resources is becoming increasingly important.

Proposed AIOS Architecture

To address these issues, the paper proposes the AIOS architecture, which isolates LLMs and other tool resources from agent applications, creating an AIOS kernel. The AIOS kernel provides fundamental services such as scheduling, context management, memory management, storage management, and access control, and efficiently manages resources like LLMs and external tools. AIOS also includes the AIOS-Agent SDK, a comprehensive suite of APIs designed to help developers utilize the functionalities provided by the AIOS kernel.

Core Design of AIOS

The design of the AIOS kernel involves breaking down LLM-related queries into sub-execution units to enhance concurrency, which are then scheduled and dispatched by an agent scheduler. Memory, storage, and tool managers handle the execution of these dispatched units, while a context manager is designed to handle long-context requests and interruptions in the LLM core. An access manager verifies the access rights of agents to ensure secure operation execution.

Development

Extensive evaluations of AIOS on agents developed using various agent frameworks demonstrate that AIOS significantly improves execution efficiency, particularly when agents run concurrently. AIOS can achieve up to a 2.1x speedup in execution. These experimental results validate the effectiveness of AIOS in optimizing both agent performance and execution speed, especially in resource-constrained environments, while maintaining high performance.