ECE Seminar - InfiAgent: A Multi-Tool Agent for AI Operating Systems

Following the launch of GPT4-Agent, GPT4 has demonstrated its flexibility in utilizing tools like Advanced Data Analytics (ADA, previously known as code interpreter) and DALLE3, although the details of GPT4-Agent have not been fully disclosed. Over the past years, we have intensively studied the core functionalities of GPT4, progressively developing a system comparable to GPT4-Agent, named InfiAgent.  Initially, we replicated Codex and discovered that while existing models such as CodeLlama, StarCoder, and WizardCoder excel in  programming capabilities, they fall short in handling FreeformQA problems for coding. To address this, we created InfiCoder—the first open-source model capable of handling text-tocode, code-to-code, and freeform code-related QA tasks simultaneously. Building on this, we developed InfiCoder-Eval (FreeformQA benchmark), which includes 270 high-quality automated test questions. Our findings indicate that even GPT4 has room for improvement in this area (achieving a score rate of only 59.13%). Based on InfiCoder, we launched the InfiAgent framework, focusing on the field of data analysis. This framework first defines the problem framework and evaluation objectives for data analysis. Then, in line with the data analysis scenarios, we developed a specialized Agent system based on the React format and LLM, effectively addressing data analysis challenges. This system integrates an LLM with programming capabilities and a sandbox environment for executing Python code, generating solutions and corresponding code through multiple rounds of dialogues. It is the industry’s first Agent framework closest to the capabilities of ADA. Additionally, we expanded the application scenarios of InfiAgent, including a mathematical assistant and multimodal LLM (MLLM) reasoning tools, achieving excellent results in math and multimodal reasoning. Particularly in multimodal reasoning, we found that there is significant room for improvement in the current GPT4V (achieving a score rate of only 74.44%). These achievements not only reveal the tremendous potential of InfiAgent but also showcase our possible directions in surpassing the capabilities of GPT4.