This study aimed to test enhancements to ChatDev from Communicative Agents for Software Development, with the goal of making ChatDev able to work on non-trivial software development projects. ChatDev is an automated software development department staffed by AI agents that can generate simple software like “Five in a Row” from a short description (Qian et al., 2023). The AI agents used in ChatDev are powered by Large Language Models (LLMs). ChatDev was found to work for small software development projects. However, ChatDev is unable to work on non-trivial software development projects, failing to implement all functions when the project size increases (Qian et al., 2023, p 8). 

New developments have been made in the LLM AI field like better LLMs, and other groups attempts that have built on LLMs to improve their performance since the creation of ChatDev. This study set out to modify ChatDev to be able to utilize MemoryGpt (MemGpt) from MemGPT: Towards LLMs as Operating Systems to enhance LLM memory (Packer et al., 2023), and Tree of Thought (ToT) from Tree of Thoughts: Deliberate Problem Solving with Large Language Models to enhance LLM reasoning ability (Yao et al., 2023). ChatDev, ToT, and MemGpt or any similar project will be collectively referred to as modules in this paper.

An iterative approach to software development, using ChatDev’s ability to work on an existing code base, was used to divide each of the five software development projects that ChatDev was tested on. Each of the projects where divided into a list of software requirements to test ChatDev with. The proposed iterative approach was tested and compared to providing ChatDev with all the project requirements at once, the configuration where ChatDev was given one attempt att implementing all the requirements is called one shot in this document. 

Different permutations of proposed improvements, where experimentally tested to characterize the impact of different configuration parameters on cost, maximum project size, and development time. The tested iterative approach was found to generate 22% more lines on average, fulfill 13% less of the project requirements, and cost 22 times more than one shot on average.

The author was unable to integrate ToT and MemGPT into ChatDev due to technical, time, and resource constraints. The proposed iterative approach to software development was found to be ineffective when applied to ChatDev without any additional enhancements. 

The theoretical work will remain in this document for anyone with more resources who wants to complete the proposed experiment. The partially complete code used to glue modules together, and test the proposed iterative method, for this study is available on Github at github.com/HeWhoThreadsLightly/LLMglue.