A study conducted by researchers from Brown University in collaboration with multiple Chinese universities explored the capabilities of AI chatbots in expediting the software development process at significantly reduced costs. Chatbots, equipped with cutting-edge language models, have showcased their remarkable efficiency in overseeing a software company with minimal human involvement. They successfully completed the entire software development process in “less than seven minutes, and the total cost incurred was less than one dollar.”
In this experiment, the researchers introduced a hypothetical chatbot named “ChatDev”, which was equipped with ChatGPT’s 3.5 model to simulate a software development company.
The experiment followed a traditional waterfall model, consisting of four sequential stages: designing, coding, testing, and documenting. Each AI bot was assigned specific roles within these stages. For instance, the ‘CEO’ and ‘CTO’ of ChatDev played roles in the ‘designing’ stage, while the ‘programmer’ and ‘art designer’ were responsible for the ‘coding’ stage.
Throughout each stage, the AI bots collaborated with minimal human intervention, engaging in conversations to complete various aspects of the software development process. This ranged from making decisions about the programming language to identifying and addressing bugs in the code until the software project was fully realised.
The researchers concluded that their experimental results clearly showcased the “efficiency and cost-effectiveness” of utilising the automated software development process driven by ChatDev.
What could ChatDev do?
As part of its tasks, ChatDev was assigned the creation of a basic “Gomoku game”, also known as “Five in a Row”, a strategy board game. The CEO and CTO quickly opted for Python as the programming language due to its versatility and simplicity. This efficient decision-making process extended to the coding stage, where the programmer and art designer collaborated to develop an aesthetically pleasing graphical user interface.
The research paper said that throughout the experiment, ChatDev successfully completed a total of 70 tasks, consistently delivering outstanding results. The AI-powered company demonstrated its remarkable efficiency and cost-effectiveness by developing software in less than seven minutes at a cost of under one dollar. An impressive 86.66% of the generated software systems functioned flawlessly.
However, the study had its imperfections. Researchers identified limitations, including errors and biases in the language models, which could potentially lead to issues in software creation. Nevertheless, the researchers believe that these findings could have practical applications, potentially assisting junior programmers or engineers in real-world scenarios.
