AI programming system competes with human coders
The system, called AlphsCode, solves new problems that require a combination of critical thinking, logic, algorithms, coding, and natural language understanding. When evaluated on recent programming competitions on the Codeforces platform, say the researchers, AlphaCode achieved on average a ranking of top 54.3% in programming competitions with more than 5,000 participants.
“We selected for evaluation 10 recent contests, each newer than our training data,” say the researchers. “AlphaCode placed at about the level of the median competitor, marking the first time an AI code generation system has reached a competitive level of performance in programming competitions.”
AlphaCode was designed to address issues with previous attempts to develop AI programming systems, which promise to assist programmers or even generate programs independently. While recent large-scale language models have demonstrated an impressive ability to generate code, and are now able to complete simple programming tasks, say the researchers, these models still perform poorly when evaluated on more complex, unseen problems that require problem-solving skills beyond simply translating instructions into code.
“Creating solutions to unforeseen problems is second nature in human intelligence – a result of critical thinking informed by experience,” say the researchers. “The machine learning community has made tremendous progress in generating and understanding textual data, but advances in problem solving remain limited to relatively simple maths and programming problems, or else retrieving and copying existing solutions.”
By combining advances in large-scale transformer models (that have recently shown promising abilities to generate code) with large-scale sampling and filtering, the researchers say they made significant progress in the number of problems the system can solve. They found that three key components were critical to achieve good and reliable performance: (1) an extensive and clean competitive programming dataset for training and evaluation, (2) large and efficient-to-sample transformer-based architectures, and (3) large-scale model sampling to explore the search space, followed by filtering based on program behavior to a small set of submissions.
AlphaCode, say the researchers, represents an advancement that demonstrates the potential of deep learning models for tasks that require critical thinking.
“These models elegantly leverage modern machine learning to express solutions to problems as code, circling back to the symbolic reasoning root of AI from decades ago,” say the researchers. “And this is only a start. Our exploration into code generation leaves vast room for improvement and hints at even more exciting ideas that could help programmers improve their productivity and open up the field to people who do not currently write code.”
The researchers say they plan to continue this exploration, and hope that further research will result in tools to enhance programming and closer to a problem-solving AI. For more, see “Competition-Level Code Generation with AlphaCode.”
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
