LLMs

OpenAI's GPT-6 Models Show Dramatic Gains on Legacy Programming Languages

GPT-6 Astra achieves 87.4% accuracy on multi-language database tasks, up from 26.8% for GPT-5.6 Sol, while COBOL migration accuracy reaches 97.9% at higher reasoning effort.

Last verified:

OpenAI demonstrated substantial improvements in its latest language models’ ability to work with low-resource programming languages—COBOL, Fortran, and Visual Basic—on two newly developed evaluations released on October 19. According to the OpenAI Blog, GPT-6 Astra achieves an 87.4% average pass rate on MultiLangBench across 20 languages, a 60.6 percentage-point jump from GPT-5.6 Sol’s 26.8%, while GPT-6.1 Sol reaches 97.9% accuracy on COBOL migration tasks at higher reasoning effort, compared with 5.4% for the prior model.

The Infrastructure Challenge Behind Legacy Languages

COBOL supports more than 40% of online banking systems and 95% of ATM transactions globally, yet remains underrepresented in AI training data because it is less commonly written in public repositories than Python or JavaScript. According to the OpenAI Blog, Fortran underpins weather forecasting and nuclear reactor safety analysis, making model performance on these languages a practical concern for teams maintaining critical infrastructure. The scarcity of publicly available code and documentation in these languages creates a “low-resource” problem: models trained on typical internet-scale corpora encounter far fewer examples to learn patterns from, leading to poor performance when asked to build or migrate software in these legacy environments.

Benchmark Design and Results

OpenAI developed two complementary evaluations to measure progress. MultiLangBench tasks models with building a SQLite-compatible database engine from scratch, testing code organization, state management, error handling, and feature addition across 20 languages. The COBOL migration evaluation specifically measures whether a model can convert existing programs while preserving their behavior—a critical skill for enterprises modernizing aging mainframe systems without changing business logic.

The performance gaps are dramatic. On MultiLangBench, GPT-6 Astra improved in every language tested, while GPT-6 Luna achieved 40.3% accuracy at an estimated 78% lower API cost than GPT-5.6 Sol. On COBOL migration, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort—39% cheaper than GPT-5.6 Sol—and 97.9% at higher reasoning effort, according to the OpenAI Blog.

Why This Matters

Legacy language support directly affects enterprise software maintenance economics. Teams maintaining COBOL or Fortran systems face declining developer pools, incomplete documentation, and high cost-per-fix. Improved AI capability reduces the friction for both bug fixes and language migrations, enabling smaller teams to manage larger codebases. The cost efficiency gains (39–78% reductions) make AI-assisted development economically viable for enterprises that previously relied solely on human expertise. However, the practical impact depends on whether these benchmark results translate to real-world migration and maintenance workflows—teams will want to validate performance on their own legacy systems before committing to AI-assisted refactoring at scale.

Frequently Asked Questions

Why does OpenAI focus on low-resource programming languages?

COBOL alone powers 40% of online banking, 80% of in-person credit card transactions, and 95% of ATM transactions. Fortran is critical for weather forecasting and nuclear safety. These legacy languages have less training data available, making them harder for LLMs to learn.

What is MultiLangBench and why does it matter?

MultiLangBench tests whether a model can build a SQLite-compatible database engine from scratch in 20 different programming languages. It measures both code organization and feature preservation—skills needed for both new development and legacy system maintenance.

How much cheaper is GPT-6 Luna on MultiLangBench?

GPT-6 Luna achieves 40.3% accuracy at approximately 78% lower estimated API cost compared to GPT-5.6 Sol, making it viable for cost-sensitive enterprise legacy-system work.

What is the COBOL migration evaluation?

OpenAI's COBOL migration evaluation tests whether a model can convert existing programs to another language while preserving their original behavior—a critical skill for modernizing mainframe systems.

#gpt-6 #programming-languages #cobol #legacy-systems #benchmarks