At the forefront of software technology, UIUC Zhang Lingming's group, together with researchers from the BigCode organization, recently announced the StarCoder2-15B-Instruct large code model.

This innovative achievement has made a significant breakthrough in the code generation task, successfully surpassing CodeLlama-70B-Instruct and reaching the top of the code generation performance list.

The uniqueness of StarCoder2-15B-Instruct lies in its pure self-alignment strategy. The entire training process is open, transparent, and completely autonomous and controllable.

The model generates thousands of instructions through StarCoder2-15B, in response to fine-tuning the StarCoder-15B base model, without relying on expensive manual annotation data, or from commercial applications such as GPT4 Obtain data from large models to avoid potential copyright issues.

In the HumanEval test, StarCoder2-15B-Instruct stood out with a Pass@1 score of 72.6%, which was improved from 72.0% of CodeLlama-70B-Instruct.

In the evaluation on the LiveCodeBench dataset, this self-aligned model even outperformed similar models trained on GPT-4 generated data. This result demonstrates that a large model can also effectively learn how to align similarly to humans using data within its own distribution, without relying on the biased distribution of the large model from an external teacher.

The successful implementation of this project has received strong support from Arjun Guha’s research group at Northeastern University, University of California, Berkeley, ServiceNow and Hugging Face and other institutions.

Technical Reveal

The data generation process of StarCoder2-Instruct mainly includes three core steps:

1. Collection of seed code snippets: The team selected high-quality, diverse seed functions from The Stack v1 , these functions are drawn from a massive corpus of licensed source code. Through strict filtering and screening, the quality and diversity of seed codes are ensured;

2. Generation of diverse instructions:Based on Different programming concepts in the seed function, StarCoder2-15B-Instruct can create diverse and realistic code instructions. These instructions cover a variety of programming scenarios from data deserialization to list concatenation, recursion, etc.;

3. Generation of high-quality responses:For each instruction, the model adopts a compilation and run-guided self-verification method to ensure that the generated response is accurate and of high quality.

The specific operations of each step are as follows:

The process of selecting seed code snippets

In order to improve the code model's ability to follow instructions, the model needs extensive exposure to and learning of different programming principles and practical operations. StarCoder2-15B-Instruct is inspired by OSS-Instruct and draws inspiration from open source code snippets, especially the well-formatted and clearly structured Python seed functions in The Stack V1.

When building its basic data set, StarCoder2-15B-Instruct conducted in-depth exploration of The Stack V1, selected all Python functions with documented instructions, and automatically analyzed them with the help of the autoimport function and inferred the dependencies required by these functions.

In order to ensure the purity and high quality of the data set, StarCoder2-15B-Instruct has carefully filtered and filtered all selected functions.

First of all, strict type checking is performed through the Pyright type checker to exclude all functions that may produce static errors, thus ensuring the accuracy and reliability of the data.

Next, through precise string matching technology, codes and prompts that are potentially related to the evaluation data set are identified and eliminated to avoid data contamination. In terms of document quality, StarCoder2-15B-Instruct adopts a unique screening mechanism.

It uses its own evaluation capabilities to show 7 sample prompts to the model, allowing the model to judge whether the document quality of each function meets the standard, thereby deciding whether to include it in the final data set.

This method based on model self-judgment not only improves the efficiency and accuracy of data screening, but also ensures the high quality and consistency of the data set.

Finally, in order to avoid data redundancy and duplication, StarCoder2-15B-Instruct uses MinHash and locality-sensitive hashing algorithms to deduplicate functions in the data set. By setting a Jaccard similarity threshold of 0.5, duplicate functions with high similarity are effectively removed, ensuring the uniqueness and diversity of the data set.

After this series of fine screening and filtering, StarCoder2-15B-Instruct finally selected 250,000 high-quality Python functions from 5 million Python functions with documentation. function as its seed data set. This approach is heavily inspired by the MultiPL-T data collection process.

Generation of diverse instructions

##When StarCoder2-15B-Instruct completes the seed After collecting functions, it uses Self-OSS-Instruct technology to create diverse programming instructions. The core of this technology is to enable the StarCoder2-15B base model to autonomously generate corresponding instructions for a given seed code fragment through contextual learning.

To achieve this goal, StarCoder2-15B-Instruct has carefully designed 16 examples, each of which follows the structure of (code snippets, concepts, instructions). The instruction generation process is subdivided into two stages:

Code concept identification: In this stage, StarCoder2-15B will conduct an in-depth analysis of each seed function and generate a code containing A list of key code concepts in this function. These concepts broadly cover the basic principles and techniques in the field of programming, such as pattern matching, data type conversion, etc., which are of extremely high practical value to developers.

Instruction creation: Based on the recognized code concept, StarCoder2-15B will further generate corresponding coding task instructions. This process is designed to ensure that the generated instructions accurately reflect the core functionality and requirements of the code fragment.

Through the above process, StarCoder2-15B-Instruct finally successfully generated up to 238k instructions, greatly enriching its training data set and providing improved performance in programming tasks. Strong support.

Response self-verification mechanism

After obtaining the Self-OSS-Instruct generated After the instruction, the key task of StarCoder2-15B-Instruct is to match a high-quality response to each instruction.

Traditionally, people tend to rely on more powerful teacher models such as GPT-4 to obtain these responses, but not only may this approach face copyright licensing difficulties, but external models are not Always within reach or accurate. More importantly, relying on external models may introduce distribution differences between teachers and students, which may affect the accuracy of the final results.

To overcome these challenges, StarCoder2-15B-Instruct introduces a self-verification mechanism. The core idea of ??this mechanism is to allow the StarCoder2-15B model to create corresponding test cases on its own after generating a natural language response. This process is similar to the self-testing process a developer goes through after writing code.

Specifically, for each instruction, StarCoder2-15B will generate 10 samples containing natural language responses and corresponding test cases. StarCoder2-15B-Instruct then executes these test cases in a sandbox environment to verify the validity of the responses. Any samples that fail in executing the test will be filtered out.

After this strict screening process, StarCoder2-15B-Instruct will randomly select one from the tested responses of each instruction and add it to the final SFT data set. Throughout the process, StarCoder2-15B-Instruct generated a total of 2.4 million response samples (10 samples per instruction) for 238k instructions. After adopting a sampling strategy of 0.7, 500,000 samples successfully passed the execution test.

In order to ensure the diversity and quality of the data set, StarCoder2-15B-Instruct also performs deduplication processing. In the end, 50,000 commands were left, each with a randomly selected, tested and verified high-quality response. These responses constitute the final SFT data set of StarCoder2-15B-Instruct, providing a solid foundation for subsequent training and application of the model.

Superior performance and comprehensive evaluation of StarCoder2-15B-Instruct

In the high-profile EvalPlus benchmark, StarCoder2- With its advantages of scale, 15B-Instruct successfully stood out and became the most outstanding autonomous and controllable large-scale model.

Not only does it surpass the larger Grok-1 Command-R+ and DBRX, it also matches industry leaders such as Snowflake Arctic 480B and Mixtral-8x22B-Instruct.

It is worth mentioning that StarCoder2-15B-Instruct is the first large independent code model to achieve a score of 70+ on the HumanEval benchmark. Its training process is completely transparent, and the use of data and methods complies with laws and regulations.

In the field of independent controllable code large models, StarCoder2-15B-Instruct has significantly surpassed the previous leader OctoCoder, proving its leading position in this field.

Even compared to large and powerful models with restricted licenses such as Gemini Pro and Mistral Large, StarCoder2-15B-Instruct still shows excellent performance and is comparable to CodeLlama-70B-Instruct have equal shares. What is even more remarkable is that StarCoder2-15B-Instruct relies entirely on self-generated data for training, and its performance is comparable to OpenCodeInterpreter-SC2-15B based on GPT-3.5/4 data fine-tuning.

In addition to the EvalPlus benchmark test, StarCoder2-15B-Instruct has also shown strong strength on evaluation platforms such as LiveCodeBench and DS-1000.

LiveCodeBench focuses on evaluating coding challenges that will arise after September 1, 2023, and StarCoder2-15B-Instruct achieved the best results in this benchmark and always leads the usage OpenCodeInterpreter-SC2-15B fine-tuned on GPT-4 data Less, but its performance in this benchmark is still strong, showing broad adaptability and competitiveness.

##Breakthroughs and Enlightenments of StarCoder2-15B-Instruct-v0.1

The release of StarCoder2-15B-Instruct-v0.1 marks an important step for researchers in the field of code model self-tuning. The successful practice of this model breaks the previous limitation of relying on powerful external teacher models such as GPT-4, and demonstrates that code models with excellent performance can also be built through self-tuning.

The core of StarCoder2-15B-Instruct-v0.1 lies in the successful application of its self-alignment strategy in the field of code learning. This strategy not only improves the performance of the model, but more importantly, it gives the model greater transparency and interpretability. This is in stark contrast to other large models such as Snowflake-Arctic, Grok-1, Mixtral-8x22B, DBRX, and CommandR+, which, while powerful, often limit their scope and trustworthiness due to a lack of transparency.

What’s even more gratifying is that StarCoder2-15B-Instruct-v0.1 has made its data set and entire training process-including data collection and training process-completely open source. This move not only demonstrates the open spirit of the researchers, but also lays a solid foundation for future research and development in this field.

There is reason to believe that the successful practice of StarCoder2-15B-Instruct-v0.1 will inspire more researchers to invest in research in the field of code model self-tuning and promote the development of this field. Technological progress and application expansion. At the same time, we also expect more innovative results in this field to continue to emerge, injecting new impetus into the intelligent development of human society.

##About the author

Teacher Zhang Lingming from UIUC is a scholar with profound attainments in the intersection of software engineering, programming language and machine learning. The research group he leads has long been committed to research on automatic software synthesis, repair and verification based on large AI models, as well as improving the reliability of machine learning systems.

Recently, the team has released a number of innovative large code models and test benchmark data sets, and has taken the lead in proposing a series of large model-based software testing and repair technologies. At the same time, thousands of new defects and loopholes were successfully discovered in multiple real software systems, making significant contributions to improving software quality.