LIP

Deep learning has made substantial progress in software engineering. Inspired by pre-trained language models in natural language processing, such as BERT, XLNet, RoBERTa, ELECTRA, GPT-2, T5, and BART, code pre-trained models have developed rapidly and been widely studied. Building on existing code pre-training techniques, we proposed a new model, SPT-Code. To help researchers in artificial intelligence and software engineering grasp the state of the art, we surveyed 20 code pre-trained models and discussed their strengths and weaknesses along four dimensions: architecture, data modality, pre-training tasks, and programming languages.