Empirical Study on Synthesis Engines for Semantics-Based Program Repair

Automatic Program Repair (APR) is an emerging and rapidly growing research area, with many techniques proposed to repair defective software. One notable state-of-the-art line of APR approaches is known as semantics-based techniques, e.g., Angelix, which extract semantics constraints, i.e., specifications, via symbolic execution and test suites, and then generate repairs conforming to these constraints using program synthesis. The repair capability of such approaches—expressive power, output quality, and scalability—naturally depends on the underlying synthesis technique. However, despite recent advances in program synthesis, not much attention has been paid to assess, compare, or leverage the variety of available synthesis engine capabilities in an APR context. In this paper, we empirically compare the effectiveness of different synthesis engines for program repair. We do this by implementing a framework on top of the latest semantics-based APR technique, Angelix, that allows us to use different such engines. For this preliminary study, we use a subset of bugs in the IntroClass benchmark, a dataset of many small programs recently proposed for use in evaluating APR techniques, with a focus on assessing output quality. Our initial findings suggest that different synthesis engines have their own strengths and weaknesses, and future work on semantics-based APR should explore innovative ways to exploit and combine multiple synthesis engines.