Measuring GitHub Copilot's Impact on Productivity

Albert Ziegler; Eirini Kalliamvakou; X. Alice Li; Andrew Rice; Devon Rifkin; Shawn Simister; Ganesh Sittampalam; Edward Aftandilian

doi:10.1145/3633453

journal article Open Access Feb 22, 2024

Measuring GitHub Copilot's Impact on Productivity

Albert Ziegler Eirini Kalliamvakou X. Alice Li Andrew Rice

Devon Rifkin Shawn Simister Ganesh Sittampalam Edward Aftandilian

Communications of the ACM Vol. 67 No. 3 pp. 54-63 · Association for Computing Machinery (ACM)

View at Publisher Save 10.1145/3633453

Abstract

Case study asks Copilot users about its impact on their productivity, and seeks to find their perceptions mirrored in user data.

Topics

No keywords indexed for this article. Browse by subject →

References

29

[1]

10.1109/saner.2016.39

[2]

Austin J. et al. Program synthesis with large language models. CoRR abs/2108.07732 (2021); https://arxiv.org/abs/2108.07732

[3]

10.1109/icse-seip52600.2021.00022

[4]

Beller, M., Orgovan, V., Buja, S., and Zimmermann, T. Mind the gap: On the relationship between automatically measured and self-reported productivity. IEEE Software 38, 5 (2020), 24--31.

[5]

Chen M. et al. Evaluating large language models trained on code. CoRR abs/2107.03374 (2021); https://arxiv.org/abs/2107.03374

[6]

10.1145/3454122.3454124

[7]

10.1109/icse.2019.00101

[8]

Hendrycks D. et al. Measuring coding challenge competence with APPS. CoRR abs/2105.09938 (2021); https://arxiv.org/abs/2105.09938

[9]

10.1109/icse.2012.6227135

[10]

10.1007/978-1-4842-4221-6_2

[11]

Kulal, S. et al. Spoc: Search-based pseudocode to code. In Proceedings of Advances in Neural Information Processing Systems 32, H.M. Wallach et al (eds), Dec. 2019, 11883--11894; https://bit.ly/3H7YLtF

[12]

Meyer, A.N., Barr, E.T., Bird, C., and Zimmermann, T. Today was a good day: The daily life of software developers. IEEE Transactions on Software Engineering 47, 5 (2019), 863--880.

[13]

10.1109/tse.2017.2656886

[14]

10.1145/2635868.2635892

[15]

Murphy-Hill, E. et al. What predicts software developers' productivity? IEEE Transactions on Software Engineering 47, 3 (2019), 582--594.

[16]

Peng S. Kalliamvakou E. Cihon P. and Demirer M. The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv:2302.06590 [cs.SE] (2014)

[17]

10.1108/14691930410567040

[18]

10.18653/v1/n19-1170

[19]

Storey, M. et al. Towards a theory of software developer job satisfaction and perceived productivity. In Proceedings of the IEEE Trans. on Software Engineering 47, 10 (2019), 2125--2142.

[20]

10.1145/3368089.3417058

[21]

10.1109/msr52588.2021.00045

[22]

Vaithilingam, P., Zhang, T., and Glassman, E. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the 2022 Conf. on Human Factors in Computing Systems.

[23]

10.1145/3491101.3519665

[24]

Wagner, S. and Ruhe, M. A systematic review of productivity factors in software development. arXiv preprint arXiv:1801.06475 (2018).

[25]

Wang, D. et al. From human-human collaboration to human-AI collaboration: Designing AI systems that can work together with people. In Proceedings of the 2020 CHI Conf. on Human Factors in Computing Systems (2020), 1--6.

[26]

10.1145/3397481.3450656

[27]

Winters, T., Manshreck, T., and Wright, H. Software Engineering at Google: Lessons Learned from Programming Over Time. O'Reilly Media (2020).

[28]

PLS-regression: a basic tool of chemometrics

Svante Wold, Michael Sjöström, Lennart Eriksson

Chemometrics and Intelligent Laboratory Systems 10.1016/s0169-7439(01)00155-1

[29]

Zhou W. Kim S. Murali V. and Ari Aye G. Improving code autocompletion with transfer learning. CoRR abs/2105.05991 (2021); https://arxiv.org/abs/2105.05991

Cited By

73

A mechanism-guided multimodal industrial copilot with incremental learning for natural products manufacturing

Qilong Xue, Yang Yu · 2026

Applied Soft Computing

LLM-based harmonized data ingestion for dataspace: a novel system for automating data ingestion across heterogeneous data sources

Parwinder Singh, Michele Gaspari · 2026

Knowledge-Based Systems

Metrics

73

Citations

29

References

Details

Published: Feb 22, 2024
Vol/Issue: 67(3)
Pages: 54-63
License: View

Authors

A

Albert Ziegler