Guidelines for including grey literature and conducting multivocal literature reviews in software engineering

Vahid Garousi; Michael Felderer; Mika V. Mäntylä

doi:10.1016/j.infsof.2018.09.006

journal article Feb 01, 2019

Guidelines for including grey literature and conducting multivocal literature reviews in software engineering

Vahid Garousi

Michael Felderer

Mika V. Mäntylä

Information and Software Technology Vol. 106 pp. 101-121 · Elsevier BV

View at Publisher Save 10.1016/j.infsof.2018.09.006

Topics

No keywords indexed for this article. Browse by subject →

References

112

[1]

Kitchenham "Evidence-based software engineering" (2004)

[2]

Systematic literature reviews in software engineering – A systematic literature review

Barbara Kitchenham, O. Pearl Brereton, David Budgen et al.

Information and Software Technology 2009 10.1016/j.infsof.2008.09.009

[3]

Guidelines for conducting systematic mapping studies in software engineering: An update

Kai Petersen, Sairam Vakkalanka, Ludwik Kuzniarz

Information and Software Technology 2015 10.1016/j.infsof.2015.03.007

[4]

Daigneault "Using systematic review methods within a Ph.D. dissertation in political science: challenges and lessons learned from practice" Int. J. Soc. Res. Methodol. (2014) 10.1080/13645579.2012.730704

[5]

Garousi "Software test-code engineering: a systematic mapping" J. Inf. Softw. Technol. (2015) 10.1016/j.infsof.2014.06.009

[6]

Olsson "Including systematic reviews in PhD programmes and candidatures in nursing – ‘Hobson's choice’?" Nurse Educ. Pract. (2014) 10.1016/j.nepr.2014.01.005

[7]

Gopalakrishnan "Systematic reviews and Meta-analysis: understanding the best evidence in primary healthcare" J. Family Med. Prim. Care (2013) 10.4103/2249-4863.109934

[8]

Schlosser "The role of systematic reviews in evidence-based practice, research, and development" FOCUS (2006)

[9]

Garousi "What we know about testing embedded software" IEEE Softw. (2018)

[10]

Glass (2006)

[11]

Wohlin (2012)

[12]

Ogawa "Towards rigor in reviews of multivocal literatures: applying the exploratory case study method" Rev. Educ. Res. (1991) 10.3102/00346543061003265

[13]

Patton "Towards utility in reviews of multivocal literatures" Rev. Educ. Res. (1991) 10.3102/00346543061003287

[14]

Alberani "The use of grey literature in health sciences: a preliminary survey" Bull. Med. Libr. Assoc. (1990)

[15]

Saleh "Grey literature searching for Health Sciences Systematic Reviews: a prospective study of time spent and resources utilized" Evid. Based Libr. Inf. Pract. (2014) 10.18438/b8dw3k

[16]

Hopewell "Grey literature in meta-analyses of randomized trials of health care interventions" Cochrane Database Syst. Rev. (2007)

[17]

Adams "Shades of grey: guidelines for working with the grey literature in systematic reviews for management and organizational studies" Int. J. Manag. Rev. (2016)

[18]

Tom "An exploration of technical debt" J. Syst. Softw. (2013) 10.1016/j.jss.2012.12.052

[19]

J.P. Higgins and S. Green, "Including unpublished studies in systematic reviews " in Cochrane Handbook for Systematic Reviews of Interventions, http://handbook.cochrane.org/chapter_10/10_3_2_including_unpublished_studies_in_systematic_reviews.htm, Last accessed: Dec. 2017.

[20]

McAuley "Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?" The Lancet (2000) 10.1016/s0140-6736(00)02786-0

[21]

Petersen "Systematic mapping studies in software engineering" (2008)

[22]

B. Kitchenham and S. Charters, "Guidelines for Performing Systematic Literature Reviews in Software engineering," in "EBSE Technical Report," 2007, vol. EBSE-2007-01.

[23]

Schöpfel "Grey literature" (2010)

[24]

Lefebvre "Searching for studies" (2008)

[25]

Giustini "Finding the hard to finds: searching for grey (Gray) literature" UBC (2017)

[26]

Mathers "Global epidemiology of injecting drug use and HIV among people who inject drugs: a systematic review" The Lancet (2008) 10.1016/s0140-6736(08)61311-2

[27]

Kitchenham (2015)

[28]

Santos "Investigating bias in the search phase of Software Engineering secondary studies" (2015)

[29]

Garousi "A systematic literature review of literature reviews in software testing" Inf. Softw. Technol. (2016) 10.1016/j.infsof.2016.09.002

[30]

Garousi "Citations, research topics and active countries in software engineering: a bibliometrics study" Elsevier Comput. Sci. Rev. (2016) 10.1016/j.cosrev.2015.12.002

[31]

Garousi "A bibliometric analysis of the Turkish software engineering research community" Springer J. Scientometr. (2015)

[32]

Garousi "Quantity versus impact of software engineering papers: a quantitative study" Scientometrics (2017) 10.1007/s11192-017-2419-6

[33]

Garousi "What we know about testing embedded software" IEEE Sof., (2017)

[34]

Garousi "A systematic mapping study of web application testing" Elsevier J. Inf. Softw. Technol. (2013) 10.1016/j.infsof.2013.02.006

[35]

Banerjee "Graphical User Interface (GUI) Testing: systematic Mapping and Repository" Inf. Softw. Technol. (2013) 10.1016/j.infsof.2013.03.004

[36]

Garousi "UML-Driven Software Performance Engineering: a Systematic Mapping and Trend Analysis" (2013)

[37]

Escoffery "A grey literature review of special events for promoting cancer screenings" BMC Cancer (2014) 10.1186/1471-2407-14-454

[38]

Favin "Why children are not vaccinated: a review of the grey literature" Int. Health (2012) 10.1016/j.inhe.2012.07.004

[39]

Kennell "A grey literature review of the cultural Olympiad" Cultural Trends (2009) 10.1080/09548960802651294

[40]

Tomka "Reconceptualizing cultural participation in Europe: grey literature review" Cultural Trends (2013) 10.1080/09548963.2013.819657

[41]

Garousi "The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature" (2016)

[42]

Raulamo "Choosing the right test automation tool: a grey literature review" (2017)

[43]

Garousi "Software test maturity assessment and test process improvement: a multivocal literature review" Inf. Softw. Technol. (2017) 10.1016/j.infsof.2017.01.001

[44]

Garousi "When and what to automate in software testing? A multivocal literature review" Inf. Softw. Technol. (2016) 10.1016/j.infsof.2016.04.015

[45]

Elmore "Comment on "Towards rigor in reviews of multivocal Literatures: applying the exploratory case study method"" Rev. Educ. Res. (1991) 10.3102/00346543061003293

[46]

Garousi "Introducing automated GUI testing and observing its benefits: an industrial case study in the context of law-practice management software" (2018)

[47]

Sahaf "When to automate software testing? decision support based on system dynamics – an industrial case study" (2014)

[48]

Amannejad "A search-based approach for cost-effective software test automation decision support and an industrial case study" (2014)

[49]

Garousi "When to automate software testing? a decision-support approach based on process simulation" Wiley J. Softw. (2016)

[50]

Yasin (2012)

Showing 50 of 112 references

Cited By

630

Deriving Empirically Grounded NFR Specifications from Practitioner Discourse: A Validated Methodology Applied to Trustworthy APIs in the AI Era

Apitchaka Singjai · 2026

Information

Evaluating a gamified MOOC

Rodrigo Rodrigues Major, Miguel Mira da Silva · 2025

Cogent Education

Digital transformation initiatives in higher education institutions: A multivocal literature review

Antonio Fernández, Beatriz Gomez · 2023

Education and Information Technolog...

Alexa, what do we know about conversational commerce? Insights from a systematic literature review

Weng Marc Lim, Satish Kumar · 2022

Psychology & Marketing

Empirical Studies on Web Accessibility of Educational Websites: A Systematic Literature Review

Milton Campoverde-Molina, Sergio Luján-Mora · 2020

IEEE Access

Metrics

630

Citations

112

References

Details

Published: Feb 01, 2019
Vol/Issue: 106
Pages: 101-121
License: View

Authors

Funding

Tekes Award: 3192/31/2017

Academy of Finland Award: 298020

Cite This Article

Vahid Garousi, Michael Felderer, Mika V. Mäntylä (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106, 101-121. https://doi.org/10.1016/j.infsof.2018.09.006

Guidelines for including grey literature and conducting multivocal literature reviews in software engineering

You May Also Like