Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study (bibtex)
by van Berkel, Niels, Goncalves, Jorge, Hettiachchi, Danula, Wijenayake, Senuri, Kelly, Ryan M. and Kostakos, Vassilis
Abstract:
The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants' perceptions of fairness in private while simultaneously allowing for structured participant discussion.
Reference:
N. van Berkel, J. Goncalves, D. Hettiachchi, S. Wijenayake, R. M. Kelly, V. Kostakos, "Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study", Proceedings of the ACM on Human-Computer Interaction - CSCW, vol. 3, 2019, 28:1-28:21.
Bibtex Entry:
@article{Berkel2019CrowdsourcingFairML,
	Abstract = {The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants' perceptions of fairness in private while simultaneously allowing for structured participant discussion.},
	Author = {van Berkel, Niels and Goncalves, Jorge and Hettiachchi, Danula and Wijenayake, Senuri and Kelly, Ryan M. and Kostakos, Vassilis},
	Doi = {10.1145/3359130},
	Journal = {Proceedings of the ACM on Human-Computer Interaction - CSCW},
	Note = {https://github.com/nielsvanberkel/SupportBot},
	Pages = {28:1-28:21},
	Title = {Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study},
	Type = {Journal Article},
	Url = {https://nielsvanberkel.com/files/publications/cscw2019a.pdf},
	Volume = {3},
	Year = {2019}}
Powered by bibtexbrowser