Please find an overview of my publications below. I publicly share the PDF of all my published work.
Google Scholar profile.
2025
S. de Jong, V. Paananen, B. Tag, N. van Berkel, "Cognitive Forcing for Better Decision-Making: Reducing Overreliance on AI Systems Through Partial Explanations", Proceedings of the ACM on Human-Computer Interaction – CSCW, 2025, to appear.
Abstract: In AI-assisted decision-making, explanations aim to enhance transparency and user trust but can also lead to negligence. In two separate studies, we explore the use of partial explanations to activate cognitive forcing and increase user engagement. In Study I (N = 264), we present participants with weighted graphs and ask them to identify the shortest paths. In Study II (N = 210), participants correct spelling and grammar mistakes in short text segments. In both studies, we provide a solution suggestion accompanied by either no explanation, a full explanation, or a partial explanation. Our results show that partial explanations reduce overreliance on incorrect AI suggestions, performing significantly better than the baseline but not as well as full explanations. Individuals with a high need for cognition benefit more from AI explanations and consequently perform better. Our work suggests that partial explanations can be valuable in domains where reducing overreliance on AI is critical, like medical diagnosis. It also underscores the need to consider explanation effectiveness across different task difficulties, a factor often overlooked in contemporary human-AI studies.Close abstract J. Delaunay, L. Galárraga, C. Largouet, N. van Berkel, "Impact of Explanation Technique and Representation on Users’ Comprehension and Confidence in Explainable AI", Proceedings of the ACM on Human-Computer Interaction – CSCW, 2025, to appear.
Abstract: Local explainability, an important sub-field of eXplainable AI, focuses on describing the decisions of AI models for individual use cases by providing the underlying relationships between a model’s inputs and outputs. While the machine learning community has made substantial progress in improving explanation accuracy and completeness, these explanations are rarely evaluated by the final users. In this paper, we evaluate the impact of various explanation and representation techniques on users’ comprehension and confidence. Through a user study on two different domains, we assessed three commonly used local explanation techniques—feature-attribution, rule-based, and counterfactual—and explored how their visual representation—graphical or text-based—influences users’ comprehension and trust. Our results show that the choice of explanation technique primarily affects user comprehension, whereas the graphical representation impacts user confidence.Close abstract 2024
N. van Berkel, H. Pohl, "Collaborating with Bots and Automation on OpenStreetMap", ACM Transactions on Computer-Human Interaction, vol. 31, no. 3, 2024, 1–30.
Abstract: OpenStreetMap (OSM) is a large online community where users collaborate to map the world. In addition to manual edits, the OSM mapping database is regularly modified by bots and automated edits. In this paper, we seek to better understand how people and bots interact and conflict with each other. We start by analysing over 15 years of mailing list discussions related to bots and automated edits. From this data, we uncover five themes, including how automation results in power differentials between users and how community ideals of consensus clash with the realities of bot use. Subsequently, we surveyed OSM contributors on their experiences with bots and automated edits. We present findings about the current escalation and review mechanisms, as well as the lack of appropriate tools for evaluating and discussing bots. We discuss how OSM and similar communities could use these findings to better support collaboration between humans and bots.Close abstract J. Wester, T. Schrills, H. Pohl, N. van Berkel, "“As an AI language model, I cannot”: Investigating LLM Denials of User Requests", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’24), 2024, 1–14.
Abstract: Users ask large language models (LLMs) to help with their homework, for lifestyle advice, or for support in making challenging decisions. Yet LLMs are often unable to fulfil these requests, either as a result of their technical inabilities or policies restricting their responses. To investigate the effect of LLMs denying user requests, we evaluate participants’ perceptions of different denial styles. We compare specific denial styles (baseline, factual, diverting, and opinionated) across two studies, respectively focusing on LLM’s technical limitations and their social policy restrictions. Our results indicate significant differences in users’ perceptions of the denials between the denial styles. The baseline denial, which provided participants with brief denials without any motivation, was rated significantly higher on frustration and significantly lower on usefulness, appropriateness, and relevance. In contrast, we found that participants generally appreciated the diverting denial style. We provide design recommendations for LLM denials that better meet peoples’ denial expectations.Close abstract S. Pareek, N. van Berkel, E. Velloso, J. Goncalves, "Effect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment", Proceedings of the ACM on Human-Computer Interaction – CSCW, vol. 8, no. CSCW2, 2024, 1–31.
Abstract: As misinformation increasingly proliferates on social media platforms, it has become crucial to explore how to best convey automated news credibility assessments to end-users, and foster trust in fact-checking AIs. In this paper, we investigate how model-agnostic, natural language explanations influence trust and reliance on a fact-checking AI. We construct explanations from four Conceptualisation Validations (CVs) — namely Consensual, Expert, Internal (Logical), and Empirical — which are foundational units of evidence that humans utilise to validate and accept new information. Our results show that providing explanations significantly enhances trust in AI, even in a fact-checking context where influencing pre-existing beliefs is often challenging, with different CVs causing varying degrees of reliance. We find Consensual explanations to be the least influential, with Expert, Internal, and Empirical explanations exerting twice as much influence. However, we also find that users could not discern whether the AI directed them towards the truth, highlighting the dual nature of automated credibility assessments to both guide and potentially mislead. Further, we uncover the presence of automation bias and aversion during collaborative fact-checking, indicating how users’ previously established trust in AI can moderate their reliance on AI judgements. We also observe the manifestation of a ‘boomerang’ effect often seen in traditional corrections to misinformation, with individuals who perceive AI as biased or untrustworthy doubling down and reinforcing their existing (in)correct beliefs when challenged by the AI. We conclude by presenting nuanced insights into the dynamics of user behaviour during AI-based fact-checking, offering important lessons for social media platforms.Close abstract J. Wester, B. Moghe, K. Winkle, N. van Berkel, "Facing LLMs: Robot Communication Styles in Mediating Health Information between Parents and Young Adults", Proceedings of the ACM on Human-Computer Interaction – CSCW, vol. 8, no. CSCW2, 2024, 1–37.
Abstract: Young adults may feel embarrassed when disclosing sensitive information to their parents, while parents might similarly avoid sharing sensitive aspects of their lives with their children. How to design interactive interventions that are sensitive to the needs of both younger and older family members in mediating sensitive information remains an open question. In this paper, we explore the integration of large language models (LLMs) with social robots. Specifically, we use GPT-4 to adapt different Robot Communication Styles (RCS) for a social robot mediator designed to elicit self-disclosure and mediate health information between parents and young adults living apart. We design and compare four literature-informed RCS: three LLM-adapted (Humorous, Self-deprecating, and Persuasive) and one manually created (Human-scripted), and assess participant perceptions of Likeability, Usefulness, Helpfulness, Relatedness, and Interpersonal Closeness. Through an online experiment with 183 participants, we assess the RCS across two groups: adults with children (Parents) and young adults without children (Young Adults). Our results indicate that both Parents and Young Adults favoured the Human-scripted and Self-deprecating RCS as compared to the other two RCS. The Self-deprecating RCS furthermore led to increased relatedness as compared to the Humorous RCS. Our qualitative findings reveal challenges people have in disclosing health information to family members, and who normally assumes the role of family facilitator—two areas in which social robots can play a key role. The findings offer insights for integrating LLMs with social robots in health-mediation and other contexts involving the sharing of sensitive information.Close abstract N. van Berkel, B. Tag, R. M. Jacobsen, D. Russo, H. C. Purchase, D. Buschek, "Impact of Interaction Technique in Interactive Data Visualisations: A Study on Lookup, Comparison, and Relation-seeking Tasks", International Journal of Human-Computer Studies, vol. 192, 2024, 103359.
Abstract: This paper presents an analysis of different interaction techniques used in interactive data visualisations to support end-users in visual analytics tasks. Our selection of interaction techniques is based on prior work and consists of the interaction techniques Select, Explore, Reconfigure, Encode, Filter, Abstract/Elaborate, and Connect. Through a within-subject study, we assessed participants’ abilities to utilise these techniques when faced with three distinct types of data-driven tasks; lookup, comparison, and Relation-seeking. Our research investigates the impact of these interaction techniques on the correctness, confidence, perceived difficulty, and cognitive load of N = 80 self-identified data scientists and N = 80 non-experts. We find that interaction technique significantly impacts answer correctness and participant confidence. Participants performed best across those interaction techniques that allow for information that is deemed least relevant to be concealed, which is reflected in lower intrinsic and extraneous cognitive load. Interestingly, participants’ expertise affected their confidence but not their accuracy. Our results provide insights useful for a more targeted and informed design and usage of interactive data visualisations.Close abstract N. K. Kollerup, J. Wester, M. B. Skov, N. van Berkel, "How Can I Signal You To Trust Me: Investigating AI Trust Signalling in Clinical Self-Assessments", in Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems (DIS’24), 2024, 525–540.
Abstract: Individuals are increasingly interested in and responsible for assessing their own health. This study evaluates a fictional AI dermatologist for assistance in the self-assessment of moles. Building on the Signalling Theory, we tested the effect of textual descriptions provided by an AI dermatologist, as manipulated across ‘Ability’, ‘Integrity,’ and ‘Benevolence’, along with the clinical assessment, ‘benign’ or ‘malignant’, affect users’ trust in the aforementioned trust pillars. Our study (N = 39) follows a 2 (Ability low/high) x 2 (Integrity low/high) x 2 (Benevolence low/high) x 2 (mole assessment benign/malignant) within-subject factorial design. Our results demonstrate that we can successfully influence perceptions of ability and benevolence by manipulating the corresponding aspects of trust but not perceived integrity. Further, in the case of a malignant assessment, participants’ perception of trust increased across all aspects. Our results provide insights into the design of AI support systems for sensitive use cases, such as clinical self-assessments.Close abstract S. Shalawadi, C. Getschmann, N. van Berkel, F. Echtler, "Manual, Hybrid, and Automatic Privacy Covers for Smart Home Cameras", in Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems (DIS’24), 2024, 3453–3470.
Abstract: Smart home cameras (SHCs) offer convenience and security to users, but also cause greater privacy concerns than other sensors due to constant collection and processing of sensitive data. Moreover, privacy perceptions may differ between primary users and other users at home. To address these issues, we developed three physical cover prototypes for SHCs: Manual, Hybrid, and Automatic, based on design criteria of observability, understandability, and tangibility. With 90 SHC users, we ran an online survey using video vignettes of the prototypes. We evaluated how the physical covers alleviated privacy concerns by measuring perceived creepiness and trustworthiness. Our results show that the physical covers were well received, even though primary SHC users valued always-on surveillance. We advocate for the integration of physical covers into future SHCs, emphasizing their potential to establish a shared understanding of surveillance status. Additionally, we provide design recommendations to support this proposition.Close abstract S. B. Kjaerulff, S. B. Pedersen, T. J. Sigvardsen, N. van Berkel, E. Papachristos, "Exploring VUI-Supported Mindfulness Techniques for Smoking Cessation", in Proceedings of ACM Conference on Conversational User Interfaces (CUI’24), 2024, 1–11.
Abstract: This study investigates the effectiveness of Voice User Interfaces (VUIs) in supporting mindfulness techniques for smoking cessation. We conducted a month-long between-subject study involving nine participants, comparing a VUI on smart speakers against an augmented VUI (a blend of VUI and Graphical User Interface) on mobile devices. Specifically, we evaluated how these interfaces support individuals in quitting smoking through mindfulness practices. Our results include qualitative insights on participants’ experiences with mindfulness, their smoking cessation motivation, and engagement with the VUI prototypes, alongside quantitative data on their usage patterns. Our findings offer insights into the potential application of VUIs in smoking cessation and suggest design guidelines for future health-oriented applications. The study underscores the importance of device context in designing effective health interventions and sets the direction for future work in HCI and mindfulness applications.Close abstract N. van Berkel, A. Visuri, S. Shalawadi, M. R. Evans, B. Tag, S. Hosio, "From Reflection to Action: Enhancing Workplace Well-being through Digital Solutions", Interacting with Computers, 2024, 1–14.
Abstract: Despite the widely acknowledged importance of well-being, our well-being can regularly be under pressure from external sources. Work is often attributed as a source of stress and dissatisfaction, so, unsurprisingly, extensive efforts are made to measure and improve our well-being in this context. This paper examines opportunities to better design supportive digital solutions through two complementary studies. In the first study, we present a longitudinal assessment of a well-being-focused self-report application deployed in two organisations. Through an analysis of one year of application usage across 219 users, we find both established and novel patterns of application usage and well-being evaluation. While prior work has highlighted substantial dropout rates and daily well-being fluctuations that peak in the morning and early evening, our results highlight that substantial breaks in usage are common, suggesting that users choose to engage with well-being applications mainly when they need them. In the second study, we expand on the topic of well-being reflection at work and the use of technology for this purpose. Through a survey involving 100 participants, we identify current practices in increasing well-being at work, obstacles to sharing and discussing mental well-being states, opportunities for digital well-being solutions, and reflections on transparency and communication. Our combined results highlight opportunities for HCI research and practice to address the ongoing challenges of maintaining well-being in today’s work environments.Close abstract J. Wester, S. de Jong, H. Pohl, N. van Berkel, "Exploring People’s Perceptions of LLM-generated Advice", Computers in Human Behavior: Artificial Humans, vol. 2, no. 2, 2024, 100072.
Abstract: When searching and browsing the web, more and more of the information we encounter is generated or mediated through large language models (LLMs). This can be looking for a recipe, getting help on an essay, or looking for relationship advice. Yet, there is limited understanding of how individuals perceive advice provided by these LLMs. In this paper, we explore people’s perception of LLM-generated advice, and what role diverse user characteristics (i.e., personality and technology readiness) play in shaping their perception.Further, as LLM-generated advice can be difficult to distinguish from human advice, we assess the perceived creepiness of such advice. To investigate this, we run an exploratory study (\textitN = 91), where participants rate advice in different styles (generated by GPT-3.5 Turbo). Notably, our findings suggest that individuals who identify as more agreeable tend to like the advice more and find it more useful. Further, individuals with higher technological insecurity are more likely to follow and find the advice more useful, and deem it more likely that a friend could have given the advice. Lastly, we see that advice given in a `skeptical’ style was rated most unpredictable, and advice given in a `whimsical’ style was rated least malicious—indicating that LLM advice styles influence user perceptions. Our results also provide an overview of people’s considerations on likelihood, receptiveness, and what advice they are likely to seek from these digital assistants. Based on our results, we provide design takeaways for LLM-generated advice and outline future research directions to further inform the design of LLM-generated advice for support applications targeting people with diverse expectations and needs.Close abstract N. K. Kollerup, S. S. Johansen, M. G. Tolsgaard, M. L. Friis, M. B. Skov, N. van Berkel, "Clinical Needs and Preferences for AI-based Explanations in Clinical Simulation Training", Behaviour & Information Technology, 2024, 1–25.
Abstract: Medical training is a key element in maintaining and improving today’s healthcare standards. Given the nature of medical work, students must master not only theory but also develop their hands-on abilities and skills in clinical practice. Medical simulators play an increasing role in supporting the active learning of these students due to their ability to present a large variety of tasks allowing students to train and experiment indefinitely without causing any patient harm. While the criticality of explainable AI systems has been extensively discussed in the literature, the medical training context presents unique user needs for explanations. In this paper, we explore the potential gap of current limitations within simulation-based training, and the role Artificial Intelligence (AI) holds in supporting the needs of medical students in training. Through contextual inquiries and interviews with clinicians in training (N = 9) and subsequent validation with medical experts (N = 4), we obtain an understanding of the shortcomings in current simulation-based training and offer recommendations for future AI-driven training. Our results stress the need for continuous and actionable feedback that resembles the interaction between clinical supervisor and resident in real-world training scenarios while adjusting training material to the residents’ skills and prior performance.Close abstract A. Alorwu, N. van Berkel, A. Visuri, S. Suryanarayana, T. Yoshihiro, S. Hosio, "Monetary valuation of personal health data in the wild", International Journal of Human-Computer Studies, vol. 185, 2024, 103241.
Abstract: The value of personal health data continues to be a debated topic in HCI and society more broadly. We investigate the monetary value people attach to their health data. Using a custom mobile app for 14 days with 55 participants, we collected health data (sleep duration, sleep quality, pain intensity, wake-up times) and a daily monetary data valuation using a reverse second-price auction. Participants bid to sell their data to a for-profit company, the government, or academia. Our findings indicate that people value their data differently based on who is buying. We also show that people are interested in monetizing their personal health data despite privacy and data protection concerns. The presented study helps us understand the data value landscape and paves way to a healthier data-driven future where people may benefit more from their own contributions, either in monetary or other forms.Close abstract J. Wester, H. Pohl, S. Hosio, N. van Berkel, "“This Chatbot Would Never…”: Perceived Moral Agency of Mental Health Chatbots", Proceedings of the ACM on Human-Computer Interaction – CSCW, vol. 8, no. CSCW1, 2024.
Abstract: Despite repeated reports of socially inappropriate and dangerous chatbot behaviour, chatbots are increasingly used as mental health services in providing support for young people. In sensitive settings as such, the notion of perceived moral agency (PMA) is crucial, given its critical role in human-human interactions. In this paper, we investigate the role of PMA in human-chatbot interactions. Specifically, we seek to understand how PMA influence the perception of trust, likeability, and perceived safety of chatbots for mental health across two distinct age groups. We conduct an online experiment (N = 279) to evaluate chatbots with low and high PMA as targeted towards teenagers and adults. Our results indicate increased trust, likeability, and perceived safety in mental health chatbots displaying high PMA. A qualitative analysis revealed four themes, assessing participants’ expectations of mental health chatbots in general, as well as targeted towards teenagers: Anthropomorphism, Warmth, Sensitivity, and Appearance manifestation. We show that PMA plays a crucial role in influencing the perceptions of chatbots and provide recommendations for designing socially appropriate mental health chatbots.Close abstract S. de Jong, J. Wester, T. Schrills, K. Laursen, C. Griggio, N. van Berkel, "Assessing Cognitive and Social Awareness among Group Members in AI-assisted Collaboration", in Proceedings of the 23rd International Conference on Mobile and Ubiquitous Multimedia (MUM’24), 2024, 338–350.
Abstract: Successful collaboration in computer-mediated teams requires awareness among group members of each other’s knowledge, skills, and goals. Large Language Models (LLMs) can play a mediating role in establishing and maintaining this awareness among group members. In an in-situ study, we explored the impact of an LLM-based chatbot on social and cognitive group awareness through a distributed text-based group task. We instructed participants (N = 48) to complete a travel-planning task in sixteen groups of three, with each member given conflicting goals. Each chat was complemented by a chatbot that could be asked for assistance. Through a survey and semi-structured interview, we gained insight into participants’ deliberations on the task and the chatbot’s role. We found that the chatbot’s presence helped increase group awareness as users are forced to clearly and transparently formulate their intentions when prompting the chatbot. The chatbot’s ability to provide suggestions that compromise between user goals based on the chat history helped participants reach a consensus. We present implications for the design of chatbots for collaborative settings.Close abstract K. Ravishan, D. Szabó, N. van Berkel, A. Visuri, C. Yang, K. Yatani, S. Hosio, "From Voice to Value: Leveraging AI to Enhance Spoken Online Reviews on the Go", in Proceedings of the 23rd International Conference on Mobile and Ubiquitous Multimedia (MUM’24), 2024, 351–364.
Abstract: Online reviews help people make better decisions. Review platforms usually depend on typed input, where leaving a good review requires significant effort because users must carefully organize and articulate their thoughts. This may discourage users from leaving comprehensive and high-quality reviews, especially when they are on the go. To address this challenge, we developed Vocalizer, a mobile application that enables users to provide reviews through voice input, with enhancements from a large language model (LLM). In a longitudinal study, we analysed user interactions with the app, focusing on AI-driven features that help refine and improve reviews. Our findings show that users frequently utilized the AI agent to add more detailed information to their reviews. We also show how interactive AI features can improve users’ self-efficacy and willingness to share reviews online. Finally, we discuss the opportunities and challenges of integrating AI assistance into review-writing systems.Close abstract M. Bahodi, N. van Berkel, M. Skov, T. Merritt, "Show Me What’s Wrong: Impact of Explicit Alerts on Novice Supervisors of a Multi-Robot Monitoring System", in International Symposium on Trustworthy Autonomous Systems (TAS’24), 2024, 1–17.
Abstract: With the rise of autonomous multi-robot systems, the role of the robot operator shifts from controlling and observing a single robot to that of a supervisor overseeing multiple robots. Previous studies suggest that timely warnings of problematic events improve a user’s ability to monitor multiple robots, however, research has not examined the influence alerts have on user monitoring behavior and their perceptions of the system. We present a 2×2 study design where we manipulate the operator’s cognitive load through the number of simultaneous robots under their control and the level of support through the absence or presence of warnings. Our findings suggest that users are more likely to fixate on alerts when shown explicitly, and task difficulty influenced the user’s willingness to allow the robots to act autonomously. Our research offers insights for advancing the design of autonomous multi-robot interfaces, emphasizing strategies to enhance the simultaneous monitoring of numerous robots.Close abstract D. Russo, S. Baltes, N. van Berkel, P. Avgeriou, F. Calefato, B. Cabrero-Daniel, G. Catolino, J. Cito, N. Ernst, T. Fritz, H. Hata, R. Holmes, M. Izadi, F. Khomh, M. B. Kjærgaard, G. Liebel, A. L. Lafuente, S. Lambiase, W. Maalej, G. Murphy, N. B. Moe, G. O’Brien, E. Paja, M. Pezzè, J. S. Persson, R. Prikladnicki, P. Ralph, M. Robillard, T. R. Silva, K. Stol, M. Storey, V. Stray, P. Tell, C. Treude, B. Vasilescu, "Generative AI in Software Engineering Must Be Human-Centered: The Copenhagen Manifesto", Journal of Systems and Software, vol. 216, 2024, 112115.
Abstract: The COVID-19 pandemic has brought significant and enduring shifts in various aspects of life, including increased flexibility in work arrangements. In a longitudinal study, spanning 24 months with six measurement points from April 2020 to April 2022, we explore changes in well-being, productivity, social contacts, and needs of software engineers during this time. Our findings indicate systematic changes in various variables. For example, well-being and quality of social contacts increased while emotional loneliness decreased as lockdown measures were relaxed. Conversely, people’s boredom and productivity, remained stable. Furthermore, a preliminary investigation into the future of work at the end of the pandemic revealed a consensus among developers for a preference of hybrid work arrangements. We also discovered that prior job changes and low job satisfaction were consistently linked to intentions to change jobs if current work conditions do not meet developers’ needs. This highlights the need for software organizations to adapt to various work arrangements to remain competitive employers. Building upon our findings and the existing literature, we introduce the Integrated Job Demands-Resources and Self-Determination (IJARS) Model as a comprehensive framework to explain the well-being and productivity of software engineers during the COVID-19 pandemic.Close abstract D. Russo, P. H. P. Hanel and N. van Berkel, "Understanding Developers Well-Being and Productivity: A 2-year Longitudinal Analysis during the COVID-19 Pandemic", ACM Transactions on Software Engineering and Methodology, vol. 33, no. 3, 2024, 1–48.
Abstract: The COVID-19 pandemic has brought significant and enduring shifts in various aspects of life, including increased flexibility in work arrangements. In a longitudinal study, spanning 24 months with six measurement points from April 2020 to April 2022, we explore changes in well-being, productivity, social contacts, and needs of software engineers during this time. Our findings indicate systematic changes in various variables. For example, well-being and quality of social contacts increased while emotional loneliness decreased as lockdown measures were relaxed. Conversely, people’s boredom and productivity, remained stable. Furthermore, a preliminary investigation into the future of work at the end of the pandemic revealed a consensus among developers for a preference of hybrid work arrangements. We also discovered that prior job changes and low job satisfaction were consistently linked to intentions to change jobs if current work conditions do not meet developers’ needs. This highlights the need for software organizations to adapt to various work arrangements to remain competitive employers. Building upon our findings and the existing literature, we introduce the Integrated Job Demands-Resources and Self-Determination (IJARS) Model as a comprehensive framework to explain the well-being and productivity of software engineers during the COVID-19 pandemic.Close abstract 2023
A. Visuri, N. van Berkel and B. Tag, "Wellbeing Insights in a Data-Driven Future", in 2023 Fourteenth International Conference on Mobile Computing and Ubiquitous Network (ICMU), 2023, 1–7.
Abstract: This article explores meaningful insights users might be able to obtain at the intersection of wearable technology and user-generated data. While wearables have become ubiquitous in monitoring health and well-being, the utility of the data they collect remains limited for end-users. Our TypeAware case study delves into users’ challenges in interpreting and deriving actionable insights from their wearable data. The TypeAware application aims to enhance user understanding of digital well-being and sleep quality data. Our results indicate that, despite engagement, participants encountered difficulties generating actionable insights from their data. Leveraging the capabilities of large language models, our results demonstrate the potential for automating insight generation: transforming raw data into meaningful, user-friendly understandings. Ulti-mately, this work calls for a shift in wearable technology design, advocating for more user-centric approaches that empower individuals to unlock the full potential of their wearable data for improved well-being.Close abstract M. Oanh Hoang, K. Andreas Rømer Grøntved, N. van Berkel, M. B. Skov, A. L. Christensen, T. Merritt, "Drone Swarms to Support Search and Rescue Operations: Opportunities and Challenges", In book: Cultural Robotics: Social Robots and Their Emergent Cultural Ecologies, B. J. Dunstan, J. T. K. V. Koh, D. Turnbull Tillman, S. A. Brown (Eds.), 2023, 163–176.
Abstract: Emergency services organizations are committed to the challenging task of saving people in distress and minimizing harm across a wide range of events, including accidents, natural disasters, and search and rescue. The teams responsible for these operations use advanced equipment to support their missions. Given the risks and the time pressure of these missions, however, adopting new technologies requires careful testing and preparation. Drones have become a valuable technology in recent years for emergency services teams employed to locate people across vast and difficult to traverse terrains. These unmanned aerial vehicles are faster and cheaper to deploy than traditional crewed aircraft. While an individual drone can be helpful to personnel by quickly offering a bird’s eye view, future scenarios may allow multiple drones working together as a swarm to reduce the time required to locate a person. Given these potentially high payoffs, we explored the challenges and opportunities of drone swarms in search and rescue operations. We conducted interviews as well as initial user studies with relevant stakeholders to understand the challenges and opportunities for drone swarms in the context of search and rescue. Through this, we gained insights to inform the development of prototypes for drone swarm control interfaces, including both technical and human interaction concerns. While drone swarms can likely benefit search and rescue operations, the significant shift from single drones to swarms may necessitate reimagining how rescue missions are conducted. We distill our findings into five key research challenges: visualization, situational awareness, technical issues, team culture, and public perception. We discuss initial steps to investigate these further.Close abstract A. Kathrine Petersen Bach, T. Munch Nørgaard, J. Christian Brok, N. van Berkel, "“If I Had All the Time in the World”: Ophthalmologists’ Perceptions of Anchoring Bias Mitigation in Clinical AI Support", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’23), 2023, 1–14.
Abstract: Clinical needs and technological advances have resulted in increased use of Artificial Intelligence (AI) in clinical decision support. However, such support can introduce new and amplify existing cognitive biases. Through contextual inquiry and interviews, we set out to understand the use of an existing AI support system by ophthalmologists. We identified concerns regarding anchoring bias and a misunderstanding of the AI’s capabilities. Following, we evaluated clinicians’ perceptions of three bias mitigation strategies as integrated into a mockup of their existing decision support system. While clinicians recognised the danger of anchoring bias, we identified a concern around the negative effect of bias mitigation on procedure time. Our participants were divided in their expectations of any positive impact on diagnostic accuracy, stemming from deviating levels of trust and reliance on the decision support. Our results provide insights into integrating bias mitigation in the clinical domain amidst a growing dependency on AI support systems.Close abstract N. van Berkel, M. Bellio, M. B. Skov, A. Blandford, "Measurements, Algorithms, and Presentations of Reality: Framing Interactions with AI-Enabled Decision Support Systems", ACM Transactions on Computer-Human Interaction, vol. 30, no. 2, 2023, 1–33.
Abstract: Bringing AI technology into clinical practice has proved challenging for system designers and medical professionals alike. The academic literature has, for example, highlighted the dangers of black-box decision-making and biased datasets. Further, end-users’ ability to validate a system’s performance often disappears following the introduction of AI decision-making. We present the MAP model to understand and describe the three stages through which medical observations are interpreted and handled by AI systems. These stages are Measurement, in which information is gathered and converted into data points that can be stored and processed; Algorithm, in which computational processes transform the collected data; and Presentation, where information is returned to the user for interpretation. For each stage, we highlight possible challenges that need to be overcome to develop Human-Centred AI systems. We illuminate our MAP model through complementary case studies on colonoscopy practice and dementia diagnosis, providing examples of the challenges encountered in real-world settings. By defining the process of Human-AI interaction across these three stages, we untangle some of the inherent complexities in designing AI technology for clinical decision-making, and aim to overcome misalignment between medical end-users and AI researchers and developers.Close abstract N. van Berkel, Z. Sarsenbayeva and J. Goncalves, "The Methodology of Studying Fairness Perceptions in Artificial Intelligence: Contrasting CHI and FAccT", International Journal of Human-Computer Studies, vol. 170, 2023, 102954.
Abstract: The topic of algorithmic fairness is of increasing importance to the Human-Computer Interaction research community following accumulating concerns regarding the use and deployment of Artificial Intelligence-based systems. How we conduct research on algorithmic fairness directly influences our inferences and conclusions regarding algorithmic fairness. To better understand the methodological decisions of studies focused on people’s perceptions of algorithmic fairness, we systematic analysed relevant papers from the CHI and FAccT conferences. We identified 200 relevant papers published between 1993 and 2022 and assessed their study design, participant sample, and geographical location of participants and authors. Our results highlight that studies are predominantly cross-sectional, cover a wide range of participant roles, and that both authors and participants are primarily from the United States. Based on these findings, we reflect on the potential pitfalls and shortcomings in how the community studies algorithmic fairness.Close abstract S. O’Sullivan, N. van Berkel, V. Kostakos, L. Schmaal, S. D’Alfonso, L. Valentine, S. Bendall, B. Nelson, J. Gleeson, M. Alvarez-Jimenez, "Understanding What Drives Long-term Engagement in Digital Mental Health Interventions: Secondary Causal Analysis of the Relationship Between Social Networking and Therapy Engagement", JMIR Mental Health, 2023, e44812.
Abstract: Background: Low engagement rates with digital mental health interventions are a major challenge in the field. Multicomponent digital interventions aim to improve engagement by adding components such as social networks. Although social networks may be engaging, they may not be sufficient to improve clinical outcomes or lead users to engage with key therapeutic components. Therefore, we need to understand what components drive engagement with digital mental health interventions overall, and what drives engagement with key therapeutic components. Objective: Horyzons was an 18-month digital mental health intervention for young people recovering from first-episode psychosis, incorporating therapeutic content and a private social network. However, it is unclear whether use of the social network leads to subsequent use of therapeutic content, or vice versa. This study aimed to determine the causal relationship between the social networking and therapeutic components of Horyzons. Methods: Participants comprised 82 young people (16-27 years) recovering from first-episode psychosis. Multiple convergent cross mapping was used to test causality, as a secondary analysis of the Horyzons intervention. Multiple convergent cross mapping tested the direction of the relationship between each pair of social and therapeutic system usage variables on Horyzons, using longitudinal usage data. Results: Results indicated that social networking aspects of Horyzons were most engaging. Posting on the social network drove engagement with all therapeutic components (r = 0.06-0.36). Reacting to social network posts drove engagement with all therapeutic components (r = 0.39-0.65). Commenting on social network posts drove engagement with most therapeutic components (r = 0.11-0.18). Liking social network posts drove engagement with most therapeutic components (r = 0.09-0.17). However, starting a therapy pathway led to commenting on social network posts (r = 0.05) and liking social network posts (r = 0.06), and completing a therapy action led to commenting on social network posts (r = 0.14) and liking social network posts (r = 0.15). Conclusions: The online social network was a key driver of long-term engagement with the Horyzons intervention and fostered engagement with key therapeutic components and ingredients of the intervention. Online social networks can be further leveraged to engage young people with therapeutic content to ensure treatment effects are maintained and to create virtuous cycles between all intervention components to maintain engagement.Close abstract Z. Sarsenbayeva, N. van Berkel, D. Hettiachchi, B. Tag, E. Velloso, J. Goncalves, V. Kostakos, "Mapping 20 Years of Accessibility Research in HCI: A Co-word Analysis", International Journal of Human-Computer Studies, vol. 175, 2023, 103018.
Abstract: We employ hierarchical clustering, strategic diagrams, and network core-periphery analysis to assess and visualise the intellectual progress of accessibility research within HCI in the past two decades. The study quantifies and explains the development of accessibility research and its thematic evolution based on 1,535 papers published at TACCESS, ASSETS, IJHCS, and CHI and their respective 3470 author-assigned keywords. The novelty of this work is based on employing a quantitative methodological approach to provide an overview of accessibility research progress and insights into its driving and trending themes through the period 2001–2021. In addition, we identify declining, emerging, and core backbone themes of accessibility research. Finally, we discuss the opportunities for research that arise from our findings. These contributions provide a roadmap for researchers working on accessibility.Close abstract N. van Berkel, K. Hornbæk, "Implications of Human-Computer Interaction Research", ACM Interactions, vol. 30, no. 4, 2023, 50–55.
Abstract: The field of human-computer interaction covers a broad set of methods, viewpoints, and application areas. While the real-world impact of our research is typically impossible to predict, HCI researchers generally seek for their work to have implications that go beyond an individual paper. For example, a study on the use of datalogging tools by hospital staff can provide design implications for patient information systems. A paper on privacy on the Web might provide policy implications on the regulation of tracking technologies. Finally, an analysis of methodological flaws in published papers can provide new analysis guidelines with implications for applying research methods. As these examples highlight, implications from a single study may span a variety of levels and types of potential impact. Here, we present seven types of implications commonly encountered in HCI research and outline the specific aspects that reinforce their impact.Close abstract E. Kuosmanen, E. Huusko, N. van Berkel, F. Nunes, J. Vega, J. Goncalves, M. Khamis, A. Esteves, D. Ferreira, S. Hosio, "Exploring crowdsourced self-care techniques: A study on Parkinson’s disease", International Journal of Human-Computer Studies, 2023, 103062.
Abstract: Living with Parkinson’s Disease introduces a range of significant challenges into one’s daily life. While medical interventions exist to overcome some of these challenges, patient self-care techniques often form an essential complement to the treatments recommended by medical doctors. Knowledge on these self-care techniques often originates from those living with Parkinson’s themselves or their close caregivers, as they have the knowledge and experience required to assess self-care techniques. This so-called `patient knowledge’ is usually exchanged in peer meetings or discussion forums. Although vital to the Parkinson’s Disease community, this information is often difficult to access due to its unstructured format and the difficulty of navigating through online forums. We present an online tool that allows for contributing, assessing, and finally discovering Parkinson’s Disease self-care techniques. The custom discovery tool was populated with self-care knowledge by over 300 people with Parkinson’s and dozens of their carers, spanning areas such as daily well-being and using assistive equipment. Then, we invited patients to explore the discover features in a smaller scale trial. While well-received, our deployment highlighted several challenges that we further discuss in this paper. Overall, our study contributes to crowdsourced digital health solutions and provides both design and research implications to this challenging domain with a vulnerable user group.Close abstract N. van Berkel, S. Shalawadi, M. R. Evans, A. Visuri, S. Hosio, "A Longitudinal Analysis of Real-World Self-Report Data", in Proceedings of the 19th IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT’23), 2023, 611–632.
Abstract: While self-report studies are common in Human-Computer Interaction research, few evaluations have assessed their long term use. We present a longitudinal analysis of a web-based workplace application that collects well-being assessments and offers suggestions to improve individual, team, and organisational performance. Our dataset covers 219 users. We assess their first year of application use, focusing on their usage patterns, well-being evaluations, and behaviour towards notifications. Our results highlight that the drop-off in use was the steepest in the first week (-24.2%). However, substantial breaks in usage were common and did not necessarily result in dropout. We found that latency periods of eight days or more predicted a stronger intention to drop out than stay engaged and that reminder notifications did not result in more completed self-reports but significantly prolonged the usage period. Our work strengthens findings related to high drop out rates, but also provides counter-evidence by showing that despite individuals appearing to drop-off in short-term studies, individuals can and do return to self-report applications after extensive breaks. We contribute an analysis of usage behaviour drivers in the area of technology-enabled well-being measurement, responding to the call for longer-term research to extend the growing literature on self-report studies.Close abstract Z. Sarsenbayeva, C. Fleming, B. Tag, A. Withana, N. van Berkel, A. McEwan, "A Review on Mood Assessment using Smartphones", in Proceedings of the 19th IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT’23), 2023, 385–413.
Abstract: Due to their abundance of sensors, today’s smartphones can act as a scientific tool to collect contextual information on users’ emotional, social, and physical behaviour. With the continuously growing amount of data that can be unobtrusively extracted from smartphones, mood-tracking and inference methods have become more feasible. However, this does raise critical implications for end-users, including accessibility and privacy. Following a structured selection process, we reviewed 32 papers from the ACM Digital Library on mood inference and tracking using smartphones. We conducted an in-depth analysis of used sensors, platform and accessibility, study designs, privacy, self-reporting methods, and accuracy. Based on our analysis, we provide a detailed discussion of the opportunities for research and practice that arise from our findings and outline recommendations for future research within the area of smartphone-based mood tracking and inference.Close abstract N. van Berkel, "Making AI Work: Designing and Evaluating AI Systems in Healthcare", In book: AI in Clinical Medicine: A Practical Guide for Healthcare Professionals, M. F. Byrne, N. Parsa, A. Greenhill, D. Chahal, O. Ahmad, U. Bagci (Eds.), 2023, 448–458.
J. Wester, M. Lee and N. van Berkel, "Moral Transparency as a Mitigator of Moral Bias in Conversational User Interfaces", in Proceedings of the 5th International Conference on Conversational User Interfaces (CUI’23), 2023, 1–6.
Abstract: From straightforward interactions to full-fledged open-ended dialogues, Conversational User Interfaces (CUIs) are designed to support end-user goals and follow their requests. As CUIs become more capable, investigating how to restrict or limit their ability to carry out user requests becomes increasingly critical. Currently, such intentionally constrained user interactions are accompanied by a generic explanation (e.g., “I’m sorry, but as an AI language model, I cannot say…”). We describe the role of moral bias in such user restrictions as a potential source of conflict between CUI users’ autonomy and system characterisation as generated by CUI designers. Just as the users of CUIs have diverging moral viewpoints, so do CUI designers—which either intentionally or unintentionally affects how CUIs communicate. Mitigating user moral biases and making the moral viewpoints of CUI designers apparent is a critical path forward in CUI design. We describe how moral transparency in CUIs can support this goal, as exemplified through intelligent disobedience. Finally, we discuss the risks and rewards of moral transparency in CUIs and outline research opportunities to inform the design of future CUIs.Close abstract B. Tag, N. van Berkel, S. Verma, B. Zi Hao Zhao, S. Berkovsky, D. Kaafar, V. Kostakos, O. Ohrimenko, "DDoD: Dual Denial of Decision Attacks on Human-AI Teams", IEEE Pervasive Computing, vol. 22, no. 01, 2023, 77–84.
Abstract: Artificial Intelligence (AI) systems have been increasingly used to make decision-making processes faster, more accurate, and more efficient. However, such systems are also at constant risk of being attacked. While the majority of attacks targeting AI-based applications aim to manipulate classifiers or training data and alter the output of an AI model, recently proposed Sponge Attacks against AI models aim to impede the classifier’s execution by consuming substantial resources. In this work, we propose Dual Denial of Decision (DDoD) attacks against collaborative Human-AI teams. We discuss how such attacks aim to deplete both computational and human resources, and significantly impair decision-making capabilities. We describe DDoD on human and computational resources and present potential risk scenarios in a series of exemplary domains.Close abstract V. Paananen, A. Visuri, N. van Berkel, S. Hosio, "Eliciting Empathy towards Urban Accessibility Issues", in CHItaly 2023: 15th Biannual Conference of the Italian SIGCHI Chapter (CHItaly’23), 2023, 1–13.
Abstract: Empathy is an integral part of what it means to be human. Empathy refers to the ability to sense other people’s emotions, coupled with the ability to imagine what they might be thinking and feeling. Architectural and urban design have identified empathy as a crucial factor in the design process and especially in user-centered participatory methods. Although empathy has been recognized as important for relating to other people’s issues, current research has not explored how urban accessibility issues elicit empathy. We conducted a between-subjects online study where 202 participants observed five scenarios on different accessibility issues. Our results show that empathic traits and previous experience are significant factors in empathizing with accessibility issues. Additionally, storytelling and photos can influence perceptions of accessibility issues. The study highlights the importance of empathic traits and personal experience in understanding and addressing accessibility issues, as well as the potential of storytelling and photos in shaping perceptions of accessibility issues and evoking empathy. Our contribution demonstrates the advantages of incorporating narrative multimedia into design processes for improved urban accessibility.Close abstract J. Moilanen, N. van Berkel, A. Visuri, U. Gadiraju, W. van der Maden, S. Hosio, "Supporting Mental Health Self-Care Discovery Through a Chatbot", Frontiers in Digital Health, 2023, 1–10.
Abstract: Good mental health is imperative for one’s wellbeing. While clinical mental disorder treatments exist, self-care is an essential aspect of mental health. This paper explores the use and perceived trust of conversational agents, chatbots, in the context of crowdsourced self-care through a between-subjects study (N = 80). One group used a standalone system with a conventional web interface to discover self-care methods. The other group used the same system wrapped in a chatbot interface, facilitating utterances and turn-taking between the user and a chatbot. We identify the security and integrity of the systems as critical factors that affect users’ trust. The chatbot interface scored lower on both these factors, and we contemplate the potential underlying reasons for this. We complement the quantitative data with qualitative analysis and synthesize our findings to identify suggestions for using chatbots in mental health contexts.Close abstract W. Jiang, D. Yu, C. Wang, Z. Sarsenbayeva, N. van Berkel, J. Goncalves, V. Kostakos, "Near-Infrared Imaging for Information Embedding and Extraction with Layered Structures", ACM Transactions on Graphics, vol. 42, no. 1, 2023, 1–26.
D. Russo, P. H. P. Hanel, S. Altnickel, N. van Berkel, "Satisfaction and Performance of Software Developers during Enforced Work from Home in the COVID-19 Pandemic", Empirical Software Engineering, vol. 28, no. 2, 2023, 1–53.
Abstract: Following the onset of the COVID-19 pandemic and subsequent lockdowns, the daily lives of software engineers were heavily disrupted as they were abruptly forced to work remotely from home. To better understand and contrast typical working days in this new reality with work in pre-pandemic times, we conducted one exploratory (N = 192) and one confirmatory study (N = 290) with software engineers recruited remotely. Specifically, we build on self-determination theory to evaluate whether and how specific activities are associated with software engineers’ satisfaction and productivity. To explore the subject domain, we first ran a two-wave longitudinal study. We found that the time software engineers spent on specific activities (e.g., coding, bugfixing, helping others) while working from home was similar to pre-pandemic times. Also, the amount of time developers spent on each activity was unrelated to their general well-being, perceived productivity, and other variables such as basic needs. Our confirmatory study found that activity-specific variables (e.g., how much autonomy software engineers had during coding) do predict activity satisfaction and productivity but not by activity-independent variables such as general resilience or a good work-life balance. Interestingly, we found that satisfaction and autonomy were significantly higher when software engineers were helping others and lower when they were bugfixing. Finally, we discuss implications for software engineers, management, and researchers. In particular, active company policies to support developers’ need for autonomy, relatedness, and competence appear particularly effective in a WFH context.Close abstract T. Dowrick, G. Xiao, D. Nikitichev, E. Dursun, N. van Berkel, M. Allam, B. Koo, J. Ramalhinho, S. Thompson, K. Gurusamy, A. Blandford, D. Stoyanov, B. R. Davidson, M. J. Clarkson, "Evaluation of a calibration rig for stereo laparoscopes", Medical Physics, vol. 50, no. 5, 2023, 1–9.
Abstract: Background: Accurate camera and hand-eye calibration are essential to ensure high quality results in image guided surgery applications. The process must also be able to be undertaken by a non-expert user in a surgical setting. Purpose: This work seeks to identify a suitable method for tracked stereo laparoscope calibration within theatre. Methods: A custom calibration rig, to enable rapid calibration in a surgical setting, was designed. The rig was compared against freehand calibration. Stereo reprojection, stereo reconstruction, tracked stereo reprojection and tracked stereo reconstruction error metrics were used to evaluate calibration quality. Results: Use of the calibration rig reduced mean errors: reprojection (1.47mm [SD 0.13] vs 3.14mm [SD 2.11], p-value 1e-8), reconstruction (1.37px [SD 0.10] vs 10.10px [SD 4.54], p-value 6e-7) and tracked reconstruction (1.38mm [SD 0.10] vs 12.64mm [SD 4.34], p-value 1e-6) compared with freehand calibration. The use of a ChArUco pattern yielded slightly lower reprojection errors, while a dot grid produced lower reconstruction errors and was more robust under strong global illumination. Conclusion: The use of the calibration rig results in a statistically significant decrease in calibration error metrics, versus freehand calibration, and represents the preferred approach for use in the operating theatre.Close abstract 2022
N. van Berkel, J. Opie, O. F. Ahmad, L. Lovat, D. Stoyanov, A. Blandford, "Initial Responses to False Positives in AI-supported Continuous Interactions – A Colonoscopy Case Study", ACM Transactions on Interactive Intelligent Systems, 2022, 1–18.
Abstract: The use of Artificial Intelligence in clinical support systems is increasing. In this paper we focus on AI support for continuous interaction scenarios. A thorough understanding of end-user behaviour during these continuous Human-AI interactions, in which user input is sustained over time and during which AI suggestions can appear at any time, is still missing. We present a controlled lab-study involving 21 endoscopists and an AI colonoscopy support system. Using a custom-developed application and an off-the-shelf videogame controller, we record participants’ navigation behaviour and clinical assessment across 14 endoscopic videos. Each video is manually annotated to mimic an AI recommendation, being either true positive or false positive in nature. We find that time between AI recommendation and clinical assessment is significantly longer for incorrect assessments. Further, the type of medical content displayed significantly affects decision time. Finally, we discover that the participant’s clinical role plays a large part in the perception of clinical AI support systems. Our study presents a realistic assessment of the effects of imperfect and continuous AI support in a clinical scenario.Close abstract Z. Sarsenbayeva, N. van Berkel, E. Velloso, J. Goncalves, "Methodological Standards in Accessibility Research on Motor Impairments: A Survey", ACM Computing Surveys, vol. 5, no. 7, 2022, 1–35.
Abstract: The design and evaluation of accessibility technology is a core component of the Computer Science landscape, aiming to ensure that digital innovations are accessible to all. One of the most prominent and long-lasting areas of accessibility research focuses on motor impairments, deficiencies that affect the ability to move, manipulate objects, and interact with the physical world. In this survey paper, we present an extensive overview of the last two decades of research into accessibility for people with motor impairments. Following a structured selection process, we analysed the study details as reported in 177 relevant papers. Based on this analysis, we critically assess user representation, measurement instruments, and existing barriers that exist in accessibility research. Finally, we discuss future directions for accessibility research within the Computer Science domain.Close abstract R. M. Jacobsen, N. van Berkel, M. B. Skov, S. S. Johansen, J. Kjeldskov, "Do You See What I Hear? — Peripheral Absolute and Relational Visualisation Techniques for Sound Zones", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’22), 2022, 1–13.
Abstract: Sound zone technology allows multiple simultaneous sound experiences for multiple people in the same room without interference. However, given the inherent invisible and intangible nature of sound zones, it is unclear how to communicate the position and size of sound zones to users. This paper compares two visualisation techniques; absolute visualisation, relational visualisation, as well as a baseline condition without visualisations. In a within-subject experiment (N=33), we evaluated these techniques for effectiveness and efficiency across four representative tasks. Our findings show that the absolute and relational visualisation techniques increase effectiveness in multi-user tasks but not in single-user tasks. The efficiency for all tasks was improved using visualisations. We discuss the potential of visualisations for sound zones and highlight future research opportunities for sound zone interaction.Close abstract T. Dingler, B. Tag, D. A. Eccles, N. van Berkel, V. Kostakos, "Method for Appropriating the Brief Implicit Association Test to Elicit Biases in Users", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’22), 2022, 1–16.
Abstract: The Implicit Association Test (IAT) has been widely used to assess people’s associations of target concepts with qualitative attributes, such as the likelihood of being hired or convicted depending on race, gender, or age. The condensed version–the Brief IAT–elicits implicit biases by measuring the reaction time to concept classifications. In this paper, we introduce and evaluate a new method to appropriate the BIAT using crowdsourcing to measure people’s leanings on polarizing topics. We present a web-based tool to test participants’ bias on custom themes, where self-assessments often fail. We validated our approach with 14 domain experts and assessed the fit of crowdsourced test construction. Our method allows researchers of different domains to create and validate bias tests that can be geographically tailored and updated over time. We discuss how our method can be applied to surface implicit user biases and run studies where cognitive biases may impede reliable results.Close abstract B. Tag, N. van Berkel, A. W. Vargo, Z. Sarsenbayeva, T. Colasante, G. Wadley, S. Webber, W. Smith, P. Koval, T. Hollenstein, J. Goncalves, V. Kostakos, "Impact of the Global Pandemic upon Young People’s Use of Technology for Emotion Regulation", Computers in Human Behavior Reports, 2022, 1–16.
Abstract: Technology plays an increasingly prominent role in emotional lives. Researchers have begun to study how people use devices to cope with and shape emotions: a phenomenon that has been called Digital Emotion Regulation. We report a study of the impact of the COVID-19 pandemic upon young people’s digital habits and emotion regulation behaviors. We conducted a two-wave longitudinal survey, collecting data from 154 university students both before and during the COVID-19 pandemic. During the pandemic, participants were subject to increased emotional distress as well as restrictions on movement and social interaction. We present evidence that participants’ emotion regulation strategies changed and became more homogeneous during the pandemic, with participants resorting to digital tools when offline strategies were less available, while also becoming more emotionally dependent upon their devices. This study underscores the growing significance of the digital for contemporary emotional experience, and contributes to understanding the potential role for technology in supporting well-being during high-impact events.Close abstract E. Kuosmanen, A. Visuri, S. Kheirinejad, N. van Berkel, H. Koskimäki, D. Ferreira, S. Hosio, "How Does Sleep Tracking Influence Your Life? Experiences from a Longitudinal Field Study with a Wearable Ring", Proceedings of the ACM on Human-Computer Interaction – MobileHCI, vol. 6, no. MHCI, 2022, 1–19.
Abstract: A new generation of wearable devices now enable end-users to keep track of their sleep patterns. This paper reports on a longitudinal study of 82 participants who used a state-of-the-art sleep tracking ring for an average of 65 days. We conducted interviews and questionnaires to understand changes to their lifestyle, their perceptions of the tracked information and sleep, and the overall experience of using an unobtrusive sleep tracking device. Our results indicate that such a device is suitable for long-term sleep tracking and helpful in identifying detrimental lifestyle elements that hinder sleep quality. However, tracking one’s sleep can also introduce stress or physical discomfort, potentially leading to adverse outcomes. We discuss these findings in light of related work and highlight the near-term research directions that the rapid commoditisation of sleep tracking technology enables.Close abstract N. van Berkel, T. Merritt, A. Bruun, M. B. Skov, "Tangible Self-Report Devices: Accuracy and Resolution of Participant Input", in Proceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI’22), 2022, 1–14.
Abstract: Tangible input has been explored as a means for participants to self-report experiences while minimising disruption and allowing for discrete data collection. However, the accuracy of these tangible devices has not been studied systematically. We compared six input techniques, including slider, slider with resistance, capacitive touch slider, squeeze, rotary knob, and joystick, to understand their accuracy and resolution profile. Each of these wireless devices was designed in a similar form factor and intended to be operated discretely with one hand. We assessed input accuracy and participant perceptions across devices through a controlled lab study (N=20), highlighting diverging limits to the accuracy of the input technique and possible explanations for the differences in resolution. Our results indicate that participant accuracy was highest using a slider, and lowest using a squeeze-based input. We discuss the suitability and challenges of discreet tangible self-report techniques, and highlight open research questions for future work.Close abstract S. S. Johansen, N. van Berkel and J. Fritsch, "Characterising Soundscape Research in Human-Computer Interaction", in Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems (DIS’22), 2022, 1394–1417.
Abstract: ‘Soundscapes’ are an increasingly active topic in Human-Computer Interaction (HCI) and interaction design. From mapping acoustic environments through sound recordings to designing compositions as interventions, soundscapes appear as a recurring theme across a wide body of HCI research. Based on this growing interest, now is the time to explore the types of studies in which soundscapes provide a valuable lens to HCI research. In this paper, we review papers from conferences sponsored or co-sponsored by the ACM Special Interest Group on Computer-Human Interaction in which the term ‘soundscape’ occurs. We analyse a total of 235 papers to understand the role of soundscapes as a research focus and identify untapped opportunities for soundscape research within HCI. We identify two common soundscape conceptualisations: (1) Acoustic environments and (2) Compositions, and describe what characterises studies into each concept and the hybrid forms that also occur. On the basis of this, we carve out a foundation for future soundscape research in HCI as a methodological anchor to form a common ground and support this growing research interest. Finally, we offer five recommendations for further research into soundscapes within HCI.Close abstract S. Wijenayake, N. van Berkel, V. Kostakos, J. Goncalves, "Quantifying Determinants of Social Conformity in an Online Debating Website", International Journal of Human-Computer Studies, vol. 158, 2022, 102743.
N. van Berkel, S. D’Alfonso, R. Kurnia Susanto, D. Ferreira, V. Kostakos, "AWARE-Light: a smartphone tool for experience sampling and digital phenotyping", Personal and Ubiquitous Computing, 2022, 1–11.
Abstract: Due to their widespread adoption, frequent use, and diverse sensor capabilities, smartphones have become a powerful tool for academic studies focused on sampling human behaviour. While packing many technological advances, the need for researchers to develop their own software packages in order to run smartphone-based studies has resulted in a clear barrier to entry for researchers without the financial means, time, or technical knowledge required to overcome this technical barrier. We present AWARE-Light, a new smartphone application for data collection from study participants, which is accompanied by a website that provides any researcher the possibility to easily configure their own study. To highlight the possibilities of our tool, we present a research scenario on digital phenotyping for mental health. Furthermore, we describe the methodological configuration possibilities offered by our tool, and complement the technical configuration possibilities with recommendations from the existing literature.Close abstract A. Oncevay, D. Ataman, N. van Berkel, B. Haddow, A. Birch, J. Bjerva, "Quantifying Synthesis and Fusion and their Impact on Machine Translation", in Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL’22), 2022, 1–14.
Abstract: Theoretical work in morphological typology offers the possibility of measuring morphological diversity on a continuous scale. However, literature in NLP typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the theoretical rigidity of such claims, by quantifying the morphological typology at the word and segment level. We consider Payne (2017)’s approach to classify morphology using two indices: synthesis (from 1 for analytic to 3 or more for polysynthetic) and fusion (from 0 for agglutinative to 1 for fusional). For computing synthesis, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study. Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish, and verbs in English-Spanish) and segment level (previous language pairs plus English-German in both directions). We complement the word-level analysis with human evaluation, and overall, we observe a consistent impact of both indexes on machine translation quality.Close abstract M. Pakanen, P. Alavesa, N. van Berkel, T. Koskela, T. Ojala, "“Nice to see you virtually”: Thoughtful Design and Evaluation of Virtual Avatar of the Other User in AR and VR Based Telexistence Systems", Entertainment Computing, vol. 40, 2022, 100457.
Abstract: This paper presents two studies investigating how physically remote telexistence users wish to see other users visualized as virtual avatars in a) augmented reality, and b) immersive virtual reality while conducting a collaborative task. To answer this research question, a telexistence system was designed and implemented with simple avatar designs. After that, visual examples of alternative avatar representations for both use cases were designed by thoughtfully altering the visual parameters of 36 virtual avatar examples. The avatar designs were first evaluated in a user study with 16 participants in conjunction with using an implemented telexistence system. As a follow-up an online survey with 43 respondents was used to record their preferences regarding virtual avatar appearance. The results suggest that users prefer the other user to be represented in a photorealistic full-body human avatar in both augmented reality and virtual reality due to its humanlike representation and affordances for interaction. In augmented reality, the choice for a hologram full body avatar was also popular due to its see-through appearance, which prevents a mix-up with a real person in the physical space.Close abstract J. Dexe, U. Franke, K. Söderlund, N. van Berkel, R. Hagensby Jensen, N. Lepinkäinen, J. Vaiste, "Explaining automated decision-making—A multinational study of the GDPR right to meaningful information", The Geneva Papers on Risk and Insurance – Issues and Practice, vol. 47, no. 3, 2022, 669–697.
Abstract: The GDPR establishes a right for individuals to get access to information about automated decision-making based on their personal data. However, the application of this right comes with caveats. This paper investigates how European insurance companies have navigated these obstacles. By recruiting volunteering insurance customers, requests for in- formation about how insurance premiums are set were sent to 26 insurance companies in Denmark, Finland, The Netherlands, Poland and Sweden. Findings illustrate the practice of responding to GDPR information requests and the paper identifies possible explanations for shortcomings and omissions in the responses. The paper also adds to existing research by showing how the wordings in the different language versions of the GDPR could lead to different interpretations. Finally, the paper discusses what can reasonably be expected from explanations in consumer oriented information.Close abstract A. Alorwu, A. Visuri, N. van Berkel, S. Hosio, "(Re)Using Crowdsourced Health Data: Perceptions of Data Contributors", IEEE Software, vol. 39, no. 1, 2022, 36–42.
Abstract: Open data is often contributed by various governments and public sector actors. An increasingly popular way to collect large bespoke datasets is crowdsourcing. In this work we explore crowdsourced open data as an enabler of future software solutions. We recruited participants from an online paid crowdsourcing platform to provide open mental health related data that was used to create an interactive data-driven decision support system for self-care. We then invited a sub-sample of 80 participants back to explore the tool that was created using their own data and to provide a rich account of perceptions on issues around such health data reuse in software. Our results unfold a range of different perceived threats and opportunities in using crowdsourced data to enable software solutions, and our work contributes a topical case study and discussion toward the use of crowdsourced data in an open fashion.Close abstract 2021
N. van Berkel, J. Goncalves, D. Russo, S. Hosio, M. B. Skov, "Effect of Information Presentation on Fairness Perceptions of Machine Learning Predictors", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), 2021, 1–13.
Abstract: The uptake of artificial intelligence-based applications raises concerns about the fairness and transparency of AI behaviour. Consequently, the Computer Science community calls for the involvement of the general public in the design and evaluation of AI systems. Assessing the fairness of individual predictors is an essential step in the development of equitable algorithms. In this study, we evaluate the effect of two common visualisation techniques (text-based and scatterplot) and the display of the outcome information (i.e., ground-truth) on the perceived fairness of predictors. Our results from an online crowdsourcing study (N = 80) show that the chosen visualisation technique significantly alters people’s fairness perception and that the presented scenario, as well as the participant’s gender and past education, influence perceived fairness. Based on these results we draw recommendations for future work that seeks to involve non-experts in AI fairness evaluations.Close abstract W. Jiang, Z. Sarsenbayeva, N. van Berkel, C. Wang, D. Yu, J. Wei, J. Goncalves, V. Kostakos, "User Trust in Assisted Decision-Making Using Miniaturized Near-Infrared Spectroscopy", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), 2021, 1–16.
Abstract: We investigate the use of a miniaturized Near-Infrared Spectroscopy (NIRS) device in an assisted decision-making task. We consider the real-world scenario of determining whether food contains gluten, and we investigate how end-users interact with our NIRS detection device to ultimately make this judgment. In particular, we explore the effects of different nutrition labels and representations of confidence on participants’ perception and trust. Our results show that participants tend to be conservative in their judgment and are willing to trust the device in the absence of understandable label information. We further identify strategies to increase user trust in the system. Our work contributes to the growing body of knowledge on how NIRS can be mass-appropriated for everyday sensing tasks, and how to enhance the trustworthiness of assisted decision-making systems.Close abstract A. Alorwu, S. Kheirinejad, N. van Berkel, M. Kinnula, D. Ferreira, A. Visuri, S. Hosio, "Assessing MyData Scenarios: Ethics, Concerns, and the Promise", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), 2021, 1–11.
Abstract: Public controversies around the unethical use of personal data are increasing, spotlighting data ethics as an increasingly important field of study. MyData is a related emerging vision that emphasizes individuals’ control of their personal data. In this paper, we investigated people’s perceptions of various data management scenarios by measuring the perceived ethicality and level of personal felt concern regarding the scenarios. We deployed the set of 96 unique scenarios to a paid crowdsourcing platform for assessment and invited a representative sample of the participants to a second-stage questionnaire about the MyData vision as well as its potential in the field of healthcare. Our results provide a timely investigation into how topical data-related practices affect the perceived ethicality and the felt concern. The questionnaire analysis reveals great potential in the MyData vision. Through the combined quantitative and qualitative results, we contribute to the field of data ethics.Close abstract N. van Berkel, M. B. Skov and J. Kjeldskov, "Human-AI Interaction: Intermittent, Continuous, and Proactive", ACM Interactions, vol. 28, no. 6, 2021, 67–71.
Abstract: With the rise in artificial intelligence (AI)—driven interactive systems, both academics and practitioners within human-computer interaction (HCI) have a growing focus on human-AI interaction. This has resulted in, for example, system-design guidelines and reflections on the differences and challenges when designing for AI-driven interaction as opposed to more-traditional applications. We argue that the current work on human-AI interaction is defined primarily by a focus on what we refer to as intermittent interaction scenarios, in which there is a clear line between the human initiator of an interaction and an almost immediate system response. However, user interaction with AI systems does not necessarily follow this rigid interaction pattern. Inspired by Kristina Höök and Yang et al., we define human-AI interaction as the completion of a user’s task with the help of AI support, which may manifest itself in non-intermittent scenarios. By overlooking these other interaction paradigms, we neglect the opportunity to define and support alternative human-AI scenarios. In this article, we present and outline three types of human-AI interaction paradigms, which we refer to as intermittent, continuous, and proactive, highlighting a diverse set of interaction scenarios and pointing to a need for HCI considerations across different types of human-AI interaction. While a wide range of existing AI-powered systems operate continuously in the background of our lives (e.g., step counters, spam filters), these applications do not engage directly with their users. Here, we focus on AI applications that interact directly with their users.Close abstract N. van Berkel, V. Kostakos, "Recommendations for Conducting Longitudinal Experience Sampling Studies", In book: Advances in Longitudinal HCI Research, E. Karapanos, J. Gerken, J. Kjeldskov, M. B. Skov (Eds.), 2021, 59–78.
Abstract: The Experience Sampling Method is used to collect participant self-reports over extended observation periods. These self-reports offer a rich insight into the individual lives of study participants by intermittently asking participants a set of questions. However, the longitudinal and repetitive nature of this sampling approach introduces a variety of concerns regarding the data contributed by participants. A decrease in participant interest and motivation may negatively affect study adherence, as well as potentially affecting the reliability of participant data. In this chapter we reflect on a number of studies that aim to understand better participant performance with Experience Sampling. We discuss the main issues relating to participant data for longitudinal studies, and provide hands-on recommendations for researchers to remedy these concerns in their own studies.Close abstract M. Kjærup, M. B. Skov and N. van Berkel, "E-Scooter Sustainability – A Clash of Needs, Perspectives, and Experiences", in Proceedings of the 18th IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT’21), 2021, 365–383.
Abstract: Electric stand-up scooters (e-scooters) are introduced in several cities worldwide, providing new means for people to travel around the city. While praised for their flexibility, e-scooters are also met with negative sentiments due to fatal accidents and chaotic parking. In this paper, we seek to understand the mobility of shared e-scooters and point to gaps in the user interaction between the digital and physical world. We carried out three data collections, including interviews, in situ observation, analysis of news media coverage. Our findings illustrate integration with alternate modes of transportation in urban context, and how technologies facilitate or hinder (micro-) mobility. We found that users of e-scooters primarily view these devices as an alternative to walking rather than other transportation forms. Additionally, we found that users’ and non-users’ needs, perspectives and experiences of e-scooters clash, in particular with regard to perceptions of sustainability. Based on these findings, we present three relevant perspectives of sustainability, extending the ongoing debate of sustainable HCI research. We contribute with an empirically supported understanding of the perception of mobility and sustainability for e-scooters in a Scandinavian urban context.Close abstract S. Shalawadi, A. Alnayef, N. van Berkel, J. Kjeldskov, F. Echtler, "Rainmaker: A Tangible Work-Companion for the Personal Office Space", in Proceedings of the International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI’21), 2021, 1–13.
Abstract: Routines are an important element of day-to-day work life, allowing people to structure their day around required tasks. Effectively managing these routines is, however, experienced as challenging by many – an issue further amplified by the current work from home lockdown measures. In this paper we present Rainmaker, a tangible device that aims to support people in their working life in the context of their own homes. We evaluate and iterate on our prototype through two qualitative studies, spanning respectively three days (N= 11) and 15 days (N= 2). Our results highlight the perceived advantages of the use of a primarily physical rather than digital tool for work support, allowing users to stay focused on their tasks and reflect on their work achievements. We present lessons for future work in this area and publicly release the software and hardware used in the construction of Rainmaker.Close abstract N. van Berkel, O. F. Ahmad, D. Stoyanov, L. Lovat, A. Blandford, "Designing Visual Markers for Continuous Artificial Intelligence Support: A Colonoscopy Case Study", ACM Transactions on Computing for Healthcare, vol. 2, 2021, 7:1–7:24.
Abstract: Colonoscopy, the visual inspection of the large bowel using an endoscope, offers protection against colorectal cancer by allowing for the detection and removal of pre-cancerous polyps. The literature on polyp detection shows widely varying miss rates among clinicians, with averages ranging around 22–27%. While recent work has considered the use of AI support systems for polyp detection, how to visualise and integrate these systems into clinical practice is an open question. In this work, we explore the design of visual markers as used in an AI support system for colonoscopy. Supported by the gastroenterologists in our team, we designed seven unique visual markers and rendered them on real-life patient video footage. Through an online survey targeting relevant clinical staff (N = 36), we evaluated these designs and obtained initial insights and understanding into the way in which clinical staff envision AI to integrate in their daily work-environment. Our results provide concrete recommendations for the future deployment of AI support systems in continuous, adaptive scenarios.Close abstract N. van Berkel, S. Dennis, M. Zyphur, J. Li, A. Heathcote, V. Kostakos, "Modeling Interaction as a Complex System", Human-Computer Interaction, vol. 36, no. 4, 2021, 1–27.
Abstract: Researchers in Human-Computer Interaction typically rely on experiments to assess the causal effects of experimental conditions on variables of interest. Although this classic approach can be very useful, it offers little help in tackling questions of causality in the kind of data that are increasingly common in HCI – capturing user behavior ‘in the wild.’ To analyze such data, model-based regressions such as cross-lagged panel models or vector autoregressions can be used, but these require parametric assumptions about the structural form of effects among the variables. To overcome some of the limitations associated with experiments and model-based regressions, we adopt and extend ‘empirical dynamic modelling’ methods from ecology that lend themselves to conceptualizing multiple users’ behavior as complex nonlinear dynamical systems. Extending a method known as ‘convergent cross mapping’ or CCM, we show how to make causal inferences that do not rely on experimental manipulations or model-based regressions and, by virtue of being non-parametric, can accommodate data emanating from complex nonlinear dynamical systems. By using this approach for multiple users, which we call ‘multiple convergent cross mapping’ or MCCM, researchers can achieve a better understanding of the interactions between users and technology – by distinguishing causality from correlation – in real-world settings.Close abstract D. Russo, P. H. P. Hanel, S. Altnickel, N. van Berkel, "Predictors of Well-being and Productivity of Software Professionals during the COVID-19 Pandemic – A Longitudinal Study", Empirical Software Engineering, vol. 26, no. 4, 2021, 62.
Abstract: The COVID-19 pandemic has forced governments worldwide to impose movement restrictions on their citizens. Although critical to reducing the virus’ reproduction rate, these restrictions come with far-reaching social and economic consequences. In this paper, we investigate the impact of these restrictions on an individual level among software engineers who were working from home. Although software professionals are accustomed to working with digital tools, but not all of them remotely, in their day-to-day work, the abrupt and enforced work-from-home context has resulted in an unprecedented scenario for the software engineering community. In a two-wave longitudinal study (N = 192), we covered over 50 psychological, social, situational, and physiological factors that have previously been associated with well-being or productivity. Examples include anxiety, distractions, coping strategies, psychological and physical needs, office set-up, stress, and work motivation. This design allowed us to identify the variables that explained unique variance in well-being and productivity. Results include (1) the quality of social contacts predicted positively, and stress predicted an individual’s well-being negatively when controlling for other variables consistently across both waves; (2) boredom and distractions predicted productivity negatively; (3) productivity was less strongly associated with all predictor variables at time two compared to time one, suggesting that software engineers adapted to the lockdown situation over time; and (4) longitudinal analyses did not provide evidence that any predictor variable causal explained variance in well-being and productivity. Overall, we conclude that working from home was per se not a significant challenge for software engineers. Finally, our study can assess the effectiveness of current work-from-home and general well-being and productivity support guidelines and provides tailored insights for software professionals.Close abstract E. Schneiders, E. Papachristos and N. van Berkel, "The Effect of Embodied Anthropomorphism of Personal Assistants on User Perceptions", in Proceedings of the Australian Conference on Human-Computer Interaction (OzCHI’21), 2021, 231–241.
Abstract: We investigate the impact of anthropomorphism on embodied AI through a study of personal assistants (PA). The effects of physical embodiment remain underexplored while the consumer market for PAs shows an increase in the diversity of physical appearances of these products. We designed three fictional personal assistants with varying levels of embodied anthropomorphism. We validated that our prototypes differed significantly in levels of anthropomorphism (N=26). We developed a set of identical videos for each device, demonstrating realistic end-user interaction across six scenarios. Using a between-subject video survey study (N=150), we evaluate the impact of different levels of embodied anthropomorphism on the perception of personal assistants. Our results show that while anthropomorphism did not significantly affect the perception of Overall Goodness, it affected perceptions of Perceived Intelligence, Likeability, and the device’s Pragmatic Qualities. Finally, we discuss the implications of the identified relationships between anthropomorphism and user confidence in embodied AI systems.Close abstract D. Russo, P. H. P. Hanel, S. Altnickel, N. van Berkel, "The Daily Life of Software Engineers during the COVID-19 Pandemic", in Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE SEIP’21), 2021, 364–373.
K. Sharma, K. Mangaroska, N. van Berkel, M. Giannakos, V. Kostakos, "Information Flow and Cognition affect each other: Evidence from Digital Learning", International Journal of Human-Computer Studies, vol. 146, 2021, 102549.
S. Klakegg, K. Opoku Asare, N. van Berkel, A. Visuri, E. Ferreira, S. Hosio, J. Goncalves, H. Huttunen, D. Ferreira, "CARE: Context-awareness for elderly care", Health and Technology, vol. 11, 2021, 211–226.
Abstract: We present CARE, a context-aware tool for nurses in nursing homes. The system utilises a sensors infrastructure to quantify the behaviour and wellbeing (e.g., activity, mood, social and nurse interactions) of elderly residents. The sensor data is offloaded, processed and analysed in the cloud, to generate daily and long-term summaries of residents’ health. These insights are then presented to nurses via an Android tablet application. We aim to create a tool that can assist nurses and increase their awareness to residents’ needs. We deployed CARE in a local nursing home for two months and evaluated the system through a post-hoc exploratory analysis and interviews with the nurses. The results indicate that CARE can reveal essential insights on the wellbeing of elderly residents and improve the care service. In the discussion, we reflect on our understanding and potential impact of future integrated technology in elderly care environments.Close abstract A. Visuri, N. van Berkel, R. Rawassizadeh, J. Goncalves, V. Kostakos, D. Ferreira, "Understanding Usage Style Transformation During Long-Term Smartwatch Use", Personal and Ubiquitous Computing, 2021, 1–15.
Abstract: Despite large investments in smartwatch development, the market growth remains smaller than forecasted. The purpose of smartwatch use remains unclear, indicated by the lack of large-scale adoption. Thus, we aim to better understand the early adoption and everyday smartwatch use. We investigate a diverse usage data of smartwatches logged over a period of up to 14 months from 79 individuals between December 2015 and March 2017, one of the largest wearable datasets collected. First, we identify both explorative and accepted behaviours that users exhibit and further investigate how the individual usage traits and features differ between the two categories. Our analysis offers an insightful perspective on how smartwatch use evolves organically. Our results improve our shared understanding of smartwatch use and users adapting their use of smartwatch over time to match the capabilities of the technology by validating numerous findings from previous literature.Close abstract M. Joosse, M. Lohse, N. van Berkel, A. Sardar, V. Evers, "Making Appearances: How Robots Should Approach People", ACM Transactions on Human-Robot Interaction, vol. 10, no. 1, 2021, 1–24.
Abstract: To prepare for a future in which robots are more commonplace, it is important to know what robot behaviors people find socially normative. Previous work suggests that for robots to be accepted by people, the robot should adhere to the prevalent social norms, such as those related to approaching people. However, we do not expect that socially normative approach behaviors for robots can be translated on a one-on-one basis from people to robots, because currently robots have unique and different features to humans, including (but not limited to) wheels, sounds, and shapes. The two studies presented in this article go beyond the state- of-the-art and focus on socially normative approach behaviors for robots. In the first study, we compared people’s responses to violations of personal space done by robots compared to people. In the second study, we explored what features (sound, size, speed) of a robot approaching people have an effect on acceptance. Findings indicate that people are more lenient toward violations of a social norm by a robot as compared to a person. Also, we found that robots can use their unique features to mitigate the negative effects of norm violations by communicating intent.Close abstract 2020
S. Hosio, N. van Berkel, "COGNET: The Planetary Cognition Delivery Network", in Blue Sky Ideas Track – Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’20), 2020, 1–4.
Abstract: Crowd-powered innovation platforms act to a large degree as silos: they cater the cognitive surplus of a bespoke, selfselected audience to a limited amount of high-value clients in a model where both stakeholders typically have to jump through a series of hoops to enrol to the services. We propose a fundamentally disruptive way for discovery with distributed crowds, by orchestrating already established online audiences for serendipitous crowdsourcing. Two recent developments make our proposition, The Planetary Cognition Delivery Network, compelling right now. First, online properties are struggling due to declining advertising revenues, caused by the proliferation of ad-blockers and a few key Internet giants taking an increasingly larger cut of the available revenue. Second, and perhaps somewhat counter-intuitively to the immediate perception in the Western world, the Internet is just now becoming widely available in many corners of the world, which provides an opportunity for a truly worldwide reach during the next decade or two. COGNET is designed to offer a frictionless participation mechanism for all three key stakeholders: requesters, providers, and contributors. It essentially orchestrates a distributed network of human cognition pools for arbitrary discovery tasks that could benefit from the vast cognitive surplus available through the internet.Close abstract N. van Berkel, M. J. Clarkson, G. Xiao, E. Dursun, M. Allam, B. R. Davidson, A. Blandford, "Dimensions of Ecological Validity for Usability Evaluations in Clinical Settings", Journal of Biomedical Informatics, vol. 110, 2020, 103553.
Abstract: The development, evaluation, and eventual deployment of novel medical devices is a complex process involving various areas of expertise. Although the need for a User Centred Design approach to the development of both hardware and software has long been established, both current regulatory guidelines and widespread evaluation approaches fail to reflect the challenges encountered during day-to-day clinical practice. As such, the results from these evaluations may not provide a realistic account of the problems encountered by users when introduced to clinical practice. In this paper, we present a case study on designing the evaluation of a novel device to support laparoscopic liver surgery. Through a reflective account of the design of our usability evaluation, we identify and describe seven primary dimensions of ecological validity encountered in clinical usability evaluations. These dimensions are: ‘user roles’, ‘environment’, ‘training’, ‘scenario’, ‘patient involvement’, ‘software’, and ‘hardware’. We analyse three recently published clinical usability evaluation articles to assess (and illustrate) the applicability and completeness of these dimensions. Finally, we discuss the compromises encountered during clinical usability evaluations and how to best report on these considerations. The framework presented here aims to further the agenda of ecologically valid evaluation practice, reflecting the constraints of medical practice.Close abstract N. van Berkel, B. Tag, J. Goncalves, S. Hosio, "Human-Centred Artificial Intelligence: A Contextual Morality Perspective", Behaviour & Information Technology, 2020, 1–17.
Abstract: The emergence of big data combined with the technical developments in Artificial Intelligence has enabled novel opportunities for autonomous and continuous decision support. While initial work has begun to explore how human morality can inform the decision making of future Artificial Intelligence applications, these approaches typically consider human morals as static and immutable. In this work, we present an initial exploration of the effect of context on human morality from a Utilitarian perspective. Through an online narrative transportation study, in which participants are primed with either a positive story, a negative story or a control condition (N = 82), we collect participants’ perceptions on technology that has to deal with moral judgment in changing contexts. Based on an in-depth qualitative analysis of participant responses, we contrast participant perceptions to related work on Fairness, Accountability and Transparency. Our work highlights the importance of contextual morality for Artificial Intelligence and identifies opportunities for future work through a FACT-based (Fairness, Accountability, Context and Transparency) perspective.Close abstract N. van Berkel, E. Papachristos, A. Giachanou, S. Hosio, M. B. Skov, "A Systematic Assessment of National Artificial Intelligence Policies: Perspectives from the Nordics and Beyond", in Proceedings of the 11th Nordic Conference on Human-Computer Interaction (NordiCHI’20), 2020, 1–12.
Abstract: Echoing the evolving interest and impact of artificial intelligence on society, governments are increasingly looking for ways to strategically position themselves as both innovators and regulators in this new domain. One of the most explicit and accessible ways in which governments outline these plans is through national strategy and policy documents. We follow a systematic search strategy to identify national AI policy documents across twenty-five countries. Through an analysis of these documents, including topic modelling, clustering, and reverse topic-search, we provide an overview of the topics discussed in national AI policies and contrast the differences between countries. Furthermore, we analyse the frequency of eleven ethical principles across our corpus. Our paper outlines implications of the differences between geographical and cultural clusters in relation to the future development of artificial intelligence applications.Close abstract D. Hettiachchi, N. van Berkel, V. Kostakos, J. Goncalves, "CrowdCog: A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in Crowdsourcing", Proceedings of the ACM on Human-Computer Interaction – CSCW, 2020, 110:1–110:22.
Abstract: While crowd workers typically complete a variety of tasks in crowdsourcing platforms, there is no widely accepted method to successfully match workers to different types of tasks. Researchers have considered using worker demographics, behavioural traces, and prior task completion records to optimise task assignment. However, optimum task assignment remains a challenging research problem due to limitations of proposed approaches, which in turn can have a significant impact on the future of crowdsourcing. We present ‘CrowdCog’, an online dynamic system that performs both task assignment and task recommendations, by relying on fast-paced online cognitive tests to estimate worker performance across a variety of tasks. Our work extends prior work that highlights the effect of workers’ cognitive ability on crowdsourcing task performance. Our study, deployed on Amazon Mechanical Turk, involved 574 workers and 983 HITs that span across four typical crowd tasks (Classification, Counting, Transcription, and Sentiment Analysis). Our results show that both our assignment method and recommendation method result in a significant performance increase (5% to 20%) as compared to a generic or random task assignment. Our findings pave the way for the use of quick cognitive tests to provide robust recommendations and assignments to crowd workers.Close abstract S. Wijenayake, N. van Berkel, V. Kostakos, J. Goncalves, "Quantifying the Effect of Social Presence on Online Social Conformity", Proceedings of the ACM on Human-Computer Interaction – CSCW, 2020, 55:1–55:22.
N. van Berkel, J. Goncalves, K. Wac, S. Hosio, A. L. Cox, "Human Accuracy in Mobile Data Collection", International Journal of Human-Computer Studies, vol. 137, 2020, 1–4.
Abstract: The collection of participant data ‘in the wild’ is widely employed by Human-Computer Interaction researchers. A variety of methods, including experience sampling, mobile crowdsourcing, and citizen science, rely on repeated participant contributions for data collection. Given this strong reliance on participant data, ensuring that the data is complete, reliable, timely, and accurate is key. Although previous work has made significant progress on ensuring that a sufficient amount of data is collected, the accuracy of human contributions has remained underexposed. In this article we argue for an emerging need for an increased focus on this aspect of human-labelled data. The articles published in this special issue demonstrate how a focus on the accuracy of the collected data has implications on all aspects of a study – ranging from study design to the analysis and reporting of results. We put forward a five-point research agenda in which we outline future opportunities in assessing and improving human accuracy in mobile data collection.Close abstract S. Wijenayake, N. van Berkel, V. Kostakos, J. Goncalves, "Impact of Contextual and Personal Determinants on Online Social Conformity", Computers in Human Behaviour, vol. 108, 2020, 106302:1–106302:11.
Z. Sarsenbayeva, G. Marini, N. van Berkel, C. Luo, W. Jiang, K. Yang, G. Wadley, T. Dingler, V. Kostakos, J. Goncalves, "Does Smartphone Use Drive our Emotions or vice versa? A Causal Analysis", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’20), 2020, 36:1–36:15.
Abstract: In this paper, we demonstrate the existence of a bidirectional causal relationship between smartphone application use and user emotions. In a two-week long in-the-wild study with 30 participants we captured 502,851 instances of smartphone application use in tandem with corresponding emotional data from facial expressions. Our analysis shows that while in most cases application use drives user emotions, multiple application categories exist for which the causal effect is in the opposite direction. Our findings shed light on the relationship between smartphone use and emotional states. We furthermore discuss the opportunities for research and practice that arise from our findings and their potential to support emotional well-being.Close abstract D. Hettiachchi, Z. Sarsenbayeva, F. Allison, N. van Berkel, T. Dingler, G. Marini, V. Kostakos, J. Goncalves, "“Hi! I am the Crowd Tasker” – Crowdsourcing through Digital Voice Assistants", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’20), 2020, 193:1–193:14.
Abstract: Inspired by the increasing prevalence of digital voice assistants, we demonstrate the feasibility of using voice interfaces to deploy and complete crowd tasks. We have developed Crowd Tasker, a novel system that delivers crowd tasks through a digital voice assistant. In a lab study, we validate our proof-ofconcept and show that crowd task performance through a voice assistant is comparable to that of a web interface for voicecompatible and voice-based crowd tasks for native English speakers. We also report on a field study where participants used our system in their homes. We find that crowdsourcing through voice can provide greater flexibility to crowd workers by allowing them to work in brief sessions, enabling multitasking, and reducing the time and effort required to initiate tasks. We conclude by proposing a set of design guidelines for the creation of crowd tasks for voice and the development of future voice-based crowdsourcing systems.Close abstract D. Hettiachchi, N. van Berkel, S. Hosio, M. B. Lopez, V. Kostakos, J. Goncalves, "Augmenting Automated Kinship Verification with Targeted Human Input", in Proceedings of the Pacific Asia Conference on Information Systems (PACIS’20), 2020, 141:1–141:14.
Abstract: Kinship verification is the problem whereby a third party determines whether two people are related. Despite previous research in Psychology and Machine Vision, the factors affecting a person’s verification ability are poorly understood. Through an online crowdsourcing study, we investigate the impact of gender, race and medium type (image vs video) on kinship verification – taking into account the demographics of both raters and ratees. A total of 325 workers completed over 50,000 kinship verification tasks consisting of pairs of faces shown in images and videos from three widely used datasets. Our results identify an own-race bias and a higher verification accuracy for same-gender image pairs than opposite-gender image pairs. Our results demonstrate that humans can still outperform current state-of-the-art automated unsupervised approaches. Furthermore, we show that humans perform better when presented with videos instead of still images. Our findings contribute to the design of future humanin-the-loop kinship verification tasks, including time-critical use cases such as identifying missing persons.Close abstract T. Dingler, S. Li, N. van Berkel, V. Kostakos, "Page-Turning Techniques for Reading Interfaces in Virtual Environments", in Proceedings of the Australian Conference on Human-Computer Interaction (OzCHI’20), 2020, 454–461.
Abstract: Virtual Reality (VR) environments ofer new ways and formats to consume and process information. Despite multimedia oferings, most information remains to be presented via text. VR has the potential to deliver immersive reading experiences while compen- sating for some of the drawbacks of rather static e-books. To allow readers to step into virtual books, we developed a 3D reading envi- ronment with three page-turning techniques for VR. Readers either move the camera position from page to page or control the page fow as positioned in a sequential or radial arrangement. Results from a user study with 18 participants show that moving pages is perceived as more comfortable than moving the camera position while allowing for higher fuency and reading speeds. Linear page movements support readers’ focus on a single page whereas the radial arrangement enables readers to jump between pages quickly. Our fndings inform the design of immersive reading experiences in VR.Close abstract A. Alorwu, N. van Berkel, J. Goncalves, J. Oppenlaender, M. B. Lopez, M. Seetharaman, S. Hosio, "Crowdsourcing Sensitive Data using Public Displays: Opportunities, Challenges, and Considerations", Personal and Ubiquitous Computing, 2020, 1–16.
Abstract: Interactive public displays are versatile two-way interfaces between the digital world and passersby. They can convey information and harvest purposeful data from their users. Surprisingly little work has exploited public displays for collecting tagged data that might be useful beyond a single application. In this work, we set to fill this gap and present two studies: (1) a field study where we investigated collecting biometrically tagged video-selfies using public kiosk-sized screens, and (2) an online narrative transportation study that further elicited rich qualitative insights on key emerging aspects from the first study. In the first study, a 61-day deployment resulted in 199 video-selfies with consent to leverage the videos in any non-profit research. The field study indicates that people are willing to donate even highly sensitive data about themselves in public. The subsequent online narrative transportation study provides a deeper understanding of a variety of issues arising from the first study that can be leveraged in the future design of such systems. The two studies combined in this article pave the way forward towards a vision where volunteers can, should they so choose, ethically and serendipitously help unleash advances in data-driven areas such as computer vision and machine learning in health care.Close abstract N. van Berkel, J. Goncalves, S. Hosio, Z. Sarsenbayeva, E. Velloso, V. Kostakos, "Overcoming Compliance Bias in Self-Report Studies: A Cross-Study Analysis", International Journal of Human-Computer Studies, vol. 134, 2020, 1–12.
Abstract: A popular methodology used for in situ observations is the Experience Sampling Method (ESM), in which participants intermittently answer short questionnaires. We analyse a set of recent ESM studies and find substantial differences in the number of collected responses across participants. These differences amount to ‘compliance bias’, as the experiences of responsive participants skew the results. Our work develops ways for researchers to ensure the collection of an adequate number of responses across participants. Through a cross-study analysis of ESM studies, we construct a model that describes the effect of contextual, routine, and study-specific factors on participants’ response rate. In addition to previous work, which aims to maximise the number of total responses, this work also aims to achieve a more equal distribution of responses between participants. In order to achieve this goal, we analyse which contextual cues can be personalised to achieve a higher response rate. Our results highlight a number of factors that have a strong effect on participants’ response rate and can guide the design of future experiments.Close abstract Y. Chen, N. van Berkel, C. Luo, Z. Sarsenbayeva, V. Kostakos, "Application of miniaturized near-infrared spectroscopy in pharmaceutical identification", Smart Health, vol. 18, 2020, 100126.
Abstract: NIRS is a spectroscopic method that propagates near-infrared waves through objects and measures the absorbance by diffuse reflection, users could analyze the composition information of objects based on that. The technology has fast speed and non-destructive analysis features with relatively simple requirements for operators, making it very friendly to non-expert users. Traditional NIRS scanners used in research laboratories are large and expensive, while recently more and more affordable smaller NIRS scanners are appearing, which attract more end-users to buy and use. Besides, pairing the technology with mobile devices (smartphones, tablets, etc.) could get rid of other professional operation problems, and bring much more possibilities to non-expert users in realistic scenarios. We will explore one such use case in this paper with the extension of work by (Klakegg et al., 2018), namely Smart Pillbox for elderly care. We develop a prototype solution consisting of a hardware-software assistance to support non-expert users.Close abstract S. Hosio, N. van Berkel, J. Oppenlaender, J. Goncalves, "Crowdsourcing Personalized Weight Loss Diets", IEEE Computer, vol. 53, no. 1, 2020, 63–71.
Abstract: The Diet Explorer is a lightweight system that relies on aggregated human insights for assessing and recommending suitable weight loss diets. We compared its performance against Google and suggest that the system, bootstrapped using a public crowdsourcing platform, provides comparable results in terms of overall satisfaction, relevance, and trustworthiness.Close abstract 2019
N. van Berkel, "Data Quality and Quantity in Mobile Experience Sampling", The University of Melbourne, 2019.
Abstract: The widespread availability of technologically-advanced mobile devices has brought researchers the opportunity to observe human life in day-to-day circumstances. Rather than studying human behaviour through extensive surveys or in artificial laboratory situations, this research instrument allows us to systematically capture human life in naturalistic settings. Mobile devices can capture two distinct data streams. First, the data from sensors embedded within these devices can be appropriated to construct the context of study participants. Second, participants can be asked to actively and repeatedly provide data on phenomena which cannot be reliably collected using the aforementioned sensor streams. This method is known as Experience Sampling. Researchers employing this method ask participants to provide observations multiple times per day, across a range of contexts, and to reflect on current rather than past experiences. This approach brings a number of advantages over existing methods, such as the ability to observe shifts in participant experiences over time and context, and reducing reliance on the participant’s ability to accurately recall past events. As the onus of data collection lies with participants rather researchers, there is a firm reliance on the reliability of participant contributions. While previous work has focused on increasing the number of participant contributions, the quality of these contributions has remained relatively unexplored. This thesis focuses on improving the quality and quantity of participant data collected through mobile Experience Sampling. Assessing and subsequently improving the quality of participant responses is a crucial step towards increasing the reliability of this increasingly popular data collection method. Previous recommendations for researchers are based primarily on anecdotal evidence or personal experience in running Experience Sampling studies. While such insights are valuable, it is challenging to replicate these recommendations and quantify their effect. Furthermore, we evaluate the application of this method in light of recent developments in mobile devices. The opportunities and challenges introduced by smartphone-based Experience Sampling studies remain underexplored in the current literature. Such devices can be utilised to infer participants’ context and optimise questionnaire scheduling and presentation to increase data quality and quantity. By deploying our studies on these devices, we explore the opportunities of mobile sensing and interaction in the context of mobile Experience Sampling studies. Our findings illustrate the feasibility of assessing and quantifying participant accuracy through the use of peer assessment, ground truth questions, and the assessment of cognitive skills. We empirically evaluate these approaches across a variety of study goals. Furthermore, our results provide recommendations on study design, motivation and data collection practices, and appropriate analysis techniques of participant data concerning response accuracy. Researchers can use our findings to increase the reliability of their data, to collect participant responses more evenly across different contexts in order to reduce the potential for bias, and to increase the total number of collected responses. The goal of this thesis is to improve the collection of human-labelled data in ESM studies, thereby strengthening the role of smartphones as valuable scientific instruments. Our work reveals a clear opportunity in the combination of human and sensor data sensing techniques for researchers interested in studying human behaviour in situ.Close abstract N. van Berkel, J. Goncalves, D. Hettiachchi, S. Wijenayake, R. M. Kelly, V. Kostakos, "Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study", Proceedings of the ACM on Human-Computer Interaction – CSCW, vol. 3, 2019, 28:1–28:21.
Abstract: The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants’ perceptions of fairness in private while simultaneously allowing for structured participant discussion.Close abstract S. Wijenayake, N. van Berkel, V. Kostakos, J. Goncalves, "Measuring the Effects of Gender on Online Social Conformity", Proceedings of the ACM on Human-Computer Interaction – CSCW, vol. 3, 2019, 145:1–145:24.
W. Jiang, G. Marini, N. van Berkel, Z. Sarsenbayeva, T. Zheyu, C. Luo, H. Xin, T. Dingler, J. Goncalves, Y. Kawahara, V. Kostakos, "Probing Sucrose Contents in Everyday Drinks Using Miniaturized Near-Infrared Spectroscopy Scanners", in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 3, no. 4, 2019, 136:1–136:25.
Abstract: Near-Infrared Spectroscopy (NIRS) is a non-invasive sensing technique which can be used to acquire information on an object’s chemical composition. Although NIRS is conventionally used in dedicated laboratories, the recent introduction of miniaturized NIRS scanners has greatly expanded the use cases of this technology. Previous work from the UbiComp community shows that miniaturized NIRS can be successfully adapted to identify medical pills and alcohol concentration. In this paper, we further extend this technology to identify sugar (sucrose) contents in everyday drinks. We developed a standalone mobile device which includes inter alia a NIRS scanner and a 3D printed clamp. The clamp can be attached to a straw-like tube to sense a liquid’s sucrose content. Through a series of studies, we show that our technique can accurately measure sucrose levels in both lab-made samples and commercially available drinks, as well as classify commercial drinks. Furthermore, we show that our method is robust to variations in the ambient temperature and lighting conditions. Overall, our system can estimate the concentration of sugar with ±0.29 g/100ml error in lab-made samples and < 2.0 g/100ml error in 18 commercial drinks, and can identify everyday drinks with > 99% accuracy. Furthermore, in our analysis, we are able to discern three characteristic wavelengths in the near-infrared region (1055 nm, 1235 nm and 1545 nm) with acute responses to sugar (sucrose). Our proposed protocol contributes to the development of everyday “food scanners” consumers.Close abstract N. van Berkel, J. Goncalves, P. Koval, S. Hosio, T. Dingler, D. Ferreira, V. Kostakos, "Context-Informed Scheduling and Analysis: Improving Accuracy of Mobile Self-Reports", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), 2019, 51:1–51:12.
Abstract: Mobile self-reports are a popular technique to collect participant labelled data in the wild. While literature has focused on increasing participant compliance to self-report questionnaires, relatively little work has assessed response accuracy. In this paper, we investigate how participant context can affect response accuracy and help identify strategies to improve the accuracy of mobile self-report data. In a 3-week study we collect over 2,500 questionnaires containing both verifiable and non-verifiable questions. We find that response accuracy is higher for questionnaires that arrive when the phone is not in ongoing or very recent use. Furthermore, our results show that long completion times are an indicator of a lower accuracy. Using contextual mechanisms readily available on smartphones, we are able to explain up to 13% of the variance in participant accuracy. We offer actionable recommendations to assist researchers in their future deployments of mobile self-report studies.Close abstract A. Visuri, N. van Berkel, T. Okoshi, J. Goncalves, V. Kostakos, "Understanding Smartphone Notifications’ User Interactions and Content Importance", International Journal of Human-Computer Studies, vol. 128, 2019, 72–85.
Abstract: We present the results of our experiment aimed to comprehensively understand the combination of 1) how smartphone users interact with their notifications, 2) what notification content is considered important, 3) the complex relationship between the interaction choices and content importance, and lastly 4) establish an intelligent method to predict user’s preference to seeing an incoming notification. We use a dataset of notifications received by 40 anonymous users in-the-wild, which consists of 1) qualitative user-labelled information about their preferences on notification’s contents, 2) notification source, and 3) the context in which the notification was received. We assess the effectiveness of personalised prediction models generated using a combination of self-reported content importance and contextual information. We uncover four distinct user types, based on the number of daily notifications and interaction choices. We showcase how usage traits of these groups highlight the requirement for notification filtering approaches, e.g., when specific users habitually neglect to manually filter out unimportant notifications. Our machine learning-based predictor, based on both contextual sensing and notification contents can predict the user’s preference for successfully acknowledging an incoming notification with 91.1% mean accuracy, crucial for time-critical user engagement and interventions.Close abstract J. Goncalves, S. Hosio, N. van Berkel, S. Klakegg, "Addressing Cooperation Issues in Situated Crowdsourcing", In book: Macrotask Crowdsourcing: Engaging the Crowds to Address Complex Problems, V. Khan, K. Papangelis, I. Lykourentzou, P. Markopoulos (Eds.), 2019, 127–145.
Abstract: Situated crowdsourcing has been growing in popularity as an alternative way to collect complex and often creative crowd work. However, previous situated crowdsourcing deployments have not successfully leveraged cooperation possibilities with their audiences, which can improve the data quality of deployed macrotasks. In this chapter, we present three situated crowdsourcing case studies that used different situated technologies and identify the reasons behind their missteps regarding promoting cooperation between workers. Then, based on the identified issues, we propose the design of a novel situated crowdsourcing platform that aims to effectively support cooperation without alienating solo workers. In order to gather insights on our proposed design, we built a prototype platform and evaluated it using a laboratory study with 24 participants. In general, participants were positive about the idea as it provided an easy way to cooperate with friends when completing tasks, while also allowing them to adjust the working environment to their liking. Finally, we conclude by offering insights towards improving cooperation in future situated crowdsourcing deployments and how this can assist in completing macrotasks.Close abstract Z. Sarsenbayeva, N. van Berkel, D. Hettiachchi, W. Jiang, T. Dingler, E. Velloso, V. Kostakos, J. Goncalves, "Measuring the Effects of Stress on Mobile Interaction", in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 3, no. 1, 2019, 24:1–24:18.
Abstract: Research shows that environmental factors such as ambient noise and cold ambience can render users situationally impaired, adversely affecting interaction with mobile devices. However, an internal factor which is known to negatively impact cognitive abilities – stress – has not been systematically investigated in terms of its impact on mobile interaction. In this paper, we report a study where we use the Trier Social Stress Test to induce stress on participants, and investigate its effect on three aspects of mobile interaction: target acquisition, visual search, and text entry. We find that stress reduces completion time and accuracy during target acquisition tasks, as well as completion time during visual search tasks. Finally, we are able to directly contrast the magnitude of these effects to previously published effects of environmentally-caused impairments. Our work contributes to the growing body of literature on situational impairments.Close abstract S. Hosio, A. Alorwu, N. van Berkel, M. Bordallo, M. Seetharaman, J. Oppenlaender, J. Goncalves, "Fueling AI with Public Displays? A Feasibility Study of Collecting Biometrically Tagged Consensual Data on a University Campus", in Proceedings of the ACM International Symposium on Pervasive Displays (PerDis’19), 2019, 14:1–14:7.
Abstract: Interactive public displays have matured into highly capable two-way interfaces. They can be used for efficiently delivering information to people as well as for collecting insights from their users. While displays have been used for harvesting opinions and other content from users, surprisingly little work has looked into exploiting such screens for the consensual collection of tagged data that might be useful beyond one application. We present a field study where we collected biometrically tagged data using public kiosk-sized interactive screens. During 61 days of deployment time, we collected 199 selfie videos, cost-efficiently and with consent to leverage the videos in any non-profit research. 78 of the videos also had metadata attached to them. Overall, our studies indicate that people are willing to donate even highly sensitive data about themselves in public but that, at the same time, the participants had specific ethical and privacy concerns over the future of their data. Our study paves the way forward toward a future where volunteers can ethically help advance innovations in computer vision research across a variety of exciting application domains, such as health monitoring and care.Close abstract D. Hettiachchi, N. van Berkel, S. Hosio, V. Kostakos, J. Goncalves, "Effect of Cognitive Abilities on Crowdsourcing Task Performance", in Proceedings of the 17th IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT’19), 2019, 442–464.
Abstract: Matching crowd workers to suitable tasks is highly desirable as it can enhance task performance, reduce the cost for requesters, and increase worker satisfaction. In this paper, we propose a method that considers workers’ cognitive ability to predict their suitability for a wide range of crowdsourcing tasks. We measure cognitive ability via fast-paced online cognitive tests with a combined average duration of 6.2 minutes. We then demonstrate that our proposed method can effectively assign or recommend workers to five different popular crowd tasks: Classification, Counting, Proofreading, Sentiment Analysis, and Transcription. Using our approach we demonstrate a significant improvement in the expected overall task accuracy. While previous methods require access to worker history or demographics, our work offers a quick and accurate way to determine which workers are more suitable for which tasks.Close abstract Z. Sarsenbayeva, N. van Berkel, W. Jiang, D. Hettiachchi, V. Kostakos, J. Goncalves, "Effect of Ambient Light on Mobile Interaction", in Proceedings of the 17th IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT’19), 2019, 465–475.
Abstract: The adverse effect of ambient noise on humans has been extensively studied in fields like cognitive science, indicating a significant impact on cognitive performance, behaviour, and emotional state. Surprisingly, the effect of ambient noise has not been studied in the context of mobile interaction. As smartphones are ubiquitous by design, smartphone users are exposed to a wide variety of ambient noises while interacting with their devices. In this paper, we present a structured analysis of the effect of six distinct ambient noise types on typical smartphone usage tasks. The evaluated ambient noise types include variants of music, urban noise and speech. We analyse task completion time and errors, and find that different ambient noises affect users differently. For example, while speech and urban noise slow down text entry, being exposed to music reduces completion time in target acquisition tasks. Our study contributes to the growing research area on situational impairments, and we compare our results to previous work on the effect of cold-induced situational impairments. Our results can be used to support smartphone users through adaptive interfaces which respond to the ongoing context of the user.Close abstract N. van Berkel, J. Goncalves, L. Lovén, D. Ferreira, S. Hosio, V. Kostakos, "Effect of Experience Sampling Schedules on Response Rate and Recall Accuracy of Objective Self-Reports", International Journal of Human-Computer Studies, vol. 125, 2019, 118–128.
Abstract: The Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of the methodological decisions faced by researchers is deciding on the question scheduling. The literature defines three distinct schedule types: randomised, interval-based, or event-based (in our case, smartphone unlock). However, little evidence exists regarding the side-effects of these schedules on response rate and recall accuracy, and how they may bias study findings. We evaluate the effect of these three contingency configurations in a 3-week within-subjects study (N = 20). Participants answered various objective questions regarding their phone usage, while we simultaneously establish a ground-truth through smartphone instrumentation. We find that scheduling questions on phone unlock yields a higher response rate and accuracy. Our study provides empirical evidence for the effects of notification scheduling on participant responses, and informs researchers who conduct experience sampling studies on smartphones.Close abstract C. Luo, A. Visuri, S. Klakegg, N. van Berkel, Z. Sarsenbayeva, A. Möttönen, J. Goncalves, T. Anagnostopoulos, D. Ferreira, H. Flores, E. Velloso, V. Kostakos, "Energy-Efficient Prediction of Smartphone Unlocking", Personal and Ubiquitous Computing, vol. 23, no. 1, 2019, 159–177.
Abstract: We investigate the predictability of the next unlock event on smartphones, using machine learning and smartphone contextual data. In a 2-week field study with 27 participants, we demonstrate that it is possible to predict when the next unlock event will occur. Additionally, we show how our approach can improve accuracy and energy efficiency by solely relying on software-related contextual data. Based on our findings, smartphone applications and operating systems can improve their energy efficiency by utilising short-term predictions to minimise unnecessary executions, or launch computation-intensive tasks, such as OS updates, in the locked state. For instance, by inferring the next unlock event, smartphones can pre-emptively collect sensor data or prepare timely content to improve the user experience during the subsequent phone usage session.Close abstract P. Markkanen, N. van Berkel, A. Visuri, A. LeSaint, D. Ferreira, A. Herneoja, "Exploring Work Environment Usage Behaviour through Context-Aware Mobile Methods", in Proceedings of the eCAADe + SIGraDi Conference on Architecture in the Age of the 4th Industrial Revolution (eCAADe & SIGraDi’19), 2019, 837–846.
Abstract: This paper presents our findings on knowledge work environment usage behaviour through a combined automated mobile indoor positioning system and self-reports collected from the environment’s inhabitants. Contemporary work environments are increasingly flexible multi-occupant environments as opposed to cellular offices. Understanding persons’ task-related and situation-related environmental needs is critical to improve the design of future knowledge work environments. This study is conducted in a team office environment prior to and following an intervention in which the office layout was re-organized. The combined methodological approach described in this paper provides a new tool for architecture researchers aiming to understand the use of workspaces. Importantly, combining self-reports with context-aware location data collection provides researchers an efficient in situ tool to access participants experiences and decision-making process in choosing their workstation or workspace.Close abstract J. Liono, F. D. Salim, N. van Berkel, V. Kostakos, A. K. Qin, "Improving Experience Sampling with Multi-view User-driven Annotation Prediction", in Proceedings of IEEE International Conference on Pervasive Computing and Communications (PerCom’19), 2019, 22–32.
Abstract: A fundamental challenge in real-time labelling of activity data is user burden. The Experience Sampling Method (ESM) is widely used to obtain such labels for sensor data. However, in an in-situ deployment, it is not feasible to expect users to precisely label the start and end time of each event or activity. For this reason, time-point based experience sampling (without an actual start and end time) is prevalent. We present a framework that applies multi-instance and semi-supervised learning techniques to perform to predict user annotations from multiple mobile sensor data streams. Our proposed framework estimates users’ annotations in ESM-based studies progressively, via an interactive pipeline of co-training and active learning. We evaluate our work using data collected from an in-the-wild data collection.Close abstract 2018
N. van Berkel, J. Vega, A. Kariryaa, E. C. Smith, Y. Yuan, "CHI 2018 – Conference Report", IEEE Pervasive Computing, vol. 17, no. 3, 2018, 58–63.
Abstract: The 2018 ACM CHI Conference on Human Factors in Computing Systems took place 21-27 April in Montreal, Canada, and attracted over 3,000 participants. This years CHI marked the 50th anniversary of the Mother of all Demos and showcased many exciting new technologies and techniques that will help shape the future of pervasive computing.Close abstract S. Hosio, J. Karppinen, N. van Berkel, J. Oppenlaender, J. Goncalves, "Mobile Decision Support and Data Provisioning for Low Back Pain", IEEE Computer, vol. 51, no. 8, 2018, 34–43.
Abstract: The authors present Back Pain Buddy, a mobile application offering decision support and coaching for people with low back pain (LBP). The application takes advantage of smartphones powerful capabilities and provides a crowd-sourced decision support system for discovering treatments and a mobile sensing solution for collecting data about users activities that are crucial in LBP research.Close abstract Z. Sarsenbayeva, N. van Berkel, V. Kostakos, E. Velloso, J. Goncalves, "Effect of Distinct Ambient Noise Types on Mobile Interaction", in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 2, no. 2, 2018, 82:1–82:23.
Abstract: The adverse effect of ambient noise on humans has been extensively studied in fields like cognitive science, indicating a significant impact on cognitive performance, behaviour, and emotional state. Surprisingly, the effect of ambient noise has not been studied in the context of mobile interaction. As smartphones are ubiquitous by design, smartphone users are exposed to a wide variety of ambient noises while interacting with their devices. In this paper, we present a structured analysis of the effect of six distinct ambient noise types on typical smartphone usage tasks. The evaluated ambient noise types include variants of music, urban noise and speech. We analyse task completion time and errors, and find that different ambient noises affect users differently. For example, while speech and urban noise slow down text entry, being exposed to music reduces completion time in target acquisition tasks. Our study contributes to the growing research area on situational impairments, and we compare our results to previous work on the effect of cold-induced situational impairments. Our results can be used to support smartphone users through adaptive interfaces which respond to the ongoing context of the user.Close abstract P. Kostakos, A. Pandya, M. Oussalah, S. Hosio, C. Breidbach, V. Kostakos, N. van Berkel, "Correlating Refugee Border Crossings with Internet Search Data", in Proceedings of the International Conference on Information Reuse and Integration for Data Science (IEEE IRI’18), 2018, 264–268.
Abstract: Can Internet search data be used as a proxy to predict refugee mobility? The soaring refugee death toll in Europe creates an urgent need for novel tools that monitor and forecast refugee flows. This study investigates the correlation between refugee mobility data and Internet search data from Google Trends. Google Trends is a freely accessible tool that provides access to Internet search data by analyzing a sample of all web queries. In our study, we surveyed refugees in Greece (entry point) and in Finland (destination point) to identify what search queries they had used during their travel. Next, we conducted time series analysis on Google search data to investigate whether interest in user-defined search queries correlated with the levels of refugee arrival data recorded by the United Nations High Commissioner for Refugees (UNHCR). Results indicate that the reuse of internet search data considerably improves the predictive power of the models.Close abstract S. Klakegg, J. Goncalves, C. Luo, A. Visuri, A. Popov, N. van Berkel, Z. Sarsenbayeva, V. Kostakos, S. Hosio, S. Savage, A. Bykov, I. Meglinski, D. Ferreira, "Assisted Medication Management in Elderly Care Using Miniaturised Near-Infrared Spectroscopy", in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 2, no. 2, 2018, 69:1–69:24.
Abstract: Near-infrared spectroscopy (NIRS) measures the light reflected from objects to infer highly detailed information about their molecular composition. Traditionally, NIRS has been an instrument reserved for laboratory usage, but recently affordable and smaller devices for NIRS have proliferated. Pairing this technology with the ubiquitous smartphone opens up a plethora of new use cases. In this paper, we explore one such use case, namely medication management in a nursing home/elderly care centre. First, we conducted a qualitative user study with nurses working in an elderly care centre to examine the protocols and workflows involved in administering medication, and the nurses’ perceptions on using this technology. Based on our findings, we identify the main impact areas that would benefit from introducing miniaturised NIRS. Finally, we demonstrate via a user study in a realistic scenario that miniaturised NIRS can be effectively used for medication management when leveraging appropriate machine learning techniques. Specifically, we assess the performance of multiple pre-processing and classification algorithms for a selected set of pharmaceuticals. In addition, we compare our solution with currently used methods for pharmaceutical identification in a local care centre. We hope that our reflection on the multiple aspects associated with the introduction of this device in an elderly care setting can help both academics and practitioners working on related problems.Close abstract J. Goncalves, S. Klakegg, N. van Berkel, S. Hosio, "Facilitating Analysis of Big Data on Reddit via an Easy to Use Visualisation Tool", in Proceedings of the British Human Computer Interaction Conference (British HCI’18), 2018, 1–6.
Abstract: With the rapid proliferation of social media sites, researchers have increasingly turned to data generated from these platforms to investigate human behaviour. In this paper we report the design and implementation of the RDV (Reddit Data Visualisation) platform, a visualisation tool aimed at facilitating the analysis of a publicly available Reddit dataset, which contains 1.7 billion JSON objects collected from October 2007 to October 2015. RDV allows for researchers without advanced coding skills to easily analyse this dataset, while also providing a tailor-made platform to account for the intricacies of any dataset originating from Reddit. We showcase the features of the platform through an example of data analysis using the Reddit dataset: the 2015 United Kingdom general elections. Finally, we conclude by discussing the need for better and simpler visualisation tools for non-technical researchers to analyse Big Online Behavioural Datasets, and report our ongoing work in this area.Close abstract S. Hosio, J. Karppinen, E. Takala, J. Takatalo, J. Goncalves, N. van Berkel, S. Konomi, V. Kostakos, "Crowdsourcing Treatments for Low Back Pain", in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’18), 2018, 276:1–276:12.
Abstract: Low back pain (LBP) is a globally common condition with no silver bullet solutions. Further, the lack of therapeutic consensus causes challenges in choosing suitable solutions to try. In this work, we crowdsourced knowledge bases on LBP treatments. The knowledge bases were used to rank and offer best-matching LBP treatments to end users. We collected two knowledge bases: one from clinical professionals and one from non-professionals. Our quantitative analysis revealed that non-professional end users perceived the best treatments by both groups as equally good. However, the worst treatments by non-professionals were clearly seen as inferior to the lowest ranking treatments by professionals. Certain treatments by professionals were also perceived significantly differently by non-professionals and professionals themselves. Professionals found our system handy for self-reflection and for educating new patients, while non-professionals appreciated the reliable decision support that also respected the non-professional opinion.Close abstract S. Hosio, J. Goncalves, N. van Berkel, S. Klakegg, S. Konomi, V. Kostakos, "Facilitating Collocated Crowdsourcing on Situated Displays", Human-Computer Interaction, 2018, 1–37.
Abstract: Online crowdsourcing enables the distribution of work to a global labor force as small and often repetitive tasks. Recently, situated crowdsourcing has emerged as a complementary enabler to elicit labor in specific locations and from specific crowds. Teamwork in online crowdsourcing has been recently shown to increase the quality of output, but teamwork in situated crowdsourcing remains unexplored. We set out to fill this gap. We present a generic crowdsourcing platform that supports situated teamwork and provide experiences from a laboratory study that focused on comparing traditional online crowdsourcing to situated team-based crowdsourcing. We built a crowdsourcing desk that hosts three networked terminal displays. The displays run our custom team-driven crowdsourcing platform that was used to investigate collocated crowdsourcing in small teams. In addition to analyzing quantitative data, we provide findings based on questionnaires, interviews, and observations. We highlight 1) emerging differences between traditional and collocated crowdsourcing, 2) the collaboration strategies that teams exhibited in collocated crowdsourcing, and 3) that a priori team familiarity does not significantly affect collocated interaction in crowdsourcing. The approach we introduce is a novel multi-display crowdsourcing setup that supports collocated labor teams and along with the reported study makes specific contributions to situated crowdsourcing research.Close abstract 2017
N. van Berkel, D. Ferreira and V. Kostakos, "The Experience Sampling Method on Mobile Devices", ACM Computing Surveys, vol. 50, no. 6, 2017, 93:1–93:40.
Abstract: The Experience Sampling Method (ESM) is used by scientists from various disciplines to gather insights into the intra-psychic elements of human life. Researchers have used the ESM in a wide variety of studies, with the method seeing increased popularity. Mobile technologies have enabled new possibilities for the use of the ESM, while simultaneously leading to new conceptual, methodological, and technological challenges. In this survey, we provide an overview of the history of the ESM, usage of this methodology in the computer science discipline, as well as its evolution over time. Next, we identify and discuss important considerations for ESM studies on mobile devices, and analyse the particular methodological parameters scientists should consider in their study design. We reflect on the existing tools that support the ESM methodology and discuss the future development of such tools. Finally, we discuss the effect of future technological developments on the use of the ESM and identify areas requiring further investigation.Close abstract Z. Sarsenbayeva, N. van Berkel, C. Luo, V. Kostakos, J. Goncalves, "Challenges of Situational Impairments during Interaction with Mobile Devices", in Proceedings of the Australian Conference on Human-Computer Interaction (OzCHI’17), 2017, 477–481.
Abstract: User interaction with mobile devices can be negatively affected by contextual factors, known as situationally-induced impairments. In this paper, we provide a systematic overview of established situational impairments and their impact on interaction with mobile devices, as well as existing methods for their detection and design guidelines to overcome them. We also propose a research roadmap for this topic where we argue that more experiments are required regarding the less investigated situational impairments. Furthermore, we argue that successful detection of the presence of a specific situational impairment is paramount before solutions can be proposed to adapt mobile interfaces to accommodate potential situational impairments.Close abstract N. van Berkel, J. Goncalves, S. Hosio, V. Kostakos, "Gamification of Mobile Experience Sampling Improves Data Quality and Quantity", in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 1, no. 3, 2017, 107:1–107:21.
Abstract: The Experience Sampling Method is used to capture high-quality in situ data from study participants. This method has become popular in studies involving smartphones, where it is often adapted to motivate participation through the use of gamification techniques. However, no work to date has evaluated whether gamification actually affects the quality and quantity of data collected through Experience Sampling. Our study systematically investigates the effect of gamification on the quantity and quality of experience sampling responses on smartphones. In a field study, we combine event contingent and interval contingent triggers to ask participants to describe their location. Subsequently, participants rate the quality of these entries by playing a game with a purpose. Our results indicate that participants using the gamified version of our ESM software provided significantly higher quality responses, slightly increased their response rate, and provided significantly more data on their own accord. Our findings suggest that gamifying experience sampling can improve data collection and quality in mobile settings.Close abstract Z. Sarsenbayeva, N. van Berkel, A. Visuri, S. Rissanen, H. Rintamaki, V. Kostakos, J. Goncalves, "Sensing Cold-Induced Situational Impairments in Mobile Interaction using Battery Temperature", in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 1, no. 3, 2017, 98:1–98:9.
Abstract: Previous work has highlighted the detrimental effect of cold ambience on fine-motor skills during interaction with mobile devices. In this work, we develop a method to infer changes in finger temperature of smartphone users without the need for specialised hardware. Specifically, we demonstrate that smartphone battery temperature is a reliable gauge for determining changes to finger temperature. In addition, we show that the behaviour of smartphone battery temperature in cold settings is consistent across different smartphone models and battery configurations. Our method can be used to determine cold-induced situational impairments, and trigger interface adaptations during mobile interaction.Close abstract J. Goncalves, S. Hosio, N. van Berkel, F. Ahmed, V. Kostakos, "CrowdPickUp: Crowdsourcing Task Pickup in the Wild", in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), vol. 1, no. 3, 2017, 51:1–51:22.
Abstract: We develop and evaluate a new ubiquitous crowdsourcing platform called CrowdPickUp, that combines the advantages of mobile and situated crowdsourcing to overcome their respective limitations. In a 19-day long field study with 70 participants, we evaluate the quality of work that CrowdPickUp produces. In particular, we measure quality in terms of worker performance in a variety of tasks (requiring local knowledge, location-based, general) while using a number of different quality control mechanisms, and also capture workers’ perceptions of the platform. Our findings show that workers of CrowdPickUp contributed data of comparable quality to previously presented crowdsourcing deployments while at the same time allowing for a wide breadth of tasks to be deployed. Finally, we offer insights towards the continued exploration of this research agenda.Close abstract A. Visuri, N. van Berkel, J. Goncalves, C. Luo, D. Ferreira, V. Kostakos, "Predicting Interruptibility for Manual Data Collection: A Cluster-Based User Model", in Proceedings of the International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI’17), 2017, Article 12.
Abstract: Previous work suggests that Quantified-Self applications can retain long-term usage with motivational methods. These methods often require intermittent attention requests with manual data input. This may cause unnecessary burden to the user, leading to annoyance, frustration and possible application abandonment. We designed a novel method that uses on-screen alert dialogs to transform recurrent smartphone usage sessions into moments of data contributions and evaluate how accurately machine learning can reduce unintended interruptions. We collected sensor data from 48 participants during a 4-week long deployment and analysed how personal device usage can be considered in scheduling data inputs. We show that up to 81.7% of user interactions with the alert dialogs can be accurately predicted using user clusters, and up to 75.5% of unintended interruptions can be prevented and rescheduled. Our approach can be leveraged by applications that require selfreports on a frequent basis and may provide a better longitudinal QS experience.Close abstract A. Visuri, N. van Berkel, C. Luo, J. Goncalves, D. Ferreira, V. Kostakos, "Challenges of Quantified-Self: Encouraging Self-Reported Data Logging During Recurrent Smartphone Usage", in Proceedings of the British Human Computer Interaction Conference (British HCI’17), 2017.
Abstract: We argue that improved data entry can motivate Quantified-Self (QS) users to better engage with QS applications. To improve data entry, we investigate the notion of transforming active smartphone usage into data logging contributions through alert dialogs. We evaluate this assertion in a 4-week long deployment with 48 participants. We collect 17,906 data entries, where 68.3% of the entries are reported using the alert dialogs. We demonstrate that QS applications can benefit from alert dialogs: to increase data precision, frequency, and reduce the probability of forgetfulness in data logging. We investigate the impact of usage session type (e.g., sessions with different goals or durations) and the assigned reminder delay on frequency of data contributions. We conclude with insights gathered from our investigation, and the implications they have on future designs.Close abstract S. Klakegg, N. van Berkel, A. Visuri, C. Luo, J. Goncalves, S. Hosio, H. Huttunen, D. Ferreira, "Informing Caregivers Through an Assistive Tool: An Investigation of Elderly Care Metrics", in Proceedings of the British Human Computer Interaction Conference (British HCI’17), 2017.
Abstract: Elderly care is a pressing societal challenge: government’s financial burden is expected to exponentially increase in the next 20 years as the population is aging rapidly. Solutions to mitigate this challenge include the use of IoT and software solutions to minimise the effort of elderly care, in care centres and at home. To accomplish this, we set to quantify what are the most important elderly care metrics (i.e., what is important to support caregivers’ work) through field observations and interviews at a local care centre housing 14 old adults. We designed iteratively and evaluated the usefulness of a mobile application with 8 caregivers, to summarise and communicate the care metrics, juxtaposed with wellbeing data (e.g., social interaction, mobility and others), part of a larger elderly care support platform, CARE. The goal of the mobile application is to enable a better care service by raising awareness to daily needs and routines of the elderly and to provide quick access to their wellbeing information. Our findings advocate that our design could positively benefit the care personnel and assist them carrying out the daily duties at the care centre.Close abstract S. Klakegg, J. Goncalves, N. van Berkel, C. Luo, S. Hosio, V. Kostakos, "Towards Commoditised Near Infrared Spectroscopy", in Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems (DIS’17), 2017, 515–527.
Abstract: Near Infrared Spectroscopy (NIRS) is a sensing technique in which near infrared light is transmitted into a sample, followed by light absorbance measurements at various wavelengths. This technique enables the inference of the inner chemical composition of the scanned sample, and therefore can be used to identify or classify objects. In this paper, we describe how to facilitate the use of NIRS by non- expert users in everyday settings. Our work highlights the key challenges of placing NIRS devices in the hands of non-experts. We develop a system to mitigate these challenges, and evaluate it in a user study. We show how NIRS technology can be successfully utilised by untrained users in an unsupervised manner through a special enclosure and an accompanying smartphone app. Finally, we discuss potential future developments of commoditised NIRS.Close abstract A. Visuri, Z. Sarsenbayeva, N. van Berkel, J. Goncalves, R. Rawassizadeh, V. Kostakos, D. Ferreira, "Quantifying Sources and Types of Smartwatch Usage Sessions", in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’17), 2017, 3569–3581.
Abstract: We seek to quantify smartwatch use, and establish differences and similarities to smartphone use. Our analysis considers use traces from 307 users that include over 2.8 million notifications and 800,000 screen usage events, and we compare our findings to previous work that quantifies smartphone use. The results show that smartwatches are used more briefly and more frequently throughout the day, with half the sessions lasting less than 5 seconds. Interaction with notifications is similar across both types of devices, both in terms of response times and preferred application types. We also analyse the differences between our smartwatch dataset and a dataset aggregated from four previously conducted smartphone studies. The similarities and differences between smartwatch and smartphone use suggest effect on usage that go beyond differences in form factor.Close abstract J. Goncalves, Z. Sarsenbayeva, N. van Berkel, S. Hosio, S. Rissanen, H. Rintamaki, V. Kostakos, "Tapping Task Performance on Smartphones in Cold Temperature", Interacting with Computers, vol. 29, no. 3, 2017, 355–367.
Abstract: We present a study that quantifies the effect of cold temperature on smartphone input performance, particularly on tapping tasks. Our results show that smartphone input performance decreases when completing tapping tasks in cold temperatures. We show that colder temperature is associated with lower throughput and less accurate performance when using the phone in both one-handed and two-handed operations. We also demonstrate that colder temperature is related to higher error rate when using the phone in one-handed operation only, but not two-handed. Finally, we identify a number of design recommendations from the literature that can be considered as a countermeasure to poorer smartphone input performance in completing tapping tasks in cold temperature.Close abstract 2016
N. van Berkel, C. Luo, T. Anagnostopoulos, D. Ferreira, J. Goncalves, S. Hosio, V. Kostakos, "A Systematic Assessment of Smartphone Usage Gaps", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), 2016, 4711–4721.
Abstract: Researchers who analyse smartphone usage logs often make the assumption that users who lock and unlock their phone for brief periods of time (e.g., less than a minute) are continuing the same “session” of interaction. However, this assumption is not empirically validated, and in fact different studies apply different arbitrary thresholds in their analysis. To validate this assumption, we conducted a field study where we collected user-labelled activity data through ESM and sensor logging. Our results indicate that for the majority of instances where users return to their smartphone, i.e., unlock their device, they in fact begin a new session as opposed to continuing a previous one. Our findings suggest that the commonly used approach of ignoring brief standby periods is not reliable, but optimisation is possible. We therefore propose various metrics related to usage sessions and evaluate various machine learning approaches to classify gaps in usage.Close abstract S. Hosio, D. Ferreira, J. Goncalves, N. van Berkel, C. Luo, M. Ahmed, H. Flores, V. Kostakos, "Monetary Assessment of Battery Life on Smartphones", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), 2016, 1869–1880.
Abstract: Research claims that users value the battery life of their smartphones, but no study to date has attempted to quantify battery value and how this value changes according to users’ current context and needs. Previous work has quantified the monetary value that smartphone users place on their data (e.g., location), but not on battery life. Here we present a field study and methodology for systematically measuring the monetary value of smartphone battery life, using a reverse second-price sealed-bid auction protocol. Our results show that the prices for the first and last 10% battery segments differ substantially. Our findings also quantify the tradeoffs that users consider in relation to battery, and provide a monetary model that can be used to measure the value of apps and enable fair ad-hoc sharing of smartphone resources.Close abstract G. V. Georgiev, M. Oja, I. Sanchez, M. Pyykkönen, T. Leppänen, J. Ylioja, N. van Berkel, J. Riekki, "Assessment of Relatedness to a Given Solution in 3D Fabrication and Prototyping Education", in Proceedings of the International Conference on Design Creativity (ICDC’16), 2016.
Abstract: This study outlines initial steps to define a new framework to measure relatedness, originality and creativity of student projects in FabLab environment. A default project topic provided to students in a 3D fabrication and prototyping class served as a basis to investigate originality on the functional component level. The added components with their input and output methods, along with the control logic, were used to judge the relatedness to a given solution of the generated design ideas. An example set of ideas given by the students was evaluated with the proposed framework. The framework can complement existing measures of originality and creativity in general.Close abstract Extended abstracts, workshops, late breaking work
J. Oppenlaender, S. Malacria, X. Fang, N. van Berkel, F. Chevalier, K. Yatani, S. Hosio, "Meta-HCI: First Workshop on Meta-Research in HCI", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’25 Workshop), 2025, to appear.
L. Meegahapola, D. Spathis, M. Constantinides, H. Zhang, S. Yfantidou, N. van Berkel, A. K. Dey, "FairComp: 2nd International Workshop on Fairness and Robustness in Machine Learning for Ubiquitous Computing", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’24 Adj.), 2024, 996–-999.
Abstract: How can we ensure that Ubiquitous Computing (UbiComp) research outcomes are ethical, fair, and robust? While fairness in machine learning (ML) has gained traction in recent years, it remains unexplored, or sometimes an afterthought, in the context of pervasive and ubiquitous computing. This workshop aims to discuss fairness in UbiComp research and its social, technical, and legal implications. From a social perspective, we will examine the relationship between fairness and UbiComp research and identify pathways to ensure that ubiquitous technologies do not cause harm or infringe on individual rights. From a technical perspective, we will initiate a discussion on model generalization and robustness, as well as data processing methods to develop bias mitigation approaches tailored to UbiComp research. From a legal perspective, we will examine how new policies shape our community’s work and future research. Building on the success of the First FairComp Workshop at UbiComp 2023, we have established a vibrant community centered around the topic of fair, robust, and trustworthy algorithms within UbiComp, while also charting a clear path for future research endeavors in this field.Close abstract A. Gammelgård-Larsen, N. van Berkel, M. B. Skov, J. Kjeldskov, "Designing for Human-AI Interaction: Comparing Intermittent, Continuous, and Proactive Interactions for a Music Application", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’24 LBW), 2024, 1–8.
Abstract: Designing effective and user-centred interactions between humans and AI systems poses fundamental challenges. The behaviour of AI systems is complex and uncertain, making it difficult to envision and craft optimal user experiences. Improved frameworks are needed to guide the design of human-AI interaction. In this paper, we develop and evaluate prototypes for a music application, representing three distinct paradigms of human-AI interaction: Intermittent, Continuous, and Proactive. Through qualitative user interviews with 12 participants, we compare the user experience across these prototypes, shedding light on potential challenges and opportunities for the paradigms represented. We found that the three prototypes exhibit distinct characteristics in terms of supported goals and user control. This case study contributes to a deeper understanding of the complexities involved in designing AI systems and offers insights for the development of more user-centred AI applications.Close abstract J. Wester, R. M. Jacobsen, S. de Jong, N. K. Kollerup, H. B. Djernæs, N. van Berkel, "Theory of Mind and Self-Presentation in Human-LLM Interactions", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’24 EA), Association for Computing Machinery, 2024, 1–4.
Abstract: The use of large language models (LLMs), such as ChatGPT, for social support and other activities is growing. LLM-based interactions require users to express themselves through text, a medium in which people’s distinct self-presentation styles (SPS) present themselves. While the divergence of people’s SPS is well-established, the effect of SPS on users’ LLM interactions has not been explored. In this position paper, we point to this gap by drawing on insights from prior work on people’s SPS online. Moreover, we discuss how Theory of Mind (ToM) can be used to increase our understanding of the possible effects of SPS on LLM output. Through this exploration, we shed light on how LLM responses are dependent on and sensitive to how people present themselves in their interactions with LLMs. We discuss the broader implications and suggest future research directions for HCI centred around people’s SPS in interacting with LLMs—providing concrete suggestions on how effects of SPS on LLM output can be empirically explored.Close abstract U. Ehsan, E. A. Watkins, P. Wintersberger, C. Manger, S. S. Y. Kim, N. van Berkel, A. Riener, M. O. Riedl, "Human-Centered Explainable AI (HCXAI): Reloading Explainability in the Era of Large Language Models (LLMs)", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’24 Workshop), 2024, 1–6.
Abstract: Human-centered XAI (HCXAI) advocates that algorithmic transparency alone is not sufficient for making AI explainable. Explainability of AI is more than just “opening” the black box — who opens it matters just as much, if not more, as the ways of opening it. In the era of Large Language Models (LLMs), is “opening the black box” still a realistic goal for XAI? In this fourth CHI workshop on Human-centered XAI (HCXAI), we build on the maturation through the previous three installments to craft the coming-of-age story of HCXAI in the era of Large Language Models (LLMs). We aim towards actionable interventions that recognize both affordances and pitfalls of XAI. The goal of the fourth installment is to question how XAI assumptions fare in the era of LLMs and examine how human-centered perspectives can be operationalized at the conceptual, methodological, and technical levels. Encouraging holistic (historical, sociological, and technical) approaches, we emphasize “operationalizing”. We seek actionable analysis frameworks, concrete design guidelines, transferable evaluation methods, and principles for accountability.Close abstract J. Wester, A. Brännström, J. C. Nieves, N. van Berkel, "“You’ve Got a Friend in Me”: A Formal Understanding of the Critical Friend Agent", in Proceedings of the 11th International Conference on Human-Agent Interaction (HAI’23), Association for Computing Machinery, 2023, 443–445.
Abstract: State-of-the-art intelligent and interactive agents, such as Alexa or Siri, often present overly conforming behaviour during interactions with humans. This can result in a misalignment between end-user expectations and agent behaviour. To overcome this barrier in human-AI interactions, we introduce the Critical Friend (CF), a conceptual idea that guides critical behaviour in human-human interactions. We present our results as a formal understanding that can be described through description logic and utilised for reasoning capabilities, enabling implementations of the CF as an intelligent interactive agent.Close abstract M. Oanh Hoang, N. van Berkel, M. B. Skov, T. Merritt, "Challenges and Requirements in Multi-Drone Interfaces", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’23 LBW), 2023, 1–9.
Abstract: Unmanned aerial vehicles (UAVs), commonly known as drones, have been deployed across various applications. These applications range from entertainment to critical situations, such as search and rescue (SAR) operations. The use of single drones is most common–one pilot controls one individual drone. Research has begun to explore the benefits of deploying a group of drones as a coordinated swarm. It is, however, uncertain how a multi-drone system should be designed to facilitate interaction in real-world contexts. We report initial findings from three study sessions involving prototype evaluations and co-design sessions we conducted in collaboration with the emergency services of Denmark. The results of our study open new questions and provide input on the features and functions that impact the future adoption of multi-drone systems, including interactions with multiple video feeds, ecology of screens, team communication, and flight control methods.Close abstract E. Schneiders, E. Papachristos, N. van Berkel, R. M. Jacobsen, "“Briefly Entertaining but Pointless”: Perceived Benefits & Risks of Social Robots in the Home", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’23 LBW), 2023, 1–10.
Abstract: In contrast to the adoption of personal assistants, social robots have yet to break into the domestic market. Several manufacturers have introduced and quickly retracted their social robots for the home. We report on a survey study (\textitN = 50) to understand potential users’ perceptions towards these social robots. Participants were presented with videos of three domestic social robots and subsequently provided their perception of these in terms of perceived benefits, attraction, privacy risk, usage intention, and capabilities. While participants perceived hedonic and utilitarian benefits, we found a low intention of future adoption of these devices. Further, our findings showed that owners of personal assistants perceived significantly higher hedonic benefits, fewer privacy risks, and higher intention to use domestic social robots. Our work provides an initial step towards understanding perceptions towards social robots and how previous exposure to domestic AI shapes users’ perceptions.Close abstract N. van Berkel, S. de Jong, J. Wester, N. K. K. Als, "The Challenge of Bias Mitigation in Clinical AI Decision Support: A Balance Between Decision Efficiency and Quality", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’23 EA), 2023, 1–2.
Abstract: An increasing number of intelligent data-driven health systems seek to support patients and clinicians in decision making tasks. However, the recommendations provided by such systems can negatively impact the reasoning abilities of its users, giving rise to cognitive biases. Such mental processes can subsequently harm the quality of the user’s decision. While decision support systems are typically designed to increase user efficiency, known approaches to mitigate such biases primarily rely on slowing down the decision making process—offsetting any efficiency benefits. This position paper calls attention to the efficiency–quality trade-off in bias mitigation and outlines a future research direction for bias mitigation in AI decision support.Close abstract J. Wester, J. Delaunay, S. de Jong, N. van Berkel, "On Moral Manifestations in Large Language Models", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’23 EA), 2023, 1–4.
Abstract: Since OpenAI released ChatGPT, researchers, policy-makers, and laypersons have raised concerns regarding its false and incorrect statements, which are furthermore expressed in an overly confident manner. We identify this flaw as part of its functionality and describe why large language models (LLMs), such as ChatGPT, should be understood as social agents manifesting morality. This manifestation happens as a consequence of human-like natural language capabilities, giving rise to humans interpreting the LLMs as potentially having moral intentions and abilities to act upon those intentions. We outline why appropriate communication between people and ChatGPT relies on moral manifestations by exemplifying `overly confident’ communication of knowledge. Moreover, we put forward future research directions of fully autonomous and semi-functional systems, such as ChatGPT, by calling attention to how engineers, developers, and designers can facilitate end-users sense-making of LLMs by increasing moral transparency.Close abstract J. Delaunay, L. Galárraga, C. Largouët, N. van Berkel, "Adaptation of AI Explanations to Users’ Roles", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’23 EA), 2023, 1–7.
Abstract: Surrogate explanations approximate a complex model by training a simpler model over an interpretable space. Among these simpler models, we identify three kinds of surrogate methods: (a) feature-attribution, (b) example-based, and (c) rule-based explanations. Each surrogate approximates the complex model differently, and we hypothesise that this can impact how users interpret the explanation. Despite the numerous calls for introducing explanations for all, no prior work has compared the impact of these surrogates on specific user roles (e.g., domain expert, developer). In this article, we outline a study design to assess the impact of these three surrogate techniques across different user roles.Close abstract S. Yfantidou, D. Spathis, M. Constantinides, T. Xia, N. van Berkel, "FairComp: Workshop on Fairness and Robustness in Machine Learning for Ubiquitous Computing", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’23 Adj.), 2023, 777–783.
Abstract: How can we ensure that Ubiquitous Computing (UbiComp) research outcomes are both ethical and fair? While fairness in machine learning (ML) has gained traction in recent years, fairness in UbiComp remains unexplored. This workshop aims to discuss fairness in UbiComp research and its social, technical, and legal implications. From a social perspective, we will examine the relationship between fairness and UbiComp research and identify pathways to ensure that ubiquitous technologies do not cause harm or infringe on individual rights. From a technical perspective, we will initiate a discussion on data collection and modeling practices to develop bias mitigation approaches tailored to UbiComp research. From a legal perspective, we will examine how new policies and regulations shape our community’s work and future research. We aim to foster a vibrant community centered around the topic of responsible UbiComp, while also charting a clear path for future research endeavours in this field.Close abstract N. Boonprakong, G. He, U. Gadiraju, N. van Berkel, D. Wang, S. Chen, J. Liu, B. Tag, J. Goncalves, T. Dingler, "Workshop on Understanding and Mitigating Cognitive Biases in Human-AI Collaboration", in Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing (CSCW’23 EA), 2023, 512–517.
Abstract: AI systems are increasingly incorporated into human decision-making. Yet, human decision-makers are often affected by their cognitive biases. In critical settings, such as medical diagnosis, criminal judgment, or information consumption, these cognitive biases hinder optimal decision outcomes, thereby resulting in unfair decisions and negative societal impact. The use of AI systems can amplify and exacerbate cognitive biases in their users. In this workshop, we seek to foster discussions on ongoing research around cognitive biases in human-AI collaboration and identify future research directions to understand, quantify, and mitigate the effects of cognitive biases. We will explore cognitive biases appearing in various contexts of human-AI collaboration: what can cause them?; how can we measure, model, mitigate, and manage cognitive biases?; and how can we utilise cognitive biases for the greater good? We will reflect on workshop discussions to form a research community around cognitive biases and bias-aware systems.Close abstract J. Wester, E. Schneiders and N. van Berkel, "Perceived Moral Agency of Non-Moral Entities: Implications and Future Research Directions for Social Robots", in Adjunct Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI’23 EA), 2023, 1–3.
Abstract: Humans tend to perceive human qualities in interactive systems. This particularly applies to social robots that utilise human attributes such as human body characteristics and natural language capabilities. Social robots with such characteristics are increasingly deployed in critical settings, such as health and well-being, where it is key to align robot behaviour with end-user expectations. Relatively little is known about how people perceive these social robots’ moral agency. In this position paper, we stress the difference between moral agency and perceived moral agency, and argue that the latter is a timely concern. We discuss the implications of perceived moral agency and outline research directions to explore how humans make sense of social robots in critical settings through perceived moral agency.Close abstract D. Ustalov, S. Savage, N. van Berkel, Y. Liu, "4th Crowd Science Workshop — CANDLE: Collaboration of Humans and Learning Algorithms for Data Labeling", in Adjunct Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM’23 Workshop), 2023, 1.
Abstract: Crowdsourcing has been used to produce impactful and large-scale datasets for Machine Learning and Artificial Intelligence (AI), such as ImageNET, SuperGLUE, etc. Since the rise of crowdsourcing in early 2000s, the AI community has been studying its computational, system design, and data-centric aspects at various angles. We welcome the studies on developing and enhancing of crowdworker- centric tools, that offer task matching, requester assessment, instruction validation, among other topics. We are also interested in exploring methods that leverage the integration of crowdworkers to improve the recognition and performance of the machine learning models. Thus, we invite studies that focus on shipping active learning techniques, methods for joint learning from noisy data and from crowds, novel approaches for crowd-computer interaction, repetitive task automation, and role separation between humans and machines. Moreover, we invite works on designing and applying such techniques in various domains, including e-commerce and medicine.Close abstract A. Alorwu, S. Savage, N. van Berkel, D. Ustalov, A. Drutsa, J. Oppenlaender, O. Bates, D. Hettiachchi, U. Gadiraju, J. Goncalves, S. Hosio, "REGROW: Reimagining Global Crowdsourcing for Better Human-AI Collaboration", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’22 Workshop), 2022, 1–7.
Abstract: Crowdworkers silently enable much of today’s AI-based products, with several online platforms offering a myriad of data labelling and content moderation tasks through convenient labour marketplaces. The HCI community has been increasingly interested in investigating the worker-centric issues inherent in the current model and seeking for potential improvements that could be implemented in the future. This workshop explores how a reimagined perspective on crowdsourcing platforms could provide a more equitable, fair, and rewarding experience. This includes not only the workers but also the platforms, who could benefit e.g. from better processes for worker onboarding, skills-development, and growth. We invite visionary takes in various formats on this topic to spread awareness of worker-centric research and developments to the CHI community. As a result of interactive ideation work in the workshop, we articulate a future direction roadmap for research centred around crowdsourcing platforms. Finally, as a specific interest area, the workshop seeks to study crowdwork from the context of the Global South, which has been arising as an important but critically understudied crowdsourcing market in recent years.Close abstract P. Wintersberger, N. van Berkel, N. Fereydooni, B. Tag, E. L. Glassman, D. Buschek, A. Blandford, F. Michahelles, "Designing for Continuous Interaction with Artificial Intelligence Systems", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’22 SIG), 2022, 1–4.
Abstract: The increasing capabilities of Artificial Intelligence enable the support of users in a continuously growing number of applications. Current systems typically dictate that interaction between user input and AI output unfolds in discrete steps, as is the case with, for example, conversational agents. Novel scenarios require AI systems to adapt and respond to continuous user input, e.g., image-guided surgery and AI-supported text entry. In and across these applications, AI systems need to support more varied and dynamic interactions in which users and AI interact continuously and in parallel. Current methods and guidelines are often inadequate and sometimes even detrimental to user needs when considering continuous usage scenarios. Realizing a continuous interaction between users and AI requires a substantial change in perspective when designing Human-AI systems. In this SIG, we support the exchange of cutting-edge research contributing to a better understanding and improved methods and tools to design continuous Human-AI interaction.Close abstract N. van Berkel, E. Schneiders and R. M. Jacobsen, "Addressing Repetition in Crowdsourcing: A Concept for Fast-Form Entry", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’22 EA), 2022, 1–6.
R. M. Jacobsen, S. S. Johansen, N. van Berkel, M. B. Skov, J. Kjeldskov, "In the Zone! – Controlling and Visualising Sound Zones", in Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’22 Interactivity), 2022, 1–4.
Abstract: Sound zones enable multiple simultaneous sound experiences in the same physical room without interference. In this paper, we present an interactive sound zone setup that can produce two sound zones within a confined space. Through a tangible remote controller, users can change the volume, size, and position of these sound zones. In addition, we have built a custom visualisation display that provides real-time feedback of the sound zones to support users’ understanding. Sound zone systems pose novel challenges for the HCI community, including how users may understand and interact with sound zones. Our setup offers a concrete solution into investigating these challenges.Close abstract N. K. Kollerup, M. B. Skov and N. van Berkel, "Using Signals to Support Trust Building in Clinical Human-AI Collaboration", in Adjunct Proceedings of the EUSSET ECSCW European Conference on Computer-Supported Cooperative Work (ECSCW’22 EA), 2022, 4.
Abstract: Artificial Intelligence (AI) has the technological potential to transform healthcare by assisting medical personal in their everyday workflow. For successful collaboration and adoption of AI technology, end-users need to trust the AI system. In this paper we outline the use of Relational Signalling Theory, an established theory on Human-Human trust building, as a conceptual lens for designing trust-building signals in Human-AI interaction. We argue that the use of a theoretical foundation in the design and evaluation of interactions supports the development of Human-Centered AI in healthcare.Close abstract E. Schneiders, N. van Berkel and M. B. Skov, "Hybrid Work for Industrial Workers: Challenges and Opportunities in using Collaborative Robots", in Adjunct Proceedings of the 12th Nordic Conference on Human-Computer Interaction (NordiCHI’22 EA), 2022, 1–4.
Abstract: The COVID-19 pandemic has drastically impacted how work is conducted, shifting many jobs to a hybrid nature with an emphasis on work-from-home. This shift has, however, not affected all job types equally. In this position paper, we argue that the advancement of collaborative robots in the industrial setting presents a unique and underexplored opportunity for robot-supported hybrid work in the industrial setting. We present five challenges that act as contributing factors that stifle access to hybrid work opportunities in the industrial context. These point to (i) the need for spacial awareness of both the robot and its surroundings, (ii) the, at times, need for physical presence for breakdown intervention and recovery, (iii) the need for contextual awareness, (iv) the need for additional employee training, and (v) a clear interface to map the varying degrees of freedom to a remote interface. We present future research opportunities with the potential to address some of the presented challenges.Close abstract M. Oanh Hoang, N. van Berkel, M. B. Skov, T. Merritt, "Challenges Arising in a Multi-Drone System for Search and Rescue", in Adjunct Proceedings of the 12th Nordic Conference on Human-Computer Interaction (NordiCHI’22 EA), 2022, 1–5.
Abstract: Unmanned aerial vehicles (UAVs) have been shown to effectively support search and rescue (SAR) operations, usually through manual control of each UAV. Research has started to move towards multi-drone systems with increasing levels of autonomy to support faster and more challenging SAR operations. However, how such a system should be implemented into the SAR procedures remains an open challenge. We seek to thoroughly understand the issues that arise with drone usage in SAR and how semi-autonomous multi-drone systems can help alleviate those. We conducted a pilot study with emergency services SAR pilots and our multi-drone control prototype. Initial findings have given insights into useful features and potential pitfalls in designing a multi-drone system. The results inform our continued work toward more refined prototypes that will serve as a platform for search and rescue operations.Close abstract A. L. Christensen, K. A. R. Grøntved, M. Oanh Hoang, N. van Berkel, M. Skov, A. Scovill, G. Edwards, K. R. Geipel, L. Dalgaard, U. P. S. Lundquist, I. Constantiou, C. Lehrer, T. Merritt, "The HERD Project: Human-Multi-Robot Interaction in Search & Rescue and in Farming", in Adjunct Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’22 EA), 2022, 1–4.
Abstract: Large-scale multi-robot systems have numerous potential real-world applications. It is, however, still unclear how a human operator can effectively engage and control a system composed of multiple autonomous robots, especially in unstructured and outdoor environments. This paper reports on ongoing work in the project HERD — Human-AI Collaboration: Engaging and Controlling Swarms of Robots and Drones, in which we focus on two concrete use cases from industrial partners, namely farming and search & rescue. One of the industrial partners, Agro Intelligence ApS, currently sells autonomous farming robots, while the other, Robotto ApS, develops autonomous drone-based monitoring solutions for emergency responders. Both partners aim to scale their technologies to multi-robot/multi-drone operations. In this paper, we present the two use cases, their differences and similarities, challenges and preliminary results.Close abstract F. Bemmann, R. Schoedel, N. van Berkel, D. Buschek, "Chatbots for Experience Sampling – Initial Opportunities and Challenges", in Adjunct Proceedings of the ACM Intelligent User Interface Conference (IUI’21 EA), 2021, CUI@IUI: Theoretical and Methodological Challenges in Intelligent Conversational User Interface Interactions, 1–7.
Abstract: The Experience Sampling Method is a widely used methodology for the collection of self-report data. These self-reports are typically collected through bespoke mobile applications or text messages. Recently, an increasing number of social messaging applications have introduced chatbots – automated services that operate inside existing chat applications. In this paper, we present an initial study on the use of chatbots for self-report studies. Furthermore, we outline three use cases in which the use of chatbots enables new research opportunities, namely personalised and empathic chatbots, psychometric instrument construction, and group-based assessments. We conclude with an overview of the opportunities and challenges that chatbots offer to researchers employing the Experience Sampling Method.Close abstract V. Paananen, P. Markkanen, J. Oppenlaender, L. Hang Lee, H. Akmal, A. F. gen. Schieck, J. Dunham, K. Papangelis, N. Lalone, N. van Berkel, J. Goncalves, S. Hosio, "2VT: Visions, Technologies, and Visions of Technologies for Understanding Human Scale Spaces", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21 EA), 2021, 1–9.
Abstract: Spatial experience is an important subject in various fields, and in HCI it has been mostly investigated in the urban scale. Research on human scale spaces has focused mostly on the personal meaning or aesthetic and embodied experiences in the space. Further, spatial experience is increasingly topical in envisioning how to build and interact with technologies in our everyday lived environments, particularly in so-called smart cities. This workshop brings researchers and practitioners from diverse fields to collaboratively discover new ways to understand and capture human scale spatial experience and envision its implications to future technological and creative developments in our habitats. Using a speculative design approach, we sketch concrete solutions that could help to better capture critical features of human scale spaces and allow for unique possibilities for aspects such as urban play. As a result, we hope to contribute a road map for future HCI research on human scale spatial experience and its application.Close abstract D. Bennett, F. Feng, T. Froese, A. Dix, P. Eslambolchilar, V. Kostakos, S. Lerique, N. van Berkel, "Emergent Interaction: Complexity, Dynamics, and Enaction in HCI", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21 EA), 2021, 1–7.
Abstract: We propose a workshop on methods and theories for dealing with complex dynamical systems, and their application in HCI. Such methods are increasingly relevant across a wide range of disciplines which focus on human behaviour, applied to understand the role of context and interactions in the behaviour of individuals and groups, and how they unfold over time. Traditional approaches to quantifying and modelling behaviour in HCI have tended to focus primarily on individuals and components. Complexity methods shift the focus onto interactions between components, and the emergence of behaviour from complex networks of interactions, as for example in Enactivist approaches to cognitive science. While we believe that complexity methods can be highly informative to HCI researchers, uptake in the community remains low due to widespread unfamiliarity. This one-day workshop will introduce, support, and encourage the development and adoption of complexity methods within HCI. Reflecting the multidisciplinary mix within complexity science, we will draw on examples of complexity-oriented theories and methods from a range of disciplines, including Control-Theory, Social Science, and Cognitive Science. Attendees will engage in group discussions and a Q&A with a panel, and a discussion group will be set up ahead of time to encourage exploratory conversations. In this way, diverse backgrounds can be brought together, matched, and inform one another.Close abstract S. Wijenayake, N. van Berkel and J. Goncalves, "Bots for Research: Minimising the Experimenter Effect", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’20 EA), 2020, 1–8.
Abstract: Experimenter-induced influences can trigger biased responses from research participants. We evaluate how digital bots can be used as an alternative research tool to mitigate these biases, as based on existing literature. We note that the conversational interactivity provided by bots can significantly reduce biased responses and satisficing behaviour, while simultaneously enhancing disclosure and facilitating scalability. Bots can also build rapport with participants and explain tasks at hand as well as a human experimenter, with the added benefit of anonymity. However, bots often follow a predetermined script when conversing and therefore may not be able to handle complex and unstructured conversations, which could frustrate users. Studies also imply that bots with human-like features may induce experimenter effects as similar to humans. We conclude with a discussion on how bots could be designed for optimal utilisation in research.Close abstract N. van Berkel, A. Exler, M. Gjoreski, T. Kolenik, T. Okoshi, V. Pejovic, A. Visuri, A. Voit, "UbiTtention 2020: 5th International Workshop on Smart & Ambient Notification and Attention Management", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’20 Adj.), 2020, 431–435.
Abstract: Users are increasingly confronted with a tremendous amount of information proactively and without explicit consent through notifications from a variety of applications and services. This information load is increased due to the ubiquity of end-user (mobile) devices. Novel computing paradigms such as IoT and smart cities may further overload end-users, despite the clear indication from literature that human attention is limited. To counter this challenge, “attention management”, including attention representation, sensing, prediction, analysis, personalization, and adaptive behavior is needed in our computing systems. Following the successful UbiTtention workshop series as organised from 2016 on-wards, the UbiTtention 2020 workshop brings together researchers and practitioners from academia and industry to explore the management of human attention and notifications across versatile devices and contexts. UbiTtention presents and elicits research to overcome information overload and overchoice – tailoring device or application behavior to user needs.Close abstract N. van Berkel, S. Hosio, B. Tag, J. Goncalves, "Capturing Contextual Morality: Applying Game Theory on Smartphones", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’19 Adj.), 2019, 888–892.
Abstract: In order to build more fair Artificial Intelligence applications, a thorough understanding of human morality is required. Given the variable nature of human moral values, AI algorithms will have to adjust their behaviour based on the moral values of its users in order to align with end user expectations. Quantifying human moral values is, however, a challenging task which cannot easily be completed using e.g. surveys. In order to address this problem, we propose the use of game theory in longitudinal mobile sensing deployments. Game theory has long been used in disciplines such as Economics to quantify human preferences by asking participants to choose between a set of hypothetical options and outcomes. The behaviour observed in these games, combined with the use of mobile sensors, enables researchers to obtain unique insights into the effect of context on participant convictions.Close abstract A. Visuri, N. van Berkel, "Attention Computing: Overview of Mobile Sensing Applied to Measuring Attention", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’19 Adj.), 2019.
Abstract: The measurement of participant attention is a frequent by-product of mobile sensing-based studies, which typically focus on user interruptibility or the effectiveness of notification deliveries. We note that, despite the popularity of interruptibility research within our discipline, research focused on attention is surprisingly scarce. This omission may be due to (a combination of) methodological, technological, or disciplinary constraints. In this paper, we argue how attention levels can be effectively measured with existing technologies and methodologies by adapting continuous measurements of attention fluctuations. Many clinically researched technologies, as well as sensing-based analysis methods, could be leveraged for this purpose. This paper invites co-researchers to assess the use of novel ways to measure attention in their future endeavours.Close abstract N. van Berkel, J. Goncalves, B. Tag, S. Hosio, "Contextual Morality for Human-Centered Machine Learning", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19 EA), 2019, 1–4.
Abstract: Big data and the increased use of Artificial Intelligence (AI) and Machine Learning (ML) have opened many new opportunities for continuous decision-support by autonomous systems. While initial work has begun to explore how human morality can inform the decision-making of future AI’s [4], these approaches consider human morality as a static concept. We note that human morality and decision-making is affected not only by cultures and personalities but is to a large degree affected by an individual’s context. In order to align with the moral judgements of their users, future ML applications should adjust their decision-making accordingly based on user context. In this work, we discuss our critical take on the importance of contextual morality for AI and identify opportunities for future work.Close abstract D. Hettiachchi, N. van Berkel, T. Dingler, F. Allison, V. Kostakos, J. Goncalves, "Enabling Creative Crowd Work through Smart Speakers", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19 EA), 2019, 1–5.
Abstract: Digital voice assistants or smart speakers have rapidly changed the landscape of voice user interfaces over the past few years. In this paper we discuss how we could utilise the affordances of these devices to create a novel crowdsourcing platform that deliver crowd tasks through voice with particular focus on creative tasks. We describe the potential benefits and challenges of using this technology for these purposes, and outline our future work in this research area.Close abstract N. van Berkel, M. Budde, S. Wijenayake, J. Goncalves, "Improving Accuracy in Mobile Human Contributions: An Overview", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’18 Adj.), 2018, 594–599.
Abstract: The collection of human contributions through mobile devices is increasingly common across a range of methodologies. However, possible quality issues of these contributions are often overlooked. As the quality of human data has a direct impact on study reliability, more should be done to improve the accuracy of these contributions. We identify and categorise solutions aimed at increasing the accuracy of contributions prior, during, and following data collection. Our categorisation assists in the positioning of future work in this area and fosters the usage of cross-methodological practises.Close abstract W. Jiang, G. Marini, N. van Berkel, Z. Sarsenbayeva, X. He, T. Dingler, Y. Kawahara, V. Kostakos, "A Mobile Scanner for Probing Liquid Samples in Everyday Settings", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’18 Adj.), 2018, 1172–1177.
Abstract: Our work investigates the use of a Near InfraRed Spectroscopy scanner for the identification of liquids. While previous work has shown promising results for the identification of solid objects, identifying liquids poses additional challenges. These challenges include light scattering and low reflectance caused by the transparency of liquids, which interfere with the infrared measurement. We develop a prototype solution consisting of a 3D printed clamp that attaches to a tube, such that it blocks ambient light from interfering. Our preliminary results indicate that our prototype works, and we demonstrate this by measuring sugar levels in a liquid solution.Close abstract A. Visuri, K. Opoku Asare, E. Kuosmanen, Y. Nishiyama, D. Ferreira, Z. Sarsenbayeva, J. Goncalves, N. van Berkel, G. Wadley, V. Kostakos, S. Clinch, O. Matthews, S. Harper, A. Jenkins, S. Snow, m. c. schraefel, "Ubiquitous Mobile Sensing: Behaviour, Mood, and Environment", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’18 Adj.), 2018, 1140–1143.
Abstract: Our world is increasingly interconnected via a wide variety of computers, IoT, wearable and mobile devices. The information provided collectively through these devices offers insightful information on our everyday lives, daily patterns, mood, behaviour, and surrounding environment. Our workshop brings together researchers interested in collecting and augmenting context to understand device specific behaviour and routines, human behaviour and mood, and changes in the environment. The outcomes of this workshop are new tools, methodologies, and potential collaborations for sensing the outlying world as well as ourselves.Close abstract N. van Berkel, S. Hosio, J. Goncalves, K. Wac, V. Kostakos, A. L. Cox, "MHC’18 – Workshop on Mobile Human Contributions: Opportunities and Challenges", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’18 Adj.), 2018, 590–593.
Abstract: Ubicomp/HCI researchers are increasingly using smartphones to collect human-labelled data ‘in the wild’. While this allows for the collection of a wide range of interesting data in authentic settings and surroundings, humans are notoriously inconsistent in the quality of their contributions. Improving the quality of data collected with mobile devices is a largely unexplored, but highly relevant field. The primary objective of this workshop is to share insights, ideas, and discoveries on the quality of mobile human contributions. The work presented in the International Workshop on Mobile Human Contributions (MHC ’18) explores methods, tools, and novel approaches towards increasing the reliability of human data submissions with mobile devices.Close abstract N. van Berkel, S. Hosio, J. Goncalves, V. Kostakos, "Informed Diet Selection: Increasing Food Literacy through Crowdsourcing", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’18 EA), 2018, International Workshop on Designing Recipes for Digital Food Futures, 1–2.
Abstract: The Internet offers plenty of options for those who want to lose weight. Choosing among the practically unlimited number of weight loss diets, exercises, and pills, advertised as borderline magical, is however challenging. We present The Diet Explorer, a crowd-powered, knowledge base that can be queried in real-time to discover weight loss diets that best match personal needs. Our long-term goals are to help people in making better-informed dieting decisions and ultimately reach more satisfactory diet outcomes.Close abstract S. Klakegg, N. van Berkel, A. Visuri, H. Huttunen, S. Hosio, C. Luo, J. Goncalves, D. Ferreira, "Designing a Context-Aware Assistive Infrastructure for Elderly Care", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’17 Adj.), 2017, 563–568.
Abstract: We present an assistive healthcare platform, CARE, which aims to provide daily support for elderly caregivers with context-aware, unobtrusive, and actionable information. This information is collected through a plethora of IoT sensors installed strategically at an elderly care centre and is accessed through an Android tablet application. The application’s goal is to empower nurses with a better understanding of elderly needs and ultimately, improve the care service. We investigate how IoT devices and sensors can enable a pervasive healthcare system, and discuss a wide-range of important parameters for integration of elderly care practices.Close abstract Z. Sarsenbayeva, D. Ferreira, N. van Berkel, C. Luo, M. Vaisanen, V. Kostakos, J. Goncalves, "Vision-Based Happiness Inference: A Feasibility Case-Study", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’17 Adj.), 2017, 494–499.
Abstract: To humanize interaction between users and computers, one needs the ability to infer the users’ mood. One approach is to use a vision-based approach. We quantify the ‘preview effect’ bias in visual mood assessment. We demonstrate that automated tools which infer user mood from photographs or video may be affected by the presentation methodology used while performing image capture. Specifically, we demonstrate that showing a “preview” of oneself, i.e., a mirror, increases the accuracy of the visual mood inference algorithms present in Google’s Mobile Vision API. Our findings show that studies that incorporate visual mood assessment should include “preview” images to reduce bias and increase the reliability of vision-based happiness inference.Close abstract N. van Berkel, S. Kopytin, S. Hosio, J. Malmberg, H. Järvenoja, V. Kostakos, "Measuring Group Dynamics in an Elementary School Setting Using Mobile Devices", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’16 Adj.), 2016, 652–656.
Abstract: Mobile instrumentation provides researchers and professionals the opportunity to collect data on several aspects of human life. In this paper we discuss our initial experiences on collecting data via mobile instrumentation in an elementary school. We augmented a classroom with mobile phones and Bluetooth beacons to capture student experiences as well as their relative distance to each other during a collaborative group project. We describe the study, and present lessons learned when instrumenting such a unique school setting with young participants.Close abstract S. Hosio, J. Goncalves, N. van Berkel, S. Klakegg, "Crowdsourcing Situated & Subjective Knowledge for Decision Support", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’16 Adj.), 2016, 1478–1483.
Abstract: In this paper we present a study on crowdsourcing subjective knowledge. We introduce a mobile app that was built for this purpose, and compare results from two datasets collected using the app. One dataset was collected during a workshop and the other one during a one-week long field trial. We present interview findings on mobile knowledge collection. Further, we discuss the types of information that should optimally be collected on the go, and show how our data analysis supports the qualitative findings. This work directly continues our earlier efforts on creating a platform that encapsulates wisdom of the crowd for decision support.Close abstract N. van Berkel, S. Hosio, T. Durkee, V. Carli, D. Wasserman, V. Kostakos, "Providing Patient Context to Mental Health Professionals Using Mobile Applications", in Adjunct Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16 EA), 2016, International Workshop on Computing and Mental Health, 1–4.
Abstract: The quantified-self movement entails self-tracking of physical activity, often using wearable devices and mobile applications. In parallel, mobile applications focusing on mental health are increasingly popular, and they often rely on active user input to track the user progress and to deliver feedback and motivation. In this paper we discuss the potential benefits of bridging these two distinct yet highly relevant application domains. We argue for the benefits of combining explicit (user-provided) and implicit (devicecollected) data sources in the context of mental health care. We argue that this combination allows for improved methods to observe patients’ lives, and thus provide a more in-depth overview of their progress. This may enable mental health professionals to establish more personalised and adaptive care plans.Close abstract N. van Berkel, C. Luo, D. Ferreira, J. Goncalves, V. Kostakos, "The Curse of Quantified-Self: An Endless Quest for Answers", in Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’15 Adj.), 2015, 973–978.
Abstract: Quantified Selfers are individuals that take a proactive stance to collect and act upon their personal data. However, these endeavours towards a better insight into one’s life often do not last long. An important challenge for QS is sustaining data collection over a long period of time (i.e., months, years, decades). In this paper we discuss the drivers, needs and concerns of longitudinal QS-data collection. We argue that to support longitudinal QS various obstacles have to be overcome, including i) integration and sharing of data between a variety of (new) devices, ii) incorporating human input for psychological data collection and iii) providing answers to the questions people really have.Close abstract