Session: 16-01-01: Government Agency Student Poster Competition

Paper Number: 150284

150284 - Predicting Creativity Ratings on Conceptual Design Ideas Using Eye-Tracking Data 

Introduction: While the Consensual Assessment Technique (CAT) has been widely used to assess the creativity of design ideas, it remains unclear whether it scales to ideas with broader representations and which cognitive factors are related. This study aims to address the following research questions: (1) Does the CAT approach provide robust creativity ratings across the different idea representations? (2) Can creativity rating be predicted using eye-tracking data, and which factors are associated with it? (3) What factors differentiate novice raters from expert raters?

Methods: We randomly selected ten conceptual design ideas for milk frothers from a larger public dataset. We then identified three datasets, each including ten milk frother design ideas in one of three representations: sketch, text description, or both. A total of 14 novice raters were randomly assigned into three groups (i.e., A, B, and C), and each group assessed ideas across all three modalities without overlap. We asked raters to read the instructions, perform the 9-point calibration of the eye tracker, and assess ideas using the CAT approach through a 6-point Likert scale. The study utilized a Tobii Pro Fusion, a screen-based eye tracker that captures gaze data at 60 to 120Hz with an accuracy of 0.3 degrees. The rating sheet and design ideas were presented on a 27-inch LCD monitor with a 2560x1440-pixel screen display. We collected raters' fixation points represented in XY coordinates on the design areas. We used the HDBSCAN algorithm to cluster raters' fixation points into areas of interest (AOIs) and defined five variables that estimated values for each AOI: the percentage of fixations, duration of fixation, time to first visit, number of revisits, and maximum pupil diameter. For statistical analysis, we first evaluated the average creativity rating for each idea modality. We developed three generalized estimating equation (GEE) machine-learning models for each idea representation to predict the creativity ratings. The dependent variable was the creativity ratings, and the independent variables included the five variables. Similarly, we developed a GEE model to predict the validity of creativity ratings to understand factors that differentiate novices from expert raters.  The dependent variable was the difference in average creativity ratings between experts and novices, and the independent variables included the five variables. We obtained expert creativity ratings from a previous study—recruiting two design experts to assess the original milk frother ideas using the CAT approach on a 6-point Likert scale.

Preliminary results: There was a difference in the average creativity ratings of ideas across different modalities: 3.61 for ideas in both sketch and text description, 3.21 for ideas in sketches, and 2.98 for ideas in text descriptions. In the GEE regression using ideas in both sketch and text description, the percentage of fixations was positively associated with creativity rating, while the number of revisits was negatively associated with it. In an analysis of ideas in the sketch, a lower percentage of creativity rating and larger maximum pupil diameter were associated with creativity rating. In an analysis of ideas in a text description, creativity rating was associated with fewer revisits and a larger maximum pupil diameter. Additionally, in the GEE regression analysis to predict the validity of creativity rating, the percentage of fixations and the number of revisits were associated with the difference in the average creativity ratings between novice and expert raters.

Conclusion: We found that the percentage of fixations, number of revisits, and maximum pupil diameters were associated with creativity ratings. The increase in the percentage of fixation points and decrease in revisits to AOIs by novice raters were associated with creativity ratings that closely matched those of the expert raters.

Presenting Author: Duk Hee Ka The Pennsylvania State University

Presenting Author Biography: I am a PhD student in the Department of Industrial and Manufacturing Engineering at Penn State, where I also received my M.S. degree in 2024. I received my B.S. degree in the Department of Industrial and System Engineering from Dongguk University in 2020. My research interests include data analysis, machine learning, and computational cognitive modeling. I have experience in data analysis in healthcare, particularly in understanding cancer drug development.

Authors:

Duk Hee Ka The Pennsylvania State University
Scarlett Miller The Pennsylvania State University
Farnaz Tehranchi The Pennsylvania State University