«by Swapnaa Jayaraman A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Industrial and ...»
Supporting Monitoring and Interruption Management in Complex Domains
through Graded Multimodal Notifications
A dissertation submitted in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
(Industrial and Operations Engineering)
in the University of Michigan
Professor Nadine B. Sarter, Chair
Professor Yili Liu
Professor Bernard J. Martin
Professor Colleen M. Seifert
© Swapnaa Jayaraman _______________________________
Dedication To appamma ii Acknowledgements My very first thanks go to appa and amma who have vicariously earned their doctorate! I thank them for being the most amazing parents on earth, for trusting my choices, for encouraging me to do anything (everything) I wished to, and for taking sheer pleasure in watching me do it all. Your unending love and support makes all the difference in the world.
I must thank my advisor Nadine Sarter for her patience and faith in me and for seeing my Ph.D. journey through to the very end. Nadine has been my greatest critic as well as my source of motivation. Her help in shaping my research ideas, her detailed feedback and her keen insights on my work have been invaluable. Her perseverance and hard work will continue to inspire me throughout my research career.
I would also like to thank my committee members who are all wonderful researchers and inspiring educators. I have been fortunate to be their graduate student even before they agreed to mentor me. I am grateful to Bernard Martin for being one of my first research inspirations and for persuading me to consider a career in academia. The delightful combination of his peculiar sense of humor and his passion for research and teaching and has helped me tide over some trying moments in school. I am also grateful to Yili Liu for enthusiastically sharing his worldly wisdom and experiences both concerning the subject iii of ergonomics and otherwise. I can only hope to someday inspire and captivate students the way he does. I also want to thank Colleen Seifert for her unrelenting support. She always made time to discuss my research over a cup of coffee anytime I asked. Her calming smile and strong presence have been a source of constant encouragement.
My graduate training and dissertation could not have been completed without the support of my funding sources. I am thankful to Thomas Finholt from the School of Information for trusting my abilities and providing me with my very first source of complete funding – a Graduate Student Research Assistant position – which helped me launch into my research path. I want to thank my department, Industrial and Operations Engineering, for supporting my research work through their generous fellowship. Funding acknowledgments would not be complete without thanking the National Science Foundation and the National Aeronautics and Space Administration for their project grants that gave me immense research latitude. I would also like to thank Rackham Graduate School, Ginsberg Center and Wallenberg Foundation for their generous fellowships which truly took my learning experience well beyond the classroom. Special thanks to the awesome individuals from Rackham, particularly to Jill McDonough and John Godfrey, who work so hard to keep up our celebrated “Michigan difference”.
I must acknowledge the support I have received from my department and my lab. My heart-felt thanks to Tina Blay for the warm teas, Michigan apples, aspirins and hugs that never failed to ward off the basement lab blues. I want to thank Tom Ferris for being an incredibly accommodating and supportive research partner, and for extending a helping hand whenever it was needed. His ability to be a dream graduate student while raising
Hameed for being an innovative lab programmer while also being a great research partner. With his technical and research skills, the sky is really the limit. Our other lab programmers Hari Molabanti, Joseph Phillips and Clif Reeder have been amazing in their abilities to quickly grasp our research requirements and create impressive simulations that resulted in several successful experiments. Many thanks in particular to Joe who graciously worked extra nights on my project to help me meet deadlines.
I must also thank some outstanding women from our lab who have contributed to my research in various ways: Wan-Ting Su for being a great friend and an invaluable research colleague, even after she moved to Taiwan; Melissa Ballard and Ashley Issa for their bright scholastic inputs; and Stephanie Widrig for being the best undergraduate lab partner one could ask for, and for repeatedly outdoing all expectations. My acknowledgements would not be complete without the mention of Huiyang Li, one of the most sincere and hardworking friends I am proud to say I have, and the rest of the gang Richard Ho, Sara Lu, and Julie Prinet – for enlivening our existence in the basement with a constant supply of food, warmth, and cheer!
Two faculty members outside my department have been strong sources of support. I take immense pleasure in expressing my gratitude to my true guru, Michael Gordon, who has been my friend, philosopher, and guide. He regularly reassures me that one can pursue several interests and also combine them in a meaningful and socially relevant way. He lives his life as an example and constantly inspires me to follow my heart. I also want to thank Lalit Patil for his ready support and insightful advice every time they were sought.
my side all the way. First thanks go to these true Wolverines, my very first roommates in Ann arbor - my sister Aneetha Jayaraman, my brother-in-law Vikalpa Jetly, and my brand new husband Shyam Venkatesan – who were there to celebrate every little milestone from my admission to my thesis defense over countless cakes, dinners and drinks. Thanks to my beloved brothers Samir Parekh, Deepak Haran, Arvind Haran, and Vasuki Seshadri for their loving long-distance encouragement. Thanks also to my grandparents, aunts, uncles, cousins and new in-laws for all the phone calls and emails bearing their best wishes at every turn. Every one of those wishes has touched my heart and inspired me to make them proud.
I also want to thank my dear friends, who were not only there when I needed them emotionally, but were also willing scapegoats for the incessant pilot experiments I conducted in my lab: Janani Mukundan for her unshakeable love for research and faith in me, Manickam Narayanan for his immense patience and unconditional love, Gaurav Rai for being a great editor and for constantly reminding me of the value of my research, Arvind Rao for his kindness and keen intellect which he shares so freely, Mekhala Raghavan for setting a great example as a researcher with a social conscience, Dinesh Krithivasan for his shining wisdom and clarity of thought, and the three stooges Sivakumar Natarajan, Saradwata Sarkar, and Balachandran Radhakrishnan for their constant presence, thoughtfulness, and all the laughter they provided during gloomy times. Special thanks to Arvind for giving me intellectually inspiring pep-talks when I
wonderful friend and foster-parent one could ever ask for. I cannot count the number of days he has brought me food when I was working in hunger, times he has driven me to school when it was too dark or cold, and hours he has spent caring for me when I was physically or emotionally low. I give my heartfelt gratitude to him and his wife Aarthi for opening their home to me and Shyam when we needed it most. A very special mention goes to Maithreyi Gopalan Sandilya for putting up with me and my crazy lifestyle over the past seven years and for being the most special friend one could ever ask for!
For the one person without whom all this literally would not have been possible, I am at a loss of words to fully express my gratitude. Shyam has done for me all the wonderful things that the people mentioned above have done, and so much more. In all these years, not a single day has passed by without Shyam giving me his intellectual, emotional, sometimes unspoken, and perpetually dependable support. I could not have hoped to go through graduate school, or my life, with a better partner.
And now for that delectable phrase: I’m all done!
List of Figures
List of Tables
List of Appendices
Chapter 1 Introduction
1.1 Attention and Interruption Management and the Alarm Problem
1.1.1 Attention and interruption management
1.1.2 Preattentive reference and the alarm problem
1.2 Multimodal Information Presentation
1.2.1 Multiple Resource Theory as the Basis for Multimodal Information Presentation
1.2.2 Peripheral Vision
1.2.4 Graded Notifications in Support of Interruption Management
Chapter 2 Guiding Visual Attention by Exploiting Crossmodal Spatial Links with the Sense of Touch: An Application in Air Traffic Control
2.2.1 Detection rates for shape change events
2.2.2 Response times to shape changes
2.2.3 ATC task performance
Chapter 3 Informative Peripheral Visual and Tactile Cues in Support of Interruption Management..
3.1.5 Data Collection
3.1.6 Experimental Design
3.2.1 Cue Detection Rate
3.2.3 Task-Switching Behavior
3.2.4 Performance on Arithmetic Task
Chapter 4 Design of Multimodal Graded Notifications
4.1 Basic design requirements
4.2 Notification properties
4.2.1 Arrangement of cues
4.2.2 Cue salience
4.2.3 Cue patterns
Chapter 5 Evaluation of Multimodal Graded Notifications
5.1 Purpose, goals and expectations
5.2 Identification of Promising Tasks/Events
5.5 ATC scenario: tasks
5.5.1 Primary tasks
5.5.2 Interruption tasks
5.7 Experimental design and analysis
Chapter 6 Evaluation of Multimodal Graded Notifications
6.1 Results and discussion
6.1.2 Goal 3 : Improve performance of interruption tasks under cognitive load 116 6.1.3 Goal 2: Improve decision making about attending to interruptions......... 127 6.1.4 Goal 4: Maintain or improve performance on primary tasks
6.1.5 Goal 5: Improve time sharing performance and management of primary and interrupting tasks
6.1.6 Participant experiences
6.2 Summary of findings
Chapter 7 General Discussion
Chapter 8 Conclusion
Figure 1-1: The process of interruption management (Sarter, 2005)
Figure 1-2: Potential outcomes of interruption management during simultaneous interruptions
Figure 1-3: Structure of multiple resources (Wickens, 2000)
Figure 1-4: View of an air traffic control station showing the various sources of information
Figure 2-1: Detection rate for shape change
Figure 2-2: Detection rates for shape change events classified by event types................ 31 Figure 2-3: Response time for shape change events
Figure 3-1: Design and location of baseline visual and peripheral visual interruption cues as well as arithmetic task.
Figure 3-2: Detection rate as a function of cue modality.
Figure 3-3: Overall interpretation accuracy for each parameter encoded in informative cues.
Figure 3-4: Task-switching rates for each parameter level (correctly interpreted informative cues only)
Figure 3-5: Accuracy on arithmetic task for cued and uncued trials as a function of cue type.
Size and brightness of cues are exaggerated for better visibility.
Figure 4-2: Vertical and horizontal visual angles at which peripheral visual cues were presented
Figure 4-3: Arrangement of peripheral visual cues within the binocular field of vision.. 62 Figure 4-4: Placement of tactors on various test locations
Figure 4-5: Mean detection rates and standard error for various levels of salience......... 69 Figure 4-6: Accuracy of detecting interruptions (mean and SE) in uncued and cued trials
Figure 4-7: Performance in main task (mean and SE) in uncued and cued trials............. 72 Figure 4-8: Median values for crossmodal matching of peripheral-visual and tactile cues
Figure 4-9: Pulse rates used in gentle, moderate and severe cues
Figure 4-10: Cue stages and cue durations within a single notification
Figure 5-1: Desktop ATC simulation. Panel 1: Main radar screen with runways (center), speed zones (concentric circles), landing wedges (diagonal lines), way points, airplanes with their data blocks. Panel 2: Wind wane. Panel 3: Message log. Panel 4: Active airplanes. Panel 5: Interruption commands and “transmit” button.
Figure 5-2: Accurate detection rates (mean and SE) under low, medium and high workload
Figure 5-3: Accuracy of detection (mean and SE) for every cue type
Figure 5-4: Left: Participant seated at ATC simulation. Right: ATC screen as seen with ambient lights turned off
Figure 5-6: Left: Screenshot of airplane with heading (horizontal line), and data block displaying speed (top, yellow) and altitude (bottom, white). Right: Enlarged and retouched to show heading and data block
Figure 5-7: Drop-down menus for (a) rectifying airspeed and (b) heading deviations.... 89 Figure 5-8: Potential participant responses and award scheme for minor altitude deviations
Figure 5-9: Additional task (detecting 3 ‘+’ symbols hidden among ‘x’ symbols) in case of moderate and severe altitude deviations