Bridging Vision, Language, and Gaze for Trustworthy Foundation Models

Abstract

Foundation models have reshaped the development of artificial intelligence (AI) by introducing a paradigm in which a single pretrained system can generalize across a wide spectrum of language and multimodal tasks. Built upon large-scale data and representation learning, these models reduce the need for task-specific engineering and enable reusable semantic and perceptual knowledge. This shift from specialized models to general-purpose learning architectures has significantly expanded the scope and applicability of AI technologies. As foundation models transition from controlled benchmarks to real-world deployment, new demands arise that extend beyond raw capability. Systems are increasingly expected to behave in ways that are reliable, interpretable, and aligned with human expectations, particularly in settings involving multimodal reasoning and interaction. These requirements expose limitations that are not visible in traditional performance evaluations and motivate a shift in research focus from scaling performance to ensuring trustworthy behavior.

In this dissertation, trustworthiness is analyzed as a property that emerges from the structural coupling of groundedness, alignment stability, faithfulness, and controllability. These dimensions correspond to interdependent stages of the trustworthiness pipeline, encompassing how multimodal signals anchor meaning, how pretrained representations are adapted, how generative processes reconcile internal knowledge with external evidence, and how model behavior can be guided in transparent ways. When these stages are treated in isolation, characteristic failure modes arise, including context-insensitive grounding, instability under adaptation, hallucinated outputs, and safety interventions that disrupt communicative intent.

The dissertation therefore investigates trustworthiness through coordinated interventions at different interfaces of the modeling pipeline. It develops methods that strengthen context-sensitive multimodal grounding while preserving representational structure, examines inference-time mechanisms that regulate the interaction between prior knowledge and conditioning signals, and introduces cognitively informed analyses to identify interpretable loci for efficient behavioral steering. Rather than addressing isolated symptoms, these contributions target complementary sources of unreliability across data construction, representation maintenance, and generation dynamics.

Overall, the findings indicate that trustworthy foundation modeling must be engineered as a lifecycle property rather than achieved through post hoc alignment alone. Reliability arises from the deliberate coordination of grounding, adaptation, inference, and control, suggesting a pathway toward foundation models whose general capabilities are matched by predictability, transparency, and human-centered usability.