Stress detection with wearable physiological sensors is vital in digital health and affective computing. Conventional machine learning techniques usually examine physiological signals separately, missing the intricate inter-signal connections involved in the human stress response. While deep neural networks offer high accuracy, they function as black boxes, offering minimal understanding of the physiological processes behind stress detection. This study introduces a hierarchical graph neural network framework for WESAD stress detection, establishing a methodology for affective computing that emphasizes interpretability and extensibility while maintaining strong predictive performance. We proposed PAMG-AT (Physiological Attention Multi-Graph with Adaptive Topology) which is a hierarchical graph neural network architecture, for stress detection using multimodal physiological signals. In this framework, physiological features serve as nodes within a knowledge-driven graph, while edges represent established physiological relationships, including cardiac-electrodermal coupling and cardio-respiratory interaction. The architecture employs a three-level attention mechanism: spatial encoding via Graph Attention Networks (GAT) to assess feature importance, temporal modeling with a Transformer to capture dynamics across time windows, and global pooling for classification. The model is evaluated using three sensor configurations (chest-only, wrist-only, and hybrid) on the WESAD dataset, employing rigorous Leave-One-Subject-Out (LOSO) cross-validation. PAMG-AT achieves competitive performance, with 94.59% accuracy (+/- 6.8%) for chest sensors, 91.76% accuracy (+/- 9.2%) for wrist sensors, and 92.80% accuracy (+/- 8.33%) for the hybrid configuration. The proposed method provides interpretability via attention weights, revealing that ECG-EDA relationships (cardiac-electrodermal coupling) are most predictive of stress. Three low-responder subjects (S2, S3, S9) with atypical physiological stress patterns demonstrate lower accuracy (81-87%), offering clinically valuable insights for personalized stress management. The effective wrist-only configuration, achieving 91.76% accuracy, supports practical deployment in consumer wearables.