Cross-Modal Communication Technology

This book offers a systematic and in-depth investigation on Cross-Modal Communication Technology. In chapter 1, the authors present the background and motivation of this domain. In chapter 2, the authors outline multi-modal services and a general cross-modal communication architecture, including typical service categories, communication requirements, architectural details, challenges, and design principles. In chapters 3, 4, and 5, the authors respectively propose three key techniques within the cross-modal communication architecture: cross-modal coding, cross-modal transmission and cross-modal signal recovery. For each technique, the authors provide an in-depth presentation of the representative algorithms and the related experimental results. chapter 6 describes an established database and the three developed cross-modal communication prototype systems. Chapter 7 introduces extended techniques, such as cross-modal semantic communications, cross-modal communications in extremely low-resource scenarios. Finally, the authors provide a summary of the entire book and discuss future research directions in chapter 8. The readers will gain a deep understanding of the development and technical details of this new communication paradigm for the emerging multi-modal services.

Currently, haptic information is gradually being integrated into traditional audio-visual-dominant multimedia services, forming multi-modal services. Multi-modal services have been considered killer applications in the B5G and 6G eras, including remote industrial robotic grasping, tele-surgery and diagnosis, online immersive shopping and gaming, etc. To support the thriving multi-modal services, the cross-modal communication paradigm is proposed. Compared with traditional multimedia communications, the cross-modal communication paradigm is characterized by the collaborative transmission and processing of audio, visual, haptic signals and streams. By fully exploring the potential correlations among audio, visual, and haptic modalities, it enhances natural human-machine interactions and immersive experiences in multi-modal services, while meeting the requirements of low latency, high reliability, and high throughput.

This book targets graduate and undergraduate students majoring in the areas of wireless communications, computer science and engineering, and electrical engineering as well as researchers working in this field. Professionals seeking multimedia communications for the emerging multi-modal services will also want to purchase this book.