Physical Layer Security for Keyhole-based NOMA Downlink Systems with a Multi-antenna Eavesdropper

Abstract

This paper investigates the physical layer security of downlink nonorthogonal multiple access (NOMA) systems operating over a degenerate keyhole channel in the presence of a multi-antenna eavesdropper. We propose a joint antenna selection framework with transmit antenna selection at the source and receive antenna selection at both legitimate users and eavesdroppers, thus striving to reduce hardware complexity while maximizing secrecy performance. In this framework, the efficacy of confidentiality is assessed for a specific user allocation methodology by deriving the closed-form approximate expression of secrecy outage probability (SOP). Extensive Monte Carlo simulations validate analytical results and reveal that increasing the number of antennas at the source and legitimate users dramatically lowers SOP, whereas a more capable eavesdropper raises the risk of secrecy. Our findings demonstrate that strategic antenna deployment and non-orthogonal access can effectively safeguard communications even through severely scattering environments.