Abstract: In view of the low recognition accuracy of telephone voice gender detection, an effective gender detection scheme for telephone speech is proposed. Firstly, the Convolutional Neural Network (CNN) is used to extract the effective information of Power-Normalized Cepstral Coefficient (PNCC), and then Support Vector Machine (SVM) is selected to realize gender classification based on the optimized fundamental frequency features. The proposed scheme can effectively study the differences of male and female's pronunciation and auditory perception characteristics, and can benefit from the ability of CNN feature extraction and SVM robust classification. Experimental results show that the proposed scheme outperforms the traditional methods in gender recognition accuracy for the telephone speech data set in practical scenarios.