Naïve Bayes and named entity recognition for requirements mining in job postings

Abstract

This paper analyses how the required skills in a job post can be extracted. With an automated extraction of skills from unstructured text, applicants could be more accurately matched and search engines could provide better recommendations. The problem is optimized by classifying the relevant parts of the description with a multinomial naïve Bayes model. The model identifies the section of the unstructured text in which the requirements are stated. Subsequently, a named entity recognition (NER) model extracts the required skills from the classified text. This approach minimizes the false positives since the data which is analyzed is already filtered. The results show that the naïve Bayes model classifies up to 99% of the sections correctly, and the NER model extracts 65% of the skills required for a position. The accuracy of the NER model is not sufficient to be used in production. On the validation set, the performance was insufficient. A more consistent labelling guideline would be needed and more data should be annotated to increase the performance.