Question Answering (QA) requires understanding of natural language queries along with information content to obtain an answer to the question. In this survey, we focus on the question answering methods specifically based on the neural network frameworks which are the state-of-art for many QA datasets. The crux of a neural network model lies in the representation of both question and answer along with auxiliary knowledge as a continuous real-valued representation, called vectors or embeddings. Powerful QA models require processing of large information content accessible from Knowledge Bases (KBs). Many KBs are readily available and involve colossal quantities of information. KBs have been successfully incorporated to neural QA by embedding the relations and entities present in KBs and then using the learned embeddings. We survey several successful applications of KBs to neural question answering problem and study the role KBs play in neural QA.