As the Internet of Things (IoT) continues to penetrate into our lives, the security becomes a crucial issue for IoT devices to communicate with each other. Many standardized network protocols and cryptographic protocols have been proposed recently for IoT to secure their communication. However, due to the heterogeneous nature and the constraint resource in IoT, many IoT devices failed to protect their communication with the existing standardized protocols. To this end, in this work we explore the area of secure communication in IoT environment. In particular, we first survey the IoT communication protocols in a layered architecture (from physical layer to application layer), show what security services are provided by the protocols, and provide a comprehensive study of challenges in designing communication protocols for IoT. We then examine the key exchange protocols in IoT which is the core component to efficiently adopt secure protocols on different networking layers. We aim to give a critical overview of the security challenges and requirements in IoT ecosystem and provide a taxonomy of communication protocols and key exchange protocols for IoT networks. Finally, we end our work with a discussion of newly emerging cryptographic primitives that may enhance the security and efficiency in IoT communication.