Information Extraction (IE) is one of the important fields in Natural Language Processing. IE models can be exploited to obtain meaningful information from raw text and provide them in a structured format which can be used for downstream applications such as question answering. An IE system consists of several tasks including entity recognition, relation extraction, and event detection, to name a few. Among all recent advanced deep learning models proposed for IE tasks, one of the potential directions to improve performance is to incorporate structural information. Structural information refers to encoding any interactions between different parts of the input text. This information is helpful to overcome long distances between related words or sentences. In this dissertation, we study novel deep learning models that integrate structural information into the representation learning process. In particular, three major categories, i.e., existing structures, inferred structure at the sample level, and inferred structure at dataset levels are studied in this dissertation. We finally showcase the novel application of structure-based models for the less-explored setting of cross-lingual IE.