In order to provide a robust multi-sensor image matching algorithm to satisfy the scale and rotation invariance, we put forward a new matching method based on key line segment pairs, inspired by the visual system in which most of the information of image is contained in sketch.Firstly, the line segments that can express the feature of image are found and screened out.Then, the characteristic of line segment pairs is constructed by using the information and the relative position of line segments.And initial matching is carried out by the similarity between line segment pairs of two images.Finally, the topological relationship between line segments is used to achieve accurate registration.Experiment results tested by multi-sensor images show that the accuracy of image matching by using this proposed algorithm is more than 75% and the computing time is about 1/5 of traditional algorithm which satisfies the requirement of real-time and veracity in the multi-sensor image matching.