Optimization of a software-only H.264 encoder

Abstract

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT AUTHOR'S REQUEST.] With the booming of semiconductor and information technology, digital video have become more popular than ever. The large volume nature of video data has made the capturing, storage, transmission and communication of video data a challenging task. H.264, the latest video coding standard as the joint efforts of MPEG and ITU-T has been considered the state of the art of video coding. However, the performance gain comes at the cost of much higher implementation complexity driving by the Moore's law. To utilize the better coding efficiency of H.264 in video communication, this thesis is focused on improving the encoding speed of an existing H.264 video encoder - T264, leveraging the powerful specific multimedia instructions provided in INTEL/AMD CPUs. The encoder speed optimization was carried out at three different layers: high level language (C-language) level optimization using techniques such as trading code size with speed, move constants outside loop, optimizing large probability events and etc. The second layer takes advantage of the SIMD (Single Instruction Multiple Data) instructions supported by INTEL/AMD MMX/SSE/SSE2 technologies. At this layer, assembly code is written to leverage these powerful parallel instructions. And at the third layer, the optimization is carried out at algorithm level to try to make motion estimation/mode decision faster. After all these changes, the software-only encoder speed is improved around 50% percent with un-noticeable video quality difference.