Computed tomography (CT) has a trend towards higher resolution and higher noise. This development has increased the interest in anisotropic smoothing techniques for CT, which aim to reduce noise while preserving structures of interest. However, existing smoothing techniques are slow, which makes clinical application difficult. Furthermore, the published methods have limitations with respect to preserving small details in CT data. This paper presents a widely applicable speed optimized framework for anisotropic smoothing techniques. A second contribution of this paper is an extension to an existing smoothing technique aimed at better preserving small structures of interest in CT data. Based on second-order image structure, the method first determines an importance map, which indicates potentially relevant structures that should be preserved. Subsequently an anisotropic diffusion process is started. The diffused data is used in most parts of the images, while structures with significant second-order information are preserved. The method is qualitatively evaluated against an anisotropic diffusion method without structure preservation in an observer study to assess the improvement of 3-D visualizations of CT series and quantitatively by determining the reduction of the difference between low and high dose CT scans of in vitro carotid plaques.