A Review of Image Inpainting Techniques & Their ApplicationsVamshi Krishna.
M Department of ECE, Vignan’s Foundation for Science, Technology ; Research Guntur, Andhra Pradesh, India. mvk. [email protected] com Ebenezer Daniel Department of ECE, Vignan’s Foundation for Science, Technology & Research, Guntur, Andhra Pradesh, India [email protected]
comABSTRACT Image inpainting is a type of image restoration technique. Images may be corrupted or may have undesired regions which are to be removed/replaced. The objective of inpainting is to reconstitute the missing or damaged portions in such a way that they are unnoticeable. The main methods of inpainting include structure based, texture based and hybrid inpainting. These methods are used in different applications. One of the major applications is restoration of our heritage digitally. Restoring heritage digitally is one important application of image inpainting. Historical monuments and paintings are enormous source of knowledge for mankind as they depict our culture and history.
They get damaged due to many reasons like aging, human interference etc. Paintings get faded with respect to time. Defects over structures appear like faint lines, loss of paint on paintings, cracks and disfigured regions in statues. Our aim is to throw a light on the works based on the areas where inpainting was used. Mainly on the areas like restoration of paintings and sculptures digitally. KEYWORDS Image inpainting, Digital Heritage, Restoration, Mural Images, Monuments.
I. INTRODUCTION Image inpainting is an important application in the field of image processing. Inpainting is the process of rejuvenating lost or destroyed parts of images or videos. Inpainting itself is an ancient art. The main goal of inpainting is to restore the lost parts in photos and videos or remove the unwanted portions in the image 1. The image with damaged or unnecessary portions is either removed or filled using inpainting. Inpainting has extensive applications.
It can be applied in the field of multimedia, in the field of archaeology like safeguarding inestimable historic ruins etc. There is a need for an algorithm to restore only the damaged portions in the image. It can be used to remove the red eye, stamped dates from the photos and is well suited in the field of adding effects in movies. The idea of inpainting is to fill the missing regions or distorted regions with the most information available from the neighbour location of images. Mural images and historic objects are very crucial and so there is a need to preserve them as they are important communicators of civilization and culture of that particular land.
They get damaged with respect to age. Historic objects here are divided into two categories namely mural images and statue images. Mural images are the paintings on the surfaces like walls or ceilings. Heritage statues are devastated with respect to their appearance like deformed eyes, nose, mouth etc.
Reconstructing the mutilated structures or paintings physically is impossible, because of their delicacy and doing so may deviate them from their original appearance. So, an alternate solution is to inpaint the demolished regions. Generally professional curators do inpainting of art works. The task of inpainting involves the following steps. 1.
The whole image determines how to fill the gap. 2. Continue the boundaries of the regions inside the region to be inpainted. 3. Regions inside gap are filled with colors which their boundaries consist of. 4. Tiny spots remaining in the image are filled. The target of inpainting is not just to restore the image but to make the image equivalent with an original picture.
This paper gives an overview of image inpainting, their techniques and their applications. The inpainting techniques have been divided into 3 categories based on the area of application, namelyWOODSTOCK’18, June, 2018, El Paso, Texas USA Vamshi Krishna. M et al. 1. General Applications 2. Mural Images Restoration and 3.
Restoration of disfigured statues images Rest of the paper is organized as follows. Section II describes the techniques involved in inpainting which are used for general applications like removal of text on images, removal of objects in images, restoring of old and deteriorated photo graphs etc. Section III discusses about restoration of faded or disfigured faded murals. Restoration of demolished facial images of heritage statue images is discussed in Section IV.
Summarizing the review is done in Section V. II. INPAINTING TECHNIQUES FOR GENERAL APPLICATIONS General applications include removal of text over photos, red eye removal from photos, removing microphones in special effects, removal of large objects like human beings, towers etc. The above-mentioned applications are commonly used and addressed in the case of image inpainting by many researchers.
Bertalmio et al., 1 introduced Partial Differential Equation (PDE) based inpainting. In Partial Differential Equation (PDE) based inpainting algorithm the information from the boundary of region into the region to be inpainted. The main focus in PDE based inpainting is to maintain uniform structure all over the image. PDE based inpainting is useful in filling small regions. The main drawback of structured inpainting is it takes a long time to inpaint a bigger image. It can only retain the structure of the image but not the texture.
In 2, it was shown how to fill-in the missing regions in an image using joint interpolation algorithm. In this, the isophote lines are extended into the missing regions of the image. This method fails in restoring the texture data.
Bertalmio et al. in 3 compared the image to a stream of fluid flowing. This was based on fluid dynamics and Navier-Stokes equation. Although this is a novel method this doesn’t have a mathematical algorithm. An Exemplar based inpainting technique was proposed by Antonio Criminisi et al. 4. This was used to remove large objects in images.
Combining the advantages of both texture and structure based inpainting approaches. This method performs well when removing larger objects. But the limitation of this work is that it fails in handling curves in the images. Huang Ying et al. 5 improved the inpainting algorithm by using segmentation techniques.
Watershed segmentation combined with curvature features are extracted. Curvature features are the texture details. Prof Amol J. Pawar et al., 6 extended the Partial Differential Equation (PDE) based algorithm for inpainting of videos.
Inpainting of videos is a complex task as the region to be inpainted changes in every frame. All the works mentioned above are used for general applications which include removing objects, text, even removing large objects like humans from the image. All these used either structure or texture based inpainting or even combined both. One such was proposed in 7.
This inpainting the structure and texture region of an image at a time. Images with large texture regions was not addressed properly in this algorithm. In texture inpainting the sample texture is filled in the holes i.
e., areas to be inpainted. III. MURAL IMAGES RESTORATION One application of image processing is for the conservation of art works called Mural Images. Murals are the paintings drawn on the walls or ceilings of structures. As time passes by, these art works may be deteriorated 8.
Losses like faded lines, loss of paint etc., can be observable in mural paintings. Physical treatment to these losses may lead to complete destruction of them, which is not recommended by many conservators. So, image processing can be used to restore these art works digitally. This makes the interaction of two different worlds arts and sciences. The first and foremost use of image processing in art field is for analysing the features of painting like the artist, time of the artist, brush strokes etc. Virtual restoration of paintings is possible with the help of image processing algorithms. This process of digitization must not disturb the paintings.
As art works are immovable, it is challenging to analyse them at their place. The main hurdle in this area is to bring a researcher with a technical background and researchers from humanistic area on to same line. 9 Baogang Wei et al., 10 linked artificial intelligence and image processing methods to virtually restore the colors in murals. Divided the knowledge of colors into 4 categories. A hybrid reasoning strategy was developed to determine the colors. Next image segmentation techniques like region and growing and merging was used to combine the regions with same colors. There is a scope to improve the segmentation techniques to distinguishA Review of Image Inpainting Techniques and their Applications WOODSTOCK’18, June, 2018, El Paso, Texas USA devastated regions.
In 11 Mauro Barni et al., proposed a system to virtually clean dirty paintings and removing cracks in artworks. Table I shows the techniques proposed for restoration of mural images. S. No Work of Techniques Used 1 Manikanta Prasanth Kumar 12 1. Line detection 2. K-Means Clustering 3. Averaging of Pixels 4.
Weighted Average 2 Pulak Purkait 13 1. Coherent Texture Synthesis 2. Edge Sharpening Anisotropic Diffusion 3 Barbara Zitová 14 1. Pre-processing 2. Image Restoration 4 Michail Pappas 15 1. Sample Mean Matchings 2. Linear Approximation 3. Closer Point Approximation 4.
White Point Transformation 5. Radial Bias Function 5 Paul Nemes 16 1. Color Restoration 2. Color Correction 3. Support Vector Regression (SVM) IV. RESTORATION OF HERITAGE STRUCTURES Heritage structures are very important for mankind.
They get mangled due to natural calamities and even human intervention. Disturbances like cracks in the structures and disfigured faces like evacuated eyes, nose and mouth regions. They are important to understand the history and civilization of mankind. Repairing the disfigured structures physically may result in complete damage to the entire structure. With the implementation of virtual reality in museums, they can showcase their great treasure i.e., monuments in a digital form.
So, one best way to work with them is to restore them virtually. Contorted structures can’t attract tourists to visit them. Restoring the disfigured statues images virtually may increase their beauty. The amount of work done in this aspect is very low. Milind G. Padalkar et al., 17 worked on identifying the disfigured face regions and inpainting them.
He classified facial regions as dislocated using K-Means clustering algorithm and inpainted the defect regions. Recognizing of other parts of the statue images as damaged regions was not addressed. In 18 Chintan Parmar et al., used basic edge detection techniques to identify the distorted facial regions.
And later inpainted the defect regions. Using Auto Regressive (AR) parameters which were estimated using non-negatively constrained least squares (NNLS) in 19 inpainted using exemplar based inpainting technique. Edge detection and morphological dilation was used to fill the cracks in heritage structure images 20. Milind G. Padalkar et al.
, in 21 extended the crack detection and inpainting algorithm for videos. This algorithm first detects cracks in the images using edge strength calculation, thresholding and refinement techniques. Then this is applied to videos by identifying the crack regions independently in every frame. V. CONCLUSION As discussed different applications of image inpainting, it can be observed that inpainting is a great technique which can be applied in many areas.
There is a need to develop algorithms which can use the best of both structure and texture inpainting. Decreasing the computational cost and time is a major criterion. Inpainting becomes successful only when the original image is restored properly. Sharp and curved edges are leading cause for errors. Conservation of paintings or art works is an important task. As the structures or paintings are not mobile there is a need to safeguard the place of their very existence. Undoubtedly it is desirable to digitize our heritage by which we can safeguard and transmit the knowledge to the future generations. Inpainting of other regions like hands or structures of animals is a challenging task where much light was not thrown.
REFERENCES 1 Bertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. (2000, July).
Image Inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (Pp. 417-424). ACM Press/Addison-Wesley Publishing Co. 2 Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G.
, & Verdera, J. (2001). Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing, 10(8), 1200-1211. 3 Bertalmio, M., Bertozzi, A. L.
, & Sapiro, G. (2001). Navier-stokes, fluid dynamics, and image and video inpainting.WOODSTOCK’18, June, 2018, El Paso, Texas USA Vamshi Krishna.
M et al. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on (Vol. 1, pp. I-I). IEEE.
4 Criminisi, A., Pérez, P., ; Toyama, K. (2004). Region filling and object removal by exemplar-based image inpainting.
IEEE Transactions on image processing, 13(9), 1200-1212. 5 Ying, H., Kai, L., ; Ming, Y. (2017).
An Improved Image Inpainting Algorithm Based on Image Segmentation. Procedia Computer Science, 107, 796-801. 6 Pawar, A. J., Jadhav, H. L., Ukarande, V.
V., ; Chavan, S. R.
(2018, January). A Comparative Study of Effective Way to Modify Different Object in Image and Video Using Different Inpainting Methods. In 2018 2nd International Conference on Inventive Systems and Control (ICISC). IEEE. 7 M. Bertalmio, L. Vese, G.
Sapiro and S. Osher, “Simultaneous structure and texture image inpainting,” in IEEE Transactions on Image Processing, vol. 12, no.
8, pp. 882-889, Aug. 2003. DoI: 10. 1109/TIP. 2003. 815261 8 V. Cappellini and A.
Piva, “Opportunities and issues of image processing for cultural heritage applications,” 2006 14th European Signal Processing Conference, Florence, 2006, pp. 1-5. 9 M. Barni, A.
Pelagotti and A. Piva, “Image processing for the analysis and conservation of paintings: opportunities and challenges,” in IEEE Signal Processing Magazine, vol. 22, no. 5, pp. 141-144, Sept. 2005.
doi: 10. 1109/MSP. 2005.
1511835 10 Baogang Wei, Yonghuai Liu and Yunhe Pan, “Using hybrid knowledge engineering and image processing in color virtual restoration of ancient murals,” in IEEE Transactions on Knowledge and Data Engineering, vol. 15, no. 5, pp.
1338-1343, Sept. -Oct. 2003. DoI: 10. 1109/TKDE.
2003. 1232282 11 M. Barni, F. Bartolini and V. Cappellini, “Image processing for virtual restoration of artworks,” in IEEE Multimedia, vol. 7, no. 2, pp.
34-37, April-June 2000. DOI: 10. 1109/93.
848424 12 K. M. P. Kumar, M.
Kumar, B. V. S.
Bhargav and M. Ghorai, “Digital Restoration of Deteriorated Mural Images,” 2014 Fifth International Conference on Signal and Image Processing, Bangalore, India, 2014, pp. 36-41. doi: 10. 1109/ICSIP.
2014. 10 13 Purkait, P., ; Chanda, B. (2012, December). Digital restoration of damaged mural images. In Proceedings of the Eighth Indian Conference on Computer Vision, Graphics and Image Processing (p.
49). ACM. 14 Zitová, B.
, Flusser, J. ; Šroubek, F. Pattern Anal Applic (2004) 7: 18. https://doi. org/10. 1007/s10044-003-0200-3 15 M.
Pappas and I. Pitas, “Digital color restoration of old paintings,” in IEEE Transactions on Image Processing, vol. 9, no. 2, pp. 291-294, Feb. 2000. DOI: 10.
1109/83. 821745 16 Nemes P., Gordan M.
, Vlaicu A. (2013) Color Restoration in Cultural Heritage Images Using Support Vector Machines. In: Bock H., Jäger W., Winckler M. (eds) Scientific Computing and Cultural Heritage. Contributions in Mathematical and Computational Sciences, vol 3. Springer, Berlin, Heidelberg.
17 Padalkar M. G., Vora M.
V., Joshi M. V., Zaveri M.
A., Raval M. S. (2013) Identifying Vandalized Regions in Facial Images of Statues for Inpainting. In: Petrosino A., Maddalena L., Pala P. (eds) New Trends in Image Analysis and Processing – ICIAP 2013.
ICIAP 2013. Lecture Notes in Computer Science, vol 8158. Springer, Berlin, Heidelberg.
18 Parmar, C. M., Joshi, M. V., Raval, M. S., Zaveri, M.
A.: Automatic image inpainting for the facial images of monuments. In: Proceedings of Electrical Engineering Centenary Conference 2011, pp. 415–420. IISc Bangalore, India (2011). 19 Padalkar, Milind ; Joshi, Manjunath ; Zaveri, Mukesh ; Parmar, Chintan. (2012).
Exemplar based Inpainting using Autoregressive Parameter Estimation. 154-160. 20 Turakhia, Naman ; Shah, R ; Joshi, Manjunath. (2012). Automatic crack detection in heritage site images for image inpainting.
Eighth Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP). 21 Padalkar, M. G. ; Joshi, M. V.
Machine Vision and Applications (2015) 26: 317. https://doi. org/10.
1007/s00138-015-0661-6.
