logo

SCIENTIA SINICA Informationis, Volume 48 , Issue 3 : 248-260(2018) https://doi.org/10.1360/N112017-00252

3D human face transplanting via depth camera

More info
  • ReceivedJan 4, 2018
  • AcceptedFeb 5, 2018
  • PublishedMar 19, 2018

Abstract


Funded by

国家自然科学基金(61672482)

国家自然科学基金(61672481)

国家自然科学基金(11626253)

中国科学院“百人计划"研究项目基金


References

[1] Blanz V, Scherbaum K, Vetter T. Exchanging Faces in Images. Comput Graphics Forum, 2004, 23: 669-676 CrossRef Google Scholar

[2] Bitouk D, Kumar N, Dhillon S, et al. Face swapping: automatically replacing faces in photographs. ACM Trans Graph, 2008, 27: 39. Google Scholar

[3] Kemelmacher-Shlizerman I. Transfiguring portraits. ACM Trans Graph, 2016, 35: 94. Google Scholar

[4] Jing Tong , Jin Zhou , Ligang Liu . Scanning 3D full human bodies using Kinects.. IEEE Trans Visual Comput Graphics, 2012, 18: 643-650 CrossRef PubMed Google Scholar

[5] Zhou J, Pan J J, Tong J, et al. Fast 3D human body reconstruction from Kinects. J Comput-Aid Des Comput Graph, 2013, 25: 873--879. Google Scholar

[6] Yang H Z, Lu Y, Fang Q, et al. Automatic 3D scanning system via depth camera. J Comput-Aid Des Comput Graph, 2015, 27: 2039--2045. Google Scholar

[7] Zhou W, Guan Z Q, Lu Y, et al. Easy 3D self-portraits. In: Proceedings of the 10th China Computer Graphics Conference and the 18th National CAGD Conference Papers. Hefei: USTC, 2014. 178. Google Scholar

[8] Chandawarkar A A, Diaz-Siso J R, Bueno E M. Facial appearance transfer and persistence after three-dimensional virtual face transplantation.. Plastic Reconstructive Surgery, 2013, 132: 957-966 CrossRef PubMed Google Scholar

[9] Pérez P, Gangnet M, Blake A. Poisson image editing. ACM Trans Graph, 2003, 22: 313-318 CrossRef Google Scholar

[10] Reinhard E, Ashikhmin M, Gooch B, et al. Color transfer between images. IEEE Comput Graph Appl, 2001, 21: 34--41. Google Scholar

[11] Lempitsky V, Ivanov D. Seamless mosaicing of image-based texture maps. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, 2007. Google Scholar

[12] Sorkine O, Cohen-Or D, Lipman Y, et al. Laplacian surface editing. In: Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, Nice, 2004. 175--184. Google Scholar

[13] Belkin M, Sun J, Wang Y. Discrete Laplace operator on meshed surfaces. In: Proceedings of the 24th Annual Symposium on Computational Geometry, College Park, 2008. Google Scholar

[14] Newcombe R A, Izadi S, Hilliges O, et al. KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, 2011. 127--136. Google Scholar

[15] Kazhdan M, Hoppe H. Screened Poisson surface reconstruction. ACM Trans Graph, 2013, 32: 29. Google Scholar

[16] Kazhdan M, Bolitho M, Hoppe H. Poisson surface reconstruction. In: Proceedings of the 4th Eurographics Symposium on Geometry Processing, Cagliari, 2006. 61--70. Google Scholar

[17] Floater M S. Parametrization and smooth approximation of surface triangulations. Comput Aided Geometric Des, 1997, 14: 231-250 CrossRef Google Scholar

[18] Field D A. Laplacian smoothing and Delaunay triangulations. Int J Numer Method Biomed Eng, 1988, 4: 709--712. Google Scholar

  • Figure 1

    (Color online) 3D face transplanting effects. (a) Testing subject; (b) transplanting based on an image; (c) result of this paper

  • Figure 2

    (Color online) Symbolic description

  • Figure 3

    (Color online) Face scanning

  • Figure 4

    System flow

  • Figure 5

    (Color online) Face extracting procedure. (a) Original point cloud; (b) crop via depth image; (c) crop via feature points; (d) cropping result; (e) geo-reconstruction; (f) texture reconstruction and feature points

  • Figure 6

    (Color online) Geo-detail comparisons between Poisson and Screen Poisson reconstructions (${\rm~Depth}=8$). Details of (a) Poisson and (b) Screen Poisson reconstruction method

  • Figure 7

    (Color online) Laplace vector

  • Figure 8

    (Color online) Geo-warping of scanned face. (a) Rigid transform; (b) correspondence of boundary; (c) result of warping

  • Figure 9

    (Color online) Rigid transform effects between 5 pairs and 78 pairs of feature points. (a) Orthographic view; (b) overhead view; (c) side view

  • Figure 10

    (Color online) Boundary smoothing. (a) Before smoothing; (b) after smoothing

  • Figure 11

    (Color online) Texturing effect comparisons. (a) Initial texture; (b) color optimization; (c) Poisson editing; (d) combine both methods

  • Figure 12

    (Color online) Female character with male face

  • Figure 13

    (Color online) Avatar with female face

  • Figure 14

    (Color online) Cartoon model with human face

  • Figure 15

    (Color online) Facial detail enhancement

  •   

    Algorithm 1 自动三维面部提取算法

    Require:人脸三维点云$C_s$, 采样$(I_c,I_d,P)$;

    Output:裁剪后的人脸三角网格$F_s$.

    借助采样数据进行点云的初步裁剪 (如图4(b)): 将深度图$I_d$转化为点云, 并借助相机位置$P$将点云注册到$C_s$位置处, 记为$C_d$, 定义$C_s$ 上一点$\boldsymbol{p}$到$C_d$的距离: $$d(\boldsymbol{p},C_d)={\rm min}_{\boldsymbol{q}\in{C_d}}{{\rm length}(\boldsymbol{p}-\boldsymbol{q})},$$ 给定阈值$d_{\rm~max}$, 若$d(\boldsymbol{p},C_d)>d_{\rm~max}$, 则此$\boldsymbol{p}$点舍弃.

    借助特征点位置对点云进一步裁剪 (如图5(c)): 从采样图$I_c$可自动检测人脸特征点, 利用$I_c$ 与$I_d$的对应关系及相机位置$P$, 可获取$C_d$ 上特征点位置, 特征点的最低高度与最高高度分别设为$h_{\rm~min}$与$h_{\rm~max}$, 由于人脸基本处于竖直状态, 预设offset为0.01 m, 若$p_y<h_{\rm~min}-{\rm~offset}$或$p_y>h_{\rm~max}+{\rm~offset}$, 则此$\boldsymbol{p}$点舍弃.

    使用Screen Poisson [15]表面重建算法提取三角网格 (如图5(e)).

  •   

    Algorithm 2 面部网格变形算法

    Require:$F_s$, $F_m$和$M_l$;

    Output:与$M_l$贴合情况较好的人脸网格.

    利用特征点刚性变换$F_s$至$F_m$ (图8(a));

    寻找$F_s$与$M_l$人脸边缘的对应关系, 以确定人脸新的边界点位置 (图8(b));

    使用基于网格Laplace的变形算法, 构造sect. 5.1小节中的线性系统求解顶点坐标, 变形$F_s$以适应$M_l$的面部边界轮廓 (图8(c)).