SCIENCE CHINA Information Sciences, Volume 64 , Issue 9 : 192101(2021) https://doi.org/10.1007/s11432-019-2707-6

TZ-Container: protecting container from untrusted OS with ARM TrustZone

More info
  • ReceivedJun 15, 2019
  • AcceptedNov 7, 2019
  • PublishedAug 19, 2021



This work was supported in part by National Key Research Development Program (Grant No. 2016YFB- 1000104), National Natural Science Foundation of China (Grant No. 61772335), and Program of Shanghai Academic Research Leader.


[1] Merkel D. Docker: lightweight linux containers for consistent development and deployment. Linux Journal 2014, 2: 12. Google Scholar

[2] Amd launching “hierofalcon” 64bit arm embedded chips in 1h 2015 - zen and k12 next year. 2015. http://wccftech.com/amd-launching-arm-serves-year-wip/#ixzz3Yef58mtq. Google Scholar

[3] MorganT P. Arm servers: Cavium is a contender with thunderx. 2015. https://www.nextplatform.com/2015/12/09/arm-servers-cavium-is-a-contender-with-thunderx/. Google Scholar

[4] Amd opteron a1100. AMD. 2016. http://www.amd.com/en-gb/products/server/opteron-a-series. Google Scholar

[5] Sverdlik Y. Paypal deploys arm servers in data centers. 2015. http://www.datacenterknowledge.com/archives/2015/04/29/paypal-deploys-arm-servers-in-data-centers. Google Scholar

[6] Rath J. Baidu deploys marvell arm-based cloud server. 2013. http://www.datacenterknowledge.com/archives/2013/02/28/baidu-deploys-marvell-arm-based-server/. Google Scholar

[7] Introduction of Rancher-labs. Rancher-labs. 2017. http://rancher.com/rancher-labs-2017-predictions-rapid-adoption-and-innovation-to-come/. Google Scholar

[8] Kubernetes on arm. Martin Jensen. 2016. http://kubecloud.io/kubernetes-on-arm-cluster/. Google Scholar

[9] Docker on arm. Uli Middelberg. 2015. https://github.com/umiddelb/armhf/wiki/Installing,-running,-using-docker-on-armhf-(ARMv7)-devices. Google Scholar

[10] Linux cve. CVE Details. 2016. http://www.cvedetails.com/vulnerability-list/vendor_id-33/product_id-47/Linux-Linux-Kernel.html. Google Scholar

[11] Chen H, Zhang F, Chen C, et al. Tamper-resistant execution in an untrusted operating system using a virtual machine monitor. Parallel Processing Institute Technical Report 2007. FDUPPITR-2007-08001. Google Scholar

[12] Chen X, Garfinkel T, Lewis E, et al. Overshadow: a virtualization-based approach to retrofitting protection in commodity operating systems In: Proceedings of the 13th international conference on Architectural support for programming languages and operating systems ACM, 2008. 2--13. Google Scholar

[13] Arnautov S, Trach B, Gregor F, et al. Scone: secure linux containers with intel sgx. In: Proceedings of USENIX Symposium on Operating Systems Design and Implementation USENIX Association, 2016. Google Scholar

[14] Yang J, Shin K G. Using hypervisor to provide data secrecy for user applications on a per-page basis. In: Proceedings of the fourth ACM SIGPLAN/SIGOPS international conference on Virtual execution environments ACM, 2008. 71--80. Google Scholar

[15] Software guard extensions programming reference. Intel. 2015. https://software.intel.com/site/default/files/329298-001.pdf. Google Scholar

[16] Checkoway S, Shacham H. Iago attacks. SIGPLAN Not, 2013, 48: 253 CrossRef Google Scholar

[17] Hofmann O S, Kim S, Dunn A M. InkTag. SIGPLAN Not, 2013, 48: 265-278 CrossRef Google Scholar

[18] Kwon Y, Dunn A M, Lee M Z, et al. Sego: pervasive trusted metadata for efficiently verified untrusted system services. In: Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems ACM, 2016. 277--290. Google Scholar

[19] Mitsuishi T, Nomura S, Suzuki J. Accelerating Breadth First Search on GPU-BOX. SIGARCH Comput Archit News, 2014, 42: 81-86 CrossRef Google Scholar

[20] Chhabra S, Rogers B, Solihin Y, et al. SecureME: a hardware-software approach to full system security In: Proceedings of the international conference on Supercomputing 2011. Google Scholar

[21] Azab A M, Ning P, Zhang X. Sice: a hardware-level strongly isolated computing environment for x86 multi-core platforms. In: Proceedings of the 18th ACM Conference on Computer and Communications Security ACM, 2011. 375--388. Google Scholar

[22] Strackx R, Piessens F. Fides: selectively hardening software application components against kernel-level or process-level malware. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security ACM, 2012. 2--13. Google Scholar

[23] Sun H, Sun K, Wang Y, et al. Trustice: hardware-assisted isolated computing environments on mobile devices. In: Proceedings of the 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) IEEE, 2015. 367--378. Google Scholar

[24] Li Y, McCune J, Newsome J, et al. Minibox: a two-way sandbox for x86 native code. In: Proceedings of 2014 USENIX Annual Technical Conference (USENIX ATC 14) 2014. 409--420. Google Scholar

[25] Baumann A, Peinado M, Hunt G. Shielding applications from an untrusted cloud with haven. In: Proceedings of ACM Transactions on Computer Systems (TOCS) 2015. 33: 8. Google Scholar

[26] Tsai C-C, Porter D E, Vij M. Graphene-sgx: a practical library OS for unmodified applications on SGX. In: Proceedings of USENIX Annual Technical Conference (ATC) 2017. 8. Google Scholar

[27] Guan L, Liu P, Xing X, et al. Trustshadow: secure execution of unmodified applications with arm trustzone. 2017,. arXiv Google Scholar

[28] gvisor. Google. 2018. https://github.com/google/gvisor. Google Scholar

[29] Alves T, Felton D. Trustzone: Integrated hardware and software security. ARM white paper 2004, 3: 18--24. Google Scholar

[30] Lipp M, Schwarz M, Gruss D, et al. Meltdown. arXiv preprint arXiv:1801.01207 Jan. 2018. Google Scholar

[31] Arm trusted firmware. ARM. 2017. https://github.com/ARM-software/arm-trusted-firmware. Google Scholar

[32] Xu Y, Cui W, Peinado M. Controlled-channel attacks: deterministic side channels for untrusted operating systems. In: Proceedings of 2015 IEEE Symposium on Security and Privacy (SP) IEEE, 2015. 640--656. Google Scholar

[33] Hähnel M, Cui W, Peinado M. High-resolution side channels for untrusted operating systems. In: Proceedings of 2017 USENIX Annual Technical Conference (USENIX ATC 17) 2017. 299--312. Google Scholar

[34] Kocher P, Genkin D, Gruss D, et al. Spectre attacks: Exploiting speculative execution. 2018,. arXiv Google Scholar

[35] Weisse O, Van Bulck J, Minkin M, et al. Foreshadow-ng: breaking the virtual memory abstraction with transient out-of-order execution. Technical Report,KU Leuven 2018. Google Scholar

[36] Ta-Min R, Litty L, Lie D. Splitting interfaces: making trust between applications and operating systems configurable. In: Proceedings of the 7th symposium on Operating systems design and implementation USENIX Association, 2006. 279--292. Google Scholar

[37] Peinado M, Chen Y, England P, et al. Ngscb: a trusted open system. In: Proceedings of Australasian Conference on Information Security and Privacy Springer, 2004. 86--97. Google Scholar

[38] J. M. McCune, Y. Li, N. Qu, Z. Zhou, A. Datta, V. Gligor, and A. Perrig. Trustvisor: Efficient TCB reduction and attestation. In: Proceedings of 2010 IEEE Symposium on Security and Privacy (SP), pages 143--158. IEEE, 2010. Google Scholar

[39] Dautenhahn N, Kasampalis T, Dietz W, et al. Nested kernel: an operating system architecture for intra-kernel privilege separation. In: Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems ACM, 2015. 191--206. Google Scholar

[40] Dan W, Martin L, Ricardo K, et al. Unikernels as processes. In: Proceedings of 2018 ACM Symposium on Cloud Computing 2018. Google Scholar

[41] Wang H, Shi P, Zhang Y. Jointcloud: a cross-cloud cooperation architecture for integrated internet service customization. In: Proceedings of 2017 IEEE 37th international conference on distributed computing systems (ICDCS) IEEE, 2017. 1846--1855. Google Scholar

[42] Cao D G, An B, Shi P C. Providing Virtual Cloud for Special Purposes on Demand in JointCloud Computing Environment. J Comput Sci Technol, 2017, 32: 211-218 CrossRef Google Scholar

[43] Wang H, Shi P, Yin H. Collaboration environment for JointCloud computing. Sci Sin-Inf, 2017, 47: 1129-1148 CrossRef Google Scholar

[44] Azab A M, Ning P, Shah J, et al. Hypervision across worlds: real-time kernel protection from the arm trustzone secure world. In: Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security ACM, 2014. 90--102. Google Scholar

[45] Cho Y, Shin J, Kwon D, et al. Hardware-assisted on-demand hypervisor activation for efficient security critical code execution on mobile devices. In: Proceedings of 2016 USENIX Annual Technical Conference (USENIX ATC 16) USENIX Association, 2016. 565--578. Google Scholar

[46] Hua Z, Gu J, Xia Y, et al. vtz: virtualizing arm trustzone. In: Proceedings of the 26th $\{$USENIX$\}$ Security Symposium ($\{$USENIX$\}$ Security 17) 2017. Google Scholar

[47] Brasser F, Gens D, Jauernig P, et al. Sanctuary: arming trustzone with user-space enclaves. In: Proceedings of the 26th Network and Distributed System Security Symposium 2019. Google Scholar

  • Figure 1

    Layers of attacks. Previous researches usually focused on direct attacks and Iago attacks [16]. In this paper we target the MUMA attacksat the layer of container abstraction.

  • Figure 2

    (Color online) Sample code of multi-process synchronization attacks. The malicious OS ignores $P$() and $V$() operations of an IPC semaphore to violate the mutual exclusiveness of the two code snippets.

  • Figure 3

    Design overview of the TZ-Container Each container process is protected by an IEE in the normal world. Each IEE is maintained by an IEE-managerrunning in the secure world. The container shield defends against Iago attacks and MUMA attacks.

  • Figure 4

    The procedure of creating an IEE. The kernel is responsible for creating a process, including constructing the page table. The created page table must be registered in the IEE-manager. Before entering a new process, the IEE-managerchecks the page table and enforces the memory isolation.

  • Figure 5

    (a)The overhead of all integer (INT) applications in SPEC_CPU 2006 benchmark, lower the better; (b), (c) the throughput of Nginx and Memcached, the higher the better; The $x$-axesof (b) and (c) represent the number of processes/threads used by the application.

  • Figure 6

    (a) and (b) The throughput of Redis and SQLite3; (c) the throughput of Redis with different numbers of containers. The $x$-axisrepresents the number of processes/threads/containers used by the applications. The higher the better.

  • Table 1  

    Table 1Attack considerations$^{\rm~a)}$

    Direct attacksIago attacksMUMA attacks
    Disk I/O
    User access
    Attack apps HasHasHas
    Attack containers HasHasHasHasHasHas
    SICE [21] $\checkmark$
    Fides [22] $\checkmark$
    TrustICE [23] $\checkmark$
    Overshadow [12] $\checkmark$$\checkmark$
    SP$^{3}$ [14] $\checkmark$$\checkmark$
    Virtual Ghost [19] $\checkmark$$\checkmark$
    MiniBox [24] $\checkmark$$\checkmark$
    InkTag [17] $\checkmark$$\checkmark$$\checkmark$$\bigcirc$
    Sego [18] $\checkmark$$\checkmark$$\checkmark$$\bigcirc$
    SecureME [20] $\checkmark$$\checkmark$$\bigcirc$
    Haven [25] $\checkmark$$\checkmark$$\checkmark$
    SCONE [13] $\checkmark$$\checkmark$$\checkmark$
    Graphene-SGX [26] $\checkmark$$\checkmark$$\checkmark$$\bigcirc$
    TrustShadow [27] $\checkmark$$\checkmark$$\checkmark$
    gVisor [28] $\checkmark$
    TZ-Container $\checkmark$$\checkmark$$\checkmark$$\checkmark$$\checkmark$$\checkmark$


  • Table 2  

    Table 2Security properties for protecting a container

    Security properties to be enforced
    P-1.1: OS cannot access container process's memory.
    Memory & CPU context P-1.2: OS cannot tamper with container process's CPU context.
    P-1.3: OS can only enter the container process from fixed points.
    Disk I/OP-2.1: OS cannot break the confidentiality and integrity of container file.
    P-2.2: One container's file cannot be accessed by any other container.
    Defending against Iago attacks P-3.1: OS cannot arbitrarily return value for syscalls.
    Multi-application synchronizationP-4.1: OS cannot tamper with the functionality of semaphore.
    P-4.2: OS cannot arbitrarily inject signal to container process.
    P-4.3: OS cannot tamper with the functionality of flock/futex syscalls.
    Inter-application communication P-5.1: Enforce the integrity and confidentiality of the communication data
    User access controlP-6.1: The permission bit of file and IPC instance cannot be tampered with.
    P-6.2: The permission of each container process cannot be tampered with.
    P-6.3: Only the process with correct permission can access a file or an IPC instance.
    P-6.4: Only the process with correct permission can send a signal.
  • Table 3  

    Table 3Single operation overhead

    Test case Docker ($\mu$s) TZ-Container ($\mu$s)
    null syscall 0.21 1.85
    open/close 7.37 12.2
    mmap 252 404
    page fault 1.24 2.53
    fork+exit 1865 6712
    fork+exec 3334 8875
    ctxsw 2p/0k 8.82 14.1

Contact and support