Experimental Demonstration of DistributedMulti-tenant Cloud/Fog and HeterogeneousSDN/NFV Orchestration for 5G ServicesRicard Vilalta, Arturo Mayoral, Ramon Casellas, Ricardo Mart ´nez, Raul Mu ˜nozCommunication Networks DivisionCentre Tecnol ogic de Telecomunicacions de Catalunya, CTTCAv. Carl Friedrich Gauss 7, 08860 Castelldefels, Spain Email: [email protected].
esAbstract —It is expected that in 5G networks billions of smartdevices will generate huge aggregated volumes of data that willbe processed in distributed cloud/fog infrastructure. To this end,it is required an integrated management of the network and thecloud resources forming a converged end-to-end system. SoftwareDened Networking (SDN) and Network Function Virtualization(NFV) architectures are the key enablers to integrate bothnetwork and cloud resources, enabling cross-optimization in bothsides.
This paper presents the experimental activities related to5G services using the ADRENALINE testbed located at CTTCpremises in Castelldefels (Barcelona, Spain). SDN orchestrationis presented as a feasible and scalable solution for providingend-to-end connectivity between heterogeneous networks andcloud/edge computing. Moreover, we present the demonstration ofan SDN/NFV orchestrator to dynamically create virtual backhaultenants over a multi-layer (packet/optical) aggregation networkand deploy virtual network functions to better adapt the capacityincrease of Mobile Network Operators.I. IN T RO D U C T I O NThe fth generation of mobile networks technology (5G)is not only focused on the evolution of the radio technologies,but with the design of a new End-to-End (E2E) convergednetwork and cloud infrastructure.This converged infrastructure, illustrated in Fig. 1, is com-posed of: E2E heterogeneous network segments covering radioand xed access, metro aggregation, and core transport involv-ing heterogeneous wireless and optical technologies; massivedistributed cloud computing and storage infrastructure; andlarge amounts of heterogeneous smart devices and terminalsfor traditional mobile broadband services (e.g.
, smartphones,tablets, etc.) and IoT services (e.g. sensors, actuators, robots,cars, drones, etc.).From the network perspective, the 5G architecture needs toprovide high exibility, low-latency, and high-capacity in orderto support the forecasted 1000x growth in mobile data trafcwith sub-millisecond latency 1. On the control/managementside, E2E connectivity services need to be provisioned betweendistributed cloud infrastructures and end users.
These require-ments can only be met by efciently integrating heterogeneousaccess (RAN, xed access, satellite, Wi-Fi, personal area net-works), optical/wireless crosshaul (fronthaul/backhaul), metro aggregation packet networks and high-capacity optical coretransport networks.For this integration, SDN Orchestration is proposed tocoordinate, in a hierarchical, logically centralized manner,the heterogeneous control plane technologies of the differentnetwork segments, which may remain separated as independentadministrative domains. Moreover, the current SDN controllersnorthbound interface (NBI) is highly heterogeneous and tech-nology and vendor dependent. The STRAUSS project hasdened the rst Transport API named Control OrchestrationProtocol (COP), that abstracts the particular control plane tech-nology of a given transport domain. COP provides a research-oriented multi-layer approach using YANG/RESTconf 2.At the cloud level, the demand of massive computing andstorage will dramatically be increased by new 5G services,which will require processing and storage capabilities (e.g.
,Big Data). In addition, the impending growth of NetworkFunction Virtualization (NFV) 3 and Mobile Edge Comput-ing (MEC) 4 also require cloud services for the deploymentof software functions (e.g., mobile Evolved Packet core (EPC),local cache, rewalls). Originally, cloud services have beenimplemented in core data centers (DCs) for high-computationalor long-term processing. However, the cloud is being spread tothe edge of the network (e.g.
, in edge DCs located in the metronetwork, or even in network nodes or mobile base stations withcloud capabilities) in order to reduce the latency of servicesfor the end user. This concept is referred to as fog computing6. Therefore, 5G networks need a global orchestration forthe distributed cloud/fog implementation and the managementof heterogeneous networks.The ongoing efforts carried out by CTTC towards theaforementioned integration architecture are condensed in thepresented 5G SDN/NFV experimental platform for testingadvanced end-to-end IoT and mobile services 5. In this paper,we present in detail three different use cases available for real-life demonstration in our platform: End-to-End SDN Orchestration of IoT Services Usingan SDN/NFV-enabled Fog Node Hierarchical SDN Orchestration of Wireless and Op-tical Networks Integrated SDN/NFV Orchestration for the DynamicDeployment of Mobile Virtual Backhaul NetworksFig. 1: ADRENALINE Testbed for 5G servicesThis paper is organized as follows, section 2 introduces theimplementation details of the CTTC 5G experimental tested;section 3 includes the detailed description of the three 5G usecases available for real-life demonstration in our testbed; andnally section 4 summarizes the conclusions and future work.II.
EX P E R I M E N TA L S E T U P D E S C R I P T I O NThe cloud computing platform and transport network ofthe ADRENALINE Testbed (Fig. 1) is composed by theCloud/Fog infrastructure, the intra-DC networks, the multi-domain heterogeneous Wireless/Optical networks and the con-trol/management planes, all physically installed at CTTCpremises in Castelldefels (Barcelona, Spain). The presenteddemonstrations exploit the ADRENALINE testbed systemfeatures, as well as they also integrate EXTREME testbed(SDN-enabled wireless domain) and IoTworld testbed (basedon wireless sensor networks).For the cloud/fog computing platform, we have deployedOpenStack Liberty into Commercial Off The Shelf (COTS)servers. Two availability zones have been dened in order toemulate distributed DC locations, which are interconnected asdepicted in Fig. 1. The Fog/Edge node has been implemented using an Intel Next Unit of Computing (NUC) on top of whichis deployed an OpenStack compute node instance running ina third availability zone.
For the intra-data center network, OpenFlow switches havebeen employed. All inter-DC trafc is aggregated to the corethrough the access/metro segments which are composed ofOpenFlow 1.4 switches deployed on COTS hardware withseveral 1G NICs implemented by xDPD software switch. EachMetro/Core border node includes a 10 Gb/s XFP tunabletransponder interface. Both the intra-data center networks, andthe access/metro segments are controlled with OpenDayLight(ODL) SDN Controller, Hydrogen Service Provider releaseinstances using OpenFlow.
The inter-data center interconnection traverse across thecore GMPLS-controlled optical network segment. This is com-posed of an all-optical WSON with 2 ROADMs and 2 OXCsproviding re-congurable (in space and in frequency) end-to-end lightpaths, deploying a total of 610 km of G.652 andG.
655 optical ber, with six DWDM wavelengths per opticallink. The optical SDN controller is responsible for the inter-data center network connectivity and it has been implementedfollowing the Active Stateful Path Computation Element (AS-PCE) architecture.Optical SDN ControllerE2E SDN Orchestrator (pABNO)LTE/WiFi/mmWav e EXTREM E Te stbe d ADRENALINE Te stbe dM e tro M PLSDWDM Cor eSDN ControllerUESDN ControllerllllerererererererererererererererererererererererererererererererCoCoCoCoCoCoCoCoCoCoCoCoCoCoIoTworldIoT CO2 WSN IoT Heat WSNIoT GW 1IoT GW 2Core DCEdge SDN CtlEdge Fog CtlCore DC SDN CtlCore DC Cloud CtlCloud/FogOrchestratorSDN Transport Orchestration(cABNO)Wireless SDN ControllerDN Network HypervisorSDN IT and Network Orchestrator (SINO)NFV OrchestratorVNF Manager #NvEPC ManagerAcce s s /Metro MPLSMMES-PGWHSSvEPCFig. 2: 5G Cloud/Fog and SDN/NFV orchestratorFig. 3: a) Message exchange for hierarchical SDN orches-tration; b) Network Topology abstraction at the E2E SDNorchestrator levelFig. 2 shows the user interface for the overall 5G Cloud/Fogand SDN/NFV orchestrator software stack. It is composed ofthe NFV orchestrator, several VNF managers, an SDN IT andNetwork Orchestrator (SINO), a network hypervisor and an hi-erarchical SDN orchestrator. The 5G Cloud/Fog and SDN/NFVorchestrator software stack has been developed entirely inPython by CTTC and it is able to provide multi-tenancyover SDN/NFV-enabled multi-vendor multi-technology net-work and cloud/fog computing resources.
III. 5G SDN/NFV E X P E R I M E N TA L P L AT F O R MD E M O N S T R AT I O N SFollowing subsections include the architectural and func-tional description of the 5G SDN/NFV experimental platformthrough the three different use cases presented in this paper.A. Hierarchical SDN orchestration of Wireless and OpticalNetworksHierarchical SDN Orchestration has been proposed asa feasible solution to handle the heterogeneity of differentnetwork domains, technologies, and vendors. It focuses onnetwork control and abstraction of several control domains,whilst using standard protocols and modules. The need ofhierarchical SDN orchestration has been previously justiedin 7 with two basic purposes: the ability to scale and theincrease of security.Fig. 4: E2E conectivity provisioning workow between IoTgateway and virtual machine running at Edge nodeIn this use case, the hierarchical SDN Orchestration isapplied for the integration of wireless and optical transportnetworks 8, to provide E2E connectivity between the UserEquipment (UE) and a cloud service deployed in the Core DClocation.
In the wireless segment (Fig. 1), implemented overthe EXTREME Testbed, an SDN controller is in charge of theprogramming of the wireless network (access and backhaul).This SDN controller tackles the specics of the wirelessmedium, implementing the proper extensions to control wire-less devices. In the optical segment, implemented over theADRENALINE Testbed, we consider an SDN-enabled MPLS-TP aggregation network, while the control of the core networkrelays on an AS-PCE over a GMPLS distributed control plane.A parent E2E SDN Orchestrator, based on the IETFABNO architecture in 9 (pABNO), is responsible of theE2E provisioning across the different network segments. ThepABNO orchestrates several network segments: an SDN-enabled wireless segment and the MPLS/Metro and Corenetwork segments orchestrated by a child ABNO (cABNO).The cABNO is responsible for abstracting the multi-domaintransport segments, and it offers a simplied view to thepABNO, thus improving scalability and security.
Fig. 3.a shows the orchestration workow. It can be ob-served that an E2E connection is requested (POST Call) tothe pABNO. The pABNO computes the involved network con-trollers (Wireless SDN/cABNO) and requests the underlyingconnection to them. We can observe how the workow followsinside a cABNO, which is responsible for another level ofhierarchy of the SDN orchestration process.
B. End-to-End SDN Orchestration of IoT Services Using anSDN/NFV-enabled Fog NodeSDN is a key technology to address all the technical net-working challenges posed by the IoT. SDN aims to overcomethe limitations of traditional IP networks, which are complexand hard to manage in terms of network conguration andreconguration due to faults and changes. An SDN controllercan be viewed as a network operating system which interactswith the data plane and the network applications by means ofApplication Programmable Interfaces (APIs).
In this regard,also the different needs in networking resources such asbandwidth and delay can be managed more easily thanks to thesoftware programmability approach facilitated by SDN in thenetwork control. Another important benet of SDN is that itpaves the way for the integration of smart objects with fog andcloud computing. More specically, thanks to the exibility ClientOptical SDN ControllerCall ACKPOST CallcABNOpABNOPOST CallEstablish FlowsPCInitiate {src,dst}PCRpt{LspId}Flows ACKCall ACK…Establish FlowsEstablish FlowsFlows ACKFlows ACKSDN CTL 1 W ireless SDN CTLSDN CTL 2AP1… FLOW_M OD Request data processingRequest data processingSINOSDNOrche stratorEdgeSDN CTLSDSDSDSDSDSDSDSDSDSDSDSDSDSDSDSDRequest VMVM AckPOST CallCall ACKDataCreate Flows Flows ACKRequest VM Fog VMVM ACK Cloud/FogOrche stratorIoT GWSDN/NFV Edge NodeCloud VM infoCloud VM info1234Fig. 5: Setup delay. (left) Edge node, (right) Core DCprovided by SDN, the data ows of information between IoTnodes and fog or cloud computing can be easily managed.
Thisenables collaborative analytics between geo-distributed smartthings.Integrating IoT and SDN can also increase the efciencyof the network by responding to changes or events detectedby the IoT which might imply network reconguration. Forexample, SDN can be used in IoT applications where the dataare transmitted from the sensors periodically in specic timeframes to schedule the requested bandwidth on the transmis-sion paths only during the active duty cycles. Such dynamicreconguration of the forwarding devices is only possiblevia centralized applications which orchestrate IoT collectedinformation and network resources information jointly.We have deployed an SDN/NFV-enabled Edge Node inADRENALINE Testbed for integrating wired IoT gatewaysfrom the IoTWorld Testbed by means of E2E SDN Orchestra-tion of integrated Cloud/Fog and network resources 6. E2ESDN orchestration provides network connectivity between IoTgateways and deployed virtual machines (VMs) which mightbe allocated in the proposed edge node or in a DC located inthe core network.The SDN IT and Network Orchestrator (SINO) is re-sponsible for handling Virtual Machine (VM) and networkconnectivity requests, which are processed through the Cloudand SDN orchestrators. The orchestration process consists oftwo different steps: the VM creation and network connectivityprovisioning (see Fig.
4). The SINO requests the creationof virtual instances (VMs) to the Cloud Orchestrator, which,is responsible for the creation of the instances. It is alsoresponsible to attach the VMs to the virtual switch inside thehost node (at the edge node or in a core DC). When the VMscreation is nished, the Cloud Orchestrator replies the VMsnetworking details to the integrated Cloud/Fog and networkorchestrator (MAC address, IP address and physical computingnode location). The SDN orchestrator is the responsible toprovision E2E network services. The SDN orchestrator willprovide the E2E connectivity between the requested IoT gate-way and the deployed VM. Finally, data from IoT gatewaywill ow to the processing resources located in the proposedSDN/NFV-enabled edge node.E2E connectivity setup delay has been measured 100 timesbetween IoT GW and edge node (Fig.
5.a) or core DC (Fig.5.b). The histograms and CDFs are showed. In average thesetup delay between the IoT GW and the edge node is 456ms, while towards the core DC is 4070 ms, due to the factthat a bidirectional optical lightpath needs to be dynamicallyestablished.Fig.
6: Multi-layer aggregation network connecting RANs andDCs and abstracted view of the backhaul networks per MNO(left).C. Integrated SDN/NFV Orchestration for the Dynamic De-ployment of Mobile Virtual Backhaul NetworksIn order to cope with the augmenting data trafc, MNOsexpect that virtualization of network functions (NFV) andinfrastructure (SDN) are appealing to obtain a more scalable,cost-efcient and exible MNO deployment, in particular,in the backhaul infrastructure. We assume that a number ofMNOs owning their radio area network (RAN) are connectedto a common physical multi-layer (packet and optical) ag-gregation infrastructure.
This shared physical infrastructure ispartitioned to compose individual virtual backhaul tenants ontop of it. Furthermore, the MNO Evolved Packet Core (EPC)functions are as well virtualized into the cloud connected tothe aggregation network. This model enables MNOs to exiblyadjust their virtual backhaul and EPC necessities to the actualtrafc loads.This demonstration use case experimentally assess thedynamic computation and automatic deployment of a MNOvirtual backhaul along with a virtual EPC (vEPC) 10. The5G Cloud/Fog and SDN/NFV orchestrator coordinates thevirtualization of heterogeneous transport technologies withinthe aggregation segment as well as compute cloud resourcesat the DCs.
Fig. 6 shows the physical multi-layer aggregation networkto connect MNO’s RAN and DC domains wherein virtual SDNcontroller (vSDN) and vEPC are instantiated. The aggregationnetwork leverages the statistical multiplexing provided byMPLS packet switching and the huge transport capacity ofoptical switching applying multi-layer grooming techniques.An MNO creating/increasing its backhaul capacity is builtupon the aggregation network as interconnected virtual packetdomains. The MNO SDN controller’s vision is an abstractionof a set of connected packet domains (via an optical connec-tion) providing the connectivity between the RAN and vEPC.Each abstracted packet domain is represented by a virtualpacket node whose interfaces are mapped to the physicalincoming/outgoing links of a packet ow.The network topology and packet resource status are used VEPC1VM NOVS…OVSvSDN1OFSOFSOFSvEPC2VM NOVS…OVSvSDN2Aggre ga tion MPLS pa cke t doma inEnB VMNO2EnB VMNO1Optica l doma inMPLS Pa cke t Core Doma inDC infra structurePhysical TopologyOFSAB CD EF GMMES-PGWHSSS1-MMEovsAbstracted view of vMNO1vEPC 1S1-UvSDN1OFP OFPOFPAbstracted MPLS nodeAC EGHMMES-PGWHSSS1-MMEovsAbstracted view of vMNO2vEPC2S1-UvSDN2OFP OFPOFPAbstracted MPLS nodeB C FHAbstracted MPLS nodeAbstracted MPLS nodeFig. 7: Workow for provisioning MNO virtual backhaulnetwork and VNFsto dynamically set up packet MPLS tunnels for backhaulingupcoming mobile LTE signaling and data bearers (i.
e., S1-MME and S1-U interfaces) between the RAN and vEPC 12.The vSDN controller for the virtual backhaul is provided as aVNF in the DC. Last but not least, the connectivity within theDC network is virtualized connecting the core packet domainand the deployed cloud VNFs.Fig. 7 shows the workow between the involved functionalblocks of the SDN/NFV orchestrator to manage the creationof an SDN-controlled virtual backhaul and the correspondingvEPC.Step 1 allows the NFV orchestrator to request the pro-visioning of the vSDN controller (for the virtual backhaul)and the vEPC. This is handled by the corresponding VNFmanagers sending requests to the Compute controller of VMswith the respective implementation (image) of the VNFs(vSDN and vEPC).
Next, in step 2, the creation of the MNOvirtual backhaul is conducted allowing the connectivity of thecreated vSDN controller to congure such an infrastructure.To do that, the MNH receives the request and computes thedomain sequence within the aggregation network in order toconnect at the packet level the MNO RAN and the vEPC.This requires that at rst the traversed packet domains areinterconnected via an optical connection which is triggeredby the SDN orchestrator. When the optical connection is setup (by the network hypervisor 11) at the packet level allthe domains are interconnected. For those packet domainsthe SDN Orchestrator subsequently requests the packet owprovisioning specifying ingress/egress links of those domainsto derive the abstracted (virtual) packet node forming thetargeted virtual backhaul.Finally, a L2 ow in the DC infrastructure (e.g.
, Ethernet)is created to connect the virtual (MPLS) node with the vEPC.Once the virtual backhaul connectivity is ready, this is notiedto the NFV orchestrator, and at that time, the vSDN has a viewof the virtual packet backhaul used to transport LTE bearersbetween the RAN and the vEPC. IV. CO N C L U S I O NConducting real-life demonstrations of an end-to-end 5Gscenario including both IoT and mobile broadband services,requiring the integration of heterogeneous wireless access andoptical transport networks, distributed cloud computing, andwireless sensor and actuators networks is a very challengingtask.CTTC has been working on the development of the rst-known end-to-end 5G platform capable of reproducing suchan ambitious scenario. This paper has described the exist-ing demonstrations, supported functionalities, use cases, andpreliminary results among the different experimental facilitiesavailable at CTTC.
Further research will consist on introducing service func-tion chainning in the ADRENALINE testbed, as well as opti-mal resource allocation algorithms which based on constraints(e.g., latency) decide the optimal allocation of virtualizednetwork functions.AC K N OW L E D G M E N TThis work was partially funded by EU FP7 STRAUSS(FP7-ICT-2013-EU-Japan 608528), EU FP7 COMBO(317762), and Spanish MINECO project DESTELLO(TEC2015-69256-R).RE F E R E N C E S1 5GPPP white paper, the 5G Infrastructure Public Private Partnership: the next generation of communication networks and services, March2015,2 A. Mayoral, et al., “First experimental demonstration of distributed cloud and heterogeneous network orchestration with a common Trans-port API for E2E service provisioning and recovery with QoS”, in Proc.
OFC 2016 , Anaheim (CA), USA.3 Network function virtualization (nfv): Architectural framework, ETSI GS NFV 002 v.1.1.1,, 2013.
4 Mobile-Edge Computing – Introductory Technical White Paper, ETSI MEC ISG, September 2014.5 R. Mu ˜noz, et al., The CTTC 5G end-to-end experimental platformintegrating IoT, SDN, and distributed cloud , in Proceedings of WirelessWorld Research Forum Meeting 35 (WWRF), 14-16 October 2015,Copenhagen (Denmark).6 Ricard Vilalta, et al., “End-to-End SDN Orchestration of IoT Services Using an SDN/NFV-enabled Edge Node”, in Proc.
OFC 2016, Anaheim(CA), USA.7 R. Vilalta, et al., Hierarchical SDN Orchestration for Multi-technology Multi-domain Networks with Hierarchical ABNO, ECOC 2015.8 R. Vilalta, et al.
, “Hierarchical SDN Orchestration of Wireless and Optical Networks with E2E Provisioning and Recovery for Future 5GNetworks”, in Proc. OFC 2016, Anaheim (CA), USA.9 D. King, and A. Farrel, “A PCE-based Architecture for Application- based Network Operations”, IETF RFC 7491, 2015.10 R.
Martinez, et al., “Integrated SDN/NFV Orchestration for the Dy- namic Deployment of Mobile Virtual Backhaul Networks over a Multi-layer (Packet/Optical) Aggregation Infrastructure”, in Proc. OFC 2016,Anaheim (CA), USA.11 R. Vilalta, et al.
, “Multi-Tenant Transport Networks with SDN/NFV”, IEEE/OSA Journal of Lightwave Technology, Vol. 34, No. 8, 2016.12 R. Martinez, et. al.
, “Experimental Validation of a SDN Orchestrator for the Automatic Provisioning of Fixed and Mobile Services”, in Procof ECOC 2015.NFV orch.vSDNmnger.
vEPCmnger.SINOCloud CtlNetwork HypervisorSDN Pkt D1SDN Opt. D2SDN Pkt D3SDN DC D4VM req w/ SDN ctrler imageVM rep w/ IP & MAC addressVM req w/ EPC imageVM rep w/ addressing of EPC elements (MME, SGW/PGW, et c.
Creation of the VMs for the vSDN ctrlr and vEPC1Req for MNO virtual backhaul w/ SDN ctrlr IP addressReq for pkt flow on D1 b/w in/out ports Rep pkt flowReq for opt. tunnel on D2 between in/out portsReq for creating an opt. connectionRep created opt. connectionRep for opt. tunnel on D2Req for pkt flow on D3 b/w in/out ports Rep pkt flowReq for pkt flow towards the vEPCRep pkt flowCreation of MNO virtual backhaul 2All the requests are duplicated to allow bidirectional connectivityRep MNO virtual backhaul Req for e2e connectivity D1, D3 and D4 w/ in/out portsRep for e2e pkt connectivity SDN Orchestrator
