Nexus Data Center Switches (NX-OS)

The Cisco Nexus switches allow a very high level of bandwidth connectivity positioned as data center switches. The Nexus switches do not support the same features and flexibility as local area network (LAN) based switches, but provide the very high capacity and cost effectiveness needed in data center environments. Nexus switches run a new operating system called Cisco NX-OS and provide various Ethernet based interface speeds varying from 100Mbps to 10Gbps depending on the Nexus platform. The Cisco Nexus product line currently consists of the following product families:

  • Nexus 1000
  • Nexus 2000
  • Nexus 5000
  • Nexus 7000

All Nexus platforms run the Cisco NX-OS operating system that is very familiar to Cisco IOS, but offers much more flexibility and reliability than Cisco IOS. The Nexus 1000 supports many virtualization technologies that are in VMWare ESX platforms and VMWare’s newer vSphere technologies. The Nexus 2000 and 5000 platforms provide layer 2 connectivity, while the Nexus 7000 provides layer 3 connectivity.

This blog will focus on a deployment model leveraging the Nexus 5020 and Nexus 2148 devices. The Nexus 5000 platforms provide non-blocking backplane architectures with onboard capacity of twenty (5010) or forty (5020) 10 Gigabit Ethernet interfaces with one to two expansion slots respectively. The expansion modules have the ability to provide additional 10 Gigabit interfaces and fiber channel over Ethernet (FCoE) connectivity. The first 8 port (5010) or 16 (5020) ports of the 5000 platform can be configured to provide 1 Gbps connectivity using small form-factor pluggable (SFP) optics, while all the ports can provide 10 Gbps connectivity over SFP+ optical interfaces. The Nexus 5000 platforms support connectivity to the Nexus 2148 through a fabric extender (fex) configuration as of NX-OS 4.0(1a).

The Nexus 2148 platform does not have a local console port or any way to directly configure the device. The Nexus 2148 data center switch is managed through the Nexus 5000 configuration. The Nexus 2148 appears as a line card in the configuration of the Nexus 5000 after the fabric extender interface is properly configured and comes online. The Nexus 2148 downloads NX-OS software from the Nexus 5000. After the 2148 is online, 48 new configurable interfaces appear in the Nexus 5000 configuration.

The Nexus 2148 platform provides 48 Gigabit Ethernet interfaces and up to four fabric extender (fex) interfaces to the Nexus 5000 platforms. The Nexus 2148 supports up to four 10 Gigabit Ethernet (GE) fabric extender interfaces to the Nexus 5000 platform, but the Nexus 2148 can only uplink to one Nexus 5000 with NX-OS 4.0(1a). NX-OS 4.1(3) supports redundant uplink connections to two different Nexus 5000 switches providing interface and path redundancy.

IEEE 802.1d Spanning Tree Protocol (STP) is enabled on the Nexus 5000 by default and all redundant paths between the Nexus 2000 and 5000 are placed in the blocking state by STP. Two channel groups can be configured to each Nexus 5000 which will result in two forwarding (channel group 1) and two blocking interfaces (channel group 2) which still results in sub-optimal bandwidth utilization. See comments below for correction.

Virtual port channel (vPC) technology introduced with Nexus 5000 NX-OS 4.1(3) allows the 2148 platform to use all four interfaces uplinked to the Nexus 5000. vPC technology is similar to the virtual switching system (VSS) technology introduced in the Catalyst 6500 VSS supervisor module which is similar in nature to multi-chassis Etherchannel (MCEC). All the above technologies allow the uplink switch to appear as one virtual switch to the downstream devices. Loop avoidance is provided through a vpc link interface between the Nexus 5000 switches which requires a special configuration. Path recalculation is performed in under one second on the Nexus platforms in the event of a link or device failure.

References:
Data Center Switches

Cisco Nexus 5000 Series Switches

Special thanks to Guest Author Dennis Hartmann for this post.

In this article

Join the Conversation

6 comments

  1. Brad Hedlund Reply

    Dennis,
    Nice article. One point of clarification though… the Nexus 2148 has no Spanning Tree control plane. Spanning Tree is not the mechanism that blocks uplinks connected to a second Nexus 5000. Rather, the second Nexus 5000 learns from the Nexus 2148 through SDP (satellite discovery protocol) that it is already connected to another Nexus 5000 and prevents the connection.
    The link between the Nexus 2148 and Nexus 5000 is not a Layer 2 link where Spanning Tree needs to prevent loops, rather it’s a Layer 1 fabric extension negotiated by SDP.

    Cheers,
    Brad

    1. mark cohen Reply

      nice post ..
      but lets be open..
      because this sounds like marketing ..
      add the following ..
      the nexus 2000 is not like a switch it is useless outside the fabric..don’t expect a full featured switch.
      you will find that there are many things you cant do with the platform ..

      nexus 5k great , nexus 7k great , nexus 2000 sucks

      1. Dennis Hartmann Reply

        What limitations did you run into? The Nexus 5000 is definitely a better platform for 10 Gigabit Ethernet, but cannot deliver many Gigabit Ethernet connections. A Unified Ethernet based fabric will eventually connect all of our devices.

    2. Dennis Hartmann Reply

      Good points of clarification Brad. Thank you very much for joining the conversation and sharing your expertise.

      Brad is 100% correct. The Nexus 2148’s FEX functionality is a Cisco pre-standard implementation of the IEEE 802.1Qbh bridge port extension standard: http://www.ieee802.org/1/pages/802.1bh.html

      The Nexus 2148 is a replacement for TOR (top of rack) switches that normally connected to server equipment with Gigabit Ehternet connections. The TOR switch normally sends traffic to an EOR (end of rack) switch via 10 Gigabit Ethernet to an aggregation switch like the Catalyst 6509. The Nexus increases this capacity 4 fold (40Gbps) with virtual port channel (VPC) technology.

      TOR/EOR network designs are not entirely going away. The TOR and EOR switch with now be connected by the fabric (backplane) of the EOR switch. The Nexus 5000 can deliver up to 40Gbps to the Nexus 2148 creating a near non-blocking architecture. If the Nexus 2148 is limitted to 40 10GE connections, a non-blocking architecture can be guaranteed.

      Here’s some of the advantages:

      Manageability: Every TOR switch is managed from one console as if it was a line card on a Catalyst 6000.

      Bandwidth: Nexus 2148 delivers up to 48 Gigabit Ethernet ports. The Nexus 5000 can be used in the case of 10GE.

      Cost: The cost of the Nexus 2148 is lower than a comparable switch and the potential of spanning tree protocol bringing down your network are minimized.

      ROI: The Nexus switch solution is a no brainer for management because they can easily quantify the ROI based on energy and power conservation alone.

      The GREEN data center should NOT be sales & marketing hype!

      If we’re going to continue to be the technology leaders in the world, we need to upgrade our infrastructures! I’m proud to say that Verizon FIOS is available in my area and I just upgraded my Internet access to 30Mbps… but ONLY because I’m trying to stimulate the economy. 🙂 Read the bottom of this blog if you want to read about my speed tests on FIOS with this new access.

      The energy savings are substantial which makes it quite easy to come with an ROI model for CAPEX justification. Energy costs will continue to rise unless we invest in our energy infrastructures and conserve what we currently have.

      As the cost of energy rises, it means budgets get cut… and that’s not good for anyone… at a personal level, the more money energy costs, the less money you have to buy a new laptop, car, or Randy Rhoads RR1 flying V guitar (made in USA).

      So for the sake of humanity… will you help me get a new Randy Rhoads flying V guitar… just kidding! Replace your power hungry data center network equipment with faster, leaner equipment that can perform more work using less energy… you’ll make management happy with ROI… you’ll make the nerds/geeks happy because they have new toys to play with… you’ll make everyone happier because they can get their job done early on Friday and go play some golf… or snowboarding this time of year.

      Verizon FIOS Story

      I still CANNOT believe that I have 30Mbps to my house! This is NERD NIRVANA! When moving from Woodbridge, NJ to Fishkill, NY we were moving from an apartment to a house, but… we also went from a 3Mbps Inernet link with Comcast to a dialup modem at 26.4kbps… these were the dark days of technology my friends… the pain! the torture! the frustration… I cannot begin to tell you.

      I hung in there for one solid year and then Starband two-way Internet satellite came out! whew! I’m saved… for only $600- (CAPEX) and $60- per month (OPEX). It hurt, but we dealt with the financial pain and got connected. The download speeds were much better, but the DELAY was painful. In theory… the propagation delay of light to a geo-synchronous satellite and back down to earth from the satellite (bi-direction) is 250ms (milliseconds or 1/1,000th of a second). In practical application, you would wait five seconds for each click of the mouse… but the downloads had throughput over 1Mbps.

      Cablevision ran fiber down our road and about 18 months after satellite Internet we moved to cable… I forgot to mention the size of the dish. It was about 3 times the size of a standard dish. Ugly with a capital U! Looked like I was launch rockets out of my back yard.

      Now I’m connected at 30Mbps… that’s 2/3 of a T3 circuit (44.736Mbps)… To my house!!!! Did I mention that it’s a symettrical link (30Mbps up/30Mbps down)?

      Unfortunately my upload speeds from http://www.speakeasy.net only measure 8 or 9Mbps up on each test… but the download speed measures over 34Mbps!!!! I’m thinking that there’s no bandwdith filter whatsoever because I thought 30Mbps was the theoretical limit of RG-6 coaxial cable.

      This got me thinking about DOCSIS standards, so I checked out the following link:

      http://en.wikipedia.org/wiki/DOCSIS

      DOCSIS standards keep on finding a new coding scheme to push that bandwidth up though. DOCSIS 2.0 delivers up to 42Mbps, while the 3.0 standard uses the same rate, but can allocate that bandwidth per channel. The standard says that the equipment must have at least 4 channels to be DOCSIS compliant. That’s 4 x 42Mbps or 168Mbps (just like 3DES, but that’s (56bit X 3).

      1. Prabhu Reply

        we need to connect cisco3020 switch(HPC enclosure) with Nexus2148T.Is port channeling option available? ..we are not able to confgure to commnication between this two devices.what are all options is there to configure and making it work

  2. Dennis Hartmann Reply

    The Nexus 2148T is configured as a line card in the Nexus 5000. Use the Etherchannel section of the Nexus 5000 Layer 2 Switching Configuration Guide:

    http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/Cisco_Nexus_5000_Series_NX-OS__chapter7.html

    Etherchannel is a terminology that Cisco used for their pre-standard implementation of the IEEE 802.3ad Link Aggregation Control Protocol (LACP). Etherchannel used port aggregation protocol (PaGP). NX-OS on the Nexus 5000 uses LACP, but oddly the technology is still referred to as Etherchannel.

    There are a lot of configuration examples and TechNotes at Cisco’s website as well. These configurations vary based on switch platform, but they I thought they may help. Good luck.

    http://www.cisco.com/en/US/tech/tk389/tk213/tech_configuration_examples_list.html