Category: Ethernet

Etherchannel: Port Channel Discovery

When ports are added to a bundle the following configuration items must be identical:

  • Same speed and duplex
  • Same operating mode
  • If not trunking, same access VLAN
  • If trunking same trunk type, allowed VLANs and native VLAN
  • Each port in a port channel must have the same STP cost per VLAN on all links in the port channel
  • No ports can have SPAN configured

When a port channel is made an interface is also automatically added to the configuration, this interface inherits the configuration of the port it was added to.  If member port configurations differ the port channel will be in a suspended state and will not become working until the member whose configuration is different is corrected and identical to the port channel interface.

Any interface commands configured on the port channel interface are pushed down to the member links for conformity.  Here are the recommended guidelines for configuring port channels

  • do not create the interface port channel manually before bundling the ports, let the switch create it and populate its config automatically
  • When removing a port channel make sure to manually remote the interface port channel from the running config so that it’s config does not cause issues when a port channel with the same number may be created later
  • be sure to make the config of the physical ports identical before adding them to the port channel
  • if the physical ports config differs from the port channel interface config correct the port config first, only then proceed to perform changes to the port channel interface config.
  • port channel interface can either be a layer 2 or layer 3 depending on the physical bundled ports are configured for layer 2 or layer 3.  If needed you can change the type of interface it is after you’ve already created it.
  • Whenever resolving issues with err-disabled ports under a port channel be sure to shut down both the physical and port channel interfaces, then try to reactivate them, unbundling them and then recreating the port channel.


Etherchannel: Load Balancing

Etherchannel increases the available bandwidth by carrying multiple frames over multiple links.  A single frame is always transmitted across a single link in an Etherchannel bundle when traffic enters a switch a hashing function is performed on the address fields in the frame producing a number that identifies which link the frame will transmit over.

The sequence of frames having the same value in the address fields fed into the hashing function is called a conversation or a flow.  The hashing function is deterministic, meaning that all frames in a flow will produce the same hash value, therefore forwarding of the frames in a flow will all traverse a single link and will not be per packet forwarded across multiple links.

This allows for multiple flows to traverse multiple links increasing higher aggregate throughput.  This also prevents frames from being received out of order.

You want to ensure there is a mix of frames sent over a port channel, usually access ports are sending frames from hosts who are transmitting only to their default gateway, and vice versa on return traffic, gateway sourced traffic is being sent back to the unique hosts.  To mix this up you could balance based on source MAC address for outgoing traffic, and then balance using destination MAC on the return traffic, so there are unique flows generated for the hash value and its not all the same going over a single link.

The maximum number of member links in a bundle is 8, the hashing function produces a 3 bit result in the range of 0-7 whose values are assigned to individual member links.   If there are fewer physical links some of the links will be assigned multiple values from this range and ultimately those links will carry more traffic than others as a result.

Traffic will only be distributed evenly across multiple links in a bundle when the number of links equals 8, 4, or 2.



Etherchannel is also referred to as link aggregation which is used to bundle multiple physical Ethernet links interconnecting devices into a single logical communication channel with increased throughput.

After Etherchannel has been established, it creates a single logical interface that utilizes the bandwidth of all the member links in the bundle.  This allows traffic to be load balanced across the multiple member links to take advantage of the combined bandwidth.

Also if links in an Etherchannel bundle fail, traffic is spread out over the remaining up and active member links in the bundle without taking the logical interface down.


PPPoE: Point to Point Protocol over Ethernet

Point to Point Protocol over Ethernet (PPPoE) visualizes Ethernet multiple point to point sessions between client hosts and an access concentrator….essentially turning a broadcast Ethernet domain into a point to multi-point environment.

The PPPoE client feature in IOS allows the router as opposed as an end user host workstation to serve as the client in the network.  This allows entire LANs to connect to the internet over a single PPPoE connection terminated to a router.

PPP interface IP addresses are assigned using an upstream DHCP server and the IP Configuration Protocol (IPCP), which is a sub protocol of PPP.  IP address negotiation must be enabled on the dialer interface in the router for it to obtain an IP address.

PPPoE also introduces an additional 8byte transport overhead, (2 bytes for the PPP header and 6 bytes for PPPoE)  in order to adjust for this in the 1500 byte MTU, you have to decrease the MTU to 1492 bytes so the entire encapsulated packet fits within the 1500 byte Ethernet frame.  For TCP sessions the Max segment size is reduced to 1452 this allows for 40 bytes in TCP and IP headers and 8 bytes in the PPPoE totaling 1500 bytes that fit into an ordinary Ethernet frame.

MTU mismatches can prevent a PPPoE connection from establishing or carrying large datagrams, so this is a good place to check when troubleshooting connections.


Ethernet: Virtual Switch System (VSS)

To provide high availability and redundancy on an ethernet LAN, network administrators will usually try to add additional hardware to create multiple layer 2 paths traffic can failover to.

Redundant connections and port density to support redundancy can quickly become a problem with more devices added to a switching fabric.  One method to avoid over complication and network design hurdles is to use a virtual switching technology as a solution.

VSS combines multiple devices into a single logical network element.  VSS manages redundant links in such a fashion that they will be seen by external devices as a single port channel.

This simplifies network configuration and operation by reducing the number of layer 3 routing neighbors and simultaneously providing a loop free layer 2 topology.


VSS interacts with with access and core networks as if it were a single switch.

VSS operates within a role-based model, where one device participating in VSS will be the ‘Active’ switch and the other will be the ‘Standby’ switch.

VSS Active switches controls the VSS running the layer 2 and 3 control protocols for the switching modules on both switches.  the active switch also provides management functions for the VSS such as module online insertion and removal and the console interface.

the VSS Active AND standby switch performs packet forwarding for ingress data traffic on their locally hosted interfaces, however the VSS standby switch also sends control traffic to the VSS active switch for processing.

To act as one virtual network element, the two switches need a method to share control information and data traffic … VSS does this by using what’s called Virtual Switch Link (VSL).  VSL is typically implemented via etherchannel for link redundancy and can support up to eight links in a bundle.  control and management traffic are given higher priority across this link in an effort to ensure the the control and management traffic is never discarded.



Ethernet: SPAN, RSPAN, and ERSPAN

Cisco switches have a feature that allows all traffic from one port to be mirrored and directed to another port.  This feature is called SPAN (Switch Port Analyzer).

There are many reasons for wanting to SPAN traffic from one port to another, it could be for compliance reasons, data collection purposes, or supporting a particular monitoring application.  It can also be used to troubleshoot VoIP or networking issues to verify what data is actually being passed through a port.

SPAN sessions can be sourced from a port or a VLAN, the session then directs the traffic to a destination port or VLAN.  When the destination port is local to the same switch that the SPAN is sourcing from, is referred to as simply a SPAN session.

Sessions that direct traffic to a remote port via a VLAN is referred to as an RSPAN (Remote SPAN) session.  In RSPAN a specific VLAN is configured across the network to carry the mirrored source traffic from the source port to the remote port off a different switch.

The VLAN that’s created for the RSPAN is referred to as the RSPAN VLAN.


Encapsulated Remote Source Port Analyzer (ERSPAN) actually does what it says and encapsulates the source traffic in a GRE header and tunnels the SPAN traffic to a remote layer 3 endpoint through a GRE tunnel.  This is primarily used to send traffic to a remote SPAN port across a layer 3 domain.

A SPAN source port can be any kind of port, on a SPAN VLAN all ports on a VLAN are monitored.  As you add or remove ports on that VLAN ports are dynamically added or removed from the SPAN session.  The destination port that traffic is being directed to cannot be part of the VLAN being monitored.

Other key restrictions include the following:

  • When a port is configured to be a destination port its original configuration is overriden…once the SPAN is removed the original configuration is restored on the port.
  • When a destination port configured is part of an etherchannel bundle, it is removed from the bundle while the SPAN is active.
  • Destination ports do not support port security, 802.1x authentication, or private VLANs.
  • Destination ports do not support any layer 2 protocols including CDP, SPT, VTP, DTP, and so on.
  • The source port can be one or more ports or a VLAN, but not a mix of these.
  • up to 64 SPAN destination ports can be configured on a switch.
  • Be careful to avoid overloading the destination port, for example if you’re monitoring a VLAN with multiple ports in it, and sending all that traffic to a single destination port, could cause problems with congestion at the destination port.
  • A SPAN destination port cannot also be a source port or vice versa
  • Only one SPAN session can be sent to any particular destination ports, you cannot send multiple SPAN sessions to the same destination port.
  • When a trunk is set as the source of a SPAN all VLANs are monitored in the session on the trunk.  You can use the filter vlan command to prune the VLANs you’re uninterested in.
  • Any traffic routed to a souce VLAN that’s being monitored from another VLAN will not be monitored.  Only ports participating in the SPAN VLAN will mirror their traffic to the destination port.

SPAN sessions can support three types of traffic: transmitted, received, or both.

  • For receive SPANs, all traffic received on the source port is transmitted to the destination port, no traffic transmitted on the port outbound is mirrored to the destination.
  • For transmit SPANs, all traffic transmitted out the source port is mirrored to the destination port, no traffic received on that port is sent to the destination port in the SPAN.
  • SPAN sessions will usually ignore layer 2 frames such as CDP, BPDUs, or VTP however you can include these frames in a SPAN by configuring the encapsulation replicate command.



Ethernet: Switching Logic

The primary goal of any switch is to forward Ethernet frames to their desired destination based off the MAC address in the destination MAC field of the Ethernet header.  The logic by which a switch forwards the frame is based on whether or not the switch is aware of where the destination MAC address is.

LAN Switch Forwarding Behavior

If the switch knows the MAC Address (Known Unicast) it will forward the frame out the single interface associated with the destination MAC Address.

If the switch does not know the MAC Address (Unknown Unicast) it will flood the frame out all interfaces, except the interface it received the frame on.

If the switch receives a broadcast frame, it will flood the frame identically out all ports.

If the switch receives a multicast frame, it will flood the frame identically to unknown Unicasts, unless multicast optimizations are configured.

Switches learn MAC addresses and the port associated with them by reading the source MAC address of received frames.


Ethernet: Address Formats

In order for a MAC address to be unique the IEEE assigns each vender a code to use as the first 3 bytes of the MAC.  This is called the Organizationally Unique Identifier (OUI).  Along with the OUI the vender then assigns a unique value in the low-order 3 bytes for each Ethernet card  that it manufactures, thereby ensuring global uniqueness of MAC addresses.


IEEE documentation lists the Ethernet addresses with the most significant byte on the left.  Inside each byte, the leftmost bit is the most significant bit and the rightmost bit is the least significant bit.  This is referred to as the canonical bit order.

I/G bit – binary 0 means that the address is a unicast, binary 1 means that the address is Multicast or Broadcast

U/L bit – Binary 0 means that the address is vender assigned, Binary 1 means that the address has been administratively assigned, overriding the vender assigned address.

Ethernet: Addressing

There are three types of Ethernet MAC Addresses:

  • Unicast – An address that represents a single LAN interface.  The I/G bit is set to 0.
  • Broadcast – An address that represents ‘all devices on a LAN’ Always a value of hex FFFFFFFFFFFF
  • Multicast – A MAC address that implies some subset of all devices on a LAN.  The I/G is set to 1.

When an Ethernet NIC is aware of the specific MAC address it intends to send frames to, it places the specific Unicast MAC address of the device in the destination field of the header and forwards the frame to that single device only.

When an Ethernet NIC needs to send traffic to the Broadcast address (FFFF.FFFF.FFFF) that Frame will be forwarded to ALL devices on the same LAN/VLAN.

Multicast frames are used to communicate with a dynamic subset of devices on a LAN. For example if only 3 out of 100 network devices on a LAN want to participate and watch a video stream.  These devices interested in participating can do so by listening for frames sent to the specific multicast MAC address.  Other devices that may receive the frame will drop the frame if they are not participating in the group.



Ethernet: Framing

The original Ethernet specification was owned by Digital Equipment Corp, Intel, and Xerox..hence the name Ethernet DIX.


Later in the early 1980’s the IEEE standardized Ethernet, defining parts of Layer 1 and some Layer 2 in the 802.3 Media Access Control (MAC) standard, and other parts of Layer 2 in the 802.2 Logical Link Control (LLC) standard.


Later the IEEE realized that the 1 byte Destination Service Access Point (DSAP) field in the 802.2 LLC header was too small.  The IEEE then introduced a new frame format with the Sub-Network Access Protocol (SNAP) header after the 802.2 header.


Descriptions of each of the fields in the header:

  • Preamble (DIX) -Allows synchronization and signal transitions for proper clocking of the transmitted signal.  This header contains 62 alternating 1s and 0s, and ends with a pair of 1s. ex: 10101010 10101010 10101010 10101010 10101010 10101010 10101010 10101011
  • Preamble and Start of Frame Delimiter (802.3) – Also used for synchronization and signal transitions for clocking, however 802.3 simply renames the 8byte DIX preamble as a 7byte preamble with a 1byte Start of Frame Delimiter (SFD).
  • Type (or Protocol Type) (DIX) – 2 byte field that identifies the type of protocol or protocol header that follows the header.  This allows the receiver of the frame to know how to process a received frame.
  • Length (802.3) – This field describes the length in bytes of the data preceding the Length field up to the Ethernet trailer.  This allows a receiver to know when the end of the frame is received.
  • Destination Service Access Point (DSAP) (802.2) – This is a 1 byte protocol type field.  the size limitations of a 1 byte field along with other uses of low order bits required the later addition of SNAP headers.
  • Source Service Access Point (SSAP) (802.2) – This is a 1 byte protocol tyupe field that describes the upper layer protocol that created the frame.
  • Control (802.2) – This is a 1 or 2byte field that provides mechanisms for connectionless and connection oriented operations.  Modern protocols generally only use this for connectionless operation with a 1byte value of 0x03
  • Organizationally Unique Identifier (OUI) – this is a 3 byte field used to determine the manufacturer of a network device (ie: Ethernet NIC)
  • Type (SNAP) – This is a 2byte field using the same values as the DIX type field creating a larger space with increased number of bits and use of the DSAP field.