Skip to content

2.5gbe-migration.md

Latest commit

 

History

History
128 lines (96 loc) · 4.35 KB

2.5gbe-migration.md

File metadata and controls

128 lines (96 loc) · 4.35 KB
Raw

2.5 GbE Network Migration Guide

Overview Migrate all Proxmox hosts, MAAS, OPNsense, k3s VMs, and Frigate VM from 1 GbE to 2.5 GbE with zero IP changes. This guide includes lessons from missteps.

Prerequisites

  • Cable to 2.5 GbE Unifi switch and uplink to 1 GbE switch remaining for migration
  • USB 2.5 GbE adapters plugged into each Proxmox host
  • Bridges configured: vmbr25gbe on each host

Pre-Migration Setup – Create vmbr25gbe

Before cutting over, create a separate bridge on each host for testing the 2.5 GbE adapters. Do this on pve, still-fawn, chief-horse, and fun-bedbug (skip rapid-civet, currently offline).

  1. GUI → System → Network → Create → Linux Bridge

    • Name: vmbr25gbe (no underscores)
    • Bridge ports: USB adapter (e.g. enx803f5df...)
    • IPv4/CIDR: leave None for now

  2. Apply settings and verify the link comes up:

    ethtool <usb-iface> | grep Speed  # expect 2500Mb/s

  3. Lesson learned: avoid adding the USB adapter to vmbr1. Doing so bridged the existing 1 GbE port and the new 2.5 GbE adapter on the same L2 segment with no spanning tree to break loops. Broadcast and multicast frames multiplied, flooding the bridge. The mDNS service avahi-daemon processed thousands of duplicated packets and spiked to 100% CPU.

Stage 1 – MAAS VM (192.168.4.53)

  1. Shut down the MAAS VM.

  2. Edit its single network device:

    • Bridge: vmbr1 → vmbr25gbe
    • Keep the same MAC and netplan interface name.

  3. Start MAAS, verify:

    ip addr show                 # 192.168.4.53 on ens19
ping -c4 192.168.4.1         # via OPNsense on vmbr1→need OPNsense migration first
ss -ulpn | grep -E ':67|:53|:69'

  4. Lesson learned: moving the NIC breaks netplan if interface names change. Always match on MAC or update netplan YAML accordingly.

Stage 2 – OPNsense VM (192.168.4.1)

  1. Shut down OPNsense VM.

  2. Add a second NIC on vmbr25gbe.

  3. In OPNsense UI:

    • Interfaces → Assignments → add the new port as “LAN.”
    • LAN → Static IPv4 → 192.168.4.1/24 → Save & Apply.

  4. Remove the old vmbr1 NIC.

  5. Verify:

    ping -c4 192.168.4.1
dig @192.168.4.1 example.com

  6. Lesson learned: attempting to add DHCP on the new NIC without removing the old caused dual-listening confusion. Always stage add → test → remove.

Stage 3 – Proxmox Hosts

For each host—pve:192.168.4.122, still-fawn:192.168.4.17, chief-horse:192.168.4.19, fun-bedbug:192.168.4.186—(rapid-civet is currently down):

  1. Configure GUI → System → Network → Create → Linux Bridge:

    • Name: vmbr25gbe
    • Bridge ports: USB adapter (e.g. enx803f5df...)
    • IPv4/CIDR: <host-IP>/24
    • Gateway: blank

  2. Apply and Verify link:

    ethtool <usb-iface> | grep Speed   # 2500Mb/s
ping -c4 192.168.4.1

  3. Remove old vmbr1 cable and interface:

    ip addr flush dev enp3s0
ip link set enp3s0 down

  4. Lesson learned: forgetting to flush the old IP left dual interfaces with the same address. Always clean up stale IPs.

Stage 4 – k3s & Frigate VMs

For each VM:

  1. Shutdown VM.

  2. Edit its Network Device: Bridge → vmbr25gbe, keep MAC.

  3. Start VM, verify same IP, connectivity:

    ip addr show
ping -c4 192.168.4.1

  4. Lesson learned: hot-plugging sometimes failed; safer to shutdown, edit, restart.

Final Cleanup

  1. Physical: remove any remaining 1 GbE uplinks/cables.
  2. Switch: decommission old 1 GbE switch.
  3. Docs: update README.md to reference 2.5gbe-migration.md
  4. Note: Always schedule a 2–4 s maintenance window for bridge reconfig.

Missteps summary

  • Attempted dual‐NIC add without matching netplan interface—broke static IP.
  • Tried adding multiple default gateways—Linux only allows one.
  • Forgot to flush old-IP on physical NIC—caused duplicate-IP confusion.
  • Assumed switch had STP support—fabric flooding without L2 uplink.
  • Added both 1 GbE and 2.5 GbE adapters to vmbr1. The bridge looped back into the same switch segment, so every broadcast multiplied. The mDNS avahi-daemon service processed a flood of packets and maxed out the CPU.