Nxge interface

nxge = nVidia Gigabit Ethernet

Numbering
Interfaces are numbered 0,1,2,3 with 3 being nearest the PCI slot.

Driver
In Solaris 10 U4 and above apparently, otherwise here or from http://www.sun.com/download/products.xml?id=45a593ce.

On U3 or below you must update to Jumbo Kernel Patch 118833-36 or later, reboot immediately following the patch and before applying any more patches.

For SPARC systems, change to the following directory: cd 10_GigabitEthernet/Solaris_10/sparc/Packages

For sun4v systems, install the software packages by typing the following at the command line: /usr/sbin/pkgadd -d. SUNWnxge.v SUNWnxgem

For sun4u systems, install the software packages by typing the following at the command line: /usr/sbin/pkgadd -d. SUNWnxge.u SUNWnxgem

Docs
http://docs.sun.com/source/820-0115-10/ (for Sun x8 Express Quad Gigabit Ethernet UTP Low Profile Adapter)

Configuration
Use /kernel/drv/nxge.conf. Here's an example


 * 1) ---Link Configuration --
 * 2)       The link parameters depend on the type of the card
 * 3)       and the port.
 * 4)       10 gigabit related parameters ( i.e adv_10gfdx_cap)
 * 5)       apply only to 10gigabit ports.
 * 6)       Half duplex is not supported on any NIU card.
 * 7)       adv-autoneg-cap
 * 8)               Advertise auto-negotiation capability.
 * 9)               default is 1
 * 10) adv-autoneg-cap = 1;
 * 11)       adv_10gfdx_cap
 * 12)               Advertise 10gbps Full duplex  capability.
 * 13)               default is 1
 * 14) adv_10gfdx_cap = 1;
 * 15)       adv_1000fdx_cap
 * 16)               Advertise 1gbps Full duplex  capability.
 * 17)               default is 1
 * 18) adv_1000fdx_cap = 1;
 * 19)       adv_100fdx_cap
 * 20)               Advertise 100mbps Full duplex  capability.
 * 21)               default is 1
 * 22) adv_100fdx_cap = 1;
 * 23)       adv_10fdx_cap
 * 24)               Advertise 10mbps Full duplex  capability.
 * 25)               default is 1
 * 26) adv_10fdx_cap = 1;
 * 27)       adv_asmpause_cap
 * 28)               Advertise Asymmetric pause capability.
 * 29)               default is 0
 * 30) adv_asmpause_cap = 0;
 * 31)       adv_pause_cap
 * 32)               Advertise pause capability.
 * 33)               default is 1
 * 34) adv_pause_cap = 1;
 * 35) --- Jumbo frame support -
 * 36) To enable jumbo support for all nxge interfaces,
 * 37) accept_jumbo = 1;
 * 38) To disable jumbo support for all nxge interfaces,
 * 39) accept_jumbo = 0;
 * 40) Default is 0.  See the example at the end of this file for
 * 41) enabling or disabling jumbo for a particular nxge interface.
 * 42) --- Receive DMA Configuration
 * 43)  rxdma-intr-time
 * 44)       Interrupts after this number of NIU hardware ticks have
 * 45)       elapsed since the last packet was received.
 * 46)       A value of zero means no time blanking (Default = 8).
 * 47) rxdma-intr-pkts
 * 48)       Interrupt after this number of packets have arrived since
 * 49)       the last packet was serviced. A value of zero indicates
 * 50)       no packet blanking (Default = 20).
 * 51) Default Interrupt Blanking parameters.
 * 52) rxdma-intr-time = 8;
 * 53) rxdma-intr-pkts = 20;
 * 54) --- Classification and Load Distribution Configuration --
 * 55) class-opt-****-***
 * 56)       These variables define how each IP class is configured.
 * 57)       Configuration options range from whether TCAM lookup ie
 * 58)       is enabled to flow hash generation.
 * 59)       This parameters also control how the flow template is
 * 60)        constructed and how packet is distributed within RDC
 * 61)       groups.
 * 62)       supported classes:
 * 63)       class-opt-ipv4-tcp class-opt-ipv4-udp class-opt-ipv4-sctp
 * 64)       class-opt-ipv4-ah class-opt-ipv6-tcp class-opt-ipv6-udp
 * 65)       class-opt-ipv6-sctp class-opt-ipv6-ah
 * 66)       Configuration bits (The following bits will be decoded
 * 67)       by the driver as hex format).
 * 68)       0010:           use MAC Port (for flow key)
 * 69)       0020:           use L2DA (for flow key)
 * 70)       0040:           use VLAN (for flow key)
 * 71)       0080:           use proto (for flow key)
 * 72)       0100:           use IP src addr (for flow key)
 * 73)       0200:           use IP dest addr (for flow key)
 * 74)       0400:           use Src Port (for flow key)
 * 75)       0800:           use Dest Port (for flow key)
 * 76) class-opt-ipv4-tcp = fe0;
 * 1)       These variables define how each IP class is configured.
 * 2)       Configuration options range from whether TCAM lookup ie
 * 3)       is enabled to flow hash generation.
 * 4)       This parameters also control how the flow template is
 * 5)        constructed and how packet is distributed within RDC
 * 6)       groups.
 * 7)       supported classes:
 * 8)       class-opt-ipv4-tcp class-opt-ipv4-udp class-opt-ipv4-sctp
 * 9)       class-opt-ipv4-ah class-opt-ipv6-tcp class-opt-ipv6-udp
 * 10)       class-opt-ipv6-sctp class-opt-ipv6-ah
 * 11)       Configuration bits (The following bits will be decoded
 * 12)       by the driver as hex format).
 * 13)       0010:           use MAC Port (for flow key)
 * 14)       0020:           use L2DA (for flow key)
 * 15)       0040:           use VLAN (for flow key)
 * 16)       0080:           use proto (for flow key)
 * 17)       0100:           use IP src addr (for flow key)
 * 18)       0200:           use IP dest addr (for flow key)
 * 19)       0400:           use Src Port (for flow key)
 * 20)       0800:           use Dest Port (for flow key)
 * 21) class-opt-ipv4-tcp = fe0;
 * 1)       0400:           use Src Port (for flow key)
 * 2)       0800:           use Dest Port (for flow key)
 * 3) class-opt-ipv4-tcp = fe0;
 * 1) class-opt-ipv4-tcp = fe0;

See the linked documentation for details on how to specify individual port details.

You can view current interface speeds like this:

root@host # dladm show-dev e1000g0        link: down      speed: 0     Mbps       duplex: half e1000g1        link: down      speed: 0     Mbps       duplex: half e1000g2        link: unknown   speed: 0     Mbps       duplex: unknown e1000g3        link: up        speed: 100   Mbps       duplex: full nxge0          link: down      speed: 0     Mbps       duplex: unknown nxge1          link: up        speed: 1000  Mbps       duplex: full nxge2          link: unknown   speed: 0     Mbps       duplex: unknown nxge3          link: unknown   speed: 0     Mbps       duplex: unknown

Performance
Add this line to /etc/system (use '8' on UltraSparc-T1 systems)


 * set ddi_msix_alloc_limit=4

Add this line to a ndd start up script somewhere (set to 16 on UltraSparc-T1 systems)


 * ndd set ddi_msix_alloc_limit=4

Firmware
Updating the firmware on these cards is one scary-arse piece of work!

You get the firmware and driver pack from http://www.sun.com/download/products.xml?id=45a593ce (this is the same line as above on this page). You need to unzip the HTML instructions from 10_GigabitEthernet/Firmware and copy them somewhere you can point a browser at (here). Follow the instructions VERY carefully!

I've written a function to generate the firmware commands from the OS of a server as it's easy to get things wrong, escpeially if you have two or more nxge cards. However, I strongly recommend you verify it's output with what you'd expect before proceeding as I don't want to be held responsible for shafting an expensive server!

nxge_fw_upgrade_cmds { echo load-base dload NXGE_OBP.FC echo load-base 1 byte-load echo load net:,QGCEM3.12 for i in 0 4 8 12 16; do test -e /dev/nxge$i || continue ifconfig nxge$i|nawk '/ether/{print $2,"set-mac-addr"}'|sed 's/:/ /g' echo "\" `ls -l /dev/nxge$i|awk '{print $NF}'|\ sed 's|../devices||;s/:.*//'`\" begin-select-dev" echo load-base update done echo reset-all }