Debugging my new network, when 10 Gigabit Ethernet Runs at 300 Megabits

I've been moving my location retention complete to a UniFi UNAS Pro 8 arsenic part of a larger homelab moderization. My main IRONHEART (the eventual PC from a fewer years back) desktop now has an Intel E610-XT2 10GbE card, the NAS is connected 10GbE, and there's a Minisforum MS-01 miniPC connected the aforesaid web pinch a 10GbE SFP+ relationship moving Immich and Portainer and a fewer different things.

Everything says 10 gigabit. Windows says 10 gigabit. UniFi says 10 gigabit. SMB copies are utilizing the correct NIC but my record copies are moving astatine astir 100 to 200 megabits per 2nd which is sad making.

Naturally, I blamed the NAS, and the spinning rust within. The UNAS has six 16 TB spinning disks successful RAID 6 and a brace of NVMe SSDs being utilized arsenic cache. I'm besides moving Immich connected the MS-01, pinch its photograph room surviving connected the UNAS, truthful location are tons of thumbnails, metadata reads, and small inheritance writes happening. All look for illustration reasonable suspects.

I switched the UNAS SSD cache from read-write to read-only. No meaningful difference. I stopped Immich completely. No difference. I looked astatine iostat; the disks weren't saturated. We looked astatine SMB signing and Windows Defender web scanning. Still slow.

Then I stopped testing the NAS and ran iperf3 straight betwixt the Windows desktop and the MS-01:

iperf3 -c 192.168.1.222 -P 4

133 Mbit/sec

Oops. The reverse trial was better, but still wrong:

iperf3 -c 192.168.1.222 -P 4 -R

1.33 Gbit/sec

That's weird. Now the disks, SMB, Immich, RAID, and the NAS itself were wholly retired of the equation. This was a Windows/NIC problem and it's weirdly asymmetrical.

Looking astatine the Intel adapter statistic shows me...

Get-NetAdapterStatistics -Name "Ethernet - 10 Gig Intel"

There were astir a cardinal ReceivedDiscardedPackets. During 1 ten-second iperf3 test, the antagonistic accrued by different 268. Why?

The E610 driver had its person buffers astatine the default 512, though it supported up to 4096. I accrued them. I emotion an accrued buffer.

Set-NetAdapterAdvancedProperty ` -Name "Ethernet - 10 Gig Intel" ` -DisplayName "Receive Buffers" ` -DisplayValue "4096"

The discarded-packet count during the adjacent trial went from 268 to zero, and person throughput jumped from 1.33 Gbit/sec to 5.15 Gbit/sec. The transmit guidance was still terrible, fundamentally 313 Mbit/sec. The adjacent research was disabling Large Send Offload (LSO) V2 for IPv4:

Set-NetAdapterAdvancedProperty ` -Name "Ethernet - 10 Gig Intel" ` -DisplayName "Large Send Offload V2 (IPv4)" ` -DisplayValue "Disabled"

Then I ran the aforesaid iperf3 trial again.

7.03 Gbit/sec

That's not a typo. 313 Mbit/sec to 7.03 Gbit/sec by changing 1 NIC setting. Sweet sassy molassey.

LSO exists for a bully reason: Windows tin manus ample TCP buffers to the NIC and fto the adapter/driver conception them into network-sized packets, reducing CPU work. Microsoft does, however, explicitly constituent retired that segmentation offload tin trim maximum sustainable throughput pinch immoderate web adapters/configurations. LSO is usually useful, but not successful this case.

In my peculiar operation of Windows and the Intel E610-XT2, thing successful the IPv4 LSO way was very, very sad. I don't yet cognize whether this is an Intel driver bug, firmware issue, Windows interaction, aliases thing peculiar to this machine, truthful I wouldn't move this into random tech blogger proposal that everyone should disable LSO. Measure first, trim once. Er, twice. Just enactment woke.

Finally I went backmost to the trial that started each this and copied the aforesaid ample record to the UNAS and Robocopy reported:

Speed : 350,201,354 Bytes/sec. Speed : 20,038.682 MegaBytes/min.

About 350 MB/sec, aliases 2.8 Gbit/sec of sustained real-world SMB writes to a six-disk RAID 6 NAS.

That's overmuch much for illustration it. The useful instruction isn't "disable LSO." It was that erstwhile retention is mysteriously slow, yet you person to extremity testing storage. iperf3 removed the NAS, filesystem, RAID, cache, SMB and disks from the research successful 1 move. Once the earthy web was besides slow, the problem became dramatically smaller. And, sometimes the small checkbox branded Large Send Offload is tin of making your 10 gigabit paper tally for illustration it's 2004.

TL;DR - pinch LSO V2 for IPv4 enabled, Windows-to-Linux iperf3 managed astir 313 Mbit/sec. Turning disconnected that azygous offload took the nonstop aforesaid trial to 7.03 Gbit/sec. I’m deliberately saying on this machine because LSO is usually useful and this isn’t broad proposal to disable it everywhere.