Currently I have it like this: I did test it with a couple of other "dumb" switches, but since they have EEE, I get the drops with them as well... I can give you the model numbers of the 2 other switches I tested if you want. I had to disable the EEE (called "Green Ethernet") on the Dell X1026 switch that had it enabled in order to solve the issue. I'm willing to test beta firmware, in the interest of not getting spammed by publicly posting my email address you can email it to dueltmp+iqdj2@gmail.com (or the one on my profile here).

So I've "fixed" my problem... After doing more research, it turns out that the "dumb" switches I have already don't support EEE so being unable to disable the setting on them wasn't necessary... but one of my Dell switches had it turned on. Turning it off on that switch seems to have fixed the issue completely, at least so far. I guess disabling EEE is okay as a workaround, but there really should be a firmware fix.

@goCoax Not sure if I need to ping to get your attention or not, so...