Symptom: when you're trying to access a webserver, you can connect fine, send the HTTP request fine, but then the client waits forever for a reply. Interestingly enough, if you upload a really small (Say, 10 byte) file to the server, you can retrieve it without fail.
Possible Cause: something is blocking Path MTU Discovery.
What is Path MTU Discovery?
Every hop on the Internet has an MTU, or Maximum Transmission Unit. This is the maximum size that IP packets sent over that link are allowed to be. The MTU for something like ethernet will be quite high (usually 1500 bytes), but other transmission media might run more efficiently with smaller packets.
If a router receives a packet that is larger than the MTU of the hop it needs to send it over, the only way it can send the packet is to break it into fragments. The problem with fragmented packets is that they're rather inefficient. One host has to break them up, another host has to knit them back together, and you end up transmitting far more packets than you'd need to if they were just the right size in the first place.
Path MTU Discovery is a way of calculating the largest packet that will traverse a particular path between hosts. The algorithm is simple. Hosts send the largest packets they can, but with the "Don't Fragment" bit set. That way, if the packet turns out to be too big, routers don't just break it up and keep going. Instead they drop the packet, and send back an ICMP Destination Unreachable (Datagram Too Big) message, which tells the originating host the largest MTU the next hop will allow. On receiving this ICMP, the originating host creates a new MTU for that specific destination (known as the Path MTU) at the lower value. Then it resends all the lost data.
It's all pretty simple, really.
So What Goes Wrong?
Sometimes, overzealous firewall administrators decide that ICMP is a bad thing, and block it. This is fair enough on the surface: ICMP can be used both as a convenient flooding tool and a way to map networks. The thing is, you have to be careful which ICMP you block. If, specifically, you block the Datagram Too Big ICMP, then any attempt at MTU path discovery will fail quietly: packets will be dropped on the floor, and the request to re-send a smaller packet will never get back to the originating host.
You end up with a really weird error condition that tends to drop in and out as routes change, and is very hard to track down unless you know exactly what you're looking for, because by definition all of the evidence of the problem is being either blocked, or dropped on the floor.
