L2TP / MTU interactions with NetSource / ESat?

Tenshot

I was talking to someone recently about how third-party ISP DSL traffic is carried over Eircom's network, and I realised something that hadn't clicked before.

An ISP like NetSource has their user's PPP sessions delivered to them using L2TP (Layer 2 Tunnelling Protocol), which means the PPP packets are encapsulated inside standard UDP packets and then routed over an IP network to the ISP's Broadband Access Server where they are turned back into standard PPP packets and processed as if the user was directly connected to the BAS.

This is grand, except that because L2TP runs over an IP network (and one which may include Ethernet links in telco offices), the maximum packet size is 1500 bytes. When you strip off the IP header, UDP header and L2TP wrappings, along with the PPP header, you're down to about 1450 bytes remaining for the PPP data. A direct link, which didn't involve L2TP, could carry up to 1492 bytes of data per PPP packet.

No problem: if a user tries to send (or receive, by way of a download) an IP packet > 1450 bytes, the corresponding L2TP packet is fragmented into two smaller packets, and then reassembled at the opposite end of the tunnel. This works fine (otherwise you'd have lots of problems surfing the web). However, it does impose a CPU load on the intermediate servers that are performing the tunnelling -- it takes a lot more CPU to fragment a packet (on both the sending and receiving side) than it does to simply route it as-is. It also uses additional bandwidth.

How often do packets get fragmented? For a standard web download, it turns out that just about every packet will end up getting fragmented by default, because TCP tries to make maximum use of the available space in each packet. Thus, it will try and send IP packets with 1492 bytes when possible. (Actually, it may try to send packets of 1500 bytes in some circumstances, leading to yet more fragmentation, but that's another day's work.) This is because the Maximum Transfer Unit (MTU) for the link is set to 1492 or 1500.

There's a trick called MTU Path Discovery, which some TCP/IP stacks take advantage of. This tries to find the largest packet size that can be transmitted without needing to fragment, so that all the fragmentation overhead can be avoided. Unfortunately, that won't help here: L2TP fragmentation happens at Layer 2 (PPP) rather than Layer 3 (IP), so it is invisible to MTU Path Discovery.

So, imagine one user downloading a file and having every packet in that file split into two at the ISP's BAS, then reassembled at Eircom's access point in the DSLAM. Now multiply that by the number of users using the DSLAM and you can end up with a much higher CPU load on the server than perhaps it was designed for. (In a well architected network, few or no packets will require fragmentation.) I wonder if this might be at least partly responsible for some of the performance degradation on Netsource that's been discussed here in recent weeks? (I've been away on holiday, so I'm not quite up to date yet).

What's the solution? Well, there are a few:

- Set the MTU on your PC to 1450 or maybe even a bit less. There are plenty of "Speed up your DSL" utilities available that let you alter this.

- Set the MTU on your access router (if you have one) to 1450 or less; this works best if your router can transparently reduce the MSS value in TCP SYN packets accordingly.

- Eircom (or the third-party ISP?) could install some sort of transparent proxy that would reduce the MSS value in TCP SYN packets on the fly, as necessary, to ensure TCP packets didn't need to be fragmented.

- The third-party ISP could configure their BAS to advertise a smaller MRU (Maximum Receive Unit) during PPP negotiation. In principle, this would cause the MTU to be automatically reduced at the client side, but depending on circumstances may instead simply cause packets to be discarded; this would need to be tested carefully.

Unfortunately, the first two solutions require that the majority of users make the change - if only a few do it, it doesn't have much effect. (Every little helps, of course.)

This problem isn't unique to Ireland, by the way: it affects BT Land as well. There's a somewhat relevant thread at:

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&th=f529e89acd7ce897&seekm=CDYC22417477B%40cd-yacht.demon.co.uk

Essentially, any wholesale provider that uses L2TP to backhaul customer connections to a retail ISP will run into this. (BT use PPPoA rather that PPPoE, so they have eight more data bytes to play with than we do.)

If ThomasB is still reading here, maybe he could comment on whether NetSource are already tackling this problem in some way?

talla

Excellent informative post!

spongebob

you can set these values in the windows registry as well. You must add Extra Keys as shown so a search will not find them. on a small lan, set them in every registry otherwise you have the same problem locally. you must specify both MTU and MSS values ISTR
See This Too

also check Karl Jeacles page Here on related issues. He may incorporate any discoveries that come from this thread.

M

crowbar

karl's page is very well written, definitely worth a read for the nitty gritty details on how dsl works here. i though the backhaul to the isp are 2mbps links though rather than the 1mbps/512kbps that the article seems to imply. (i may well have read it wrongly.)

another way to solve the problem could be to increase the mtu in the dsl core network to include the overheads caused by all this encapsulation, so that full 1500 byte packets + overhead is carried all the way through the core to the internet. i thought i read somewhere that bt in the uk were doing exactly that to eliminate all the problems associated with a lower mtu in the middle of the path.

Tenshot

Another way to solve the problem could be to increase the mtu in the dsl core network to include the overheads caused by all this encapsulation, so that full 1500 byte packets + overhead is carried all the way through the core to the internet.

This would be great if it works. I think it gets scuppered by any Ethernet links in the middle (carrying the L2TP traffic), because most Ethernet interfaces have a fairly hard limit of 1518 bytes (1500 for IP, 18 for the Ethernet headers + CRC).

If BT are doing it though, perhaps they've found a workaround (or are just not using any Ethernet links internally)?

crowbar

i'd be surprised if bt were still using bog standard ethernet and not gig+jumbo frames or something a bit more advanced. dunno about eircom, though.

here's the link to the news article from [url]www.adslguide.org.uk:[/url]

http://www.adslguide.org.uk/newsarchive.asp?item=1210

longword

L2TP is a very specific term - I'm not sure that any ISPs use it at all. L2TP is for stuffing ethernet frames into IP packets. Your problem is with PPPoE which stuffs IP packets into PPP packets and those PPP packets into Ethernet frames. The ethernet frames are then shattered into bazillions of tiny little ATM packets before they hit the modem bits of your DSL modem but they're re-assembled back into full size ethernet frames. The PPPoE wrapper is what thieves the 8 bytes from your usual 1500 byte payload.

It's perfectly possible for an ISP to run something like DHCP instead of PPPoE to give you an IP address, and leave you with straightforward regular common-or-garden IP over ethernet. A handful of DSL lines in this country are configured to do that. ISPs don't like it because it's difficult to control and isn't tied to a login name and password.

Tenshot

Originally posted by longword
L2TP is a very specific term - I'm not sure that any ISPs use it at all.

Eircom use L2TP internally to deliver PPP sessions to independent ISPs like NetSource. It's only used between Eircom's exchange and the ISP's central office; it never gets as far as the customer's equipment.

The article crowbar pointed out is very interesting. If Eircom followed a similar path, it sounds like it would remove what is potentially a big headache. (Perhaps they've already done so. Something tells me they haven't though...)

monkey_magic

Just a couple of comments on the origional post to clarify the workings of L2TP, and Path MTU Discovery.

L2TP is not as the name sugests a layer 2 protocol, it's a way of transparently sending layer 2 information across a WAN, and operates at layer 3. It does add some overhead to IP packets being sent from one end of the tunnel to the other, and therefore fragmentation of the IP packets can sometimes be required to facilitate this.

Because it's a Layer 3 protocol Path MTU Discovery has no problems working within a network that implements L2TP. The only way it won't work is if it's not supported by your OS, or if ICMP type 3 code 4 messages are being filtered.

http://www.faqs.org/rfcs/rfc2661.html
http://www.faqs.org/rfcs/rfc1191.html

Tenshot

Because it's a Layer 3 protocol Path MTU Discovery has no problems working within a network that implements L2TP. The only way it won't work is if it's not supported by your OS, or if ICMP type 3 code 4 messages are being filtered.

I believe this is incorrect, at least for the scenario we're discussing.

With MTU path discovery, an IP packet arrives in from the network, within the MTU size. It has the Do-Not-Fragment bit set (as per the RFC you quoted). Because it is within the MTU size for the interface it arrived at, no fragmentation is required.

The packet is now encapsulated as PPP data, which produces a slightly larger packet. This PPP packet now gets encapsulated inside an L2TP frame producing a noticeably bigger IP packet. This new packet does not inherit any of the characteristics of the original packet (and in particular, it doesn't inherit the Do-Not-Fragment bit) -- as far as it's concerned, a PPP packet is just a big chunk of data.

This L2TP frame is now transmitted across the internal IP network of the provider, fragmented (because it exceeds the minimum MTU of that network) and is reassembled at the far side. Then the L2TP encapsulation is removed, and the original PPP frame is delivered down the DSL line to the end user (still containing the original IP packet with the do-not-fragment bit set). The PPP wrapper is removed by the user's PC or router, and the original IP packet is finally delivered to the user's PC.

The problem really is that MTU path discovery here is applied at the client/server ends of the link, and not to the internal ISP network.

monkey_magic

Very Few sites have the df bit set on packets, but some of the major sites for whatever reason have decided to set this bit, Google and Yahoo are two.

When a packet needs to be fragmented to put into an L2TP tunnel but the df bit is set then it's the job of the L2TP LNS or LAC (whichever one is recieving the packet) to send an ICMP message to the sending host letting it know the packet is too large, and then drop the packet. The sending host should then send packets of a smaller size for the remainder of the session.

monkey_magic

Just reread your reply, and yes you're right if a packet has the df bit set, and the packet is small enough to fit into the L2TP tunnel it could very easily be fragmented along the path of the tunnel to be reassembled again when it gets out the other side.

It's unlikely however that any provider buying wholesale from Eircon would have a network setup that could cause packets to be fragmented once they enters the tunnel though. Most, if not all providors would have a combination of ATM, and Ethernet, and the MTU should be the same throughout the Ethernet segments of the network.

longword

AFAIK the popular Apache web server sets DF by default.

And are you sure about Eircom delivering sessions to wholesaler customers with L2TP? I could swear they deliver ATM to other providers. I know at least one wholesale DSL customer that's using straight AAL5 ethernet over ATM with DHCP to assign an IP address. No PPPoE. No MTU issues.

crowbar

uh, can you do that ... don't you at least talk pppoe to eircom's dslam so you can specify your service's parameters to the dslam?

edit: there seems to be more than just pppoe at least on eircom's i-stream. i can get 1454 byte pings (1426 bytes payload + 8 byte icmp header + 20 bytes ip header) through to www.cisco.com with the df bit set but no more. so there seems to be a maximum of 46 bytes overhead somewhere in the network. i'm not getting icmp frag required packets back from the network either, but it could be my router filtering them. at work, i can get full 1500 byte pings through to www.cisco.com no bother so it's probably not www.cisco.com rejecting large icmp packets.

the consensus on google is that l2tp overhead is 40 bytes ... so what are the other 6? or is my math wrong?

Tenshot

monkey_magic: hmm.... I can see that a sufficiently clever implementation might take into account the fact that a packet is about to be PPP encapsulated and then L2TP-wrapped, at the point the packet is being routed, and reject it (when the df bit is set) on that basis. But I have my doubts -- the routing engine probably only has visibility of the PPP layer (which could be going out a direct link for all the routing engine knows), and the L2TP layer only sees PPP frames. The links crowbar posted earlier about problems in the BT network tend to support this.

longword: Karl Jeacle's iStream page gives NetSource and BT (ESat) as examples of ISPs that receive customer connections from Eircom using L2TP. (This is for the RADSL product; higher-end services may use a different protocol.)

Tenshot

This is a bit speculative, but maybe someone with access to a Netsource link could try a few ping tests as follows:

ping www.netsource.ie -l 1412 -t
ping www.netsource.ie -l 1422 -t
ping www.netsource.ie -l 1432 -t
ping www.netsource.ie -l 1442 -t

The first two should be below the threshold for fragmentation down an L2TP tunnel, while the second two should be above the threshold. If the first two have similar response times, and these are lower than the response times for the second two, then it suggests that the L2TP tunnel is indeed resulting in fragmentation of larger packets.

It's not conclusive though, since there are other factors like ATM cell size that could come into play. Also, the speed of the link is possibly fast enough that the difference won't show up on a standard ping. Still, worth a try for the sake of a few minutes.

Also possibly worth adding the -f flag (do not fragment) to the pings; if the first two get through and the second two don't, that would prove that the L2TP tunnel is paying attention to the DNF bit.

(Edit: great minds think alike -- I see Crowbar has just finished doing this test already!)

Tenshot

I'm on Eircom's iStream starter pack, and I can get successful replies from Cisco up to the 1492 byte maximum I would expect for a PPPoE link:

c:>[b]ping [url]www.cisco.com[/url] -l 1464 -f[/b]
Pinging [url]www.cisco.com[/url] [198.133.219.25] with 1464 bytes of data:
Reply from 198.133.219.25: bytes=1464 time=327ms TTL=240
Reply from 198.133.219.25: bytes=1464 time=328ms TTL=240
Reply from 198.133.219.25: bytes=1464 time=331ms TTL=240
Reply from 198.133.219.25: bytes=1464 time=329ms TTL=240

c:>[b]ping [url]www.cisco.com[/url] -l 1465 -f[/b]
Pinging [url]www.cisco.com[/url] [198.133.219.25] with 1465 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

I just tried on our work connection (iStream business, no cap) and got the same results. Maybe your router is the one rejecting the pings? Is it doing any IPSec stuff?

(As an aside, Windows XP seems a bit inconsistent with its fragmentation messages -- sometimes it displays the IP address of the host that returned the message and sometimes it doesn't. Anyone know why?)

longword

Originally posted by Tenshot

c:>[b]ping [url]www.cisco.com[/url] -l 1464 -f[/b]

Just to so there's no confusion, that should be clarified. Windows ping takes the ping packet payload for a size parameter, ignoring the ICMP and IP headers that are added to the packet, so a ping with 1464 bytes of data appears as a 1492 byte IP packet on the wire.

longword

Originally posted by crowbar
uh, can you do that ... don't you at least talk pppoe to eircom's dslam so you can specify your service's parameters to the dslam?

The DSLAM in the exchange (as Eircom uses it at least) doesn't understand Ethernet, IP, or PPPoE. All it knows is ATM. Where that ATM stream ends up depends on the VCI/VPI settings of the client's modem and/or Eircom's DSLAM configuration (likely the latter since Netsource use the same VCI/VPI as Eircom).

monkey_magic

I'm pretty sure IOL at least get their wholesale via L2TP. The clue is in the username. all IOL usernames end in @iolbb this kind of username is typical of L2TP implementations. What happens is when you're authentaciting a PPP session is recieved by the Eircon BAS, it sees the @iolbb part of the username and that's how it knows what tunnel to put a particular PPP session into. Do NetSourct/UTV have @blaah in their usernames?

Tenshot..

"the routing engine probably only has visibility of the PPP layer"

PPP operates at layer 2 of the OSI model, so the routing engine would have no interest in the workings of PPP. It has to be remembered that L2TP is a layer 3 protocol, and works by adding overhead to existing IP packets, it does not change the packet in any way. If a packet has the DF bit set, and is too large to be encapusalated by L2TP the router must drop the packet.

tomk

Originally posted by monkey_magic
Do NetSourct/UTV have @blaah in their usernames?

Great thread for interested lurkers like me - keep it up lads!

My UTV username does indeed end in @adsl.utvinternet.ie, if that's of any interest to anyone.

Tenshot

Originally posted by monkey_magic
PPP operates at layer 2 of the OSI model, so the routing engine would have no interest in the workings of PPP. It has to be remembered that L2TP is a layer 3 protocol, and works by adding overhead to existing IP packets, it does not change the packet in any way. If a packet has the DF bit set, and is too large to be encapusalated by L2TP the router must drop the packet.

Not quite -- although L2TP itself operates at layer 3, the packets going down the tunnel are layer 2 PPP packets (complete with PPP headers etc.) They've already been turned into PPP packets before L2TP gets visibility of them.

The routing engine doesn't care about the internals of PPP, but it does care about the MTU configured on the PPP link. This MTU could be configured to be smaller than usual (to take into account the L2TP encapsulation that's about to happen); in that case, larger packets will be fragmented before going into the PPP layer (which is still CPU load on one of the ISP's server, but at least the reassembly overhead is now handled by the user's CPE equipment).

However, for sites using MTU Path Discovery, you're now back to square one -- if discovery fails for any reason (typically blocked ICMP messages, something likely to be happening a lot more recently with the Blaster worm variants in circulation) then the site becomes inaccessible to end users unless they manually configure their own MTU to something smaller. Since there is no standard way to detect the need for this reduced MTU at the user side, the ISP will no doubt get a load of calls from confused users wondering why they can't access half the sites on the web.

(There is also potential for this with PPPoE, but in this case the PPPoE stack is terminated on the user side, so the MTU can be reduced automatically to take this into account.)

monkey_magic

I've done a bit more reading about this one, and you're absolutely right, appologies if I caused any confusion.

Found an interesting article on the Cisco website describing exactly this scenario, along with workarounds hopefully implemented by providers (if they use cisco that is).

http://www.cisco.com/en/US/tech/tk801/tk703/technologies_tech_note09186a0080094c4f.shtml

The issue of ICMP filtering is an interesting one, I think only time will tell if there is an increase in the number of sites inaccessable to iolbb/UTV/NetSource.

I remember a thread on boards a while ago about iolbb customers having problems viewing sites and www.iol.ie was one of the sites mentioned. I wonder if it's related to Path MTU Discovery? was there a solution in the end? Can't find the thread now. Out of curosity I tried to ping www.iol.ie, and got some destination net unreachables, along with some timeoutes. I don't know if they have the df bit set or not, but if they do it could certainly cause problems if they are also filtering "friendly" ICMP packets.

C:\>ping www.iol.ie

Pinging www.iol.ie [194.145.128.36] with 32 bytes of data:

Reply from 193.95.132.34: Destination net unreachable.
Reply from 193.95.132.34: Destination net unreachable.
Request timed out.
Request timed out.

Ping statistics for 194.145.128.36:
Packets: Sent = 4, Received = 2, Lost = 2 (50% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\>

It's probably a non runner because the problem would be more widespread, still might be worth noting, then again maybe not.

thomasb

The RADSL Service is delivered to all non Eircom ISPs over L2TP - which means your PPP is encapsulated in IP and then delivered over a "private" IP network to the ISP - be it IOL, UTV or Netsource.

All DSL connections on the Eircom network (ie everything but the Esat Business service) use an ATM circuit which terminates on a Broadband Access Server. For Netsource SME/Premier packages, this BAS is actually administered by Netsource, and there is no L2TP involved. For RADSL customers, regardless of ISP, the connection terminates on an Eircom BAS and if the username format is @string - the BAS determines which ISP's L2TP tunnel to forward to based on the value of string.

thomasb

I meant to add on the IP fragmentation.

The L2TP circuit is delivered at the last hope to us over ATM. The MTU on this is 4470 bytes. We are "four hops" from the BAS, so it's possible that fragmentation does take place between our termination point and the Eircom BAS where the PPP frame is removed from the L2TP packet.

We have had a couple of issues with customers accessing remote sites, because of over enthusiastic firewall admins who filter out all ICMP traffic. Setting the MTU lower (to say 1452) on the PPP connection has resolved this issue.

Thomas.

Tenshot

ThomasB, thanks for the clarification on Netsource's setup. It sounds like the MTU/fragmentation problem might be a non-issue if the path from Eircom's BAS to your server is completely ATM-based.

Although most of the TCP/IP traffic is probably downstream, is it possible for you to check upstream traffic and identify the largest packet size being received at your server from Eircom's BAS? If nothing larger than 1500 bytes (including L2TP wrappers) is arriving in, that would imply that there probably is an intermediate link with a smaller MTU...

Monkey_magic, thanks for that Cisco link -- it does an excellent job of putting all the bits and pieces from this thread into context.

(Also, I'm seeing the same unreachable messages when I ping www.iol.ie, but I can access the webserver at that site fine -- I imagine their firewall is just blocking pings at the moment.)

dahamsta

I seem to be getting pretty nasty speeds on UTV so I'm wondering if tuning my MTU would be any help, and if so if anyone has recommended settings for UTV?

Ta,
adam

dahamsta

Anyone? Anyone? Bueller? Any tweaks for UTV welcome.

Frank Grimes

Set it to 1452. It doesn't matter what ISP you're on, anything above that and you might have problems.

dahamsta

Yeah, I was just fiddling around with the SpeedGuide advanced registry stuff to figure out my MaxMTU, however I had to go down as far as 1426 to stop packets fragmenting. So that makes me wonder if the 1452 setting mentioned here is actually achieving anything?

adam

Frank Grimes

Originally posted by dahamsta
So that makes me wonder if the 1452 setting mentioned here is actually achieving anything?

I've seen that setting fix problems a few times.

dahamsta

Any chance a few other people would have a go, out of interest? Just open a DOS window (Start > Programs > Accessories > Command Prompt) and type:

ping -f -l 1472 utvinternet.com

You'll get a message saying "Packet needs to be fragmented, but DF set" four times, so now try it again replacing 1472 with 1468; and again with lower and lower values until that message disappears and you get a proper ping response. As I said above, my packets were fragmenting until I hit 1426.

adam

thomasb

The -f function defines the payload. You need to add 28 bytes for the IP and ICMP header, which added to 1426 gives you 1452.

dahamsta

Excellent! Thanks for the explanation Thomas.

adam

dahamsta

I'm not going away until I figure this out.

ipconfig is bringing up three adaptors: one for my LAN, one for the DSL modem and one for the WAN (PPP/SLIP) Interface. Both of the DSL ones have public IP addresses. Should I set the MTU for all of them, or just the DSL ones, or just one of the DSL ones?

adam

tomk

I went through this exercise a while ago, also on UTVbb, because I was unable to send mail, or post to the web, or upload FTP - anything that required transmission of data. My max MTU turned out to be 1470 (payload) = 1498 total. I had to set this value on every interface in my network i.e. the public DSL connection, and all my internal network connections. If there was a machine on my network that didn't need internet access, I suppose I could have left it at the default 1500, but there isn't, so I didn't.