Join Telecom Blog

This article is in English for everybody to understand:

printk: dst cache overflow is a pain in the a$ for (almost) every linux sysadmin. There are literally tens of thousands of articles on the Internet pertaining to this and very few of them are close to solving the issue.

What happens:

Due to high traffic (virused workstations, torrent/dc applications, malware-especially worms), lots of connections are opened by one or more workstations and are traversing the router. Each connection (the route itself) goes into the kernel cache. If these routes come in an alarming rate and they grow at a pace higher than the expiry time from the kernel cache, this one is “flooded” and will overflow once the value specified in /proc/sys/net/ipv4/route/max_size becomes lower than the routes in the cache.

How to deal with this (Red Hat Linux):

1. In rc.local (or other autorun script) add:

echo 2000000 > /proc/sys/net/ipv4/ip_conntrack_max

echo 200000 > /proc/sys/net/ipv4/route/max_size

The first line helps if you use NAT, giving you the ability to have 2 million connections from NAT clients.

The second line raises the route cache to 200000 routes, allowing you to flush them at a slower pace (see point 2).

2. In order to keep cached routes to a level lower than that specified by max_size, you have to flush them at a regular interval. This is best done by crond, adding a single line in crontab:

/2 * * * * root echo 1 >/proc/sys/net/ipv4/route/flush

Don’t forget to issue in a command line

service crond restart

after altering crontab, in order to let the cron service know about your modifications.

In the example above I have chosen to flush the route cache every two minutes. Depending on your network and real traffic, this can go from one minute up to several minutes, depending on how the route cache level raises.

You can monitor the route cache level by issuing in a command line:

grep ip_dst_cache /proc/slabinfo

and looking at the first value.

issuing cat /proc/sys/net/ipv4/route/max_size will show you the current maximum size of the route cache.

example:

#grep ip_dst_cache /proc/slabinfo

ip_dst_cache 129720 129720    256   15    1 : tunables  120   60    8 : slabdata   8648   8648      0

#cat /proc/sys/net/ipv4/route/max_size

200000

Whenever the first value (129720 in the example above) becomes bigger than max_size (200000 in the example above), you may experience network lazyness and/or connectivity loss. You should reduce the flushing interval in crontab at that moment in order to never let this happen on your server.

This method kept alive all my servers for months and clients didn’t notice any traffic issues:

# uptime

17:10:47 up 93 days, 21:44,  1 user,  load average: 0.04, 0.17, 0.13