IPsec Pain
For anyone looking to configure the size of the anti-replay window (aka replay protection window) when using automatic keying for IPsec with racoon ... you can't. For evidence, search for sa_args.wsize in the following source code: http://ipsec-tools.sourcearchive.com/documentation/0.7/racoon_2pfkey_8c-source.html and note that it's got a magic number of 4. GRR! Damn magic numbers! I suppose the next step is to replace racoon with a different ISAKMP daemon.
Also of note is that even if you do manual keying (which causes all sorts of other issues) you can barely get the anti-replay window large enough under Linux to meet the RFCs. The RFC for ESP states that you must support an anti-replay window of at least 32 packets, and the default should be 64. As noted in this bug report, Linux can't support over 32. They're not intending to fix it.
Also of note is that even if you do manual keying (which causes all sorts of other issues) you can barely get the anti-replay window large enough under Linux to meet the RFCs. The RFC for ESP states that you must support an anti-replay window of at least 32 packets, and the default should be 64. As noted in this bug report, Linux can't support over 32. They're not intending to fix it.