Is that ipfilter? Oooh, don't know that one. :(

I'll second ewx's suggestion: log everything. Even the packets you're already accepting (ie. the packets that are returned from your outgoing connections). It might show that they're weird in some way, which then affects the stuff that's causing difficulty.

eg. if your upstream hardware is NATting incoming traffic so your machines think it's coming from an unroutable address, you'd probably drop it before anything else (Remember: I don't read ipfilter, so I'm guessing at what the rules mean). Of course, such hardware will cause further problems. But by logging everything you should get some clues.