I have removed quite a number of my affiliates. If those people who were affiliated with me and want to stay affiliated, that's fine with me.
Why did I remove them? When you want to be affiliated with a community, you generally want to be affiliated with whatever that person does with their community and the work that they do. I think what makes good affiliates is if you actually talk to those people. I always keep the person's LJ on my friends list so I can check it everyday. When I see their updates, I automatically look at their icons if they interest me. You don't need to even take their icons, but I think that if you want to be an affiliate, then you need to stay in touch with their work, since that's why you had affiliated yourself with that person and their work.