Is this shelter trying to scam me?

When you bought the cat, you bought the cat. You shouldn't have to pay. If they want you to pay for the kittens they should have told you before you bought the cat. I would not give the kittens back. If they have a problem they can take you to court, but they probably won't since what they are telling you isn't fair. And yes, they probably are scamming you.