I think that most Finns must be a bit bewildered about your situation. If you don't go to work, don't clean/do the laundry and ironing, or even don't take care of your small children yourself - what on earth would you do then? Many won't know how to behave around you, what to talk with you. Finnish moms/dads either go to work or are at home taking care of their children.
Having said all this, I must add that it's very unfortunate that people would get mean about it. But maybe they are just amazed and don't know how to behave... no excuse, but still. I can't actually think why anyone would envy such an empty life, I don't think it's about jealosy. Aren't you a bit bored, why not go to work then, for some social life?
My husband is actually rich, but I'd rather go to work and "have a life" of my own and a career; we also take care of the ordinary household things (cleaning, laundry, ironing) ourselves. It isn't hard. But this is the Finnish way, it is a bit difficult to explain it.