i've lived in hk and while i haven't had a baby there, i have taken my son to hk and have very good friends there who have lots of babies between and they're all just fine and dandy.
hk is an amazing place to visit if you've never been and at 14 weeks your child is still young enough to cart around without too much trouble.
depends on what kind of spirit of adventure you have??
if you know people there to visit then even better but even if you don't and if your husband is free over the weekend to hang out with you guys then i'd say it's worth a visit. you can take the tram up to the peak, the star ferry across to kowloon to see the harbour front and just soak in the atmosphere -it's like no where else on earth.
it's getting cooler there so if you go, you'd need some warm clothes for the baby.
if you're not of the exploring kind then maybe don't go but if it's one of those places you've always wanted to visit then go for it!