I can't tell if it's random, or factors I'm unaware of. Some family members will quickly lose loyalty while in the capital with the leader, while others, seemingly more slowly, gain a little.

I should mention this is 2.2b, so if there have been specific changes to this since then, I'd be interested to hear about them anyway, and any insight on older version would be appreciated too.