The more Affection Points spent on a character, the more outfits you unlock for that character, maybe.
You don't pick what character you want in your "hub area" like in My Pocket Hayami. You pick the outfit you want them to wear at their respective area 