Madness will be increased ambiently, albeit slowly. It can be prevented and even decreased by staying safe in a built base, reminding yourself who you are in Journal Entries, and staying away from danger. It's increased whenever you run into something dangerous, like a monster or armed guard, and even as you stare into the darkness. Your mind likes to play tricks on you.

When fully insane (below 30%), you'll find that rooms aren't how you remember them, and sometimes you may even walk into a room for it to change right in front of you. Some monsters in the shadows may even start to roam the halls, looking for you in your vulnerable state.