There's a kind of person that hypnotists label as "analytical" who tend to be more difficult to hypnotize. "Analytical" people tend to see the alternative being a "ignore reason and listen to your feelings" kind of person. The thing is, this "analytical" thing is also a failure mode.
Yes, it makes sense to use system 2 a lot. Many people often fail to use it when they should. However, system 1 is important too. Yes, LessWrong is already aware of this, however, it is still full of people in the analytical failure mode. They can say "system 1 is important" and believe it too, but it's not incorporated in revealed beliefs because their system 1 doesn't seem to get it. Oh system 1, Believe in yourself! Well, that's a gross oversimplification. Moving on...
So let me get to a more specific oversimplification. If you're an analytical guy and you're struggling to quit smoking, you might want to visit a hypnotherapist. While there, you'll have a harder time getting results than the stupid gullible idiot that just accepts things uncritically. So unfair, right? But of course, rationalists should win. This is not a perverse game where the winning criteria is to select the losers. It's actually possible to be a good subject and a good rationalist. It's just that you're a good subject when it makes sense to be. Just accept the right suggestions, or turn your filters off when it's called for. I say "just", but I don't mean to trivialize it. It can feel genuinely hard from the inside - I've been there. Yet the winning actions remain simple - it's your implicit reasoning for not taking them that gets complicated.
When you have seen the horrible and pervasive tragedies of poor thinking, it can scare you into taking thinking seriously. I think that this aversion to being wrong is the thing that defines the "rationalist" cluster - though it doesn't feel like that for reasons touched upon below. I think that we're not weird because of some innate preference for Truth, but because we've seen what "irrationality" does, and it scares the shit out of us. (and rightly so!)
And from this place, it's too scary as hell to jump blindly into the unknown. "I know that irrationality wrecks peoples lives and that from the inside, it's freaking invisible and that it's tough as shit to predict where it'd go in advance... but... I'm gonna jump in headfirst and willingly commit irrationalities like believing anything the hypnotist says without doubt or question - even if he says I'm a carrot!". Hah! No. Fucking. Way. "But.. I'll get cool things out of it like remapping my taste preferences for healthy foods!". Nope. No. Fucking. Way. Are you kidding!? You know that promises of irrationality are devils offers. Even if you can't see the downsides. Even if it works out well for others. It's not safe. There's just too much to lose.
Analyticals implicitly recognize the need for a theory of when it's safe to let your thoughts go wonky - and that they don't have it.
And so even if you talk yourself into giving hypnosis a go, you work extra hard to make sure everything stays understood, which causes major problems. When you're desperate enough, you'll grasp at anything that you can "verify" as true... which is a much lower bar than actually being usefully true. Ironically enough, the problems that truth-craving analyticals struggle with while being hypnotized come from avoiding other truths. I had a fun one recently where I showed this guy magnetic fingers and his fingers instantly started moving - then he stopped them and denied their motion. The problem comes when the
fact checking organization into preconceived mental bins is so intense as to come in and screw everything up - for example, rounding down finger motion into "no motion". In essence, it's refusing to direct your attention where the hypnotist says to because things have to make sense god dammit!.
And of course, all the value comes where it applies to everyday life. Not when your mind is being bent by a hypnotist, but when you're trying to learn from someone who is more than one inferential step away. You can immediately rule out things they say from apparent contradictions with how you see things, but maintaining a search for a useful truth - maintaining knowledge that there is something important which you don't understand means spending time in that unknown.
And it's not an easy fix - it's scary at the meta level too. For that, you have to consider that sometimes you're better off taking a blind leap - that all the probing you do to "understand" fucks things up and even gets in the way of understanding. The need to be strictly epistemically rational looks unattainable. That irrationality which you try so hard to avoid? Yeah, you can't avoid it. Not only that, but you can't just say "okay, whatever" and not act on that information. Sometimes the safest thing to do is to run head first into an experience without even allowing yourself to try to stay safe with analysis. Like not only walking up to steal the lion's kill, but doing so after committing to do it with seemingly irrational "confidence" - not even allowing yourself to hold some quite reasonable doubts and fears - so that they don't see your fear and eat you. It's not an easy thing to do, and when it seems like you don't have a choice, it's not a very comfortable mental space.
So no wonder people will choose to fool themselves into believing their fingers weren't moving, even when it gets in the way of what they want. It's too scary to allow yourself to not know, and it's too scary to admit it.
If fear of irrationality is the defining feature of "rationalists", then I suspect acceptance of this risk is the defining feature of "post rationalists". Not a giving up, of course, but an understanding of the risks and a lack of paralyzing fear.
Acceptance of calculated risks is pretty important, and definitely the direction we want to go, so it can be tempting to conclude "they should get over their fear" (and I wouldn't say that would be wrong) and perhaps brute force it. But it's oh so much more pleasant to just allow it for the time being while you build lines of retreat. You're doing alright, so keep on doing your thing until your new thinking ship is done.