You can try Feliway  it is used to calm down animals with anxiety
conditions.  I have seen it work wonders.

You also have to understand some cats are like that  they do what they want.
He obviously trust you to sleep with you

Our cat limo  doesn't at all like to be picked up and to be petted when she
wants to only.   My spirit is a nervous nit  any noise or anything gout of
the ordinary he becomes a wreck.  I am always worried I  he would have a
heart attack

NO KEEP HIM IN  letting him out is asking for a world of trouble with a
nervous cat

I do a little volunteer work in animal rescue. This has
included visiting with a few dozen cats in a cattery every
week, to help them get used to humans. Some like to be
picked up, and some (even the old-timers) do not. All
ultimately will accept being stroked, at least a bit. Cat
personalities really do vary about as much as human
personalities.

I no longer believe in letting cats roam outside. They can
get bitten by other cats or dogs, or hurt by cruel people,
or (very common) hit by a car. One of my cats loves busting
out of the house (by accident, and making me a nervous wreck
until I lure him back in around supper time), but he's also
one who, a few years ago before I got him, broke a leg on
one of his forays into the world.

I avoid sudden moves or sounds around any cat that seems a
bit nervous. I always hold my hand a few inches from his/her
nose, and wait for him/her to move his face close to my
hand, before trying to stroke him/her. I think they're
adjusting to one's odor. Once they get used to it, they're
more willing to let the person get close the next time.

You can try a heating pad set on low during the day, and see
if your cat favors this spot. In the winter time, this has
always been a big hit with the cats I've had. A little
catnip now and then seems to make a happy cat, too.

Sounds like you have a fine cat, just not one that is a
"picker upper."

You already have some good advice here, but I wanted to add that trying
something like Rescue Remedy could be of help. You get it at a health
food store and add a few drops to the water. (it's herbs, not harmful
and people can use it too).

I also echo the NO in letting him out.

Rene