I need some cat advice!

I have two cats that, until about two months ago, were perfectly happy
to go out and about.  There are quite a few cats in the neighbourhood,
and they all lived happily together.

A big ginger tom cat has now appeared from somewhere (I think he lives
a few doors down) and terrifies them so they don't want to go out.

This means I either make them stay in all day or force them out, which
means they start crying and growling.  And I've seen the ginger brute
chase them both up the garden and really scare them on several
occasions

I don't have a cat flap, and the ginger cat will always come in and
steal the food if I leave the door open, pushing my wimpy cats aside.
He has a magnet on his collar so there's no point in installing a
secure cat flap for my cats to get in and out as he'd still be able to
get in.  He's sat outside my back door at every possible opportunity.

I don't know what to do! he's a bully and no matter how much I've
tried to shoo him away, even trying a bit of water, he just sits and
looks at me.  I spoke to his owner, but she just laughed and said he
was a bit of a bully.

Has anyone got any advice on how to deal with this?  Otherwise, I'm
going to have to move house.

Thanks
Alee
(England)