The diplomatic penalty is pretty large too since any new clans you meet refuse to open a trade route with you and randomly declare war if you researched only the bushido tech for most of the beginning. Building a church and parking a missionary in there speeds up the the time they are held up there by a little, but those are extra costs that being buddhist doesn't have. Taking one province required 4-5 stacks of ashigaru to sit on the province for a long time to keep it from revolting.
I have played otomo clan and christianity heavily influences the early game.