Now… you could certainly try being a Buddhist with wrong views. There’s lots out there. And it’s much better to be a Buddhist whow has a wrong view or two than to not be a Buddhist at all.
Tha said, there is a very common misunderstanding out there that you have to see something with your direct experience before you believe it. Even many Arahants can’t see their previous lives, so what about an ordinary Buddhist?
When we have faith in the Buddha’s enlightenment, then we believe in rebirth. It’s not that complicated.