Not really. Regarding speculation, there are other words for that, views, opinions and so on. Yes, unless you can see past lives, unless you have the ability to see other beings getting reborn, which is one of the powers the Buddha evidently had, unless you can do that, then rebirth stays as a theory. Fine, we can accept it as a probability. We don't say it is absolutely true. But because it makes so much sense, we accept it as a probability. Now that doesn't mean that we accept it 100% though, that's different.
The Kalama Sutta states, "When you know for yourself that a teaching is unskillful, blamable, rejected by wise people, and if followed and performed will lead to harm and ill, then don't follow those teachings." There is no way in the world that you can take the thought of rebirth and say that this is an unskillful thought, this is blamable, this is rejected by wise people, this will lead to harm and ill. Because actually, the thought of rebirth helps any spiritual person, whether they are Buddhist, Christian, Jew, whatever. If they have a thought of heaven and so on, it helps a person to do good. It helps a person to avoid harm, it helps a person to purify the mind, because we're thinking of what's going to happen when the body dies. We're thinking about a reward, or a result that's going to come due to our Kamma. So in that way, to consider rebirth as probable is a beneficial thought, it's a beneficial teaching, it will lead to benefit and happiness. So in that way, it is not at all against the Kalama Sutta.