When asking if time travel is theoretically feasible, one must ask both whether it is conceptually possible, and whether it is attainable.
Before it can be said that time travel is theoretically possible, one must first choose a theory to base the prediction on. In modern physics, there are a number of discrepancies between the behavior of matter at the subatomic level and at an astrophysical level. Because of these discrepancies, varying theories to explain observable phenomena have emerged. Theories about space and time on the subatomic scale can not be extrapolated to explain relativistic observations. Until a single unified theory is created that can both explain the observable and be explicitly tested, the theories of each field will remain just that: theories.
But when discussing time travel in a purely theoretical sense, we are given the freedom to choose the theories that best suit our needs.
Current models of subatomic space, such as string theory (where space is composed of multi-dimensional closed loops) or M-theory (our universe exists in space-time as a 2-dimensional plane) tend to close the possibility of time travel. This is because these theories are mathematical models created to match mathematical data. Their purpose is conceptualize formulas, and are thus imaginary to an extent. It is impossible to prove one theory over another, because all theories conform to the same known data. Until new data can directly test a theory, all subatomic views of space-time cannot be taken as literal interpretations of the universe.
From a relativistic stand point, the theories are much more unified, and also friendlier to the possibility of time travel. According to general relativity, a limited form of time travel exists and is technically feasible. By accelerating close to the speed of light, time appears to pass slower to the traveler than it does to a stationary observer. If one were to travel from earth at near-light speed, he would return to find that more time as past on earth than did in-flight. However, mathematically the difference is negligible.
When talking about time travel, the nonscientist will undoubtedly bring the issues of black holes to the table. Black holes are not a rip in space-time; at best they are a severe depression. Matter is simply absorbed, and eventually re-emitted as radiation. Matter and energy are conserved, and there is no reason to believe that a black hole can transport anything through time.
More conducive to time travel are theories that permit the warping of the space-time continuum. Einstein-Rosen Bridges, better known as wormholes, are said to able to connect different regions of space-time. The possibility is based in the formulas of general relativity, which has not been fully proven. But for the sake of argument, let us assume that these wormholes are theoretically possible. This does not make it feasible to create, much less control. The energy required would be on the scale of millions of nuclear bombs, all detonated in space of a pin point. This is more energy than a black hole, supernova or any cosmic event known to man. Even in the best of circumstances, this hole wouldn't allow so much as a lab rat the time and room to pass through before it collapses.
So when Dr. Stephen Hawking said "...The best evidence we have that time travel is not possible, and never will be, is that we have not been invaded by hordes of tourists from the future," the reason we have yet to see time tourists may not be because time travel is impossible, but because time travel is a practical impossibility.