Forgive me for pointing out the obvious, but the question has only one possible answer: Fact. Let me explain that first before moving on.
Think of the term: Unidentified flying object. Key in on the first word of this; unidentified. This means that it is something you cannot identify, nothing more. Even highly trained military observers will occasionally see an aircraft that they cannot identify. Usually, we take this term to mean that we cannot even identify the type of aircraft, much less the origin of it. But logically speaking, even if we knew that a flying craft was an airplane, but we had no idea of what type, it would still be a UFO. It is very literally an unidentified flying object.
The really interesting questions occur when we do consider objects that cannot be identified at all, and when we start considering where they may have come from. There have simply been far too many sightings of objects that are totally unrecognizable as something of terrestrial origin. There are dozens if not hundreds of thoughts about where these craft may have come from.
It is very true that many can be attributed to military projects that we know nothing about. Logically, though, the sheer volume of the sightings that cannot be explained cannot all be military projects.
There is a problem, however, in thinking that they could be from somewhere else in the galaxy. Star Trek and Star Wars notwithstanding, even if we could attain light speed, it would take over four years to get to the nearest star other than our sun. It is unlikely that there are even planets in that system since it is a ternary star system. Even if there was life there, to make the trip here and back, dozens of years would pass on their world because time passes differently for something moving at the speed of light than for those that are traveling at small fractions of it. This puts substantially further distance stars out of reach, even at the speed of light.
The idea of FTL or Faster Than Light travel doesn't work much better. Why? Because of the Theory of Relativity, and keep in mind that this hasn't been disproved in even a small way. Basically, since energy is equal to mass times the speed of light squared, as the speed grows greater (and the energy expended also grows), the mass would also grow greater. At the speed of light, mass becomes infinite, which means that no amount of energy would be enough to move it over the threshold of the speed of light.
From here, theory becomes bazaar. Some people have suggested using black holes to travel at a greater rate of speed. The problem becomes the same. As an object falls within the event horizon of a black hole, where even light cannot escape, gravitational forces become so tremendous that nothing in nature can resist the crushing pull. This is important. Science emulates nature, never exceeding it. So if nature cannot resist the force of a black hole, neither can man, and one would suppose, other beings.
It is interesting, however, that if a naked singularity could ever be found, and none have been even hinted at yet since it would mean that the original collapsing star could not be spinning at the time of collapse (we have never detected a star that doesn't spin), it would not have an event horizon and theoretically if a carefully calculated tangent could be found, it would be possible to virtually skim over the singularity. It would then in fact be possible to move backward or forward in time!
This is highly speculative, and we have nowhere near the knowledge to do it, but it is possible that we are being visited by beings in our own future. It is a stretch, but no more than any of the ideas put forth to explain "unexplainable" UFO visitations.
At any rate, the one thing we can say for sure, whether we are actually being visited or not, is that UFO's do exist. After all, what we don't know and can't identify is at least more numerous that the things we do and can.