Energy is never created or destroyed. By reacting materials, you are simply taking advantage of energy that is already there. When you burn gasoline, you're taking advantage of the energy stored in the bonds between carbon and hydrogen. When you react matter and antimatter, the annihilation is releasing energy in the form of particle neutralization (though the majority of that is lost in the formation of unrecoverable neutrinos). it's not creating anything.
Converting matter directly into energy aka squishing quarks until they're electrons, is horrendously inefficient. It would be an incredible miracle to convert even a microgram of anything into energy at an efficiency of 0.0001%. (That is, 99.9999% loss due to waste radiation, friction or other things.) But the problem is real world efficiency of such an attempt would be even lower. it would probably be so inefficienct (like 1*10^-20 %) that it would be questionble to even attempt a postive return. The cores of neutron stars are probably composed of quark-gluon plasma, and even that doesn't burn hot enough to turn matter into energy. (This is to say, proton destruction is a seperate energy release mechanism than antimatter-matter annihilation; it's much harder to do.)
The simple fact is its cheaper, easier, smaller, more efficient and in all practical ways better to do nearly anything else than direct matter-energy conversion. Fusion and Antimatter annihilation for example.
Back to antimatter annihilation. The specific impulse of antimatter annihilation is fantastic compared to any other fuel. (Gunpowder has an ISP of about 60, ammonium perchlorate ~270, liquid oxygen and H2 ~390, hot uranium and hydrogen ~1200, ion-xenon 3500, ion-argon 4000, airbreathing external combustion steam-power ~6000, airbreathing internal combustion petroleum ~10 000, airbreathing Jet A1 turbines ~30 000, airbreathing SCRAM-Jet ~70 000, antimatter-matter annihilation ~250 000.)
However the biggest problem within our lifetimes is going to remain the fact that antimatter is difficult to manufacture-- it can only be made in particle accelerators, so far. The largest amount of it we've ever had was about 1000 atoms of antihydrogen at once. That's less than a sexillionth of a microgram.
Another problem with antimatter beyond being able to make it, is storing it. Antimatter has a habit of exploding when it comes in contact with matter, which means it must be suspended by high energy magnetic fields. These fields are not easy to generate; they take expensive equipment and high electrical demands, which is going to make resource extraction expensive, storage even moreso, and make your spaceship very heavy with the addition of a pretty beefy power generation system (whose absence will make your spacecraft explode due to propellant containment failure).
With the Large Hadron Collider working at full power for a century, we would drain the world's oil reserves three times over and make a few tons of antimatter. This might be enough delta-V to put a spacecraft anywhere in the Solar System within a few months (which would be incredible) but it's still piddle compared to large fractions of lightspeed. Also remember that however fast you intend to go, you'll need quadruple the initial fuel requirement and more: 1 ration for the initial acceleration, 1 to slow down again, and 2 more for the return journey, plus a bunch more to account for the weight of those four rations, plus a bunch more to account for the weight of that first accounting ration, plus more for the second accounting ration... All in all, you're going to need either more efficiency than what antimatter can produce, or a very clever way of making or gathering antimatter. You could harvest it from the sun using a massive space station near Venus or something, but that's going to be expensive and time-consuming to built too.
Simply put, there is no way of squaring up the logistics easily. One can ignore them but to do so is to firmly plant your foot in science fiction away from the hard direction and on the soft side of the wall. There's no problem with being in soft science fiction. Star Wars is soft science fiction, where lasers can blow up planets and spaceships have artificial gravity that makes their internal reference frames feel like cars, ships, busses or airplanes. But when talking about the nitty gritty of distances, travel times, speeds and so on, physicsts beat you to the punch 50 years ago because they were paid to crunch the numbers on exactly what you're thinking about-- who were thinking about it for the same reason you want to, except they went to university for years in the pursuit. Don't think you can outsmart them in an afternoon.