I was reading some old rpg books and they had this rule: if a monk hits he has a 75% chance of stunning his target and a 25% chance of killing it.

This is a grammar and math issue. The conjunction "and" means both. Therefore if the monk hits then he rolls % dice to see if he stuns. He then rolls % dice again, regardless of the results of the stun roll, to see if he kills. So he rolls the % dice two times, and determines the stun and kill rolls separately.

How does the math work on this? Is it better to roll one time or two times? If the rule said the he either stuns or kills, but not both, then it would have said, "if the monk hits then roll % dice. A result of 1 to 25 means he kills. A result of 26 to 100 means he stuns."

In that case he is guaranteed to either stun or kill. If you roll separately then there is a chance that you will fail at both rolls. Let us calculate these probabilities.

In the case where you roll once, this is easy. 0.25 to kill and 0.75 to stun, with no other possibilities.

In the other case there are 4 outcomes. Stun and kill, no stun and kill, stun and no kill, no stun and no kill. The probabilities for these are, respectively

0.75×0.25 = 0.1875

0.25×0.25 = 0.0625

0.75×0.75 =0.5625

0.25×0.75 = 0.1875

Adding these up gives 1, as it should. Assume that a kill supersedes a stun and is more desirable. In other words, a stun and a kill results in a kill. The probability of stun and kill, or no stun and kill is = 0.1875 + 0.0625 = 0.25 which is the probability of getting a kill in the alternate situation discussed above.

The probability of getting just a stun is 0.5625 as opposed to 0.75 in the alternate situation, and the probability of getting nothing is 0.1875. So this sitauation is worse than the case where you roll one time.