Choosing the right propeller for your boat is a challenging feat! Here is a guide for selecting the right one as well as some resources.
The proper propeller size for your boat and engine combination is based in part on the wide open throttle (WOT) operating range for your particular engine. You can find this in your operator's manual, expressed in terms of a certain horsepower at a certain r.p.m.

The goal in propeller selection is to determine what style and size will maximize your boat's performance, while allowing your engine to operate in the recommended r.p.m. range. The correct propeller will prevent the engine from over-revving, yet allow it to reach the minimum r.p.m. where the maximum horsepower is produced, with ideal engine loading.

Using your existing propeller, determine your maximum obtainable r.p.m.. If during this test, you begin to exceed the maximum rated r.p.m. of the engine, reduce the throttle setting. If the engine over-revs beyond the maximum recommended r.p.m., you may need to increase the pitch of the propeller.  Increasing the pitch increment by 2" will result in approximately a 200-400 r.p.m. drop. Also, switching from an uncupped to a cupped propeller will reduce your r.p.m (see "Cupping" paragraph below). The cupped propeller of the same pitch and diameter will typically reduce your r.p.m. by approximately 200. If you cannot reach maximum r.p.m., then pitch should be decreased. These recommendations apply to single engine installations only. For most twin engine installations it is necessary to increase pitch by 4".

Once your WOT r.p.m. falls within the recommended range of the engine manufacturer, you have a propeller that is suited to your boat with respect to r.p.m. However, you may not be satisfied with your boat's skiing performance or trolling speed. It may be advisable in these circumstances to have multiple propellers, each to accommodate different boating activities. In all likelihood, more than one propeller will be suitable for your boat and motor combination, depending on your usage. Ski boats need more top end speed, and should choose a prop with a higher pitch. Cruisers and houseboats need more performance at displacement speeds, and should use a prop with a lower pitch to acheive low-end power. It is imperative, however, that the WOT r.p.m. fall within the range specified by your engine manufacturer. If your engine is not able to reach this r.p.m. range, it's operating under an extremely loaded condition and premature failure is highly likely. Our Manager of Technical Services Bob Adriance would like to remind you that the wrong prop can wreck an engine. "I've talked to mechanics that think using the wrong prop is the single greatest cause of premature engine failure," says Bob.

While most of these comments are geared to outboard engines, some also pertain to inboards.  There are other factors that can adversely affect the performance of your propeller. One of these is dings on one or more blades. Another would be having the blades out of alignment, as would occur if you hit something. Either of these could cause vibration or undue stress and ultimately damage your transmission, cutlass bearing and other components. Usually the best way to deal with a propeller issue with an outboard is to get a new prop if it’s damaged or, if you’re not sure you nave the correct prop, try on different ones, with the recommendation of a qualified dealer, until you’ve got it right. But with inboards typically you’ll need to have the propeller(s) pulled and sent to a good prop shop so that they can work their “magic” on your existing prop to repair it or determine that you need another. This includes things such as “swinging it” to determine balance, checking for alignment of the blades, and actually working on blades, such as adding or removing cup and many other adjustments. You would give them all the information about your boat that they’d request and describe fully all the issues you’re experiencing.   

Propeller Size
The size of a propeller is defined with two sets of numbers, diameter and pitch, with pitch always following the diameter.

Diameter
The diameter is two times the distance from the center of the hub to the tip of the blade. It can also be looked at as the distance across the circle that the propeller would make when rotating.

Pitch
Pitch, the second number listed in the propeller description, is defined as the theoretical forward movement of a propeller during one revolution. Since there is almost always a small amount of "slip" between the propeller and the water, the actual distance travelled is slightly less.

Cupping
Many of today's propellers incorporate a cup at the trailing edge of the propeller blade. This curved lip on the propeller allows it to get a better "bite" on the water, resulting in reduced ventilation and slipping, and allows for quicker acceleration, or "hole shot," in many cases. A cupped propeller also works well in applications where the motor can be trimmed so that the propeller is near the surface of the water. The cup will also typically result in a higher top end speed.

Some Problems to Avoid:

Ventilation
Ventilation occurs when surface air or exhaust gasses are drawn into the propeller blades. When this happens, boat speed is lost and engine r.p.m. climbs rapidly. This can result from excessively tight cornering, a motor that is mounted very high on the transom, or by over-trimming the engine.

Cavitation
Cavitation (which is often confused with ventilation) is a phenomenon of water vaporizing or "boiling" due to the extreme reduction of pressure on the back of the propeller blade. Many propellers partially cavitate during normal operation, but excessive cavitation can result in physical damage to the propeller's blade surface due to the collapse of microscopic bubbles on the blade. There may be many causes of cavitation, such as incorrect matching of propeller style to application, incorrect pitch, physical damage to the blade edges, water flow obstruction caused by parts of the boat’s hull or running gear to close to t the propeller and others.

Source: BoatUS Expert Advice