Although my husband is just a carrier of CAH his experience may be helpful to you. My husband was also told that his testes were malformed (one was much larger than the other one) and they were concerned about what it might be. After having a couple of ultasounds they found it to be something called a varicosele (sp?). A varicosele is a group of veins that are clustered together. This causes a concentrated area to have blood flowing through them which keeps the testicles too warm and is not a condition under which sperm will flourish.
It is not necessarily uncommon but it can cause the sperm to die and/or become malformed enough to cause infertility problems.
After some outpatient surgery we were able to have two beautiful babies. I hope this is something that can happen for you as well.
If it is more than this, there are ways that they can go in and remove just one sperm and physically place it inside the egg to increase your chances of achieving pregnancy.
I wish you the best of luck with whatever it may be.(:>)