In general, you cannot buy a vacation home with a 203(b) loan. 203b loans are intended only for owner-occupied primary residences, and, in some cases, HUD-approved second residences. HUD approves second residences on an individual basis in the case of hardship, such as divorces, relocations, and growing families, but not for vacation homes.