Plans have been approved to transform a half-built office block into new homes – despite fears it will impact a nearby tourist attraction.