A former military barracks used to house asylum seekers for more than five years is set to close its doors.