Duterte's lawyer Kaufman already said earlier that his family might not be able to visit on Christmas and New Year's Day because these are official court holidays