According to the sources cited by The Daily Telegraph, the refusal may worsen his relations with the US president