Kristoffer Borgli's unsettling new film balances dark humour, psychological tension but struggles to stick the landing