Wales' controversial 20mph limit in built-up areas came at a cost to Welsh Labour, says former minister.