The problem that auction houses are facing for works under $1 million is squeezed margins, as overhead rises and sellers demand more of the profits.