Plans to redevelop an abandoned leisure centre remain in limbo amid claims the scheme cannot afford to provide a single affordable home.