Is It A Bad Idea To Change Your Oil Filter Without Changing Your Oil?

When you need to change your oil filter, it's tempting to skip the messy oil change and just swap the filter, but is that the right thing to do?