Go to the revise page for the listing. Then click on the EDIT button next to VARIATIONS. This takes you to the variation details page to edit. This page can also control your "default" photos.
First you will see the "Add default photos". Remove all photos and re-add the 12 you originally wanted in there. Scroll down to "Add variation photos."
MAKE SURE you choose "SIZE" in the pulldown menu under where it says "Change photos in your listing based on this attribute. This determines which photos buyers see when they select a variation option"
You will then see your two Sizes listed underneath. Click on each respective size and remove all the photos. Re-add the respective 10 photos intended for the size (basically like what you did with the default photos earlier). Do the same thing for the next size. Click "Save and Close" at the bottom.
It will take you to the default revision page and the changes you've made will be reflected on this page (sometimes this takes up to 10 seconds). Click "Update listing" on the bottom and follow thru to save it.
Have a beer. Done.