The first souvenir is a toy boat found in the Safe Room, above Emily's bed. While it is possible to return to this room much later in the game, the item will be gone by then, so make sure to snatch it the first time out.
Mission 2 -
This souvenir is the helmet of the elite guard, and is gained by taking out an elite guard in the Dockyard Quarter or by Addermire station. This encounter does not need to be fatal.
This souvenir is the head of a clockwork soldier from Jindosh's mansion. It is possible to get this souvenir later, but if you never took out a single clanker in all of mission 3, you missed out on some loot and an achievement. Remember than clankers don't count as kills, and can be taken out using rewire tools to maintain ghostly status (a rewired clanker explodes after a few minutes).
This souvenir is the syringe you used to make a cure for the Crown Killer in Addermire. See the Mission 4 Walkthrough for further details.
This souvenir is a whalebone sword, which appears as a lootable item on the top floor Conservatory. In situ it is a fragmented sword on pillows in a glass display case.
Two souvenirs this time- Corvo's old trophy hidden behind some bricks in his childhood home, and a silvergraph of the Wale, found in the silvergraph studio (the studio is initially locked, but you can snag this early if you like just by smashing open a window).
This souvenir is a calendar that can be found the very first time you travel into the past- it rests just by the door, inside the piano room.
This souvenir is found inside the palace Vault, on a table with some whalebone. If you listened to Foster's audigraph on the Wale, you know the grave significance of this particular wooden gazelle head.
This souvenir is acquired after you've left the Wale for good. When you reach the throne room in Dunwall Tower, retrace your steps from Mission 1 to the safe room. It's changed a lot, but this doll (which is present but unlootable in Mission 1) is your final souvenir.