Rather than defining the "self" (map or territory), let's keep it very practical.
If you ever talk to someone who cannot remember their identity, such as a victim of traumatic total amnesia or a victim of senile dementia, you can directly recognize the tragedy and horror the person is suffering.
Even worse is to experience the acute pain that someone who does know the amnesiac experiences. The wife or husband or children have a still living body that looks like their spouse or parent, but the person inside is "gone", lost.
We may debate about what the self "really" is, but take away the memories and that individual self is, to all intents and purposes, gone.