Yup - we train ourselves to ignore distinctions as "not meaningful" because of our native languages, and then when we learn another language, one that uses those distinctions, it bites us back. You can get it later on, mind you, but it's always a bit of a pain.

My personal example of that is from Italian (L2): it took me a few years to be able to reliably distinguish pairs like "pena" (pity) and "penna" (feather), simply because Portuguese (L1) doesn't care about consonant+vowel length.