No, it's not swap in the usual sense. It just writes the texture data to the sd and reads it back, if it uploads the texture to the GPU. As long as the context doesn't change, there's no difference if you load a texture directy or via the virtualizer. Normal Android background operations or web browsing will most likely cause more IO than the virtualizer.
I really don't see how one can draw the relation from your app to a broken (whatever that means...) device. It could be anything. Just the fact that your app was running (which is only natural if it's a wallpaper) doesn't mean that it's in any way responsible for this.
End users are...difficult. They usually have no clue what they are talking about, have no proof of anything but they always have an opinion. And it's never them who's responsible. It's always somebody else. So as long as the Galaxy Y has no design flaw that causes this, I see no relation. And if it has...so be it. You can't fix Samsung's shit. If you want to be sure, you can exclude that particular device in Google Play, so that people which are using it won't see your app in the store (I think you need the proper internal name for that, not just Galaxy Y).