You're right, but you're missing a key point. Every edit changes the way the image looks. With destructive editing those changes are "baked" into the object you're changing, and that is data loss. If you want to make a change to that edit you want it to still have the information from the original image so it can be included and changed into the new result you want. Destructive editing doesn't allow for that. It's like if you bend a metal wire, you just crumple it up, and then you want to straighten it again - you won't be able to get it perfectly straight. Non destructive editing does allow for that because it still has the original information, it just doesn't display it in its original form, it displays it with the edit you've made to it, and the edit is "live" so you can change it. It has nothing to do with layers per se, but using layers can be a way to do certain edits in a non destructive way.

If you don't grasp the difference just open Gimp and do the transform test. Paste an image into a new layer, change the transform and squish it to the extreme (non uniformly), make it a few pixels wide only. Apply the transform. Change the transform again and pull the image out to its original aspect ratio. You'll have a blurry image because of all the data that was lost in the first edit. Non destructive editing has been like the most requested feature for Gimp for the past forever for a reason.