I totally hear you. I grew up riding dirt bikes myself or on the back of one with my dad (horrible rider) and we crashed a few times (helmetless, of course). I wasn't attached to him, so I could get my hands out to help break my fall. As could the people you were taking piggy back. IMO, a kid under the age of 2 doesn't react to falls nearly so well- they rarely are able to break their own falls unless it's just a trip forwards. Being strapped to something or someone will hinder this even more.
Do what works for you, I'll do what works for me. Clearly we have very different risk thresholds, and that's ok.
Both times I've taken adults down it was an act of necessity, not casual recreation. Both were relatively petite women. One was injured, the other exhausted, cold, crying, and completely gripped. The terrain was an easy blue, the area refused to let her download. There wasn't much choice. The other was returning to a ski area boundary. A young woman had fallen, sprained her ankle and to top it off, rolled into a small creek. She was injured, cold, unable to ski, but otherwise fine. There were no emergency services available without considerable wait and huge expense. Sure, we could have made a sled from her skis, etc... But efficacy dictated the speedier solution. It was about a 1.5k glide on a path through the woods to get her to a patrol shack. I was on teley gear for that one. Yes, I felt absolutely rock solid and comfortable. She probably weighed 105 lbs, and she was in fact pretty wet even after we put together as much dry warm clothing as we could muster. It was pretty much like carrying an over sized pack in both cases.