I became a parent and a secondary school teacher in the same year. During my first decade raising two boys and teaching hundreds of children, I began to feel a creeping sense of unease, a suspicion that something was rotten. But it was only when my elder child started secondary school that my worlds collided and the source of the problem became clear to me: today’s overprotective, failure-avoiding parenting has undermined the competence, independence and academic potential of an entire generation.

Over a decade of teaching, I’d seen my students descend in to a constant state of fear and trepidation, a state that spells disaster for learning. When students are too afraid to take intellectual risks, too hesitant to raise their hand and take a chance on an idea that could change the course of class discussion, teachers can’t teach. It’s that simple.

We’ve ended up teaching our kids to fear failure – and, in doing so, we have blocked the surest path to their success. Out of love and a desire to protect our children’s self-esteem, we have bulldozed every uncomfortable bump and obstacle out of their way, depriving our children of the most important lesson of childhood: that setbacks, mistakes and failures are the very experiences that will teach them how to be resourceful, persistent, innovative and resilient.

I didn’t intend to teach my children to be helpless. On the contrary, I thought my kids would grow up brave, in the sort of wild, free idyll I experienced as a child. I wanted them to explore the woods with a pocket knife and a couple of cookies shoved in their pockets, build tree houses, shoot handmade arrows at imaginary enemies. I wanted them to have the time and the courage to try new things, explore their boundaries and climb one branch beyond their comfort zones.

But somehow, somewhere, that idyllic version of childhood morphed into something else: a high-stakes, cut-throat race to the top. The pressure to succeed from an early age has ramped up. There’s no longer room in our children’s schedules for leisure time in the woods, let alone opportunities for them to problem-solve their way out of the muck and mire they encounter out there. The more successful our kids are as students, athletes and musicians, the more successful we judge ourselves to be as parents. What kind of negligent mother allows her kids to play alone in the woods, with pockets full of sugar?

Modern parenting is dictated by fear. Risks seem to lie around every corner – antibiotic-resistant germs, bullying kids, unfair teachers, lurking paedophiles – so when we tuck our kids in to bed at night, free of cuts, bruises or emotional hurt, we have, for one more day, found tangible evidence of our parenting success. Maybe tomorrow I’ll let them walk to school, we tell ourselves, but today they got to school safely. Maybe tomorrow they will do their own homework, but today they are successful in maths. “Maybe tomorrow” continues until it’s time for them to leave home, and by then they have learned that we will always be there to save them from themselves.

When tomorrow arrives, and the responsibilities, freedoms and risks inherent in adult life arrive with it, overparented children will be less likely to manage all of it successfully. It’s vital that we give children experience managing this autonomy, to build competence in everything from filling out paperwork to making responsible decisions about risk to expressing their needs and wants to adults.

I’m as guilty as the next parent; I’ve extended my children’s dependence in order to feel good about my parenting. Every time I pack my child’s lunch for him or drive his forgotten homework to school, I am rewarded with tangible proof of my conscientious mothering. I love, therefore I provide. I provide, therefore I love.

While I know, somewhere in the back of my mind, that my children really should be doing these kinds of tasks for themselves, it makes me feel good to give them small displays of my deep, unconditional love. My kids will have their entire lives to pack their lunches and remember their school bags, but I have only a brief window of time to be able to do these things for them.

There’s a term for this behaviour in psychiatric circles. It’s called enmeshment, a maladaptive state of symbiosis that makes for unhappy, resentful parents and “failure-to-launch” children who move back in to their bedrooms after university.

In 2012, 36% of 18- to 31-year-olds in the US were living with their parents; in the UK, 26% of 20- to 34-year-olds were living at home in 2013, a 25% increase since 1996. While part of this is due to increased housing costs, these numbers are part of a trend that has been rising for decades. Doing what feels good as parents has fostered a generation of narcissistic, self-indulgent children unwilling to take risks or cope with consequences.

So, what would be a better way? What parenting practice can help our children acquire the skills, values and virtues on which a positive sense of self is built? Parenting for autonomy. Parenting for independence and a sense of self, borne out of real competence, not misguided confidence. Parenting for resilience in the face of mistakes and failures. Parenting for what is right and good in the final tally, not for what feels right and good in the moment. Parenting for tomorrow, not just for today.

The first step is to get honest with ourselves and our children that mistakes have been made, but that we’re learning to learn. The day I finally came to terms with my overparenting, I was determined to start making amends with my children.

I needed to do something immediate, something symbolic, and I knew where to start. My younger son, then aged nine, had never learned to tie his shoes. I blamed this oversight on the invention of Velcro and his preference for slip-on shoes, but, if I’m completely honest, I knew I was falling down on the job. He freaked out when I mentioned the situation, even in my most enthusiastic “Won’t this be a fun project we can do together?” voice. He got frustrated with my instruction, I got frustrated with his helplessness and the entire endeavour dissolved into anger and tears. Tears. Over shoelaces. When I began to look closely at the source of his issue with the shoelaces, I realised that what he was feeling – the frustration and helplessness – was my fault, not his. I had taught him that.

Every time I tied his shoes, rather than teach him to do it himself, I reinforced his perception that I believed the task was too hard for him. Eventually, he and I both began to wonder whether he’d prevail. One day before school, when he’d left his Velcro shoes at a friend’s house and had to wear the backup pair with laces, he said he’d rather wear his wellington boots than try to tie his shoes. He didn’t even care that wearing boots meant he’d have to sit out PE.

So, that afternoon, I took out his backup trainers. I empathised with his worry, and I told him that, while the task might be hard for him at first, I knew he could conquer it with some effort and perseverance. I told him I was so confident that we were going to stick with it until he mastered those darned shoelaces. In less than an hour, the embarrassment he’d felt about being the only child in his year who couldn’t tie his shoes was gone. He’d succeeded, and I’ve hardly seen him so proud of himself. All it took was a little time, a little faith in each other and the patience to work through the tangle of knots and loops.

It’s not always going to be this simple. The stakes get higher and the consequences get bigger as our children get older. Lumpy knots and uneven shoelaces give way to flawed university dissertations and botched job interviews, and there’s only so much time available to instil confidence and resilience.

The work of raising a resourceful adult takes time, but it begins with a simple equation. We need to give our children autonomy, allow them to feel competent, and let them know we support them as they grow. This process begins the moment our babies fail to grasp a toy or fall as they toddle across the room and continues until they head out into their own lives. The sooner parents learn to appreciate the positive aspects of hardship and allow children to benefit from the upside of failure, the sooner all of us will have the opportunity to share in the moments of pride such as the one I saw on my son’s face as he tied those laces.

• This is an edited extract from The Gift of Failure by Jessica Lahey, published by Short Books, £14.99. To order a copy for £11.99, including free UK p&p, go to bookshop.theguardian.com or call 0330 333 6846