Quick. What's the point of estimating stories? If you said determining how long they will take to implement, you only got part of the equation. In addition to helping the team to project the contents of future iterations, estimation can be an invaluable way of getting the team up to speed quickly on what is coming soon.
When agile teams estimate stories, they typically do so by estimating the relative size. I.e., if story A is a '3' and story B is a little under twice as difficult, then it is a '5'. Over time, teams prove how many points they are able to accomplish in an iteration (their velocity). And this is then used to determine how much they'll likely get done in future iterations.
So if the goal is to get an accurate estimate, you'd be best off getting the person most familiar with it to do the estimate, right? Not necessarily. The goal is a team goal. Estimate as a team. Have each person put out a number at the same time. Have the people giving the low and high number explain their logic. Re-estimate until you converge.
There are several advantages of this approach. First, it is valuable for the whole team to understand what lies ahead of them. Since everyone must give a number, they have a vested interest in becoming familiar enough with the story to estimate it. As a result, the team members can leverage this knowledge of what's coming in their work leading up to it. And there is more flexibility in who does the work when the story comes to the top of the priorities since everyone understands what it is.
Second, there is value in the wisdom of crowds. James Surowiecki wrote a book about this topic. Each person brings their own experiences and perspective. This tends to give a more well rounded estimate. Those less familiar with the area may ask questions that seem obvious at first, but when you have to answer them, you realize they're not as simple as you thought.
But doesn't this slow the meeting down? Not too much. The first few stories might take a bit longer, but people get up to speed pretty quickly and patterns emerge. Once you get into a rhythm, it starts to flow. As a result, the team can quickly come up to speed on what's likely coming soon and can provide a reasonable estimate of how quickly they'll be able to get it done.