Here's an essay about male vs. female writers and how they feel about writing across gender.
The male writers felt that gender influenced their writing, and some preferred not to write female characters. The female writers did not feel that gender influenced their writing, that it was all about creating plausible characters. This fits a pattern I know from various sociology classes, which is that in any pairing of an advantaged and a disadvantaged group, the disadvantaged people have to know a lot more about the advantaged people in order to accommodate them for sake of personal survival. In a patriarchal society, women will therefore tend to know more about men, and it will be easier for female writers to handle male characters well than vice versa. Of course there's variation; some female writers don't handle male characters well, and some male writers produce brilliant female characters.
My own thoughts on this issue include ...
1) You don't "have to" include any particular thing in your creative output. What projects you undertake is up to you. So if you don't want to write female characters, or male characters, or whatever -- that's okay. Just be aware that it will affect what audience you attract. That guy who said he'd leave writing female characters to women, who are better at it? He's almost certainly right. His female characters would probably suck, and he probably wouldn't enjoy writing them, and that would make nobody happy. Don't use a screwdriver to pound nails.
2) Writing outside your traits is harder than writing inside them. Some writers are diverse and love to push their envelopes. Others are comfortable establishing a niche. It's great to venture outside writing what you personally know, but it takes a lot of extra work to make it plausible and effective. If you don't want to do that work, or you happen to suck at it, then you're probably better off writing what you know. If you want to explore farther, be aware that sometimes you will fall into a pit of alligators and that will suck.
3) All of the stories need to be told. They do not all need to be told by any one person. The more people writing, the more diversity of representation will happen naturally. If you put together stories by people with a certain trait, there are often trends. Men and women have different favorite topics, although there is plenty of crossover. Even in the same field or genre, they often tell different kinds of stories, and that's awesome. Genderqueer folks of all flavors widen the spread yet again. Whenever there's a split with most of the stories being written by one group, it is advantageous to encourage people in some other group(s) to write their stories.
4) There are different character types. Some are active (changing over the story) while others are static (not changing). Some are simple (they only have one role) while others are complex (they have layers, even contradictions). All of those types are useful in different stories, in different ways. Not every character needs to be dynamic and complicated. Not every story does, even; some people love formulaic plots, and there are entire pulp genres that cater to those tastes. That's okay too. The folks who want more complexity and variation have other branches of pulp, and literary fiction, to enjoy.
5) Write what you like. Read what you like. Use your folding vote to influence entertainment and politics. If you dislike someone's output, you can pick something else. If you don't feel that the mainstream is representing you or your interests, ask for and support things that are different. Crowdfunding makes this much easier than it used to be. You can look for open prompt calls or browse hub sites like Kickstarter. You don't have to settle for oppressive crud.