Here is an excerpt from my 2nd book, Skin Care A to Z, that explains my opinion in full:

Facial Brushes
The sales clerk at my department store told me to start using a facial
b rush because it would reach into my pores to clean them better.
Facial brushes do have the ability to reach down into the pores.
Therefore they can cause damage, like particles in scrubs. Presumably,
the reason you are using the brush in the first place is for exfoliation
benefits, yet you can exfoliate in other ways without causing harm
to your skin. Of course it depends on the actual brush and what type
of bristles it has. I’m sure there are some facial brushes that won’t cause
any harm to your skin, but why use them? They harbor bacteria, they
can cause irritation, and then there is the potential to “reach into your
pores” and do some real damage. Use brushes on your body if you
want to—this skin is far less susceptible to irritation than on your face.
But bottom line, I don’t recommend using these brushes on your face.

A good use for a facial brush would be under a beard or even just
a moustache. Due to the dense hair growth, the skin underneath doesn’t
get any action and certainly little if any exfoliation. This can cause
itching and possibly bumps under the skin, although you may not be
aware of their existence.

To exfoliate the area under a beard or moustache, take a facial
brush and gently nudge it underneath the hair onto your skin. Use
this brushing technique prior to washing your face. Gently massage
with the brush until you can feel the stimulation on the surface of
your skin. Go over the entire area under your facial hair. Afterwards,
take some cleanser and wash your face as you normally do, paying
extra attention to the areas you just brushed, and work the cleanser
into your facial hair. Be sure to get a thorough rinse to finish. Use your
spray toner, and moisturize your skin as usual. Do try to get some
moisturizer under your beard, but don’t glob it on there. Just massage
a little bit into the hair to help moisturize the skin you brushed
underneath. By using a facial brush under your beard, you are benefiting
the skin by exfoliating that untouched area.

HOT TIP: Brush your teeth, not your face.