Hi Bizzy Boo, in your example fairly is used as a qualifier, and what it ultimately qualifies is the family, which is a noun. This makes "fairly large" an adjective/adjectival phrase. The only verb in that sentence is "were" and "In a fairly large farmily" is not really part of the main sentence "there were four children", so fairly isn't connected to the verb.

I hope this helps. This is also the kind of question you can ask under the video Intro to adverbs.