Think of it like the reproduction of any organism -- the male parent affects
the offspring's characteristics, but not the appearance of the mother. Apples
are maternal tissue, except for the seeds (which do differ, at least
genetically, based on the source of pollen). But there's no reason for the
pollen source to have any effect on the rest of the fruit. There are a few
plants in which the pollen source does affect fruit characteristics -- "Indian"
corn, dates, etc. But not so in most plants.