I think you can sort-of see a science fiction community of authors; the ones who go to cons, or read each others' work and respond to its arguments or pull out themes from it (so, say, there might have been a "Singularity SF" community for the last 10 years or so) or at least agree they're writing SF. Though since Ballard had stories in Interzone I think it'd still be bloody difficult to say he wasn't writing in an SF community of some sort!

But I don't think that means something written by an author who doesn't hang around with other SF authors can't be SF. I mean, 1984 was set in the future, it's about the effects that a technologically and socially different society have on a person within it- it's quite Le Guin-ish, except bleaker- and it's infulenced later SF dystopias. Saying that "literary" is the One True Way to consider it would seem daft.
(If Atwood acknowledged that she does bloody well write spec-fic and met some other authors doing similar stuff, her SFnal elements might be better and less reinventing of boring old wheels. But that's a whole nother snark.)