Lost Theorizing! (A BIG ONE!)
"Jacob" is the spirit of the island. I know it's hokey, but give it a chance. Jacob has been imprisoned by the original hostiles for a very, VERY long time. My guess is that they were originally a tribe of islanders who worshipped the spirit (aka Jacob). They tried to summon Jacob in some sort of manifest form and succeeded. They might have made some sort of Faustian deal with him, most likely a sacrifice of the father/head chief (reverse Abraham and Isaac thing). They got greedy though and Jacob decided they had had enough, but they found a way to imprison and control him. Everlasting knowledge, power, that sort of thing. It can explain why the island has healing properties; one of his 'gifts' could be long life/health/regeneration. And so it was that generation after generation the methods to control him have been passed down and Jacob is nothing more than their prisoner. That the hostiles, now the Others are so smart and so strong is because they sap power from him. Something went awry though (the incident!) and part of Jacob broke off and this is what we know as Smokey. The Others have been trying to find ways to reclaim mini-Jacob (Jakey?) so that they have full control over Jacob.
Fast forward to Flight 815. Now I can't explain the whole cover-up, stuff like that, but on the island Jakey saw these strangers as a possible hope for Jacob to be released. That's what the whole 'scanning' thing is. It looks into the soul of a person to determine whether or not they are fit to rise to the challenge. If not, well...DEAD! It manipulates them with the things it know will most affect them (Jack and Christian, Kate and her horse, Hurley and Dave, hell, Ben and his mom) to try to lead them closer to what can release it. What that is, we don't know yet...
This can also explain the inability to make babies on the island. The only way Jacob can be held captive is if the current Hostiles pass on the instructions to their children. Due to the incident and Jakey getting free, Jacob has some power once more and uses it to render everyone unable to make babies, thus ending this generation of Hostiles and eventually freeing it. The Others are frantic to make babies, and Jacob is frantic to be set free.
I think it's interesting that in this theory, what empowers people to be one of Jacob's 'priests' of sorts is to kill your father. Ben did and it's reasonable to assume that countless before him did as well. Why else would Ben order Locke to kill his father? To make him worthy of their ranks of Jacob-quashing-Others. In that sense Sawyer, Locke, Kate and Jack seem able to release him (Jack in the sense that he drove his father to drink, so the onus is on him).
AH AH! A new thought: My guess is the Incident obviously was due to energy from the hatch being released, thus empowering Jacob and allowing Jakey to split. The reason that things went crazy in the Shack of Doom was Locke having the flashlight allowed Jacob to absorb a tiny amount of energy. Shack of Doom must clearly be positioned over the spot where Jacob is held. That explains why the Others can have power and such in their commune. It really makes you wonder then, what kind of effects triggering the fail-safe did. My guess is that Jacob is more empowered than before and the Others need to recruit as many people as possible to help keep Jacob controlled. If Jacob is getting more powerful, he could be diminishing his 'gifts' to the people, explaining why Ben got cancer (anyone sick before should be wary...Rose, I'm looking at you).
Short and sweet: Jacob is the spirit/lifeforce of the island. He's pissed and getting stronger. The Others need to get him back in control before things go craaaaazy.
The major setpiece for next season? Apparently, it will be The Ruins (like the caves, the beach, the hatch, the commune). RUINS?! Hello spirit-worshipping natives!!