Lost: Did Jacob's brother turn into the smoke monster, or die upon its unleashing?
For me, the irrefutable evidence is that Jacob finds his brother's body after throwing him into the heart-cave, proving his brother died; if his brother had turned into the smoke monster, than it is not likely that his body would have been found.
The argument then might be, "his mind/soul/what-have-you separated from his body", but remember that Jack was spit out of the heart-cave in the same manner and was still whole, and I am also reminded of what Ben says in the episode of the same name, "Dead is dead", and Widmore saying "that thing masquerading as John Locke", as it can also be said to be masquerading as Jacob's brother. It has shown it will go at great lengths to pretend to be something it's not to get what it wants (i.e. to get off the island to destroy everything). In "Across the Sea" Jacob's brother is shown to be capable of lying and mischievous when it came to Mother, but nothing nefarious like what we see with the monster.
Did Jacob believe it was his brother? He never refers to it as his brother, and although he refers to it as "him" or "man" it may just be for smoother communication since the people he speaks with may not fully grasp what it is they are dealing with (who could?). I also find there is a suspicious air to the way Jacob talks to it in the few conversations they do have; he doesn't speak to it with the love someone might express to a sibling they have wronged.
Could Jacob have killed it himself? When speaking to the candidates, he says "You must do what I couldn't; what I wasn't able to do"; and, as Mother says, "Jacob can't lie". To kill the monster required a Desmond, whose ability to withstand the amounts of electromagnetism in the heart-cave seems very rare, and we see no evidence that Jacob had that ability, and even if he did, it would take a team to help him get in and out and locate and kill the monster, as Desmond went unconscious after pulling the plug, and from what we see from Jacob he is a loner, at least while on the island. Perhaps he could have assembled a team as we see him doing some recruiting in the show (e.g. Ilana's group), but he may not have deemed himself capable of pulling it off; the love and dedication the group that did kill the monster had for each other was not something he could have manufactured for himself. Remember too that Jacob had to die before the monster could be stuck in a single human form, making it easier to find
I agree, it's fun to think about. Thanks for giving me some points to argue, LOL.