I had to give up.
Thank you everyone for the advice about little Joe- but after a lot of thought I had to take him back :(
I know that eventually it probably would have worked out but I spend sooooooo much of my time trying to keep my older cat alive that I couldnt really keep going. It wasnt fair on my cat or the kitten. I dont think my old ginger boy will be with us much longer and I will wait until that sad day comes before getting a new kitten from the same wonderful lady as little Joe!!!! :)