AOL Notification problems
**Jan 15, 19:18 UTC/GMT UPDATE**
As of about 2 hours ago we received confirmation from AOL that we've been placed on their whitelist and that LJ is unblocked from their automated spam detection programs; this was verified using marta's email test account. Part of the delay was explained by AOL as such:
"[AOL's] automated high volume spam URL detection process was not allowing automated removal of the livejournals URLs because the sites were generating too many complaints... as soon as [AOL] removed the URL manually, it'd get re-added."
We're still going to be trying to implement some of the other, longer-term solutions to lessen the possibility that this will happen with AOL or any other email provider.
Also, unfortunately this helps only our users that are using AOL email, we've still got a lot of other underlying notification delivery problems to track down and fix.
-end update-
WHAT DO WE THINK IS GOING ON?
We currently have over 21 million notification ERRORS to AOL users sitting in our database. This is an insanely large amount as we might have a couple hundred errors to any one domain, this includes the large domains like hotmail, yahoo, or gmail. So *millions*?? Something is going on. Note that 21,000,000 errors do not equal 21,000,000 notifications as 1 notification will be re-tried many times and each time it fails, it will record an error.
marta has done extensive testing with our notifications to her AOL test account. She literally pulled apart a notification line by line, sent it in 10 different parts, and isolated the problem to... news.livejournal.com! Basically, any notification that *INCLUDES* a link to news.livejournal.com is getting blocked by AOL. (Which means if you're 1) subscribed to this community, 2) receive email notifications and 3) are an AOL user... you're not going see this in your AOL inbox!)
Of course this includes news posts, but also all notifications on our PERSONAL journals. Notifications like: journal responses, friend's birthday's, addition/deletion from friend's lists, etc. Those types of notifications have a footer that includes links to our FAQ, Support board and News, among other things.
WHY DID THIS HAPPEN?
We *think*, and without verification from AOL this is purely speculative on our part, that enough AOL users have flagged our News posts as spam (wait, what? How could you?!) or "abusive" in some way and that AOL is now outright rejecting any email that even contains LINKS to news.livejournal.com.
They are blocking us at the AOL mail servers; they are rejecting these emails before it even gets to your inbox. You, as the user, are protected from these emails and will not find them at all, even in your spam filters. Our servers see the rejection, take their hurt feelings, put an error into our notification database, and wait to retry them at a later time. Because of the severity of this problem however we are suspending all notifications to AOL email addresses for 24 hours until we can get resolution on this. If the errors continue to pile up it could risk bringing our whole system down.
WHAT IS LJ DOING TO FIX THE PROBLEM?
WHAT ABOUT NON-AOL USERS NOT GETTING NOTIFICATIONS?
We've checked most of the popular lists of known spammers/IPs and we're not on them. That's the good news. The bad news is that any ISP, company or domain is able to (and should) develop their own list of what is acceptable email. And it's quite possible that LJ is on their unacceptable list. I'm hoping that maybe removing the news reference might help out with other email domains but it's probably just wishful thinking on my part.
WHAT CAN WE DO TO HELP?
I'm so glad you asked!
bt
As of about 2 hours ago we received confirmation from AOL that we've been placed on their whitelist and that LJ is unblocked from their automated spam detection programs; this was verified using marta's email test account. Part of the delay was explained by AOL as such:
"[AOL's] automated high volume spam URL detection process was not allowing automated removal of the livejournals URLs because the sites were generating too many complaints... as soon as [AOL] removed the URL manually, it'd get re-added."
We're still going to be trying to implement some of the other, longer-term solutions to lessen the possibility that this will happen with AOL or any other email provider.
Also, unfortunately this helps only our users that are using AOL email, we've still got a lot of other underlying notification delivery problems to track down and fix.
-end update-
WHAT DO WE THINK IS GOING ON?
We currently have over 21 million notification ERRORS to AOL users sitting in our database. This is an insanely large amount as we might have a couple hundred errors to any one domain, this includes the large domains like hotmail, yahoo, or gmail. So *millions*?? Something is going on. Note that 21,000,000 errors do not equal 21,000,000 notifications as 1 notification will be re-tried many times and each time it fails, it will record an error.
marta has done extensive testing with our notifications to her AOL test account. She literally pulled apart a notification line by line, sent it in 10 different parts, and isolated the problem to... news.livejournal.com! Basically, any notification that *INCLUDES* a link to news.livejournal.com is getting blocked by AOL. (Which means if you're 1) subscribed to this community, 2) receive email notifications and 3) are an AOL user... you're not going see this in your AOL inbox!)
Of course this includes news posts, but also all notifications on our PERSONAL journals. Notifications like: journal responses, friend's birthday's, addition/deletion from friend's lists, etc. Those types of notifications have a footer that includes links to our FAQ, Support board and News, among other things.
WHY DID THIS HAPPEN?
We *think*, and without verification from AOL this is purely speculative on our part, that enough AOL users have flagged our News posts as spam (wait, what? How could you?!) or "abusive" in some way and that AOL is now outright rejecting any email that even contains LINKS to news.livejournal.com.
They are blocking us at the AOL mail servers; they are rejecting these emails before it even gets to your inbox. You, as the user, are protected from these emails and will not find them at all, even in your spam filters. Our servers see the rejection, take their hurt feelings, put an error into our notification database, and wait to retry them at a later time. Because of the severity of this problem however we are suspending all notifications to AOL email addresses for 24 hours until we can get resolution on this. If the errors continue to pile up it could risk bringing our whole system down.
WHAT IS LJ DOING TO FIX THE PROBLEM?
- We are trying to remove ALL references to News in our regular notifications. This will allow affected users to at least get their updates and notifications and they still have the option of going to news directly if they want to check it out. We're trying to get this workaround done within the next 12-24 hours.
- We've also been working directly with AOL postmasters for the past week! Unfortunately, of the 2 we've been working with, 1 of the email addresses started bouncing back starting yesterday. We're (ok, mostly marta, she's awesome) will continue to work with AOL and to verify how and why LJ is getting blocked and most importantly, how to get us UNBLOCKED!
- We're going to try to get an "unsubscribe" feature more easily implemented into our news posts. This will be specific to just the news community. Every other notification will be adjustable from your account page. We're hoping this will cut down emails being sent to those of us that don't want news posts in our inbox and will then result in lower spam flaggings.
WHAT ABOUT NON-AOL USERS NOT GETTING NOTIFICATIONS?
We've checked most of the popular lists of known spammers/IPs and we're not on them. That's the good news. The bad news is that any ISP, company or domain is able to (and should) develop their own list of what is acceptable email. And it's quite possible that LJ is on their unacceptable list. I'm hoping that maybe removing the news reference might help out with other email domains but it's probably just wishful thinking on my part.
WHAT CAN WE DO TO HELP?
I'm so glad you asked!
- A lot of thought and effort goes into our news updates, but hey, if you don't want to get them in your inbox, we understand. Instead of just marking it as spam, PLEASE PLEASE PLEASE either remove the news community from your friends list or turn off your notifications by referring to FAQ 272.
- If you do contact your ISP or your email provider, please ask them to check if their mail servers are rejecting any livejournal.com email or emails that contain links to livejournal.com! So they'll need to see if they're blocking not just our mail servers but also if they're blocking any content that contains links to livejournal. Many times the person that answers the phone may not know how, or where, to check this information. Try to escalate it if possible as it may require a systems, network, email or security administrator (depending on the company) to actually verify if they're blocking us at different points within their mail system.
bt