ADMIN: signups tomorrow; while you wait: nominations process
First, as some of you have already noticed, the deploy with the new challenges code is chugging away done, haha, since as I was writing this we discovered bugs and I had to interrupt to go code *g*. (BTW, you can follow progress reports on twitter at @AO3_Status.)
BUT -- just to let you all know, signups will open officially TOMORROW, in the morning US Eastern time, after I have loaded up all the tags and we have banged on the archive a bit. I will be turning the signup form on and off sporadically tonight for final "omg please don't break" testing, but please do NOT jump in if you happen to see it open, since by the laws of murphy that probably ensures it will all collapse in a heap of flaming fire. *g*
If you would enjoy being a possibly-lightly-singed guinea pig, please DO feel free to hang out in #yuletide on irc.sff.net; I will poke in there if I need recruits! Otherwise, just watch this space and yuletide_admin for the announcement that signups are officially open. :D
The final updated list of fandoms has been posted (you might have to force-refresh if your browser had the other version cached), with additions and deletions and all the corrections we were able to make. It's even less alphabetical than it was, though, sorry! Many apologies if we missed yours. <3
To give you all something to gnaw on in the interim: as elyn posted the other day, we are always happy for useful suggestions for improvements to the nominations process (or anything else for that matter), but it's hard for you guys to actually offer those without some more understanding of the situation and the criteria, which I have a little time now to provide in more detail, along with the initial design for the nominations code as it will work next year (there just wasn't enough time to get it in this year, alas).
Okay, so the basic problems that a nominations system has to solve:
- Automating the process of creating missing canonical tags, including:
-- avoiding unnecessary duplication & making sure the tags do not exist already
-- oversight of the spelling/capitalization
-- ensuring they are in UTF-8 character encoding system
-- checking interconnections between tags (ie fandoms wrangled into the correct media, characters wrangled into the right fandom).
- Automating the process of winnowing out nominations in various categories (Yuletide wants to reject fandoms that are too popular; another challenge might want to reject characters that are too popular within a given fandom; another challenge might want to reject fandoms that are too small, too recent, etc.). This entails:
-- finding some reasonable internal AO3 metric to set the benchmark
-- alerting users if a nomination is rejected
-- allowing moderator to override manually
Limitations:
- This is not a Yuletide-specific system, none of the challenges code is. The gift exchange code has been heavily influenced by the Yuletide process, but it still needs to be generic, so that anyone can use it to run their own gift exchanges.
- Not adding excessive complexity to configuration. (Challenges are to some extent unavoidably complex to set up, but too many configuration options are still not desirable. Any additional option that a moderator has to choose among should have serious bang for the buck and represent a significant variation that people are likely to use.)
- Using AO3 data as the benchmark is imperfect for several reasons: the site is still young and the population of users is still relatively small; challenges tend to bring in an influx of new participants who are not already present and whose fannish experience/production isn't reflected on the site at the time of signup; not all participants use their accounts outside of the challenges they sign up to participate in. Over the long run this can be expected to improve, but in the short run it produces flaws. Manual oversight is the only effective option for fixing this at this point.
- Using external sites (eg a system that checked eg fanfiction.net, google, livejournal interests, etc), is not feasible because the external sites are not under our control, which makes the code unreliable and potentially blocking (ie something that would keep the challenge from proceeding effectively).
(For instance, if we write a screen-scraper today that goes to ff.net and picks up the # of stories in a fandom; tomorrow ff.net could make potentially even a small non-user-visible change to their interface and break this code with no warning. Or the system ignores resources when they can't be reached, and then LJ is down when nominations start. A fandom that is huge on LJ but small elsewhere gets through in hour one of signups; LJ comes back up and in hour two of signups the same fandom is blocked. These are just examples, but hopefully that makes it more clear why it is not great design to depend on external sites (unless it's something like Amazon S3 where they have a set API and you pay them moneys to commit to not breaking it from under you).
That's the basic set of constraints. They are probably hard to grok in a vacuum, so here are my initial rough sketches for the system:
Basic nominating process:
Adding option for restricting nominations, and sending final tags on:
Wranglers have oversight:
And then there would be a final step where the moderator got to confirm all the tags to be made available for the challenge, and then could update them going forward.
Okay, whew. So that is the outline of the system. If you guys have suggestions or ideas for improving on this, please do hit us with them. (And if you find this kind of thing neat and/or entertaining, come volunteer. :D)
BUT -- just to let you all know, signups will open officially TOMORROW, in the morning US Eastern time, after I have loaded up all the tags and we have banged on the archive a bit. I will be turning the signup form on and off sporadically tonight for final "omg please don't break" testing, but please do NOT jump in if you happen to see it open, since by the laws of murphy that probably ensures it will all collapse in a heap of flaming fire. *g*
If you would enjoy being a possibly-lightly-singed guinea pig, please DO feel free to hang out in #yuletide on irc.sff.net; I will poke in there if I need recruits! Otherwise, just watch this space and yuletide_admin for the announcement that signups are officially open. :D
The final updated list of fandoms has been posted (you might have to force-refresh if your browser had the other version cached), with additions and deletions and all the corrections we were able to make. It's even less alphabetical than it was, though, sorry! Many apologies if we missed yours. <3
To give you all something to gnaw on in the interim: as elyn posted the other day, we are always happy for useful suggestions for improvements to the nominations process (or anything else for that matter), but it's hard for you guys to actually offer those without some more understanding of the situation and the criteria, which I have a little time now to provide in more detail, along with the initial design for the nominations code as it will work next year (there just wasn't enough time to get it in this year, alas).
Okay, so the basic problems that a nominations system has to solve:
- Automating the process of creating missing canonical tags, including:
-- avoiding unnecessary duplication & making sure the tags do not exist already
-- oversight of the spelling/capitalization
-- ensuring they are in UTF-8 character encoding system
-- checking interconnections between tags (ie fandoms wrangled into the correct media, characters wrangled into the right fandom).
- Automating the process of winnowing out nominations in various categories (Yuletide wants to reject fandoms that are too popular; another challenge might want to reject characters that are too popular within a given fandom; another challenge might want to reject fandoms that are too small, too recent, etc.). This entails:
-- finding some reasonable internal AO3 metric to set the benchmark
-- alerting users if a nomination is rejected
-- allowing moderator to override manually
Limitations:
- This is not a Yuletide-specific system, none of the challenges code is. The gift exchange code has been heavily influenced by the Yuletide process, but it still needs to be generic, so that anyone can use it to run their own gift exchanges.
- Not adding excessive complexity to configuration. (Challenges are to some extent unavoidably complex to set up, but too many configuration options are still not desirable. Any additional option that a moderator has to choose among should have serious bang for the buck and represent a significant variation that people are likely to use.)
- Using AO3 data as the benchmark is imperfect for several reasons: the site is still young and the population of users is still relatively small; challenges tend to bring in an influx of new participants who are not already present and whose fannish experience/production isn't reflected on the site at the time of signup; not all participants use their accounts outside of the challenges they sign up to participate in. Over the long run this can be expected to improve, but in the short run it produces flaws. Manual oversight is the only effective option for fixing this at this point.
- Using external sites (eg a system that checked eg fanfiction.net, google, livejournal interests, etc), is not feasible because the external sites are not under our control, which makes the code unreliable and potentially blocking (ie something that would keep the challenge from proceeding effectively).
(For instance, if we write a screen-scraper today that goes to ff.net and picks up the # of stories in a fandom; tomorrow ff.net could make potentially even a small non-user-visible change to their interface and break this code with no warning. Or the system ignores resources when they can't be reached, and then LJ is down when nominations start. A fandom that is huge on LJ but small elsewhere gets through in hour one of signups; LJ comes back up and in hour two of signups the same fandom is blocked. These are just examples, but hopefully that makes it more clear why it is not great design to depend on external sites (unless it's something like Amazon S3 where they have a set API and you pay them moneys to commit to not breaking it from under you).
That's the basic set of constraints. They are probably hard to grok in a vacuum, so here are my initial rough sketches for the system:
Basic nominating process:
Adding option for restricting nominations, and sending final tags on:
Wranglers have oversight:
And then there would be a final step where the moderator got to confirm all the tags to be made available for the challenge, and then could update them going forward.
Okay, whew. So that is the outline of the system. If you guys have suggestions or ideas for improving on this, please do hit us with them. (And if you find this kind of thing neat and/or entertaining, come volunteer. :D)