Back to previous post: The morning news

Go to Making Light's front page.

Forward to next post: Discussing Specific Changes to the Hugo Nomination Election: Another Guest Post By Bruce Schneier

Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)

April 7, 2015

On voting systems: a guest post from Bruce Schneier
Posted by Bruce Schneier at 03:18 PM *

I’ve been thinking about voting systems lately, and Teresa has graciously allowed me this space to hash out some ideas with others.

There are a lot of ways to deal with the problem: Brad Templeton has a good list. As a preliminary, this is what I currently think about the general situation.

  1. I think the best choice would be to do nothing. It’s not at all obvious that this is anything other than a temporary aberration, and that any fixes won’t be subject to a different set of abuses and need to be fixed again. I think the worst situation would be a series of rule changes in a continuous effort to stave off different abuses. I don’t think highly of a bureaucracy that tinkers with election rules until it gets the results it wants.
  2. If we choose to ignore (1), the second-best choice is to modify the electorate. The problem isn’t the rules of the vote; the problem is that a voting bloc was able to recruit voters from outside the usual community. Trying to fix that problem by changing the voting rules is very difficult, and will have all sorts of unintended consequences.
  3. If we choose to ignore (1) and (2), this is the thread to discuss how to fix the voting rules.
I’d like to limit this thread to that: discussions of various voting systems and how changing to them might help and hurt. I am less interested in the politics of the Hugo elections and the bureaucratic processes of WSFS than in these voting systems as a theoretical construct. I understand that these are inexorably linked, but think there’s value in discussing the voting systems issues separately.

The Hugo selection process is best thought of as two separate elections. The first is the nomination election, in which voters use an entirely write-in process to select a slate of five nominees in each category by simple plurality. The second is the actual Hugo election, where voters select a winner from the five nominees in each category using an “Australian ballot” instant-runoff system

Let’s talk about what happened this year.

The way to think about the Sad Puppies and their slate is as a political party. Essentially, what we saw was the rise of a political party (technically, two parties with highly correlated slates) in an election that has never had parties before. Parties are powerful for two reasons. One, they focus voters’ preferences onto specific candidates, increasing the power of their votes. Both Dave McCarty and Django Wexler explained how this works:

Let’s consider a hypothetical election between Green and Purple voters. There are 800 Greens in the voting pool, and 200 Purples. The Greens mostly prefer Green works, of which there are, say, 10 in serious contention — we’ll call those G1, G2, etc. The Purples similarly prefer Purple works, P1, P2, etc.

The Greens have no organization. Each Green picks the five works out of the ten that he or she personally likes best. Assuming each work has its fans, this will lead to a vote distribution that is reasonably even — say 95 for G3, 93 for G5, 89 for G8, down to 56 for G1.

If the Purples voted similarly, they would get a similar distribution: 34 for P2, 30 for P10, and so on. In this case, the ballot would be all Green, since the fifth-most popular Green work is more popular than all the Purples.

Instead, Purple Leader says, “Hey, lets all vote for P1, P2, P3, P4, and P5.” The Purples all go along with this. So those five works receive 200 votes each, and the others zero. Now the final ballot will be entirely Purple! The minority, by being more organized, runs the table. The Purples don’t cheat; neither have they suddenly become a majority. They simply have a more effective strategy, considered solely in terms of getting Purple on the ballot.

This is why political parties are so powerful, and why U.S. voters will generally vote either Democrat or Republican even if they prefer a third-party candidate.

The second reason political parties are powerful is that they provide a shorthand for marginal voters. I do this myself: “I have no idea who to vote for in this City Council election, but I normally prefer Democrats, so I’m going to vote for their candidates.” This makes it easier for me to vote, and therefore more likely for me to vote.

We saw both of these dynamics in the 2015 Hugo nomination election. People who normally didn’t pay attention were motivated to vote because they could vote the slate without any further thought, and the existence of the slate focused their votes to make them even more effective.

Normally, this is all good and why we generally like political parties. We don’t in this case because of a divergence between the purpose of the parties and the purpose of the Hugos. In a normal political election, the parties are strongly correlated with the issues of the election. So when we choose to vote for the Republican party, we are choosing a set of policies that a Labour government can be reasonably expected to follow. The Hugos are different. The election is supposed to choose works based on overall quality, but the parties are choosing works based on some moral/ethical/political philosophy. Elizabeth Bear touched on this. So what is normally a good thing — a political party — becomes a bad thing.

Of course, one way to fight a political party is with a rival political party. Many people expect rival slates to appear next year, and for the Hugos to forever be a battle of slates, which means that the Hugos will be a battle of ideologies rather than a referendum on the quality of fiction.

This is not a simple problem to fix. Strategic voting — modifying your vote based on what you know or believe about the votes of others — is a powerful strategy, and probably a dominant one. But there are voting systems that minimize the effects of slate voting.

But remember, no election system is perfect, and choosing one is an exercise in trading off among various problems. It’s may be easy to reconfigure an election system to reduce the effects of a current set of abuses, but it’s much harder to design an election system that is immune from future abuses. Any changes should be examined carefully before being implemented.

More in the comments.

Comments on On voting systems: a guest post from Bruce Schneier:
#1 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 03:25 PM:

This is a suggestion (not fully analyzed) from Ronald Rivest of MIT -- Josh Kronegold has independently suggested a similar idea in a SMOFs post -- to tweak the system to be harder to game. I'll describe the method, then discuss its use in a specific case and more generally.

Method:

  1. Each voter lists as many candidates as he likes on his ballot, up to five.

  2. All of the eligible ballots are tallied. Each candidate gets a tally, which is the number of ballots that list it.
  3. The highest-tallied candidate is declared to be one of the winners. If the number of winners has been reached, then stop; you are done. (If there is a tie for "highest-tallied," use some suitable tie-breaking rule, such as picking one at random.)
  4. Otherwise, eliminate from the pile of ballots any ballot that names the candidate just declared to be a winner. Go back to (2) with the remaining ballots (which are the only "eligible" ones), unless there are no ballots remaining, in which case you stop.

Here's an example.

Ballots:
1. A B C
2. A B C
3. A B C
4. A D
5. D E
6. E D
7. F G H
8. D F H
9. G H
10. A

Tally A:5 B:3 C:3 D:4 E:2 F:1 G:2 H:3
A declared a winner. Ballots 1, 2, 3, 4, 10 eliminated.

Tally A:0 B:0 C:0 D:3 E:2 F:2 G:2 H:3
D declared a winner (won coin flip with H). Ballots 5, 6, 8 eliminated.

Tally A:0 B:0 C:0 D:0 E:0 F:1 G:2 H:2
H declared a winner (won coin flip with G). All ballots eliminated.

Final ballot is A D H.

Note the reduction in power of the voting bloc. In the above example, ballots 1, 2, 3 are a "slate." The system allows them to get one winner, but not multiple.

This is generalizable, of course. You can keep ballots "valid" until 2, or some other number, of candidates win. In this system, there are three parameters: A voter can vote for x candidates. A ballot is counted until y of those candidates win. We need z winners total.

Looking at the current system, x = 5, y = 5, z =5. And x = 2, y = 2, z = 5 is a simple system of only allowing each voter to nominate two candidates.

Things get interesting, I think, when x > y. Perhaps voters could vote for up to five candidates, and their ballots continue to be counted until two of those candidates won. This would reduce the power of slates -- they would only get two candidates on the ballot -- while at the same time allowing voters to vote for many candidates.

There will be other ways to game this system, of course. You could imagine a slate of A, B, C, D where supporters were told: "If your birthday is Jan-June, vote A B, and if it's July-December, vote C D." You could imagine a rival slate optimizing its supporters 75%/25%. But this seems to be an interesting place to look.

#2 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 03:26 PM:

The above method -- with a ballot being eliminated after one candidate wins -- is similar to the Single Transferrable Vote system described by B.L. Meek in this paper.

It's better than the more simple system above, because "excess" ballots above the quota needed to elect the candidate can get their votes transferred to another candidate.

#3 ::: abi ::: (view all by) ::: April 07, 2015, 03:31 PM:

Small correction: ballots 2, 3, and 4 are a slate, not 1, 2, and 3, right?

#4 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 03:38 PM:

Abi: It's actually a bigger correction, but I corrected it. Thanks.

#5 ::: Chris Gerrib ::: (view all by) ::: April 07, 2015, 03:40 PM:

I'm the author of at least one "4 and 6" proposal for this year's Worldcon. The proposal is everybody gets to pick four names, the top six win.

Bruce - care to comment?

#6 ::: dh ::: (view all by) ::: April 07, 2015, 03:42 PM:

Mr. Schneier, you have a lot of fans among the people who volunteer and work with Carter Center, and I can, only speaking for myself, say thanks for the thoughtful post. Having just left Nigeria, it was a real joy to see you commenting on this issue.

Reading your post, I have three questions/concerns:

1. The two closely related slates (party stand-in's) seem to have taken over as many as six or seven categories. Is there any special reason why an award system should have an intensity enhanced nomination process and a popularity based award process? Is this the worst case scenario?

2. There is supposition that the slates won because of bloc voting, but before that is known, don't we have to analyze the balloting and look to see what correspondence the slates have to votes cast? Meaning, there is not yet firm evidence (in terms of science, we can all have strong gut feelings), that slate voting took place. Based on the voting information released so far, do you believe there is data to support that bloc voting according to the slates occurred?

3. The Hugo Awards seem to be very well run, and your suggestions 1&2 seem to be wise, especially since giving up on something that has so far produced acceptable results seems to be hasty given that there are not a lot of very good well tested alternatives to move towards. In general, and in this case, does it seem odd that on average only between 10 and 20% of eligible voters actually vote in the final ballot? In the voting community, there seems to be consensus that the US's outdated voting system leads to voter disengagement, and even with a billion dollars in advertising and millions of hours of outreach, voting levels never get to the universal levels, and often rise much above half. Do the Hugo's have an awareness problem, an electorate problem, or something else?

#7 ::: Cheradenine ::: (view all by) ::: April 07, 2015, 03:46 PM:

The Single Transferable Vote system is currently used for the Academy Awards nominations (more or less - there are a few idiosyncrasies). It does require that nominations be ranked, so it would put some additional load on nominators to not just come up with a set of works they feel should be considered but to also rank them. Given the current situation, I'm a bit wary of making it more challenging for nominators to nominate.

STV does significantly mitigate the impact of slates, though of course it doesn't eliminate tactical voting. (Fun voting theory fact: the only voting systems in which there is never an incentive for tactical voting are dictatorship and random systems.) I think it's superior to the system described in #1, which has the disadvantage of potentially pulling an excessive number of ballots out of the pool if there's strong support for a particular candidate.

#8 ::: rcade ::: (view all by) ::: April 07, 2015, 03:50 PM:

The "4 in 6" idea still seems vulnerable to gaming the vote. The bloc vote organizer lists six nominees for each category and randomly generates a ballot for each bloc voter in which four are picked. All six get 80% of the total bloc vote.

#9 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 07, 2015, 03:51 PM:

Cheradenine @7: That is, all voting systems are susceptible to tactical voting.

#10 ::: Miramon ::: (view all by) ::: April 07, 2015, 03:57 PM:

The solution presented by Schneier in the post above appears to me to be a good one for breaking the power of a plurality-supported slate for forcing a complete set of nominees in a voting category.

But given the appearance of other parties, no doubt next year the SPs will change their strategy from showing off by destroying to the nomination system to simply winning the Hugo in all categories.

With that goal change, they need only to nominate one candidate per category, which if anything is even easier under the proposed system as in the old system.

In the end it comes down to which party can turn out the largest number of voters, regardless of the voting system. So while I support the suggested change above, just as Schneier says in his OP changing the voting system is unlikely to have any substantial effect on the outcomes in any event.

#11 ::: Martin Schafer ::: (view all by) ::: April 07, 2015, 04:00 PM:

This is a stronger version of a suggestion I tried to post on one of the earlier threads during a time when comments were turned off.

Mine was to give voters five points to assign to their nominees in each category, which ups the power of nominating a single person rather than a slate of 5.

I think your suggestion here truly fixes the thing that we have a working consensus on wanting to stop, an organized slate taking an entire ballot category.

The secondary effect will be to include more works that are strongly supported by a group of people who don't share the majority tastes. I'm comfortable with that effect at the nominating phase as the voting system ensures that the award will not go to anything that the majority is not comfortable with.

So adopting this means one faction can't dominate but more factions will be represented (at the nominating phase). This probably will require some change to the 5% qualifying rule if it is made into a proposal.

The two year process for making a change would allow people to see what happens next year and decide if change was still needed.

#12 ::: Martin Schafer ::: (view all by) ::: April 07, 2015, 04:05 PM:

Responding to 7: This system does not require ranking your nominations. All of your nomination receive one vote from you until one of them is on the ballot.

#13 ::: Andrew M ::: (view all by) ::: April 07, 2015, 04:07 PM:

rcade: Yes, but that would require a much greater degree of organisation than exists now. At the moment the organisers simply have to make a post saying 'If you want to increase diversity/upset those nasty leftists/whatever, here is a slate you can vote for'. On this system they have to communicate with each voter personally telling them which works to support. Can they really do that? If they tried, how many takers would they get - people who are ready to vote a publicly announced slate might not be so ready to write to headquarters asking for instructions.

#14 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:07 PM:

Chris @6: Your "4 in 6" proposal is an instantiation of the general proposal at @1: x = 4, y = 6, z = 6.

This feels like the right direction to look, although the parameters might be off.

See rcade @8: "The "4 in 6" idea still seems vulnerable to gaming the vote. The bloc vote organizer lists six nominees for each category and randomly generates a ballot for each bloc voter in which four are picked. All six get 80% of the total bloc vote."

That's right. All of these are going to be gameable, but they'll be less gameable. The proposal reduces the power of a bloc by 20%. Is that enough today? Will that be enough in the future? I don't know.

The real issue, I think, is not the power of any individual bloc. It's the power of blocs in general. Bloc voting is always going to be more powerful than individual voting for the reasons I laid out in my initial post. The question is how to reduce the power of blocs without undermining the underlying system.

This -- and not the politics of the Hugos or the voting -- is why this is interesting to me.

#15 ::: Tim Bartik ::: (view all by) ::: April 07, 2015, 04:09 PM:

The proposed voting system seems to penalize correlated choices. Correlated choices may arise from party slate voting. They may also arise from correlated tastes.

In other words, suppose a number of voters like a certain style of SFF, and in year t, works A and B are the best examples of that. Other voters like a different style, and works C and D are the best examples of that style in year t.

Then there will be some tendency for only one work in each style to end up on final ballot.

This may be viewed as a feature and not a bug if one wants to encourage diverse styles of SFF on the final ballot.

As an alternative, I would go back to Bruce's suggestion about the electorate. One could make the nominating electorate different from the final voting electorate, for example one could restrict the nominating electorate to those who have been sustaining members for at least 3 years. The voting electorate among the finalists could include any sustaining member.

I believe the Academy Awards, for example, for several categories has a much narrower nominating electorate than the final voting electorate, so it is not unheard of.

#16 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:12 PM:

dh @6:

"1. The two closely related slates (party stand-in's) seem to have taken over as many as six or seven categories. Is there any special reason why an award system should have an intensity enhanced nomination process and a popularity based award process?"

I think the best way to think about this is two separate elections: the first to choose five winners, and the second to choose a single winner. There's no reason why the two elections need to be similar, and there might be important reasons why they should not be. It's going to depend on the goals of the two elections.

"2. There is supposition that the slates won because of bloc voting, but before that is known, don't we have to analyze the balloting and look to see what correspondence the slates have to votes cast? Meaning, there is not yet firm evidence (in terms of science, we can all have strong gut feelings), that slate voting took place. Based on the voting information released so far, do you believe there is data to support that bloc voting according to the slates occurred?"

Yes. I think it can be mathematically proven that bloc voting has occurred, even without seeing the actual ballots. I was writing something on that, and will post it later today.

"3. The Hugo Awards seem to be very well run...."

I very much want this discussion not to get into the politics of the Hugos, who votes, who is eligible to vote, and so on. I am interested in the voting theory problem of reducing the influence of bloc voting. Whether any solutions should be applied in any particular case is something I have no expertise in, and not much interest in. So that would be another thread somewhere else.

#17 ::: James ::: (view all by) ::: April 07, 2015, 04:14 PM:

One side effect of the suggested system derives from the fact that readers' preferences are not independent. Thus there may be a strong correlation between, say, readers liking works by Stross and Stephenson.

In a year in which novels by both authors are on the ballot, eliminating all the ballots containing Stephenson (assuming he has more votes than Stross) will eliminate a disproportionate number of votes for Stross on the next pass.

Note that this is entirely without any slate voting.

The net effect would be that works getting on the ballot would be more widely distributed through the phase space of possible SFF novels than would otherwise be the case. Whether this is a net good or not is another question.

#18 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 07, 2015, 04:19 PM:

Just to keep things clear:

This is a discussion of the mechanics of voting and how it affects bloc voters.

Talking about the politics of the Hugos, or the politics of the various puppies, should be done in other threads.

Failure to appreciate this distinction will meet with the usual penalties/mt wth th sl pnlts.

#19 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:22 PM:

On identifying bloc voting.

Dave McCarty wrote:

If you look at an average nomination ballot for folks stating personal preferences.....if you compare what they put in on any category against everyone else....well, they're likely to be mostly unique submissions but they'll likely have a few categories where they match some number of people (a handful if they submit several nominations in that category....possibly a lot if they submit only one).

Such that if you say "what is the average amount of ballots this person matched by category", you'd get a number (for most voters) of maybe 2 to 15 or 20 (it they did a lot of single nominating).

This produces a bell curve.

Imagine a world where some small number of folks submit nearly identical ballots....such that they're match by category could be 50 to 200 depending....for an average match of somewhere WELL ABOVE 100 in each category.

Bloc ballots look different from indepenedent ballots because the number of candidates is very large (every work published during the calendar year) compared to the number of nominees in a ballot.

This means that blocs can be discovered from the ballots themselves, without any need to investigate the greater world to determine if bloc voting is being advertised, advocated, and so on.

Think of it this way. Imagine that each full ballot is an eigenvector in "book space." By taking at all of the vectors together, it becomes a straightforward clustering problem to determine what the blocs are.

This also means, that at least in theory, to imagine an election system that searches for and discounts voting blocs. You could invalidate those ballots, or at least reduce their voting power, with a reasonable degree of accuracy. You would also end up with a voting system that is unexplainable to a normal person -- which I think violates an important rule of a voting system. But that there is a theoretical solution means that there might be practical solution.

#20 ::: jnfr ::: (view all by) ::: April 07, 2015, 04:24 PM:

Niall Harrison has some analysis of the impact of the bloc voting in the comments of this photo, for those who want more detail on that.

#21 ::: dh ::: (view all by) ::: April 07, 2015, 04:25 PM:

Bruce--

Thanks. Very interested to see what you can come up with mathematically to address #2. There is some evidence to suggest that voters following either of the two slates had difference levels of discipline and differing levels of adherence to the slate.

#22 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:26 PM:

Miramon @10:

"But given the appearance of other parties, no doubt next year the SPs will change their strategy from showing off by destroying to the nomination system to simply winning the Hugo in all categories.

"With that goal change, they need only to nominate one candidate per category, which if anything is even easier under the proposed system as in the old system."

Turning out more voters than the other side is a time-honored way of winning elections, and I don't think we can design a moral election system that doesn't allow it.

I don't mean to be trite, here. Increasing the diversity of winners in a multi-winner election (the Hugo nomination process) is something that a voting system can do. Modifying the final election system feels beyond the scope of this discussion. It gets back to changing the electorate.

#23 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:30 PM:

Martin Schafer #11:

"Mine was to give voters five points to assign to their nominees in each category, which ups the power of nominating a single person rather than a slate of 5."

This is generally known as cumulative voting, and much has been written about it.

It's certainly worth discussing in this case.

#24 ::: Cheradenine ::: (view all by) ::: April 07, 2015, 04:31 PM:

Teresa in #9: "That is, all voting systems are susceptible to tactical voting."

Essentially. Voting theorists tend to use a wider definition of "voting systems" than any we would consider actually acceptable - possibly because of mathematicians' general preference to include as few conditions in a theorem as possible.

Martin in #12: "This system does not require ranking your nominations. All of your nomination receive one vote from you until one of them is on the ballot."

I was referring to the Single Transferable Vote system (linked in #2) when I mentioned rankings. The proposal from #1 doesn't require it, but the more I think about it, the more I'm worried that this proposal creates a tactical incentive not to vote for anything that seems likely to be extremely popular, because it will take out all your other nominations. In 2013, about 48% of ballots in Dramatic Presentation - Long Form listed "The Avengers," which means that all other nominations from that half of the pool would be disregarded. So if you're pretty sure plenty of other people will list "The Avengers," there's strong incentive not to list it. If enough people follow this line of thought, of course, it might not make the ballot at all. It's why the "transferable" part of "Single Transferable Vote" is so important.

Sorry to go on after mostly lurking for years. I teach voting systems and it's a subject of intense interest to me, and I'm really angry at the Puppies, and the combination seems to be making me voluble.

#25 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:32 PM:

Martin Schafer @11:

"I think your suggestion here truly fixes the thing that we have a working consensus on wanting to stop, an organized slate taking an entire ballot category."

I know you're not doing this, but this is as good place to say this as any.

I want to caution everyone against jumping to solutions without lots -- say, months -- of thought and analysis. Every system replaces old problems with new problems, and it can take a while before we'll understand what the new problems are and if they're worth the change.

#26 ::: dh ::: (view all by) ::: April 07, 2015, 04:33 PM:

Sorry to go on after mostly lurking for years. I teach voting systems and it's a subject of intense interest to me, and I'm really angry at the Puppies, and the combination seems to be making me voluble.

From an academic point of view, you should be excited to see such a use of voting systems to enforce a point of view. Put it this way, with even just as much as 150 more votes, the two slates could have taken over every single category of note, not just a solid half of them.

#27 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:34 PM:

Tim Bartik @15:

"The proposed voting system seems to penalize correlated choices. Correlated choices may arise from party slate voting. They may also arise from correlated tastes. In other words, suppose a number of voters like a certain style of SFF, and in year t, works A and B are the best examples of that. Other voters like a different style, and works C and D are the best examples of that style in year t. Then there will be some tendency for only one work in each style to end up on final ballot."

This is true, although I suspect that bloc voting will look fundamentally different from the sorts of random correlations we might find. But, yes, it is certainly possible any deliberate behavior we seek to penalize might occur randomly.

#28 ::: albatross ::: (view all by) ::: April 07, 2015, 04:35 PM:

The simple version of bloc voting could easily be detected, but not the slightly more complicated version where I distribute a program to the people who want to vote my slate, which randomizes them so that we don't get above whatever threshhold there is for detecting it.

An interesting problem is, given something like Rivest's proposal Bruce described in #1, what's my optimal strategy? If I knew about how many people were voting (nominating) my slate, and I also had a pretty good idea how many total nominations there were going to be, I could put the required number of nominations to get Alice nominated in (each nomination with a randomized set of other choices), then the required number to get Bob nominated, etc. My intuition is that this never gets as powerful as the slate voting that happened this year, but I'm not sure of that.

#29 ::: Chris Gerrib ::: (view all by) ::: April 07, 2015, 04:35 PM:

I've been thinking a lot about this, even after my rush to get the 4 and 6 into the Worldcon business meeting.

It seems to me that slate voting is a problem because 10% of the voters picking 5 names can overwhelm 90% of voters picking 700+ names (which, mathematically, is what happened).

Mathematically, what is needed is a way to concentrate that 90% on a more manageable set of names. And we need to bear in mind that, no matter what, a straight majority or even a large minority (40% instead of 10%) will be able to dominate.

So, the other way to concentrate the 90% is to have three rounds of voting. Round one creates a long list - say 15 names. Round two creates a short list of 5, and round 3 narrows it to one.

It can still be gamed, in that the slate will get all five of their slots in. But now, the number to lock in the short list goes to 33%. In addition, it might be easier to get people to vote on the long list round - they don't have to write stuff in, just go "oh, yeah, I liked that."

Again, comments appreciated.

#30 ::: lorax ::: (view all by) ::: April 07, 2015, 04:36 PM:

The difference between bloc voting and correlated preferences, I think, is that bloc voters are likely to have all or nearly all items on their ballot the same, not just in a single category but across all categories, while correlated tastes may mean people share two or three out of five nominations in a category. Is there a simple way to reflect this distinction in a voting system that penalizes the former - completely or nearly identical ballots - while allowing for the latter? It may help to consider the specific case of the Hugo nominations not as just having five candidates (for a single category) but however many they have across all categories (sixty-some?)

Any system is gameable, of course, but systems requiring overt and obvious collusion to be gameable, such as "If your birthday is in January or February, vote for A; if it's in March or April, vote for B" may mean fewer people are willing to game the system than if the gaming isn't necessarily obvious, such as "Vote for these candidates if they're the sort of thing you like."

#31 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:37 PM:

On identifying bloc voting:

Dave McCarty wrote:

If you look at an average nomination ballot for folks stating personal preferences.....if you compare what they put in on any category against everyone else....well, they're likely to be mostly unique submissions but they'll likely have a few categories where they match some number of people (a handful if they submit several nominations in that category....possibly a lot if they submit only one). [] Such that if you say "what is the average amount of ballots this person matched by category", you'd get a number (for most voters) of maybe 2 to 15 or 20 (it they did a lot of single nominating).

This produces a bell curve.
[]
Imagine a world where some small number of folks submit nearly identical ballots....such that they're match by category could be 50 to 200 depending....for an average match of somewhere WELL ABOVE 100 in each category.

Bloc ballots look different from independent ballots because the number of candidates is very large (every work published during the calendar year) compared to the number of nominees in a ballot. While there certainly will be correlations in independent ballots, they won't look the same.

This means that blocs can be discovered from the ballots themselves, without any need to investigate the greater world to determine if bloc voting is being advertised, advocated, and so on.

Think of it this way. Imagine that each full ballot is an eigenvector in "book space." By taking at all of the vectors together, it becomes a straightforward clustering problem to determine what the blocs are.

This also means, that at least in theory, to imagine an election system that searches for and discounts voting blocs. You could invalidate those ballots, or at least reduce their voting power, with a reasonable degree of accuracy. You would also end up with a voting system that is unexplainable to a normal person -- which I think violates an important rule of a voting system. But that there is a theoretical solution means that there might be practical solution.

#32 ::: Jeff R. ::: (view all by) ::: April 07, 2015, 04:37 PM:

The ideal system, obviously logistically impossible on several different levels, is to have every voter rank every eligible work as is currently done on the final ballot. So we should be looking for systems that more or less* produce the same top five results that that method would.

I think that the original nomination system with nearly-universal good faith and a norm towards slate-avoidance did a reasonably good job at hitting that ideal. What's not obvious at all is if there's any possible system that can do the job in a post-slate world.


*The main unavoidable and probably generally positive part of that 'more or less' being people reading books they otherwise wouldn't have because of their getting a nomination.

#33 ::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:39 PM:

Cheradenine @7:

"The Single Transferable Vote system is currently used for the Academy Awards nominations (more or less - there are a few idiosyncrasies). It does require that nominations be ranked, so it would put some additional load on nominators to not just come up with a set of works they feel should be considered but to also rank them. Given the current situation, I'm a bit wary of making it more challenging for nominators to nominate."

I agree with this. I don't think that ranking is very useful in the election to choose five nominees.

#34 ::: nathanbp ::: (view all by) ::: April 07, 2015, 04:39 PM:

I'm not sure what the proposed system in #1 has to offer over STV except that it's slightly simpler to understand.

As I see it, the basic problem exposed by the slates is that under the current system each voter is given 5 votes per category, but effectively uses only 1 of them (or less), on average. Where by effectively uses I mean votes for a work that actually becomes a nominee. For example, last year for Best Novel, there were 930 votes total for the nominees from 1595 total ballots (some of these ballots may have voted for more than one of the final nominees). On the other hand, slate voters effectively use all 5 of their votes. Switching to STV reduces all voters to only 1 effective vote at maximum, while still allowing everyone to express the same number of preferences as under the current system. This would reduce the impact of slates to be proportional to the number of voters they manage to gather (or perhaps slightly more since they're voting in an organized fashion), instead of having the outsized influence they had this year.

#35 ::: tavella ::: (view all by) ::: April 07, 2015, 04:40 PM:

Doing nothing doesn't seem to be much of an option. This is the third year, and given they have a successful strategy, they will do the exact same thing next year. And given that the supporting memberships their cadre brought this year cover nominations next year, there's no reason to think they won't succeed. Probably not to the same degree as this year if more are motivated to nominate, unless they recruit new numbers, but enough to still massively distort and possibly block the smaller categories.

However, pretty much every technical suggestion I've seen is gameable, and gameable loses. You aren't talking about ethical actors here; they will be perfectly happy to have a web page that assigns ballots to finesse whatever system is put into place. And when going against a system that wants to encourage diversity of votes, a slate will win in most cases. Which is why politics quickly turns into slate vs slate. Except no one who loves the Hugos wants to start creating counter slates, because even loving slates break the system as it becomes about people maneuvering to be on slates, not people individually suggesting what they think are the best of the year.

I'll say what I said in the other thread: the only real solution is going to be either restricting the pool of nominating voters, or establishing a jury filter.

Sure at some point some of them will get bored, but it won't take many years of either no award or vile winners to utterly destroy even the idea of the Hugos.

#36 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 07, 2015, 04:42 PM:

Cheradenine @24:

Sorry to go on after mostly lurking for years. I teach voting systems and it's a subject of intense interest to me --
And therefore you belong in this conversation. Welcome, in case that wasn't clear.

#37 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 07, 2015, 04:46 PM:

Tavella @35: Don't despair so soon. This process is at its beginning, not its end. All voting systems are imperfect, and all of them have vulnerabilities that can be gamed.

#38 ::: Phil Palmer ::: (view all by) ::: April 07, 2015, 04:49 PM:

There is a proverb, "Bad cases make bad law", that everyone quickly forgets. This corresponds to Bruce's point number 2, in the OP. I would say that the voting system doesn't need to change just because of the puppies. Indeed, doing so is more likely to entrench slates into the system. What you could change, if you wanted to, would be to make nominations transparent, so that every nomination, successful or not, was published with all the nominators' names and addresses.

#39 ::: John Mark Ockerblom ::: (view all by) ::: April 07, 2015, 04:52 PM:

As you note in your post, the Hugo process involves two elections: the nominations, and the final award. The final award election process, which uses instant runoff, seems relatively robust (not totally immune to attacks, but I suspect that few if any of this year's slate will actually win, and most I expect to finish below No Award.)

The nomination election, which currently consists of just picking the 5 top vote-getters, though, does seem more vulnerable to gaming. The main complaint I've seen being is that works that a lot of people though were Hugo-worthy were pushed off the ballot due to the takeover of limited slots.

So what do you think of proposals to make the ballots variable in size, instead of always the top 5 picks? It doesn't prevent slates from getting their candidates on, but it might prevent them from pushing off other works that non-slate voters tend to favor. I'm having a frustratingly hard time locating one technical proposal that others in this forum must have seen-- but even a system as simple as "anything with more nominations than a certain threshold gets on the ballot", a system that many real-world elections use (substituting voter signatures for nominations) could potentially work.

The ballots might get a bit more unwieldy, but in the case of slates of interest only to a distinct minority, most voters could probably figure out quickly which candidates were worth more than cursory attention, and the IRV in the "awards" election would tend to keep the stuff that's only on the ballot thanks to a distinctive minority faction out of the winner's circle.

Would this be an improvement over the current system, do you think? What problems do you see with it?

#40 ::: albatross ::: (view all by) ::: April 07, 2015, 04:53 PM:

If we assume bloc voting/nominations in the modern world, we should assume willingness to at least write a little code. That makes it easy to make the strategic voting or nomination into a computer program, perhaps a bit of Javascript that runs in your browser as you visit the page and generates a recommended nomination ballot. There's no need to assume cumbersome stuff like basing your ballot on your date of birth or some such thing.

One question I have that's specific to the Hugo nominations: How correlated are they, normally? Is it common for lots of people to nominate the same two or three works on top because most people think they're the best? That could make it a lot harder to distinguish bloc voting from just natural chance. (In the world of political elections, if we took all the party affiliations off ballots, I expect you'd still see enormous correlations between votes in different races. Even without any awareness of party affiliations, very few people are going to vote for both Mitt Romney and Bernie Sanders on the same ballot.

#41 ::: rcade ::: (view all by) ::: April 07, 2015, 04:55 PM:

Here's a recommendation to expand the number of nominees in a category dynamically based on nominating patterns:

http://drplokta.livejournal.com/166650.html

From the link: "I would propose that for each category we take the total number of nominations received in that category, subtract the number of nominations received by the most popular nominee in the category (thus removing the effect of a slate, if there is one, on the numbers), and then the shortlist consists of everything that got at least 10% of the remaining number, but with a minimum of five per category and scrapping the existing 5% rule (which has already been causing problems)."

#42 ::: Tim Bartik ::: (view all by) ::: April 07, 2015, 04:55 PM:

#38: Bad cases may make bad law. But in the current case, the power of slate voting has been clearly demonstrated, and there is little probability that within the current system that it can be defeated without forming a counter-slate, which is undesirable.

The system proposed here encourages diversity of candidates, because once you have gotten a candidate named in a category (I assume that is how it would work), you get no further voice. So we would get some very diverse works in each category.

#43 ::: Bruce Baugh ::: (view all by) ::: April 07, 2015, 04:55 PM:

There's a category of posts and threads where I have literally nothing to contribute and everything to learn. This is one of them. Thanks, Bruce S. and everyone - I'm working on the learning part.

#44 ::: albatross ::: (view all by) ::: April 07, 2015, 04:58 PM:

Phil #38:

Publishing everyone's ballot with their names attached seems like a spectacularly bad idea. A small amount of targeted harassment of people who nominated the wrong sort of works would have a big effect of suppressing nominations for controversial stuff, or simply convincing lots of people it wasn't worth the hassle.

#45 ::: dh ::: (view all by) ::: April 07, 2015, 04:59 PM:

@Bruce--

This also means, that at least in theory, to imagine an election system that searches for and discounts voting blocs. You could invalidate those ballots, or at least reduce their voting power, with a reasonable degree of accuracy.

It's not quite that easy, because would not be able to distinguish, except perhaps by some fairly mean regressions, between ballots who are voting block, and ballots that overlap to a degree and who find candidates in common with the bloc.

Imagine the block consists of 10 candidates who are all acceptable, and all other candidates who unacceptable. Each voter can only select 5 candidates, and each individual voter in the bloc selects the 5 candidates who he or she feels most closely related.

Any algorithm that can detect votes that fall into the parameters of that block should also be expected to pickup a voter who selection of acceptable candidates overlaps with that of the block.

This is a common voting technique that can be used to hide the voting bloc, which trades rigidity and requires more participants, giving it a lower gaming power. However, the risk is of pulling voters and ballots who are not part of the bloc and diluting their vote.

Taken to its game theory maximum, an effective block would be a large slate of acceptable candidates, and a large slate of unacceptable candidates. The controller of the voting bloc divides half of the available votes to spread votes among the acceptable candidates, while focusing the remaining half on the candidates most likely to win if the block was not active. In this way they can neutralize no block voters by subjecting them to the disfavoring algorithm.

It is a philosophical question if the collaterally damaged votes and voters are worth throwing out with the bath water, but I think it disproves the idea of a theoretical way to punish bloc voters without looking at anything other the ballots themselves.

#46 ::: dh ::: (view all by) ::: April 07, 2015, 05:01 PM:

@44

Yeah, that's a non-starter. People who runs slates are already hardened against criticism and retribution. What are you going to say to Vox Day to make him regret submitting a nomination? There is nothing.

But that's not true for the countless fans who would prefer not have a struggling author turn his or her fans lose on the prior years electorate.

#47 ::: Louis Patterson ::: (view all by) ::: April 07, 2015, 05:04 PM:

The current nomination system is effectively Block Voting, which has known majoritarian problems [if you have N slots and N votes then it's pretty similar to N fptp elections and the N candidates selected will largely represent the largest group-of-preferences in the electorate]; what happened this year isn't an artifact or an edge case, but pretty much exactly what you'd expect out of block voting.

The size of the effect represents a larger-than-usual division in the electorate, but I'd guess that you'd see the same effect happening in previous years, that the hugo nominees somewhat underrepresented the diversity of SF.

I don't think the current setup is a good choice for assembling shortlists. It'll get you "chose from among these works that represent the preferences of the largest segment of the fandom", while what you want -- I think -- is "chose from these works that are each supported by a different group of people": SP reveals a real structural problem in the nomination process, even if it's worked OK-ish or decently before.

For the nomination process you want -- I think you want -- a system that if anything _under_represents the works liked by the larger blocks of fandom, in the name of "diversity", which suggests to me single non-transferable vote, which has exactly this problem. "problem".

[sntv has the advantage that the change required is very very simple: simply cut the number of nominations you can make in a category to "one". Any work that can't get enough first-preference nominations to make it to the ballot won't get enough first-preference votes to make it out of the first round of the main vote.]

Or so I think. Not a hugo voter, because I run five to fifteen years behind what's coming out of the presses.

#48 ::: P J Evans ::: (view all by) ::: April 07, 2015, 05:05 PM:

38
Or names and membership numbers. Still traceable, but far less personally hazardous.

For what it's worth, in 1984, Hugo ballots were stamped with a serial number as they came in, and the serial number was marked on a list of members. No way to tell whose ballot it was, without that list. (Site selection ballots that came in got the same serial number if they were with Hugo ballots, and a different one without.)

#49 ::: Martin Schafer ::: (view all by) ::: April 07, 2015, 05:08 PM:

One modification might be rather than eliminating a ballot once one member has won cut the rest of the votes in half (or potentially some other fraction) and once you had two winners down to a quarter etc. That would allow the nominator the satisfaction of being able to give at least some support to more than one work.

I realize with viewing this as a five winner election we have essentially been picking all 5 members at large and this is a proposal to create districts. Districts of taste rather than geography. Setting Y to more than one is essentially deciding to have some at large and some district.

#50 ::: dh ::: (view all by) ::: April 07, 2015, 05:11 PM:

PJ -- I think you are trying to solve a problem that is not imagined, one of fake ballots. Is that a concern all of the sudden? Being done online seems to have sewn up this issue completely unless there is a theory that there are voters who are not actually eligible.

#51 ::: Tom Whitmore ::: (view all by) ::: April 07, 2015, 05:14 PM:

Chris Gerrib @29: adding another round of voting is expensive in terms of time and people. And it's still possible to game it by slate voting -- so it's probably not a cost-effective way of dealing with the problem. It's got some theoretical advantages, but not enough to overcome its disadvantages.

#52 ::: Louis Patterson ::: (view all by) ::: April 07, 2015, 05:14 PM:

It's better than the more simple system above, because "excess" ballots above the quota needed to elect the candidate can get their votes transferred to another candidate.

I actually don't think you want this, for the nomination process in this particular case: two slots for the most popular style of SF is one slot less for a more-niche work.

#53 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 07, 2015, 05:16 PM:

Phil Palmer's idea may not be workable, but it's interesting to think about. In the Hugos as we know them, each of us casts a single nearly-anonymous vote that becomes part of an as-yet-unknown whole, as though we were singing a note without knowing yet what the combined word or chord will be. If every nomination comes with a list of nominators, our attention's drawn to persons and factions.

#54 ::: dh ::: (view all by) ::: April 07, 2015, 05:20 PM:

@53--

It's a really bad idea. It raises the bar to nomination to being a fan, having $50, and having the patience and ability to stand up to sore losers, the mob, and future deal making.

But if it should be considered, maybe it can be done retroactively.

#55 ::: Fragano Ledgister ::: (view all by) ::: April 07, 2015, 05:22 PM:

If there is going to be only one winner, then a preference ranking system is Alternative Vote, not Single Transferable Vote. Single Transferable Vote will produce multiple winners in a multi-member constituency. Alternative Vote will produce a single winner with a majority of preferences in a single-member constituency.

The objective, in a ballot for a Best X contest, is to produce a single winner aggregating a majority of preferences. This means, when preferences are being ranked, that this is an AV rather than an STV system.

#56 ::: Jeff R. ::: (view all by) ::: April 07, 2015, 05:23 PM:

I'm not seeing the value in deanonymizing the votes here. In fact, it looks like all downside to me, enabling petty thuggery and harassment, in exchange for, what? Making secret slates visible after the fact?

(I don't see any way to reveal this information between the nomination and actual vote without eliminating the tradition of keeping hidden the popularity order of the nominees, which is probably something that should be kept.)

#57 ::: Fragano Ledgister ::: (view all by) ::: April 07, 2015, 05:28 PM:

TNH #9: All voting systems can be gamed. Some voting systems are less susceptible than others.

The best method to limit gaming a nomination vote for the Hugos is the limited ballot (i.e., since there can be five nominees per category, limit voters to no more than n-x/n). A cabal like the Sock Puppets, given its limited numbers, is going to have a harder time pushing a single slate of candidates.

Bruce Scneier is correct that it has constituted itself, effectively, as a political party. This does not mean that the response to it has to be the formation of a rival party (the historic norm). There have been cases (the Cayman Islands is one such) where political parties emerged and disappeared because the electorate did not find that they satisfied their actual needs.

#58 ::: P J Evans ::: (view all by) ::: April 07, 2015, 05:31 PM:

50
Not fake ballots - multiple votes by one person. And just, generally, tracking that member x had voted, because the question does sometimes arise ('Hey, did I send in my ballot?').
The ballots that year were not particularly susceptible to faking, for technical reasons.

#59 ::: Cheradenine ::: (view all by) ::: April 07, 2015, 05:34 PM:

rcade@#41: "Here's a recommendation to expand the number of nominees in a category dynamically based on nominating patterns:

http://drplokta.livejournal.com/166650.html"

It's an interesting proposal that I think winds up essentially equivalent to John Mark Overblum's suggestion in #39 of setting a threshold percentage of the total nominations that guarantees a work appears on the final ballot. I looked at what the threshold numbers would be for this year (slate) and 2013 (no slate) in a few categories.

For Best Novel, the threshold to make the ballot would be 7.88% in 2013, 8.27% this year.
For Best Related Work, the threshold would be 7.62% in 2013, 8.46% this year. The slate's presence does affect the thresholds, but not enough that it seems worth the extra complexity compared to simply setting the threshold around 8% (or wherever seems reasonable).

I think this proposal would be fairly robust against a single slate, but in a multiple-slate situation, it may exacerbate the problem by guaranteeing ballot presence to *every* slate that can exceed the threshold - if 5 slates were sufficiently popular, we'd have a ballot with at least 25 works, and a strong incentive for people to vote one of those slates instead of their own ballot. So I'm leaning toward the conclusion that it would fix the exact situation that happened this year, but not be very robust against future permutations.

#60 ::: Q. Pheevr ::: (view all by) ::: April 07, 2015, 05:49 PM:

I wasn't sure if the nomination system I imagined in the other thread was relevant here (since it's not really about the mathematics of voting systems), but it occurs to me that it might have an effect similar to the intent behind Phil Palmer's proposal for transparency, but hopefully without the same potential for exposing nominators to harassment. So I'll repeat it here, if that's okay:

Here's a hypothetical nomination system: Every nomination must be accompanied by a one-sentence rationale. The sentences would not be evaluated for sense, or coherence, or grammaticality, or anything else but uniqueness (which could be done by computer). Each nominee gets a score equal to the number of unique sentences submitted in its favour. This is obviously a very low bar for anyone trying to muster an army of sock puppets, in this age of automated text generation, but suppose further that the sentences are made public once nominations have closed. Any nominee whose supporting sentences read like lorem ipsum, or are hopelessly generic, or primarily express a political gudge, would be revealed to voters and to the world at large as having attracted a particular kind of support. I don't know whether this would actually do any good, but it struck me as a potentially interesting idea.

#61 ::: ULTRAGOTHA ::: (view all by) ::: April 07, 2015, 05:52 PM:

Bruce Schneier @25
I want to caution everyone against jumping to solutions without lots -- say, months -- of thought and analysis. Every system replaces old problems with new problems, and it can take a while before we'll understand what the new problems are and if they're worth the change.

Under the rules, any change voted on this year must be voted on again next year and can't take effect until at least 2017.

That gives us a LOT of time to think about changes, and to see what will happen next year before a potential change is finalized. Months until August to come up with a preliminary proposal, and then all of next year's cycle to refine it, or drop it.

Thanks to all the ML moderators and Bruce for this discussion. It's a positive and thoughtful response.

#62 ::: albatross ::: (view all by) ::: April 07, 2015, 05:56 PM:

Fragano:

Is the difference between AV and what's being described in #1 (once you account for needing to get multiple nominees) just that you guarantee that each nominee got a majority rather than a plurality (after removing the least popular choices enough times to get a majority)?

#63 ::: Q. Pheevr ::: (view all by) ::: April 07, 2015, 05:58 PM:

Oh, and an afterthought—I don't know that my proposal would do anything to deter a group interested in putting forward a slate for the sake of breaking things and annoying people, and perhaps it's redundant merely to expose such a plot (since it would be hard to recruit enough slate supporters in secrecy), but perhaps it would provide an incentive for authors/artists/editors/etc. to decline nominations of that sort.

#64 ::: nathanbp ::: (view all by) ::: April 07, 2015, 06:02 PM:

Fragano Ledgister, #55: The Hugos already use IRV for selecting the winner. For the nominees, we're picking 5 "winners" (nominees), so it'd be STV or something similar. Or maybe I'm not understanding something?

Louis Patterson, #47: Cutting the nominations to one per category per ballot (1/5, in the terminology of 4/6) would probably solve the slate problem, but I feel like it would significantly reduce the chance for niche works to get on the ballot because it encourages people to vote for the candidates that are perceived to be popular.

#65 ::: Christopher Hensley ::: (view all by) ::: April 07, 2015, 06:11 PM:

Releasing ballots with names doesn't actually increase the transparency of the process in any meaningful way. The anonomized ballots are sufficient to show deliberate voting pattern. Either one is a matter of too little, too late.

#66 ::: James ::: (view all by) ::: April 07, 2015, 06:13 PM:

The real edge case for the system in #1 is one where there are, say, two works which are generally acknowledged masterpieces in the year. Then everybody, to a good approximation, votes for both; one gets included, having an edge of a few votes, and #2 basically vanishes. For novels, where the cost and effort barrier is higher, this probably still comes out OK, as enough people have read only #2 for it to be still the next one selected, but for movies it might be a bigger problem.

BTW, the legal maxim I learned was that "hard cases make bad law", and it applies to situations where there is a temptation to bend the rules to provide a remedy for a circumstantially needy or deserving party. (This originally comes from equity but is often most applicable these days to tort law.) This does not correspond to the current situation...

#67 ::: Eemeli Aro ::: (view all by) ::: April 07, 2015, 06:30 PM:

For one alternative to changing the nominating system as such, we could make it more open: the rankings of the candidates could be published during the nomination phase, allowing nominators to observe the current state of the process, and to update their own choices accordingly.

The pool of nominators is fixed on January 31; after that, the most democratic interest is in motivating the largest possible fraction of those eligible to participate. Right now, the nomination process is closed and provides no information until its end. If, instead, the state of the election could be (partially) observed, it would act as a strong motivator towards participation, as well as shining a light on the actions of any organised minority group.

Sure, this would lead to nominators acting in a different way than they currently act, and nominating works that they might not have nominated otherwise. On the other hand, it would also promote participation and discussion about the works in a way that is not currently done. In a way, this would embrace the nomination process as a popularity contest rather than shying away from it.

To be clear, I'm not talking of the full state being revealed; rather, something like a weekly listing of the top fifteen condidates, in order but without the running nomination counts; possibly with an added minimum of say, five nominations to get listed.

One benefit of this proposal over others is that it's technically allowed within the current rules, so it could be implemented already next year. It would, however, be a more radical change than e.g. a switch to a single tranferable vote system, which is the option I like best of the ones I've heard.

#68 ::: nathanbp ::: (view all by) ::: April 07, 2015, 06:43 PM:

Eemeli Aro, #67: I'm not sure how I feel about that in theory, but in practice anyone trying to game the system would just wait until the last minute to vote, so I don't think there's much point in it. Or they would put in one set and change their nominations at the last minute, if that's allowed (I haven't nominated before, so I don't know).

#69 ::: albatross ::: (view all by) ::: April 07, 2015, 06:44 PM:

#67:

My intuition is that that would be more beneficial to organized blocs than disorganized ones. "Hey, everyone, Alice is pretty much a lock to be on the ballot now, so everyone start putting Bob at the top of their nominations."

#66:

I don't have any legal training at all, but my impression was always that those hard cases could involve extremely unsympathetic people as well. (Thus, free speech cases almost always involve really offensive people speaking; few people want to use the law to shut down basially inoffensive points of view.)

#70 ::: Mark Z. ::: (view all by) ::: April 07, 2015, 06:59 PM:

TNH #9: "Random systems" do include systems that give the voters a meaningful choice.

For example, the simple random ballot method defeats tactical voting. The best choice is always to vote for the candidate you like best.

If there are multiple seats to fill, like the Hugo nominations, then you use a ranked random ballot, where you rank your first N choices, draw N ballots at random, and for each of them, select the first candidate on the list who's not already selected. This is also immune to tactical voting.

For example, if someone's pushing a slate of 5 candidates, and can somehow get 20% of the voters to vote for exactly those 5 candidates (in any order), then there's a 67% chance that at least one of them will be chosen (most likely only one). But the chance of getting the entire slate elected is 0.032%. Note that the math doesn't depend on how anyone else votes--whether the other voters are organized blocs or not. If you have 20% of the voters you're probably getting about 20% of the seats.

It has its problems--most importantly, the random selection can only be done once, so everyone has to be committed to accept the result, and any inspection of ballots has to be done before that. This might not work in an environment like the Hugos where you want to announce a winner shortly after closing the vote. But short of actual fraud, there's no gaming the system.

#71 ::: rcade ::: (view all by) ::: April 07, 2015, 07:00 PM:

Protecting the Hugos from manipulation is hard. Perhaps we should be discussing how to game the Sad Puppies instead.

#72 ::: Hmm ::: (view all by) ::: April 07, 2015, 07:09 PM:

In the method described in #1, are there any less arbitrary options for resolving ties than flipping coins?

I understand that if you're trying to reduce the influence of a bloc, you don't want to just include everything that ties (because if all the bloc voters voted for the same things, and no one else did, then the tie is because of a bloc and you actually want to eliminate some of the bloc's candidates), but flipping a coin to determine the winner (especially in a case where there's another round still to go) does not seem very satisfactory to me.

You could use previously eliminated ballots to try to resolve a tie for everything but the first winner (though there may be issues with doing this?), but if that doesn't help, or if the first winner is a tie...

I suspect (haven't done any math, and my math is probably not up to it) that you could come up with a situation where the first winner is an actual, natural tie -- say A and B were tied and B was selected arbitrarily -- and then eliminating all ballots that contain B drops A below the threshold for ever winning, even if there's not a super-strong bloc level correlation between A and B. (This is probably more likely in a scenario where there aren't any blocs, even? Say that there are a lot of candidates grouped toward the top, A and B happened to tie at the very top, and eliminating all ballots with B just happened to hit A a little bit harder than the other candidates near the top?)

I guess maybe this is just the price you pay for this system that reduces the impact of blocs? It just seems extra strange and potentially unfair when it comes to ties (unfair to the works under consideration, that is, maybe not unfair to the voter, who does get to vote for some number of things, though it will be hard for the voter to predict how their vote is going to work out).

Although it makes me wonder, if you assume that there aren't any blocs, just how arbitrary does/could this system end up being? For example, if everything is distributed totally randomly (I know this won't happen in the real world, but I'm just curious for the sake of analysis), and you look at a list ranked by number of votes and compare it to the five winners, what's the likelihood that the thing ranked 20th in the original list will end up in the group of winners? What about the thing ranked 10th?

I guess basically, I would be curious about how this system performs in the absence of blocs? Is there any good way to measure that?

And this is something that would change depending on the parameters, yes? So does it end up being a trade-off between bloc-reducing strength and weird effects in the absence of a bloc?

#73 ::: Jo Walton ::: (view all by) ::: April 07, 2015, 07:12 PM:

Eemeli -- while there are things I hate about your suggestion, it is still the best suggestion I have seen anyone make so far.

#74 ::: John Mark Ockerbloom ::: (view all by) ::: April 07, 2015, 07:16 PM:

rcade@41: Yes, Dr Plotka's proposal at http://drplokta.livejournal.com/166650.html was the one I'd seen before, but couldn't place in my previous comment. I'm not sure his formula is the ideal one to use, but the general idea of "list *any* qualifying title that has more nominations than some pre-determined threshold" seems like a good one to me.

Cheradenine@59 raises the objection that this would result in unworkable ballots: "if 5 slates were sufficiently popular, we'd have a ballot with at least 25 works". This is a reasonable concern, but I don't think it's a show-stopper.

First of all, it's not at all clear to me that you'd *get* 5 slates with significant following. Particularly if it does turn out that voters penalize slate candidates in this year's award IRV election, anyone who's serious about actually *winning* a Hugo, as opposed to just cadging a nomination or counting coup against one's perceived enemies, would have a disincentive to participate in a slate campaign.

Second, while readers may need to take a different approach sifting through 20 candidates than sifting through 5, we do this sort of thing all the time, when looking for things to read, people to hire, manuscripts to accept, and other choices. When readers have access to things like the Hugo packet, information on slate attempts, and other information and discussions on the Internet, I suspect most could do a pretty quick triage on which entries in a 20-candidate field (should one occur) are worth their serious attention, and which are not. (And 20 or 25 candidates is still a much narrower pool to consider than all of the works published in a year in a particular SF category.)

#75 ::: Jeff R. ::: (view all by) ::: April 07, 2015, 07:29 PM:

@71: Game as in crash their nomination process? To the end of putting good books into it or extremely awful ones?

Making it more difficult for them to gain the approval of next years equivalents to Butcher and Weber is always a thing. If we take them at their word they'll at least try to get The Dark Forest on their list, that might be stoppable. But there's a Card and a Stirling coming next year, so there's no shortage of NYT-bestseller-list-contender-SF-by-very-rightwing-authors on the horizon...

#76 ::: Ctein ::: (view all by) ::: April 07, 2015, 07:46 PM:

Dear Phil (38) and Teresa (53),

Aughhh! No, no, no!

And furthermore, no!

Do not put down the can opener and step slowly away from the worms. Run away as fast as you can.

Think real world, for a mo', 'cause that's the playpen, not fannish abstraction.

The best way to kill or corrupt a democratic process is to eliminate the secret ballot. You all know this! It is a sure road to disaster, if your desire is an uncoerced electorate.

On the one side, I do not see what it possibly gains us, except it allows us to identify those Bad People Who Do Not Think Like We Do. Such a fine thing for a democracy.

"Oh my," he said in his best Pogo-ish voice, "We have met the enemy and he is us."

On the other side, if I were a GGer, I'd be thinking "Cool, I bet I can track down at least 10% of those SJWs (meaning everyone who ain't a GG or SP) and then I'll DOX'em."

(Bruce has written at length about how much easier it is to build up a detailed profile of someone than most people realize. Semi-anonymous? Not really.)

So, ya wanna play Russian Roulette with your private life? Ya feeling lucky, punk?

Worst.
Idea.
Ever.

pax / Ctein

#77 ::: Ctein ::: (view all by) ::: April 07, 2015, 07:48 PM:

P.S. Oh yeah, I do understand that the Hugo ballots may not be internal-to-the-system anonymous, but they are externally.

#78 ::: John A Arkansawyer ::: (view all by) ::: April 07, 2015, 07:56 PM:

Mark Z. @ 70: My chaotic side LOVES random ballot, but my time-binding side asks, won't there be a tendency for people over time to make more and more one-off votes? Will the indeterminate nature of the process make people more inclined to vote for fringe, parody, or protest candidates? And what happens when Pat Paulsen becomes the president?

#79 ::: Mike D ::: (view all by) ::: April 07, 2015, 08:15 PM:

One thing that does affect how voting works is the size of the pool of voters. Some things that work well when you have millions of people voting on a thing don't work nearly as well when scaled down, and vice versa.

For example, at smaller scales it's pretty reasonable to make and enforce a rule that says "look, we expect you to behave honorably and vote your actual preferences rather than forming a bloc. As voting officials, if we think there's shenanigans going on, we reserve the right to ban the people involved from future participation". That kind of thing can't fly at national-office scales, because it's a lot of power and eventually you'll get an administrator who abuses it. But for things like eg. Class President, it's perfectly reasonable to let the voting officials use their discretion to identify people trying to game the system.

I mean, basically what we're talking about here is pattern recognition, I think? There is a problem that has been identified, and the goal is to spot it happening in the future and reduce its impact. It may be possible to set up the voting system to detect it algorithmically by design, but if the people handling the votes can be trusted (and that's an IF) then frankly it's a lot simpler to just empower them to act as moderators and block abuse when they see it.

#80 ::: Cheradenine ::: (view all by) ::: April 07, 2015, 08:19 PM:

Mark Z. @#70: I realized I mentioned it first, but I don't think random ballot is viable here, unless we want to rebrand the Hugos as "the top work as recognized by WorldCon and the fickle, fickle hand of fate." Getting people to accept a voting system with a nondeterministic outcome would be a hard sell, and it inevitably would eventually produce a really bad set of nominations.

John Mark Ockerbloom @74: First of all, so sorry for getting your name wrong in my last response! In general, I think we want to minimize the assumptions we make about future voter behavior in examining options for changing the voting system. The current system worked more or less fine until people behaved in a way that wasn't anticipated by its designers. Before we change to anything else, we want to look at how it will behave in edge cases, especially when we have at least some people showing active interest in provoking those edge cases. So I agree that it's unlikely that, say, an 8% threshold would lead to the maximum number of nominees (60 if I'm doing my math right, assuming we stuck to 5 max nominations per voter) - but the fact that it COULD lead to close to 60 items on the ballot is worth considering.

#81 ::: Josh Berkus ::: (view all by) ::: April 07, 2015, 08:21 PM:

All:

I'm interested that nobody seems to be addressing Bruce's points (1) and (2). I'd like to make a suggestion specifically around Bruce's recommendation (2), namely: change the electorate.

It seems to me that online-only sustaining memberships (OOSM) -- meaning someone who has never attended WorldCon -- open Hugo voting systems up to gaming by all sorts of individuals; per my comment in a prior thread, OOSM are even more vulnerable to a *really* popular author with good social networking skills and no inhibitions than they are to the SPs. Further, OOSM make political parties more powerful because online-only members are far less immersed in WorldCon and thus more interested in determining how to vote in 15 minutes or less.

So one thing WorldCon could do is restrict sustaining memberships to people who have attended a WorldCon in the last X years (2, 3 or 5, probably).

And to forestall a pro-forma objection: I have never attended a WorldCon, and therefore am eliminating my own eligibility to vote if this carries. I'm OK with that; I didn't vote on Hugos in the past, and was mostly good with what won.

#82 ::: P J Evans ::: (view all by) ::: April 07, 2015, 08:24 PM:

77
As far as internal-to-the-system-anonymous: I think that's the ideal.
There's really no reason why anyone outside the process has to know that member n has returned a Hugo ballot. (Sign the envelopes, not the ballots; membership transfers are just a wrinkle, although annoying if membership hasn't passed the changes through to the Hugo group.)
There's also no reason why more than one person (or two, because backup) needs to know who won (because the plaques have to be made, and that means someone has to get the information to the engraver).

#83 ::: Ctein ::: (view all by) ::: April 07, 2015, 08:33 PM:

Dear Josh (81),

I addressed this at length on Sclazi's blog, so I won't repeat myself, but the short form version is that any scheme that substantially raises the bar to voting is based on two assumptions.

1) The Loyal Opposition is less financially able to bear the burden than Us Righteous Ones.

2) The LO is less ideologically committed to voting than URO. (Barriers work most strongly to weed out less committed voters.)

The first is entirely unproven and I'd not want to place any bets on the second. The LO is pretty fanatical.

pax / Ctein

#84 ::: TexAnne ::: (view all by) ::: April 07, 2015, 08:35 PM:

Josh, 81: That's a terrible idea. You would be limiting the voting pool to people who had a ton of money. Your lack of interest is fine for you, but don't make my choices for me. (Wait. Where have I heard that before?)

#85 ::: Mark Z. ::: (view all by) ::: April 07, 2015, 08:39 PM:

John Arkansawyer #78: What happens when Andrew Jackson becomes the president?

#86 ::: Nicole Poweleit ::: (view all by) ::: April 07, 2015, 09:00 PM:

I have a suggestion for identifying slate voting based on a specific application of vector clustering with which I'm familiar due to my research.

The process is called "class averaging" and it's used in cryo electron microscopy of biological samples. Data collected from this method is very noise. In order to boost the signal to noise ratio, class averaging is done. This is a process by which images from within the data set are compared to each other and based on how alike they are assigned to class with other like images.

To adapt this to the Hugo nomination process I would suggest a random-reference class average. Basically, a random number of ballots would be selected as references from the whole. Say there are 200 ballots and 5% are selected as references providing 11 classes, the 10 randomly selected ballots and an 11th empty ballot which serves as a repository for ballots which do not match any of the others. The remaining 190 ballots would then be compared to the 11 classes and each assigned to a class. After the class averaging is complete, the classes' statistics can be examined and outliers determined based on how alike the ballots within a single class are to each other (i.e. select for classes with the strongest signal and eliminate them.)

The plus sides: 1) This would differentiate between true slates and "The Avengers were very popular" example. 2) The statistical results of the class averaging could be posted without damaging ballot anonymity. 3) There are many open source programs which perform class averaging for cryo electron microscopy data, I'm most familiar with EMAN2 and Relion, which could be potential sources for code. 4) Conceptually, this is fairly easy to explain: We selected 10 random ballots and compared the remaining 190 ballots to them. We found that 20 ballots shared 90% of their nominees; therefore meeting the threshold to be considered a slate and were invalidated.

Downsides: 1) The math is actually quite complicated and would be difficult to explain, making the process something of a black box. 2) While I suspect this would be sensitive enough to identify a slate such as last years SP2, it's difficult to say without testing. 3) Test data would likely be needed to validate the approach.

Point 3 actually brings me to a question. Is it possible to get the Hugo nomination data from previous WonderCons? I'm a WonderCon newbie and don't know how these things work. Last year's data set and data sets from a few more "normal" years would go a long way towards validating any new voting method (if such data is available.)

If my suggestion is considered too unwieldy, (I totally understand if it is) I'd like to put my support behind an open election process as described by #67 Eemeli or SVT. Neither seems perfect to me, but I believe they would reduce the effect of the current issues.

#87 ::: Tammy Coxen ::: (view all by) ::: April 07, 2015, 09:07 PM:

I think that an open voting system as described by Eemeli in #67 would cause there to be a huge amount of overt and ongoing campaigning. "Slaughterhouse 7 has dropped in the rankings - quick everyone, go vote and get your cats to vote too!" I don't think that's something we want to encourage.

#88 ::: heresiarch ::: (view all by) ::: April 07, 2015, 09:14 PM:

Tim Bartik @ 15: "This may be viewed as a feature and not a bug if one wants to encourage diverse styles of SFF on the final ballot."

If anything can convince me that changing the voting system is a good idea, that might be it: having a consistently more diverse group of Hugo nominees has everything to recommend it. This voting system might actually be more valuable to me for that property than for its slate-defeating properties. (I still think that Bruce's options 1 and especially 2 are preferable to any change to the system, but tempts me.)

Bruce Schneier @ 19: "You would also end up with a voting system that is unexplainable to a normal person -- which I think violates an important rule of a voting system."

This is worth remembering. An essential characteristic of any functional voting system is that voters think it's comprehensible and fair. As it happens, that is another reason I really like this system: I feel it can be explained very simply. "You get to nominate five choices, but once one of your choices makes it on the ballot then the rest are discarded." It even, I think, seems fair: isn't getting one of your picks enough?

Jame @ 66: "The real edge case for the system in #1 is one where there are, say, two works which are generally acknowledged masterpieces in the year. Then everybody, to a good approximation, votes for both; one gets included, having an edge of a few votes, and #2 basically vanishes."

As soon as #1's ballots are eliminated, a bunch of down-ballot votes not only for #2, but for #3, #4, etc. are eliminated as well. If everything is randomly distributed, then #2 should still be the biggest fish in the (much) smaller pool. The only time when this isn't the case is if #2 is disproportionately "in the shadow" of #1, i.e. #1 voters are unusually likely to vote for #2 compared with voters who did not vote for #1. That's a sign that however excellent #1 and #2 are, they appeal to a similar demographic--in other words, they are running afoul of Rivest's system's preference for a diverse ballot.

#89 ::: Tom Whitmore ::: (view all by) ::: April 07, 2015, 09:19 PM:

P J Evans @82: That also requires the engraver to know the names of the winners, so it's at least 3....

But seriously -- when I chaired the Worldcon, I had no idea who had won the Hugo until the announcement happened, so it can be kept to a "need to know" basis. We actually do need a fairly large number of people who know at least shortly before: there's a handout given out right after the Awards Ceremony which includes all the voting breakdowns, and *somebody* has to print that and have it ready for distribution. Not to mention having the people ready to distribute it. That's a few hours before the announcement, but it is noticeable (especially in Internet time, which is the opposite of geological time).

The number should be kept small, but it can't approach the minimum you suggest, on a practical level.

#90 ::: P J Evans ::: (view all by) ::: April 07, 2015, 09:19 PM:

88
You can see that kind of similar-demographic pattern if you look at the voting results for the final ballots. (I believe that Locus often points that out in their report.)

#91 ::: P J Evans ::: (view all by) ::: April 07, 2015, 09:26 PM:

89
Tom, you only need one person to do most of that. Most of what you need people for is labor: handling the ballots and the trophies. Or the slides for the ceremony.

(I recall there were three, maybe four, of us dealing with the Hugos in 1984. It was very much need-to-know when it got to the actual results, and I wasn't need-to-know.)

#92 ::: John A Arkansawyer ::: (view all by) ::: April 07, 2015, 09:30 PM:

Mark Z. @ 85: Second time today genocide has been brought up in an unexpected context. A fair point.

And now that I give it more thought:

Random ballot has a lot to recommend itself to the nomination process. It's non-deterministic, but it should give good results on average, and the math is relatively simple to understand and, more to the point, to explain. It's easier to explain and understand than the current final voting system. And it matches a current feature: The results will never satisfy everybody.

Let me amplify that final point. How often does everyone agree that all their obvious, slam-dunk, no-brainer nominees are on the ballot? I'll say it approaches never. This is no worse than that.

#93 ::: Rene ::: (view all by) ::: April 07, 2015, 09:38 PM:

Nicole's question (#88) to her third point identifies an interesting issue. The Sad Puppies claim that prior to the Sad Puppies campaigns, there were "whisper campaigns" that encouraged block voting and that the Sad Puppy campaign was a direct reaction to the "whisper campaigns." If the nominating data is available and if Bruce Schneier is correct that block voting can be determined by data analysis, then the prior data should be analyzed. If the data does not confirm pre-Sad Puppies block voting, then this will take a lot of the wind out of the Sad Puppies campaign. If, on the other hand, the data confirm the presence of pre-Sad Puppies voting blocks, then the Sad Puppies campaign becomes much more understandable.

#94 ::: rcade ::: (view all by) ::: April 07, 2015, 09:43 PM:

Tammy Coxen @ 87: Tammy Coxen I think that an open voting system as described by Eemeli in #67 would cause there to be a huge amount of overt and ongoing campaigning.

True, but that could be good marketing for the Hugo awards and for the books competing for nominations. During the whole process in which nominations are accepted, people would be talking about the works and how they are doing in the nominations vote.

#95 ::: Randolph ::: (view all by) ::: April 07, 2015, 09:46 PM:

Some modest system reforms that reduce the impact of partisan voting on the Hugos seem worth the trouble, if we can work out good ones. It will at least reduce the impact of the less-committed partisans.

I don't think swinging the final ballot is likely; that would require hundreds, perhaps thousands of people. It would have to be paid for and, really, there aren't very many crazy people willing to spend money for such a minor win. Perhaps legal action against people who openly fund block elections would be appropriate.

On the other hand, I doubt this is a temporary problem. Vox Day will likely stick it out until he is jailed like his father. The hardcore GGs also probably aren't going away without legal pressure; stalkers (if you study them) don't quit unless pressured. Sometimes handcuffs and steel bars are the necessary amount of pressure.

#96 ::: Doctor Science ::: (view all by) ::: April 07, 2015, 09:46 PM:

I'm a big fan of final ballots that adjust to fit the year. Elsewhere, I saw readingsff suggest everything that gets more than 5%, and then there's DrPlotka's more complex suggestion.

This would make the Hugos a little more like the Locus Awards, in that the final vote is based on a longer ballot that's more like a curated reading list.

#97 ::: Cheradenine ::: (view all by) ::: April 07, 2015, 09:47 PM:

In general, the more information people have about how other people are voting, the more effective tactical voting is. So while it may have other merits, publicizing the status of the nomination counts is likely to encourage tactical voting.

#98 ::: Ravi ::: (view all by) ::: April 07, 2015, 09:48 PM:

#40: Yes, I'm sure that there's plenty of willingness to write some code. But, in my opinion, writing and deploying the code changes the optics of the matter.

This year, when pushed, the Sad Puppies often claim they weren't really voting as a bloc. They were just listing things they liked. People weren't slavishly voting the slate. They had no idea they'd be this successful (trying to imply there wasn't a calculated strategy at work, they were just genuinely popular) and so on. They could game the system but still maintain some superficial deniability.

If you study the details, this isn't credible, but you actually have to get into those details to understand that. If, alternatively, the Sad Puppies had to deploy a web page that tossed random numbers to maximize the power of their bloc, it would be easy to point to that to say something fishy was going on. For that matter, the less favorable optics might also make it harder for some of the Sad Puppies voters to fool themselves about what they are actually doing.

More generally, instead of just focusing on how easy or hard it is to game a particular voting system, I think it is important to consider how deniable that gaming is. A voting system that can be easily gamed, but where that gaming is undeniable might work better in practice than a system that is harder to game, but where that gaming is easy to deny.

#99 ::: Randolph ::: (view all by) ::: April 07, 2015, 10:09 PM:

I think the bottom line of my comments @95 is that, while voting system changes can help some, they will work best in conjunction with a political and possibly a legal solution.

That is not a conclusion I come to happily.

I wonder what combination of political and voting system changes might work.

#100 ::: Greg M. ::: (view all by) ::: April 07, 2015, 10:27 PM:

I'm a newcomer to the intricacies of the Hugos and voting blocks, though I've been following this keenly, and I don't know if this thought-path is workable, but it seems like there were multiple 'exploits' that were gamed here, one of which was that nominations were determined by pure popular vote in a world in which hundreds of items were eligible. The award itself should be determined by popular vote, and every WorldCon supporting member should get to nominate, but do the nominations have to be determined entirely by popular vote? The Academy Awards--and yes, as an Errol Morris fan and Hoop Dreams fan, this may be a *very* poor analogy--but they do have a volunteer committee that determines the documentary shortlist, made up entirely of documentarians.

Animation would be better--it's animators alone who determine the short list of animated short films.

I suspect this may not be workable, but I'm throwing it out there, because I'd really like to see the Hugos continue to be an award of merit and not be turned into some frakking toy a couple of very angry people want to smash.

#101 ::: Randolph ::: (view all by) ::: April 07, 2015, 10:34 PM:

Greg@100: a juried award is of course possible, but who would trust the jury? The Nebula Award is entirely given by members of he Science Fiction Writers of America; that's a different thing.

#102 ::: Phil Palmer ::: (view all by) ::: April 07, 2015, 10:45 PM:

Ctein, #76

I am not suggesting eliminating the secret ballot. The ballot isn't broken; don't fix it.

What's broken is the nominations. Should people really be putting items onto the Hugo ballot if they aren't prepared to say who they are?

Of course there are people who are nutty enough to try to - and who do - send SWAT teams around to someone's house and they may conceivably do this just because someone has said that they like a book enough to nominate it. But this is the fault of your nutters and your out-of-control SWAT teams and to be dictated to by this "real world" consideration is close to saying that the US should stop hosting worldcons.

#103 ::: JonW ::: (view all by) ::: April 07, 2015, 10:45 PM:

Here’s another voice in favor of some system like Mike Scott’s (drplotka) that adds names to the ballot. When we think about voting system fixes, if our plan is “let’s figure out a new voting scheme under which, instead of the five names that get on the ballot under the current system, we get a different set of five names,” then all of Bruce’s cautions become really salient. We’re implementing a rules change that causes certain people no longer to be nominated, and that’s a weighty thing. The joy of a system like Mike’s is that it ameliorates the problem caused by the puppies without eliminating anybody from the ballot. What we want from a Hugo nominating system, after all, is that it generate a list that most Hugo voters will think includes most of the works they think most worthy. If it also includes bad works, that’s not a problem; an IRV majority won’t vote for them. (And if a majority will vote for them, we have a problem no voting system can solve.) So a pretty good solution – I feel confident that Mike’s isn’t the best possible implementation, but it’s a start – is one that reacts to slate voting by keeping the slate nominees on the ballot, but also adding other names, of works that would have made it but for the slate voting. Yes, such an approach could lead to long lists of nominations, and that’s not ideal, but it's much less not-ideal than the alternatives. I’d love to hear from the voting-system experts as to how Mike’s approach (nominate every work in a category that got votes > 10% (total noms – noms of first place finisher)) could be improved.

#104 ::: JonW ::: (view all by) ::: April 07, 2015, 10:48 PM:

Sorry, Mike -- that's drplokta, not drplotka

#105 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 07, 2015, 10:54 PM:

Hi, everyone. I'm about to shut down for the night. We'll see you in the morning.

#106 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:36 AM:

Wow, you people are chatty.

I had an hour to go through everything since yesterday afternoon, and I will now attempt to post the comments in this otherwise closed thread.

#107 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:37 AM:

dh @21:

"There is some evidence to suggest that voters following either of the two slates had difference levels of discipline and differing levels of adherence to the slate."

While that certainly may be true, the statistical correlation between slate members is still going to be significantly different from the statistical correlation between others.

albatross @40:

"One question I have that's specific to the Hugo nominations: How correlated are they, normally?"

Dave McCarty talks about this.

#108 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:37 AM:

Cheradenine @24:

"I was referring to the Single Transferable Vote system (linked in #2) when I mentioned rankings. The proposal from #1 doesn't require it, but the more I think about it, the more I'm worried that this proposal creates a tactical incentive not to vote for anything that seems likely to be extremely popular, because it will take out all your other nominations. In 2013, about 48% of ballots in Dramatic Presentation - Long Form listed 'The Avengers,' which means that all other nominations from that half of the pool would be disregarded. So if you're pretty sure plenty of other people will list 'The Avengers,' there's strong incentive not to list it. If enough people follow this line of thought, of course, it might not make the ballot at all. It's why the 'transferable' part of 'Single Transferable Vote' is so important."

First, I agree with you. Right now, I think that some sort of Single Transferable Vote system (see here) is the best answer. I would want a variant that doesn't require ranking the nominees. We'd also have to decide how many nominations a ballot has to successfully make before it is removed -- my guess is 2-3 -- but that seems to both limit the power of voting blocs while not adding any new problems.

But I don't think the system in #1 has that strategic incentive for normal voters. Bloc voters have a different goal: to get their slate on the final ballot. Normal voters are just voting for what they like, and are presumably going to vote for their favorate in the final election. So not including your favorite in the nominating election doesn't make sense, even if you think it is wildly popular. Still, STV removes the problem and hence is better.

nathanbp #34:

"I'm not sure what the proposed system in #1 has to offer over STV except that it's slightly simpler to understand."

I agree with you. There are no advantages of #1 over STV.

#109 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:38 AM:

Cheradenine @24:

"Sorry to go on after mostly lurking for years. I teach voting systems and it's a subject of intense interest to me, and I'm really angry at the Puppies, and the combination seems to be making me voluble."

Thank you for being here, then. It's good to have someone with actual expertise. I feel like I'm mostly pretending.

#110 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:38 AM:

albatross @28:

"The simple version of bloc voting could easily be detected, but not the slightly more complicated version where I distribute a program to the people who want to vote my slate, which randomizes them so that we don't get above whatever threshhold there is for detecting it."

Yes. Given better coordination, all sorts of things are possible. In general, though, we can expect only the most simplest of coordination.

"An interesting problem is, given something like Rivest's proposal Bruce described in #1, what's my optimal strategy? If I knew about how many people were voting (nominating) my slate, and I also had a pretty good idea how many total nominations there were going to be, I could put the required number of nominations to get Alice nominated in (each nomination with a randomized set of other choices), then the required number to get Bob nominated, etc. My intuition is that this never gets as powerful as the slate voting that happened this year, but I'm not sure of that."

My intuition is the same as yours. Strategic voting -- that is, voting with knowledge of how others will vote -- is more powerful than straight preferential voting, but strategic voting by voting blocs is more powerful still.

#111 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:39 AM:

Chris Gerrib @29:

"So, the other way to concentrate the 90% is to have three rounds of voting. Round one creates a long list - say 15 names. Round two creates a short list of 5, and round 3 narrows it to one."

The part of the strategy that works here is making the number of candidates a person can nominate less than the number of winners. Even without the middle round of voting, a slate of 15 is harder to game than a slate of 5 -- assuming we keep the number of nominations a person can make constant at 5.

I hesitate to add another round of voting without some serious evidence that it is significantly better. Otherwise it's just more complicated, and one important solution here is to get more non-bloc voters.

John Mark Ockerblom #39:

"So what do you think of proposals to make the ballots variable in size, instead of always the top 5 picks? It doesn't prevent slates from getting their candidates on, but it might prevent them from pushing off other works that non-slate voters tend to favor. I'm having a frustratingly hard time locating one technical proposal that others in this forum must have seen-- but even a system as simple as "anything with more nominations than a certain threshold gets on the ballot", a system that many real-world elections use (substituting voter signatures for nominations) could potentially work."

This feels like a simple fix that does a lot of good. Again, the goal is to make the number of winners greater than the number of candidates any individual can nominate. Whether you reduce the number of candidates to 2, or the number of winners to 15 -- or some variable number as Dr. Plokta suggests -- works.

Fragano Ledgister @57:

"The best method to limit gaming a nomination vote for the Hugos is the limited ballot (i.e., since there can be five nominees per category, limit voters to no more than n-x/n). A cabal like the Sock Puppets, given its limited numbers, is going to have a harder time pushing a single slate of candidates."

I don't know about "best," but I agree that this is a simple change that seems to do the trick against one voting bloc. Once the number of voting blocs exceeds the number of winners, though, only the blocs will win. So maybe this is just a temporary fix.

nathanbg @64:

"Cutting the nominations to one per category per ballot (1/5, in the terminology of 4/6) would probably solve the slate problem, but I feel like it would significantly reduce the chance for niche works to get on the ballot because it encourages people to vote for the candidates that are perceived to be popular."

Agreed. You want each individual to be able to nominate more than one candidate. My guess is that 2-3 is the right number, but people who know more about the details of this particular election will have to make that decision.

#112 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:39 AM:

Jeff R @32:

"The ideal system, obviously logistically impossible on several different levels, is to have every voter rank every eligible work as is currently done on the final ballot. So we should be looking for systems that more or less* produce the same top five results that that method would."

Fragano Ledgister @55:

"If there is going to be only one winner, then a preference ranking system is Alternative Vote, not Single Transferable Vote. Single Transferable Vote will produce multiple winners in a multi-member constituency. Alternative Vote will produce a single winner with a majority of preferences in a single-member constituency.

"The objective, in a ballot for a Best X contest, is to produce a single winner aggregating a majority of preferences. This means, when preferences are being ranked, that this is an AV rather than an STV system."

I don't think the nominations election benefits from any ranking system. It doesn't do anything to limit the power of voting blocs, and it's another barrier to voting. This is my main objection right now to a cumulative voting system: it forces voters to rank their nominations.

#113 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:40 AM:

tavella @35:

"However, pretty much every technical suggestion I've seen is gameable, and gameable loses. You aren't talking about ethical actors here; they will be perfectly happy to have a web page that assigns ballots to finesse whatever system is put into place. And when going against a system that wants to encourage diversity of votes, a slate will win in most cases. Which is why politics quickly turns into slate vs slate. Except no one who loves the Hugos wants to start creating counter slates, because even loving slates break the system as it becomes about people maneuvering to be on slates, not people individually suggesting what they think are the best of the year."

I don't think it's that bleak. Yes, every voting system is gameable, but some are more gameable than others and different ones are gameable in different ways. It's a problem worth thinking about.

#114 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:40 AM:

Phil Palmer @38:

"What you could change, if you wanted to, would be to make nominations transparent, so that every nomination, successful or not, was published with all the nominators' names and addresses."

This is going to have all sorts of unforeseen consequences, and is pretty radical and scary. Also, I feel, off the topic of election systems.

#115 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:40 AM:

Martin Schafer @49:

"One modification might be rather than eliminating a ballot once one member has won cut the rest of the votes in half (or potentially some other fraction) and once you had two winners down to a quarter etc. That would allow the nominator the satisfaction of being able to give at least some support to more than one work."

I have been thinking of a system along these lines. Something like: define a "happiness" level of each voter that's a function of the number of his nominations that win. Something like 1 if 1 nomination wins, 1.5 if two nominations win, 1.75 if three win, etc. Then computer a list of winners that maximizes happiness across the voters.

I was going to try to write this up this afternoon.

Nicole Poweleit @86:

Clever, but you're right about the downside: "The math is actually quite complicated and would be difficult to explain, making the process something of a black box." I think one of the important characteristics of any good election system is that it can be explained to a layman.

This is another reason why the random ballot method won't work. Try explaining how it's fair to someone who doesn't understand statistics.

#116 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:41 AM:

Eemeli Aro @67:

"For one alternative to changing the nominating system as such, we could make it more open: the rankings of the candidates could be published during the nomination phase, allowing nominators to observe the current state of the process, and to update their own choices accordingly."

I have been thinking about this all night, and I think it's just too chaotic. We have no idea what the effects of this will be. It certainly will be a useful recruiting tool for the voting blocs, and since they are already inherently more powerful I think it will in balance help them more than it will hurt them.

nathanbp @68:

"I'm not sure how I feel about that in theory, but in practice anyone trying to game the system would just wait until the last minute to vote, so I don't think there's much point in it. Or they would put in one set and change their nominations at the last minute, if that's allowed (I haven't nominated before, so I don't know)."

I also thought of that. One strategy for a voting bloc would be collect their votes at some intermediate point and then submit them all at the last minute.

#117 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:41 AM:

Mark Z @70:

While I have always had a soft spot for the random ballot method, I don't think it will work on our case. The results would be too, er, random.

It definitely does reduce the power of voting blocs significantly.

Cheradenine @80:

"I realized I mentioned it first, but I don't think random ballot is viable here, unless we want to rebrand the Hugos as 'the top work as recognized by WorldCon and the fickle, fickle hand of fate.' Getting people to accept a voting system with a nondeterministic outcome would be a hard sell, and it inevitably would eventually produce a really bad set of nominations."

Yes.

#118 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:41 AM:

Cheradenine @80:

"In general, I think we want to minimize the assumptions we make about future voter behavior in examining options for changing the voting system."

I agree with this 100%. Don't overly focus on the tactics of bloc voting that happened to be successful this year. They might only be successful this year, and different tactics might be successful in a future year. And, if we change our system in reaction to this year's tactics, the voting blocs of future years will change their tactics. A longer term more general view is essential.

#119 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:41 AM:

Josh Berkus @81:

"I'm interested that nobody seems to be addressing Bruce's points (1) and (2). I'd like to make a suggestion specifically around Bruce's recommendation (2), namely: change the electorate."

My hope is the reason is that I declared those points to be not under discussion in this thread. I hope there is another forum where people can talk about them, but I'd prefer that we stick to the mechanics of elections.

Greg M @100:

"The award itself should be determined by popular vote, and every WorldCon supporting member should get to nominate, but do the nominations have to be determined entirely by popular vote? The Academy Awards--and yes, as an Errol Morris fan and Hoop Dreams fan, this may be a *very* poor analogy--but they do have a volunteer committee that determines the documentary shortlist, made up entirely of documentarians."

I consider this to be a "change the electorate" solution, which I would like to be discussed in some other thread somewhere else

#120 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 06:45 AM:

If I can focus the discussion on Day 2, let's talk about the Single Transferable Vote System (see here and here).

In our case, votes wouldn't necessarily be ranked and wouldn't have to be single. They would be double or triple, in that they would be continued to be counted until 2 or 3 of their nominations win. And they would get some derived ranking from their popularity.

It's somewhat explainable. And it reduces the power of bloc voting.

What do people think? Can someone who has more time than I have this morning work through an example so people can more easily see how it works.

#121 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 08:21 AM:

One more reference.

We've been talking about dynamically expanding the number of nominees based on the voting patterns. Here's a good paper by Steven Brams on one way to do that.

#122 ::: D. Eppstein ::: (view all by) ::: April 08, 2015, 10:55 AM:

I suggested essentially the same system as in comment #1 (which in more technical terms is essentially the greedy algorithm for the hitting set problem) a couple days ago, here on Making Light. A couple of comments down from there, I added some analysis, which can be tightened a little: if there are n people making nominations for k slots, and D is the set of nominators who don't get any of their nominations onto the ballot, then every candidate who fails to get onto the ballot must be supported by fewer than n/(k + 1) of the members of D. Additionally, either D has fewer than 2n/(k + 2) people (meaning that most of the nominators are happy because they have at least one candidate in the final ballot) or the failing candidates all have less than a majority of D (no clear favorite remains among the disenfranchised).

For k = 5 (the Hugo ballot size) this means that, with this system, it's not possible for the dissatisfied voters to put together votes for a single candidate that would have more than a 16% share of the total votes, and also either over 70% of the nominators succeed in getting one of their candidates on the ballot or the remaining candidates have no clear favorite. The worst cases are when the nominators split into six different factions with non-overlapping sets of favorites, in which case five of the factions get one candidate onto the ballot and the sixth faction is shut out. In realistic situations where the support for different candidates is likely to be more diffuse, the numbers will be even better.

By the way, I understand the reasons why the moderators need to do this, but the current schedule of shutting down comments in the early evening (Pacific time) and reopening them only well past midnight is kind of exclusionary to working west-coasters. And by working west-coasters I mean me. I've had this comment ready to go since yesterday sometime and am getting it in only by squeezing in some morning internet time. Is there some other way of handling this? Some other regular in another time zone who could be pressed into moderator service?

#123 ::: Guesso ::: (view all by) ::: April 08, 2015, 11:00 AM:

I think the best option is to allow more than 5 nominees. I like the idea of anything getting 5% of the vote getting nominated. I am making this statement as a fan who has only voted once (signed up to vote for Wheel of Time last year).

As a regular fan who has never attended a conference this type of nomination makes me more likely to pay $40 to vote. The more books and stories I get in the voter packet the more value I get for $40. If I'm looking at getting 8 novels plus novella, short stories, etc... that is a much better deal for $40. I think this would appeal to alot more people and convince alot more people to buy supporting memberships. This approach is very fan friendly. Worldcon is a fan organization. If you give me more for my money I am more likely to participate.

The 2nd thing I would do is limit how many nominees you can make per category. You really don't need to nominate 5 things. Limit it to 3-4 and a total number of nominees of some sort. How many people actually nominate 5 things in each category?

Right now Worldcon represents a small slice of fandom. It appears to be heavily skwewed to SF fans over Fantasy fans. Urban Fantasy seems to do particularly poorly. The nominees tend to skew more liberal and more white male. By allowing more nominees more voices can get heard. There is more room for minorities to get nominated and there is room for conservatives to get on the ballot. This just increases participation.

As has been stated by the original poster is we have a problem of 2 elections. Nominations and then picking a winner. Its easy to game the nomination process due to political parties, well if you let in more nominees... this balances it out. Conservatives can still nominate stuff they like and have it appear on the ballot. Yeah Vox Day will get on every year, but he will just have crowd noise and he won't block anyone else. I don't have a problem with conservative voters putting Larry Correia or Baen Military SF writers. Its what they like to read.

I think this is a win for fans since we get more things to read for $40. Its a win for different groups since there is room for all of them to be nominated. Also, with the Australian rules voting, a small minority is not going to take home the Hugo. Authors all seem to like being nominated. The academy awards expanded to more nominees. I think the Hugos doing the same is a good idea.

#124 ::: Reimer Behrends ::: (view all by) ::: April 08, 2015, 11:01 AM:

To begin with, my interest is more or less the same as Bruce's here; namely, in voting systems as a theoretical construct, rather than the politics of the Hugo process (while I do have opinions on the latter, too, this thread isn't the time and place for those opinions).

I see two possible defects with Ron Rivest's original proposal when applied to the Hugo (both of which, I admit, may be imaginary). Both would seem to primarily occur in the absence of slate voting. However, assuming that a new system is successful in defeating slate voting, this new system needs to keep functioning in its absence.

One concern is that if a category has a naturally strong candidate (or two strong candidates with little overlap), this will heavily skew the ballot for the remaining candidates. If, say, 80% of the ballots are eliminated from consideration after 1-2 candidates have been selected, the selection of remaining ballots may be very random and may not even approximately reflect voter intent.

A second concern is a field that has a number of naturally outstanding candidates (but less than five). In this case, whoever leads the field could decimate the chances of the other strong candidates. While this might not be a problem if those aren't very diverse (say, all of them Dr. Who episodes), it is also easily imaginable that two or three completely unrelated strong candidates might emerge; in this case, the nomination process may preempt the election.

Rather than eliminating a ballot after a number of candidates on it have been selected, I am wondering if combining the procedure with Sainte-Laguë or d'Hondt might smooth out the above effects. In such a scheme, the weight of each ballot when selecting the next winner is not zeroed, but reduced according to the number of existing winners on the ballot (i.e. ballots are given weight 1/(2*W+1) for Sainte-Laguë or 1/(W+1) for d'Hondt, where W is the number of already declared winners on the same ballot). Sainte-Lagué's formula would have a stronger preference against voting blocks, d'Hondt's would come closer to a proportional scheme and wouldn't hurt natural ties or near-ties as much.

Applying this to the original example, using Sainte-Laguë's formula: A is selected as the first winner as before, since all ballots have weight 1. In the next round, ballots 1-4 and 10 (which A appears on) receive weight 1/3.

Tally for round 2: B: 1, C: 1, D: 3 1/3, E: 2, F: 2, G: 2, H: 3.

D is the second winner, ballots 1, 2, 3, 5, 6, 8, 10 now have weight 1/3 and ballot 4 has weight 1/5 (ballots 4 and 10 also have no remaining candidates left, of course).

Tally for round 3: B: 1, C: 1, E: 2/3, F: 1 1/3, G: 2, H: 1

G is the third winner, ballots 1-3 and 5-10 now have weight 1/3, ballot 4 has weight 1/5.

Tally for round 4: B: 1, C: 1, E: 2/3, F: 2/3, H: 1 (assuming we continue past round 3 for the sake of the example).

B, C, or H becomes the fourth winner (let's assume it's B, per some fair random procedure). Now ballots 1-4 have weight 1/5 and ballots 5-10 have weight 1/3.

Tally for round 5: C: 3/5, E: 2/3, F: 2/3, H: 1.

With that, H becomes the fifth winner. Winners are: A D G B H.

This approach has the interesting property that in the case of only non-overlapping slates being voted for (and assuming a suitable tiebreaker), the result would approximate the proportional representation of the slates (as it is then reduced to the original d'Hondt/Saint-Laguë procedure with slates representing party lists).

Running some quick and dirty simulations, it appears that the above procedure approximates the top five votes for some typical random distributions (though it's not always exactly those) and makes it very hard to nominate more than a couple of candidates (out of five) through block voting, unless the block voters greatly outnumber the regular voters.

I note here that any time a single group controls 50% or more of the ballots, they can always collude to get at least one of their candidates nominated and to then ensure that the same candidate wins during the final vote (assuming a voting system that other than that collusion is impartial). Thus, for purposes of dealing with attempts to game the vote, it should in practice only be necessary to consider those cases where one block controls less than half the ballots.

The obvious downside of the above procedure is that it is significantly more difficult to understand than the original (though, of course, d'Hondt and Sainte-Laguë by themselves are used in existing voting systems).

#125 ::: Peter Hentges ::: (view all by) ::: April 08, 2015, 11:12 AM:

In looking for ways to test how various voting strategies might work, I ran into OpaVote, which looks like it would allow small tests of various schemes using different voting and counting methods. This might provide us with ways to simulate the effect of block voting on various strategies.

#126 ::: Seth Gordon ::: (view all by) ::: April 08, 2015, 11:19 AM:

I’d like there be some way for me to fill out my final Hugo award ballot that says “I think A is better than B and B is better than ‘No Award’, but since I haven’t read C, D, and E, I abstain from expressing any opinion regarding them”. If I understand correctly, the current voting rules make that impossible: if I leave C, D, and E off my ballot, that’s like ranking all of them below “No Award”.

And this kind of abstention becomes even more desirable if we’re going to expand the pool of nominees for a category.

The nomination system that my statistically-unsophisticated brain came up with, before reading this thread, was to shuffle the ballots and divide them into five equally-sized piles. The work that gets the most votes within Pile #1 gets nominated. The work that gets the most votes within Pile #2 gets nominated, unless that is the same as the Pile #1 winner, in which case the Pile #2 runner-up gets nominated. And so on and so forth. Since the ballots from bloc-voters are unlikely to be equally distributed among the piles, a work favored by the bloc is actually more likely to get a slot on the final ballot, but the slate is less likely to sweep it.

#127 ::: Eimear Ní Mhéalóid ::: (view all by) ::: April 08, 2015, 11:19 AM:

Bruce at 120:

Irish elections are mostly STV (General elections, local elections, European Parliament). We tend to think of IRV as a special limited case of same.

Here's a link showing an animated version of the counts in the last local elections in one area
http://clairebyrne.ie/counts/animation.php?count=wexford_wexford
The "back" link on that page brings you to a page for all the areas: I picked one that had both distribution of surpluses and elimination of lower candidates.
It seems to me STV absolutely requires ranking of preferences to make the elimination process work. The real difficulty in using it for nominations is that the candidate pool would be so wide, the first count would take ages.

#128 ::: albatross ::: (view all by) ::: April 08, 2015, 11:21 AM:

This link has a game that illustrates STV.

One wrinkle that's a bit confusing at first about STV is the idea of transferring fractional votes. That is, if you have a quota to get elected of 5 votes, and you have ballots like this:

a b (3 ballots)
a c (3 ballots)
b a (2 ballots)
c b (4 ballots)


Now, a wins the first election (so a gets nominated). But a got 6 ballots, and it only needed 5 to get nominated, so there's one vote to be transferred. Half of a's ballots had b as their second choice, half had c as their second choice. So the fairest way to handle this is to transfer half a vote to b, and half a vote to c. Then for the second round, you get

b (2.5 ballots)
c b (4.5 ballots)

#129 ::: David Dyer-Bennet ::: (view all by) ::: April 08, 2015, 11:22 AM:

Cheradenine @80: I'm not absolutely sure about the counting rules already used for Hugo ballots, but the very similar system we use in Minn-StF (intended to be the same, but at the time we didn't have access to their *exact* rules and had to make up our own for some corner cases) already has non-deterministic outcomes.

It was, as I recall, to handle something like the case where the candidates have been reduced to three, the first does NOT have a majority, and the second and third have equal counts, and examining the votes for the second and third in the pile for the first shows no preference there either; that is, very unlikely but perfectly possible. Eliminating both as a tie would give the victory to #1, and we ended up choosing for our procedure to make a random drop in this situation. When I was running the counts, it never happened in an actual election.

A quick look for what's online about the actual Hugo process doesn't find me anything that goes into nearly that much detail. We found the problem, of course, when implementing software to count "Australian ballot" elections; that forces one to look at the possibilities in each situation rather more carefully, I find.

#130 ::: Carrie S. ::: (view all by) ::: April 08, 2015, 11:24 AM:

Guesso @#123: You do have to keep in mind that the Hugo packet isn't a requirement; it's not something the publishers are obliged to provide, and it's quite recent--I think this will be the fourth or fifth year it happens*. Saying "I want more nominations so I get better value for my con fee" strikes me as a little...tacky? I mean, I totally sympathize with being short of funds for recreational reading! But relying on the Hugo packet to make up that shortfall is a bad plan. (Nor is it particularly apropos to the voting system itself or whether allowing more slots on the shortlist would be good for ameliorating the effects of slate nominations.)

*: I'm in the SCA; I well grasp how something that happens two years in a row becomes a time-honored tradition. :)

#131 ::: nathanbp ::: (view all by) ::: April 08, 2015, 11:25 AM:

Bruce, #121: That paper seems to be behind a paywall.

After thinking about it overnight and reading Bruce's references, I'm no longer such a fan of STV for picking the nominees, and I think actually a system more like Bruce's in comment #1 would be a better solution. A primary part of STV is that "wasted" votes in excess of those needed to elect a candidate are transferred to other candidates on the ballots of those voting for the winning candidate. However, if the goal of the Hugo nomination process is to pick a diverse set of nominees, then it's better to just eliminate the ballots of those voting for the winner entirely. This reduces the power of slate voting and also reduces the power of non-slate factions within the voters. Basically the message Bruce's system in #1 says is "You should be happy one of your choices made the final ballot, now let everyone else have a chance."

It would be interesting to know how correlated or not ballots from previous years were.

#132 ::: Jo ::: (view all by) ::: April 08, 2015, 11:27 AM:

What about this?

Each nominator gets four slots in each category. They're not ranked, exactly, but they are classed. For example:

BEST NOVEL
Hedgehog: 25 HP, +5 damage vs. witch
Dalek: 25 HP, +5 damage vs. hedgehog
Witch: 25 HP, +5 damage vs. dalek
Mithril Mech: 30 HP, begins in herald slot

... so for best novel, my ballot might look like this:

Hedgehog: Jeff VanderMeer, Southern Reach
Dalek: Cixin Liu, The Three Body Problem
Witch: Adam Roberts, Bete
MM: Ann Leckie, Ancillary Sword

With each round of voting, each party is randomly paired with another. If the heralds are the same (i.e. in round one, if my ballot encounters another Ancillary Sword mithril mech) then both ballots survive intact and unchanged into the next round. Otherwise, a champion is randomly selected from the non-herald party members of each ballot. Then:

(1) if the champions happen to be the same (e.g. my Southern Reach bumps into another Southern Reach) then the champions move into the heralds slots, but no damage is inflicted, and both ballots survive otherwise unaltered into the next round.

(2) otherwise, both nominations take damage according to their class. For example, say my Southern Reach hedgehog gets paired against a John Scalzi Lock In dalek. My nomination loses fifteen Hit Points, and the Lock In nomination loses ten (my quills aren't much use against the dalek's armour plating and selfie-stick).

Nominations that have fallen to zero Hit Points are eliminated, and a new round begins.

The cycle continues until all except five novels have been eliminated.

(So there is a slight element of STV-style ranking, a nod toward Arrow's impossibility theorem via incorporation of chance procedures, a slight anti-organised fringe basin of attraction).

#133 ::: Andrew M ::: (view all by) ::: April 08, 2015, 11:27 AM:

Guesso@123: The more books and stories I get in the voter packet the more value I get for $40. If I'm looking at getting 8 novels plus novella, short stories, etc... that is a much better deal for $40.

I think it is unlikely that the voter packet would continue if there was a significant increase in the number of nominees. Publishers would have to supply more, and would have a reduced chance of a win.

It appears to be heavily skwewed to SF fans over Fantasy fans. Urban Fantasy seems to do particularly poorly.

Heavily? Six out of fifteen Novel winners since 2000 have been fantasy (more by some reckonings), plus one AH and one unclassifiable speculative. I don't know the figures for short fiction, but two out of three winners last year were fantasy. As I mentioned earlier, I don't think urban fantasy does worse than any other well-defined subgenre. The works that tend to get nominated are those which have cross-group appeal. The current voting system encourages this. I'm worried that a system where everything with substantial support gets nominated would lead everyone simply to nominate the things they are fans of, making the process less of a meeting of minds.

To me, the main problem with expanded nominations is the difficulty of reading them all (which may interact with the previous point; voters stop actually considering the whole field).

#134 ::: Cheradenine ::: (view all by) ::: April 08, 2015, 11:31 AM:

Reimer Behrends @124: I think the procedure you describe (or a slight variation on it) has been called "sequential proportional approval voting" (whew) and I was just going to post about it, since it seems like the closest approach to STV that's applicable and practical for unranked lists. I want to do some more research and experimentation with it. Some versions seem to favor cloning, which is bad news if we're trying to discourage slates, but I don't know how this approach would fare. I may not be able to work on this further until this evening, though (darn work).

#135 ::: Julian ::: (view all by) ::: April 08, 2015, 11:33 AM:

Tried to get this up last night, but took too long to write it:

I think the best approach is attempting to identify block votes and eliminate or seriously weaken their influence.

Approaches like that in 1 may work, but need testing on actual data sets to make sure they don’t overly hurt works with strong overlaps between their fandoms.

My proposal is that ballots are not counted in the 5% eligibility calculation at all if they are substantially similar to at least 3% of the other ballots in their category. (So, if you have 1100 ballots, and 100 of them are substantially similar, then works have to have at least 50 nominations on the other ballots to be eligible.)

If a work can hit the 5% threshold without the suspect ballots, they still count toward determining the actual nomination. This ought to make it less likely that works with strongly overlapping audiences will be hurt, and that an organized block can’t hurt a work’s chances by including it.

(If the category doesn’t get five nominations, then the remaining slots should be filled up with the highest-ranking works that would’ve reached 5% had all the ballots been counted.)

The hard part is defining “substantially similar” in a way that is both hard to manipulate and still comprehensible to the average voter.

My first attempt is:

A nomination of only one work is never substantially similar to another ballot. (This means that simple “vote for me!” campaigns and slates like SP2 can still affect the nominees. The former is not stoppable, and I think we can live with the latter; they’re far less disruptive to the awards than large slates.)

A nomination of two works is substantially similar to another ballot if it is identical to it. (This shouldn’t produce enough false positives to be a problem. It does allow for a “vote for these two, plus some other stuff” slate, but that’s more work for the voters, and that other stuff is legitimate nominations. (Though this requires invalid nominations to be eliminated before the votes are counted.))

A nomination of three or more works is substantially similar to another ballot if at least three of the nominees overlap. (By this point, I expect extremely few false positives — people’s tastes and reading habits are too high-variance to be concerned about it.)

It’s not perfect, and there are obvious ways to work around it by dividing the slate up, but doing that rapidly dilutes the slate’s effectiveness. (And if the block vote’s big enough, all their permutations could end up hitting the threshold.)

A possible additional rule would be something to the effect of “a ballot that hits the substantially similar threshold in three or more categories will not count for the 5% calculation in any category”, which would make workaround attacks even more complex.

#136 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 11:38 AM:

Aesthetically, I love the x = 5, y = 1, z =5 system for the first round.
("A voter can vote for x candidates. A ballot is counted until y of those candidates win. We need z winners total")

In practice it does have some problems, as commented above (which system doesn't?). It can encourage tactical voting.

It also can eliminate a very popular candidate if it is highly correlated with an even more popular candidate, but I tend to see that as a feature rather than a bug. If it's so highly correlated, why do we need both in the nominees list, if voters are telling us those two are basically the same? Comment #66 says that two works which are generally acknowledged as masterpieces of the year so that most people vote for both would get one of them eliminated by this system, but I think he is wrong. One of the masterpieces would be eliminated only if the correlation between the two masterpieces is extremely high, very much higher than with other candidates.

It would highly reduce the power of political parties.

Yes, I would love to see this in practice. Aren't we all going on about how we want more diversity in the lists? Well, this is the way to get true diversity, as opposed to having a list of nominees different from the traditional profile of the Hugo winner but very similar to one another (this happened for example in the 2014 Short Story category).

#137 ::: Bruce Schneier ::: (view all by) ::: April 08, 2015, 11:42 AM:

Here is a proposal for a nominating process that tries to spread nominations over a broader base of nominees, at the expense of slates that many people have nominated.

Each voter submits a list of nominations.

Each individual voter is unhappy if none of his candidates are on the final slate of nominees, and happier as more of his picks are.

So, to figure out what the final slate is, we create some numerical value for each voter's happiness, and choose a slate that maximizes the summation of that value.

So, to take the simplest case, let's call a voter's happiness 0 if none of his candidates win, and 1 if at least one of them wins. Maximizing the summation will maximize the number of voters who have at least one of their candidates on the final slate of nominees.

Or you could do something more complicated. Zero nominees = a happiness of 0. One nominee = a happiness of 1. Two nominees = a happiness of 3/2. Three nominees = a happiness of 7/4. x nominees = a happiness of 2 - 2^{1-x}. So no ones happiness is ever more than 2. Again, we choose a final slate that maximizes the summation.

The simplest case -- maximizing the number of voters who have at least one pick on the final ballot B -- is easy to explain. The second case is a refinement, and harder to explain.

#138 ::: Q. Pheevr ::: (view all by) ::: April 08, 2015, 11:55 AM:

Jo: The idea being that the system is less damaged by gaming if it is designed as a game in the first place?

#139 ::: Jon Lennox ::: (view all by) ::: April 08, 2015, 11:59 AM:

On publishing the nomination results:

As far as I can tell, there's nothing in the WSFS constitution that forbids a Worldcon from releasing raw Hugo nominating or voting (or Site Selection) ballots. (And indeed, I don't think voting is even currently required to be done by secret ballot!)

I was considering proposing a Business Meeting continuing resolution for Sasquan, to the effect of:

"A Worldcon Committee may, at its discretion, release additional information about nominations or votes for the Hugo Awards or Site Selection, beyond those required by sections 3.11.4 and 4.1.4 of the Constitution, provided the identities of voters are kept confidential. Committees are encouraged to do so in the event of controversy or allegations of impropriety, when additional information would be enlightening. This resolution applies to past as well as current and future Worldcon Committees."

The motivation here is to allow a committee to release of enough information to disprove (or prove, though I think that very unlikely) the existence of "secret slates" in the past, as well as to determine precisely the effect of the SP / RP slates this year. Since as I said it doesn't look like the constitution forbids this, I think this could just be a Business Meeting continuing resolution rather than a constitutional amendment, and thus take effect immediately and apply retroactively.

(This would also allow a committee to release the raw voting ballots, such that people could test alternate counting mechanisms such as the ones proposed in this thread for nominations, or Condorcet methods for the voting.)

I am a Supporting but not Attending member of Sasquan, so I should be able to propose it for the Business Meeting, assuming I get a second, but I won't be able to be there as a proponent.

Is anyone interested in seconding this and/or being a proponent for it at Sasquan? If so, I'll work with Kevin Standlee to get it into correct technical shape for the Business Meeting.

#140 ::: Buddha Buck ::: (view all by) ::: April 08, 2015, 11:59 AM:

Bruce @137:

It seems to me that the major problem with that approach is efficiency. It looks to be O(n^z) (where n is the total number of unique potential nominees appearing on ballots, and z is as defined earlier, the final number of nominees). There may be a more efficient way of optimizing this than the naive approach of trying all (n choose z) combinations of nominees, but I don't see it offhand.

In the simplest case (happiness is at least one successful nomination), it seems equivalent to the (x=5,y=1,z=5) case discussed earlier, but I only have a strong hunch, not a proof, that that's the case.

#141 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 12:00 PM:

The other option I like, which I suggested elsewhere, is basically what comment #29 suggested.

(three rounds of voting. Round one creates a long list - say 15 names. Round two creates a short list of 5, and round 3 narrows it to one.)

It has the problem of having an extra round, but otherwise it looks very good to me (and besides, the second round would be very interesting and generate a lot of debate and analysis). Candidates in the long list would NOT be called Hugo nominees. Only those who get to the short list are Hugo nominees.

Political parties representing minority interests could get their candidates in the long list, but they would find it hard to get into the short list.

Comment #29 suggested x=5, y=5, z=5 for the second round, whereas I had thought x=1, y=1, z=5
("A voter can vote for x candidates. A ballot is counted until y of those candidates win. We need z winners total")

My thinking was that x=1, y=1, z=5 would make it extremely difficult for a political party to shut all other candidates out of the short list, as has happened in the 2015 Hugos. They would probably not be able to place any of their candidates unless they have wider appeal. However, thinking about it now I actually prefer #29's system. My system encourages tactical voting, and x=5, y=5, z=5 is probably enough to get rid of voting blocks at this round.

#142 ::: Cheradenine ::: (view all by) ::: April 08, 2015, 12:00 PM:

Bruce @#137: Satisfaction Approval Voting uses this basic approach (with a satisfaction rating of m/n, where m is the number of a voter's candidates elected and n is the number they listed). It's described in the paper at:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1608051

Per the authors, it has a few properties that have come up as desirable in this conversation: it's clone-inhibiting (which disincentivizes slates), and it tends to yield better representation of diverse constituencies. I was worried about computational complexity with a lot of nominees, but it looks like, at least with their formula, it can be computed in polynomial time by computing a score for each candidate and selecting the most satisfying candidates. With a different formula (like that in #137) it might take a lot longer if we need to compute satisfaction over all possible selections of candidates.

#143 ::: albatross ::: (view all by) ::: April 08, 2015, 12:09 PM:

Not counting ballots selectively based on voting patterns that aren't fraud strikes me as a very bad idea. Part of the point of an election is to convince the losers they lost fairly. A scheme that throws some ballots out because they voted in a pattern (which coincidentally happens to be a pattern one "party" did well with the previous year) is not going to convince the people whose ballots were not counted.

#144 ::: Chris Gerrib ::: (view all by) ::: April 08, 2015, 12:09 PM:

Farseeker @ 141 - Although the 3-round vote seems simpler, I'm getting a lot of pushback from the people who actually have to administer it.

The more I think about it, the more interested I am in Dr. Plokta's idea. The good doctor has ran analysis on the past couple of year's data, and all that happens is we pick up a couple of names, usually in Long Form and Editor.

#145 ::: Douglas Knight ::: (view all by) ::: April 08, 2015, 12:19 PM:

Bruce, in your original post you seem to use "Australian ballot" to mean "instant-runoff." While the term is no longer common in America, "Australian ballot" already has an established meaning in other English-speaking countries. Australians I have met are prouder of their earlier role in voting innovation, the secret ballot.

#146 ::: Fragano Ledgister ::: (view all by) ::: April 08, 2015, 12:21 PM:

Albatross #62: The difference between the two types of preference systems is this:

The alternative vote is designed to agglomerate a majority of preferences among a substantial number of rivals. That is, when all the votes are tallied a majority of people should have listed the winner in first, second, third, or maybe fourth place. Assuming, that is, that there are a large number of rivals for first place, and a lot of different rank-orderings.

Single transferable vote is designed to produce more than one winner. To win, therefore, the candidate must receive a 'quota'. Now, there are two different ways of calculating a quota. The simpler is the Hare quota (named after the Irish mathematician who invented STV) which is total votes/total seats (or prizes or whatever). Then there's the Droop quota (named after an English mathematician, not after the way it makes you feel), this is (total vote/total seats+1)+1. The Droop quota is the one in more frequent use in countries using STV, like Ireland.


#147 ::: spacefaringkitten ::: (view all by) ::: April 08, 2015, 12:28 PM:

I'm too mathematically-challenged to make sure whether STV or the solution in #1 is better.

Some commenters have pointed out that under the system proposed in #1, nominating one relatively popular work (that is likely to make it to the shortlist) renders all your other nominations meaningless.

Perhaps one possible solution could be something like this:
-Pick two winners (instead of one) first
-Eliminate all ballots that nominate those two works
-Pick the next winner
-Eliminate all ballots that include two of the three selected winners
-Pick the next winner
-Eliminate all ballots that include two of the four selected winners
-Pick the last winner

A slate would still be able to force two works on the final ballot, but no more. There would be no problem with nominating both Stross and Stephenson, who were mentioned in #17. In addition to them, though, you can't get Peter Watts in as well. Is that too generous for the slate-voters, then? I feel that a system where only more nominees are added and none of the items on a slate is taken out is even more generous.

Another (admittedly more political than mathematic) solution is to have some sort of a jury in the nomination phase. For example 50% of the nominating votes could be given by them and the other half by the Worldcon members (or 40/60 or whatever), so that fans aren't cut out of the process completely. Choosing a jury is problematic, of course. The member vote could be organized with any of the different proposed voting systems.

If Eemeli Aro's proposal in #67 is possible to be implemented without any rule change, that is surely worth thinking about. But doesn't that sort of system require something that encourages people to put their votes in before the very last deadline? There's also the possible (or very probable) problem of all voting becoming very tactical in this scenario. By the way, if this could be carried out without a rule change, who has the power to make it happen? Worldcon 2016? WSFS people?

At the end of the day, convincing a big enough majority of potential nominees of the fact that being on a slate is despicable may be easier than designing an adequately gaming-proof system.

#148 ::: Stephen Rochelle ::: (view all by) ::: April 08, 2015, 12:30 PM:

Farseer2 @136 wrote: "It [discarding a ballot once y=1 work on it places] also can eliminate a very popular candidate if it is highly correlated with an even more popular candidate, but I tend to see that as a feature rather than a bug. If it's so highly correlated, why do we need both in the nominees list, if voters are telling us those two are basically the same?"

"Those two are basically the same" is not at all a reasonable assumption to make for the scenario you posit. Books aren't that fungible. We can pretty safely assume that Skin Game and Trial by Fire are heavily correlated on the nominations, but having read them, they're nothing alike. One is "Wizard, P.I. Pulls a Heist" and the other is "Near-Future Counterinsurgency with Aliens". I liked both, and while I don't think either is their respective authors' strongest work, I plan to read the next in each series. But I wouldn't read them interchangeably, nor would I recommend them to a friend interchangeably. Why should it be assumed that I'd nominate them for an award interchangeably?

Nope. "Mutually popular" should not be conflated with "redundant".

(BDP Short is another excellent category to talk about the problems with this approach, but as I've bounced off Dr Who, I don't have much specific to say about it)

#149 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 12:32 PM:

Dr. Plokta's idea is elegant, and also simple enough to be acceptable by the general public as another apparent quirk of the Hugo elections (unlike the "maximizing happiness" approach, which would be unacceptable for people who already find it hard to understand how "No Award" works... unless we reduce "maximixing happiness" to the trivial case which is equivalent to x=5 y=1 z=5).

The Plokta system should prevent a voting block from shutting all other candidates out of the nominees list. However, the voting block would still be able to get all their candidates nominated, which for me is a problem even if a couple of other candidates get into the ballot. Of course the Hugo win is the big prize, but to be able to easily dominate the nomination lists... in the Hugos that's still a big prize for a relatively small but disciplined block. They would still be able to get disproportionate attention, even if they got no wins.

#150 ::: Fragano Ledgister ::: (view all by) ::: April 08, 2015, 12:34 PM:

Bruce Schneier #112: I think you misunderstand me. I'm not proposing cumulative voting for nominations, I'm proposing a limited ballot.

That is, there are currently five nominees. I'm proposing a nominating ballot that is smaller. At most four, ideally three. (That is n-x where n is the number of nominees and x is a smaller number.) This is a lot harder for a cabal to game, especially a cabal of a couple of hundred out of a few thousand potential voters.

A cumulative vote isn't a preferential system by the way, it's a semi-preferential system. The voter is given a number of votes and can spread them out or pile them up (cumulate them) depending on their will. One of my colleagues is an advocate of that system in another context.

#151 ::: Cheradenine ::: (view all by) ::: April 08, 2015, 12:35 PM:

Buddha @140: They're not quite equivalent. Consider an example where two candidates are chosen from the following ballots:
1 A B C
2 A B
2 A C
1 B
1 C

A has 5 votes, B 4, and C 4. So the system in #1 will choose A, discard the first 5 ballots, and then choose either B or C. Under Bruce's simpler case from #137, it would choose B and C because that maximizes satisfaction and omit A despite it being the single highest vote-getter.

#152 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 12:38 PM:

Stephen @148: If those books are actually quite different I find it difficult to believe they would be so highly correlated. Of course every book is different, but really, with nominations so diversified, having two books almost always appear together or not at all is a sign of block voting or of both books satisfying exactly the same kind of reading taste. If we are going to push for diversity, do we really need both in the ballot?

#153 ::: dh ::: (view all by) ::: April 08, 2015, 12:46 PM:

@137 Bruce--

That smells like a recipe for disaster and disorder.

In a binary election like US Presidential elections, you have (usually) a close race, and in the end, a small majority get EXACTLY what they want - their candidate wins. The minority gets nothing of what they want. This is stable at least because a lot of people get what they want.

In this system, it seems like everyone will get (in your example) either 0, 20, or maybe 40% of what they want.

So beyond having a system which is pretty hard to audit and explain, you also have a system where more people get less of what they want. The only upside is more people will get a non-zero amount of what they want.

#154 ::: Chris Gerrib ::: (view all by) ::: April 08, 2015, 12:49 PM:

Farseer2 @ 149 - but to be able to easily dominate the nomination lists... at the end of the day, the Sad Puppies are in fact Worldcon members and most years, 200 members are able to get some works on the ballot. We won't know what would have happened this year - my guess is we would have had 9 or 10 nominations in each category.

If, after a massive slate and a final list of 9 or 10, the Sad Puppies got spanked, then the wind goes out of their sails.

#155 ::: perlhaqr ::: (view all by) ::: April 08, 2015, 01:00 PM:

Ctein @76: "On the other side, if I were a GGer, I'd be thinking "Cool, I bet I can track down at least 10% of those SJWs (meaning everyone who ain't a GG or SP) and then I'll DOX'em." "

If the Hugos publish name and address of people who nominate, as suggested in #38, it won't be the GGers who are "doxing" people. The Hugos will have taken care of that for them already.

#156 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 01:06 PM:

Chris Gerrib @154: Probably less nominations, because in short fiction there is a lot of dispersion. We would have to know the numbers, but probably we would have around 7 nominations, 5 of them from the voting block.

I have nothing against Sad Puppies candidates or anyone else winning the award or dominating the nomination lists. My problem is that they do so not by bringing a really large number of voters, but by bringing a respectable but still relatively small number of voters and getting them to behave like a political party. Other people are not able to organize their own political parties because they believe it defeats the purpose of the award, and the SP obtain great results not because they are a lot of voters, but because they behave like a party when the rest of voters can't. Most people find that situation unfair, and that is why we are having this conversation.

#157 ::: Arkady Martine ::: (view all by) ::: April 08, 2015, 01:06 PM:

dh @153: You are missing a significant and fundamental difference between a US presidential election and a nominating election which ends in a multiple-option ballot: i.e., partial success is a possible, and perhaps an ideal, end-state. While in a presidential election in the US only one person can win.

The model Bruce is working with trends toward more diversity on the ballot; I personally find this a positive outcome.

#158 ::: Tammy Coxen ::: (view all by) ::: April 08, 2015, 01:12 PM:

#157 - It's also in keeping with the reasons we use IRV for the final determination - the goal is to identify the work that the most people have some level of support for, not that just 51% of people have a lot of support for.

I think it's probably a non-starter for administrative and explanatory reasons, but it's philosophically consistent.

#159 ::: Rail ::: (view all by) ::: April 08, 2015, 01:13 PM:

Eemeli Aro's proposal in #67 is what Woot Shirts uses for their weekly design derby. I'm not sure what they find the plusses and minuses of the system to be.

#160 ::: Andy Brazil ::: (view all by) ::: April 08, 2015, 01:16 PM:

Umm, one thing that's been bothering me. How easy is it to game the business meeting? Because logically, having learnt how to get a slate nominated, the next step is to get rid of no-award as an option? Which, if I were so inclined, I would do as a long term plan starting with seemingly innocent and deeply technical rule changes. (Actually you might want to think about any rule changes that were proposed last year and are up for a vote this year, with an eye to how they might be gamed in future years.)

#161 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 01:18 PM:

@Rail: For me Eemeli Aro proposal in #67 is not a good idea at all. It relies on people to detect voting blocks and organize themselves to defeat them. However, it is extremely easy for a voting block to defeat this system, simply by waiting until the last day to vote. Besides, this system rewards exactly the kind of organized political behavior that we would like to discourage in a literary award.

#162 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 01:26 PM:

@160: Andy Brazil: It is very difficult to change the rules in an abusive manner. You could appear by surprise at the meeting with a large enough number of cronies and change the rules in an outrageous way. However, the new rules thus approved are not yet in operation. They must be ratified the next year at the next WorldCon, and only then are they in operation. If you made some outrageous change in this manner, there would a big storm in fandom, and at the next WorldCon and very large crowd would turn up at the meeting and vote against ratifying your changes.

#163 ::: Stephen Rochelle ::: (view all by) ::: April 08, 2015, 01:41 PM:

Farseer2 @152: But your 136 is "if there are two masterpieces, then they're probably redundant". The logical extension of that is a prioritization of variety above that of quality.

Last year, 23% of nomination lists (368, in raw numbers) included Ancillary Justice. A y=1 proposal says that a quarter of the Best Novel nominations should have been immediately discarded at that point, as those people have already all had their preference considered. I suspect, but cannot demonstrate via the public data, that chopping a quarter of the data would have an outsize effect on the rest of the ballot. 368 ballots eliminated when the gap between 5th and 10th place in the nominations was just 17 ballots? A 5% margin of error for nominally-uncorrelated data to avoid a five-place shake-up?

It gets worse in the other categories. Top nominee in BDP Short, BDP Long, and Best Semiprozine (again 2014 data) all got better than 40% of the nomination pool; the 5th-thru-10th gaps meanwhile remain comparably small.

Discarding ballots at one success doesn't look like it will break the power of slates so much as guarantee that any focused slate can grab one slot while the final spaces are filled in by comparatively low-popularity niche stuff, assuming they can stay above the 5% threshold (note that Short Story, the most-varietal / least-correlated category, is also the one that has the most 5% issues).

Bleh. Count me out of that "solution".

#164 ::: Kimiko ::: (view all by) ::: April 08, 2015, 01:49 PM:

Bruce,
I'm so glad you are here.

I think I read the whole thread, but please forgive me if this has already been rejected:

What if the nomination list was unlimited? Let the nomination process produce a ranked list, however long, and the voters could look for themselves to see if they like something in the long tail of the distribution, while at the same time they could "see" what the community thinks is significant. So long as only a reasonable number (5? 7?) were allowed to be nominated per person, people would still have to value their votes as a scarce resource. I don't think this would defeat slate voting altogether, but I think it would make a given slate bunch up conspicuously in the distribution. Maybe more information would help people pick a better strategy? Maybe this would help people find really good books they would have otherwise missed? That might be a really good outcome!

There are some practical issues with this, but if the voting population is around 2000 members, and by chance half of everyone picks a unique work, then 2000/2 = 1000*5= 5000 works with only one vote. If the cutoff is at least two nominating votes, the final nomination list could still be conceivably printed in a 60-ish page letter-size booklet.

#165 ::: Stephen Rochelle ::: (view all by) ::: April 08, 2015, 02:02 PM:

Kimiko @164: I'm not Bruce, but I can talk to some of your comment.

The issue with an arbitrarily long nominations list is that IRV loses effectiveness if you can't vote for all of the nominees. So if there are 50 nominees, but we keep the voting to 5 and you love 5 fairly obscure ones, your vote will quickly be converted to "no preference". That's probably not seen as desirable.

OK, so open things up to an arbitrarily-long IRV ballot. Expressing full preference now becomes a real chore on the voter's part. You might be able to keep your ballot that champions those 5 obscure works relevant all the way through, but it's gonna take some work.

Fine, pick the middle ground: there are n works you actually care about from the 50, so you rank those n and leave the rest of your ballot as "don't care". Great! We've solved everything... unless you also want to usefully vote No Award above Terrible Travesty Thing. Because once No Award is on your ballot, everything you didn't list on your ballot is also below No Award.

So, if you're going to have a 50-item nomination list with IRV and a No Award option, you're asking voters to deliberately and exhaustively rank a huge list in order to express a reasonably simple preference of "Thing A, Thing B, Thing C, whatever, as long as it's not Thing X". It doesn't scale well.

#166 ::: Julian ::: (view all by) ::: April 08, 2015, 02:12 PM:

albatross @ 143:

I agree it’s unacceptable for a normal election, but the goals of an award nomination are not the same as a normal election. Block voting encourages people to nominate work they have not actually read, and reduces the viewpoints represented on the final ballot, both of which are undesirable.

If there are effective anti-block rules, the way a group that feels that the things they love aren’t getting nominated can best influence the award moves closer to the desirable behavior for an award nomination process: getting out the vote and getting people to nominate a diverse slate of things. If there’s enough of them, and there certainly seemed to be this year, they’ll see things they like on the ballot, but so will other people.

#167 ::: Cheradenine ::: (view all by) ::: April 08, 2015, 02:21 PM:

Following up on Stephen Rochelle @#165: The other problem is that if most people rank only part of the nomination list, the final vote is much more likely to elect something with less than majority support - meaning it makes the final vote more gameable.

#168 ::: Buddha Buck ::: (view all by) ::: April 08, 2015, 02:30 PM:

Stephen Rochelle @165:

It could be done moderately well if you used a ranked ballot system which allowed you to express tied opinions other than "worse than everything expressly voted upon", and even better if it allowed you to express no opinion on choices you haven't read.

Imagine there were 15 books on the ballot, A-O (or A-Q if you choose to not use I and O), and you've read, or tried to read, 7 of them, A-G, before voting time came. If you thought one was superb, two more were Hugo-worthy but couldn't decide which was better, one was good, but not up to Hugo standards, two were so-so, and one left a dent in your wall when thrown, it would be nice to be able to cast a ballot like "A>B=C>NA>D>E=F>G" to express exactly those preferences, including a lack of opinion about H-O(Q).

Unfortunately, I don't think IRV or it's variants (STV, etc) are compatible with such flexible ballots, as they depend heavily on the concept of "first-place votes", which would fail on the ballot above once A is eliminated.

Fortunately, there is real-world experience that at least some Condorcet election methods can handle such ballots.

#169 ::: Steve ::: (view all by) ::: April 08, 2015, 02:33 PM:

#137 seems really good to me. Actually solving it would required a computer, but it's easy to explain to the voters, and it looks very hard for a slate to dominate. In order for a group of voters to guarantee a final slate of 5 they need to comprise 83% of the nominators, at which point they pretty much deserve to dominate the final slate.

#170 ::: Brad from Sunnyvale ::: (view all by) ::: April 08, 2015, 02:41 PM:

Ron's proposal (#1) is interesting, and would push a lot of diversity in the ballot -- as some suggest perhaps even more diversity than desired.

However, my biggest concern is over strategic voting. As those who study voting systems know, some systems are very prone to strategic voting, some are more robust against it. When there is a value to strategic voting, you deliberately vote something other than your true opinion in order to help your true opinion indirectly (or hurt its opponents.)

Preferential ballots are used by the Hugos (though we use the wrong one, the Hale IRV, instead of Condorcet) because they greatly reduce strategic voting on the final ballot. Collusion actually also has little value in the final ballot.

In the nominating ballot, collusion is obviously powerful, as we're seeing here. Ron's approach reduces that but creates other strategems. This is bad for two reasons. First, the stratagems may work and bias the result, or second, they may not work but still cause people to vote differently.

Hugo nominations, and even the voting are really supposed to be a survey, not a vote. Indeed, one cause of this problem is the SP treated this as an election (where political parties are common) and not as a survey (where they entirely invalidate the survey.)

Surveying is of course a highly studied and decently understood field. The risk of self-selection (which is how Hugos work) is well known and feared.

As a survey, the question of the Hugos is, "What is the most representative view among Worldcon fans, of the greatest works of this year?" Or at least that's what I think it should be. Some might argue it should be what the view is of "fans who care most strongly," particularly because only a minority nominate or vote, a very small minority in some categories.

One simple strategy -- choosing con members at random to nominate -- does not defeat slates, because many chosen members do not wish to nominate, but all slate supporters do.

Any algorithm to find clusters can likely be defeated, particularly because the algorithm will probably be public, allowing an attacker to test their approach against it.

Major increases in the number of nominees could present real problems, as the ballot becomes too large for most people to then go out and read them all. A final ballot of 15 novels would result in a win simply for the novel that was mostly widely chosen to be read. (This is already an issue, and a larger ballot increases it.) The whole purpose of a final ballot is to reduce the number of choices to give people a chance to actually judge fairly among them all.

A jury (such as the Nebula uses) is a likely approach, though quite unlike what is done now. Jurors could be chosen by some impartial method (former Hugo or Nebula winners) or elected by attending fans at the previous con, or delegated (anybody who gets 100 fans to support them is a juror.) Editors would actually make great jurors but they have too much skin in the game. The ideal juror consumes works voraciously but has no conflicts of interest or agenda.

Though if the jurors and their biases are known, you could get strategic nominating among them!

#171 ::: Sam M-B ::: (view all by) ::: April 08, 2015, 02:42 PM:

Bia Jason Sanford, "A modest Hugo Award proposal: [I]nstead of simply tweaking the Hugo rules, perhaps a better solution is to find common ground and agree to fix the Hugo Awards once and for all."

http://www.jasonsanford.com/blog/2015/4/a-modest-hugo-award-proposal

The proposal being Condorcet count with proportional representation:

"Proportional representation means basically that if 60% of ballots are A-B-C-D-E and 40% of ballots are F-G-H-I-J that the 5 nominees are A-B-C-F-G. This is what I actually favor: minority representation is important no matter which "side" one might be on."

http://civs.cs.cornell.edu/proportional.html

#172 ::: lorax ::: (view all by) ::: April 08, 2015, 02:57 PM:

Sam M-B @171:

Proportional representation concedes the existence of political parties or slates, though, which is not something most of the contributors here are willing to do. The existing situation isn't 60% ABCDE and 40% FGHIJ; it's 10% ABCDE and 90% everything else, including some that have A or C or both, 15% that have Q and 15% that have Z (some of them have both, or also A and/or C), and so forth.

(Or was the use of "modest proposal" intended to indicate this proposal wasn't intended seriously, and I just missed the satire indicator?)

#173 ::: Cheradenine ::: (view all by) ::: April 08, 2015, 03:01 PM:

Sam @171: Many of the methods that have come up in the thread are proportional representation methods of one sort or another. The Condorcet count with proportional representation has the same disadvantage as STV in that it requires ranked ballots, and the general sentiment seems to be against requiring nominators to rank. (That particular implementation, according to the Cornell page, also has the significant issue that there's no method proven to yield the best result which is computationally feasible.)

Looking over the thread, it looks like the main proposals so far (at least in Bruce's 3rd category, which we were aiming to discuss here) fall into one or both of the following categories:
1) Use some sort of proportional-representation system (STV, satisfaction approval, etc.).
2) Increase the number of nominated works, or the ratio of nominated works to works on the ballot.

I suspect that both approaches would probably be effective at preventing this year's particular failure state (a set of works supported by a minority completely dominating the ballot). It's worth considering the possible side effects of each in general, as well as the specific implementations. (Also, is there a category of suggestions I'm missing that doesn't fit more into the "change who's voting" idea?)

#174 ::: Cheradenine ::: (view all by) ::: April 08, 2015, 03:03 PM:

lorax @172: Sometimes "proportional" is used more generally to denote methods which will tend to include choices supported by minority groups among the winners. The proportional Condorcet method doesn't require that parties or slates exist; it just handles the situation where they do gracefully.

#175 ::: Doire ::: (view all by) ::: April 08, 2015, 03:07 PM:

Bruce Schneier @116
Eemeli Aro @67

A few years ago there was a Livejournal election (for members of the advisory board) that used instant runoff, but allowed for changing your votes until close of the election. The current voting results were visible to voters.

I'm not sure that they realised voters could transfer their votes themselves for maximum effect.

My thoughts at the time

#176 ::: Jo Walton ::: (view all by) ::: April 08, 2015, 03:16 PM:

The objections to Eemili's proposal would rely on a much greater degree of organization than the Puppies have or are likely to have. They're not a monolithic block, they're not actually cheating, and they're not in anyone's control. Making them do something really hard -- all nominate on the last day -- seems really unlikely.

There is a tendency to consider one's enemies as much more powerful and scary and organized than they are. Don't sell yourself to fear in your mind.

The Puppies are about 200 or 300 people with mixed motivations. They wouldn't be doing it if they had to pay attending memberships. They wouldn't be doing it if they all had to do it the same day.

This isn't an endorsement of Eemeli's plan, (though I do like the fact it could work next year and not have to wait) just an objection to the objections.

#177 ::: Joshua Kronengold ::: (view all by) ::: April 08, 2015, 03:19 PM:

Jon@#139: I might be willing to second if I was convinced it was useful and would have a chance of passage. I'm not at the moment, although I'm also not convinced that having open (in aggregate, with a threshold) nominations would work badly in the end.

The problem with the Plotka rule is that a block would still dominate very diverse (Best Short Story, Best Related Work, Best Fan Writer) categories -- often, very few to no nominations in a normal year make a 10% threshold, and they wouldn't here either, so if there was a block they'd take all the nominations.

STV would work and well, but as said it would require more from nominators than we require now, as their ranking would matter.

How about Single Divisible Vote? Every nominator gets 1 vote, which is ordinarily divided up among their nominations. Just as in STV, we then eliminate the weakest candidate -- but then, we reapportion the voters among their new set of nominees. When there's a tie for elimination, we either eliminate all the tied nominees (if doing so would not put us within our cutoff, or if we currently have twice our cutoff (for instance; some number that's too many to put up for a vote) candidates left) -- or stop, as we do when we hit our cutoff.

Under this system, if we have five voters:

Alice: 1, 2, 3
Bob: 1,2 3
Carol: 4, 5, 10
Dave: 5,6, 9
Ed: 6,7, 8

Alice and Bob are part of a voting bloc. Carol, Dave, and Ed are not, but they do like a few things in common.

Under the current system, if our threshold is 3 (ie, three things get nominated), then the AB bloc just wins, nominating 1, 2, and 3 over the wishes of the other three voters.

Under my proposed system, it's a little more complicated -- we first eliminate 10, 9, and 8, 4, and 7 as being tied at 1/3 of a vote each, and we're then left with:

1,2,3: 2/3 each

5,6: 3/2 votes

And stop; we can't eliminate the bloc votes without dropping below our theshold, so our nominees are 1, 2, 3, 5, and 6.

#178 ::: Joshua Kronengold ::: (view all by) ::: April 08, 2015, 03:27 PM:

FWIW, re #1, I think my smofs suggestion is closer to #177 than to #1 (I don't know which of mine it was referring to, but my initial thoughts were stv redoing the stv process after we had a "winner" without the necessary ballots to win, and a weaker SDV system where once you win one we apply a negative weight to the rest of your nominations, to make it harder for a bloc to get multiple noms while not eliminating most of our signal if most nominators included the same thing). But I think I like SDV best at the moment--although increasing the threshold (going to 4/6, say) past the number of things you can nominate might also be necessary if we want to avoid the issue where a large bloc wipes out categories where the winning ballot typically has less than 10% of the nominations.

#179 ::: Peter Hentges ::: (view all by) ::: April 08, 2015, 03:32 PM:

Jo @176: I don't think we can accurately discern the organizational capacity of future slate-voters based on our perception of this most-recent set.

#180 ::: Joshua Kronengold ::: (view all by) ::: April 08, 2015, 03:50 PM:

Peter@179: While we should look at whatever we do not being trivially breakable, I don't think we need to solve problems we don't have yet.

#181 ::: Lee ::: (view all by) ::: April 08, 2015, 04:12 PM:

nathanbp, #68: We already have a model for how that works, in a different context: eBay. What you're talking about here is called "sniping", and is a well-known strategy complete with tips on how to make your snipe more successful and software to make the process automatic. I don't think it's something we want to import into the Hugos.

Rene, #93: It's worth noting here that Larry Correia, who has a background in statistics, undertook to analyze at least one year's worth of Worldcon ballot results and actually said that he found no evidence suggesting that sort of collusion.

This has not, of course, stopped the SPs from insisting that such collusion must exist; they just assume that it's far better hidden than even they thought it was.

Phil, #102: Non-anonymous nominations suffer from exactly the same problems as non-anomymous final balloting, and for exactly the same reasons. Also, I for one would greatly prefer not to be one of the people thrown under the bus for the sake of your ideology. You don't get to tell me what risks I should, or should not, be willing to undertake.

#182 ::: Steve Halter ::: (view all by) ::: April 08, 2015, 04:25 PM:

Cheradenine@142:That Satisfaction Approval Voting paper was quite interesting and does seem to display many of the characteristics we want here. In addition, it is fairly easily computable.

#183 ::: Mongoose ::: (view all by) ::: April 08, 2015, 04:25 PM:

GRRM has just posted on the subject, in pessimistic mood. He will be making further posts later.

#184 ::: Brad from Sunnyvale ::: (view all by) ::: April 08, 2015, 04:32 PM:

I want to reiterate an important point, one definitely not understood by the bloc voters but also by many others.

I believe the Hugos are not, in spite of the trappings of voting, an election. They are a measurement. The goal of the measurement is to provide the best estimate of the most well regarded work by Worldcon fans.

It's important to understand that while the individual opinions are opinions (and it is totally OK to try to influence them with arguments of all sorts) the measurement is not an opinion, it is an estimate of a fact. The aggregate of the opinions is a fact about the opinions.

You must not have biases or politics or compromises when judging facts. You just want the most accurate measurement of the facts.

In doing a survey to learn the fact, collusion among participants completely invalidates the survey as a means to the facts, and this is the problem with slates.

There are other problems as well, of course. Our survey is self-selected, which also huts its accuracy. Not self-selected to Worldcon members (because they award is defined as being the choice of Worldcon members) but self-selected to those who take the effort to nominate or vote.

We mathematicians dislike self-selected surveys, they have very limited validity. They tell us only the opinion of those who cared, which may differ from the broad opinion. Of course if you declare that the fact we are seeking is the aggregate opinion of those who care, then the survey is valid for that.

The error of the SP approach is obvious from this context, the slate corrupts the measurement of facts.

But beware: This also gives credence to arguments that a combination of groupthink, social pressure, popularity and lobbying can bias the self-selected sample away from the truth, which is a more polite expression of their claimed grievance.

#185 ::: Neil W ::: (view all by) ::: April 08, 2015, 04:38 PM:

Hmm @72 "In the method described in #1, are there any less arbitrary options for resolving ties than flipping coins?"

Football ("soccer") matches that needed a result for a competition used to use this method after all other attempts to get a result (extra time, replay etc.) The modern game uses the penalty shootout, which suggests the possibility of a live write-off/edit-off (etc.) at the ceremony in the event of a tie.

Cheradenine @80 - "the top work as recognized by WorldCon and the fickle, fickle hand of fate."

If I were setting up a new award with this system, I'd use a version of your description as the award subtitle.

#186 ::: abi ::: (view all by) ::: April 08, 2015, 04:42 PM:

Brad @184:
I believe the Hugos are not, in spite of the trappings of voting, an election. They are a measurement. The goal of the measurement is to provide the best estimate of the most well regarded work by Worldcon fans.

Yes, this. That's what I was trying to get at with this bit in my blog post:
...the weirder and more subtle belief that the “right result” of a Hugo vote is unknowable, and can only be achieved by using the right means to go about it.

#187 ::: spacefaringkitten ::: (view all by) ::: April 08, 2015, 04:42 PM:

Jo @176 & Peter @179:

It's not about slate voting, necessarily. I think the proposed solution would possibly turn everybody into tactical voters, because it would be worthwhile to vote on the last minute to push in something you might like that is not going to make the cut otherwise (and abandon anything you love that is so far behind that it has no chance).

#188 ::: Sam M-B ::: (view all by) ::: April 08, 2015, 04:43 PM:

Cheradenine @173 and Bruce @many,

I have to admit to not understanding the allergy to ranked ballots in the nomination round. We rank ballots in the final round, and adding the number 1-5 in the nomination round is hardly a "burden" is it? (Especially when using a voting method which is resistant to strategic ranking, meaning you don't have to double-think your rankings. And it's even less cumbersome than in the final round because you don't have to squirrel around "No Award" tactics. And, with Condorcet counts of which I am aware, I can vote everything the same rank, or unranked, if I feel it's not worth my time to rank them, and that's fine. People who want to rank, can rank, people who don't, don't have to, in the nomination round.)

That said, even being a bit of an amateur voting methods wonk I'd never heard of "sequential proportional approval voting" or "Satisfaction Approval Voting" and so I have some reading to do.

#189 ::: Buddha Buck ::: (view all by) ::: April 08, 2015, 04:51 PM:

Neil W @185:

The method described in #1, and in general, most of this discussion, centers around selecting nominees, not selecting final winners. Live write-offs/edit-offs to select a single final winner would take place long after this procedure was completed.

Besides, I think the current rule is that if there is a tie, multiple Hugos are awarded, not a coin-flip or a showdown.

But it would be interesting to have an edit-off: "OK, you both have 4 hours. You each have a pile of identical unopened slush. You need to put together a 50-page magazine. You will be judged on quality of stories you choose, how well you hit the 50-page mark, and overall unified theme. Ready? Steady? Go!".

#190 ::: Chris Gerrib ::: (view all by) ::: April 08, 2015, 05:01 PM:

Joshua @ 177 - the 10% of Plokta's rule is NOT the same as the 5% rule for getting stuff on the ballot. It's 10% of *the most popular item* which actually adds items to the final list.

#191 ::: Jon Lennox ::: (view all by) ::: April 08, 2015, 05:04 PM:

Josh@177: I don't think my suggestion is itself a solution to the problem. I do, however, think it is a useful prepatory step for deciding on a solution for the problem.

One of the concerns I think it's appropriate to have for any of these proposed balloting changes is, how do you beta-test them? Can you be sure that it won't break anything?

The existing nominations statistics don't give us enough information to know what effect algorithms like STV would have had in past years, in the absence of slate voting, or to simulate what adding a slate would do.

The point of this resolution is to make it clear that Worldcons are allowed to release the information we'd need to make these judgments.

#192 ::: heresiarch ::: (view all by) ::: April 08, 2015, 05:15 PM:

Reimer Behrends @ 124: "I note here that any time a single group controls 50% or more of the ballots, they can always collude to get at least one of their candidates nominated and to then ensure that the same candidate wins during the final vote (assuming a voting system that other than that collusion is impartial)."

Well, I should hope so.

Bruce Schneier @ 137: "So, to take the simplest case, let's call a voter's happiness 0 if none of his candidates win, and 1 if at least one of them wins. Maximizing the summation will maximize the number of voters who have at least one of their candidates on the final slate of nominees."

I'm skeptical of this direction for reasons of computational load--less on the system than on voters. (It looks like a traveling salesman problem to me, but as I am not a mathematician if others think the problem is calculable then I'll take their word for it.) Either way though, it seems to me that the ultimate outcome will depend a lot on fairly abstract weighting decisions (is the happiness of having two on the ballot 1.5 or 1.4?) that are difficult to explain, predict the impact of, or justify. Any system in which a clear frontrunner can lose (per @151) is going to be a tough sell.

Stephen Rochelle @ 148: ""Those two are basically the same" is not at all a reasonable assumption to make for the scenario you posit."

It is a reasonable assumption not in terms of content, but only in terms of "why this work was nominated"--and in that sense, they are quite similar. Rivest's system discourages correlation agnostic of whether it is driven by content or strategy.

#193 ::: JonW ::: (view all by) ::: April 08, 2015, 05:20 PM:

Chris @190 - DrPlokta's proposal is 10% (total noms – noms of first place finisher). So in the novel category this year, it would have dropped the threshold from 256 votes to (1827-387)/10 = 144 votes. That said, 10% may be too high; the proposal could be tweaked with a lower value.
Also, I agree with @188 that we shouldn't rule out an otherwise-attractive proposal simply because it incorporates ranked voting.

#194 ::: Cheradenine ::: (view all by) ::: April 08, 2015, 05:32 PM:

Jon @#191, that would be a good reason to incline toward a voting system that's seen more real-world use (e.g. Single Transferable Vote) since we'll at least have case studies of how they've worked in practice. We can do some work with simulations - generating a set of ballots that at least share the characteristics we know about of historical Hugo ballots - but won't be able to capture the data sets exactly, much less know the effect of changes in voting behavior. (The effect of this year's change in voting behavior caught a lot of people by surprise!)

Chris @#190, not quite - it's 10% of (the number of votes cast minus the number of votes for the most popular item). I stand by my argument that the variation in thresholds is low enough that just setting a universal threshold would be simpler and have essentially the same effect. Josh is right that the proposal wouldn't do much in highly dispersed categories - in 2013, only one short story was over 6%, and stories under 6% would only be included if the slate had more than 40% of the total votes.

Sam @#188, I have a strong hunch that ranked nomination ballots will reduce the number of nominations received (people already talk about having trouble deciding on nominations, having to order them is further cognitive load), but they certainly have strong advantages from a theoretical perspective. I'm not opposed to using a ranked method, but it might well be a harder sell at a WorldCon.

#195 ::: Jo ::: (view all by) ::: April 08, 2015, 05:36 PM:

Q. Pheevr @ 138: Yes exactly! & also: gaming ultimately more or less ineliminable, in some form? & also: it could be fun? (E.g. are the Rabid Puppies et al. having fun? ARE THEY HAVING MORE FUN?). & how to make the mechanics of preference aggregation something which stays interesting & important to a broad base, not just those who enjoy handling the math and/or those who bone up on voting systems when the current system throws up anomalies? & to be cutting edge, to be sf?

#196 ::: felice ::: (view all by) ::: April 08, 2015, 05:44 PM:

#67 looks good to me. I don't think sniping would be a big problem; publishing some information about the state of nominations before nominations close would increase participation and concentrate votes. It's much easier to look at a list and think "ok, I like the look of that, that, and that" than it is to answer "what were the best five short stories published last year". That means more nominations for the most popular works, and a higher bar to reach for block nominations. We don't need to know if there's a going to be a last minute slate or not for this to help counter it.

Rather than a weekly ranked top 15, I think a one-off unranked (alphabetical or random) top 15 a week before the deadline would be sufficient information to achieve the desired benefits. It's basically equivalent to releasing a longlist, though it's possible for a work not on the longlist to get on the shortlist if it gets a late surge in nominations.

#197 ::: Cat ::: (view all by) ::: April 08, 2015, 05:45 PM:

I have been reading with interest. I am apparently pretty late to the party because I didn't realize that I should regularly enter Making Light at the main page to be apprised of such new developments. My bad, now I know.

The major problem I see with making the number of nominations smaller than the final number of nominees is that that works until you have two slates. We already have two slates, the Sad Puppies and the Rabid Puppies. It's too late for that.

I have been watching the suggestions about having ballots count for less after one or two of the nominees on them have been put on the ballot, and am still working on understanding those.

In the meantime, has anyone considered Django Wexler's anti-vote proposal?

An open slate like the Sad Or Rabid Puppies would tend to attract anti-votes from annoyed fen. I suspect a "secret slate" would be hard to make workable, because if you enlist enough people to put stuff on the ballot, the secret gets out, and attracts anti-votes from annoyed fen, and if you don't enlist enough people I don't particularly care if you collude.

There might be attempts to game this in anti-slates directed at popular works like Ancillary Justice or betes noires like John Scalzi, but I would expect, to be public enough to make a difference, they'd have to be public enough that fen with a spare nomination or two (since many Hugo Nominators appear to me not to nominate a full slate) might sprinkle them generously over the anti-nomination slate.

So far this seems to me to be the best bet for making slate-making behavior counterproductive.

What do people think?

#198 ::: C. Wingate ::: (view all by) ::: April 08, 2015, 05:52 PM:

The issue I have with any attempt to solve this with a voting system is that targetting the symptoms of slate voting necessarily targets anything else that produces similar symptoms, including any group of people who (through common interest, for example) produce a strongly similar pattern of voting. It also seems to me that in any system there will come a point at which a large enough bloc is going to overwhelm the voting no matter what (Best Novella shows distinct signs of that this year).

#199 ::: felice ::: (view all by) ::: April 08, 2015, 06:06 PM:

heresiarch @ 88: "I feel it can be explained very simply. "You get to nominate five choices, but once one of your choices makes it on the ballot then the rest are discarded." It even, I think, seems fair: isn't getting one of your picks enough?"

What if everyone has the same first choice? Then there are no ballots left to fill the other places. Not a particularly likely scenario, but I think it illustrates a fundamental flaw. (And being every nominator's first choice doesn't mean it would automatically win the actual vote, because there's a bigger pool of voters, and people may change their opinions after reading shortlisted works they were unfamiliar with at the time they nominated.)

STV with transfer of fractional surplus votes seems like the best option to me. If everyone has the same first choice, 80% of their vote is transferred to their second choices. I'd make ranking your nominations optional, and assign a random ranking to unranked works for purposes of counting the nominations. Eg on the sample ballot below, Good Book and Another Good Book would be randomly allocated the 3rd and 4th positions.

[1] Best Book
[2] Second Best Book
[ ] Good Book
[ ] Another Good Book
[ ]

#200 ::: kimiko ::: (view all by) ::: April 08, 2015, 06:08 PM:

#184, Brad from Sunnyvale,

I want to reiterate an important point, one definitely not understood by the bloc voters but also by many others.

I believe the Hugos are not, in spite of the trappings of voting, an election. They are a measurement. The goal of the measurement is to provide the best estimate of the most well regarded work by Worldcon fans.

This is what I was thinking. Regardless of how many or few nomination slots are present, I'd want to see the distribution of nominated works. That tells me what the electorate is really thinking. I want to see the measurement. I want to see if people are manipulating the data.

Picking a winner is something different.

Maybe I should put quotes around "winner?" What is the meaning of winning, if it is not a contest where you defeat the other person, but rather, a measure of what share of fandom thinks you are important?

What if we gave awards to anything in the first standard deviation of the data? Wouldn't that be an important win? If everyone agrees that A and B are super important, and C through Z are so far down the tail that they aren't even in the same league, shouldn't the desired outcome be that A and B get recognition?

To get that kind of measurement, we'd have to tweak the v/o/t/i/n/g/ survey method in some way. More slots than can conveniently be manipulated might help. More votes than slots might help too?

#201 ::: DavidR ::: (view all by) ::: April 08, 2015, 06:31 PM:

Given a procedure which is repeated each year, you can also carry information about the ballot forward from one year to the next. One idea that I haven't seen mentioned above would be to make the rule that any work that gets more nominations than last year's winner in that category gets on to the shortlist, even if that means a longer shortlist.

#202 ::: JonW ::: (view all by) ::: April 08, 2015, 06:39 PM:

Felice@199: Actually, under the voting system in #1, it only takes every ballot listing the same work as one of its N choices to eliminate all ballots after the first round.

#203 ::: soru ::: (view all by) ::: April 08, 2015, 07:53 PM:

What's the general feeling on nomination procedures that are a mathematical black box?

I'm thinking something that takes in all the raw votes, and uses statistical information theory/big data/machine learning techniques to generate the _most interesting_ set of nominations in some completely opaque way.

Obviously, it means you completely lose the ability to count nominations by hand. But the source code can be published on github and the results reproduced by anyone who cares to. So it would be impossible to subvert the process without changing the votes going in (which would of course break any system).

And its not like more than 1 person in a 1000 correctly understands the details of any voting system more complex than FPTP. So everyone not understanding it equally is a strike against elitism.

The Hugos are, after all, supposed to cover science fiction. So why not just put a robot in charge?

#204 ::: P J Evans ::: (view all by) ::: April 08, 2015, 07:58 PM:

203
When nominations are handwritten, you need a human to read and make sense of them. Humans don't do well at following instructions.

#205 ::: Steve Halter ::: (view all by) ::: April 08, 2015, 08:01 PM:

Brad @184 & abi@186:I also like that approach of framing this as a measurement rather than a vote. In that case, the nomination presents us with information and a slate attempts to obscure that information.
We can think of the nomination data as the signal and the slate as the noise.

#206 ::: Lee ::: (view all by) ::: April 08, 2015, 08:16 PM:

Brad, #184: That's a very good distillation.

Buddha Buck, #189: Now I'm trying to figure out how to turn that thought into an "Editing Workshop" at a con! (And being severely handicapped by not being an editor and so having only the dimmest idea of the process.)

#207 ::: Reimer Behrends ::: (view all by) ::: April 08, 2015, 08:18 PM:

Joshua @ 177: That's also a system I've played around with, following Bruce Schneier's suggestion to focus on STV systems. Its behavior seems to be similar to the weighted version of Ron Rivest's proposal or cumulative voting with elimination in that it condenses ballots into a small subset of the most popular nominees. As long as there are still multiple candidates on a ballot that haven't been eliminated (whether because it's a slate or because votes are randomly assigned to semipopular candidates), their weight is much lower compared to ballots that have been reduced to 1-2 popular candidates. Finally, it also doesn't need tiebreakers.

I've run a small simulation and the effect seems to be that even a heavily stacked slate appears to have a hard time overwhelming voters voting normally (unless the votes are very evenly and randomly scattered).

A nice feature of this approach is that it's very easy to explain and intuitive (vote off the least popular candidate, but preserve the weight of their votes by given them to remaining candidates on the same ballots). Another nice feature is that it's not purposely punishing block voting; it simply focuses the votes and should actually also produce sensible results in settings where block voting is unproblematic or even encouraged.

heresiarch @ 192: I was not making a value judgement, but describing a mathematical property.

#208 ::: Farseer2 ::: (view all by) ::: April 08, 2015, 08:26 PM:

@197: Django Wexler's anti-vote proposal is a bad idea in my opinion. It would be used to get rid of these block-voters, but it would also be used by extremists of both sides to blacklist works or authors they dislike. People would organize campaigns and countercampaigns and it would become a complete political mess. Please, no negative votes or anything else that encourages or rewards political campaigning.

#209 ::: Doctor Science ::: (view all by) ::: April 08, 2015, 08:37 PM:

#184, Brad from Sunnyvale,

I believe the Hugos are not, in spite of the trappings of voting, an election. They are a measurement. The goal of the measurement is to provide the best estimate of the most well regarded work by Worldcon fans

I think you're almost right, but not quite -- and the difference is probably crucial.

The goal is "to provide the best estimate of the work that would be most well regarded by Worldcon fans, if they all read every work -- which is impossible."

This is why the nomination process and the final "vote" are fundamentally different, and reflect the biggest unwritten rule the Puppies have broken.

The purpose of the nomination round is to collect works that have already been well-liked by a subset of Worldcon fans, and which thus might be liked by the rest. The rule the Puppies have broken is: you don't nominate a work you haven't read (or seen). I will eat John C. Wright's goddamn hat if all the Puppies voters have read all the stories they voted for -- among other things, too many of them are difficult to track down, especially at short notice.

Anyway, once we have a pile of nominations, most of us use the time between the ballot announcement and the deadline actually *reading* the works that we hadn't known about or had time for in the course of the year.

This is why, if you compare the nominations results to the voting results, the winning work is frequently not the one that got the most nominations. Sometimes it is, but quite often the work that people turned out to like best is one they didn't get a chance to read before the voting period.

Because of this, the nomination stage isn't really estimating a fact, because "the work best-liked among the ones we've all read" is a null set until we have the reading list. What we want to get out of the nominations is something like the Locus Recommended Reading List.

The LRRL is especially useful because the Locus staff divide the work of reading among them, so they cover a much higher proportion of the total published output than does a random group of the same size (or even an order of magnitude higher). Even so, they admit that there are huge swathes of the genre that they cover poorly or not at all, and which they don't really know how to whittle down: paranormal romance, for instance, and all kinds of ebooks.

What do those of you who are experts in surveys and/or voting think is the way to deal with this sort of problem? Basically, the nomination round problem is that the number of possible candidates is too large for any single voter to deal with, so what an individual nominates is based both on what they like, but also on what they happen to have run across.

As Cat has been pointing out, the other unwritten rule the Puppies have broken is the one about making your own, idiosyncratic decisions. The Puppy slates greatly narrowed the choices of all Puppy voters: most of the works & people on their slates were put there due to a handful of commenters, and largely, I suspect, by Torgerson, VD, and a few of their friends.

They didn't even have a real "primary", just comments & suggestions (as far as I can tell). The slates would still be a problem if 200-300 had their own pre-vote and agreed to all vote for the winners, but that would at least reflect a set of individual choices.

#210 ::: Kevin Standlee ::: (view all by) ::: April 08, 2015, 08:40 PM:

Andy Brazil @160:

How easy is it to game the business meeting?
Define "game." Attendance is typically between 100 and 200, usually at the lower end. The largest one I personally remember was ~350 (10% of the attendance, which is huge) in Winnipeg in 1994, due to multiple non-overlapping hot-button issues with different constituencies turning up to fight.

The very fact that constitutional changes require two years works against packing single meetings. Few people can keep up the enthusiasm for two years, and getting enough people to the second meeting, particularly when the word gets out about what's going on and a counter-force mobilizes. (See: Semiprozine, which was one vote away from being removed as a category, but the act of first passage mobilized a bunch of semi-pro publishers and the category morphed instead.)

Which, if I were so inclined, I would do as a long term plan starting with seemingly innocent and deeply technical rule changes.
If you're thinking of the common Congressional trick of taking a moribund bill, deleting the entire bill and substituting an unrelated bill in it, that trick won't work in WSFS. 1. Unlike the US Congress, amendments have to be germane to the thing they're amending. 2. Anything that increases the scope of an amendment resets its ratification clock, so it has to hold over another year.

PJ Evans @204:

Your observation leads into why I'm allergic to trying to apply any form of STV/IRV/Condorcet, etc. to the nominating phase: Too Much Load on the Administrator. We already put a whole lot of work on the people administering the ballot, who have to figure out what the various permutations of names boil down to. Paul Cornell lost a finalist nod in 2007 because the administrators didn't realize that the three different ways people were nominating his work were actually all the same work. Paul was a really good guy and understood there wasn't anything to be done on account of the error didn't come to light until the nomination details were published post-ceremony. Adding a preferential ballot to that makes things even worse IMO.

The worrisome thing here is that in the currently charged environment, even innocent technical errors are being used as a sign of a Deep Conspiracy. I left the "mailto:" tag off the address of the Hugo Administration Subcommittee on the finalist announcement post at the Hugo Awards Web site and was accused of nefarious schemes because the so called open door led to an invalid address. You've got a lot of people ready to assume malice where human fallibility is much more plausible.

Could it possibly be that Doing Nothing (that is, not fiddling the rules at all) and encouraging every member to vote their conscience (including No Awarding anything that they don't think should have been on the ballot in the first place) is sufficient? How many times would the Bloc have to sweep nominations and then lose everything in the final round before they get tired and Go Away?

#211 ::: Rail ::: (view all by) ::: April 08, 2015, 08:53 PM:

Kevin Standlee@210: "How many times would the Bloc have to sweep nominations and then lose everything in the final round before they get tired and Go Away?"

A lot, since VD has said publicly that his intent is to destroy the Hugos.

#212 ::: Doctor Science ::: (view all by) ::: April 08, 2015, 08:53 PM:

Kevin Standlee @ 210:

How many times would the Bloc have to sweep nominations and then lose everything in the final round before they get tired and Go Away?

Never. We have to assume that they will never go away. At the very least, it's foolish to think that VD will go away voluntarily: the dude is *obsessed*.

#213 ::: nathanbp ::: (view all by) ::: April 08, 2015, 08:53 PM:

felice, #199, others: Bruce's reference in #120 describes a way to run STV without ranked candidates (or with some at the same rank). In short you divide a vote for 2 works at equal rank into 2 votes each worth 0.5.

My primary objection to STV is that it makes things fairly easy for slates. The constructors of a slate just instruct people to rank all 5 candidates in random order, and then they'll pick up some number of spots on the ballot (possibly even all 5 in well spread out categories like Short Story). On the other hand, discarding ballots once they've elected one nominee seems much more difficult to game (if I'm missing an obvious strategy please point it out). Obviously if all participants in a slate list the same 5 works, only one will make the final ballot. So then to capture more of the ballot they'd have to split votes, but figuring out how to split votes and still get on the ballot requires guessing correctly how the final ballot will turn out. This is as opposed to STV which performs the splitting for the slate voters automatically.

#214 ::: Pandora's Lox ::: (view all by) ::: April 08, 2015, 09:02 PM:

Wouldn't multiple slates be the simplest solution to this problem? Two dueling slates would be awful, but once you have four or five factions, they would just cancel each other out. Especially if the number of nominations per category were reduced to three (or whatever).

#215 ::: Lizzy L ::: (view all by) ::: April 08, 2015, 09:07 PM:

If the SPs want Hugos that badly, my advice is -- Give them Hugos.

I teach martial arts. Every once in a while I get someone who'll come in the door and ask, "How long will it take me to get a black belt?"

I have been known to walk into my office, walk out with a fresh never worn black obi, which I then hand the questioner. (Sometimes I don't.) But in this case... The SPs want Hugos. Fine. Spend the extra money, make statues, put little tags on them, take pictures, and FedEx the statues to Wright, Day, and everyone else on the SP's nominating slate. Everyone. Do it before the convention. Announce what you've done. Don't even have a vote. Throw a big party at the convention on the night which was supposed to be award night. Dance your asses off.

#216 ::: felice ::: (view all by) ::: April 08, 2015, 09:09 PM:

nathanbp @ 213:

Dividing votes is an incentive to only nominate one work, which isn't desirable.

With STV, slates would need five times as many nominators as any other single work to sweep the entire ballot, compared to now where they just need one more nominator; I don't think even the short story category would give them that much of an advantage. It wouldn't stop a slate getting one or two nominations on the ballot, but that's democracy for you; if they're a significant portion of the membership, there's no legitimate reason to exclude their preferences entirely.

And if STV is supplemented by early publication of nominations, there's an opportunity to reduce the spread by letting people nominate works they know are in with a chance.

#217 ::: felice ::: (view all by) ::: April 08, 2015, 09:12 PM:

Lizzy L @ 215:

Ooh, that would open up interesting possibilities for base design!

#218 ::: nathanbp ::: (view all by) ::: April 08, 2015, 09:26 PM:

felice, #216: The votes are divided yes, but it's still STV, so your votes condense back as lower ranked candidates are eliminated. E.g. your ballot is Popular Work + 4 unpopular works. Initially your vote is split 5 ways at 0.2 each, but after the 4 unpopular works are eliminated, it becomes just a single vote for Popular Work.

Also, with STV, they don't need 5 times as many voters, just 5 times as many at the votes for the most popular candidate.

#219 ::: JonW ::: (view all by) ::: April 08, 2015, 09:27 PM:

If we applied STV to the 2013 short story category, and if I’m understanding it correctly, then a hypothetical slate consisting of 20% of the nominators would have swept three slots (since only one story that year got more than 6% of the nominations). That’s a point for nathanbp@#213. On the other hand, consider the 2013 best dramatic presentation, long form category. Applying the proposal in #1 might have led to some odd results, since The Avengers appeared on 49% of the nomination ballots, and discarding half of the ballots after the first round would have meant that four of the five nominations would have been decided after (non-randomly) eliminating half of the nominating pool based on whether they liked The Avengers.

#220 ::: DavidS ::: (view all by) ::: April 08, 2015, 09:29 PM:

@137 "[L]et's call a voter's happiness 0 if none of his candidates win, and 1 if at least one of them wins. Maximizing the summation will maximize the number of voters who have at least one of their candidates on the final slate of nominees."

In theory, this is a very hard problem, namely the maximum coverage problem, which is NP-Hard. To see this, imagining making a list, for each candidate, of all voters who chose that candidate. Your job is to find a collection of m (currently 5) lists which cover as many voters as possible.

Although it is NP-Hard, Wikipedia notes that the greedy approach suggested in Bruce's first post always comes within 1-1/e=0.62 of the optimum. It might be fun to run some simulations and see what happens with actual nomination lists.

What bothers me about this system is that it makes strategic voting really mind bending. I need to think "what work do I like best, which I don't expect a lot of people to pick, but I do expect some other people to pick?". In some ways, this is a good question to be asking as it adds diversity, but it is so much harder to think about than "what work do I like best?". Perhaps some people, faced with such confusion, give up and vote naively, but I know that I would spend a long time trying to figure out the strategy, and most likely get it wrong.

#221 ::: Tammy Coxen ::: (view all by) ::: April 08, 2015, 09:35 PM:

Everyone's been focusing on the x = 5, y = 1, z = 5 case, but at the end of #1 Bruce discusses the possibility of something like x=5, y=2, z=5 - which it seems would help mitigate the Avengers problem stated above (at the cost of probably allowing slightly more slated works onto the ballot).

#222 ::: P J Evans ::: (view all by) ::: April 08, 2015, 09:41 PM:

217
There's no requirement that the rockets be identical to the usual ones, is there?
Even more possibilities!

(One year they used transparent colorless acrylic. It didn't hold up really well in terms of color and transparency, but I think they're still around. I think shit-brown with an iridescent finish might be interesting.)

#223 ::: John A Arkansawyer ::: (view all by) ::: April 08, 2015, 09:48 PM:

Jo Walton @ 176: "There is a tendency to consider one's enemies as much more powerful and scary and organized than they are. Don't sell yourself to fear in your mind."

That is one of the wisest things I've ever heard. I've tried to say something like it over and over and never had it quite right. This is a little gem which is not quite too long for a t-shirt.

(My newest t-shirt comes from this song and says "We don't need old rich white men to tell us who we can kiss goodnight.")

#224 ::: nathanbp ::: (view all by) ::: April 08, 2015, 09:54 PM:

JonW, #219: Those both sound accurate to me. One thing I'd like to point out though, is that dropping 49% of the ballots would only change the results in the case that voting for the Avengers is highly correlated with voting for one of the other nominees. For example, if all of the voters for The Hobbit had also listed the Avengers, then obviously The Hobbit would drop to 0 votes and not be nominated. On the other hand, if votes are not well correlated, then removing 49% of the ballots just drops everyone's totals by 49%, which has no effect on the end result. Now, the reality is probably somewhere in the middle, and unfortunately without a set of ballots to examine we can't tell how correlated the ballots tend to be. And it may vary by category.

Tammy Coxen, #221: You could also weight further votes less instead of eliminating ballots entirely. For example, once one work on your ballot has been chosen, your votes for the less only count for 1/2 a vote, and then 1/4 after that (if y >= 3), etc. I'm not sure how the 2 systems compare. I feel like 2 nominees out of 5 is a lot to lose to a slate though.

On somewhat of a side note but still somewhat related to voting systems, I had an amusing thought for if a system that requires a coin flip or other random choice is chosen. To be as transparent as possible, a set of random numbers could be generated publicly (by say, rolling a D100) at the preceding Worldcon and then published. Then whenever a random choice is required, these pregenerated numbers could be applied in a deterministic way.

#225 ::: P J Evans ::: (view all by) ::: April 08, 2015, 10:06 PM:

224
You also can find files online with a million digits (or more!) of pi (or e). Those should be sufficiently random. They're certain large enough for most purposes.

#226 ::: John A Arkansawyer ::: (view all by) ::: April 08, 2015, 10:06 PM:

I'll tell you one thing: If I were running a Worldcon, I wouldn't be in any hurry whatsofuckingever to start taking BitCoin. I would not even go near it.

soru @ 203, nathanbp @ 224: I got an idea which kind of combines your thoughts:

Find as many good voting methods, preferably ones which can be parameterized, like the one from Ron Rivest which Bruce presents @ 1, which are computationally feasible, reasonably understandable, disadvantage slates under most circumstances, and (this is my personal thought) provide a diverse (in the sense in which the SP/RP folks claim to mean) set of nominees.

Then, after the nominations are all in, hire someone impartial to publicly, on streaming video, roll dice to pick the one being used that year, and roll them again to set the parameters.

Make this a BIG GODDAM EVENT. This is the sort of thing which can provide great publicity. It'd be playful and a lot of fun. And NO ONE can successfully game an unknown system effectively.

This combines the "black box" idea of soru and the randomness of nathanbp.

Sorry to not talk about STV, Bruce. ;-)

#227 ::: John A Arkansawyer ::: (view all by) ::: April 08, 2015, 10:09 PM:

Mongoose @ 183: Shit. I really didn't want this to be called that. I lived through Nixon.

#228 ::: DavidS ::: (view all by) ::: April 08, 2015, 10:41 PM:

Let me toss out an idea, somewhat of a mix of Plokta's and SAV. We choose a number, let's say 50%, of voters which we will attempt to satisfy. Then we add works to the final ballot in simple order of number of nominations, until at least 50% of voters have one work on the final ballot. (There should also be a rule that at least some number of works, say 4, always make it to the final ballot, in case there is one work which everyone nominates.)

Thus, if there is a slate, everything on the slate gets on the ballot but, since this doesn't satisfy the non-slate voters, more works get added. This is the same effect as Plokta's rule.

However, unlike Plokta's rule, having a very popular work does not lead to a huge expansion of the category. It also seems to me to be easier to explain. Possible downside -- it can create an incentive to nominate fewer works, in order to be harder to satisfy. But it isn't a huge incentive, since the hardest 50% to satisfy just won't be satisfied.

PS Why 50%? First of all, it is pleasantly round. But I actually obtained it in a different way. I looked at the Best Novel nominations in 2011 and 2012. (last pre-canine years). I then tried to estimate the fraction of people none of whose nominations made the final ballot by multiplying together the fraction of people who, for each finalist, failed to nominate it. So I computed (1 - 0.1827) (1 - 0.1701) (1 - 0.1357) (1 - 0.0845) (1 - 0.0741)=0.4969 and (1 - 0.1501) (1 - 0.1297) (1 - 0.1224) (1 - 0.0996) (1 - 0.0936) = 0.5298. This isn't the exact fraction who got no nominee on the ballot, but the fact that I got roughly the same number both times made me think I was in the ballpark.

#229 ::: DavidS ::: (view all by) ::: April 08, 2015, 10:48 PM:

Since I'm posting a bunch, I should probably introduce myself. I am a long time lurker and occasional poster. I enjoy reading and discussing SF but am not a congoer. I am a mathematician and, while voting systems aren't my professional area, I've always found them interesting. No poetry, I'm sorry to say.

#230 ::: Emily H. ::: (view all by) ::: April 08, 2015, 11:15 PM:

DavidS, the pitfall I see there is that it seems easy for a single very popular work to throw things off -- in both this and the system in #1, it creates an incentive not to vote for a thing that you suspect will be extremely popular. That might not be a bad thing.

Imagine 10 slate voters for ABCDE, and 50 non-slate votes of which 20 nominate A (The Avengers, let's say) and a bunch of scattered works -- 11 votes for F, 9 for G, 8 for H, and so forth. As soon as you pick A as one of your nominees, 50% of the voters are satisfied -- so it comes down to A, F, and three of BCDE. Which doesn't do much to dilute the slate. Is that an unlikely situation? I don't know; I suspect it depends on the category.

#231 ::: Brad from Sunnyvale ::: (view all by) ::: April 08, 2015, 11:39 PM:

Dr. Science in #209, you are correct. We do seek what the aggregate opinion would be if everybody read everything. This is one of the reason that award nomination juries are popular. The juries are appointed with the expectation to read a lot, and to read other jurors' suggestions to produce a ballot which is very likely to contain the best work, and small enough that the whole group has a chance of evaluating all of it. (Though we don't actually get there, either.)

The current nomination ballot is sort of a variant of the Approval voting system, and I would dare say that has one advantage over some of the models proposed, including Bruce's "maximize happiness" approach. It is very easy to understand. Even easier than STV/IRV which routinely confounds Hugo voters.

I do feel that the simplicity of the system is an important requirement. While it is not necessary that the nominators or voters understand the algorithm, they have much more confidence in it when they do. This is yet another reason that I prefer Condorcet to STV -- One can explain Condorcet reasonably well in a few sentences. ("For every pair of nominees, we compare them to see if more people put A ahead of B or B ahead of A. The winner is the one who beats everybody else in a head to head. If the rare case that doesn't happen, we have a tie (which takes slightly longer to explain.")

Anyway, to sum up some of the important constraints:

a) The system can't generate more nominees than most of our group can practically evaluate.
b) The system must be reasonably easy to explain
c) The system should be mostly immune to strategic voting
d) The system must be immune to collusion between nominators, particularly small minorities of nominators.
e) The system should be extremely likely to find the correct eventual winner (the book most would choose if we all read them all) within its set of nominees, and the other nominees should be among the best.
f) The system must be possible to explain to a WSFS business meeting and have them see the wisdom of it. Twice!
g) Related to this, the system must be fair and appear fair, with no party having any clear ability to bias it.

Bruce's proposal goes after an interesting higher goal, namely, "The system should provide the least disappointment to nominators as a whole."


All of these are a tall order.

#232 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 08, 2015, 11:50 PM:

Hello all, we're shutting down comments for the night in ten minutes or so. See you all bright and early tomorrow morning.

#233 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:08 AM:

I am seeing several ideas worth discussing:

1. Increasing the total number of nominees in a category, either statically or dynamically based on the voting. This has the benefit of denying blocs the ability to sweep, but has the problem of there being too many nominees on the final ballot. IRV breaks down when people don't rank every candidate.

2. Decrease the number of nominees each person can nominate to something like 2 or 3. This also limits the ability of blocs to sweep. The only downside I can think of is that the well-read electorate can't nominate everyone they think worthy.

3. Change the voting system to use some sort of proportional representation system, either single transferrable vote, satisfaction approval voting, and so on.

BTW, here's an IRV system with equal preferences.

These are not mutually exclusive, of course. Way back in @1, I tried to parametrize this: "In this system, there are three parameters: A voter can vote for x candidates. A ballot is counted until y of those candidates win. We need z winners total."

Right now we have an x = 5, y = 5, z =5 system.

Cheradenine @173: "I suspect that [all three] approaches would probably be effective at preventing this year's particular failure state (a set of works supported by a minority completely dominating the ballot). It's worth considering the possible side effects of each in general, as well as the specific implementations."

Yes. This is very important.

I am also seeing several different goals in the voting system:

1. It should encourage people to nominate. To me, this implies no ranking.

2. It should be fair, whatever we decide that means.

3. It should be easy to explain.

4. It should be easy to administer.

5. It should be difficult for a voting bloc to co-opt. To me, this implies that it should encourage diversity.

Anything else?

Those are our discussion topics for today.

#234 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:09 AM:

Reimer Behrends @124:

This kind of idea [from @1] is definitely worth thinking about, It's along the lines of a single transferable vote, where the "excess" votes from one candidate get transferred to another.

Cheradenine @134:

"Reimer Behrends @124: I think the procedure you describe (or a slight variation on it) has been called "sequential proportional approval voting" (whew) and I was just going to post about it, since it seems like the closest approach to STV that's applicable and practical for unranked lists. I want to do some more research and experimentation with it. Some versions seem to favor cloning, which is bad news if we're trying to discourage slates, but I don't know how this approach would fare. I may not be able to work on this further until this evening, though (darn work)."

Can you post a link to a paper? I'd like to read about it. The best I found by Googling is this.

I like this, and it seems to converge with my "maximize satisfaction across the electorate" idea -- which is Satisfaction Approval Voting -- @137.

Buddha Buck @140:

"It seems to me that the major problem with that approach [satisfaction approval voting] is efficiency. It looks to be O(n^z) (where n is the total number of unique potential nominees appearing on ballots, and z is as defined earlier, the final number of nominees). There may be a more efficient way of optimizing this than the naive approach of trying all (n choose z) combinations of nominees, but I don't see it offhand.

"In the simplest case (happiness is at least one successful nomination), it seems equivalent to the (x=5,y=1,z=5) case discussed earlier, but I only have a strong hunch, not a proof, that that's the case."

Yes, it's going to be a computational problem to compute the final slate, but it doesn't feel intractable. Surely a computer can do this in a reasonable time.

Cheradenine @142:

"Bruce @#137: Satisfaction Approval Voting uses this basic approach (with a satisfaction rating of m/n, where m is the number of a voter's candidates elected and n is the number they listed). It's described in the paper at: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1608051 Per the authors, it has a few properties that have come up as desirable in this conversation: it's clone-inhibiting (which disincentivizes slates), and it tends to yield better representation of diverse constituencies. I was worried about computational complexity with a lot of nominees, but it looks like, at least with their formula, it can be computed in polynomial time by computing a score for each candidate and selecting the most satisfying candidates. With a different formula (like that in #137) it might take a lot longer if we need to compute satisfaction over all possible selections of candidates."

Thanks for this. I consider this system one of the ones worth continuing to think about.

Steve Halter @182:

"Cheradenine@142:That Satisfaction Approval Voting paper was quite interesting and does seem to display many of the characteristics we want here. In addition, it is fairly easily computable."

Yes.

dh @153: On satisfaction approval voting: "That smells like a recipe for disaster and disorder. In a binary election like US Presidential elections, you have (usually) a close race, and in the end, a small majority get EXACTLY what they want - their candidate wins. The minority gets nothing of what they want. This is stable at least because a lot of people get what they want. In this system, it seems like everyone will get (in your example) either 0, 20, or maybe 40% of what they want. So beyond having a system which is pretty hard to audit and explain, you also have a system where more people get less of what they want. The only upside is more people will get a non-zero amount of what they want."

I think this is actually okay, because this is a nominating election. Were it the final election, I would agree more with you.

DavidS @220:

"What bothers me about [satistaction approval voting] is that it makes strategic voting really mind bending. I need to think 'what work do I like best, which I don't expect a lot of people to pick, but I do expect some other people to pick?'. In some ways, this is a good question to be asking as it adds diversity, but it is so much harder to think about than 'what work do I like best?'. Perhaps some people, faced with such confusion, give up and vote naively, but I know that I would spend a long time trying to figure out the strategy, and most likely get it wrong."

This feels like a good thing.

#235 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:09 AM:

nathanbp @131:

"After thinking about it overnight and reading Bruce's references, I'm no longer such a fan of STV for picking the nominees, and I think actually a system more like Bruce's in comment #1 would be a better solution. A primary part of STV is that 'wasted' votes in excess of those needed to elect a candidate are transferred to other candidates on the ballots of those voting for the winning candidate. However, if the goal of the Hugo nomination process is to pick a diverse set of nominees, then it's better to just eliminate the ballots of those voting for the winner entirely. This reduces the power of slate voting and also reduces the power of non-slate factions within the voters. Basically the message Bruce's system in #1 says is 'You should be happy one of your choices made the final ballot, now let everyone else have a chance.'"

I am coming around to this, too. I really need to sit down and figure out how STV -- or the scheme above -- works without ranking. But, yes, what you're saying makes sense to me.


nathanbp @213:

"My primary objection to STV is that it makes things fairly easy for slates."

This is worth talking about. STV makes voting blocs less powerful. A bloc will get one candidate on the final ballot easily, but I don't think it's possible to prevent that. What we want is for a bloc to have trouble getting many candidates on the final ballot.

#236 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:10 AM:

Cheradenine @194:

"Jon @#191, that would be a good reason to incline toward a voting system that's seen more real-world use (e.g. Single Transferable Vote) since we'll at least have case studies of how they've worked in practice. We can do some work with simulations - generating a set of ballots that at least share the characteristics we know about of historical Hugo ballots - but won't be able to capture the data sets exactly, much less know the effect of changes in voting behavior. (The effect of this year's change in voting behavior caught a lot of people by surprise!)"

Yes. Very much so.

#237 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:10 AM:

Brad from Synnyvale @184:

"I believe the Hugos are not, in spite of the trappings of voting, an election. They are a measurement. The goal of the measurement is to provide the best estimate of the most well regarded work by Worldcon fans."

This is a really interesting way of looking at things. It also explains what went wrong this year. The voting bloc treated it as an election. As their goals were orthogonal to the goals of the nomination election, they were able to subvert the process.

#238 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:11 AM:

julian @125:

"My proposal is that ballots are not counted in the 5% eligibility calculation at all if they are substantially similar to at least 3% of the other ballots in their category. (So, if you have 1100 ballots, and 100 of them are substantially similar, then works have to have at least 50 nominations on the other ballots to be eligible.)"

My worry about schemes like this is that they are arbitrary, and tied to the details of this year's particular tactics. I'd rather a solution that's more robust against future collusion.

Joshua Kronengold @180:

"Peter@179: While we should look at whatever we do not being trivially breakable, I don't think we need to solve problems we don't have yet."

I think we definitely do need to solve problems we don't have yet. We don't want to be back here again in a few years.

Sam M-B @171: "Via Jason Sanford, 'A modest Hugo Award proposal: [I]nstead of simply tweaking the Hugo rules, perhaps a better solution is to find common ground and agree to fix the Hugo Awards once and for all.' http://www.jasonsanford.com/blog/2015/4/a-modest-hugo-award-proposal The proposal being Condorcet count with proportional representation: 'Proportional representation means basically that if 60% of ballots are A-B-C-D-E and 40% of ballots are F-G-H-I-J that the 5 nominees are A-B-C-F-G. This is what I actually favor: minority representation is important no matter which "side" one might be on.'" http://civs.cs.cornell.edu/proportional.html

For those who don't know what a Condorcet count, it's worth reading the final link. I don't think it works well for a nominations ballot -- I want the election to minimize the power of bloc voting rather than moderate between different blocs, and I don't want ranking -- but I'd like to hear from others on this.

#239 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:11 AM:

Hmm @72:

"In the method described in #1, are there any less arbitrary options for resolving ties than flipping coins?"

Anything works, as long as it's random. In 2008 a tie in Nevada was resolved by a hand of poker.

Buddha Buck #189:

"But it would be interesting to have an edit-off: 'OK, you both have 4 hours. You each have a pile of identical unopened slush. You need to put together a 50-page magazine. You will be judged on quality of stories you choose, how well you hit the 50-page mark, and overall unified theme. Ready? Steady? Go!'"

Yes, there are lots of ways to have fun with this.

#240 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:11 AM:

Sam M-B @188:

"I have to admit to not understanding the allergy to ranked ballots in the nomination round. We rank ballots in the final round, and adding the number 1-5 in the nomination round is hardly a 'burden' is it? (Especially when using a voting method which is resistant to strategic ranking, meaning you don't have to double-think your rankings. And it's even less cumbersome than in the final round because you don't have to squirrel around 'No Award' tactics. And, with Condorcet counts of which I am aware, I can vote everything the same rank, or unranked, if I feel it's not worth my time to rank them, and that's fine. People who want to rank, can rank, people who don't, don't have to, in the nomination round.)"

I don't like it because 1) it's an additional barrier to voting, and 2) it violates the spirit of the nominations election. This is not a final vote, it's a nomination. It's people saying: "Out of all the books published in 2014, I think these deserve to be on the ballot." One of the things we do is encourage more people to nominate, and making it simple aids in that.

#241 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:12 AM:

Cat @197 "In the meantime, has anyone considered Django Wexler's anti-vote proposal?"

I've thought about it. I don't like it for several reasons. One, it's weirdly complicated -- which means it'll have all sorts of unforseen consequences. Two, it enshrines the battle of the blocs. And three, it gives additional power of blocs to punish people and works they don't like.

I just don't see most of the electorate rising up against the slates by anti-voting them. Sure, we on this thread will. But most people won't. So it doesn't make slate behavior counterproductive, it makes it more powerful.

#242 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 07:12 AM:

Pandora's Lox @213:

"Wouldn't multiple slates be the simplest solution to this problem?"

Yes, of course. The best defense against a political party is a rival political party. But there is a strong belief here that parties are, in general, bad. I wrote about this in the initial post on this thread.

#243 ::: John A Arkansawyer ::: (view all by) ::: April 09, 2015, 07:43 AM:

IANAModerator, but when a world-class expert drops by and says "Those are our discussion topics for today", I personally find discussions more fruitful when we stay in those topics' vague areas.

#244 ::: rahaeli ::: (view all by) ::: April 09, 2015, 08:02 AM:

Kevin Standlee @ 210:

We already put a whole lot of work on the people administering the ballot, who have to figure out what the various permutations of names boil down to. Paul Cornell lost a finalist nod in 2007 because the administrators didn't realize that the three different ways people were nominating his work were actually all the same work.

Not having ever nominated before, I don't know for certain how the process works, but if I'm reading this correctly, you're doing noms by giving people simple uncorrected/non-normed text boxes to type into, which the administrators then have to standardize by hand for every single ballot? Yeesh, no wonder you keep talking about administrator load!

This is a great opportunity for five hours of programming time now to save you thousands of hours of work in the future. When someone begins typing a nomination, the system should try to figure out what they mean by looking at what's already been nominated, and offer the N nearest and most likely matches as possible options. (Still, of course, allowing people to type in something else entirely if they want to.) If the first 20 people nominate "That Awesome Story by Jane Smith" and the 21st person starts typing "Awesome Story - Jane Smith", they should see "Do you mean 'That Awesome Story by Jane Smith'?" as a suggestion.

There are more powerful forms of normalizing your data that you can use during the nomination period to minimize your burden in the tallying phase[1] but they're more of a pain. Still, a basic distance algorithm + a database of what's already been nominated will exert significant pressure on nominators to normalize your data for you with minimal work on your part. The only downside to such a system is that people would be able to tell when they are or aren't the first person to nominate a particular work.

Generally speaking, it sounds as though all the software involved in this process could use a strong dose of 21st century usability improvements. Is it open source? If so, where's your source code repo, and if not, have you considered making it so? I would be pretty willing to do some work on the nominating and voting software, and I'm pretty sure I can round up others in the open source world who'd say likewise. (Likewise, I'd be willing to help work on redesigning thehugoawards.com, which I have a lot to say about but will spare you for the moment.)

Drop me a line at denise AT dreamwidth dot org if you're interested in talking, since I suspect this is edging close to off-topic for this particular thread.


[1] For instance, the Archive Of Our Own, the fanfic archive produced by the OTW, has a tagging system that will, for instance, let different users categorize their work as belonging to "Star Trek - The Original Series", "Star Trek TOS", "Star Trek - TV Show", "ST:TOS", or whatever the user wants to use, while still a) suggesting the 'canonical' tags and b) funnelling all of those variations into the same bucket for purpose of viewing and searching. (Repeat for every single tag someone could possibly want to use to describe their work, from fandoms to relationships to content-description tags.)

The AO3 method has many flaws -- it requires significant volunteer labor to 'wrangle' all the permutations with the right behind-the-scenes connections and it makes many librarians and archivists of my acquaintance weep bitter tears into their beverage of choice for its inefficiency -- but it is very effective at nudging people into choosing already extant fandom tags while still allowing the option of creating a new fandom tag if they want.

#245 ::: nathanbp ::: (view all by) ::: April 09, 2015, 08:17 AM:

Another interesting consideration is what to do when a nominee declines the nomination. Under the current system the next eligible work is nominated to fill out the ballot (if it meets the 5% requirement). For more complicated voting systems it's not so obvious what should be done to find the next replacement nominee.

For STV, Bruce's system in #1 and similar systems, I see two different options.
1) You can leave the ballots as they are and see what the system picks as ranking #6. This basically throws out all the other preferences of the voters who voted for the nomination that was rejected. For example, if the winner from a set of identical votes rejected the nomination, it would throw out all of the slate votes and leave that set of votes with no bearing on the nomination at all.
2) You can cross the rejected nomination off everyone's ballots and rerun the election. However, this could cause other works that were previously nominated to no longer be nominated. I guess as long as you notified nominees in order from highest vote count first that could be OK. Although I feel like under some of these proposed systems even the highest nominee could be knocked off by crossing off all votes for a lower nominee.
For example, STV:
A - 101 Votes
B - 100 Votes
C - 99 Votes
D - 98 Votes
E - 97 Votes
-----Not Nominated Below This Line----
F - 96 Votes
G - 95 Votes
If B rejects the nomination, the 2nd choices for B voters could be 20 votes each for C through G, removing A from the nominees list. I feel like having to call up A and tell them that they aren't nominated after all is a pretty terrible outcome and one that should be avoided if at all possible. I guess you could instead rerun the election and then pick the top 4 excluding A as the remaining nominees, although that feels a little strange.

If we use some algorithm to expand the ballot beyond 5, then I'd think no additional work would be nominated?


A slightly better tie-breaker for the system in #1 might be that the work with the largest number of votes in the eliminated ballots wins, and then only go on to a random choice if works are still tied after that.

#246 ::: James ::: (view all by) ::: April 09, 2015, 08:17 AM:

raheli@244:

There is no single set of software in use because each worldcon is run by a different set of people with different resources, and the resources to build and maintain a more sophisticated input system on a continuing basis aren't part of the WSFS mandate.

#247 ::: rcade ::: (view all by) ::: April 09, 2015, 08:27 AM:

I've been wondering this morning whether the way forward is for the Hugos to actively encourage slates. Let any group of five people with Worldcon attending memberships announce their slate on thehugoawards.org while nominations are open, but limit them to three picks per category. Let any other member who wants to sign on as a supporter publicly do so.

If there were 5-10 popular slates representing different parts of SF/F fandom, their impact is spread out and no slate is likely to take the entire ballot. Voters who wanted to influence the process without adopting one slate could pick from the best of several.

We could end up with a ballot that contains some picks from slates, some works with crossover appeal to multiple slates and others that got there from individual support. This wouldn't require any rules changes, though I think it would help to also expand the nominees per category from 5 to 10.

#248 ::: rahaeli ::: (view all by) ::: April 09, 2015, 08:43 AM:

James @ 246:

I know each Worldcon is run by a different group, but you're telling me that there isn't even an established software package that's passed from committee to committee, and instead each year's committee has to completely reinvent every single system from scratch? Like, Sasquan had to design and build their Hugo nominating and voting (...and vote-tallying?) webapp from the ground up, and MidAmeriCon II next year has to start over and build their own Hugo nominating and voting webapp from the ground up, and this is done every single year?

Please tell me that's not the case, because if that's the case, I am genuinely horrified and appalled at both the sheer amount of waste-of-labor involved...

I'm not suggesting "the WSFS build and maintain a more sophisticated input system on a continuing basis" -- I know the WSFS does not directly administer the awards each year. But I can't wrap my brain around the idea that there isn't source code provided to each year's Worldcon committee for them to use to administer the process.

(...if there isn't source code provided to each year's Worldcon committee for them to use to administer the process, I'm guessing there also isn't conference software that solicits panel topics, performs panel selection, handles scheduling, etc? And that too has to be redone every year? Oh my God I am so sorry.)

An example of what I mean is the MIT Mystery Hunt: it's held each year in the same place and at the same time, but each year's Hunt is organized and administered by the team that ran the previous year's. It's very much like Worldcon in that there's no continuity of organizers (although, like Worldcon, the same names pop up fairly regularly), but there's a whole collection of tools that each year's organizing team passes to the next to avoid having to recreate infrastructure every year. A committee organizing a Worldcon should be able to go to Github, fork the 'worldcon' repo, and in an hour have a working install of all the tools they need to run the con, which they can then customize as needed!

#249 ::: Peter Hentges ::: (view all by) ::: April 09, 2015, 08:50 AM:

@Bruce 233: I was wondering yesterday if STV would be workable without ranking. It seems a robust scheme for running the nomination selection if ranking was in place. Good to see someone is examining the method without ranking. I agree that people (particularly Americans) would be less likely to nominate if required to rank their choices and more nominations is a good goal.

Once we've hashed this out, it looks like there is some good opportunity for doing some open-source programming of tools to be made available to worldcon committees for running the nominating and voting procedures. I can likely contribute to those efforts should they arise.

#250 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 09:15 AM:

rcade @247 With the current system the likely outcome would be that the most successful slate would take the whole ballot, and the other slates as well as individual voters would be shut out.

#251 ::: dh ::: (view all by) ::: April 09, 2015, 09:22 AM:

5. It should be difficult for a voting bloc to co-opt. To me, this implies that it should encourage diversity.

Diversity should be a side effect of a well designed system, so long as the field is diverse. If the field is not diverse, the nominations should reflect that.

For example, if you design a system for electing the head of the LDS church, I would be surprised if the nominations were especially diverse.

Producing especially diverse nominations doesn't seem to be of any value. What's important is that each of the nominations could be a potential winner.

#252 ::: Stephen Rochelle ::: (view all by) ::: April 09, 2015, 09:22 AM:

Bruce @233: I've been thinking about how to better express my reservations about the "y=1" proposals (remove a nomination ballot from consideration after 1 of its nominees is accepted) from your {x,y,z} formulation, and what I've come up with is a sixth point to tack on to your list of goals:

6. In an idealized no-slates environment, a new nominating process should behave substantially like the current nominating process.

That is, I don't think we've seen people say, absent slate-driven considerations, that the current nomination ballot is "bad" or "confusing" or "needs work" or what have you. However, in a "y=1" environment, strategic voting is strongly incentivized.

For example, last year, it was apparent in advance that there was a groundswell of support for Ancillary Justice; it was not much surprise that it appeared on the final ballot. Under a "y=1" system, anyone who really wanted to champion a more obscure work, or even someone who loved AJ, thought it should win the award, but still wanted something else represented via nomination, would have good cause to not nominate AJ at all, because that would result in their ballot being dropped due to a comparatively trivial result. The voting incentive in many cases devolves to "x=y" (here, "x=1"), because if there's something I really want to push toward the award ballot, it behooves me to make sure that nomination isn't lost in favor of something I care less about -- and so I nominate one thing and one thing only to ensure that the vote I care most about will be counted. As I don't recall seeing anyone here advocating for "just nominate one thing", I don't think the extreme case is seen as a good outcome -- but that's what reducing y in {x,y,z} pushes towards.

To be clear, the current "y=x" system has the same structural issue: if you're persuaded that Big Work A will be on the ballot regardless, you get the most mileage for your ballot by nominating other things, even if you love BWA. But "five votes count" to me feels qualitatively different from "one vote counts" -- at five, it seems you're much more likely to be able to reach a comfortable balance of "obscure stuff I want to champion" and "strong contenders I want to support" (because there's also the failure mode that if everyone abandons BWA because it's safely going to be nominated -- it won't be).

Where's the right balance point to restrict the power of slate voting without overly disrupting an idealized nominating environment? I'm not sure. But I'm certain from where I stand that it's not "y=1", and I'm skeptical that it's "y=2".

#253 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 09:25 AM:

Thanks to Cheradenine for bringing this discussion up to a higher level of voting systems literacy. As another voting geek, I think they've made all the necessary points, but since this is a really long thread, I'd like to reiterate.

1. From a voting systems perspective, this is an issue of proportionality. The Sad Puppies used strategic voting to get a disproportionate share of the nominations - around 30% of the voters in most categories captured around 75-90% of the nominations (depending on which categories you're focused on). But this would not be a big issue if they'd gotten 30% or even 40% of the nominations.

2. There is extensive literature and body of theory on proportional and semiproportional voting systems. Among the systems that could reasonably be used here are:

A. Bruce's "STV" suggestion. Note that this is NOT standard STV, as he doesn't transfer overvotes. While this change is "desirable" in that a Puppy-like slate would be punished, it's also undesirable in several ways. For instance, if there were two odds-on favorites, it's likely only one would get the nomination, as others have discussed. Also, it would encourage people to actively avoid voting for a work that they expected to win anyway; a perverse and divisive incentive ("Everyone, please vote for my book, and make sure you don't also vote for GRRM's book because he's already got a lock on a nomination...").

B. Satisfaction approval voting: this means assigning a number to "how satisfied" a voter is based on how many nominees they voted for, and then maximizing the total satisfaction. There are serious computational issues with this idea; I actually disagree with the paper that claims otherwise, though the reasons are probably too technical for a blog comment. Even if the computation could be manageable, it's really hard to explain.

C. Proportional Condorcet systems. Good outcomes, but hard to explain, and require voters to rank, which is a pain in the neck.

D. What Cheradenine called "Sequential Proportional Approval Voting" and is sometimes also known as "Reweighted Approval Voting". This means that voters can approve as many as they want; at the start, all approvals have a weight of 1; the highest approval gets a nomination; then all ballots are reweighted based on how many nominated works they approve in the category. Technically speaking, the weights should go as D'Hondt (reciprocals of integers) or Saint-Lagüe (reciprocals of odd numbers), but in practice for 5 nominees exponential (reciprocals of powers of two) would work fine and be easier to explain (because you can just reweight ballots at each step, without having to count how many winners they supported in earlier steps). As far as I can tell, Cheradenine expressed some skepticism of this system, but I think their criticism only applies if the D'Hondt weights are used; so I'd suggest the exponential or Saint-Lagüe weights.

3. This is a really long thread and it's impossible for me to respond to everything in it. As a voting geek, a lot of it looks like an impressive but somewhat misguided attempt to reinvent the wheel. People raise ideas, other people raise objections; most of the time, their instincts are good; but almost all of it has already been debated in the voting theory community, and even though the issues are not fully resolved there, there is at least a useful body of knowledge about what doesn't work.

3a. I think it's probably time for a new thread.

#254 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 09:31 AM:

Oh, one more thing: I've posted a question about this on Quora. I understand that some people don't like Quora's registration policy, but on the other hand, I do find it to be a good place to work out ideas like this because, unlike a pile of 250+ blog comments, it's possible for the good ideas to float to the top. Also, it's refreshingly troll-free (or nearly so).

#255 ::: GuruJ ::: (view all by) ::: April 09, 2015, 09:44 AM:

Living in Australia, where we use both IRV and STV (for our lower and upper houses respectively), and being a member of a political party that tends to hold no prisoners when nominating candidates for preselection/primaries the whole concept of bloc voting is very familiar to me.

My view is both systems are very effective at aligning your actual preferences with the best 'tactic' for voting. That is, in virtually every situation it is correct to vote [1] for your most favoured candidate, [2] for your second-most preferred and so on.

However, there is one important caveat. In variations of this system where preferences are *optional* and not exhaustive, bloc votes regain the advantage. In effect, not having exhaustive voting means that you can't put nominees last, and that's critical to avoiding the bloc vote situation where:

40% vote for A
15% vote for B, hate A
15% vote for C, hate A
15% vote for D, hate A
15% vote for E, hate A

but then A wins. However, as others have noted the time investment needed to evaluate and rank all candidates for something like the Hugos makes STV impractical.

My proposal (as an outsider) would be simple: don't make it a vote. Just have an nomination process where people "sign" the endorsement for a work to be included in a category. Do it live and online. While the organisers will know who has nominated (obviously), names don't have to be visible publicly. Once a work reaches the designated threshold for that category, it's in the nominee list, period -- no maximum or minimum number of nominees guaranteed.

You could give people one vote or multiple votes. Either way will tend to have the same outcomes overall assuming broad popular support for certain works, since once the quota for the most popular work is reached, those who weren't in that quota block can redirect their nomination elsewhere.

Of course, this doesn't prevent any particular nominee getting on the list. But no-one else gets excluded either. Just as in any other election, you're free not to evaluate anyone you consider to be unworthy or 'fringe'. And then STV largely mitigates the impact of blocs on the final outcome.

#256 ::: Andrew M ::: (view all by) ::: April 09, 2015, 09:46 AM:

Farseer@250: rcade's proposal ensures that one slate couldn't take over the whole ballot, by limiting slates to three works.

Nevertheless I find the proposal problematic, because I think it would encourage polarisation. The present system encourages works which have cross-group appeal. A system with official slates might well encourage people to put forward works which are central to a subgenre but don't have much appeal outside it, or are the latest work of a popular author but only appeal to those already familiar with them.

#257 ::: rcade ::: (view all by) ::: April 09, 2015, 09:58 AM:

Andrew M @ 256: A two-party system would be polarized, but how much polarization is possible in a robust 10- or 15-party system?

#258 ::: JonW ::: (view all by) ::: April 09, 2015, 10:08 AM:

Jameson@#253: Serious question -- you name four approaches that could reasonably be used here, and then identify substantial problems with three of them. So does that mean that in your view the best approach in this context (the only one without substantial disadvantages) is Sequential Proportional Approval Voting / Reweighted Approval Voting?

#259 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 10:22 AM:

Jameson Quinn @253 Regarding "Satisfaction approval voting" (assigning a number to "how satisfied" a voter is based on how many nominees they voted for, and then maximizing the total satisfaction):

Maybe a way to make the computations simple would be choosing nominees one by one, maximizing satisfaction at each step instead of trying to find the global maximum of the satisfaction function.

The first step would be trivial, with the most voted candidate being chosen as nominee. At the second step, among the remaining candidates the one who would maximize satisfaction at that step would be chosen as nominee, and so on.

At step n, since n-1 nominees have already been chosen, it's only a matter of checking the satisfaction result of choosing each of the remaining candidates with votes, one by one, and choosing the one that gives the best result.

This should be done by computer rather than by hand, but the algorithm would be quite simple. The result would not necessarily be the global maximum of the satisfaction, but it would be deterministic, simple and reasonable enough.

#260 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 10:27 AM:

Sorry, regarding my last message (259) I have just realized that my proposal would be similar to the "Reweighted Approval Voting" system... Anyway, I like that system. What are the known problems with it?

#261 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 10:32 AM:

JonW@258: Yes, I would suggest SPAV/RAV here. From now on I'll call it RAV but of course the terminology doesn't actually matter as long as we understand each other.

But still, I should probably discuss its downsides too, because, as others have pointed out, the Gibbard-Satterthwaite theorem shows that no deterministic, non-dictatorial voting system over finite alternatives is strategy free.

What would voting strategy look like in RAV? Something like what I've suggested would happen with Bruce's proposal, but at a much lower level. That is to say, there would still be an individual incentive not to vote for an odds-on favorite, in order for your vote to have full power to support your preferred lesser-known work. The difference is, this incentive would be much weaker than in the case of Bruce's crippled "STV", because unlike with his proposal, a group-level guarantee would still apply. That is, a faction of 40% of the voters could still be sure to capture 2/5 of the nomination slots, with or without strategy, so strategy would not be so necessary, and I think on the whole, dishonest strategy would be rare. Yes, there would be a few people who would strategically bullet-vote for their favorite work, but that's honest strategy (it doesn't involve voting for a less-preferred work while spurning a more-preferred one), so really there's nothing to be done about that, and in the end, I suspect such voters would not be enough to substantially impact the outcomes.

Note: if you wanted me to, I could list at least a dozen other proportional voting systems, and equal numbers of single-winner systems, and discuss all their relative advantages and disadvantages for this use case. RAV is really what I honestly think would work best in this case out of all those options. I am an activist on voting systems for political use (I'm a board member of electology.org, for instance), but RAV is actually not the system that I'd suggest for electing political representatives, so I'm not just doing the hammer/nail thing here. (For politics, my preferred solutions are approval voting, PAL representation, SODA voting, citizen advisory juries/sortition, and asset voting, in about that order. And I'd better stop there because there are another half-dozen systems that would be good.)

#262 ::: P J Evans ::: (view all by) ::: April 09, 2015, 10:35 AM:

248
The software that counts the Hugo ballots themselves is fairly standardized, and is passed along, but for the nominations, because no ranking is involved, it can be done with a spreadsheet - but it does require some kind of internal standardization for the names! (see this page for more details)

#263 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 10:40 AM:

farseer@260: I just talked a bit about the downsides above before I saw your message. As I said, I think they aren't too bad. Basically, the result would be that a work that everyone acknowledged was good, but that nobody was really passionate about, would be less likely to get a nomination; and that instead, works that inspired passion (which would probably include more things people thought were new and/or underappreciated) would get more nominations. On the whole, I think that's basically a healthy way for an award to be. If something is already so well-known that people are strategically avoiding voting for it, it probably doesn't really need the award anyway.

#264 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 10:40 AM:

GuruJ @255: the system you suggest could result in a very large number of nominees if there are several competing voting blocs, and I believe that result is not acceptable.

#265 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 10:45 AM:

Further note: we at electology.org would be happy to talk to Worldcon about the various issues here, and possibly could even help with the software. Anyone interested can gmail me, using the firstname dot lastname address (server implied by verb).

#266 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 10:51 AM:

Jameson Quinn @263 I agree. Of all the systems I have read about here so far, RAV seems to me the best for the Hugos. The encouragement for tactical voting exist but is not strong enough to be crippling, and as you say it even works in a way that could be seen as desirable for an award. It wouldn't encourage bloc voters by allowing them to sweep the nominations, but it would treat them fairly giving them a reasonable representation according to their percentage of the votes. Yes, I really like it.

#267 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 11:03 AM:

I hope I didn't offend anyone by using GRRM in my example. I am mostly wearing my voting theorist hat here, because it's bigger than my fan hat, but I still do at least know the difference between fantasy and science fiction (and no, it's not unicorns versus ray-guns either.) It's just that these days I can't think of an SF author with as big of a Hollywood boost to their fanbase as GRRM has.

#268 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 09, 2015, 11:06 AM:

Jameson Quinn @254:

Oh, one more thing: I've posted a question about this on Quora. I understand that some people don't like Quora's registration policy, but on the other hand, I do find it to be a good place to work out ideas like this because, unlike a pile of 250+ blog comments, it's possible for the good ideas to float to the top.
Where do people get the idea that it's polite to say they aren't reading the thread, or to complain that it's too much trouble?

This isn't a beauty contest where a few ideas are declared the winners after a little disengaged discussion. This is a conversation. We're all part of it. We're examining these ideas and their side-effects and implications. The processes that lead us to ideas are as important as the ideas. Arguably, they're not entirely separable.

Also, it's refreshingly troll-free (or nearly so).
There are forums that work harder to keep trolling from happening, and forums that deal with it more promptly and thoroughly when it shows up, but there are no troll-free forums.

If you mean to imply that there's undealt-with trollery in this thread, speak up and identify it now. I may have missed something that has the troll nature, but that doesn't mean I tolerate it.

Consider also that we're discussing this topic at a very fraught moment. There are strong feelings on all sides, and they're occasionally going to give rise to strong sentences. Again, if you think some go over the line, or if you feel the overall tone is becoming oppressive, say so, and be specific.

#269 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 11:07 AM:

Anybody who's still reading: pltcl lctns r s brkn, nd s fxbl, s th crrnt Hg prcss. t lctlgy.rg, w hv sltns; jn s. Th crrnt systm (plrlty) s s bd tht wn/wn fxs r pssbl; thngs tht wld mk Lbrtrns, Dmcrts, Grns, nd Rpblcn vtrs ll bttr-ff.

#270 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 11:16 AM:

Teresa@268: Sorry. I didn't mean to imply that I hadn't read this thread; I have. This stuff really interests me. What I meant was, I'm sure there are some people who don't have time to, so it's reasonable to give links to other places where back-and-forth discussion can still happen but it's easier for a passerby to see the outcome without reading the whole thing.

Similarly, my comment about Quora being reasonably troll-free was not meant as a comparison. The discussion here is doing a great job of staying on-topic. While I may think its a bit of a waste to try to reinvent the voting-theory wheel, I'm actually seriously impressed by how good a job of reinventing it this thread has done. I have seen a lot of threads where people are discussing some democratic failure and occasionally propose new voting systems, and outside of voting-specific mailing lists, I've never seen one which does as good a job as this one of exploring the pros and cons of the various proposals. If you're one of the moderators here, you're obviously doing something right.

#271 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 11:17 AM:

One comment about RAV (or any other of the systems being considered here): After the Hugo awards are given, the organization publishes the results of both voting rounds. For the first round (the nominations) the number of votes for each candidate is given. Obviously that would not be enough now, because with any system different from the current one the five candidates with more total votes would not necessarily be the final nominees. Therefore I guess that all the votes would have to be published (obviously without identifying the voter)...

#272 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 11:18 AM:

@269: OK, I get the message. I didn't think it was too spammy but I understand you have to draw the line.

#273 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 09, 2015, 11:20 AM:

Jameson Quinn @269:

Anybody who's still reading:
I beg your pardon? This is Making Light. Go look at the length of the other threads.
pltcl lctns r s brkn, nd s fxbl, s th crrnt Hg prcss. t lctlgy.rg, w hv sltns; jn s. Th crrnt systm (plrlty) s s bd tht wn/wn fxs r pssbl; thngs tht wld mk Lbrtrns, Dmcrts, Grns, nd Rpblcn vtrs ll bttr-ff.
If you can make me a good argument for why that wasn't an advertisement, I'll restore your vowels.

#274 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 09, 2015, 11:22 AM:

Thank you. I have a soft spot for people who get messages.

#275 ::: John Mark Ockerbloom ::: (view all by) ::: April 09, 2015, 11:24 AM:

"I agree that it's unlikely that, say, an 8% threshold would lead to the maximum number of nominees (60 if I'm doing my math right, assuming we stuck to 5 max nominations per voter) - but the fact that it COULD lead to close to 60 items on the ballot is worth considering."

It is worth considering what *could* happen, but not as important, IMO, as considering what's *likely* to happen (which can of course be a range of possibilities), and how to robustly adapt as conditions change.

Here in Philadelphia, for instance, there's a race on for mayor. Anyone can get on the ballot if they get at least 1000 nominations (voter signatures). In a city our size, that *could* results in a ballot with hundreds of candidates. But we can safely assume it won't. There are two big reasons for this: First of all, obtaining 1000 valid voter signatures is no easy feat. But also, past experience tells us the general order of magnitude of ballot listings a given threshold is likely to yield, and while the pattern of voters changes somewhat from year to year, the changes are limited in size and somewhat predictable.

That latter point seems to be true of the Hugos as well. Even with invitations in new venues to jump onto a slate, the number of Hugo nominations went up only by about 10% (2122 noms vs. 1923 last year). And the Puppies have taken 3 years to build up steam (and online indications suggest their momentum will be continuing into year 4).

So what about this simple modification to the voting process: Keep the second IRV round the same, and keep the nomination voting the same as well, but authorize the voting committee to decide in advance each year (after one year's final ballots have been counted and reported on, and before the next year's nomination balloting begins) how many nominations will qualify for the next year's final ballot in the various categories?

So, for instance, the committee could say something like "Given the number of nominations and bloc voting patterns we saw this past year and other recent years for Best Filk Opera, next year's Best Filk Opera category will have a threshold of _180_ nominations, or a minimum of _3_ candidates. (Past results indicate that should result in about 5 non-bloc candidates in a typical year, give or take, and not produce an overly large final ballot.)"

Doing things this way does mean that voters would need to put a bit more trust in the skill and integrity of the voting committee. But it seems the Hugo committee already puts a fair bit of trust in them to get it right, and the task set out for them here is doable. (One can run various content-neutral algorithms to detect nomination voting blocs and their associated works, and the kind of historical size analysis that one would run to estimate likely vote pool sizes is similar to the sort of analysis one would run to estimate the sizes of venues one should book for a WorldCon.) It can be adjusted year to year as Hugo voting pools and slate influences rise and fall. And importantly, *it doesn't make the voting process any more complex or harder to understand for the voters themselves*.

I'm sure a committee like this would occasionally be off in their estimates, for one reason or another. But this could well be a manageable thing: the occasional 12-work or 3-work final ballots might be grist for snark among some Hugo voters, but you generally wouldn't see 60-work final ballots.

I should conclude by noting that I'm not a Hugo voter, and have rarely ever been to a con. But I wonder if those who are, and who have, would find this sort of system workable, or not, for the Hugos.

#276 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 11:26 AM:

farseer2@271: For RAV, as long as you're using the exponential weights, you can just publish a 2-dimensional table of how many votes there were which supported each pair of candidates. The raw votes for each candidate would be along the diagonal of the matrix. (That is another advantage for the nearly-proportional exponential weights over the truly-proportional D'Hondt or Saint-Lagüe ones. Note that the exponential weights are proportional for 0-2 nominees, and only non-proportional for factions with 3 or more nominees. Since there are 5 slots, the only people who would have a valid complaint against exponential weights would be majority factions. Since a true majority can pick the winner anyway, it doesn't really matter if the exponential weights "cheat" them out of a 4th nomination slot they "deserved".)

#277 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 09, 2015, 11:27 AM:

Also, re your comment #267, if you're looking for an example of an author whose fanbase has gotten a huge boost from Hollywood, George R. R. Martin is the one to choose. There's nothing offensive about it.

#278 ::: CHip ::: (view all by) ::: April 09, 2015, 11:35 AM:

Andy @ 160, Farseer2 @ 162: The business meeting is massively harder to game even once, because there is no proxy voting; voters must have an attending membership (minimum 3x the supporting membership*) AND get to the site AND feed&house themselves for some number of days, because a rule can be rejected either in a preliminary ~agenda-setting round or in a final vote. The mechanics Kevin @ 210 discusses are an additional layer, but ISTM that the ban on proxies is a much higher first barrier to the Puppies' type of mischief.

Hentges @ 179: you are technically correct; however, one of the major themes of this discussion is that a system hardened against all interference will also be hardened against most voting. (See, for starters, the last paragraph of Bruce's introit.)

rahaeli @ 248: PJ Evans is confident the Hugo software is passed on; I'm not, but I may be more out of the loop. However, you're certainly going overboard talking about other systems not being transferred, because most Worldcon committee members also work on at least one regional. Hugo voting is Worldcon-specific; other computerizable tasks have been solved (many times...) at the local level, and many of these solutions are scalable to Worldcon level. (The complexity difference between a regional large enough to nucleate a Worldcon bid and a Worldcon is a lot smaller than the difference between (e.g.) J. Random Regional and San Diego Comic Con.)
      wrt transferable code: every committee has some degree of we-can-do-that-better; I don't think you could get a committee able to run a Worldcon without this. Where it pops out varies widely, but trying to suppress it for any case will have costs -- aside from the maxim that portable systems aren't.

Jameson @ 253: Best estimate is that the Puppies were 10-15% of the nominorate, not 30%; when they played fair last year, they got <<10% of the nominations, which IMO is proportional to their extremism and non-fact-based approach.

(*) my one contribution to WSFS, changed from 2x in 1989

#279 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 11:39 AM:

Whoops. No, actually, I lied; the 2-D matrix would not actually be enough to verify the correct RAV winner. But it would be enough to get very close to doing so, and put some strict bounds on the possible RAV winners. In fact, I'd say that you could use the matrix instead of doing true RAV. I guess I'll call that proposal "Summable RAV" or "SRAV". Here's how it would work:

1. People vote and the matrix is tallied
2. The candidate i with the largest diagonal (i,i) element is chosen.
3. For every j≠i, subtract (j,i)/2 from (j,j)
4. Repeat steps 2 and 3 four more times (to get 5 nominees total).

This is not entirely equivalent to RAV, but it's damn close.

#280 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 11:41 AM:

@Teresa Nielsen Hayden: I respect your decisions as owner of the blog, but I don't think there's any need to be harsh. Jameson Quinn has made very useful contributions and has offered his organization's help should the WorldCon want it. That help is valuable because they have knowledge that most of us here don't have and, as he said, there's no need to reinvent the wheel. I don't think it's so bad if he wants to mention his organization (which has a relation to the subject of this thread), and may be of interest to people in this thread.

#281 ::: Mark Z. ::: (view all by) ::: April 09, 2015, 11:42 AM:

rcade #257: I think you're missing the point.

We're not talking about the system for choosing the next Parliament, where there's actually some benefit in electing a group of people who can function as a team. The purpose here is to identify the best works of science fiction of the year.

As such, there is no honest reason for anyone to promote a slate of nominees. The nominating ballot asks me, "Mark, of all the science fiction you've seen this year, which five TV episodes do you think are likely to be the best?" It's very unlikely that someone else would come up with the same list of five, even if they'd seen all the same stuff I have. So if several people do turn in identical ballots, they are most likely not answering that question.

If there is a group of people that passionately believes some work deserves to win the award--and this doesn't have to be anything sinister, simple fannish enthusiasm will do it--then we expect they'll all nominate that thing, intending to vote for it. What they won't do is all nominate the same four other things.

In summary: slates are bad. They screw up the process. What you're offering is exactly the outcome everyone here is hoping to avoid, where one slate induces the formation of other slates, and the whole process degenerates into Who Runs Barter Town.

#282 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 11:46 AM:

CHiP@278: "Best estimate is that the Puppies were 10-15% of the nominorate, not 30%" -- I'd say you're about right, if you're counting total ballots. But a large fraction of the ballots didn't vote in many categories, while the puppies mostly did. So my 30% estimate was based on the less-prominent categories. I think that's a reasonable way to look at it, because I don't see that issue going away. Organized, puppy-like factions will always be able to spam all categories, while bona-fide fans will always be less likely to have a considered opinion in some categories.

#283 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 12:02 PM:

Jameson Quinn @279 The complete 2-D matrix would be quite large, since I'm sure there is a very large number of candidates with at least one vote, but publishing only a submatrix with maybe the 15 candidates with most votes would be enough.

#284 ::: P J Evans ::: (view all by) ::: April 09, 2015, 12:08 PM:

278
Based on a fast search online, and the Hugo site itself: it's Jeff C's program (which has been in use for some years, and he's apparently maintaining it).

(Beats writing a new one every year or so. Been there, done that.)

#285 ::: rcade ::: (view all by) ::: April 09, 2015, 12:14 PM:

Mark Z @ 281: "What you're offering is exactly the outcome everyone here is hoping to avoid ..."

True, but the successful emergence of a party in a voting system is a genie that won't go back in the bottle unless there are significant voting system changes. Any such changes can't take effect until 2017 given how Worldcon voting works.

Given the difficulties and drawbacks of voting system proposals being floated here, it seems like we should at least consider whether the official encouragement of slates would diffuse their power enough to make them as weak as the normal Hugo recommendations that pros and fans have always made when nominations open.

#286 ::: andyl ::: (view all by) ::: April 09, 2015, 12:16 PM:

@Mark at 281.

I'm not so sure about that. I think that the Doctor Who ones (or at least 3 of them) last year might have been on a lot of people's ballots. Partly because it is a popular show and partly because of the 50th anniversary.

I think this is only likely to happen for the dramatic presentation categories.

#287 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 12:17 PM:

Me @121:

"We've been talking about dynamically expanding the number of nominees based on the voting patterns. Here's a good paper by Steven Brams on one way to do that."

nathanbp @131:

"Bruce, #121: That paper seems to be behind a paywall."

Try this link.

#288 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 09, 2015, 12:20 PM:

rcade: There is no way to make slates so weak that they're the equivalent of recommendation lists. This goes double if the electorate gets bigger, broader, and more diffuse.

#289 ::: JonW ::: (view all by) ::: April 09, 2015, 12:29 PM:

Bruce@287 -- the Brams paper seems to be about identifying candidates who are closely bunched, so that it would be reasonable for them all to make the runoff ballot. But in a Hugo election like this year's, that would mostly just identify the slate candidates, no?

#290 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 09, 2015, 12:39 PM:

Mark Z. @281: I can imagine several voters independently coming up with the same five nominees in one category. Not a problem. What I'd question would be them doing it in more than one category.

At that point I'd want to hear that they're all members of the same little SF club and are voting in the Hugos for the first time, or are online BFFs who came up with their nominees during a long chatty groupthink session. And you know, even then I'd expect variations, because anyone who cares enough to vote in the Hugos is bound to Have Opinions.

#291 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 12:47 PM:

So, I have to edit my SRAV proposal @279. Grrr... After reading a whole thread full of smart people stumbling around in territory I already know, I should realize how hard it is propose voting systems on the fly.

Here's the better rules. The change is that I've added rule 4. The other rules are slightly rewritten but mean the same as above.

1. Vote and tally the matrix. Cell [i,j] of the matrix counts how many ballots approve of both candidates i and j. In practice, you could ignore overlaps between any two candidates which are below a certain threshold, leaving a truncated matrix plus a list of individual tallies below that, as farseer@283 basically suggests.
2. Choose candidate i with the highest [i,i] as a nominee.
3. For every j≠i, subtract [i,j] from [j,j].
4. For every k∉{i,j}, multiply (k,j) by max(0, ([j,j]-[i,j])/[j,j], ([k,k]-[i,k])/[k,k]) That is, reduce the "overlaps" proportionally to how much you reduced the "individual tallies".
5. Repeat steps 2-4 four more times to get 5 nominees per category.

I can't be 100% certain until I've actually coded it up and played with it, but I'm pretty sure the above algorithm is now solid; that in practice it would almost always amount to the same as RAV, but simply be easier to tally and verify.

Note: the equation in step 4 uses the versions of [k,k] and [j,j] from before step 3. In practice you'd do step 4 before 3 but it's harder to explain like that.

#292 ::: nathanbp ::: (view all by) ::: April 09, 2015, 12:48 PM:

The results from any staged nominating process could be reported in a way similar to the way IRV results are now now. So something like show the vote totals for the top 15 candidates at each stage of the nominating process. There's no need to provide enough information to allow the election to be rerun by an outside group.

#293 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 12:49 PM:

@290: That's true. But if you made a rule based on that, it would be easy for the faction to game the system by simply adding a few random "noise" nominations to their ballots.

#294 ::: spacefaringkitten ::: (view all by) ::: April 09, 2015, 01:01 PM:

rcade @285:
True, but the successful emergence of a party in a voting system is a genie that won't go back in the bottle...
...it seems like we should at least consider whether the official encouragement of slates would diffuse their power...

I think that the opposite is true, to some extent. If there's a strong, public and popular backlash against slate-voting this year, possible nominees will consider very carefully whether they want to be on one the next year. That could take some wind off Sad Puppy 4's sails.

#295 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 01:15 PM:

spacefaringkitten @294 I don't really agree. It might make high profile candidates that have nothing to do with them ask not to be included, but there's nothing to stop them from making a slate with people who are sympathetic to their cause and do not care (or even including high-profile candidates without their permission, if they are so minded). After all, what can you do to threaten them? They are already convinced they will never be considered fairly for the award.

#296 ::: J Thomas ::: (view all by) ::: April 09, 2015, 01:15 PM:

The argument is made that STV will stop slates and increase diversity in nominations. Here is an emotional argument why that might not be what we want:

Imagine that one year we get some novels that are obviously great, that tower over the crowd.

Stranger In A Strange Land.
Dune.
Venus Plus X
The War for the Oaks
Jhereg.

These so obviously great that there are a thousand nominations that list four or more of them.

There are also 8 votes that only list Dhalgren.
7 votes that only list Raped Slavegirls of Gor.
6 votes that only list Roger’s Raiders #27: Slaughter on Saskatoon
5 votes that only list Goblins and Trolls #16: Hobbit’s Dilemma

Who wins the nomination with STV?

Stranger in a Strange Land
Dhalgren
Slavegirls
Roger’s Raiders #27
Goblins and Trolls #16

That’s diversity.

The problem is that STV does not just hurt slates, it also hurts the second-most-popular choices and the third-most-popular choices etc.

However, it might help with the immediate issue. The Sad Puppies think their choices don’t win because the dominant worldview excludes them.
If we try to have fewer popular choices and more less-popular nominations, they will be more likely to get some of their choices nominated whether they block-vote or not.
So if they are satisfied maybe they will not raise so much fuss.

#297 ::: JonW ::: (view all by) ::: April 09, 2015, 01:22 PM:

J Thomas @#296 -- Your argument is part of the motivation for looking at an approach like SRAV, which recent posts have been discussing, as an alternative to STV.

#298 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 01:24 PM:

JThomas@296: You're right that Bruce's proposed "STV" could have this result. But STV as practiced in actual countries, with transfers of overvotes, would not end up like this. Again, the lesson is, don't try to reinvent the wheel, or even to "improve" it without thinking seriously about the downsides.

(I didn't entirely follow that advice with SRAV above, and I got it wrong, at least the first couple of times. I think that nathanbp@292 has it right that SRAV is unnecessary; RAV is plenty.)

#299 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 01:25 PM:

J Thomas @296 Your example is a bit exaggerated. How come there is such an incredible correlation between the votes of the five great novels? There could be a lot of correlation, but not one so extreme. You basically have no one voting for the great novels unless it is together with absolutely all the other ones.

But in any case, I agree that y = 1 systems are too extreme. However, the RAV system handles things much more smoothly and would work perfectly well in your example.

#300 ::: JonW ::: (view all by) ::: April 09, 2015, 01:25 PM:

Correcting myself: As an alternative to Rivest's version of STV presented in #1. Conventional STV wouldn't present the problem in the first place.

#301 ::: Fragano Ledgister ::: (view all by) ::: April 09, 2015, 01:26 PM:

rcade #285: I can think of one case of parties appearing in an open electoral system and then disappearing, the Cayman Islands.

#302 ::: nathanbp ::: (view all by) ::: April 09, 2015, 01:29 PM:

J Thomas, #296: That's true for Bruce's system in #1, and one of the reasons people are talking about alternatives (like making further choices count as 1/2 a vote instead of a full vote). For STV, the quota for 1026 votes is (1026/6)+1 = 172 (see wiki). So your situation would result in your 5 great novels winning.

Assuming ranked STV and all 1000 votes in the order you listed, Stranger In A Strange Land wins the first nominee slot. Then (1000-172)=828 votes spill over to Dune. Dune wins the second nominee slot. 656 votes spill over to Venus Plus X which wins the third nominee slot. And so on. For non-ranked STV, all 5 nominees tie with 200 votes to start and are all chosen since 200 > 172.

For the half a vote system with no ranks we'd pick from the tied winners randomly. Assuming the same order, Stranger In A Strange Land gets first with 1000 votes, Dune second with 500 votes, etc down to Jhereg fifth with 62.5 votes.

Guess other people pointed this out while I was typing but since I typed up some examples I'll post it.

#303 ::: John A Arkansawyer ::: (view all by) ::: April 09, 2015, 01:33 PM:

J Thomas @ 296: "That’s diversity."

I like it. It increases the number of books I've read and puts the best choice on the ballot.

#304 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 01:34 PM:

fs@299: RAV with Saint-Lagüe weights would handle JThomas's example well, but RAV with exponential weights might short-change Jhereg. But if 5 such novels were published in one year, I think even Brust would acknowledge that a Jhereg Hugo just wasn't in the cards anyway.

(Also: TWftO? That was certainly a landmark at the time, and still not a bad book overall, but if it were published today I don't think it would "tower over the crowd".)

#305 ::: Carrie S. ::: (view all by) ::: April 09, 2015, 01:34 PM:

This may be hopelessly naive.

Is there a reason we can't say, "Any nomination ballot that shares 4 or 5 choices in a given category with a published slate is not counted in that category"?

That is, if SP 4 comes out with ABCDE in Best Novel, and a nomination ballot shows ABCDE or ABCDF, that ballot's choices for Best Novel just aren't counted. But if the slate has ABCDE for Short Story and the same ballot's Short Story is FGHIJ, it is counted.

At the very least it would force the slate-makers to either keep their slate(s) really quiet or spread out their picks and do a whole lot of backchannel coordinating: "If your surname begins with A-E, nominate ABCDE. If it begins with F-H, nominate ABFGH." Etc.

#306 ::: Aaron ::: (view all by) ::: April 09, 2015, 01:42 PM:

This may engender more work, and might not entirely solve the problem, but given that the slate forced a ballot into existence that a majority of the voters dislike, could (in the future) a middle round be put in place in which the voters were asked to vote on whether to accept or reject the proposed nominee list in toto?

#307 ::: Lee ::: (view all by) ::: April 09, 2015, 01:51 PM:

Brad, #231: So you're saying that Condorcet is essentially a bubble-sort? That makes sense to me, because I have a background in programming.

When I talk about IRV, the first thing I say is that its goal is to choose the most broadly liked option out of a group. I think having that idea in place makes it easier to understand the process.

Bruce Schneier:
@235: STV makes voting blocs less powerful. A bloc will get one candidate on the final ballot easily, but I don't think it's possible to prevent that. What we want is for a bloc to have trouble getting many candidates on the final ballot.

Yes. The goal is to make the overall system less gameable by a small group.

@241: The "negative votes" proposal strikes me as subject to the same issues that regularly occur on websites that allow downvoting of comments, which are in turn similar to the issue we're discussing here -- it becomes too easy for a small but closely-coordinated group to take control of the process.

#308 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 01:52 PM:

Aaron @306 That's been suggested. It would not be a bad solution, but it's discarded because adding a new round would be too much work for the already overworked organizers.

There are reasonable solutions that do not require another round, fortunately.

#309 ::: Fragano Ledgister ::: (view all by) ::: April 09, 2015, 01:54 PM:

J Thomas #296: Would you mind demonstrating how that is possible?

#310 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 01:55 PM:

Carrie S. @305 Wouldn't work in practice, Carrie. You would need a permanent committee of judges scouting the internet to identify slates and arguing about which are slates and which are recommendations. A real mess.

#311 ::: Carrie S. ::: (view all by) ::: April 09, 2015, 02:09 PM:

Farseer2: "If it says 'slate' or has exactly the right number of picks, it's a slate." And again, broken by category--if your List has five Best Novels but three Best Short Stories, Best Novel counts as a slate but BSS doesn't.

Yes, this makes life slightly more difficult for folks who want to recommend, but if you can think of five works you like enough, surely you can think of six, or one that isn't quite up to the snuff of the others.

It would add a task; how much of one depends on how covert you want to force the slate-makers to be. Crowdsourcing might help too.

I just think it pretty clearly sends the message that the problem is slates qua slates, not their specific contents.

#312 ::: Nancy Lebovitz ::: (view all by) ::: April 09, 2015, 02:17 PM:

Here's a notion-- instead of weakening the votes for highly correlated nominees, label those nominees on the ballot, and also mark non-correlated nominees. Non-correlated nominees would also be listed before correlated nominees.

I'm hand-waving my way past the difficulties of identifying correlated nominees. With the stakes a little lower, perhaps being on a slate should be added as evidence to votes which look correlated to an algorithm. I don't think a secret slate could work.

I'm not sure how many should be on the ballot-- maybe most-nominated four (or five, in case of ties) of non-correlated, and the same for correlated, nor what to do with a work which has gotten a lot of independent votes and slate nominations.

#313 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 02:20 PM:

Carrie S: You would have to make detailed rules on defining objectively what a slate is. It would be an incredible amount of work, generate endless discussions and accusations and be very easy to game. No, the organizers cannot be in the business of deciding what vote contents are acceptable. That's a very bad place to go. There are much better alternatives. If you read this thread you'll see that there are voting systems that are fair but won't let voting blocs get more representation than their numbers warrant.

#314 ::: Cheradenine ::: (view all by) ::: April 09, 2015, 02:23 PM:

Here's a paper that discusses reweighted approval voting (as well as two other proportional forms of approval voting). It's focused on computational complexity, which may make it denser reading than everyone here is interested in, but I mention it in part because it has a pretty thorough bibliography. I'll try to follow the links there and come up with more (& hopefully more accessible) references.

http://www.nickmattei.net/docs/av1.pdf

One takeaway from the paper is that the winner can be reasonably determined by a computer using reweighted approval voting, but that finding a strategy guaranteed to elect a particular candidate (or whether such a strategy exists) is computationally hard. This is probably a good thing if we want to limit the effect of strategic voting.

#315 ::: Buddha Buck ::: (view all by) ::: April 09, 2015, 02:43 PM:

Lee @307:

OK, maybe Brad's 1-sentence explanation of Condorcet can be misunderstood ;-). Let me try (more than one sentence).

It's easy to understand a 2-candidate election. People cast ballots saying they prefer Abbott over Costello, or Costello over Abbott, and if more people prefer Abbott over Costello than vice versa, Abbot wins. Unless there's a tie, one candidate or the other will be preferred by a majority of the voters.

But in a 3-, 4-, or more-candidate election, this fails. It's easy for a third of the voters to want Groucho, a third to want Chico, and a third to want Harpo, so there's no clear majority. It gets worse when Zeppo or Gummo appear on the ballot.

So imagine each candidate as if they were in a 1-on-1 race with each other candidate: instead of 1 race with Groucho, Chico, Harpo, and Zeppo, imagine the six 1-on-1 races of Groucho v. Chico, Groucho v. Harpo, Chico v. Zeppo, etc. Each of those 6 races is going to yield a winner (or, in rare cases, a tie).

Most of the time, one candidate (like Harpo) is going to win all their 1-on-1 races. More voters prefer Harpo to Groucho (but some of them might prefer Chico to either), more voters (not necessarily the same voters) prefer Harpo to Chico, and more prefer Harpo to Zeppo. It makes sense that Harpo should win, since regardless of what other candidate you choose, a majority of voters will say that the other candidate was worse. Harpo is, in this case, the "Condorcet Winner".

It's even possible that this happens even if Harpo is no one's favorite. Imagine 30 voters, 10 each who like Groucho, Chico, and Zeppo best, and all like Harpo second-best. Then Harpo would beat Groucho 20-10, Chico 20-10, and Zeppo 20-10. (Incidentally, IRV would eliminate Harpo in this scenario in the first round, which is one major way IRV and Condorcet differ).

The vast majority of the time, there is a Condorcet Winner, but occasionally there isn't. Sometimes more people will prefer Moe to Larry, more will prefer Larry to Curly, and more will prefer Curly to Moe. Essentially, there's a complex tie. There are different ways to resolve such ties, but I think for the Hugos that as long as No Award is not involved in the tie, just accepting it is reasonable.

But I'm not recommending Condorcet for nominations.

#316 ::: Stephen Rochelle ::: (view all by) ::: April 09, 2015, 02:59 PM:

John A. Arkansawyer #303 wrote: J Thomas @ 296: "That’s diversity."
I like it. It increases the number of books I've read and puts the best choice on the ballot.

If that's the case, then you're also implicitly arguing that the actual voting round is unnecessary. Just declare whoever got the highest number of nominations as the winner; they're "the best choice".

#317 ::: james woodyatt ::: (view all by) ::: April 09, 2015, 03:00 PM:

It seems to me that one of the perceivable problems with the system that Mr. Schneier describes @1 is that it might get squirrelly in the event there are N > 1 mostly balanced parties, each voting slates, and there is a limit L of less than 2N nominations in a category.

What I would expect to happen when N=3 and L=5 is that two of the parties would win two lines each on the ballot and the remaining party would win only one line on the ballot. Now, if the party that wins only one line on the ballot happens to have a plurality, or worse, a majority, the rules of the nominating round might be viewed as having unfairly disadvantaged them. This seems like it could serve as invitation to tactical voting.

What can be done to prevent that outcome from arising?

#318 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 03:04 PM:

Man, I wish I'd been more careful earlier when I was trying to promote my organization, ryrpgbybtl.bet. If instead of going into salesman mode, I somehow dreamed up some ironic meta joke, I might have avoided disemvowelment. And then this impressive bunch of people who, based on the last few comments, actually have a pretty great understanding of voting theory, would be exposed to our organization and its attempts to mprv pltcl vtng. W hv sch gd ntntns...

#319 ::: Carrie S. ::: (view all by) ::: April 09, 2015, 03:07 PM:

Farseer2 @#313:

One, I have already given you a rule that objectively defines a slate: It says "slate" at the top, or has exactly five picks on it. (Or N, where N is the number of finalist slots.)

Two, I rather wonder what I've said that makes you assume I haven't read the thread.

Three, allow me to state explicitly that the idea was meant to allow the Hugo nomination process to proceed in as close to its current mode as possible, while still mitigating the influence of slates by making them more trouble for their compilers. Is there a reason it wouldn't do that, aside from "We can't define a slate"?

I'm not married to this idea by any means, but it seems minimally disruptive to the process as a whole. Which possibly falls under the "simple, elegant, and wrong" rule, but I'd be pleased to get engagement that wasn't "We can't define terms", if you don't mind.

#320 ::: Stephen Rochelle ::: (view all by) ::: April 09, 2015, 03:12 PM:

Carrie S. @317 wrote: I have already given you a rule that objectively defines a slate: It says "slate" at the top, or has exactly five picks on it.

OK, but from memory, the SP slate had 4 items listed for several categories. Suppose the next one limits itself to 4 (or N-1) in all categories. Now suppose it's called the "SP Voting List". What happens? It fails my sniff test, but it passes your rule (and the "uses the word 'slate'" is the part I find particularly fragile).

To be clear, I like the principle of your suggestion. I just don't see how it can be implemented without credible claims of bias on the part of the organizers.

#321 ::: dh ::: (view all by) ::: April 09, 2015, 03:13 PM:

Bruce--

Did you end up posting a study of the data available that shows the correlation between the published slates and the actual results?

#322 ::: Stephen Rochelle ::: (view all by) ::: April 09, 2015, 03:13 PM:

Er, Carrie @319. Hopefully that's obvious, given that it's the next comment up.

#323 ::: John A Arkansawyer ::: (view all by) ::: April 09, 2015, 03:20 PM:

Stephen Rochelle @ 316: I'll admit it--I was trolling (though lightheartedly rather than meanspiritedly). I've actually read two of the books on your original list, though Venus Plus X was some time ago and I don't recall it as clearly as I'd like. And by "best", I meant "my favorite", which is also not quite true. I'm not sure I like Dhalgren better than Stranger.

So my apologies for cluttering (and now further cluttering) the thread. My other suggestions here have all been straightforward suggestions (I think), though playfully expressed. This whole argument is very depressing to me at the same time that it is very ludicrous, and one of the few things that lets me stare into that void without terror is laughing.

#324 ::: Carrie S. ::: (view all by) ::: April 09, 2015, 03:22 PM:

Stephen @#320: You could, if you liked, make it "four or five". Or even "five or fewer"--you have to propose more picks than there are slots to stick 'em in.

But that latter would get the guy who just read the one book and thinks it's great and should get a Hugo.

I think I may in fact be simple, elegant and wrong. :)

#325 ::: jnfr ::: (view all by) ::: April 09, 2015, 03:28 PM:

The more you precisely define what is unacceptable, i.e. 'slates', the harder the min-maxxers will work to dance around your rules and still accomplish their purposes. That's an old, old game.

I'm hoping that this focus on actual changes to the voting algorithms can find some way to dilute their impact without encouraging the game-playing.

#326 ::: felice ::: (view all by) ::: April 09, 2015, 03:46 PM:

"1. It should encourage people to nominate. To me, this implies no ranking."

I'm not convinced ranking is a big disincentive in the specific case of the Hugos, given that ranking is already required for the actual vote, and that gets a much bigger turnout than the nominations. In fact, people have to rank their nominations anyway because they're listing them themselves, rather than selecting from an existing list; we just currently ignore that ranking. I'd suggest simply downplaying the importance of ranking - instead of "The nominations are equally weighted: the order in which you list them has no effect on the outcome", say something like "The order in which you list nominations may have some effect on the outcome, but in most cases it won't matter and there's no need to overthink it"?

I believe the biggest barrier to participation in the process is people simply not knowing what to nominate, and #67 still looks like the best solution to that.

#327 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 03:55 PM:

Steven Rochelle @252:

"Where's the right balance point to restrict the power of slate voting without overly disrupting an idealized nominating environment? I'm not sure. But I'm certain from where I stand that it's not 'y=1', and I'm skeptical that it's 'y=2.'"

You make some good points. Whatever system we end up with is going to be a compromise between different requirements. Limiting the number of nominees that an individual can vote for -- either by a hard restriction on the number of nominations, or a software restriction in a STV-like system -- isn't a perfect solution. But it does limit the power of bloc voting.

The question is how do we do that? We can limit x: the number of candidates any person can nominate. We can limit y: the number of nominations that win before a ballot is no loner counted. You're right, y=1 is too low. The incentive to game the system is great. On the other hand, y=5 is probably too high. My thinking is that y=3 is about right.

Does anyone have data on how many people currently nominate five works in a category? How many nominate less? That is, do people actually use all the nomination slots the system gives them?

#328 ::: Bruce Schneier ::: (view all by) ::: April 09, 2015, 03:56 PM:

Jameson Quinn @253:

I am starting to collect papers and essays on the various systems we're considering. If you have any, please email them to me.

"What Cheradenine called "Sequential Proportional Approval Voting" and is sometimes also known as "Reweighted Approval Voting". This means that voters can approve as many as they want; at the start, all approvals have a weight of 1; the highest approval gets a nomination; then all ballots are reweighted based on how many nominated works they approve in the category. Technically speaking, the weights should go as D'Hondt (reciprocals of integers) or Saint-Lagüe (reciprocals of odd numbers), but in practice for 5 nominees exponential (reciprocals of powers of two) would work fine and be easier to explain (because you can just reweight ballots at each step, without having to count how many winners they supported in earlier steps). As far as I can tell, Cheradenine expressed some skepticism of this system, but I think their criticism only applies if the D'Hondt weights are used; so I'd suggest the exponential or Saint-Lagüe weights."

I am still thinking about this system. Again, references would be appreciated.

#329 ::: nathanbp ::: (view all by) ::: April 09, 2015, 04:18 PM:

Bruce, 327: Isn't the more interesting statistic how many people tend to vote for more than one of the nominees? I don't think anyone has released that data.

#330 ::: Vivien ::: (view all by) ::: April 09, 2015, 04:20 PM:

Hi, I am just a fan who is somehow discouraged by the current situation. Wandering on the internet looking for someone with a solution :-). Please excuse my english as I am not a native speaker.

I am really enthused by those alternative voting systems you discuss, especially the RAV one. What I find exhilarating is that it actually is a system that even a Sad Puppy could approve (at least, a not too disingenuous one) ! It does keep slates away, but it would also effectively guarantee that no strong minority of the voters is ever left totally aside...

An advantage of this system (and of the system presented in 1, by the way) that has not been presented yet is that it could be easily proof checked on old Hugo data as it doesn't require a modification of the way the ballot is filled... As long as those data has been kept unaltered after the counting. How better to assess the potential of the voting system then to test it on real life data ?

Do you know if those data still exist and if the WSFS could be willing to share it ? At least, it could definitely be done on the 2016 edition waiting for a definite approval in 2017.

#331 ::: Fragano Ledgister ::: (view all by) ::: April 09, 2015, 04:36 PM:

Bruce Schneier #328: Saint-Lagüe quotients work best when there is a large number of candidates. That approach tends to produce a slightly higher degree of proportionality.

In any case, the real issue isn't the quotient, which is a simple matter of mechanics, but the threshold. Accepting the 'natural' threshold is part of what makes it easy for a small group to game the system. Setting a specific percentange of votes for a nomination to count makes it harder to game the system. However, it also makes the representation less proportional and the process less democratic.

#332 ::: Teemu Kalvas ::: (view all by) ::: April 09, 2015, 04:42 PM:

DavidS @228: The perverse incentive of trying to be hard to satisfy can be countered by only using those nominations which have all five entries filled for the 50% calculation.

#333 ::: kimiko ::: (view all by) ::: April 09, 2015, 05:07 PM:

I've been thinking about the problem of voters who can't suggest more than two or three nominees for a position that takes five. What if each voter gets 5 votes per category, that they can spend any way they wish? Here are some scenarios:

A slate voter would have to risk not getting their slate to fill all five positions if they spread their votes out. If they were content to merely spam one or two of the positions, they could probably manage that, but they'd be guaranteed that some of their "opponents" would be on the ballot too. In other words, they would "lose".

An anti-slate voter would may want to spam a single position - that is, spend all five of their votes on a single non-slate candidate of reasonably popularity to make sure they achieve their victory condition: at least one non-slate candidate on the ballot.

A slate-agnostic voter might use a mix of strategies, weighing the quality of the works they like against the power they'd want to assign them. Would obscure, but really good works prosper?

I'm interested in the Winter Soldier Problem too. If everyone knows that Winter Soldier is a good movie, AND of course it's popular, how do we make sure it's not over-represented relative to the rest of the ballot? In a 5-vote system does it make sense to spend five votes on it? Probably not. It would be reduced in power relative to it's perceived popularity.

So, what's going on here with having people estimate popularity and quality? If what we are trying to measure is "what fandom thinks is a worthy work", then instead of asking fans what they like, we should ask fans what fandom likes. We know that. Human primates are really really good en masse both identifying what other people in their group like and voicing their own opinion.

---

I think a good strategy would not be to try to defeat slate voters altogether, but to make it sufficiently difficult for them to dominate the ballot, so that more good stuff bubbles to the surface. Our victory condition should be that the ballot should contain all the same possibilities that it would on a non-slate year. If the ballot gets a little unwieldy, or if some of the slate works are actually good enough to be on there, I guess I'm okay with that.

#334 ::: Farseer2 ::: (view all by) ::: April 09, 2015, 05:07 PM:

Well, thank you everyone for this highly instructive conversation. I am leaving since I am already convinced that there is a good and fair solution to the Hugo bloc-voting problem (some variant of RAV). It would not only solve the Sad Puppies problem, but also any other special interest group having a disproportionate impact on the nominations, and I really like that it does so without shutting out those voting blocs or being unfair to them.

Unfortunately I don't believe the voting system will be changed any time soon (Worldcon fandom is extremely resistant to change), so we'll have more years of slates, counterslates, No Awards and all associated nonsense. If at any time Worldcon members decide that enough is enough, the solution is there.

#335 ::: Vivien ::: (view all by) ::: April 09, 2015, 05:21 PM:

Hi Kimiko (333)

Your system would reward monomaniacs who are there only to push one work and do not care about the rest. Or people who are not well aware of the field and have fewer stuff to nominate.

Please remember that before this slate-thing, previous issues with the hugo nomination process was more about group of people aggressively campaining for one precise work, not entire slates...

#336 ::: kimiko ::: (view all by) ::: April 09, 2015, 05:31 PM:

Right! Monomaniacs! Yeah...I seem to have overlooked a consistent feature of fandom, eh?

#337 ::: Lori Coulson ::: (view all by) ::: April 09, 2015, 05:38 PM:

Farseer -- it takes several years (3?) to get a rules change enacted for Worldcon, so any thing done this year wouldn't take effect until 2018.

We'd still have a cubic buttload of puppy piddle and manure to deal with in the meantime.

#338 ::: Vivien ::: (view all by) ::: April 09, 2015, 05:41 PM:

Not that I have anything againt monomaniacs :-).

But yes, think Wheel of Time last year. One of the line of defense of people supporting it was that they played the game fair, and nominated it along other stuff they liked in the year. Which is perfectly fine with me.

But don't tempt them, would you ?

#339 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 09, 2015, 05:49 PM:

Jameson Quinn @318:

Man, I wish I'd been more careful earlier when I was trying to promote my organization, ryrpgbybtl.bet. If instead of going into salesman mode, I somehow dreamed up some ironic meta joke, I might have avoided disemvowelment. And then this impressive bunch of people who, based on the last few comments, actually have a pretty great understanding of voting theory, would be exposed to our organization and its attempts to mprv pltcl vtng. W hv sch gd ntntns...
You know, there are days when that might slide past me. You'd have a few extra hours before Abi or Avram or Patrick spotted it -- or, more likely, before comments started appearing in the "Recent Comments" list that said "Sylvester sees spam, Thecla sees spam, Marcellinus sees spam" where they'd normally just show the commenter's name.

I get that you're excited about your website. That's a nice thing. And doing your best to make your comment a clever one? Definitely a move in the right direction. But next time you find yourself thinking it would be a good idea to post camouflaged ad links on my site, would you please remind yourself that I used to moderate Boing Boing, and have seen that trick more times than I can count?

(Your URL's in ROT13 now. You probably noticed that already.)

Aside from the ad links, you're more than welcome to stick around and keep participating. This world has no oversupply of people who are genuinely excited about the mechanics of voting.

#340 ::: Brad from Sunnyvale ::: (view all by) ::: April 09, 2015, 05:57 PM:

Rethinking all this brings me to a different (non puppy) related thought. We vote the hugos with an instant-runoff system. In the modern world of the interwebs, is it so necessary to have the runoff be instant?

For example, one could imagine a race where an instant runoff system eliminates all but 3 or even all but 2 nominees, and those are published, and there is a time gap for those who haven't read them to read them, and everybody gets to vote again, or at least everybody who didn't specify a preference before gets to specify one.

The theory is that with only 2 books to choose between, you can get people to actually read them. You could even ask (though not enforce) that everybody at least try to read them both or don't vote. Would they be honest enough? I don't know.

This fixes one of the classic flaws of IRV and other systems. You can get a case with two candidates, one very popular and one less so, and Popular gets 1,000 votes from people who read only it, and Obscure gets 500 votes from people who read both, and almost all of of those who read both liked Obscure more than Popular. Most people, even fans of Popular, would agree that Obscure should win.

Of course, in this case, in the ceremony, only the last 2 candidates are in suspense. The rest can go to the loser's party early.

As I said, just a thought when you open up the whole thing.

------------

The more I've thought about the big problem, the more I am thinking about juries to pick nominees. A hybrid approach I have been thinking about is delegation. Let each member of the convention support a delegate, and the top delegates (or any delegate who gets 50 supporters) becomes a juror. There will thus be a few Sad Puppy jurors, and a few jurors with political or genre agendas, but if most jurors are just people who read voraciously, who listen to the input of the members (both those who delegated and others); who go out and read suggestions of other jurors; who enjoy all the piles of free books you get as a juror :-) -- then I think they might be trusted to create a final ballot that is a decent list of what the ordinary members should be evaluating for the actual Hugo vote.

Yeah, there is history of agents, writers and publishers trying to sway jurors. I might try to delegate to folks who will not be influenced much by that. Or some might instead try to delegate to a juror who will nominate only my preferred genre. It doesn't matter, it will be very hard to game the system, I suspect.

#341 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 06:10 PM:

Bruce@328: The best reference on this area is probably Kilgour and Marshall: Approval Balloting for Fixed Size Committees; in DuBaffy, 2010. That laid out the various options of this type.

In terms of further discussions, such as which weighting sequence has which effects, it's mostly a matter of the voting systems experience I've gained as an active participant in the electorama mailing list for over 15 years. So no, I don't have specific references for every claim I made above. (My offhand knowledge of the academic literature on this stuff is better for single-winner systems than for multi-winner ones.)

@330: It's not actually easy to go back and virtually re-run old elections with new rules. Setting aside the changes in ballot format — for instance, RAV allows unlimited approvals per ballot, not just 5 — there is the fact that proportional systems fundamentally depend on more than just the tallies. You'd need all the old ballots to re-run the elecions; I doubt that is available for past years. (That's what my ad-hoc SRAV suggestion was about, making it more feasible to keep and re-check that information. I still think that proposal would work, I just don't think it's really worth the complications.)

@331: I think I understand what you mean, but I'm not sure. I think you're saying that unless there's a minimum vote count for winning, RAV doesn't actually guarantee that slates can't get more than their share of seats. For instance, if 100 puppies voted for works VWXYZ, then 900 others voted 45 each for works A-U, the puppies would get 2 slots under RAV, even though proportionally they only deserve half a slot (that is, one at most). But a threshold would only "fix" that problem by giving just one nomination that year (W). So I don't think that's a good idea.

@339: Wow, you _are_ a harsh mistress. I honestly thought that given the support I got from farseer2@280, you'd let that one slide. Just one correction: the thing I'm excited about that you put in rot13 is not "my website", but a 501(c)3 nonprofit for which I am a board member, and in whose name I offered assistance to any worldcon representatives who want to email me. So, I won't repeat the link, but anybody who thinks that voting methods are important and that it sounds as if I know what I'm talking about can easily un-rot13 (re-rot?) the link, and check us out.

(Note: Yes, I realized I was testing the line with the first two comments you modded, but I genuinely thought that you'd see it as farseer2 did. However, all playful scampery aside, I think this current comment is 100% in bounds, and I strongly hope you do not mod it. You mischaracterized my organization and I realize it was without malice but I think it's absolutely reasonable for me to respond and correct the record.)

#342 ::: Steve Halter ::: (view all by) ::: April 09, 2015, 06:14 PM:

Brad@340:There are already a number of fine juried awards. I think converting the Hugo into another one would loose something for the Hugo and not really add anything.

#343 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 06:15 PM:

On second thought, my comment @341 may not be 100% in bounds. But it's not because of spam; it's because the "harsh mistress" joke (or "joke"?) may have been in not-entirely-good taste. If that's so, I sincerely apologize; it wouldn't be the first time I put my foot in my mouth in search of "humor".

#344 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 06:17 PM:

After all, humor can be a hein line.

#345 ::: heresiarch ::: (view all by) ::: April 09, 2015, 06:45 PM:

It occurs to me that all the voting methods we have looked at so far are geared towards a different kind of problem than the one we are interested in. Most, if not all, are about determining one or more winners from among a finite, predetermined set of nominees, whereas the problem before us is assembling a set of nominees from the (functionally infinite) range of "all possible candidates." This is more akin to the primary system, or even the pre-primary system, than to final elections. There aren't really any voting systems geared towards getting from "everything" to "a set of nominees": consensus around a particular ballot is built through multiple iterations of a very chaotic and diverse combination of party machinery and public/back-room campaigning.

Given that parties and campaigns are a) the nigh-universal answer to how to go from "everything" to "a set of nominees" and b) widely agreed to be undesirable developments in the context of the Hugo, this is a problem worth trying to solve, but we should recognize that a pseudo-primary voting system answers a different need than voting systems typically need to. In addition to being bloc-voting resistant, it should be robust against a very long tail, and hopefully provide some opportunity for consensus-building along the way.

#346 ::: Brad from Sunnyvale ::: (view all by) ::: April 09, 2015, 06:49 PM:

Steve #342 -- Well, the clear thing it "adds" is that the general nomination approach used right now appears, in the view of many, to have been corrupted and no longer workable. So yeah, there's a reason to look at alternatives.

I don't see the reasoning of "there are other juried awards." There are other vote-nominated awards too. There are lots of types. What matters is not so much the methods, but which people's tastes are driving the award. The goal is to find the work that is the best representation of our (WSFS members) aggregate opinion on the best work of the year. The standard is "how well does the method attain the goal?"

We want to know what fans like best if they get a chance to actually look at things. (If we just wanted their raw opinions, you could use simply total nominations in the nominating ballot to award the Hugo, or book sales, for that matter.) The goal of the nominating ballot is to produce a small pool of the likely contenders for best of the year, so that the larger group of fans have the chance to actually look at that small subset, and form their opinions, and then the system measures those opinions to produce the winner.

One way to produce that small pool of contenders is the nomination ballot, but that always was vulnerable to collusion, and now that is being done, so it's time to find another way, probably. You need a way that either blocks collusion, or expects it and handles it. A delegate system as I discuss welcomes the Sad Puppies getting together and having 3 or 4 delegates -- it's an essential part of it, actually. They can't so easily break something that embraces what they seek to do.

#347 ::: Buddha Buck ::: (view all by) ::: April 09, 2015, 06:53 PM:

Jameson @343: Rember Scalzi's Law (or at least one of them): The failure mode of 'clever' is 'asshole'.

While I can't speak for TNH, obviously, I think the "harsh mistress" line will be OK, in part because you have been a good contributor, despite pushing the line with trying to drive traffic to your site after being warned about it. You aren't someone who comes in and just posts links, or just insults our hosts.

I did visit your site, and it seems to be geared toward political election reform, and mainly pushes two election methods as "best", neither of which are IRV, STV, etc. The relevance of the site to this discussion is unclear (and this is not an invite to clarify).

#348 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 07:07 PM:

Brad@340: I agree; IRV is not the best system for the final round of voting. It's confusing that voting something below "no award" can actually help it win; and IRV is definitely subject to premature elimination (in politics, we call that "center squeeze").

If I had to recommend a single-winner system for the Hugo, I think I'd go with Majority Judgment¹. In this system, each voter independently grades each candidate using one of the pre-supplied grading categories, and the highest median wins (with a natural tiebreaker for equal medians). Since the Hugo has a "no award" option, it would be easy to say that any median below the second-lowest rating option means "no award".

Usually, if you're using it in a political context, the available grades are the letter grades A, B, C, D, F. However, this would mean that a book could win a Hugo with a median grade of "D+". While in one sense the letters are just labels, that still doesn't seem right. So I think a bit of grading-label inflation is in order: name the categories something like "Peerless", "Outstanding", "Excellent", "Deserving", and "Undeserving". Or maybe the top three categories could be "With Highest Honors", "With High Honors", and "With Honors". Then, the winner would have to be at least "Deserving+".

If you don't like MJ, there are other good systems: Range voting, approval voting, Condorcet, etc. In general, these are either more subject to strategy than MJ, or harder to explain and tally. All are clearly better than IRV, though.

¹Actually, I said "Majority Judgment" because it's a well-documented proposal; but there is actually a closely-related system which is slightly better for this case, which I call "graduated majority judgment" or GMJ. These are both "median" or "modern Bucklin" systems and both would give the same results in almost all cases, but GMJ can deal a bit more smoothly with abstentions.

#349 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 07:15 PM:

I realize I didn't explain how abstentions would work in my message @348. The idea is that leaving a nominee blank would be equivalent to rating it "undeserving"; but a voter could choose to explicitly mark "abstain" next to a nominee. Such explicit abstentions would not count in determining the median; so the median of "Highest Honors", "Honors", "Undeserving", "Abstain", and "Abstain" is "Honors", not "Undeserving".

#350 ::: Brad from Sunnyvale ::: (view all by) ::: April 09, 2015, 07:16 PM:

JQ, there are a few good systems, and STV is probably now accepted as the worst by election theorists, but all of them have the nice attribute that strategic voting is not practical.

The main point of my post was to say that we should recheck all our assumptions from back in the days the Hugo was designed. Today, nothing stops a real runoff, even a full 4 round runoff if you really want it, other than it being more work for the voters.

I think the current system does decently well, actually, even though like most I don't agree with the results all of the time.

I am actually more concerned with the nomination issue at hand. More and more often I am seeing what I judge to be the best work not even nominated, let alone winning. This may be a sign I am diverging from other fans, or it may be a sign of trouble.

Examples I would give are Super Sad True Love Story, which I presume got almost no nominations because it is not by an SF author, does not have an SF title or cover and was not marketed as SF, but was to my view the best SF novel of that year by a large margin. Another non-nominee, the Quantum Thief, was marketed as SF but by a newish writer. I was even more surprised last year at a British Worldcon, to see Black Mirror get no significant nominations, though it is to my view the best SF TV show, possibly ever.

Or, as I said, my tastes may be just getting too different. It happens.

#351 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 07:17 PM:

BuddhaBuck@347: Understood; I'll let people judge for themselves (if they care).

#352 ::: Steve Halter ::: (view all by) ::: April 09, 2015, 07:17 PM:

Brad@346:The current raw nomination approach does seem to be broken. Several of the voting systems under discussion seem like they could fix many of the current systems weaknesses.
The jury approach doesn't appeal to me personally for the Hugo's and also doesn't seem to really fix the problem. The process of choosing the jury seems that it could possess all of the flaws of the current system in whatever process is used to select the jury.

#353 ::: Jameson Quinn ::: (view all by) ::: April 09, 2015, 07:27 PM:

Brad@350: " there are a few good [single-winner] systems [for the post-nomination process], and STV [aka IRV (?)] is probably now accepted as the worst by election theorists..."

Well, of course, election theorists pretty universally agree that the absolute worst system is actually plurality (aka FPTP, the system most English-speaking places use for politics). I understand you meant that IRV was the worst except for plurality, and while I personally would pretty much agree that that's the case, I have to say that's not the consensus among theorists. For instance, at the Voting Powers and Procedures conference at the London School of Economics, in a poll of 22 election theorists, the favorite system was approval (approved by 16), the worst was plurality (approved by 0), and IRV was actually counted better than most (approved by over 10, IIRC).

Again, I think that IRV is generally not so great, and for the Hugos I'd actually recommend MJ. But I don't want to misrepresent my recommendation as a consensus.

#354 ::: Pandora's Lox ::: (view all by) ::: April 09, 2015, 08:15 PM:

Bruce @242:

Yes, I know your original argument was against "political party"-style voting. I was trying to make a slightly different point.

Rather than arguing for two parties, what I was trying to argue was that with multiple, differing slates, they would cancel each other out and basically return things to the status quo. Voting for the Hugos is small enough that you are destined to get a lot of noise, regardless of the system, especially when voting is open and relatively small. But I think having a large number of parties, rather than trying to pretend parties shouldn't exist, is the simpler and more effective answer.

#355 ::: A.J. Luxton ::: (view all by) ::: April 09, 2015, 08:24 PM:

My partner just suggested an addition to ballots and nomination ballots which would be psychologically useful: a simple checkbox by each line item. "Have you read this work? Y/N."

Yes, lying would still be possible, but it wouldn't be effortless.

#356 ::: Brad from Sunnyvale ::: (view all by) ::: April 09, 2015, 09:14 PM:

Some of the bloc this year are principled, some are not, so a pledge of having read the works, and even a pledge that the ballot is your own independent opinion and is not cast because you know others will be casting the same thing -- these could make a difference. But probably not enough.

But Steve Halter, the reason a delegate jury does not get affected by slates so much is it removes the value of collusion to overwhelm the nominations. Everybody is pooling their will via their delegate, including the slate voters. They are no longer anybody special, they no longer have special power to warp the results.

That said, I would not want to see too many of the delegates be political. Some delegates might say, "Hey, I will only nominate paranormal romance" or Steampunk or whatever. But I suspect most of the delegates would actually feel they have taken on a duty, to look at recommendations, to read works, to share recommendations with other delegates, and to truly as a group come up with a set of 5 great works for the community to consider in the final ballot. As I see it, as long as most of the community supports this approach, the fact that a small portion of the delegates want to grind axes does not corrupt the process. I see the delegates openly discussing things with one another. Not colluding, but acting in the interests of great SF, saying, "Hey, I just read this great book you might not have heard of" and others saying "thanks!" Most delegates would be established fans, rewiewers, critics and voracious readers with a love of the genre, and they would not get the support of members otherwise.

The job of the jury is not to pick the winner, but to make sure really great work does not escape the attention of the whole community on the final ballot. In fact, I see no problem with the jury releasing an extra set of suggestions; these didn't make the final but we suggest you read them.

#357 ::: Steve Halter ::: (view all by) ::: April 09, 2015, 10:02 PM:

Brad@356:So, there would be an election stage to elect these juror delegates?
That seems like it would add a lot of complexity. There would have to be lists of people who want to be delegates and really a whole election apparatus. It might solve the slate problem but seems like it might introduce problems of its own.

#358 ::: Brad from Sunnyvale ::: (view all by) ::: April 09, 2015, 10:09 PM:

There are a lot of ways one could do that. It need not be super complex. Potential jurors could apply and get an ID. Any con member could specify what ID is their delegate, using their Hugo PIN. We might even do the delegation the year before so you are a juror during the year, while you read. (It's a nice perk, publishers would send you all their books free.)

Yes, a bit of programming. Not too much. Once you have the jurors -- probably about 50 of them, I would guess -- it's a small enough number that they could even do their choice by hand or a simple program.

#359 ::: Tammy Coxen ::: (view all by) ::: April 09, 2015, 10:18 PM:

From an implementation perspective, timing is a challenge to any proposal that would add additional rounds of voting. You need to allow people enough time to nominate (without overly disadvantaging works published late in the year) and enough time to read the nominees before voting. The Hugo Administrators already struggle to strike the right balance, and adding additional rounds would be problematic for both the administrators (who are all remember that all these people are volunteers, with real lives and jobs) and the voters.

#360 ::: P J Evans ::: (view all by) ::: April 09, 2015, 10:20 PM:

For whatever it's worth, I may, in a week or two, have access to actual datasets. (Frisbie still has them from 1984. And, he thinks, 1972. He was going to trash them in another month. Timing is everything.)

#361 ::: Brad from Sunnyvale ::: (view all by) ::: April 09, 2015, 11:16 PM:

Tammy, what I'm thinking is that delegates for the 2018 worldcon would start to be selected late in 2016, though as more people become members, more delegates would be named by these new members (and old members who did not yet delegate) throughout 2017. Jurors would be busy reading all through the year and after, but no new delegates would be named after January of 2018. By this time the jurors should have examined most of the works of 2017 they would consider. They would then ponder and create the final ballot, to be released later in the year. A juror named on Jan 31 2018 would presumably have already been reading for a while, or they would have a lot of catching up to do.

So no additional rounds of voting in this approach. If you are referring to the thought that you need not do instant runoff, that does involve additional rounds of voting, but it's all during the final ballot phase, with a few weeks before the final round to evaluate the final 2 works, so that ideally, every voter in the final round has read them both, if they haven't already. (They have known them since the final ballot was released many months prior.)

There is no reason with computerized voting the final round could not take place even during the worldcon. It's not like it's that hard to get engraving done in a day.

#362 ::: P J Evans ::: (view all by) ::: April 09, 2015, 11:22 PM:

362
You underestimate the amount of time (and work) involved.

#363 ::: PhilRM ::: (view all by) ::: April 10, 2015, 12:19 AM:

This seems relevant here:

http://james-nicoll.livejournal.com/5302561.html

#364 ::: Brad from Sunnyvale ::: (view all by) ::: April 10, 2015, 12:45 AM:

#362 -- I've run online votes, and auctions, written my own software to do Hugo style votes and much more, and managed projects much greater than this, so yes, I do know the amount of time and work involved, and I don't think it is trivial but it is much closer to trivial than to intractable.

(Note I don't know if you're talking about delegates or multi round voting. I am mostly talking about delegates.)

However, that's beside the point. What matters is finding a good system that meets the goals I outlined in #231, goals that others have also described. Once we find that system, I do not think it is likely the implementation cost will be a major barrier, though of course a system that's almost as good but much easier to implement might see favour.

The more I talk with SPs, the more it's clear that some of them are not just griefers, they really feel they are right, which means this doesn't end, and there is no choice but to replace the award system.

#365 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 10, 2015, 01:40 AM:

Jameson Quinn @341:

@339: Wow, you _are_ a harsh mistress.
That's "advocate of firm moderation" to you.
I honestly thought that given the support I got from farseer2@280, you'd let that one slide.
I"m the most senior moderator here. In matters of user behavior, my opinion outweighs farseer's. Also, you might want to take a closer look at the site URL.
Just one correction: the thing I'm excited about that you put in rot13 is not "my website", but a 501(c)3 nonprofit for which I am a board member, and ...
And you posted it inappropriately on my site when you were on your second warning..
Yes, I realized I was testing the line with the first two comments you modded, but ...
Stop testing the line, please. I'm being very patient. Ask anyone who knows me.


===


We're shutting down soon. I'll see you in the morning.

#367 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 07:37 AM:

I'm sorry. Just because this place kinda reminds me of the good parts of usenet, doesn't mean I should put on the bad parts of who I was back in the usenet era. Respect to you, and it won't happen again.

#368 ::: Bruce Schneier ::: (view all by) ::: April 10, 2015, 08:07 AM:

Buddha Buck @315:

"But I'm not recommending Condorcet for nominations."

I agree; it's not suitable for what we want.

Pandora's Lox @354:

"Rather than arguing for two parties, what I was trying to argue was that with multiple, differing slates, they would cancel each other out and basically return things to the status quo. Voting for the Hugos is small enough that you are destined to get a lot of noise, regardless of the system, especially when voting is open and relatively small. But I think having a large number of parties, rather than trying to pretend parties shouldn't exist, is the simpler and more effective answer."

Voting blocs won't cancel each other out. They'll battle each other, and dominate any non-bloc votes.

#369 ::: Bruce Schneier ::: (view all by) ::: April 10, 2015, 08:07 AM:

dh @321:

"Did you end up posting a study of the data available that shows the correlation between the published slates and the actual results?"

No. I don't believe the proper data is available yet.

nathanbp @329:

"Bruce, 327: Isn't the more interesting statistic how many people tend to vote for more than one of the nominees? I don't think anyone has released that data."

Yes, that would be interesting too.

I wonder what it would take to get the Hugo Administrator to release the detailed data early this year. I know the standard practice is to release it after the awards, but it would be useful to have it before the business meeting.

Vivian @330:

"An advantage of this system (and of the system presented in 1, by the way) that has not been presented yet is that it could be easily proof checked on old Hugo data as it doesn't require a modification of the way the ballot is filled... As long as those data has been kept unaltered after the counting. How better to assess the potential of the voting system then to test it on real life data?"

Yes, it would be both interesting and useful to test some of these systems against real data.

#370 ::: Bruce Schneier ::: (view all by) ::: April 10, 2015, 08:09 AM:

Jameson Quinn @341:

"Bruce@328: The best reference on this area is probably Kilgour and Marshall: Approval Balloting for Fixed Size Committees; in DuBaffy, 2010. That laid out the various options of this type."

For those who want to read it, the link is here.

#371 ::: Bruce Schneier ::: (view all by) ::: April 10, 2015, 08:09 AM:

Brad from Sunnyvale @350:

"The main point of my post was to say that we should recheck all our assumptions from back in the days the Hugo was designed. Today, nothing stops a real runoff, even a full 4 round runoff if you really want it, other than it being more work for the voters."

I worry about changing more than absolutely necessary, even given the problems with IRV.

And a discussion of making the nomination process juried is not within the scope of this thread. Changing the electorate is an important change to consider, but not here.

#372 ::: J Thomas ::: (view all by) ::: April 10, 2015, 08:31 AM:

I want to explore an idea which was mostly rejected.

Say that you eliminate the nomination stage. Put the whole voting in a database that voters can query. Any voter can nominate one work per award, and vote for it.

Since the system tracks voters (but doesn't share their identities), each voter can change his vote at any time.

So if you want to, you can leave your vote on your nomination the whole time even if it gets no second vote. But you might want to switch to the leading candidate, or the second-place one, or whichever you want.

People who have a slate can each vote for one. That's fair.

If 10% or 30% vote for a slate's choice, that's what they choose to do. If 11% or 31% of others can agree on an alternative they like better, that's what they choose to do. There's nothing particularly wrong with this.

There's a *whole lot* of room for strategic voting. If you see that your first choice will not win, you can switch to something else you like. There's nothing wrong with that.

There would be some challenge to figuring out how to display the results. If for example you display a list with the top 5 at the top, then the ones that get into the top 5 early have a great big advantage. That turns into an advantage and a disadvantage for cabals who want to push a particular item. They get attention early, but they also get attention from people who want to stop them. It ought to be possible for a candidate that doesn't get many early votes to still be viable, to get more later. There would be some tendency for people to ignore anything that didn't get ahead at the start, and I don't consider that a good thing. But statistically, if there are 5 candidates that are each 30 or more votes ahead of yours, yours is unlikely to win regardless.

There would be a security challenge. Of course you don't want your results hacked or voters' information revealed.

You couldn't very well give people free copies of everything. So things that not many people have read would be at a disadvantage. The old way, if few people have read something but it somehow gets past the nominations, then they can read it at the last minute and vote for it.

But the really good point is that you can entirely skip the nomination stage that is so gameable. It's mostly transparent. Nominate anything you like. Vote for anything you like. Switch your vote to something else you like whenever you want. Your vote counts as much as anybody else's. Fair, open democracy as long as we trust the software to do what the rules say it does.

If you don't trust the software, you should have the option to reveal your name. When ten people who choose not to be anonymous vote for a particular candidate, then that candidate can't get less than ten votes. Or eleven, if any non-anonymous people have also voted for it.

This would be a big break with tradition, but it's simple, obvious, and fair. It's worth some thought.

#373 ::: Bruce Schneier ::: (view all by) ::: April 10, 2015, 08:38 AM:

J Thomas @372:

My problems with this are twofold. One, it's a much bigger change than we need to make. And two, it doesn't really map with the way people vote. Much of the electorate use the list of nominees as a reading list. We want to finalize this list a few months before the WorldCon.

#374 ::: Andrew M ::: (view all by) ::: April 10, 2015, 08:39 AM:

Brad from Sunnyvale: I think we may in the end need a juried award to preserve the virtues which the Hugos have (imperfectly) exemplified; of reflecting a conversation among those interested in SF; of finding interesting new works, works with cross-group appeal, works that stand out rather than just being typical of an author or field. But I don't think we could get away with calling it a Hugo. The Hugos have invested a lot in being the fans' award, the people's award, the award anyone can take part in. (GRRM notes he may be partly responsible for the current crisis by emphasising this.) This provides the standard by which they are often judged, and of which they are often seen as falling short. There's nothing intrinsically wrong with an award given by 2000 people, but given the way the Hugos are promoted this is thought of as a defect. Any attempt at limitation would provoke cries of 'See! What did I tell you? A tiny clique!'.

I think more and more that the idea I proposed in the other thread, of splitting the Hugos into something wider and something narrower, may be the only long term solution.

#375 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 08:47 AM:

Brad@350: I'd say that the problems with IRV are not in the "instant" part but in the form of the "runoff" part. IRV is a virtual sequence of plurality runoffs. Plurality is a bad system, and approval voting dominates it (that is, is better in every way). If you wanted an explicit sequence of runoffs, do that with approval; you'll get the same deliberation as with plurality runoffs, but without the dishonest strategy (see below for parenthetical caveat). If you wanted a virtual series of approval runoffs, that's MJ or a similar modern Bucklin system; which is one reason I suggested such.

(There is zero dishonest strategy in one-round approval; it has strategy, but only semi-honest strategy. This is also true for MJ, which amounts to the same thing as an instant approval runoff. However, in multi-round approval runoffs, there is in principle a possibility of dishonest strategy — "turkey raising" — but in the context of an award, where there aren't clear "ideological alliances", I don't see how such dishonest strategy could be viable in practice.)

So: If there are real problems with the final round, I'd suggest MJ as the fix. In particular, it can do a better job with voters who abstain from voting on one given work, while still voting on the others. However, if that system ain't broke, don't fix it. In particular, it's probably not the best time to fix it this year, when everybody is worried about the puppies. Bad cases make bad law.

On the other hand, the nomination process needs fixing ASAP, and I still think RAV is the solution there.

#376 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 08:51 AM:

Bruce@370: Thanks for posting the link to Kilgour and Marshall. Unfortunately, that paper is pretty impnetrably technical from a layman's point of view. Also, it assumes the D'Hondt weightings, without considering the Saint-Lagüe ones, even though both alternatives should have been obvious.

#377 ::: J Thomas ::: (view all by) ::: April 10, 2015, 09:27 AM:

Bruce Schneier @373

"My problems with this are twofold. One, it's a much bigger change than we need to make."

You of course should choose how big a change you want to make. I'm reminded of an old riddle that goes "How can I make a hole in my garden wall so that my chickens can get into my neighbor's garden but my neighbor's chickens can't get into my garden?"

It sounds like the point of this exercise is to find a way to keep the wrong nominators from having too much influence, while still making sure the right nominators have enough influence.

It might possibly work best to just let the influence spread organically.

"And two, it doesn't really map with the way people vote. Much of the electorate use the list of nominees as a reading list. We want to finalize this list a few months before the WorldCon."

That's possible. Open the nominations to people who are already eligible, as many months as you like before the WorldCon. People make their nominations. Then any time before the deadline, they can change them. So if your nomination is #100, and you think #9 is the best of the top 20, you might change to #9. At the deadline, take the top 5. We don't need complications with runoffs and transfers -- people change their choices whenever they want.

Then you can publish not just the top 5 but all the nominations and anybody who wants reading recommendations can have lots.

Similarly for the election. Vote for the one you think is best. Ten minutes before the deadline if your choice is #5 but you think #2 is much better than #1, change your vote. Or if you don't think it really matters between them, stay with #5 and it might possibly win.

Probably better to have a ten minute delay on publishing results, so you don't publish misleading data if people start voting a whole lot and slowing the system down. Particularly in the last ten minutes before the deadline.

Everybody gets one vote which they can change when they like. Strategic voting changes from a bug to a feature. All very simple, and I don't see how it could be fairer.

Of course it may not fit your needs and that's fine. I just want to keep it in mind, when we look for flaws in the complicated approaches.

#378 ::: dh ::: (view all by) ::: April 10, 2015, 09:28 AM:

Bruce--

Thanks for your reply. Way back up on #6, I wrote:

2. There is supposition that the slates won because of bloc voting, but before that is known, don't we have to analyze the balloting and look to see what correspondence the slates have to votes cast? Meaning, there is not yet firm evidence (in terms of science, we can all have strong gut feelings), that slate voting took place. Based on the voting information released so far, do you believe there is data to support that bloc voting according to the slates occurred?

You then wrote:

Yes. I think it can be mathematically proven that bloc voting has occurred, even without seeing the actual ballots. I was writing something on that, and will post it later today.

And now:

No. I don't believe the proper data is available yet.

Are you now saying that this initial feeling was incorrect and we are back to supposition that a block of slate voting has occurred? I ask because there are several unexplored scenarios that change the nature of the discussion. One such scenario is that the two related slates were not followed rigidly, and instead, were used as a guide for voting, that voters deviated it from as they saw fit. We should be able to work out how correlated the slate were to actual ballots. If it turns out that there incidence of straight block voting is not especially high (meaning, in line with historical norms about variation in the ultimately selected nominees and uniformity in the ballots that lead to a work being nominated) then this conversation about changing the rules is untimely and ill considered. IF (and it's a big if) the variation within the ballots is in line with historical norms, the nature of what has occurred is vastly different than the current unscientific assumption. IF (and it's a big if) that's the case, all we have is an electorate problem, and not a modeling problem. Do you agree?

#379 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 09:35 AM:

JThomas@377: I agree with Bruce that your proposal is probably unworkable. But even on its own terms, there is no reason to restrict people to one vote each. That's just asking for lesser-evil dynamics; a situation where an early lead turns into unbreakable advantage. If Worldcon did decide that continuously publishing the votes was the way to go, the right way to do it would be with an approval election, not a plurality one.

#380 ::: J Thomas ::: (view all by) ::: April 10, 2015, 09:44 AM:

#378 dh

If it turns out that there incidence of straight block voting is not especially high (meaning, in line with historical norms about variation in the ultimately selected nominees and uniformity in the ballots that lead to a work being nominated) then this conversation about changing the rules is untimely and ill considered. .... IF (and it's a big if) that's the case, all we have is an electorate problem, and not a modeling problem. Do you agree?

I very much disagree.

What would you expect to happen if you found an exploitable bug in Firefox, and published it, and then they checked and saw that no one had exploited that bug yet so they decided it did not need to be fixed?

If the Hugo nominating process has a bug that allows block voting to give us bad results, and that bug can be fixed, then it needs to be fixed even if it takes 2 years for the update.

We've gone to a lot of trouble telling the Sad Puppies that they can break the system with block voting. If they haven't done it this year, they'll try it in the future. And if they don't do it, somebody else will.

You don't leave a published exploit unfixed just because it hasn't caused a disaster yet.

#381 ::: J Thomas ::: (view all by) ::: April 10, 2015, 09:52 AM:

#379 Jameson Quinn

If Worldcon did decide that continuously publishing the votes was the way to go, the right way to do it would be with an approval election, not a plurality one.

Thank you! I had not thought out the details, but my immediate thought now is that you are right. Approval voting looks obviously better.

I was thinking about one vote because the winner is supposed to be the *best*, so it made sense to me that each voter would choose the one he was willing to say was the best.

Approval voting seems like it would get a better result even though each voter would not be choosing the one he considers the best, but only choosing all the ones he thinks are good enough to win the award.

Perhaps if multiple candidates get more than 50%, we might consider them all winners? If none get over 50%, declare it No Award?

#382 ::: Idumea Arbacoochee, Gardener of Threads ::: (view all by) ::: April 10, 2015, 09:59 AM:

I've cleaned up duplicate comments, requests for removal of duplicate comments, and affected references. The thread now is perfectly operational and all its references are functioning perfectly.

Also, I know that some members of this conversation do have particular interests in the outcome of this set of results. Although this may drive the particular questions asked, it's worth trying not to make them too pointed. Keep it constructive and I won't have to single anyone out for more...blunt moderation, OK?

#383 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 10:22 AM:

@381: The idea of multiple winners seems like just change for change's sake; I think skepticism is warranted.

In general, what you're now suggesting is a continuous series of approval runoffs, with no elimination. Majority Judgment can be thought of as a virtual continuous series of approval runoffs with no elimination and monotonic ballots (that is, in which voters can only add approvals, not remove them). So there's not a huge gulf between the proposals.

But as I said above, right now, when the nomination process is broken, is probably a bad time to tinker with the non-nomination process. It's perfectly OK to think about it and discuss it in this thread, but I wouldn't suggest making any official motions on this this year, and probably not next year either.

#384 ::: dh ::: (view all by) ::: April 10, 2015, 10:26 AM:

We've gone to a lot of trouble telling the Sad Puppies that they can break the system with block voting. If they haven't done it this year, they'll try it in the future. And if they don't do it, somebody else will. You don't leave a published exploit unfixed just because it hasn't caused a disaster yet.

The problem with this mindset is:

1. It's not scientific.

2. We don't even know what the problem is, with clarity. It's just as likely that any changes will create more exploits. In your analogy, you can't fix a software bug properly without reproducing it first. At this point, in the analogy, we don't even have a good bug report yet.

3. The risk is as high or higher for making a hasty, bad decision in time for this years business meeting. If the data is not available until after the ballot this year, that's part of the process and it's a feature, not a bug.

Getting any fix right is more important than doing something for the sake of it.

#385 ::: Buddha Buck ::: (view all by) ::: April 10, 2015, 10:40 AM:

I wonder if the effects of slate nominating could be mitigated by two changes:

1. Make the list of nominated works (and nomination counts) open during the nomination period, so that if a slate is at work, it can be seen and countered during the nomination period by other nominators, while keeping the "nominations are a good reading list" feature.

2. To counter the "then the slators will just dump their nominations at the last minute", have a random close time of nominations.

A statement in the opening of nominations announcement like "Nominations will close at a predetermined random time between 2015-03-05T00:00:00UTC and 2015-03-15T00:00:00UTC, as witnessed by the following MD5sum: c23f65af12aec20545c0ff8afb5b881c", followed by a statement at the close like "Hugo award nominations are now closed. The predetermined random time for this was 'The 2015 Hugo Award nomination period will end at 2015-03-11T14:23:41UTC. Correct Horse Battery Staple.'.".

Anyone who suspects that the administrators deliberately closed the nominations to lock out last-minute slate nominators can verify that the MD5sum of the string "The 2015...Staple." is indeed the c23f65... number given.

I'm sure this system can still be gamed, but it might be an effective and simpler solution than trying to change the method of tallying nominations.

#386 ::: nathanbp ::: (view all by) ::: April 10, 2015, 10:47 AM:

Bruce, #369: It might be easier to get last year's data (or some previous year) than to get this year's early. It would then be fairly easy to simulate the addition of a slate.

dh, #384: I'm not sure why you think we haven't identified the problem? It seems pretty clear to me that the problem is that because of the voting patterns, an organized small minority can sweep the nominations. The solutions are either to change those voting patterns by running a counter slate (which seems unacceptable to most commenters here) or change the system to reduce the power of slates. The problem with the old system that is being exploited is that everyone was given 5 votes each, a large majority of which were "wasted" each year. By forming a slate, the Puppies were able to use all 5 of their votes. Thus the discussion has mostly centered on reducing the effective number of votes per ballot.

#387 ::: Q. Pheevr ::: (view all by) ::: April 10, 2015, 10:53 AM:

Buddha Buck: A random close time for nominations seems like it would make things harder not just for voters who are trying to game the system, but for all voters. Anyone who wants to make sure that their vote will count has to vote before the earliest possible close time, and if a big stack of slate votes comes in just then, there might or might not be time to counter it.

#388 ::: J Thomas ::: (view all by) ::: April 10, 2015, 10:54 AM:

dh @384

"You don't leave a published exploit unfixed just because it hasn't caused a disaster yet."

The problem with this mindset is:

1. It's not scientific.

That makes no sense to me.

2. We don't even know what the problem is, with clarity.

Ah! I had supposed that you had models which showed that slates are bad. Are you saying that there are no such models? That people are just making up things the SPs may have done to try to create arguments that they have done something bad? That the argument against slates is itself unscientific and bogus?

I read people saying that one slate is bad and two slates are as bad or worse. But now it turns out that the problem isn't slates, it's whatever it is the SPs are doing?

It's just as likely that any changes will create more exploits.

Sure, you need to watch for that. If slates are a disaster waiting to happen, we don't want to just substitute some new disaster that might be worse. But the question is not whether SP actually used a slate, and if they didn't there's no problem to be solved. Unless in fact the problem is to stop SP and we haven't yet decided what it was they did that needs to be stopped.

In your analogy, you can't fix a software bug properly without reproducing it first. At this point, in the analogy, we don't even have a good bug report yet.

So -- let me get this straight -- you're saying that we in fact don't know whether slates are bad for the system. We'll only find out whether they are bad when we find out whether that's what SP did.

I'm kind of flabbergasted. We've had hundreds of posts about how to fix the inherent problems of block voting. How to keep a relatively small group of block voters from dominating the nominations and restricting the choices. And now you tell me that block voting is OK after all unless the SPs did it worse than usual. And you tell me that I'm being unscientific.

3. The risk is as high or higher for making a hasty, bad decision in time for this years business meeting. If the data is not available until after the ballot this year, that's part of the process and it's a feature, not a bug.

Getting any fix right is more important than doing something for the sake of it.

This I agree with. Better not to rush a bad fix. It sounded before like you were saying that known bugs the SPs haven't exploited (yet) are not worth fixing, which sounded utterly weird to me.

People had been talking like block voting was a known bad thing. I did not realize it's only bad if SP does it.

#389 ::: P J Evans ::: (view all by) ::: April 10, 2015, 10:54 AM:

369
Would a past year do? Because it's possible I can have that in a couple of weeks. (It would do for a start, anyway.)

#390 ::: P J Evans ::: (view all by) ::: April 10, 2015, 10:58 AM:

388
I am not entirely sure that dh's goals are compatible with the goals of the people who vote on the Hugos.

#391 ::: CHip ::: (view all by) ::: April 10, 2015, 11:01 AM:

dh @ 384: the WSFS rules prevent a bad decision from being rushed. Aside from the fact that people have 4 more months to file motions, anything that is passed this year must be ratified next year. If a system is passed and people in the next year conclude that something better is possible, the system approved in Spokane can be replaced in KC without ever having been used.

There's also the question of how much damage the Hugos take if nothing at all is done now that there's a clear problem, given that any fix will be delayed until at least 2017.

#392 ::: Cadbury Moose ::: (view all by) ::: April 10, 2015, 11:08 AM:

P. J. Evans @ #390

You might think that, I couldn't possibly comment.

#393 ::: John A Arkansawyer ::: (view all by) ::: April 10, 2015, 11:13 AM:

Spurred by but not addressed directly to P J Evans @ 390: Y'all are still way too freaked out by this to be making long-term decisions on deliberations that can't even begin till August.

Seriously.

Let me isolate one thing that has been said earlier: That some of the folks who have commented here might be trying to distract you from something being done elsewhere.

What, exactly, could be being done elsewhere that you need to counter RIGHT THIS MINUTE?

(Yes, that is a rhetorical question. No, I don't think there is anything.)

There is a contest for that kind of thinking, and we all--including me--have the skills to enter it.

I agree this is a serious issue. I disagree that anyone's hair is on fire.

What the Puppies have succeeded big time in doing is freaking people out. Don't keep feeding that!

Yes, DO think about the future. You have the attention of some world-class people in the field of information integrity here. Use it effectively! Don't piss it away.

#394 ::: Idumea Arbacoochee, Gardener of Threads ::: (view all by) ::: April 10, 2015, 11:14 AM:

Let's back off of the personalities a little, please.

And dh, I hope, given the way your participation in the previous thread went, and given some things I've seen and correlated in the back end, that you will make the effort to be a constructive and useful member of this conversation.

#395 ::: JonW ::: (view all by) ::: April 10, 2015, 11:22 AM:

@#385, 387: My problem with the proposal #67 & #385 is that it seems to assume that ordinary voters have nothing better to do with their time than to keep a browser tab open to the nominations page and monitor it so as to be able to counter the machinations of other nominators. That’s not what we want, I think. We want people who love SF to be able to cast nomination ballots listing what they believe are the best works of the year, without having to worry more than is unavoidable about strategic voting or voting strategy or countering other voters’ slates, and then have the resulting list do a good job of reflecting the community’s views. If we can agree that that’s our goal, then we're back on the question of what’s the best vote-tallying system for that purpose.

#396 ::: Bruce Schneier ::: (view all by) ::: April 10, 2015, 11:28 AM:

dh @378:

"IF (and it's a big if) that's the case, all we have is an electorate problem, and not a modeling problem. Do you agree?"

My goal in this thread is to discuss possible changes in how the Hugo nomination voting process works. I have specifically excluded two very important issues: 1) should something be done, and 2) should changes be made in the electorate.

So whether I agree or not is not something I will speak to here.

#397 ::: Vivien ::: (view all by) ::: April 10, 2015, 11:32 AM:

@PJEvans @Bruce

I think any past year would do, as long as someone can remember what the discussed work were all about and identify meaningful change in the nominations, from the old system to the new.

1984 or similarly old could be a bit painful, because I guess the data will not be in numerical form, and that the testing would imply typing in the adequate format a shoebox full of ballots... Anyway at this point I am sufficiently excited about the subject that I might do my part if it comes to that. You could also argue that fanhood structure and diversity of taste has evolved quite a bit since that time.

To me it would mainly play the role of an illustration to demonstrate the inocuity, or the advantage of the new system, indepedently of the presence of slate. I think the argument that the systems we are discussing are good barrier to slates are solid enough to stand on their own. But I feel we need to know more about what it does to naive fan voting.

#398 ::: Vivien ::: (view all by) ::: April 10, 2015, 11:43 AM:

@Jameson Quinn (341)

Yes, I did know that a re-run would require the full ballots. And that this information is unlikely to be kept for long in real life. But you forgot a factor: this is an election organized by serious data geeks. And it seems that PJEvans could prove me right :-).

I don't think the fact that the actual ballot are limited to 5 would be a strong limitation to a test. You presented yourself that diluting your preference on long nomination list was a weak strategy to defend your tastes, and I wouldn't expect ballot nominating 5+ works to be that many in real life.

As for your 2D matrix, i understood it would be a practical, reasonably simple way to display the results for proof-checking, but we still need good old intact ballots to compute it, right ?

#399 ::: dh ::: (view all by) ::: April 10, 2015, 12:10 PM:

@386-

The problem with the old system that is being exploited is that everyone was given 5 votes each, a large majority of which were "wasted" each year. By forming a slate, the Puppies were able to use all 5 of their votes.

I believe that this is supposition, not fact. We presume that the SP/RP slates acted this way, but we don't know it. There is evidence to suggest it (certain categories), and there is also evidence to tend against it (other categories). There could be a much more sophisticated voting pattern in play, which would change the discussion of what the exploit and resolution actually is. Bruce initially suggested we did have data to know it, in a scientific sense. But he has now, apparently, back peddled that level of certainity. I am not sure if it's because he was originally mistaken, or if the data is not complete enough for him to make a conclusion on the basis of math.

@388--

1. Are you unclear on how we would address the voting problem scientifically? It would start by having a testable hypothesis. The hypothesis would be "SP/RP has exposed a weakness in the nomination process where slate voting amplifies the voice of a relative few". I think that's a fair restatement. Please feel free to offer corrections. Applying a science orientated process to addressing the problem would be to test each phrase of this hypothesis. That there is a weakness, the weakness is slate voting, that it amplifies the voice of a relative few. So far these things have not been tested.

2. There is a strong consensus I believe that slates are "bad" for achieving a stated outcome. The stated outcome is "nominees which the most people find acceptable". However, we are not even to the point of establishing that a slate vote happened. We think it happened. SP/RP say it happened. But how do you know it happened? By the results? We only have a portion of the results - the nominees and the total votes for that nominee. Think of what we don't know: we don't know which categories had works who declined nomination. We have an idea of who did. We don't know with what correlation to slate votes actually happened. We presume it was straight slate voting - namely, that SP adherents voted the slate, that RP adherents voted the slate. But we don't that it's true. These are facts that are important. I'll put it this way. If the average Puppy voter only used 4 out of 5 of their slots for block voting, and 1 out of their 5 slots for their own choice, the proposed changes could enhance

3. It is not credible to propose that the recent discussion on block voting happened for not particular reason at all. It is related to the SP/RP slate voting. What I am proposing is that it is foolhardy to rush, without all the data, into a change process that will take 2 years to effect, when the probability is that we cannot make a competent decision. When the change process starts, proposals and votes will happen in real time. If nothing passes, it will be another year before it can be revisited. If something that turns out to be sub-optimal passes, and is ratified next year, it will be another year hence, and another year after that, before a fix can be applied.

There is a large continuum of outcomes that stretch from the status quo, to radical change. The greater push towards radical change, the higher the risk.

#400 ::: P J Evans ::: (view all by) ::: April 10, 2015, 12:15 PM:

397
I think that I'd be getting the digital as well as the paper versions for 1984 (both exist, AFAIK). It's 1972 that may not have digital, at least as far as the nominations go.

#401 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 12:17 PM:

Vivien@398:

"PJEvans could prove me right": Touché.

"but we still need good old intact ballots to compute [the matrix]": Right. The idea is just that, once you have the matrix (in the future), you can keep that, and then throwing out the ballots wouldn't be much of a loss.

As a statistician, I think of that matrix as being akin to a variance/covariance matrix. This kind of matrix has all the information about first and second moments in this case. And this is very handwavy and non-rigorous, but the lesson of the Central Limit Theorem is that the more data you have, the less the third and higher moments matter (which in this case means, information about how many voters approved all the works in specific sets of 3 or more). Yes, throwing away the ballots throws away that information, but what you're losing is mostly either redundant or noise.

#402 ::: P J Evans ::: (view all by) ::: April 10, 2015, 12:19 PM:

398
Not just data geeks, but data geeks with pack-rat genes. (But really - a month later and that data would have been gone. Luck is also a factor.)

And I'm really interested in seeing what these proposed counting systems will do with 1984, where we know there was ballot-box-stuffing attempted.

#403 ::: abi ::: (view all by) ::: April 10, 2015, 12:24 PM:

dh @399:

1. Please refer to Bruce @396; this discussion, though prompted by SP/RB, is not about what happened this year. So the hypothesis is more like, "slate voting amplifies the voice of a relative few". That possibility is easily provable.
2. Not relevant given the hypothesis as stated in 1.
3. Not relevant to the conversation, which is about "in what ways could we" change the process. "Should we" is another conversation.

#404 ::: dh ::: (view all by) ::: April 10, 2015, 12:36 PM:

abi--

Got it.

#405 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 12:44 PM:

Sorry, that talk about moments may have been a bit baffle-gab-y. The relevance of the central limit theorem to this case is, in a more rigorous sense: insofar as there is some amount of randomness in any given voter's ballot (that is, insofar as there is some chance that they might have voted something different), then the information that is in the third and higher moments should tend to be either redundant or noise. This is just as true for small amounts of data as for large amounts of data, but with less data, the information we have about the "true" second moments is more fuzzy, so "redundant" views on that same info is more useful.

So that point is not entirely relevant here because you can imagine that, in the purest scenario, all of the rabid puppies were just voting as puppies and not letting their own judgment enter into it at all; and in that case, the 2D matrix might give you clues that that was what happened, but you would not be able to build a solid case from the 2D matrix alone.

Still, central limit theorem aside, I think that the clues you'd get from the 2d matrix would be enough, and so that the SRAV result would closely mirror the RAV result.

In practice, I still think RAV is the best proposal. It would be instructive to publish the SRAV matrix, but I think that should be optional / epiphenomenal.

#406 ::: Rene ::: (view all by) ::: April 10, 2015, 01:22 PM:

Lee @ 181:

You wrote: " It's worth noting here that Larry Correia, who has a background in statistics, undertook to analyze at least one year's worth of Worldcon ballot results and actually said that he found no evidence suggesting that sort of collusion.

This has not, of course, stopped the SPs from insisting that such collusion must exist; they just assume that it's far better hidden than even they thought it was."

Larry Correia never stated that his analysis suggested or confirmed that there was no collusion or block voting by nominating voters in the nomination process. He stated that his analysis revealed that there was no fraud by the committee in charge of tabulating nominating votes. Here is what he wrote:

"I had heard many allegations of fraud in the nomination process from other authors. Tossed votes, far lower than expected counts, that sort of thing. I am a full time author now, but I am a retired auditor. I love looking for fraud. I do spreadsheets and statistical analysis for fun. So I wanted to see if votes were being tossed. When Sad Puppies 1 launched I kept track of who said they were voting, kept a tally, and then kept their emails so if necessary I could ask for their registration receipts. My suggested slate in other categories would help provide check figures in the smaller categories. (But for the record, everything I suggested was something that I read, enjoyed, and thought was of superior quality and deserving of an award).
The final numbers for last year were within the expected deviation. No red flags. LonCon has struck me as perfectly honest in my dealings with them. So I’m happy to say that I see no evidence of dishonesty in the nominating process. That is excellent."
http://monsterhunternation.com/2014/04/24/an-explanation-about-the-hugo-awards-controversy/

#407 ::: J Thomas ::: (view all by) ::: April 10, 2015, 01:29 PM:

@383 Jameson Quinn

In general, what you're now suggesting is a continuous series of approval runoffs, with no elimination. Majority Judgment can be thought of as a virtual continuous series of approval runoffs with no elimination and monotonic ballots (that is, in which voters can only add approvals, not remove them). So there's not a huge gulf between the proposals.

OK, good.

I can imagine that it might be better to let people remove approvals, but it is likely not to be a big deal.

@395 JonW

My problem with the proposal #67 & #385 is that it seems to assume that ordinary voters have nothing better to do with their time than to keep a browser tab open to the nominations page and monitor it so as to be able to counter the machinations of other nominators.

It doesn't have to take much of their time.

Vote. Somewhere along the line see how your choices are doing. If you care about the current front runners, you can vote for some of them and not others. If you don't like any of them but you decide you do like a dark horse that is improving, vote for that. Or do nothing.

If you hear a rumor about somebody awful who's getting too far ahead, you can log in a third time and see whether there's something you want to do about it -- like vote for one or more current front-runners you didn't vote for before, that have a good chance to stop it.

You can check just before the polls close, and see whether you want to vote for any of the front runners that you haven't voted for before.

You'll only obsess over it if that's how you like to do things.

@385 Buddha Buck

To counter the "then the slators will just dump their nominations at the last minute", have a random close time of nominations.

If you use acceptance voting, it isn't unlikely that the top 5 nominations will all have majorities of the non-cabal nominators. When you can vote for as many as you like, what's the chance that a whole lot of people will like the most popular ones? Pretty good.

Let's be conservative and say that each of the top 5 has 30% of the nominators so far. A cabal that wants to overturn that at the last minute needs to bring in more than 30% of the vote at the last minute.

And what does it get them? If they vote early, their enemies might notice and more of them vote for the front-running non-cabal choices than would otherwise, just to stop them. If they carefully coordinate voting at the last minute, maybe their choices will dominate because there are more of them that want those choices than there are who like any of the mainstream choices.

I say, if they're that dedicated and that organized, let them win. They like their stuff more than any group of other nominators likes other stuff. That's a win.

If you don't know what anybody else is doing, you will waste most of your votes. The slate guys don't waste their votes. But if you know which ones have a chance to win, you can vote for the winners you like. Your votes are not wasted. The cabal wins if they are the majority.

#408 ::: Edmund Schweppe ::: (view all by) ::: April 10, 2015, 02:08 PM:

Bruce Schneier @233:

5. It should be difficult for a voting bloc to co-opt.
I'd change this to "difficult for a minority voting bloc to co-opt" (my emphasis). I'm guessing that there were somewheres between one and three hundred SP/RP nominators (out of twenty-two-hundred total nominating ballots), based mainly on how well the SP/RP slate performed in the shorter-fiction categories and how (relatively) poorly it performed in Best Novel. (If a hypothetical Steampunk Partisan-sponsored slate were to be nominated by sixteen hundred out of three thousand members, that'd pretty clearly be the preference of a majority and it would make sense for the best works to be picked from that pool.)

I don't have any brilliant ideas to add to the mix, alas. I have a sort-of-half-baked idea, however. Presuming there's some way of mechanically identifying slate nominations, would we achieve the goals Bruce listed in @233 (as modified above) by treating each identified slate as a distinct nominee?

#409 ::: Edmund Schweppe ::: (view all by) ::: April 10, 2015, 02:10 PM:

Bruce Schneier @0:

So when we choose to vote for the Republican party, we are choosing a set of policies that a Labour government can be reasonably expected to follow.

Clearly I have not been paying sufficient attention to the rightward drift of Labour. Or was it the leftward drift of the Republicans?

#410 ::: Martin (not a glyptodont) ::: (view all by) ::: April 10, 2015, 02:37 PM:

This is an edit of something I posted on GRRMs blog. It might be repeating an extant idea, in which case I apologize. However, it might be a solution, so I want to make sure it is considered.

The election process can't be changed next year, because it takes at least two years.

So the Non-Puppies (NPs) either have to organize their own slate(s) or accept the puppy one. I'm betting on one or more NP slates. So slates are inevitable for at least one year, probably several.

Here's my suggestion: Agree, well beforehand, on which electoral reform you want to use to exclude slate voting. *Run a very transparent public mock nomination election using this system to build the NP slate.*

If the electoral reform you want is good, it will produce a NP slate that the majority of the electorate can support. After a few years of doing it, you can probably get it made into the official rule pretty easily, and can skip the slate generation step.

If it is not good, well, at least you didn't try to get a bad system written into the bylaws. Keep trying.

I realize this will take several years of heightened organization and good-faith management, but I think it would actually lead to fix of the system, both immediately and in the long term.

#411 ::: C. Wingate ::: (view all by) ::: April 10, 2015, 02:42 PM:

A comment and a question on the current system:

First (and someone else may have have said this already, but the notion came up elsewhere re strategic voting) it seems to me that in the current or for that matter any ranked system of voting in which "no award" is an option, the tallying should be disregarding any votes ranked below NA. Preferring NA to some nominee implies preferring that the nominee not win. Therefore no such system should be able to count such a "vote" for a nominee towards it winning.

Second, WRT VD's threat to force NA in the future: how difficult is it for him to carry this out? What percentage of people voting NA as their only vote in a category, under the current system, is sufficient for it to "win"?

#412 ::: Lee ::: (view all by) ::: April 10, 2015, 02:52 PM:

Jameson Quinn:
Your first response to getting a mod warning made you look like a reasonable person who can take a hint.
#318 makes you look like an over-friendly puppy who keeps jumping up on people because they don't know any better.
#341 makes you look like either a troll or a 3-year-old. "Pushing limits" is a hallmark of both.

Teresa isn't the only person who's getting annoyed with you. Stop pushing your damned website, which is AT BEST off-topic for this discussion, and start focusing on contributing honestly, which you have also shown you can do when you're not thinking about grabbing eyeballs.

P J Evans, #390: I stopped trying to engage with dh 2 threads ago, when it became obvious that any rebuttal of their arguments was only met with a re-statement of the original argument, possibly with a slight tweak.

John A., #393: I thought the whole point of this thread was to start working out ideas which could then be refined between now and the Business Meeting. Am I wrong? Because if I'm right, your comment seems to be both unwontedly scoldy and counterproductive.

Rene, #406: Ah, thank you for the exact quote. I seem to have misremembered it as being somewhat wider in scope.


*looks back over comment in Preview, dials back a few things and deletes others*

I'm getting testy again. I think it's time for me to take a break.

#413 ::: Cheradenine ::: (view all by) ::: April 10, 2015, 02:54 PM:

C. Wingate @#411: The reason for being able to rank nominees below No Award is basically the same as the reason for ranking in general - if No Award is eliminated as a possibility, you retain the ability to express a preference for which non-worthy candidate wins the award. So, for example, I can rank

A, B, No Award, C

To communicate that I think A & B are worthy of the Hugo, but if enough people disagree with me, I would rather that it went to C than D or E, because C is "not good enough" while D/E are "actively terrible."

#414 ::: P J Evans ::: (view all by) ::: April 10, 2015, 02:55 PM:

411
As I understand it (possibly incorrectly) that's how it's supposed to work, and if you hand-count the ballots (not actually recommended with the number of ballots received these days) it could easily be done that way. However, it's hard to get the computer to do it that way [/wry].
I haven't seen the current ballot-counting program, and I don't know how it handles nominees ranked below Noa Waard.

#415 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 02:57 PM:

MartinNonGlyptodont@410: Good idea.

I am willing to help with the programming on this. No matter what system seems to be the consensus, although obviously my advice is that RAV is the best choice in this case.

How would we validate the voters? Ideally, the eligibility criteria should be the same or nearly the same as the Hugo electorate.

How would we publicize this?

It would in principle be possible to do this so that the SRAV matrix was publicly voter-verifiable, using homomorphic encryption, as with Helios voting; with the caveat that you'd have to make a way to canonicalize the name of each work into a hash table, and there would be a tradeoff between probability of collisions for a small hash table, and the storage and processing burden for calculating votes for a larger hash table. Blah blah birthday paradox blah blah. Um... this line of thinking may be overkill; if the code is open source, it may be OK if the chain of custody isn't 100% independently verifiable, people may just be able to trust the organizers as they do today...

#416 ::: Steve Downey ::: (view all by) ::: April 10, 2015, 03:01 PM:

C. Wingate @ 411

It would probably be sufficient to (under current nomination rules) to nominate a poison pill slate, such that everyone else would have to vote NA. I don't think the RB/SB voters could force NA by themselves.

#417 ::: Steven desJardins ::: (view all by) ::: April 10, 2015, 03:11 PM:

PJ, if you look at last year's voting results, you can see how the No Award system works.

http://www.thehugoawards.org/content/pdf/2014HugoStatistics.pdf

See, for example, "Race For Position One" in the novel rankings. No Award has fewer first-place rankings than any of the items on the ballot, so it's eliminated first, and Ancillary Justice goes on to win. But, before its victory is complete, there's a final runoff step with No Award: out of 2657 ballots that ranked at least one of them, 2327 ranked Ancillary Justice above No Award, and only 480 ranked No Award above Ancillary Justice, so an award is presented in the category.

But now look at, say, "Race for Position 4". At this point we're only considering Wheel of Time, Warbound, and No Award. This time around, No Award has more first-place votes than Warbound, so it's not eliminated first, and Wheel of Time goes on to win in the second round. (If Wheel of Time had two more votes, it would have an outright majority in the first round, and you wouldn't even need to go to a second round.)

#418 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 03:38 PM:

Lee@412: Understood. Further mea culpas and/or defensiveness from me would be off-topic. I've already said that it won't happen again, and also apologized to Teresa for being inappropriately personal. I meant both.

#419 ::: Joshua Kronengold ::: (view all by) ::: April 10, 2015, 03:40 PM:

#410: My thought here is that someone could put up a site that would allow aggregation of nominations according to multiple suggested nomination rules, with people being able to go between the various rules and see what the results would be (and then use them to inform their actual nominations if they so choose).

This would serve the purposes of:

1. Test proposed nomination rules before they go into practice, in a very voluntary fashion.

2. Allow nominations to converge in a way that helped counter the effect of the expected slate without truly becoming a slate (particularly if the goal was that any worldcon member could nominate, though that wouldn't be enforcable; it could also let people who didn't appear on the list of members of appropriate worldcons to add nominations but let people exclude them if they wanted to see just the worldcon picture).

#420 ::: Brad from Sunnyvale ::: (view all by) ::: April 10, 2015, 03:40 PM:

I am a big fan of Approval voting for its simplicity, but it's not the right choice for a literary award at all. (It was the right choice for the nominating ballot, but that's over.)

In the voting ballot, approval voting simply gives the award to the most widely read work of sufficient quality. Unfortunately, Hugo voters traditionally make the default assumption that all or most works on the final ballot are of sufficient quality, and No Award is almost always a tiny minority choice.

I believe it is essential to ask people to compare the works in question. Even with that, there is a strong bias towards the works of greatest popularity (before the nomination) because while many try to read all the works, especially with the packet, many do not.

As such, oddly, Approval would become strategic. If you were evaluating a brand new writer with an amazing first novel, competing with but clearly superior in your mind to the 3rd novel in a popular series by a very popular writer, approval is very likely to give the award to the established writer if people like them both but the latter had more readers. As such, you might decide to leave off the series book, even though you judge it worthy (the Approval test) to help your first choice.

Bruce, I will defer on Jury discussions at your request, but I will say I don't think at all this makes the award not a Hugo! I think an award given by WSFS, voted on fairly by fans and called a Hugo is a Hugo both by definition and spirit, even if the nomination process is different, as long as its fair.

The purpose of the nomination process is not, I feel, to be a contest. It is there to help find really good works that some of us may not have read yet, so that we can feel we compared the best in picking the winner. I read the nominees I haven't yet read not just to be fair in forming my final ballot, but also because I feel, "Hey, a lot of fans felt this had merit, I should try it."

#421 ::: P J Evans ::: (view all by) ::: April 10, 2015, 04:10 PM:

417
Steve, I don't know all the details of the current counting program, but I've seen the full results from at least two past years (many pages of printouts). That's normal.

#422 ::: Joshua Kronengold ::: (view all by) ::: April 10, 2015, 04:23 PM:

JQ, I know you came in late, but what do you think of my "single divisible vote" idea compared to RAV?

I actually started with RAV on that idea, but thought it was too hard to explan. However, I believe (with some cursory examination) that SDV is mathematically equivalent to RAV, except that it's much easier to explain how it works, particularly since, like STV, it uses elimination.

#423 ::: Fade Manley ::: (view all by) ::: April 10, 2015, 04:31 PM:

I'm not able to follow this discussion closely--I mean, I'm reading all the posts, but I don't have the background to follow a lot of the technical details--but I'd like to point out that counter-slates are a terrible solution fundamentally in that many of the "right" people have already said they will reject slates on principle.

In particular, assembling a slate by having a sort of mock nominations period and working it out from there looks an awful lot like "We want to run the Hugos without letting the wrong sort of people in." (Further complicated by the number of authors who are against slates, and who would ask to be taken off that assembled list, making it increasingly less representative of even of that subset of voters.) At that point you might as well call it a new award going by some other name.

#424 ::: Douglas Henke ::: (view all by) ::: April 10, 2015, 04:33 PM:

There may be a source of information useful to vote-tallying algorithms we're not considering (or consideration of which I've missed in my initial reading of the thread):

The nomination slate has multiple categories.

I've seen some really good ideas about how to detect and mitigate block voting within a single category, as well as problems with same (mostly along the lines of "people can have similar tastes without being a bloc" and "works can get nominated a lot because they're good, and not just because a small coordinated group is pulling for them").

I submit for your consideration that while nomination ballots may correlate closely within one category for innocent reasons, ballots will not correlate strongly across multiple categories unless there is collusion amongst the voters casting those ballots.

You and I may have very similar tastes in novels. But it is far less likely that we will have both that and also very similar tastes in best New Doctor Who episode.

There may be five novellas out in a given year that are staggering works of genius towering over all others. But will the same thing happen, in the same year, for fanzines?

Whether one can design around this observation a voting system that can be easily explained to a drunk child or a sober goldfish is another matter. (Also: Such a system might prevent puppying of the entire slate yet leave the process vulnerable to future single-category shenanigans.)

#425 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 04:46 PM:

JoshuaKronengold@422: Your "single divisible vote" idea is a good one. It's a known thing in voting theory; in Kilgour and Marshall, the ref I gave and Bruce linked above, a very similar (but sadly, not identical) system is called "sequential satisfaction approval voting". That (K&M's SSAV) is not equivalent to RAV, but in practice it would usually be pretty close. I'd have to look at it further to be sure whether your SDV is actually equivalent to RAV; my intuition says it's closer to RAV than SSAV is, but still not equivalent. I'm not entirely sure. In any case, your system is definitely proportional (by the Droop definition thereof, which is the one I've been using throughout this thread).

So yes, it would solve the basic problem, and would not require a major change to ballot format. So I'd strongly support it; whether or not it's fully equivalent to RAV, it's just about equally good, and it's basically just a matter of which one people find easier to understand.

#426 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 04:49 PM:

@424: You're right, of course. But trying to design a voting system around that would be building a fence against the tide; there would always be a strategic way around such measures.

#427 ::: Cheradenine ::: (view all by) ::: April 10, 2015, 04:56 PM:

Joshua @422: They're not equivalent. RAV will always nominate the candidate appearing on the most ballots. SDV can eliminate the candidate appearing on the most ballots.

Simplified and thus unrealistic example: 12 voters fill out their ballots:
1 AB, 1 AC, 1 AD, 1 AE, 2 B, 2 C, 2 D, 2 E

Under RAV, the first thing that happens is that A gets nominated (with 4 votes to everyone else's 3); under SDV, the first thing that happens is that A gets eliminated (since they have 2 weighted votes to everyone else's 2.5).

As far as I can tell, there's no existing analysis of SDV, so I'd have to work with it a lot more to get a sense of its overall behavior and how it compares to RAV. I suspect they would often produce the same results, but can't quantify often.

#428 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 04:58 PM:

FadeManly@423 argues against the idea of using voting to assemble a "consensus slate", as a temporary stopgap until the voting system, saying that this would look like exactly what the Puppies complain of, especially if some authors asked to be taken off the slate.

If we used a good voting system, everybody participated in the voting for the consensus slate, then decided honestly whether the result seemed fair, I think that people would agree that the results were good. Unfortunately, while good-faith actors can control the voting system, there's nothing stopping bad-faith actors from boycotting it and then complaining that it excludes them. Insofar as any worthy authors sympathized with that position, that could doom the undertaking. I'd like to be an optimist here but I have to admit that success is not guaranteed. Still, I'd be willing to help give it a try.

#429 ::: Siobhan M. Murphy ::: (view all by) ::: April 10, 2015, 05:15 PM:

If the goal of a voting system is to create an effective parliamentary or representative government, slate voting seems to me to have a legitimate purpose - i.e., I would like my legislators to have enough of a common goal that they get something done.

IMO, the goal of the Hugo process is very different - it is to recognize excellence (or as close as we get to it) from the F/SF creative community.

A slate / counter slate system, particularly one based on categories of F/SF (or worse yet perceived politics of authors) undermines the rationale for the voting system. For this reason I think leaving things alone to see how things play out is not the most desirable response. I also dislike any rule that would change the openness to participation.

Anyone who is interested enough in these genres (this genre) to support and/or attend a World Con deserves an equal vote.

In my experience, simpler usually seems fairer to people.

Stated differently, the more complex the voting system is made to prevent block voting, the more the block voters will feel disenfranchised when compared to others.

For this reason, my inclination would be to reduce the total number of nominees per voter (2 or 3 of 6 or 7).

Yes, a "political party" may be rabid enough to coordinate each voter's 2 or 3 nominating votes, but any system can be gamed. Also slate voting can be fought by discussion of the quality of a potential nominee or work in social media or by 'political ads.' There are more ways to address the problem than changing only the voting system.

#430 ::: abi ::: (view all by) ::: April 10, 2015, 05:28 PM:

Jameson Quinn @428:

If we used a good voting system, everybody participated in the voting for the consensus slate, then decided honestly whether the result seemed fair...

I've bolded the biggest potential problem there. Who is "everybody" in this context? How do we reach them? How do we determine whether we've got them all?

#431 ::: Brad from Sunnyvale ::: (view all by) ::: April 10, 2015, 06:01 PM:

Small nit, when it comes to voting, it's more commonly spelled as "bloc" rather than "block" though it's so easy to confuse the two that the latter form has been showing up, and even makes more sense.

But either way, there are so many proposals here and other places, all with some validity, it seems like it will be a huge and complex business meeting this year. I think it might make sense to actually have Sasquan form a "subcommittee on redesign of the Hugo nomination process to make it more robust against collusion by nominators" and have it take input from many sources and proposals, try to narrow down to a few good ones, and then bring them up. Does not stop anybody with their own system from also doing at the meeting, but might consolidate some of the proposals. (Yes, I see the irony :-)

#432 ::: J Thomas ::: (view all by) ::: April 10, 2015, 06:03 PM:

@420 Brad from Sunnyvale

In the voting ballot, approval voting simply gives the award to the most widely read work of sufficient quality.

It depends on how the voters vote.

I believe it is essential to ask people to compare the works in question.

I think people generally expect that they should choose the best.

But if they only choose one, then if their choice loses they have no further say. With 5 choices, the winner could have as little as 21% of the vote.

So we try various methods to let voters choose second choices, and that allows weird kinds of strategic voting.

Like, if you have a precise understanding of how the others will vote, you might create a conspiracy to vote for one of the least popular choices first, one you don't actually want. If you can get just enough people to vote that way to knock out the popular choices you don't like, then when your first choice is knocked out your real second choice gets lots of votes.

This sort of thing has to be done precisely -- if you're a little bit wrong predicting the votes, you could turn it into a win for the unpopular choice and your second vote never comes into play.

I think usually the weird strategic voting will not really work right because voters don't know enough. But they are still likely to try it. Very difficult to create a voting system that does not give people the idea they can do effective strategic voting.

As such, oddly, Approval would become strategic. If you were evaluating a brand new writer with an amazing first novel, competing with but clearly superior in your mind to the 3rd novel in a popular series by a very popular writer, approval is very likely to give the award to the established writer if people like them both but the latter had more readers. As such, you might decide to leave off the series book, even though you judge it worthy (the Approval test) to help your first choice.

This is appropriate. If you don't want your vote to count for the popular novel, don't vote for the popular novel. The less popular novel probably will not win, regardless, but if you want it to have its best chance, don't also vote for the opposition. With a ranking system you can vote first for the unpopular novel and then for the popular one, and the popular vote won't count until the unpopular one has failed. But now you have complicated strategic voting and complicated plans to reduce it. And every strategy can backfire.

#433 ::: Martin (not a glyptodont) ::: (view all by) ::: April 10, 2015, 06:25 PM:

Jameson @ 428

I'm not sure that "everybody" needs to participate - you just need enough diverse folks to assemble a good "Anti-Slate Slate" via a good anti-slate voting mechanism.

If the ASS (unfortunate acronym there) election is transparent enough to show anyone who cares what happened and how it works to assemble a good list of nominees, those looking for a way to fight both slates and puppies will have a clear option. You only need about 20% of the electorate to cancel the * Puppies.

Another thought - the ASS might only list 3 results in each category. It allows folks to add their own input while still fighting the *P slate. You could call it "covering your ASS."

#434 ::: felice ::: (view all by) ::: April 10, 2015, 06:51 PM:

JonW @ 395: "My problem with the proposal #67 & #385 is that it seems to assume that ordinary voters have nothing better to do with their time than to keep a browser tab open to the nominations page and monitor it so as to be able to counter the machinations of other nominators."

A one-off advance publication of nominations would serve much the same function without requiring ongoing monitoring, or even caring about whether there's a slate or not. Early nominators would select a pool of works which are seriously in the running, and later nominators could choose the ones they like most from that "long list". That reduces the number of nominations wasted on works that have no chance, and increases the number of nominations required to get on the ballot, making it harder for a slate, irrespective of whether the slate nominates before or after the publication.

#435 ::: Parmenio ::: (view all by) ::: April 10, 2015, 09:45 PM:

If the numerology of
https://storify.com/SecretGamerGrrl/that-s-not-a-haystack-it-s-tiny-a-pile-of-needles
https://medium.com/@MikeRTrice/gamergate-in-data-perspective-part-1-d80068cbc85a?source=latest
https://medium.com/@MikeRTrice/gamergate-in-data-perspective-part-ii-gamergate-data-versus-gaming-data-9323ed68535f?source=latest

is trustworthy, there are only about 500 activists in Puppy/Gamergate, and probably a majority of them cast nominating ballots this year. It is possible that more 'non-usual-Worldcon-supporters' may be recruited - the Conservative Political Action Conference may add a free Worldcon supporting membership to their attendees swagpacket, but certainly he or she who payeth their membership hath a right to cast a nominating vote.

It is highly likely, tho, that many Supporters who usually did not cast nominating ballots will be motivated to cast one in 2016. I expect that various websites and dead-tree publications will have reminders of deadlines, and that next January will see people professional and amateur (amateur people = dogs pretending to be people on weekends ;-) ) putting up lists of 'stories I'm considering' or 'novellettes that impressed me' or 'All The Short Form Fiction Eligible For This Year's Yugos'. Or maybe even the Rainbow Kitten, Velvet Unicorn, and Purple Squirrels slates.

I predict there will be a significant jump in the number of nominating ballots cast.

The one downside I can see is that works not available through booksellers, or not online, in early 2016, will be disadvantaged. Publication in an Australian chapbook with a group of fanatic followers may not be sufficient to get a novella into the finals henceforward.

tl:dr 2015 will be remembered as an aberration. Its story will be retold. Future fen will learn that WE must make the system work.

#436 ::: Jameson Quinn ::: (view all by) ::: April 10, 2015, 10:25 PM:

Martin@433: In order to counteract the puppies (at least, at this years' size), a slate only needs a few hundred people to vote it. However, in order to "be a consensus slate", that is, not be vulnerable to claims of being exclusionary, so that worthy authors don't ask to be left off, an consensus-anti-slate-slate (CASS) voting process needs to be seen to welcome everybody, including puppies. Really, the point can not be to shut them out; an eye for an eye leaves everyone blind.

There may be some people who want to sow chaos, who have enough chutzpah to find some excuse to boycott such a "lets form a consensus slate" voting process and then complain that they were excluded. If their story is plausible enough to get anyone to turn down a nomination from the CASS, then the CASS loses. Which, in my view, would be bad for democracy.

#437 ::: J Thomas ::: (view all by) ::: April 11, 2015, 06:05 AM:

@436 Jameson Quinn

In order to counteract the puppies (at least, at this years' size), a slate only needs a few hundred people to vote it.

You brought up an important point. Until now, the sick puppies' leaders have claimed there is an invisible conspiracy against them. But now the conspiracy is completely open and aboveboard. There's a lot of overt discussion about how to stop the SPs. Discussion about how to keep them from voting, how to make sure they have less influence, etc.

They have about a year to exploit that. They can say there is a fanatical evil faction in fandom that has declared war on them, and ask who's willing to fight back.

Wars start this way. It takes a bunch of people who get interested in fighting, and if the early adopters on one side can commit enough atrocities to get the other side motivated, then they can use that motivation to motivate their own side.

So we should consider the possibility that they may have 1,000 SP fanatics voting next year, or possibly 5,000.

If it's 5000 at $40 each, that would be $200,000 the Worldcon organizers didn't expect to see. Maybe they should have a contingency plan for what to do with the money, if it does show up.

#438 ::: rcade ::: (view all by) ::: April 11, 2015, 08:56 AM:

Perhaps we should tackle this problem first by taking away the argument that slate voting is within the rules.

For instance: "3.7.4: Members should not vote for nominations by copying any slate of nominees suggested by others, but instead should make their own individual choices for what they believe are the best works."

This would make it official that the membership opposes bloc voting. Anyone acting in good faith would honor this rule, and if a slate was proposed people could point to the WSFS Constitution to show that it is not allowed. There would be some social pressure on anyone who tried to encourage people to break this rule by promoting a slate.

At that point the problem becomes how to stop people acting in bad faith, but maybe that's not as big a problem as it appears. The Hugos operated for decades without attempts to game the entire ballot, even though the strategic value of doing so was obvious.

#439 ::: Tim Bartik ::: (view all by) ::: April 11, 2015, 09:36 AM:

For next year, it is impossible to change the voting rules.

Therefore, next year, any voter who does not back SOME slate in the nominating process is likely throwing away their vote. It is implausible that the total number of nominating voters will increase to the extent needed to prevent this outcome.

Given that the SP groups are constructing slates for next year, they will dominate the nominations again unless someone constructs a counter-slate.

This is not the optimal outcome if one thinks slates are a bad idea for nominations for literary awards.

What could make counter-slates a less bad option than they otherwise would be?

First, some groups could agree to come up with a slate using some voting process similar to what has been proposed in this thread. This means that the slating process would features to encourage diversity of nominations that reflect a wide variety of views of voters.

Second, it could be agreed that this counter-slate would be suggested to potential nominators as works to consider in nominating if you have read them.

Third, the works in this counter-slate could be available for some agreed upon package price to nominators participating in this slating process, to allow folks to read them before making their Hugo nominations.

In essence, this would be running a pre-Hugo primary to identify works to be nominated.

While I think it would be preferable to modify the ACTUAL Hugo nominating rules to discourage slate voting, this cannot be done for next year. And given the likely resistance to any attempt to modify Hugo rules, it may never happen.

This is in my view a second best or third best solution, but it seems better than the alternative next year of having only SP slates.

I believe it is ethically better than the SP slates because the process of choosing the counter-slate is fairer, and because the process would urge the final Hugo nominations to be only made for works you have read. And there would be some process to try to make the works accessible so that this reading could become a reality.

#440 ::: Simon Bisson ::: (view all by) ::: April 11, 2015, 09:53 AM:

There's an interesting alternative approach, taking a leaf from Ross Anderson's book: changing the economics of the voting process.

In that vein here's a thought. What if Sasquan didn't organise a voter's packet? The SP/RP folk have been going on about it as if it was a de facto part of the deal - whereas it's actually something very new, it's not actually mandated by the rules, and as we saw last year still a matter of debate at publishers.

Reducing the perceived value of a supporting membership might have a negative effect on con funds, but it would reduce the likelihood of SP/RPs stacking the ballot in future. You get to vote, but you don't get free books for doing it.

#441 ::: Bruce Schneier ::: (view all by) ::: April 11, 2015, 10:06 AM:

Parmenio @435:

"It is highly likely, tho, that many Supporters who usually did not cast nominating ballots will be motivated to cast one in 2016. I expect that various websites and dead-tree publications will have reminders of deadlines, and that next January will see people professional and amateur (amateur people = dogs pretending to be people on weekends ;-) ) putting up lists of 'stories I'm considering' or 'novellettes that impressed me' or 'All The Short Form Fiction Eligible For This Year's Yugos'. Or maybe even the Rainbow Kitten, Velvet Unicorn, and Purple Squirrels slates. I predict there will be a significant jump in the number of nominating ballots cast."

I agree. It used to not matter if someone nominated. (I, for example, never did.) Now it does. And lists of "this is what I think is worth nominating" will be valuable for all the marginal voters out there.

J Thomas @437:

"So we should consider the possibility that they may have 1,000 SP fanatics voting next year, or possibly 5,000."

There is going to be a voting-bloc percentage at which no voting system can protect against.

rcade @438:

"Perhaps we should tackle this problem first by taking away the argument that slate voting is within the rules. For instance: '3.7.4: Members should not vote for nominations by copying any slate of nominees suggested by others, but instead should make their own individual choices for what they believe are the best works.'"

This is a perfectly reasonable solution. The devil is in the enforcement, but I think it will be easy to detect any slate with a significant following in the ballots.

Simon Bisson #440:

"There's an interesting alternative approach, taking a leaf from Ross Anderson's book: changing the economics of the voting process. In that vein here's a thought. What if Sasquan didn't organise a voter's packet? The SP/RP folk have been going on about it as if it was a de facto part of the deal -- whereas it's actually something very new, it's not actually mandated by the rules, and as we saw last year still a matter of debate at publishers."

It shouldn't surprise you, but I have been thinking about how to modify incentives. The problem is that for the SP/RP, the value of voting was disrupting the process. Getting the voter's packet was incidental. You can raise the cost of voting to people who are not members of the community, but I can't figure out a way to do that that doesn't also penalize members of the community. You can make it easier for members of the community to vote, but that also makes it easier for activists to vote according to their bloc. I don't see a lot of economic levers here.

#442 ::: Tim Bartik ::: (view all by) ::: April 11, 2015, 10:07 AM:

My previous post at 439 was drafted yesterday. I apologize for posting it without reading the latest posts. I should have read the latest posts before posting it, as it seems others are also thinking along the same lines as me, specifically 410, 415, 419, 436.

The real issue is the ethical issue. If slate voting is always unethical, then counter-slates would seem equally unethical, as argued at 423.

As was argued at 436, I think one feature that would make a counter-slate somewhat better ethically if it was constructed via a "fair process".

I should note that I think this requires not only that the voting rules be fair and award diversity in the primary process used to construct the counter-slate, but also that the counter-slate have sufficient number of votes in the primary to be considered legitimate.

An additional problem is that the counter-slate also in practice would have to be as diverse as possible in what works actually are there. This would only occur if participants in the voting process in practice end up representing fans of a wide variety of SFF styles. This may be hard to accomplish in an unofficial "Hugo primary" election.

In addition, I think any counter-slate should ask people to list the slate nominees only if they have read them and think they deserve a Hugo. If a counter-slate is asking people to vote for works they have not read, this is ethically highly dubious. For works in all categories to be read by Hugo nominating voters, the slate nominees need to be available to be read somehow for participants in the counter-slate. This requires some equivalent to the Hugo packet, at a reasonable price, to be constructed for the finalists from this Hugo primary process. Whether publishers would participate in such a primary voting packet would be questionable.

So, while I think that this might in principle be doable, the barriers to a pre-Hugo primary to determine a counter-slate are:

(1) Getting sufficient participation from primary voters so that there is some legitimacy.

(2) Getting sufficient diversity of participation that the resulting finalists are diverse and are perceived as such.

(3) Getting participation from publishers and authors to make the primary finalists available to be read.

I think those are some big barriers to deal with. I have no idea if it is feasible to overcome those barriers. Perhaps where there's a will, there's a way.

#443 ::: rcade ::: (view all by) ::: April 11, 2015, 10:19 AM:

Bruce Schneier @ 441: The devil is in the enforcement, but I think it will be easy to detect any slate with a significant following in the ballots.

I see my proposal in #438 as something that might help even in the absence of enforcement. If you take Larry Correia and Brad Torgersen at their word, they value the Hugos. Anyone who does would be reluctant to break an explicit rule by promoting a slate.

#444 ::: P J Evans ::: (view all by) ::: April 11, 2015, 11:07 AM:

443
Why should we take them at their word?
They've already gamed the rules in a way that offends much of the community responsible for the awards they say they value.
That's why we're here.

#445 ::: rcade ::: (view all by) ::: April 11, 2015, 11:18 AM:

P J Evans @ 444: Why should we take them at their word?

There are two groups of slate voters: People acting in good faith and bad faith. The passage of constitutional language that makes explicit the membership's disapproval of bloc voting will peel off the ones acting in good faith. They will join the rest of us in nominating individually.

I'm not suggesting this be the only solution considered -- just the first step and likely the one that's easiest to pass.

#446 ::: CHip ::: (view all by) ::: April 11, 2015, 12:08 PM:

J Thomas @ 437: If it's 5000 at $40 each, that would be $200,000 the Worldcon organizers didn't expect to see. Maybe they should have a contingency plan for what to do with the money, if it does show up. That isn't exactly noise level, but it's well within the swing in income that a Worldcon has to be able to cope with. And there's always Pass-Along Funds if the money comes in too late to be spent usefully that year -- for sufficient values of "usefully"; you wouldn't believe how quickly that money could be bled away if a Worldcon did serious catering for pre-Hugo reception, Hugo losers party, feeding Masquerade contestants, etc. -- catering is expensive. They could even solve the perennial problem of providing enough room for bidding parties by giving them catering allocations that would let them use function space. I'd lean against all of these, but they exist and could be arranged in the month-plus between the close of voting and the start of the convention.

#447 ::: P J Evans ::: (view all by) ::: April 11, 2015, 12:46 PM:

445
Optimistic, aren't you?
People acting in good faith wouldn't be likely to be voting for a slate in the first place.

#448 ::: Andrew M ::: (view all by) ::: April 11, 2015, 01:39 PM:

Simon Bisson@440: We discussed the possibility of dropping the packet in another thread; I feel that doing so (if it remains in in our power to have it) would be disastrous. Besides what I said elsewhere, I think bloc voters would actually be the people least deterred by the loss of it; the people who need it are those who want to consider all the stuff and make their minds up, not those who already know what to vote for.

rcade@438: I'm sure all the puppies will insist that they are already following this rule. It is clear that they are not all voting the slate without variation. How much variation there is we cannot determine from available data (and even when the fuller data is released we won't know enough), but the best evidence is the figures for Best Related Work, the only category where every nominee is a puppy and there was no difference between the two slates; there the difference between highest and lowest nominees was 67 votes.

I think 'make your own individual choices' is either too easy to satisfy, or rules out what everyone does all the time. People don't make choices in complete isolation; they take account of what's being recommended, what has a groundswell of opinion in support of it. Whole slates are different from just recommendations of individual works, but it may still be hard to say what makes something a slate; Torgersen did not fill every slot on the ballot. Suppose someone makes two recommendations for each category and a lot of people follow them; is that a slate? And so on.

#449 ::: Fade Manley ::: (view all by) ::: April 11, 2015, 01:57 PM:

Tim Bartik @442:

So, while I think that this might in principle be doable, the barriers to a pre-Hugo primary to determine a counter-slate are:

(1) Getting sufficient participation from primary voters so that there is some legitimacy.

(2) Getting sufficient diversity of participation that the resulting finalists are diverse and are perceived as such.

(3) Getting participation from publishers and authors to make the primary finalists available to be read.

Thank you for laying this out! I was struggling to articulate #s 1 and 2 in particular as the reasons why I think a counter-slate might as well be a different award. I mean, look at it this way. Imagine someone does say, "Let's make a counter-slate, by having people nominate things just the way they do for the Hugos!" At that point...

...who gets to vote on the counter-slate?

I mean this quite seriously. Are we going to ask a third party, not affiliated with WorldCon, to start checking that people have registered with supporting or attending memberships? Because that's a lot of work, and I'm not sure how it works with confidentiality, if any, much less Worldcon's willingness to vet people online saying "Oh, yes, I am totally X who is registered." And whether or not that's checked, what if the Sad Puppies just shrug, and go assemble a slate to nominate in a voting bloc in that set of nominations?

Because if that slate is available to the general public, it's no longer representative of the people who would vote in the Hugos ordinarily. And unless the organizer of this special counter-slate expects to get more people nominating for it than would nominate things in the Hugos, it's just as gameable by the Sad Puppies. Unless we're somehow carefully excluding the people who would vote for the SP block, which is rapidly developing into really icky political party/loyalist faction issues.

Basically, I don't see any way to do a counter-slate that's both usefully representative and not as easily gamed as the nominations themselves. The problem with the SPs (in the technical sense, rather than the ethical sense) is that they vote in a bloc, not that they vote. Removing their votes from the equation entirely means it's a non-representative slate, even if it's one I would personally enjoy more. Not removing their votes means their bloc tactics work just as well there; it's just added one more layer of hassle between nominations and final awards.

#450 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 02:18 PM:

There is much merit in a simple rule forbidding collusion between nominators, which would be setting your ballot in agreement with other nominators to bias the results. In this case, the use of a subjective rule, judged by the Hugo Committee, may be superior to a mathematical fix, because with the hard and fast rule, one can find the algorithm that games it, but it is much harder to game humans.

In other words, there are times when the rule of men can be superior to the rule of law, if you can trust the rulers and have ways to make the accountable. The real law actually often works that way.

A ruling of collusion would of course have to be made public, with the reasoning. The data would also have to be released, but that can only happen after the awards (probably,) and one would have the right to appeal the ruling, perhaps to the full concom (if delegated) or perhaps to the prior year's concom, or eventually the business meeting. Which they would do, but that's part of the price of a just system.

While it has been noted that WSFS constitutional changes take two years, there are things which can happen in less, including this year:

a) A convention could downplay the Hugos (award them in May by mail with no ceremony) and conduct an alternate, non-Hugo award as it sees fit.

b) The business meeting could even retroactively declare the alternate award a Hugo if it wishes.

c) If a proposed change only affects the tallying of the final ballot, it is legal to tally the ballots of 2016 after a business meeting in 2016 ratifies a 2015 change to the rules. Irregular, but within the rules. The nomination and actual voting must take place before the worldcon. So you could make a rule prohibiting victory for a slate candidate in 2016, but not stop the final ballot from being packed by a slate. ie. No Award if the ballot is all-slate. And the Hugo awards must be tabulated and awarded after the convention, which is not very satisfying, but it's better than doing corrupted awards at the convention.


Of course, such drastic steps only make sense if there is a common judgement that there is an existential threat to the award. Which I think there may well be.

#451 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 02:18 PM:

GRRM has said on his blog that he thinks weighted votes are a bad idea ("spinach"). Here's a metaphor I used that I think helps explain why RAV is actually fairer than the current system.

The current system is like a game of "grab the cookie" played one cookie at a time. Whoever's fastest will grab all the cookies. Ideally, we'd prefer to put out all the cookies at once, so that someone would have to be 5 times as fast as anyone else to get all of them. That's basically what RAV does. The "reweighting" is a way to account for "how much time you've already spent grabbing cookies". To get the first cookie, you have to be faster than the other people grabbing it; to also get the second, you have to be over twice as fast; etc. The process of choosing one nominee at a time is kinda like in matrix-style movies, where time slows down to show the especially skilled moves.

Sure, there will be a few people who would be better off under the current rules, and they can complain about how the change is unfair because fast cookie grabbing is what it's all about. But there are a lot more people (3-4 times as many...) who would be better off under the new rules.

(If you notice, the simplest version of this metaphor leads to the D'Hondt weightings. If you want the Saint-Lagüe weightings, just assume that it takes the same amount of time to stash a cookie as it does to snag it. If you want the exponential weightings, I guess you grab one cookie with each hand first and after that you have to stash the cookies in progressively harder places.)

Actually, this metaphor is too kind to the current rules. If we're talking about rockets and not cookies, the way to win under current rules is to strategically collaborate. This may not be against the rules but is certainly against the spirit of independent judgement that should animate an awards process. Whereas being a fast grabber really is the point of the "grab the cookie" game.

#452 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 02:24 PM:

FadeMenley@449 says: "Basically, I don't see any way to do a counter-slate that's both usefully representative and not as easily gamed as the nominations themselves."

But the whole point of this thread is that voting theory gives us ways to do that. That's exactly what RAV and the other proportional-representation proposals all guarantee. I see that there are still the three hurdles TimBartik@442 mentioned; but if we can overcome those hurdles it would be perfectly OK and in fact entirely desirable for the puppies to vote in the primary (with whatever degree of coordination they wished).

#453 ::: P J Evans ::: (view all by) ::: April 11, 2015, 02:37 PM:

452
You seem to keep missing the point that slates are toxic. The point is to not have them, or to find a way to minimize them, and still have a nomination process that's clear, fair, and open to all members.

This is not an election where we're chosing who we want to run things. It's a vote on who we want to be on the final ballot.

#454 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 02:42 PM:

@453: Yes, you're right. They are. But any rule you pass to explicitly outlaw them will just push them underground. The way to get rid of them is to give them exactly their fair share. If they don't get any extra power from coordinating, and they don't get to play the victim, they'll have no reason to exist.

(I understand your argument here to be: "Yes, an unofficial primary is a good temporary workaround, but we should run it under rules that actively discourage slate-based voting, not ones that merely passively discourage it." If I've misunderstood you, I apologize.)

#455 ::: rcade ::: (view all by) ::: April 11, 2015, 02:56 PM:

P J Evans @ 447: Optimistic, aren't you?

I have enough optimism to think that because the Hugos weren't gamed like this for six decades, despite it being an obvious winning strategy, a rule that means "don't do slates" would help us regress to the mean.

If we come up with a voting system that weakens bloc voting, great. But it will be hard to pass that in a highly politicized climate without also appearing political.

Right now, the people running the Hugos can't even tell the public that slates are against the rules to discourage their use. The most they can say is that they violate the "spirit," which is a nebulous concept. (Not to be confused with the Nebulas.)

#456 ::: Keith "Kilo" Watt ::: (view all by) ::: April 11, 2015, 03:05 PM:

I have to say, it seems to me that Django Wexler's proposal is the only simple, workable proposal I've seen so far. I don't see it as an "anti-vote", I see it as a targeted "no award" (even though, to be clear, we are talking about nominations, not voting -- the voting process remains unchanged). Let's face it, there are great works of SF out there, some respectable works, some mediocre works, and some truly awful ones. This is not at all unprecedented in the internet world today: Those that agree with something give a post a "thumbs' up", those who disagree give a post a "thumbs' down" and those who are indifferent don't rate the post at all. Currently, the Hugos have a "thumbs' up", an "indifferent", and an "everything else in SF is thumbs' down". It's this last part that is the problem.

Django's proposal has a number of advantages (and a few disadvantages, of course, which I'll get to).

First and foremost, it is an effective counter to slates, if the Hugo voters are inclined to counter them. It requires no intervention by the Hugo organizers (which would give credence to the Puppies claim that they are being excluded by a "shadowy cabal"). Furthermore -- and I see this as a major advantage -- if a slate lists a work that you -do- think is Hugo worthy, you can nominate it in good conscience. In this way, a slate becomes nothing more than a suggested reading list, which we all agree is a perfectly valid, acceptable way to conduct the Hugo process. If, by some extraordinarily unlikely miracle, you find that all the works on a slate are to your liking, and you like them more than any other works you've seen, then nominate them all. If there are few that you think are worthy, but others that are terrible, nominate the good ones and downvote ("under no circumstances should this particular work be considered) the bad ones. Neither the Puppies nor the non-Puppies can really complain about that.

There are some obvious disadvantages. First, of course, if non-Puppies don't make the effort to look over the Puppy slate (or any other slate), and use their downvotes as required, then the slate voters can still run the table. But that's true in any case.

A more serious disadvantage is slate voters dog-piling (sorry, could resist) on an author they hate, for example, VD and Scalzi. Keep in mind, however, that it works both ways -- there are any people here who have said they will never read VD's stuff under circumstances (which, by the way, is a perfectly valid point of view for someone to take -- about any author). What it means is that a controversial author will end up having to work harder to create a work that truly impresses those who haven't prejudged him or her in order to counter the inevitable downvotes. Is it fair? Perhaps not. But keep in mind that only someone whose works are deeply held in the public eye will ever be affected by this -- which means just maybe you're a pretty damn good writer in any event. As Nirvana said, "You know you've made it [in popular culture] when Weird Al parodies you."

In the end, I don't think the proposal is at all radical when it comes to nominations. It is certainly far less complicated than essentially all of the other proposals I've seen so far. It recognizes the state of the world that has been forced upon us and presents a system that fairly operates within that state. It removes the threat of slates without devolving into competing political parties and without giving anyone -- Puppy or otherwise -- justification for saying a cabal is controlling the Hugos. Simply put, some people will think some specific works are Hugo-worthy, just as we do today. The only difference is that now we will also be able to specify which specific works are Hugo-unworthy as well. And really, that's not a bad way to approach recognizing the best of the best.

Just my thoughts,
Kilo

#457 ::: Cheradenine ::: (view all by) ::: April 11, 2015, 03:12 PM:

Kilo @#456: " This is not at all unprecedented in the internet world today: Those that agree with something give a post a "thumbs' up", those who disagree give a post a "thumbs' down" and those who are indifferent don't rate the post at all."

And having seen how it works in the internet world today gives me great trepidation about using it for an awards system.

#458 ::: Bruce Schneier ::: (view all by) ::: April 11, 2015, 03:21 PM:

P J Evans @453:

"452 You seem to keep missing the point that slates are toxic. The point is to not have them, or to find a way to minimize them, and still have a nomination process that's clear, fair, and open to all members."

I do not believe he is missing the point. He is looking for a voting system that minimizes their effect precisely because they are toxic. That's what some of these voting systems do.

Cheradenine @457:

Agreed. Anti-votes don't solve the problem; they just create even more problems.

#459 ::: Steve ::: (view all by) ::: April 11, 2015, 04:06 PM:

It almost feels like a recommender system is more appropriate than a voting system here. If the voting body largely agrees about what constitutes a good book, then the outcome of any reasonable voting system will produce a work which the voting body can agree is worthy, even if it wasn't their favorite.

The problem with the SPs is not that they have an different opinion about what consititutes an acceptable winner, it's that their opinion is so divergent from the rest of the electorate. I don't believe there is a choice out there that everyone could agree was a worthy winner of the Hugo (the SPs would reject everything not on the slate, and many others would reject anything on the slate which didn't compete against the whole field).

Is there a voting system out there which can produce a split result? No award seems like it would appropriately capture the lack of consensus, but I wonder if it would be possible to create a system which could produce a result like: 'No consensus: Group 1 prefers X, Group 2 prefers Y'

#460 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 04:25 PM:

@459: For the nomination phase, I don't see how what you're asking for is any different from what the proportional proposals actually give. True, they don't actually say "Look, nobody voted for both A and B, so these are two warring factions". But they do give each faction their proportional share; and, if the SRAV matrix were published, the factionalism would be clear.

As to the awards phase: I think the only answer is majority rules. Yes, there are some winners that are liked by all, and others that are loved by 51% and hated by 49%. But that's life. I don't see how any process can fix that, or even really why an awards process should want to.

Actually, no. I guess there is a way to "fix" that, if you really wanted. Majority judgment in the final phase would make the system with the highest median (that is, 50th percentile) grade into the winner. If you really wanted to, you could look at, say, the 60th percentile grade, to get "more edgy" winners, or the 40th percentile, to get "broader appeal". But I'd be surprised if even one fan in 10 thought that that was a good idea.

#461 ::: rcade ::: (view all by) ::: April 11, 2015, 04:44 PM:

What if nominated works have to get X number of votes to be on the ballot, and once one reaches that total it is publicly announced so there is no more incentive to nominate that work?

X could be set to the 5th, 10th or 15th place total the previous year in that category, depending on how many nominees we want.

Under this kind of system, because the fifth place best novel nominee in 2014 got 98 votes, any novel that got 98 votes in 2015 would have made the ballot. Anyone who tried to vote for it after that would be told by the vote-taking program, "That's already on the ballot. Please nominate something else."

#462 ::: albatross ::: (view all by) ::: April 11, 2015, 04:44 PM:

Steve:

I don't think you're right about the problem. The SPs have different but not completely distinct tastes from the rest of fandom, but that's not a problem--there must be dozens of groups of identifiable fans who have similarly different tastes. The issue is that the SPs made a list and got lots of people to nominate in a bloc, and that threatens to turn every year's nominations into some kind of slate / bloc nomination.

It's a mistake to focus too much on the SPs here. Because next year, it will be half a dozen slates, not just them, maybe not even mostly them. If you don't want the Hugos dominated by slate nominations and slate votes, then we need to work out how to adjust the nomination scheme so slates don't dominate non-slate nominees.

#463 ::: felice ::: (view all by) ::: April 11, 2015, 05:04 PM:

Fade Manley @ 449 : "...who gets to vote on the counter-slate?"

One option is to get the WorldCon to run it themselves as a trial for possible future changes to the Hugo. Use the same web form for both trial and actual nominations (so people don't have to fill in the data twice), but with two submit buttons - one for "Submit Trial Nominations" and one for "Submit Actual Nominations" (I'm assuming the only change is in counting - and taking rank into account is only a counting change). Then release the results of the trial before actual nominations close, so people who haven't yet actually nominated can take the results into account.

The trial could even try multiple different counting methods, and publish the results of each one so voters could see how they affect the outcome.

Jameson Quinn @ 451 : "GRRM has said on his blog that he thinks weighted votes are a bad idea ("spinach")."

Spinach is tasty and good for you. And in this case, the nominators don't have to eat it themselves, so there's no valid reason for them to object.

#464 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 05:22 PM:

Along with Brad@420, I do not find an approval type of system at all appropriate for a literary award. I can very easily find three works acceptable outcomes for the Hugo, and yet care passionately which of the three actually wins, and our current system lets me represent all that, whereas an approval system does not.

Related, people talk about our current system as being somehow intended to produce as broadly acceptable a result as possible; whereas I find approval systems to have that property and *not* ours. (I haven't read recent technical approval articles; it's possible in the last couple of decades systems with that name have evolved further afield than I think, but descriptions here do not sound that way.)

Somewhat related, in #417 counting info for the 2014 Hugos show that "out of 2657 ballots that ranked at least one of them, 2327 ranked Ancillary Justice above No Award, and only 480 ranked No Award above Ancillary Justice." Almost 1/5 of the voters with an opinion ranked the winning novel BELOW NO AWARD! And Steven describes this as "only"! This is a view on how broadly acceptable that win was. (That book is on my fairly urgent to-read list, people whose taste I think I know something about say it's very good indeed.)

#465 ::: P J Evans ::: (view all by) ::: April 11, 2015, 05:31 PM:

I don't know if this is true of online nominating and voting systems, but with mail-in, there's a peak in ballots received at the beginning, a drop to a very low rate, and then a sharp rise at the end to a much larger peak - we called it a 'bathtub curve with a shower' for the shape.
So anything that assumes a steady rate of ballots coming in may run into real problems.

#466 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 05:32 PM:

J Thomas@372 brings up something I hadn't considered. I consider the possibility (and hence the *need*) for strategic voting to be a severe drawback in a system.

However, his willingness to accept it and an example of a system with it illuminated something I had previously missed: In many ways, the drawback of strategic voting is that it requires understanding and thought by the voter. If the voter doesn't realize it's needed, they won't do it, and hence the result won't properly reflect their preferences and, perhaps, the results will be wrong (if enough voters don't get it right to affect the end result).

If the "strategic voting" is simple enough, and publicly included as part of the system, I may be coming around to thinking it's not inherently bad. (There's still the question of complexity, and how hard it is to get the choices right.)

I'm not sure I actually like the specific system described (or the multi-vote variation brought in later), and it does seem to have some issues of software security, and it seems to pretty much freeze out the possibility of paper vote-by-mail participants, which I don't think the Worldcon Business Meeting is ready for yet. (Since it's a complete change of system, the whole thing would be before the BM, and so the current requirement of paper ballots *could* be eliminated at the same time *if* that was the will of the BM, but I don't think it is.)

#467 ::: Cheradenine ::: (view all by) ::: April 11, 2015, 05:33 PM:

David @#464: IRV (the current method for the final vote) is more likely to produce a broadly acceptable winner than first-past-the-post, but less likely than Condorcet or approval methods. That said, I think the focus here has been on the nomination round, since that's the one that's presently badly broken. Approval-type methods seem like a good way to narrow down the field, even if they might not be appropriate to identify a winner in the final round.

#468 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 05:37 PM:

Steve@459: The results from 2014 that Steven desJardins posted @417 clearly show that the electorate very much do *not* agree on what constitutes a worthy book -- the winner was ranked *below No Award* by almost 1/5 of the voters who expressed an opinion on it.

#469 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 05:41 PM:

Cheradenine#467 thanks for confirming what I thought I knew. And I was looking a bit too closely at details and not at overall. For nominating, which I agree is the main focus pretty much all along here, I do see approval-type processes as fitting quite nicely.

(I trained as a mathematician 40 years ago but have worked in computer software my entire career, did an informal dive into voting theory in the early 90s, and feel a bit strange sometimes in this conversation, a weird combination of familiarity and being out of my depth.)

#470 ::: rcade ::: (view all by) ::: April 11, 2015, 05:47 PM:

David Dyer-Bennet @ #466:

I can't find the cite, but I read somewhere this week that only 3 nominating ballots were submitted on paper.

#471 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 06:05 PM:

C. Wingate@411: I see people talking about "helping a work win" as a bad thing a lot, and I don't think I really understand their thinking. And their thinking clearly confuses them about using the current voting system, which to me is amazingly simple and transparent.

So, on the theory that how *I* think about it conforms well with the actual way the current voting system works, let me try to describe it, in hopes it may be useful to others:

What ranking a work on an IRV ballot (that's I think the most broadly used official name for the current voting system) means is: I wish to express a preference for that work relative to the other works *that I rank*.

That's all you need. If you express such a preference, it gets taken into account in the vote-counting. If you do NOT express such a preference -- then you will have no influence on which of the two works wins.

So -- if you think two works would both embarass the Hugo awards, but still think one is better than the other, rank them both below no award, in the proper order. If you don't care which of your works below No Award win, then you can stop ranking after ranking No Award.

Some people see huge harm in ranking choices below no award, but so far as I can see it's based in a misunderstanding.

TRUE: If you rank works below no award, it is possible that your ballot will be in the pile that constitutes the winning majority for that work.

(Yeah, it's done by computer, but the pile metaphor is an accurate way to explain it.)

But that doesn't mean you "helped it win". At least, it doesn't mean you helped it win *over some decent work* (i.e. one you ranked above NO Award). Those works had already been eliminated from the election before this stage; your ballot was in one of their piles until then, but eventually it was the smallest pile, was eliminated, and your ballot got redistributed. You did your best for those works, but you failed. SOME work you ranked below No Award is going to win regardless.

So if you have the slightest preference between two works, always rank them both. There's no reason not to.

#472 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 06:08 PM:

rcade@470: You'll note that I carefully commented on the readiness of the Worldcon Business Meeting to take a position, rather than on the actual need or importance for paper ballots today :-) .

Perhaps, however, I over-estimate their conservatism if this issue came up as part of a system to solve the nominating slate problem.

Also, we're not supposed to be commenting on the politics here, and I apologize for bringing that in, which lead you to make a post solely about it and me to make this post. Sorry all!

#474 ::: Andrew M ::: (view all by) ::: April 11, 2015, 06:14 PM:

David Dyer-Bennett@471: I think the worry comes from the fear that people will reason like this: 'I will put Asimov first, and Clarke second. And I've no idea who Smith and Jones are, so I won't list them at all. But I hate Hubbard so much I will put him below No Award! That'll show him!' And of course such a person may be helping Hubbard win over Smith and Jones.

#475 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 06:16 PM:

Jameson Quinn@375 I'm not sure the formal measures of voting system goodness fully capture what I care about. It's possible, i.e. I can't at the moment disprove it, that my preference is purely emotional, though.

What I'm sure of is that my IRV ballot contains information that my approval ballot does NOT contain, and that the IRV counting process uses that information in ways that represent my preferences. Furthermore, deciding where to rank No Award in my IRV ballot is not as crucial a decision as deciding when to stop approving works in an approval ballot.

I find IRV voting much more natural and easier than approval voting in any complex situation.

#476 ::: David Dyer-Bennet ::: (view all by) ::: April 11, 2015, 06:17 PM:

Andrew M@474: and if that *is* how they end up there, then thinking about it my way actually *will* prevent them from making that mistake.

#477 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 07:02 PM:

Yes, DDB #471, Andrew M is right. It is fairly common for people to leave works they didn't read off the ballot entirely. It seems like the fair thing to do -- you didn't read it, so why would you express a preference about it?

But off the ballot means "(tied for) last place" Behind anything you ranked, including No Award and works ranked below No Award.

I think people have this strange emotional need to rank items after No Award to say, "See, this stunk so much I am saying it's worse than nothing!" They don't want to accept that the strongest displeasure is expressed by leaving it off the ballot instead.

I have a strong suspicion that if I were to have access to the raw ballots of the last few years, I could find a case where ranking something below No Award caused it to win the Hugo, or at least to increase its standing in the final result. All I need to find is a close race, and signs of the winner being ranked below NA by a few voters.

That's because, counter to our intuitions with STV, a the close races end up being decided by people who ranked the contenders 4th and 5th (vs. 5th and 4th.) or 4th and not.

Just from the published results, I may not be able to determine this, because it could be that the deciding votes were all lower than NA.

But be clear -- in a close race, your 4th place choice can be just as meaningful as somebody else's 1st place choice. It's one of the problems with IRV (and Condorcet) which makes some election theorists like weighted systems because in truth, a 4th place selection, particularly under NA is not as strong a sign of support as a 1st place selection, but when it gets down to the wire, it is considered the same.

I also am shocked that 330 ballots ranked No Award higher than Ancillary Justice (not over 500 as claimed above.) AJ is, in my view, clearly a masterpiece, and I think other concur.

What is troubling is that in some ways, I think the real purpose of the Hugo award is not to honour the best of the year. Rather it is to make sure we recognize the masterpieces and tell the world about them. There is not a masterpiece every year -- I think there have been perhaps 9 in the 21st century -- but some years there are two masterpieces. I think the essence of the No Award rule is there to express this idea, that there need not be a winner every year. In practice, it's never been used outside the dreadful decade of '70s bad SF movies.

And yes, while AJ was a masterpiece, I think that Ancillary Sword was a "very good" book but not close to its predecessor, but it's likely to win its abbreviated category, I suspect.

== Voting works you like after No Award in 2015 ==

One other thought, as yet unspoken on No Award. It might be that the right ballot choice for some will be to list No Award first, the non-slate works in order of preference, and then no ranking for the slate works. One would cast this ballot to say, "This category was invalidated this year, give No Award. However, if an award is to be given, it should be for the works as ordered. No consideration should be given to the slated works, so that even if they do win, it will be with small vote totals."

#478 ::: P J Evans ::: (view all by) ::: April 11, 2015, 07:03 PM:

475
It feels to me like some people get so caught up in "let's make this as perfect a system as possible" that they forget that the people using it want something that's easy to understand and use.
It's hard enough to get them to understand IRV....

#479 ::: KayTei ::: (view all by) ::: April 11, 2015, 07:19 PM:

David @ 476

I don't see how that follows, because my internal characterization of my choice doesn't affect the mathematical outcome.

If I vote for

A
B
No Award
C

...and if the vote then comes down to a runoff between C & D, then my vote for C will be counted, even though, had I read D, I might have preferred it to C. I am still benefitting C over D, regardless of my intent to harm C by placing it lower on the ballot.

The choice your model overlooks is the "even a book I have not read might be better than C and so I am going to leave that decision up to the rest of the voting electorate rather than risk helping C win over another, potentially worthier, candidate" vote. That vote can only be accomplished by voting "No award" then leaving C off the ballot entirely.

#480 ::: KayTei ::: (view all by) ::: April 11, 2015, 07:28 PM:

Following up to my 479, I suppose there is the other alternative, which is "vote what you know, put the last guy(s) last, and randomly insert everything else on the ballot in the middle."

I mean, it has its problems as a solution, but it's one way to ensure that you help the bad guys lose, if that's your goal, even starting with imperfect information.

#481 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 07:35 PM:

@466: You say that strategy may not be too much of a problem if it's obvious what strategy would work.

In voting theory, strategy tends to be less of a concern for multi-winner systems like the nomination process we're mostly discussing here. There are enough ways to minimize it that it's not the first thing people worry about.

For single-winner systems like the final round, however, it's a bigger deal. A lot of single-winner voting theory is about strategy; for instance, how to find a system where the strategic vote and the obvious vote are as close as possible as often as possible. In that regard, plurality is clearly the worst system (except perhaps Borda); IRV is a bit better; and everything else (approval, score, Condorcet, MJ, SODA, etc.) is better than that. Approval is a bit of a special case; it's a system that's easy to explain and gets good results, but the way it deals with strategy is basically to make all votes strategic. Some people don't like that (see @475 above). The problem is, if you give up on approval's simplicity, you really are "opening pandora's box" for arguments about which of the other systems is best. Such arguments are mostly a waste of breath (see bansinglemarkballots.com for a statement on why, and no, that's not me promoting my website) but they can be hard to avoid. As an experienced voting theorist, I'd recommend MJ for this use case, but I'd support just about anything except plurality or borda as being good enough.

But anyway, fixing the single-winner system in the final round is not what's urgent here.

...

@463 suggests that if there's a "primary" for determining a "non-slate-slate", that it should

1. be run by WorldCon if possible, alongside the regular election

2. report the tally using various voting systems

I think these are good ideas. I'd add:

3. Results are not visible until you've submitted a vote. That way, the unofficial vote is basically what you would have voted if there were no poll.

4. Suggest that voters vote "3 or more" of their favorites from the top 5 poll results. So you'd still have room to officially nominate your favorite two works even if they weren't doing well in the unofficial poll. If everyone does this, it should still be robust enough not to be overwhelmed by puppy-type slates.

#482 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 07:54 PM:

One example where poor No Award ranking may have played a role. In 2010 there was a tie for best novel. In the results:

654 ballots ranked The City & The City by China Miéville higher than No Award
55 ballots ranked No Award higher than The City & The City by China Miéville
617 ballots ranked The Windup Girl by Paolo Bacigalupi higher than No Award
61 ballots ranked No Award higher than The Windup Girl by Paolo Bacigalupi

There were a total of 875 ballots. 108 of them ranked No Award. 11 ranked No Award and nothing else. One ranked No Award first and had something below it. We can't tell how many ranked the winners below NA, or if it's sure that one of the 55 or more ballots that ranked the winners below No Award happened to rank them both below it.

But if just one of those 55 or more ballots ranked one of them and not the other, then this is an example of somebody ranking a work below No Award caused it to win or tie for a Hugo. It is not the hypothetical exercise that some suggest.

#483 ::: CHip ::: (view all by) ::: April 11, 2015, 07:56 PM:

DDB: wrt agreement on a worthy book, looking at the statistics themselves rather than #417 suggests the fraction disliking AJ is 1/8 (330 out of 2657), not 1/5; if I read the table correctly, another 480 expressed no opinion. I would also note that AJ was one of the few pieces of fiction in \any/ year to have a majority when there were still two other works in the running, which I'd say is a fair indicator that it was held in high regard. On the gripping hand, how often have you seen even 80% of fans agree on anything?

#484 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 08:09 PM:

Fans do not agree, CHip, but I must admit I am surprised so many fans would say that they would rather see no award given than hand one to Ancillary Justice. (But as I said, I view it as a masterpiece, and tastes obviously do differ.)

But as I also said, the goal is not so much to measure award support perfectly, but to assure that when a masterpiece comes along, the only reason it doesn't win is that there was the very rare event of another masterpiece that year.

#485 ::: felice ::: (view all by) ::: April 11, 2015, 08:30 PM:

Jameson Quinn @ 481: "3. Results are not visible until you've submitted a vote.... 4. Suggest that voters vote "3 or more" of their favorites from the top 5 poll results."

If the current results are made visible people who've nominated in the unofficial poll, they'll be leaked publicly, but I don't think there's any need to keep the current state of nominations secret. In fact, I think the unofficial nominations need to be made public while still open, to stop the Puppies spamming them too (different counting methods help, but concentrating the non-slate vote is also necessary). I think it would be inappropriate for the WorldCon to officially suggest letting the unofficial poll influence actual nominations, but I agree completely that that's what people should do, and it should be recommended through unofficial channels.

Brad from Sunnyvale @ 482: "if just one of those 55 or more ballots ranked one of them and not the other, then this is an example of somebody ranking a work below No Award caused it to win or tie for a Hugo"

It would have caused it to win over or tie with another work that they also ranked below No Award (ranking No Award at all is ranking it above any unranked works). There is no possibility of a work they consider worthy getting the award, and no possibility of No Award winning; this is not making the outcome worse for them.

#486 ::: Keith "Kilo" Watt ::: (view all by) ::: April 11, 2015, 08:32 PM:

I hesitate to suggest this, because I think it would be a logistical nightmare (but maybe not IF all nominating and voting are 100% online). The simplest fix of all is to simply put every nomination on the ballot. You could optionally restrict people to one nomination per category. The problem with slates isn't with voting, it's with controlling the ballot that is voted on, after all. We may end up with 200 nominations per category, of course, but you still don't have to vote for anything that doesn't seem interesting.

Of course, this would likely mean no more Hugo packets and "Hugo nominated" is no longer a selling point (though "Hugo winner" is unaffected), but given the alternative of not getting a representative ballot at all, that seems to be a small price to pay.

It's simple, easy to manage (only if done online), and solves all of the problems, all at the cost of the two items mentioned above. I really don't think any of the complicated systems that are being discussed are going to be workable in the long run.

Opinions?
Kilo

#487 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 08:49 PM:

Sorry, Felice, but as I have explained, in the situation outlined, if a voter ranked one of those winners below no award, and then did not rank the other winner because they did not read it or for other reasons did not care, then their ballot gave the work a Hugo, where otherwise it would have lost. (Unless countered by another ballot with the opposite mistake.)

So yes, most definitely, you can, by ranking a work below No Award cause it to win the Hugo when it otherwise would have lost, and more to the point, it seems likely (though I don't have enough data to be sure) that this has happened, probably many times.

So just don't do it unless you are sure you know what you are doing and are ranking even works you did not read. Better still, just don't do it, unless you are ready to contribute to a victory for the unworthy work.

There is another, perhaps unique to 2015 reason not to do it. Even if you do it right (which not everybody does) and you end up simply helping decide between Piece of Shit A and Piece of crap B when they are the only choices left, you will be adding to their final vote total. The final release results will say, "This work was given the Hugo by 1,000 voters" and you will be one of them. Don't be.

If a work from one of the slates is to win the Hugo, let's have it win over its opponent with 250 total votes, not 1,000.

#488 ::: Steven desJardins ::: (view all by) ::: April 11, 2015, 09:02 PM:

Keith, if the voters don't have time to read and rank everything on the final ballot, then that wrecks the Hugos as thoroughly as anything the Puppies might attempt.

#489 ::: Cat ::: (view all by) ::: April 11, 2015, 09:08 PM:

How about a confirmation step?

Hold the nominations as usual.

The top 15 winners in each category are published, perhaps in reverse alphabetical order or something so nobody knows who has the highest scores. Everyone eligible to nominate or vote on the Hugos for that year gets an e-mail to come check out the list. The voters/nominators get to vote thumbs up or thumbs down on each nomination. Nominees that get voted down by more than half the people (or some supermajority if we want to specify it that way) who vote at this stage get cut from the finalists, and the next work down promoted. The authors of the top 15 pieces are all contacted, told they are in the top 15, and asked if they wish to decline the nomination if they receive it. If they decline, and are among the top five, the nomination passes to the next work down also in the usual way.

A work from a slate only gets passed through to the ballot if the majority of the voters (or whatever size minority defeats the supermajority specified above) agree that it deserves to be on the Hugo ballot. And if more than half (or whatever size minority) agree it belongs there, then it does; this is a democracy.

Then you have the final step, where the works that survive the confirmation and opportunity to decline end up on the Hugo ballot and the voting proceeds as normal.

Someone suggested this to me; do people see obvious holes? There is the whole adding an extra step thing, but pretty much every idea I've seen here seemed to add complexity to the process.

#490 ::: felice ::: (view all by) ::: April 11, 2015, 09:10 PM:

Brad from Sunnyvale @ 487: "if a voter ranked one of those winners below no award, and then did not rank the other winner because they did not read it or for other reasons did not care"

In that situation, yes, you're right. But we have no way of knowing if any of those 97 voters didn't know what they were doing. Some - possibly most - of them would have ranked all the works, others would have left off their least favourites understanding what that meant. We're dealing with a very small number of people who might have failed to understand the system, and would have to ask them to find out for sure; it's not something that can be determined from the data. And even in the very worst case, they're only supporting something they didn't like over something they have no opinion on, not over something they actually wanted to win. I think the advantages of being able to rank works below no award far outweigh the disadvantages.

#491 ::: Keith "Kilo" Watt ::: (view all by) ::: April 11, 2015, 09:11 PM:

Steven@488: I'm not convinced that's true. Can you explain your reasoning?

Kilo

#492 ::: Keith "Kilo" Watt ::: (view all by) ::: April 11, 2015, 09:11 PM:

Steven@488: I'm not convinced that's true. Can you explain your reasoning?

Kilo

#493 ::: Keith "Kilo" Watt ::: (view all by) ::: April 11, 2015, 09:12 PM:

Steven@488: I'm not convinced that's true. Can you explain your reasoning?

Kilo

#494 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 09:22 PM:

#489 would add another layer, but it would probably work well against slates, unless the slaters are 3 times as many as needed to fill a 5 candidate ballot.

Understand though, that people would be giving their thumbs up in most cases having read only a few of them. 15 is too many (except perhaps in short story) to ask for serious evaluation.

Slates work because the slaters are concentrated in their choices while others are broadly distributed. This makes the others concentrated as well.

------

Felice #490. As indicated, in this particular case I don't have data to be conclusive, but based on other ballots, and past Hugo patterns where many ballots are not fully filled out, I think it is a very safe conclusion that at least one of the 50 to 100 ballots in question would have done this. These two works were quite different with different fan bases. I think it's a strong assertion that every single ballot was good. If I did harder -- or get access to full ballot data -- I am guessing I could find other examples, perhaps many other examples.

So it's really simple. Tell people that "if a work is not worthy of the award, leave it off, in spite of your instinct that ranking it below No Award is somehow more satisfying. Only a computer is going to look at your ballot, so nobody else will see how strongly you expressed your distaste."

#495 ::: Randolph ::: (view all by) ::: April 11, 2015, 09:23 PM:

Cat@489: that looks pretty good to me, though I'd like to hear what the voting systems experts say about it. It does have a strong advantage of legibility--it is easy to explain--, and a strong disadvantage of requiring, at least, e-mail access and additional time on the part of voters, which is always scarce. Hunh. Opportunities for heavy-duty negative campaigning might also be a problem.

#496 ::: Soon Lee ::: (view all by) ::: April 11, 2015, 09:30 PM:

P J Evans #478:

Agree wholeheartedly. Make it fairer, but please don't make it more difficult to nominate & vote.

Not so long ago we were discussing how to increase Worldcon & Hugo participation. This current brouhaha (thank you Steve Right) shouldn't change that.

#497 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 10:00 PM:

Another chaotic result that actually happened, the 1998 Novelette - already modestly controversial:

BEST NOVELETTE 545 ballots counted

"We Will Drink A Fish Together..."                                 122 122 150 165 218 378
"Three Hearings on the Existence of Snakes in the Human Bloodstream" 115 115 135 160 218
"Moon Six" 107 108 120 160
"Broken Symmetry" 99 99 105
"The Undiscovered" 75 75
No Award 27

A bit hard to read with the formatting, but the reason this was strange was the unusual tie at the end. "Drink a Fish" had 165, and "Snakes" and "Moon Six" had 160 each. The Hugo administrator interpreted the ambiguous rule of eliminating the "last place" entrant and eliminated both #2 and #3, giving the Hugo to "Drink a Fish."

I asked the Hugo committee that time what would have happened in a Condorcet, and it was a tie between Drink a Fish and Snakes, as shown by the confusing line where they each have 218. "Moon Six" does not win.

I often wonder what that Hugo would have meant for the career of Jim Gardner, a friend and former employee of mine (thus my curiousity).

I now notice something else. Somebody ranked Moon Six behind No Award on this ballot, but did not appear to rank Snakes. When No Award was eliminated, that vote went to Moon Six, allowing it to tie with Snakes in that last round, causing them both to be eliminated. So the voter didn't give a Hugo to his/her disliked work, but it did alter who got the Hugo, and another small tweak might have had the chance to help Moon 6 itself.

#498 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 10:07 PM:

@478:

It feels to me like some people get so caught up in "let's make this as perfect a system as possible" that they forget that the people using it want something that's easy to understand and use.
It's hard enough to get them to understand IRV....

The problem is, there's a tension between "easy to understand" and "easy to use". For instance, SODA (a delegated system; not appropriate here, because you can't delegate your vote to a book, I'm just using it as an example) is extremely easy to use; most people can just vote for their favorite and go home, and (unlike in plurality) still be sure that their vote is strategically optimal. But it's hard to explain fully; a 3-step process, which includes a virtual copy of inverse Schulze voting; not for the faint of heart. On the other hand, approval can be explained in four words — "vote for any number" — but in order to vote most effectively, you have to know who the frontrunners are, and not have a problem with giving the same rating to a lesser evil as to your favorite.

In this case, the reason for this discussion is this year's disproportionate result. So it's pretty clear (to me, at least) that the solution is some kind of proportional system. No proportional systems are really easy to explain; though several, including RAV, are no worse than IRV. So I think the priority should be on "easy to vote". And in that regard, both RAV and SDV are very good.

So I'm not really sure what the complaint is about. Unless it's really about the logorrhea of people like me. In which case... point taken.

#499 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 10:11 PM:

@489, @495: This is not a problem from the point of view of voting theory. From a logistical point of view, though, I can imagine that people don't really want to add extra rounds unless they have to; more work for both voters and organizers. Especially since, for a conscientious voter, it would mean 3 times as many things to have to read.

#500 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 10:13 PM:

@497: Yep, that's exactly the kind of strange outcome that you can get with IRV, and that MJ or Condorcet would avoid. But again, right now is probably not the best time to fix that.

#501 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 10:13 PM:

@497: Yep, that's exactly the kind of strange outcome that you can get with IRV, and that MJ or Condorcet would avoid. But again, right now is probably not the best time to fix that.

#502 ::: P J Evans ::: (view all by) ::: April 11, 2015, 10:20 PM:

498
You're talking about stuff that you haven't explained clearly. Yes, there are lots of systems of voting. For any given purpose, one may be better than another.

But can you for Ghu's sake remember that real people are going to be using this, not experts in voting systems, and that many of them will not have degrees in anything relevant to whatever argument you're trying to make? (There's no lower age limit that I know of on voting, either.)

#503 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 10:35 PM:

@502: I hear you. You want simple. My proposal, reweighted approval voting, is simple to use for the voters. You just vote the same way you're doing now. The only difference is, people with slates (like puppies) can't unfairly overwhelm your vote.

I've said a lot that isn't simple here. Voting theory had its recondite corners, and I enjoy talking about all that. But the previous paragraph is all you need to know.

#504 ::: Jameson Quinn ::: (view all by) ::: April 11, 2015, 10:42 PM:

(I don't want to be patronizing, and if I sounded that way, I apologize. You can be a smart person and not want to worry about details of voting theory.

(On the other hand, in my other posts here, when I use jargon, there's a reason I don't carefully explain each term starting from first principles. Fields develop jargon because it's simply not feasible to do that all the time. If you don't understand something in particular [and it seems as if it matters], just ask, and I'd be happy to explain.)

#505 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 11, 2015, 10:47 PM:

Brad from Sunnyvale @477:

What is troubling is that in some ways, I think the real purpose of the Hugo award is not to honour the best of the year.
The purpose of the Best Novel Hugo is to give a rocket to the work nominated in that category that's still standing after the Australian Ballot eliminations and redistributions have been performed. The exact nature of this victory is not spelled out in the rules.

This is a surprising thing I discovered many years ago about basic fannish institutions. Their documentation tends to be very brief and to the point, and concentrates on describing the institution's mechanisms, not explaining its higher purposes.

Our institutions aren't the content. They're the channels through which it flows.

#506 ::: Erwin ::: (view all by) ::: April 11, 2015, 10:48 PM:

Wait and watch for a year or two would probably be best.

How much work would it be to try out a few voting systems on previously cast Hugo ballots? Particularly those of the last 5-6 years. Tests along those lines might identify some of the zero-order problems with the proposals. I'd expect that it would have helped with the last election.

Anyways, #1 might be preferable to the current system independent of slate voting - as it will tend to nominate across a broad range of tastes.

Ties are easy - just mention that, for ties, the one with more first place, et cetera, nominations will win and then mention that it'll hardly ever happen.

Effective slates really are bad because the end result - if people optimize - will be a two-party system. I wonder about the option of just asking people to nominate at most one or two works per category.

Multi-primaries are a bad idea because they are likely to be highly gameable. The problem will be that voter turnout decreases for the 'unimportant or inconvenient' elections.

#507 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 11, 2015, 10:51 PM:

Jameson, "simple" is not an adjective I often reach for in conversations with P J Evans.

#508 ::: Bruce Schneier ::: (view all by) ::: April 11, 2015, 10:54 PM:

rcade #461:

"What if nominated works have to get X number of votes to be on the ballot, and once one reaches that total it is publicly announced so there is no more incentive to nominate that work?"

I don't think any real-time feedback system is workable in practice. Sure, everyone reading this thread is going to pay attention, but the 1000 people who nominate won't. (Data from 2013, a year where the blocs didn't have much power, is ">here.) And we're going to want even more people to nominate in the future. We want to make it as easy as possible. Also, easy to administer.

felice @463:

"One option is to get the WorldCon to run it themselves as a trial for possible future changes to the Hugo. Use the same web form for both trial and actual nominations (so people don't have to fill in the data twice), but with two submit buttons - one for 'Submit Trial Nominations' and one for 'Submit Actual Nominations' (I'm assuming the only change is in counting - and taking rank into account is only a counting change). Then release the results of the trial before actual nominations close, so people who haven't yet actually nominated can take the results into account."

Think: easy. We don't want something that's more complicated for the nominating voter.

David Dyer-Bennet @464:

"Along with Brad@420, I do not find an approval type of system at all appropriate for a literary award. I can very easily find three works acceptable outcomes for the Hugo, and yet care passionately which of the three actually wins, and our current system lets me represent all that, whereas an approval system does not."

For the nominations election -- which is what we're talking about -- an approval system can work. A list of acceptable outcomes is what we want in a Hugo ballot.

P J Evans @478:

"475 It feels to me like some people get so caught up in 'let's make this as perfect a system as possible' that they forget that the people using it want something that's easy to understand and use."

Yes, it's an easy trap to fall into.

Keith "Kilo" Watt @486:

"I hesitate to suggest this, because I think it would be a logistical nightmare (but maybe not IF all nominating and voting are 100% online). The simplest fix of all is to simply put every nomination on the ballot. You could optionally restrict people to one nomination per category. The problem with slates isn't with voting, it's with controlling the ballot that is voted on, after all. We may end up with 200 nominations per category, of course, but you still don't have to vote for anything that doesn't seem interesting."

The whole point of a nomination process is to avoid this. Nominations are good for all sorts of reasons. In our case, one of the big ones is that voters like to have a reading list.

Cat @489:

"How about a confirmation step?"

Any change that adds a round of voting isn't going to fly. Two rounds are enough work for the administrators and voters as it is.

Jameson Quinn @503:

"@502: I hear you. You want simple. My proposal, reweighted approval voting, is simple to use for the voters. You just vote the same way you're doing now. The only difference is, people with slates (like puppies) can't unfairly overwhelm your vote."

RAV is simple to use, yes. My worry is that it's hard to explain.

Everyone:

The discussion of the final ballot, and how the IRV voting process interacts with No Award, is interesting -- but not the topic of this thread.

#509 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 10:56 PM:

I would sum up RAV simply as "The more of your way you are getting, the less likely you are to get more of it."

However, RAV suffers from the same thing many other proposals do, there is still strategy, there are still incentives to not vote for your honest choice.

That's always true if you have six choices you want to support -- you must leave one off. But in this case, you have an incentive to leave off the works you think will be most popular, even if you only want to nominate 2 works. Ie. if your picks are a known popular work and a more obscure one, you reduce your help for the obscure one significantly if you list more popular ones on your ballot. This is true even today, but to a much lesser degree.

A slate will still get a couple of works on the ballot with RAV, but find it hard to swamp it. The latter is indeed worse, but there are solutions that even prevent the former.

#510 ::: Lady Kay ::: (view all by) ::: April 11, 2015, 10:57 PM:

Having bought a child's membership for 2014, it did not come with voting rights.

#511 ::: Soon Lee ::: (view all by) ::: April 11, 2015, 11:06 PM:

Erwin #506:

I keep having Vorkosigan quotes running through my head:

Gregor Vorbarra: "Let's just… see what happens."

Aral Vorkosigan: "Meaning is what you bring to things, not what you take from them."

Cordelia Vorkosigan: "When you choose an action, you choose the consequences of that action."

#512 ::: Brad from Sunnyvale ::: (view all by) ::: April 11, 2015, 11:07 PM:

Bruce (#508) -- yes, I know Approval is mostly fine for nominating. It is in fact what we use today -- Approval with 5 winners and a few minor caveats. It has, sadly, failed. I was responding to what I thought was a proposal for Approval in the final ballot.

Ok, sorry about the topic drift.

Back to nomination, I would challenge main of the claims that having another round or even live data are "too much work" for administrators. While I am not necessarily supporting these proposals, I think that in today's modern web era, where I would guess that hundreds of fans make their living building interactive web applications with the large suite of modern tools we have, that if this is what needs to be done, it can be done, and done entirely by machines once working.

I have concern primarily over things that are a barrier to the members. More complexity will reduce participation, at least possibly.

Another reason I resist systems which have individual strategy (as opposed to group strategy like slates) is that such systems are inherently more complex for the member, who has to now evaluate their ballot on a strategic basis if they want to do the best they can. The closer a system is to "List what you truly thought was best," the better it is, even if that takes a bit more work to use or administer.

#513 ::: J Thomas ::: (view all by) ::: April 11, 2015, 11:40 PM:

@486 Keith "Kilo" Watt

You suggest letting the final voters vote for all nominations. Then a cabal cannot crowd out other nominations with block nominating.

Let's look at some details. Say the cabal is 20% of the voters. If we have a nominating system which lets the cabal get one candidate while everybody else gets four candidates, then the cabal cannot win provided all the other voters vote for at least one noncabal candidate and do not vote for the cabal candidate. They will have 20% of the vote and it takes more than 20% to win.

But if there are 200 nominations on the final ballot, then one cabal choice (or all cabal choices, depending on the voting system) gets 20% of the vote. Is there any reason to think that any of the others among the 200 will get 20%?

When there are only 5 choices, one of them has to get more than 20% unless No Award wins some. When there are 200, the final vote can be as scattered as traditional nominations.

The old voting system provided a way to generate consensus, by artificially limiting the choices in the final election.

If we can't stop people voting for a slate, then slate voters will win unless we find a way that other voters can agree on some other winner. If they can't tell which of 200 choices is popular with other voters, they don't know which of them has a chance to beat out cabal voters.

The attempt to prevent cabals is almost like we're saying, "Fans never agree so people who agree have an unfair advantage. We need to do something about people who agree with each other. Pass a rule against it or something."

We would do better to find a way that fans can agree. It shouldn't start out as "Here's my list, come join my slate." Better that they have some process to work it out with discussion or something, and come to an agreement if they feel like it.

#514 ::: P J Evans ::: (view all by) ::: April 12, 2015, 12:15 AM:

507
I've been known to ask for explanations intended for the ignorant. You get so much better answers then.

Jameson, I translated the previous ballot-counting program (1971-72) from PL/I (full PL/I: it used some very interesting functions) into Pascal - it was 1984, I had to use what was available on the computer that was going to be used. I don't want to have to get into it at that level again.

If you can stop thinking of this as a resume item, maybe you can understand where we're coming from. We're the people who will be implementing and using whatever is decided: we have to understand it well enough to make it work.

#515 ::: P J Evans ::: (view all by) ::: April 12, 2015, 12:18 AM:

510
The people under 18 who don't have children's membership will have voting privileges. (There are fans who have been going to conventions literally from babyhood.)

#516 ::: Bill Stewart ::: (view all by) ::: April 12, 2015, 12:24 AM:

DD-B@471 - Of course, the problem with reading and merit-ranking the slate nominees (whether above or below N/A) is that to do that, you've got to read works that Beale wanted to nominate :-)
Based on the short story and other rants by Wright, I doubt I'll want to read all three of his novellas. One of Beale and two of Torgersen were more than enough last year. Correia's writing was fun, but he withdrew.

#517 ::: David Dyer-Bennet ::: (view all by) ::: April 12, 2015, 01:35 AM:

Bill@516: Ah, but I didn't specify "merit ranking", I merely specified that you knew your preference order for the works :-). It could be based on your perception of story enjoyability based on reading them, but it could just as well be based on whether the story appears on the blacklist you get from that PO Box in Schenectady.

#518 ::: David Dyer-Bennet ::: (view all by) ::: April 12, 2015, 01:45 AM:

Brad@497: I believe that's exactly the corner case that the Minn-StF version of IRV goes to great lengths to avoid. We look for any possible way to break the tie (like relative preference between the two in the pile of votes for the leader), and finally resort to a random drop if necessary rather than throwing out the tied other two candidates. Which shows how much we didn't like the idea of what happened, happening, I guess. Interesting to see it come up in real data; we only found it in programming.

#519 ::: David Dyer-Bennet ::: (view all by) ::: April 12, 2015, 01:53 AM:

Brad@477: " the strongest displeasure is expressed by leaving it off the ballot instead " -- that's simply false. The strongest displeasure is expressed by ranking it *below every single other thing on the ballot including No Award*.

But, by definition, you can only express the *strongest* displeasure about one item; if you feel that way about two items, then it's not the strongest, there are two you feel equally strong displeasure about.

The thing is "below No Award" is not magic in the voting, and people seem to think it is.

#520 ::: KayTei ::: (view all by) ::: April 12, 2015, 02:07 AM:

Steven desJardins @ 488

You can increase participation or you can increase stringency. I'm pretty sure this is a "choose one" situation - given the scope of the ballot and the number of items on it, I suspect that "read everything" is not a reasonable expectation for many people, especially if books are being nominated mid-series.

It's a fan-favorite award; I also suspect that a lot of votes and nominations are based on cumulative choices people have been making for years that drive them to read certain types of authors and stories and not others.

I don't think this ruins the Hugos. I think it reflects the subjective nature of the award, which is not about rigorous analysis and comparison, but which seems to me rather to attempt to capture a point-in-time of the constantly changing preferences and vagaries of fans, as they react to the full range of available stories.

Differently useful is not un-useful - I appreciate having an award that tells me about what ordinary fans want to read, rather than professional lit critics or marketing departments. I don't need recommendations that perfectly rank books in objective order from best to worst; I just need a process that reliably turns out really solid and interesting recommendations that I can selectively pursue if they line up with my already existing biases.

#521 ::: Steven desJardins ::: (view all by) ::: April 12, 2015, 03:02 AM:

KayTei, "reading everything" is a much more reasonable expectation if there are five novels on the ballot than if there are two hundred. But I'm happy to substitute "making an informed judgment": it's often easy to tell from a small portion of a nominated work that it's not very good.

The majority of the voters last year do appear to have ranked all five novel nominees, which is how the Hugos manage to be a fan-favorite award. It's possible to win the first round of balloting with a narrow but passionate base of support, but you can't win the Hugo unless a broad spectrum of fandom thinks it's pretty good.

Shifting to a system where there are too many nominees for anyone to evaluate breaks that: it becomes more important to have a narrow, passionate base of support than to have a majority that actually likes the book, because no book on the ballot will have been read by a majority of voters.

I don't see how you get from my desire to have voters read and evaluate all the works on the final ballot (or, at least, to evaluate them) to the idea that I want people to rigorously analyze and compare the works, or give professional lit critics and marketing departments more influence. I think you're either ascribing opinions to me that I don't hold, or making some sort of conceptual leap that I can't follow.

(And, of course, the problem of evaluating a work mid-series applies to only a few categories; generally, it's fairly easy to read all five novella nominees, but it wouldn't be easy to read fifty.)

#522 ::: KayTei ::: (view all by) ::: April 12, 2015, 06:36 AM:

Steven @ 521

Okay, hang on. I didn't attribute anything to you except your initial statement. In a conversation about differing voting philosophies, yes, I did lay out my detailed way of thinking about this, precisely because everyone is coming from different angles and I am aware I have not yet mastered the cyberpsychic arts.

Your explanation makes sense as one valid approach. But it does sound to me like letting the perfect be the enemy of the good. I suspect that is because we are looking for two different things. I thought it was worth acknowledging that diversity of thought. That's all.

#523 ::: J Thomas ::: (view all by) ::: April 12, 2015, 08:33 AM:

@521 Steven desJardins

"reading everything" is a much more reasonable expectation if there are five novels on the ballot than if there are two hundred.

Yes.

Shifting to a system where there are too many nominees for anyone to evaluate breaks that: it becomes more important to have a narrow, passionate base of support than to have a majority that actually likes the book, because no book on the ballot will have been read by a majority of voters.

In 1957 a lot of fans might have read most of the books under consideration. No longer, there is too much published. So a great work which has not been read by many people will not get many votes if it is nominated, or it will not get many nominations if it is screened out at that stage. I don't see that there is any way around this.

But a great work which has been adequately marketed, which has had good word-of-mouth spread, will have been read a lot. A lot of people like it, or they wouldn't be recommending it to their friends. Those are the stories that have a good chance to win.

I don't see that we get better decisions by choosing five in a somewhat-arbitrary way and then getting a lot of people to read those five.

But there might be a cultural reason to do that. SF has become so diverse that people might have very different concepts. Fifty years ago there was a set of concepts you could expect readers to catch -- spacecraft, vacuum suits, relativity, psi, etc. Now there are so many more that you can't be sure what readers will get. If we can establish a canon, a set of core texts that most people have read, then we have a set of memes that writers can depend on readers to have seen before.

The Hugos give us a way to encourage a lot of fandom to read 5 particular books a year. Maybe that's valuable.

If all the nominations were published, people could use them to get an idea what's popular and how popular it is. But they might ignore them.

In other discussions I've been hearing some people say "This is an Important, Serious, Work Of Art. I didn't enjoy reading it, I didn't like it, but it has great literary value so it deserves the Hugo. I don't vote for books I like, I vote for books that are literary masterpieces regardless whether I like them or anybody likes them."

I have no respect for that idea. But somehow I can respect "This book has the concepts you will need to understand a lot of the SF coming out in the next ten years".

As opposed to "This is masterfully done, everybody will enjoy it. The sad parts will make you cry. The victories will leave you elated. The jokes are the funniest. You'll feel good for a week after you read this. But it breaks no new ground. No Hugo."

I'm not sure whether it's a good thing to have 5 nominees we can try to get everybody to read. In the short run, to me it seems arbitrary and useless. But it could have bigger implications, maybe some I don't imagine.

#524 ::: Cat ::: (view all by) ::: April 12, 2015, 08:43 AM:

Bruce @508

A confirmation step would be more work for the Hugo administrators, it is true.

But the elaborate systems of intermediate counting and reweighting of votes under discussion here strike me as calling for more work on the part of the Hugo administrators also.

And a confirmation step has the advantage of being easy to explain and understand, which these other systems don't. Furthermore it occurs to me that there may be a fear that someone acting in bad faith while writing the counting program---or even just making a mistake--would be practically invisible in a complicated system like this, a disadvantage I think a confirmation step would not share.

In addition all this reweighting of votes stuff is just giving in to slates--giving them one place on the ballot instead of all five, which I agree is better, but why does a slate deserve *any* place on the ballot when the rest of the nominators read what they thought they would like and chose their favorites honestly?

A confirmation step gives the majority a chance to squash the bad-faith actors before they get on the ballot. It uses human judgement, which I think people will see as a strength rather than a weakness.

Its weakness is that it relies on Hugo voters knowing what slates are out there, which right now is not much of a problem for anyone who is checking, but which could become a problem if slates are successful in the next couple of years, and breed more slates.

#525 ::: J Thomas ::: (view all by) ::: April 12, 2015, 09:34 AM:

@524 Cat

In addition all this reweighting of votes stuff is just giving in to slates--giving them one place on the ballot instead of all five, which I agree is better, but why does a slate deserve *any* place on the ballot when the rest of the nominators read what they thought they would like and chose their favorites honestly?

From my perspective, we are looking at ways to punish people for agreeing.

"Those people like the same books! Throw out their votes!"

Only in fandom....

I think I see how we got to here, but (to me) it looks like a place to back away from.

We don't need a way to punish people who agree. We need a way that people who start out with diverse opinions can approach agreement. If actual fans can come up with, say, five nominations that they mostly agree on, then the system is robust against minorities who agree.

The problem is not that slates agree too much, it's that fans don't agree enough. Any system that gives fans a chance to come to agreement, will beat the slates without having to throw away their votes.

#526 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 09:59 AM:

@514: OK. So, what is it that you don't understand? I think I've been clear about the RAV algorithm and its rationale, but obviously there's something that's not clear. Just ask. (Or if people like SDV better, I can explain that further too.)

#527 ::: Bruce Schneier ::: (view all by) ::: April 12, 2015, 10:15 AM:

Kat @524:

"A confirmation step would be more work for the Hugo administrators, it is true. But the elaborate systems of intermediate counting and reweighting of votes under discussion here strike me as calling for more work on the part of the Hugo administrators also."

We need to separate the two types of work. An additional round of voting/confirmation/whatever requires significantly more work on the part of the Hugo administrators every year. A more elaborate voting system requires more work on the part of a programmer once, and then exactly the same work on the part of the Hugo administrators who are simply running the voting program.

#528 ::: abi ::: (view all by) ::: April 12, 2015, 10:44 AM:

Bruce Schneier @527:
An additional round of voting/confirmation/whatever requires significantly more work on the part of the Hugo administrators every year. A more elaborate voting system requires more work on the part of a programmer once, and then exactly the same work on the part of the Hugo administrators who are simply running the voting program.

Mostly. There is additional labor, year on year, educating voters on how a more elaborate system works and addressing inevitable misconceptions, misinterpretations, and mistrust. As Kevin Standlee can testify, and as several comments on this thread demonstrate, that is already the case with the current system's treatment of No Award.

If this is necessary complexity, that's fine—but the cost of interfacing with the humans will be an ongoing one.

#529 ::: Aaron ::: (view all by) ::: April 12, 2015, 10:47 AM:

For instance: "3.7.4: Members should not vote for nominations by copying any slate of nominees suggested by others, but instead should make their own individual choices for what they believe are the best works."

Now you have to define what a slate is in the rule. This raises the question of does WorldCon want to go down the road of defining what is and is not a slate. Even if WorldCon does define what a slate is, then the people who have used the letter of the rules this time will just do so again, proposing something that is *just short* of being a slate under the rules of what a slate is and is not.

#530 ::: Andrew M ::: (view all by) ::: April 12, 2015, 10:59 AM:

I agree with Steven desJardins; but Keith's proposal is only an extreme form of something that has been coming up throughout this thread, where there have been various suggestions to massively increase the number of nominees. This worries and puzzles me. I have always taken it as axiomatic that one should either read all the works in the categories for which one is voting, or decide not to, for reasons, where 'there are too many' is not a sufficient reason. I think this is vital if the process is to be an actual consideration and comparison of works, not just a measure of popularity; the idea is to come up with a work which the voters, having considered the possibilities, have judged the best, not just a work which has a lot of supporters.

Bear in mind that this system will have to apply in every category. With novels, it's likely that one can quite easily sort them into those one might vote for and those one would never have any interest in, because they come from a subgenre one does not like; with short fiction, fan writers, artists, etc., many of whom one might never have heard of before the Hugo process began, this won't be so; without looking at them all one won't have any idea what is worthy.

I also think this may lead to the thing that wins having a quite small number of votes, proportional to the number of voters. Even if the winner is required to have a majority on the final count, that's not a majority of the voters; it's a majority of the votes that are left after others have been discarded. (I'm thinking of this in terms of IRV, but as far as I can see something similar would have to be true in other systems.) With five nominees the difference is not that great; with twenty, and even more with 587 (number of novels nominated this year), it would be.

#531 ::: J Thomas ::: (view all by) ::: April 12, 2015, 11:33 AM:

There is additional labor, year on year, educating voters on how a more elaborate system works and addressing inevitable misconceptions, misinterpretations, and mistrust. As Kevin Standlee can testify, and as several comments on this thread demonstrate, that is already the case with the current system's treatment of No Award.

If this is necessary complexity, that's fine—but the cost of interfacing with the humans will be an ongoing one.

This is one of the reasons I like the Open Nomination, Visible Voting approach.

Nominate whatever you want. Vote for whatever you want. When you get new information, including information about which nominees are doing well, you can change your vote until the deadline.

Simple. You get to do what you want. If you notice a nominee that is doing well and you want it to do better, vote for it.

If you think a bad work is too close to winning, vote for the most popular competitor that you approve of and/or for No Award.

All dead simple. If there's anything complicated going on, it's happening inside voters' heads and the voting system is not responsible for that.

If somebody says it isn't fair, ask them why. They got to nominate whatever they wanted. They got to vote for whatever they wanted. If it didn't win, that was because voters did not choose to vote for it.

What's fairer than that?

#532 ::: Erwin ::: (view all by) ::: April 12, 2015, 12:03 PM:

The problem with fan agreement is that the dominant strategy ends up being a 2 party system. If you have 2 similar slates and one different slate, the similar slates are likely to halve their voting power.

If we define a figure of merit for a voting system to include diverse genre selection and high quality, slate voting is likely to give inferior results on average to something like RAV.

The RAV system has an advantage in that even large contingents with similar tastes will select approximately one candidate.

So, it might be reasonable to argue that RAV is also likely to reduce vote splitting in the final round. (eg, 4 better sci-fi works and one fantasy...fantasy wins.)

Still, it would make sense to actually test on prior years. Is that possible? The effect of different voting systems is likely to be sensitive to the actual distribution of nominations. That'll give a guess of the effects for nonstrategic voters.

#533 ::: P J Evans ::: (view all by) ::: April 12, 2015, 12:53 PM:

526
Assume intelligence.
Don't assume educational background like your seminar attendees and students.

(Try explaining it as you would to, for example, high school students.)

#534 ::: TomB ::: (view all by) ::: April 12, 2015, 01:00 PM:

The problem with fan agreement is that the dominant strategy ends up being a 2 party system. If you have 2 similar slates and one different slate, the similar slates are likely to halve their voting power.

If it ends up being a 2 party system, it will have 2 dissimilar slates, where works of great merit will be left off of one slate solely because it is on the other slate. In other words, it will strongly resemble the current American political landscape. This does not seem like a good thing.

Or maybe it could end up like the current UK political landscape, with 3 mainstream slates that are mostly the same and a far-right radical fringe slate. I'm not sure this is a good thing either. Vote for the TORies?

#535 ::: J Thomas ::: (view all by) ::: April 12, 2015, 01:07 PM:

@532 Erwin

The problem with fan agreement is that the dominant strategy ends up being a 2 party system. If you have 2 similar slates and one different slate, the similar slates are likely to halve their voting power.

I'm starting to repeat myself. I tend to do that when I feel like my ideas get mostly ignored, since of course I don't recognize for myself that they deserve to be mostly ignored. I'm trying to watch myself and not do that too much, but of course it's a strain.

If fans mostly disagree, then it's only natural that some alien force that agrees would move in and take over. One way to avoid that is to dilute or entirely throw out votes if they agree too much.

Another way is to get a cabal of fans who agree to vote together. But having our own unrepresentative cabal that works together to win isn't that much better than being ruled by aliens.

Imagine it was people voting in person like in a Heinlein democracy. Like in Methuselah's Children. You get some big rooms, and you assign a bunch of stations. "Somebody who wants to make a nomination, go to one of these stations. Anybody who agrees with them, join them there. Anybody who isn't sure, wander around and listen to people and choose the one you like best."

If some people want to stay with nominations that don't have many people? That's fine. Maybe more will come, or maybe not. If some people want to choose among the top 5 or the top 2, that's fine too. At the end of the day, the ones that have the most people are the ones that are most popular.

A cabal can all stay at one station from the begining and try to make their nomination look popular. Maybe some people who want to be with the winners will join them because there are a lot of them. Or maybe they will be the sort of people who don't attract others.

You don't have to start a cabal to oppose the first cabal. If people want to make sure the cabal is not the most popular, they can crowd around some other standard, and they can dynamically choose which one. Or if they don't feel the need to outcompete anybody, they can spread out among the minority views they think are best. Whatever they want.

If we could reproduce a system like that, I think we'd be just fine. We don't need instant runoffs so people get a second chance when they don't know whether their first choice will lose or not, because people can stroll over to their second chance whenever they want.

The more I talk about it, the more I like it. The more it gets ignored the more I want to talk about it. This is how monomaniacal cranks get started, and I don't want to become that. I ought to walk away for awhile.

#536 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 01:09 PM:

@485:

@ 481: "3. Results are not visible until you've submitted a vote.
If the current results are made visible people who've nominated in the unofficial poll, they'll be leaked publicly, but I don't think there's any need to keep the current state of nominations secret. In fact, I think the unofficial nominations need to be made public while still open, to stop the Puppies spamming them too (different counting methods help, but concentrating the non-slate vote is also necessary).

I didn't mean that poll results should be kept a Big Secret from people who haven't submitted their polling. It's just that, ideally, we would want people to do things in a certain order:

1. Vote in the unofficial poll. Ideally this should be based principally on their own judgement and taste, though the system should be robust to organized voting blocs.

2. See the latest unofficial poll results. Comes after step 1, because if it came immediately before, it would be hard not to be influenced by the list you've just seen, and that would give early voters too much extra influence. Note that the unofficial poll results should not be visible at all until there are a reasonable number of votes; probably, 200 would be about the right number here, to give a margin of random sampling error of under 2% for the results.

3. Cast an official ballot, which includes your favorite 3 of the 5 unofficial frontrunners, plus your favorite 2 which are not in that group.

So showing the results only after a vote is a way of nudging people to do the steps in that order. If they really want to, they can get leaked results and change the order or skip steps entirely. That's on them.

#537 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 01:13 PM:

@533: OK. What is it that I should explain as if to a high school student? The RAV process? The rationale? Something else?

#538 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 01:19 PM:

(I think part of the confusion comes from 340-508, when we got distracted and started talking about the single-winner process in the second round. I started slinging around single-winner terminology because I can't resist talking about that stuff, but I didn't define it because on some level I knew we were getting off topic. I apologize, that wasn't very helpful.)

#539 ::: Keith "Kilo" Watt ::: (view all by) ::: April 12, 2015, 01:19 PM:

J.Thomas@531

This is one of the reasons I like the Open Nomination, Visible Voting approach.

Nominate whatever you want. Vote for whatever you want. When you get new information, including information about which nominees are doing well, you can change your vote until the deadline.

Huh. The more I think about this, the more interesting it seems. It could make "Hugo voting season" actually be something exciting to watch. That can't be bad for the field as whole. It also encourages readers who are watching the contest to try new things as something starts to rise in the rankings. And yet, they can still respond to it however they wish: "Wow, I've never heard of this, but it's really good (or literary, or feminist, or military, or all of the above, depending on your interest -- and since you are also part of fandom, your interests represent a valid part of fandom's interests)." You can also respond, "Wow, that's really garbage. I'm not going to vote for that, and in fact, I'll explicitly list it below no award."

Radically different from what we've had to this point, and absolutely impossible prior to the internet age. But we're SF, being forward-looking doesn't really bother me.

Very, very interesting idea...

Kilo

#540 ::: Tom Whitmore ::: (view all by) ::: April 12, 2015, 01:22 PM:

Bruce Schneier @527: It only requires more work on the part of the programmer once if the program is:

a) shared
b) used by the next convention
c) not broken/modified by later changes either to rules or hardware/software (sufficiently robust).

None of these are actual givens in the real world of Worldcon running. One of the major hurdles to overcome with anything related to changing voting is not just the rule-changing cycle for unincorporated WSFS: it's the fact that while there is a great deal of overlap in the people who run the Worldcon year to year, they frequently have different jobs and they (almost) always have different people at the top. This is a bigger problem than most people writing here seem to realize -- and it's not part of the "let's design an optimum voting system" that you want to address, but is is a part of "let's design an optimum Hugo voting system."

#541 ::: nathanbp ::: (view all by) ::: April 12, 2015, 01:39 PM:

For all that everyone is saying that slates are bad, I'm surprised the open nomination process keeps coming up when it basically amounts to an official slate. Even if you list the top 15 candidates and don't list them in order of votes it's still basically saying "Vote for one of these candidates or your vote won't count". It also requires a lot more attention from the voters and potentially a lot more reading and research on their part to figure out to change their vote to. In fact, it really shifts a lot of power to those people with enough free time to spend gaming the system, which is unfair to the rest of us that just want to vote and be some with it. Plus it still is unlikely to solve the problem of slates on its own (see comments on fans not agreeing on anything) so you'd still need to change the underlying voting system.

#542 ::: nathanbp ::: (view all by) ::: April 12, 2015, 01:42 PM:

Sorry for typos in previous comment, typing on a tablet. Should be "figure out what to change" and "be done with it".

#543 ::: Bruce Schneier ::: (view all by) ::: April 12, 2015, 01:47 PM:

Re open nominations, visible voting:

The reasons this doesn't work are 1) most voters don't vote until the last minute, and 2) a trivial voting bloc strategy is to hold votes until the end. "Just send them to me, and I will forward them to the Hugo Administrator at the last minute." It's kind of like eBay sniping.

#544 ::: P J Evans ::: (view all by) ::: April 12, 2015, 01:50 PM:

541
There's a big difference between a slate as a list of nominees in general, and a slate intended by a particular person (or group of people) to determine the nominees. (Same word, different usage and meaning.)

#545 ::: Brad from Sunnyvale ::: (view all by) ::: April 12, 2015, 02:58 PM:

I would say that the one thing worse than a self-selected survey is a self-selected survey where you can see the current totals and react to them before you vote. It gets biased even more!

But do any among the proposals here meet the goal set?

a) Simple to understand and nominate (and harder, simple enough to pass a WSFS business meeting)
b) Robust against collusion (slates)
c) Robust against individual strategizing
d) Good measurement of broad fan opinion and extremely likely to find what should be the winner
e) Produces a set of nominees small enough that most voters can judge them
f) Is fair
g) Appears to be fair


It's also important, I think, to have many instances where the top nominated work doesn't win. That's been rare recently in the novels, though it does still happen -- Graveyard book being the strongest recent example.

#546 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 03:26 PM:

@545

Here's how I'd rate using RAV officially by your criteria

a) Simple to understand and nominate (and harder, simple enough to pass a WSFS business meeting): I'd divide this in 3:

a1: simple to understand: Certainly not "simple". But, using things like my cookie grab metaphor, I hope it's not impossible either.
a2: simple to use: Yes. Basically, same as old system.
a3: can pass WSFS business meeting: I have no idea

b) Robust against collusion (slates): Yes.
c) Robust against individual strategizing: No voting system is 100%, but this one is about as good as any.
d) Good measurement of broad fan opinion and extremely likely to find what should be the winner: Yes.
e) Produces a set of nominees small enough that most voters can judge them: Yes.
f) Is fair: I certainly think so.
g) Appears to be fair: I think so. But some might disagree. For instance, on GRRM's blog, he seems to think that any combination of the words "weight" and "vote" makes something unfair.

How does that compare to other options?

SDV, or unranked STV, or other proportional system using same ballots: Same as RAV in all important respects.

unofficial poll to set slate, using proportional system: doesn't have to pass WSFS, but harder to use.

Approval with public totals: Harder to use, gamable by timing votes (depending on what you want, putting all your votes at the start or the end could be more powerful. Because of this gamability, I think people would over time not feel this system was entirely fair.)

Limited voting (such as: 3 suggestions per ballot, 7 nominations at end; or similar): Nice and simple. Reduces the slate problem to where factions would have to be about twice as big as to sweep the nominations, but doesn't solve the problem entirely. Works supported by slates would still have a significant advantage. I wouldn't trust it to resolve things, but it might.

#547 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 03:27 PM:

@545

Here's how I'd rate using RAV officially by your criteria

a) Simple to understand and nominate (and harder, simple enough to pass a WSFS business meeting): I'd divide this in 3:

a1: simple to understand: Certainly not "simple". But, using things like my cookie grab metaphor, I hope it's not impossible either.
a2: simple to use: Yes. Basically, same as old system.
a3: can pass WSFS business meeting: I have no idea

b) Robust against collusion (slates): Yes.
c) Robust against individual strategizing: No voting system is 100%, but this one is about as good as any.
d) Good measurement of broad fan opinion and extremely likely to find what should be the winner: Yes.
e) Produces a set of nominees small enough that most voters can judge them: Yes.
f) Is fair: I certainly think so.
g) Appears to be fair: I think so. But some might disagree. For instance, on GRRM's blog, he seems to think that any combination of the words "weight" and "vote" makes something unfair.

How does that compare to other options?

SDV, or unranked STV, or other proportional system using same ballots: Same as RAV in all important respects.

unofficial poll to set slate, using proportional system: doesn't have to pass WSFS, but harder to use.

Approval with public totals: Harder to use, gamable by timing votes (depending on what you want, putting all your votes at the start or the end could be more powerful. Because of this gamability, I think people would over time not feel this system was entirely fair.)

Limited voting (such as: 3 suggestions per ballot, 7 nominations at end; or similar): Nice and simple. Reduces the slate problem to where factions would have to be about twice as big as to sweep the nominations, but doesn't solve the problem entirely. Works supported by slates would still have a significant advantage. I wouldn't trust it to resolve things, but it might.

#548 ::: Keith "Kilo" Watt ::: (view all by) ::: April 12, 2015, 03:39 PM:

Brad@545:
I'm not sure how the bias worse. There's no inherent bias that isn't already in the system to begin with. Remember that under the current system, once we get a list nominations for the final ballot, those are your only choices, and too bad if something you wanted to vote for isn't on the list. Fundamentally, that's the problem with slate nominations. I'm assuming, by the way, that you mean that if a voter comes to vote late in the process, he or she will see that there are already a number of favorites and focus his or her votes on those favorites. If a voter feels that a particular work is worthy, he or she can still vote for it, even in that case. Those who vote earlier may have a greater opportunity to get their choices up where people can see them sooner, and that may be an advantage, but we can't justifiably say it's an unfair one.

Regarding your seven characteristics, no, I also haven't seen a system that meets all seven. Continuous voting fails (e), at least at the outset. But public opinion will eventually coalesce around a much smaller set of works -- and isn't the point of the Hugo to find out what the fans as a whole like? The more I consider it, the more it seems that system meets that fundamental goal better than any others.

Bruce@543:
Regarding point 1, if they wait until the last minute, that's fine. A consensus may already be growing around a number of works, but that's no different from what we have now: those who vote late -- and missed the nomination period -- have exactly the same situation now. Under open continuous voting, if there was a work that they strongly felt should be on the ballot, there was nothing stopping them from nominating early. I don't really think any system can or should have to correct for that.

Regarding point 2, your analogy to eBay sniping doesn't quite work. When sniping, the assumption is that you have an unlimited sum of money that you could potentially spend in order to swamp the competition at the last minute. Slate votes don't have that. If you only have 200 votes to spend, you should spend those votes early to get your work visibility. Sniping is actually counterproductive, except in the case when no work has many votes AND opinion isn't coalescing around certain works. This is where our high-minded goal of getting more participation in voting pays off. With enough numbers total AND if fandom is willing to work together in a crowd-sourcing way (through the voting process) to come to a consensus through the process, 200 votes won't be enough to change that.

I guess this just goes back to Bruce's point (f), is the system fair? If 200 votes -- at any point in the process -- is enough to engender a majority, is that fair? I'm not sure the answer to that question is as easy as people are making it out to be. Right now, it's -not- fair because those 200 votes (in the form of nominations) is enough to remove all -chance- of someone else's opinion even being considered.

I'd be interested to see the Puppy reps weigh in on this question: If an open, continuous voting system were implemented that allowed their preferred works to always have a chance of winning, would that satisfy their objections and remove the need to vote as a bloc in the first place? Reading their posts, their -claim- is that their works aren't even being considered, and that's why they've chosen to nuke the whole process. We all know that some of their commenters aren't interested in getting their type of works read, they simply want to break it for everyone else out of sheer joy of destruction, but it seems this type of proposal is a "put up or shut up" moment. What is it -really- that they want?

But I'm probably dreaming in that regard, I realize...

Kilo

#549 ::: Brad from Sunnyvale ::: (view all by) ::: April 12, 2015, 04:10 PM:

I presume we're talking nominating here, not voting. The problem with publicly visible voting (aside from what Bruce outlined with sniping etc.) is that it changes your vote from its "natural" state, or makes it more or less likely you will vote, which is what a bias does.

In this case, if the works you don't like are ahead, you are more likely to vote and worse, to rally the fans of another work to fight the opponent. Likewise, if you see your choices are already winning, you are less likely to vote or rally. (It's why in elections politicians always try to convince their base that the vote is close to make sure they turn out. Advance polls do affect the results they are measuring.)

I think you miss Bruce's point about sniping. eBay sniping has nothing to do with money! eBay is a sealed bid 2nd price auction that people get fooled by because they slip in the trappings of a going-going-gone public auction. Those who think it's the latter get confused when a sniper beats them it what was never a ggg auction in the first place.

Sniping is about not revealing your hand. If you are competing -- and the puppies are competing -- you don't reveal your strength until the last second.

----

As for RAV, as I pointed out it is more strategic than the basic approval we use now. It discourages you from nominating works you expect to be popular (even more than Approval does) to avoid devaluing your ballot for the works that really need your support to make the ballot.

This is one of the reasons people like ranking, though it adds complexity. With ranking, after I have ranked my own book, there is no harm to me in ranking my competitor's book after it, and so I will.

If you have a system where even a strongly interested party (such as an author) is still willing to list the works of other good authors, you have done well.

RAV is only partially robust against slates. The slate will probably get 2 entries on the ballot, and with some clever planning and a slightly larger size, possibly even 3.

#550 ::: P J Evans ::: (view all by) ::: April 12, 2015, 04:11 PM:

548
I think you're forgetting that we're discussing how to make the nomination system less gameable. We're not trying to change the system for the final ballots.

So: we have a nomination system which allows members to nominate whatever they think is within the requirements - it isn't, always; making that determination is the committee's job - and they shouldn't need to be strategizing or anything else at that point. That's why the existence of slates is a problem. They distort the nominations.

Try to keep the solutions to the problems we have now, and try to keep the solutions simple.

#551 ::: Cheradenine ::: (view all by) ::: April 12, 2015, 04:12 PM:

The last thing I want VD to have is detailed information about exactly how many wreckers he needs and how they need to vote to swoop down at the last moment to swing the nominations. Gaming the system always becomes more powerful the more information you have about what everyone else is doing.

Of course, if everyone else waits until the last minute, this is less of an issue, but then the continuous nature is largely irrelevant.

#552 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 04:49 PM:
Brad@549: As for RAV, as I pointed out it is more strategic than the basic approval we use now. It discourages you from nominating works you expect to be popular (even more than Approval does) to avoid devaluing your ballot for the works that really need your support to make the ballot.

This is not entirely wrong, but it's not that simple.

Say your favorite works in a category are A through E. If you know that C is likely to make the ballot anyway, you will probably strategically support just A and B. This is strategy, but it's not dishonest. If everybody thought like this, then C would have a harder time getting nominated — but it still has a chance, if enough other people think it's really the best. So the overall result is that nominees will be a bit "edgier"; things with deep, rather than broad, support. You could argue that that's actually a good thing for an award.

What if it's A, not C, that is the "shoo-in"? Well, then it depends what your goals are. If you want the slate of nominees to be as close to being exactly what you would have picked, then you might vote BCDE, using dishonest strategy. But that's actually a dangerous thing to do. If everybody thinks like that, A will not get nominated, precisely because everyone thought it was an obvious nominee. In other words, this may work as an individual strategy — though it's risky and hard to plan in practice — but it doesn't work as a collective strategy. I think you don't really have to worry about this kind of dishonest strategy being a big deal.

Also, remember that the Gibbard-Satterthwaite theorem shows that no voting system is completely strategy-free in all circumstances. In practice, the choices for a multi-winner system are usually between having a small amount of this kind of strategy, or to have a large amount of slate-based voting and frequent category-sweeps by a minority, as the current system is likely to devolve into in the post-puppy era unless something is done. I'd prefer the former.

#553 ::: P J Evans ::: (view all by) ::: April 12, 2015, 04:56 PM:

552
I believe you're overthinking this. Most people don't get into that level of analysis, even on the final ballot.

#554 ::: felice ::: (view all by) ::: April 12, 2015, 05:44 PM:

Cheradenine @ 551: "The last thing I want VD to have is detailed information about exactly how many wreckers he needs and how they need to vote to swoop down at the last moment to swing the nominations."

Don't publish numbers, just the current (or last week's) top 15 in each category, unranked. That doesn't give the Puppies any useful information. They already know which works they want to win, and knowing who their competition is doesn't change anything.


Brad from Sunnyvale @ 549: "I presume we're talking nominating here, not voting."

Yep, so I've changed "vote" to "nominate" in the rest of the quote - I hope that's correct!

"if the works you don't like are ahead, you are more likely to nominate and worse, to rally the fans of another work to fight the opponent. Likewise, if you see your choices are already winning, you are less likely to nominate or rally."

Incentivising more people to participate in the nomination process is a good thing; I doubt it will incentivise the Puppies any further, since they already know that without a slate they'd hate most of the final ballot. And if just the unranked top 15 are published, not actual numbers, then you don't know if your favourites are winning or just close to winning, so there's no disincentive.

"Sniping is about not revealing your hand. If you are competing -- and the puppies are competing -- you don't reveal your strength until the last second.

But sniping doesn't help you if your hand isn't strong enough, and some form of open nominations would strengthen the non-Puppy hand.

"This is one of the reasons people like ranking, though it adds complexity."

Remember nominations are already ranked! It's impossible to submit a non-ranked write-in list; we just currently ignore the ranking in the counting.

"RAV is only partially robust against slates. The slate will probably get 2 entries on the ballot, and with some clever planning and a slightly larger size, possibly even 3."

Open nominations to concentrate support into a smaller range of non-slate works in combination with RAV would reduce the slate to 1 entry at most.

#555 ::: andyl ::: (view all by) ::: April 12, 2015, 05:51 PM:

@553

I know I have not bothered to put something on my nominations because I felt others would. I can then use my spare nomination on a Hugo-worthy book which I think may have a harder task to make it to the final ballot - maybe due to release date or the author being less well known in the US. It is what I did this year, But maybe I am just atypical - people have told me that before.

#556 ::: J Thomas ::: (view all by) ::: April 12, 2015, 06:14 PM:

@548 Keith "Kilo" Watt

When sniping, the assumption is that you have an unlimited sum of money that you could potentially spend in order to swamp the competition at the last minute. Slate votes don't have that. If you only have 200 votes to spend, you should spend those votes early to get your work visibility. Sniping is actually counterproductive, except in the case when no work has many votes AND opinion isn't coalescing around certain works. This is where our high-minded goal of getting more participation in voting pays off. With enough numbers total AND if fandom is willing to work together in a crowd-sourcing way (through the voting process) to come to a consensus through the process, 200 votes won't be enough to change that.

It could be a problem as follows:

Suppose that the votes are scattered. Everybody but the cabal has voted at least once, and no nomination has more than 50 votes. The cabal has not voted so nobody is worried about them. Everybody is satisfied with the status quo -- the winning nomination has 50 votes, the next-best has 49, another has 46, and nobody who hasn't voted for one of them feels the need to vote for them.

Then in the last 5 minutes when it's too late for anybody to respond, the cabal gets 200 votes for their nomination(s). They win and nobody can stop them, because people who would otherwise stop them didn't see the need to stop them.

When your *only* reason to vote for popular nominees is to stop the cabal, the cabal can be sneaky and slip past you.

But if people tend to choose among the winners whether or not there's a cabal, then it doesn't matter what the cabal does.

If a bunch of people vote for the 2nd-highest with 49 because they want it to win, and a bunch vote for the one with 50 because they want that to win, and maybe a bunch vote for one that has 35 because they think it's the best and it's only 15 votes behind with more than a thousand people who might vote for it, very likely they will beat the cabal.

I think if your only reason to vote for a winner is to stop the cabal, and the cabal tricks you, then they deserve to win.

They shouldn't get to keep other nominations off the list, so you can't vote for the ones you want. But if you can't agree on anybody and they can, why do you get to pick a winner that gets fewer votes?

Meanwhile, you have a much better chance to agree on a winner if you know how other people are voting. Maybe there's something immoral about people influencing each other to pick a winner, but I don't see it. You start out with diversity and you have the chance to create some sort of consensus. What's wrong with that?

#557 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 06:40 PM:

@553: I'm responding to Brad's allegation that strategy would be a problem. It's not a problem if people don't overthink things, and it's not a problem if they do. I was showing the latter.

#558 ::: Keith "Kilo" Watt ::: (view all by) ::: April 12, 2015, 06:42 PM:

J Thomas@556:

You start out with diversity and you have the chance to create some sort of consensus. What's wrong with that?

Whatever system ends up getting decided upon, I do think this needs to be a guiding principle. If the Hugos are supposed to be a fan favorite, this, fundamentally, is what needs to happen. How we do that is obviously subject to much debate...

Kilo

#559 ::: P J Evans ::: (view all by) ::: April 12, 2015, 06:53 PM:

We need a system that's easy for voters to use. If you need a degree in political science or statistics to understand it, it's too damned complicated.
If you have to do analysis of the voting that's already taken place (assuming that that information is available - and I wouldn't be releasing it myself) to figure out how to vote, it's too damned complicated.
If you're worried about how someone else will vote at the last minute - maybe you shouldn't be using electronic voting. And certainly you shouldn't be telling people what the numbers are.

The juvenile canines aren't in this to win as much as they're in it to wreck the Hugos as an award valued by fans. They're atypical voters. maybe they can be defended against - but not before they do it again, given that there aren't a lot of things that can be done before 2017. If the administrators have the authority to invalidate nominating ballots based on suspicious behavior, then something might be possible - but that's up to the committee, and it likely won't do more than slow them down a little.

Remember, I've been there and done that already. Smarter voters can do it so it can slide by because it won't look suspicious - or not suspicious enough. It's the obvious ones that get caught.

#560 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 07:06 PM:

@559: You don't need to overanalyze to use RAV. Brad said you did. I took his analysis one step further and showed you basically don't. I agree most people wouldn't go that far.

I could also have just said "don't overthink it".

Either way, the lesson is the same: in RAV, you can just vote honestly.

RAV is not flawless. Brad's argument is not totally invalid. More seriously, explaining the system can be hard, and it's smart to be suspicious of what you don't understand. But "you need an advanced degree to vote effectively" is not one of the flaws.

#561 ::: felice ::: (view all by) ::: April 12, 2015, 07:09 PM:

P J Evans @ 559: "We need a system that's easy for nominators to use. If you need a degree in political science or statistics to understand it, it's too damned complicated."

Most of the suggestions don't require any change on the part of the nominators, just on the admin side.

"there aren't a lot of things that can be done before 2017."

There's nothing in the constitution requiring all details of nominations to be kept secret till after the nomination deadline, so that can be changed next year (MidAmeriCon II willing), and should be sufficient to prevent another Puppy takeover. Improvements to the way nominations are counted are still worth pursuing, though.

#562 ::: Jameson Quinn ::: (view all by) ::: April 12, 2015, 07:17 PM:

Also:

there aren't a lot of things that can be done before 2017.

The suggestion of having an unofficial poll to agree on a consensus slate (which would include one or two puppy nominees if they participate), then encouraging everyone to support that consensus slate, could be done for 2016, because it doesn't take an official rules change. There have been several variations of that suggestion made. I don't know enough about this community to know how consensus around that would be reached or expressed, but I support the idea. Would Bruce explicitly taking sides here be helpful to forming consensus? I'm not sure; possibly. Would Teresa doing so help? I doubt that's her style, and even if it is, I suspect there would be pushback from some quarters.

#563 ::: Keith "Kilo" Watt ::: (view all by) ::: April 12, 2015, 08:42 PM:

Just thinking out loud here, but couldn't you run unofficial tests of lots of nominating systems next year and create a slate from each one? We could not only see how the various systems play out, we could also have (possibly several) slates to put forward that might be more representative of fandom than any slate created by an individual. If competing slates are inevitable for next year, and I think they are, this at least puts them to good use. Slate voting with a purpose!

Kilo

#564 ::: J Thomas ::: (view all by) ::: April 12, 2015, 08:46 PM:

I want to review the bidding.

The Hugos traditionally gave people a restricted vote -- five pre-selected nominees they could choose among. This is simple for final voters and they have a chance to read all the choices.

But systems to choose the 5 nominees have problems. If you just choose the 5 that nominators like most, the nomination can be pirated by slates.

We can develop rules for nomination that prevent slates from taking over. People might reasonably ask why the rules are arranged so that the nominees with the most votes are not selected. "Because that's how we want it. We don't want the nominees with the most votes to be on the final list because we don't like your nominees."

Any system to choose 5 nominees for the final ballot can possibly be gamed so that some cabal can get the wrong 5 nominees to be selected.

It might make sense to allow open nominations for the final vote. Then the original problem moves to the voting.

We can no longer expect every voter to study all the works. And just as a diversity of nominations let cabals take over, a diversity of voting that let cabals take over the final election could give us a final cabal win.

Either for voting or for a preliminary nominating step, if a cabal agrees with each other and nobody else agrees about anything, why shouldn't they win? They have the votes. No other choice has the votes. We're going to stop them because we don't like them, after they win fair and square?

We need to start out looking at all the diversity we bring to the table, and then grope toward a consensus. The right way to win is to get more votes for one candidate than any unpopular cabal can get for theirs.

One way to do that is to have a series of runoffs, with fewer candidates each time. That could happen either in nomination or in the actual election. It is extra work for the humans who run the system, and extra work for voters.

We can approximate that with IRV and similar approaches so that people can blindly suggest alternatives in case their first choice fails. These approaches are somewhat hard to explain, and blind. You don't know who will actually be in the run-off when you make your second choice. But the extra work is all programming work and not human work (except for the voters), apart from explaining it.

To my own way of thinking, the only problem with a series of actual runoffs is that it's work for the organizers and work for the voters. You start with a diversity of nominees and you gradually narrow it down toward a winner. Apart from the work required, it does what we want. You can stop when it's down to 5 nominees and run a special election with those, or you can just keep having runoffs.

Better to get some way to automate the runoff procedure, so it isn't extra work for the organizers.

Visible Voting gives you automated continuous runoffs. Each voter decides to ignore nominees that he thinks can't win, when he himself decides they can't win. It's simple to do (apart from possible bandwidth problems, DDOS attacks, etc) and simple to explain.

But it doesn't guarantee that slates can't win. It only gives each voter a chance to vote for the best candidates he can find that have a chance to win. That *ought to* beat slates, but it might not.

You could approximate that with partial data. If for example people see which are the top 5 and the top 15, and the cumulative votes for each, they can get an idea whether their choices that aren't in the top 15 have a chance to get into the top 5. Assuming they can keep voting, then at some point they decide whether to choose among the top ones.

You could program the system to release that data weekly or whenever you wanted. You might feel it's better not to let voters know too much. Just knowing which nominees are in the running may be enough. That might be enough for them to reach a consensus or else vote No Award.

I like continuous voting because each voter gets to decide for himself how often he wants to pay attention and review his choices. If you have a series of automated runoffs, for each one he either votes or fails to vote. Some influential runoffs may have only a few voters. When it's continuous, times that there are only a few votes will be important proportional to the changes they actually make. But a series of runoffs ought to work OK. It's basicly the same concept.

#565 ::: Emily H. ::: (view all by) ::: April 12, 2015, 08:50 PM:

Any slate, however unofficial or consensus-driven, feels to me like a "we had to destroy the village in order to save it" solution.

I would much rather just wait until a sensible rules change, perhaps towards something like RAV, could be implemented, even if it takes a couple years. And if such a slate is put forward, I certainly wouldn't use it for making nominations or votes.

#566 ::: Erik Nelson ::: (view all by) ::: April 12, 2015, 09:19 PM:

I appreciate this article. It answers the question "what is everyone talking about?" in a way I wish I had had when I was beginning to read these threads:
http://www.thedailybeast.com/articles/2015/04/12/right-wing-trolls-hijack-scifi-oscars.html

#567 ::: Brad from Sunnyvale ::: (view all by) ::: April 12, 2015, 09:24 PM:

Actually JT#564, there are several systems which can handle collusion. In fact, most of our political systems fully expect it. But they are different systems, and as a consequence they tend to be more prone to strategic voting.

What the danger is of strategic voting is hard to say. Many will not care and not vote in any strategic way. Many others will read articles about strategies, or figure out their own, and create ballots that are not their true and independent desire, which is what we want. We can't learn what fans want if they are going to lie about it on their forms.

It leads to a lot of second guessing -- did I vote right? Did I hurt my favourite by listing my 2nd favourite?

Reveal the results while it's going and now it's nothing but strategy. While not in any way supporting the public approach, I will find out that it does have a defence against the dark arts, namely the vote does not close until there have been fewer than N votes in the last day. Any sudden flood of votes keeps it open and gives people who care a chance to see it. But it's still a giant, highly biased political game, rather than an attempt to figure out what the fans really loved this year!

#568 ::: Fade Manley ::: (view all by) ::: April 12, 2015, 10:03 PM:

I am very much of Emily H.'s opinion on this. A well-meant slate derived from the Right Sort Of People voting is still a slate, and I'm not interested in making nominations at all if I have to lie to do it.

#569 ::: TomB ::: (view all by) ::: April 12, 2015, 10:04 PM:

To prevent sniping, extend the deadline until the moving average of the nomination rate goes below a threshold.

In practice, the deadline will always be extended at least one day because of normal last-minute nominating. If there is a controversy, the various parties will try to get more people to nominate. The nominations will remain open until all parties are exhausted and the nomination rate goes down.

To prevent endless procrastination, charge a late fee to nominate after the initial deadline.

#570 ::: P J Evans ::: (view all by) ::: April 12, 2015, 10:22 PM:

569
That's much more common with mail-in, where you have to deal with odd pickup times. (If the deadline is Sunday midnight, there are areas that don't have Sunday pickups. Although I don't think most people will go as far as the one who postage-metered it at a time before the deadline - and then apparently dropped it in the regular mail, so it was postmarked a couple of days later and thirty or so miles (and an entire county) away. Don't think so, dude.)

#571 ::: P J Evans ::: (view all by) ::: April 12, 2015, 10:25 PM:

569
Also: it's a bathtub-shaped curve: the early-bird peak, then a long period with a fairly-steady trickle, then a much larger peak as you come to the deadline. The only actual number you have to go by is the number of memberships.

#572 ::: felice ::: (view all by) ::: April 12, 2015, 11:02 PM:

Fade Manley @ 568: "I am very much of Emily H.'s opinion on this. A well-meant slate derived from the Right Sort Of People voting is still a slate, and I'm not interested in making nominations at all if I have to lie to do it."

The Puppies would be welcome to nominate for the counter-slate too; it's not restricted to the "Right Sort", it's just counted in a way that prevents a bloc getting disproportionate representation. And nobody should be lying.

TomB @ 569: "To prevent sniping, extend the deadline"

Sniping isn't a problem if we can concentrate nominations towards the works that actually have a chance; a moving deadline is an unnecessary complication.

#573 ::: John A Arkansawyer ::: (view all by) ::: April 12, 2015, 11:05 PM:

Brad from Sunnyvale @ 549: "RAV is only partially robust against slates. The slate will probably get 2 entries on the ballot, and with some clever planning and a slightly larger size, possibly even 3."

First thought: This is probably good enough and probably as good as you're going to get.

Second thought: How does it do against two competing slates? More than two?

Third thought: I don't have a third thought.

#574 ::: J Thomas ::: (view all by) ::: April 12, 2015, 11:45 PM:

@567 Brad from Sunnyvale

What the danger is of strategic voting is hard to say. Many will not care and not vote in any strategic way. Many others will read articles about strategies, or figure out their own, and create ballots that are not their true and independent desire, which is what we want. We can't learn what fans want if they are going to lie about it on their forms.

I have the strong impression that the winners fans want are very very diverse. This is why an organized cabal can make so much difference. If 51% of the fans all wanted the same thing, 20% of interlopers could get no traction.

It's because we can't agree on anything that the SPs have an in. They *do* agree. When they are the biggest plurality they are important out of proportion to their numbers, because they agree and we don't.

Instead of finding a way that we can hash things out and come to agreement ourselves, we should instead forbid anybody to agree?

So you want to rig the voting. "We mostly don't nominate the works that got the most votes. When they get the most votes it's because too many voters agree on them. Instead we nominate other works that didn't get as many votes, because that way we increase diversity."

Good luck explaining that to people who haven't studied voting systems.

#575 ::: AnnieY ::: (view all by) ::: April 13, 2015, 01:47 AM:

#559 If you have to do analysis of the voting that's already taken place (assuming that that information is available - and I wouldn't be releasing it myself) to figure out how to vote

Why would you need to analyze anything or care about it? You nominate what you liked, right? You don't decide - oh, I loved this one but I am sure a lot of people will nominate it, so I will skip it from my ballot and nominate these other 5 instead - because this will sooner or later backfire and that best work won't make it into the final ballot because too many people decided to play strategic (thankfully there are a lot of people that actually nominate what they liked without trying to decide how to vote strategically)...

#576 ::: Bill Stewart ::: (view all by) ::: April 13, 2015, 03:47 AM:

DD-B@519 - Actually there is something magic about being below No Award, because the Hugo version of IRV has an extra "showdown against No Award" phase at the end, so works on your ballot below NA or left off (if you voted for NA) both get treated roughly.
(And hey, that PO Box is run by the Secret Masters of Fandom, or at least that's what the newsletter tells me.)

For the actual voting phase, I do vote at/near the last minute, at least for novels and sometimes novellas, which take time to read. Short stories, graphic stories, and artwork I tend to hit early on.

But for nominations, this is probably the first year I've nominated anything (short stories by people I know who make them available on the web); I may have nominated a movie or two in the past, and it depends a lot on whether I went to last year's Worldcon or got my supporting membership early enough to nominate.

#577 ::: Vivien ::: (view all by) ::: April 13, 2015, 03:48 AM:

I made quick calculation to evaluate how efficient RAV would have been to prevent slates from being too efficient in 2013. I am using exponential downweighting.

There is a small limitation, I can't model the downweghting for non-slate voters, as I don't have the ballots. I assume non-slaters don't get downweighted, and that exagerates the efficiency of RAV.

Anyway, RAV would have worked pretty well for best novels.

Ballot was :
Redshirts 193
Blackout 138
2312 135
Captain Vorpatril's alliance 133
Throne of the crescent moon 118

Total ballots: 1113

For slaters, getting one nominee would have been as easy as ever : 119, to beat "Throne of the crescent moon". Needs in vote rise then steadily : 267 to get a second nominee, 540 to get a third. Based on first evaluation, Puppies would have got 1 or 2 nominations. Getting a third nomination would have mean representing about 1/3 of the voters...

But... if you are strong enough and want more then 2 nominates, another stategy then slating gets more profitable. You should divide your fan base : if you push three works A, B and C, have one third vote A, on third B, one third C. Now it is also more risky : if you are not as strong as you thought you were, you can get totally wiped out.

For best novel, requirements become 406 votes to get three nominees, 552 to get four. Total wipe out is at 965. Still pretty solid.

***

Short fiction categories are much, much more fragile. In those, slaters wouldn't have to divide their forces. In novelette, slaters could get three nominees with 216 votes, and four with 496.
Short stories is even worse : three nominees require 136 votes, four 304.

Sad puppies would have likely got 3 novelettes and up to 4 short stories on the ballot, even without using more complex strategies then slating.

**

Now, at some point, it can't be all the slaters fault. It is my personal opinion that at least in short fiction categories, Hugo Award could use slightly more organized voting, not less.

In 2013, only three short stories reached the 5% threshold that allowed them to compete on the final ballot. The obivious explanation is that the average Hugo voter had no clue about what has been going in the (vast) short fiction field (I don't, either). Hence the dispersion of the votes, and, again IMHO, the same authors and short fiction venues getting nominated again and again. Slaters or not.

So, yes, in short fiction, a little more debate around awardable work, and more "recommended reading lists" would certainly help.

#578 ::: Soon Lee ::: (view all by) ::: April 13, 2015, 04:06 AM:

Vivien #577:

In 2013, only three short stories reached the 5% threshold that allowed them to compete on the final ballot. The obvious explanation is that the average Hugo voter had no clue about what has been going in the (vast) short fiction field (I don't, either).

I think it was more that it was a diverse category with (too) many popular stories; nominations spread over more works makes it harder to get five finalists over that 5% threshold. It's something I suspect will be more of an issue as the nominating population grows & likewise gets more diverse. Should the 5% also be considered alongside any proposed rule change?

#579 ::: felice ::: (view all by) ::: April 13, 2015, 04:59 AM:

Soon Lee @ 578: "Should the 5% also be considered alongside any proposed rule change?"

No, open nominations will solve that problem too. (If for some reason we don't get any kind of open nominations / longlist, then yes, it would make sense to do something about the 5% threshold)

#580 ::: Vivien ::: (view all by) ::: April 13, 2015, 05:25 AM:

Precision on my post in 577 : I base my estimation of the number of block voters on this estimation on Chaos Horizon :

https://chaoshorizon.wordpress.com/2015/04/06/how-many-puppy-votes-breaking-down-the-hugo-math/

150-350, grossly averaged at 250.

@Soon Lee (578)

Well, I think that this point is debatable. I would say both : yes, short fiction is getting more diverse then ever, increasing vote splitting.

But at the same time, I would say that Hugo voter average "short fiction litteracy" has decreased. I think I see several evidence of that: a decrease in the diversity of works nominated; increasing discrepancy between the Hugo list and what Dozois, Horton or Tilton would pick up; the reccuring presence of certain venues, especially free online ones.

Precision : I never considered myself "short fiction litterate" enough to consider nominating for the Hugos. It is more that it brings an important precision on the question of "increasing the voter base" : informed, dedicated readers are a finite ressource.

Anyway, whatever our opinion on the subject, the vulnerability of short fiction categories to block voting remains the same.

#581 ::: felice ::: (view all by) ::: April 13, 2015, 05:28 AM:

nathanbp @ 541: "For all that everyone is saying that slates are bad, I'm surprised the open nomination process keeps coming up when it basically amounts to an official slate. Even if you list the top 15 candidates and don't list them in order of nominations it's still basically saying "Nominate one of these candidates or your nomination won't count""

That's an improvement over the current system of "Nominate one of the more popular candidates or your nomination won't count, but we won't tell you what the popular candidates are". The list of 15 is a longlist, not a slate; it's generated democratically from the early nominations of all the participants.

#582 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 05:31 AM:

Thank you, Vivien@577.

So the problem is that under RAV in 2013, just 310 slate voters could have picked up 4 nominations in "short story". There are two related causes, and three ways that voters could act to reduce the influence of slates in such categories.

One cause is that there are typically just fewer voters in "short story" than in "novel". This could be because, as Vivien claims, voters "had no clue about what has been going in the (vast) short fiction field". The second cause is that there's a "longer tail" in short stories, and less consensus on what works stand out; in other words, as Soon Lee argues, that "it was a diverse category with (too) many popular stories".

In order to decrease the potential influence of slate voters, there are several different things that could be done. The most obvious is: vote. If the short story category had as many votes as novel, with basically the same voting patterns as what we saw, a slate would have taken at least 500 votes to get four nominees. The second thing voters could do, with RAV but no other rules change, would be to put more than 5 nominees on their ballot. That way, there would be more overlap between non-slate ballots. I estimate that if voters had put twice as many works on their ballot, it would again have taken over 500 votes for a slate to capture 4 of the 5 slots. The third thing is for non-slate voters to act a little bit more like slate voters — that is, without actually blindly voting a slate, to share reading lists more, and make sure not to fail to consider stories others may be nominating. It's hard to judge what effect this would have had. And the final thing is to change the rules yet further, and allow more nominees in the category. If there were up to 7 nominees progressing to the final round, rather than 5, it would be virtually impossible for a slate to capture all but 1 slot under RAV. This is true even if the 5% rule were still in place, so that there would be fewer than 7 nominees in a typical non-slate year.

...

@568: When people talk about using an unofficial poll to determine a consensus slate to counter the influence of minority slates, nobody is talking about restricting voting in this unofficial poll to "the right sort of people". Everyone would be welcome; the voting system, not exclusionary tactics, would protect against undue slate influence in the poll. Furthermore, devoting 3/5 of your ballot's slots in each category to the "consensus slate" is not "lying", even if you haven't read those works. It's just saying, "I think these works deserve to be nominated", which is something you can honestly believe based on your trust in the other voters of the poll, even without having read the works. (Felice@572 makes similar points.)

...

I agree with AnnieY@575.

...

My organization (the one whose name/website I won't mention because it got rot13'd above) has already provided technical assistance to the Webby awards this year. Marc Kilgour, a leading expert in proportional systems and the author of one of the academic papers cited above, is on our advisory board. Our executive board is aware of, and supportive of, my offer of our assistance here. So this is a serious offer.

#583 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 07:19 AM:

Comments on this thread have been closed. The discussion is continuing here.

Choose:
Smaller type (our default)
Larger type
Even larger type, with serifs

Dire legal notice
Making Light copyright 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 by Patrick & Teresa Nielsen Hayden. All rights reserved.