Back to previous post: On voting systems: a guest post from Bruce Schneier

Go to Making Light's front page.

Forward to next post: The 2015 Hugo finalists, v. 2.0

Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)

April 13, 2015

Discussing Specific Changes to the Hugo Nomination Election: Another Guest Post By Bruce Schneier
Posted by Bruce Schneier at 07:11 AM *

This is a continuation of an earlier thread (now closed) about the Hugo nomination process and how it might be modified. It is not a discussion of either 1) whether or not the nomination process should be changed, or 2) any potential changes in the rules about who is allowed to nominate. It is exclusively focused on voting systems and their relative merits, given what happened in the 2015 Hugo nomination process. Whatever we choose should have these properties:

  • It should be fair.
  • It should be perceived as fair.
  • It should be relatively easy to explain, both to the voters and at the WSFS business meeting.
  • It should be relatively easy to administer.
  • It should encourage people to nominate.
  • It shouldn’t result in too many nominees for the electorate to reasonably read and rank by the Worldcon.
  • It should be resilient to some degree against strategic voting: i.e., minority voting blocs.

In the earlier thread, we identified several different ways to change the voting system. I am going to number them, so we can more easily discuss and compare their properties and suitability.

Option 1: Change the number of candidates a person can nominate. Right now, that number is 5. It can be made less — or more — than 5. Call this parameter x.

Option 2: Change the number of winners of the nomination election. Right now that number is 5, with the possibility of more in the case of a tie. We can make this larger (or smaller, I suppose). We can either fix this number at a single value, or make it variable based on various characteristics of the votes (several ways of doing this is are here, here, here, and here). Call this parameter z.

Option 3: Change the mechanism by which the winners are selected. Right now it is a simple first-past-the-post system, in which the nominees that get the most votes win. There are other ways to choose a winner, some more resilient to bloc voting than others. Here, we have several possibilities:

Option 3a: A Satisfaction Approval Voting (SAV) system (see here, here, and here). In this system, the “satisfaction” of each voter with each possible set of winners is computed based on how many of their nominations win). The winners are chosen to maximize the total satisfaction of all voters. SAV computes each voter’s satisfaction as the fraction of their nominees who are elected.

Option 3b: A Proportional Approval Voting (PAV) system (see here). This system is like SAV, except each voter’s satisfaction is computed with the first nominee elected giving +1 satisfaction, the second nominee +1/2, the third nominee +1/3, and so on - so a voter who had three of their nominees chosen would have a satisfaction of 1 + 1/2 + 1/3 = 1 5/6.

Option 3c: A Reweighted Approval Voting (RAV) — also called Sequential Proportional Approval Voting — system (see here, here, here, and here). This system first nominates the candidate with the most votes. Then, all ballots featuring that candidate are “reweighted” so that votes on them are worth proportionally less. The votes are tallied again with the new weights, and then this process is repeated until all nominees are selected. Multiple values of weights has been proposed; these include (Option 3c-1) d’Hondt (1, 1/2, 1/3, 1/4, …), (Option 3c-2) Saint-Lague (1, 1/3, 1/5, 1/7, …), and (Option 3c-3) exponential (1, 1/2, 1/4, 1/8, 1/16…) as the weights for ballots with 0, 1, 2, 3, … candidates already nominated.

Option 3d: A Single Transferable Vote (STV) system (see here and here, and a simplified version here) probably without — but possibly with — ranking. (Illustration with ranking here.) Votes are “divvied up” among the candidates, according to what the ballots say. Candidates are nominated one by one, based on which one has the most votes. Each time a candidate gets nominated, it “uses up” a certain number of votes, which is taken proportionally from the votes it currently holds. If a voter has elected a pre-determined number of candidates, it is discarded. (Call this parameter y.) If no candidate has enough votes, the one with fewest votes is eliminated. Remaining votes are redistributed after each election or elimination. Variants: with or without ranking; and with or without transfers of “extra” votes.

Option 4: Banning Slates. We create a rule outlawing slates. We’d have to figure out how to define a slate, and how to enforce the rule, but it could be done.

Option 5: Making the Voting Tallies Public Throughout the Process. The voting administrator makes public the current state of the vote throughout the nominating process.

These are not mutually exclusive, of course, although we can choose only one Option 3.

Other systems people have mentioned are Cumulative Voting, Single Divisible Vote, Random Ballot, Condorcet Proportional Representation, anti-votes, and several different ways of adding a third voting/approval round. These have not garnered very much support (for good reasons, I think), and I don’t think these are worth further consideration.

Again, there are many important issues relating to Hugo voting that are not part of this discussion, but should be discussed elsewhere, including: 1) whether to do something at all, 2) whether to change the electorate, either by making voting easier, making it harder, or turning either the electorate into some sort of preselected jury, 3) changing the voting mechanism of the final election in addition to the nominating election. This discussion assumes that whoever decides these things wants something to be done about the nomination system. We’re here to figure out what should be done in that case.

(Thank you to Cheradenine and Jameson Quinn for helping with this summary post.)

Comments on Discussing Specific Changes to the Hugo Nomination Election: Another Guest Post By Bruce Schneier:
#1 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:13 AM:

Example for Options 3a-d:

Assume that 16 ballots are cast, including votes for 10 different works (labeled A through J), and that 3 works are being nominated. In principle, voters in any of these systems can put forth for as many works as they want, but in this example, no one has listed more than 3 works. There is a slate in play, consisting of ABC.

#1: ABC
#2: ABC
#3: ABC
#4: ABC
#5: ABF
#6: AC
#7: AE
#8: BD
#9: DFG
#10: DG
#11: DIJ
#12: EH
#13: EI
#14: EJ
#15: FGH
#16: H

The raw vote counts are A 7, B 6, C 5, D 4, E 4, F 3, G 3, H 3, I 2, J 2. Under the current system, the nominated works would be A, B, and C (i.e. the slate).

Option 3a: Satisfaction Approval Voting (SAV).

We compute a satisfaction score for each voter and each possible group of nominees by looking at the fraction of a voter's choices that were nominated. For example, if B, C, and G were nominated, voter #1 would have a satisfaction score of 2/3 because 2 out of their 3 nominees were nominated. We then add up the scores for each possible set of nominees. For BCG, this yields a satisfaction score (going in the same order as the table) of 2/3 + 2/3 + 2/3 + 2/3 + 1/3 + 1/2 + 0 + 1/2 + 1/3 + 1/2 +0 + 0 + 0 + 0 + 1/3 + 0 = 5.17.

There are 120 possible sets of nominees. For this group, the highest satisfaction score is 6.83 for ABE - so A, B, and E would be nominated. (ABC and ABH are close at 6.67.)

In practice, the winner can also be found by finding a satisfaction score for each candidate, and nominating the 3 candidates with the highest satisfaction score. This yields the same result and requires less computation.

Option 3b: Proportional Approval Voting (PAV).

Again, we compute a satisfaction score, but this time it is 1 if there's one nominee from the ballot, 1 + 1/2 if there are two, and 1 + 1/2 + 1/3 if there are three. The idea is that the more candidates a ballot provides, the less heavily we will weight that person getting more candidates on the ballot. Using this method, if B, C, and G were nominated, voter #1 would have a satisfaction score of 1 + 1/2. We then add up the scores for each possible set of nominees; here, for BCG they are 1.5 + 1.5 + 1.5 + 1.5 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 = 12.

Again, there are 120 sets of nominees to examine. The highest satisfaction score is 14.5 for ADE - so A, D, and E would be nominated. (ABD, ABE, and ADH are close at 14.)

There is no known more efficient way to compute the winner, which means this method is computationally difficult for high numbers of nominees.

Option 3c: Reweighted Approval Voting (RAV).

I'm using the exponential weights (1, 1/2, 1/4, ...) for this example.

We start by nominating the candidate who appears on the most ballots, in this case A. The ballots with A on them (#1-7) are now worth 1/2 the votes for their remaining works.

The new vote counts are B 3.5, C 2.5, D 4, E 3.5, F 2.5, G 3, H 3, I 2, J 2. D now has the most votes and is nominated, and ballots containing D (#8-11) are now worth 1/2 the votes. (If any ballots contained both A and D, they would now count 1/4.)

The vote counts now stand at B 3, C 2.5, E 3.5, F 2, G 2, H 3, I 1.5, J 1.5. E has the most votes left and is our third nominee. ADE is our final ballot. (If we continued another round, B would be the fourth.)

Option 3d: Single Transferable Vote (STV).

I assume that each voter's choices are ranked in the order given above (so slate voters rank A 1st, B 2nd, C 3rd).

The quota (number of votes to guarantee election) is 16/(3 + 1) + 1 = 5.

The first-place vote count is A 7, B 1, C 0, D 3, E 3, F 1, G 0, H 1, I 0, J 0. A has more votes than the quota, so it is nominated. There are 2 surplus votes over the quota, so these are evenly divided over the second choices of A's supporters. This means B receives 10/7 of a vote, C receives 2/7 of a vote, and E receives 2/7 of a vote.

New vote counts are B 2.43, C 0.29, D 3, E 3.29, F 1, G 0, H 1, I 0, J 0. No one has the quota, so we eliminate the candidate(s) with the fewest first-place votes. G, I, and J are eliminated; C is eliminated and their vote does not transfer (since the AC ballot has no candidates left on it); F and H are eliminated and their votes do not transfer (since their ballots are likewise exhausted).

We stand at B 2.43, D 3, E 3.29. No one has the quota, so B is eliminated and the remaining two candidates (D and E) are nominated. The final list of nominees is ADE.

#2 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 07:13 AM:

I think we are going to need to do some modeling of the various systems under different scenarios. Here is some data to start with.

Let's take 2013 is a "normal" year. The voting bloc had just started, and they weren't much of a force. The data is here. Two examples:

Best Novel
Total Ballots: 1113
A: 193
B: 138
C: 135
D: 133
E: 118
---Above six on ballot---
F: 101
G: 91
H 90
I: 74
J: 69
K 68
L: 62
M: 61
N: 58
O: 56
P: 55

Best Novelette
Total ballots: 616
A: 89
B: 62
C: 61
D: 54
E: 55
F: 38
---Above six on ballot---
G: 37
H: 36
I: 35
J: 31
K: 30
L: 28
M: 22
N: 22
O: 20
P: 20

This is what we know about those same two categories in 2015:

Best Novel
Total Ballots: 1827, 587 entries total
A: 387
B: ?
C: ?
D: ?
E: 256
---Above five on ballot---
---Three are bloc entries; two are not---
F: ?
... ?

Best Novelette
Total Ballots 1031, 314 entries total
A: 267
B: ?
C: ?
D: ?
E: ?
F: 165
---Above six on ballot---
---All six are bloc entries---
G: ?
... ?

We also know something about the size of the voting bloc.

#3 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:14 AM:

I generated sample ballot data using the nomination totals from the 2013 Best Novel nominations, and assuming that anything that didn't get enough votes to appear on the list wasn't in contention. (They were left as "minor" votes on the ballots because they affect the outcome of SAV, though not the others.) I used the following assumptions:

* 40% of nominators listed 5 novels, 15% listed 4, 15% listed 3, 15% listed 2, 15% listed 1
* Votes were distributed randomly (no correlation aside from coincidence)

I have no idea how accurate the first assumption is, and I'm sure that votes aren't actually randomly distributed. In the absence of real data, I tried to go with a premise that would involve relatively few biasing assumptions.

I then looked at the following four scenarios. I've numbered the candidates based on the number of ballots they appeared on; so #1 would correspond to Redshirts with 193 first-place votes.

Using the randomized ballots with 2013 vote totals: The same five candidates are nominated. #3 and #4 swapped orders using RAV, but they were only 2 votes apart so that's not very significant.

Using another set of randomized ballots, but assuming that all the supporters of book #6 listed only book #6 on their ballots: Where the current method would still nominate #1-#5, All three methods selected #1, #2, #3, #4, and #6 - so with any of them, there are potential advantages to having your book listed by itself. Using SAV, it generated such a high satisfaction score that it would still have made the ballot with only half the votes. Using PAV and RAV, it was much closer - losing 7 votes would have dropped it off the RAV ballot, and losing 14 would drop it off PAV.

Using the randomized ballots, but adding 150 "puppy" voters who voted for books #2 and #14, as well as three other books that basically no one else voted for: (Apparently this is an alternate universe where the Puppies are all Seanan McGuire fans.) Since only #1 got over 150 votes, the current method would have nominated #1, #2, #14, and two of the three "Puppy only specials." All three of SAV, PAV, and RAV would nominate #1, #2, #3, #4, and #14 - so the slate does add #14 onto the ballot in place of #5, but all 3 methods handle the slate much more robustly than the current method.

Using the randomized ballots, but adding 300 "puppy" voters and 250 "anti-puppy" voters who vote for #12 and 4 works with negligible other support: The current system would nominate #2, #12, #14, and two puppy specials - so one work from the smaller slate and four from the larger. SAV nominates #1, #2, #12, #14 and a puppy special - so the top non-puppy work makes the ballot, though it's close to losing to another puppy special. I think PAV nominates #1, #2, #4, #12, and #14 - so two puppy works (one of which would have made it anyway), one anti-puppy work, and two non-slate works. The result of RAV depends on the weights used. With Sainte-Lague weights, it nominates the same candidates as PAV #1, #2, #4, #12, and #14. With d'Hondt or exponential weights, #4 gets booted in favor of an anti-puppy candidate, so each slate gets 2 nominees and the remaining nominee is the most popular non-slate candidate.

As above, but the "anti-puppies" vote a slate closer to the overall preference, slating #1, #4, #5, #7, and #12: The current system would nominate #1, #2, #4, #5, and #14 as the five candidates with the most votes. SAV nominates #1, #2, #4, #5, and #7 - so the slightly smaller slate that's more popular with non-slate voters now gets 4 nominations. PAV and RAV (using any system of weights) nominate #1, #2, #4, #5, and #14 - so in this case they actually agree with the popular vote totals.

As above with two slates, but 1/3 of the non-slate voters remove any slated work from their ballots: This assumes they don't replace them with non-slated works, which is probably unrealistic but makes the computation manageable. Very much to my surprise, the results are completely unchanged from the case above.

Predicting the winner using STV is tricky because we really have no information about what ranked ballots would look like. The quota would be 187 for the first two situations, so #1 would immediately be nominated, its 6 excess votes would be transferred, and then we would have to start eliminating the candidates with less support and transferring their votes. In the second case, whether #6 got nominated would depend on whether enough ballots got exhausted to put 101 votes in the top 5 vote counts.

For the 150-puppy case, the quota would increase to 212. Book #2 might meet this if a large number of puppies listed it first. Assuming that was the case, the 76 excess votes would be distributed among book #2's other supporters. This would probably leave the puppy group with too few votes to get another book on the ballot, though #14 would be an outside possibility.

With 300 puppies, the quota becomes 237. As long as the puppies are in reasonable agreement on a first choice, it would clear the quota even with no non-puppy support. Depending on vote distribution, enough votes might well transfer to nominate another puppy candidate, but more than that would still be fairly unlikely.

Note that STV is likely to nominate the puppies' favorite candidate, whereas the other methods favor slate candidates with the greatest non-slate support.

Please take this analysis for STV with an extra-large grain of salt, since I don't have a good enough idea of what the ballots would look like to convincingly simulate the election using this method.

#4 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:15 AM:

I did a smaller set of experiments based on the novelette category to see what happens when the nominations are more dispersed. There were a total of 616 votes cast; the first-place story received 89 votes, and the fifth-place story received 45. I distributed votes randomly, assuming in this case that equal numbers of nominators listed 1, 2, 3, 4, or 5 works.

Using the randomized ballots with 2013 vote totals: SAV nominated the actual #1, #2, #3, #4 and #8 - it received 36 votes, but they wound up heavily concentrated on ballots which didn't list #1-#4, which pushed it just ahead of #5 for the last spot. PAV and RAV (for all weights) matched the actual nominations of #1, #2, #3, #4, and #5.

Adding 100 "puppy" voters who voted for five stories that picked up 0-18 non-slate votes each Under the current system, the puppies would completely sweep the ballot (as happened this year). SAV, PAV, and RAV with D'Hondt or exponential weights all nominate #1, #2, #3, and 2 puppy picks. RAV with Sainte-Lague weights nominates #1, #2, #3, #4, and the top puppy pick.

Adding 200 "puppy" voters as above: Under the current system, this is a dominant puppy win. SAV would also yield a sweep for the puppies. PAV, and RAV with D'Hondt weights, yield a ballot with #1, #2 and/or #3 (they tie for the last spot), and three puppy candidates. RAV with exponential or Sainte-Lague weights nominates #1, #2, #3, and two pieces of puppy chow.

It's hard to predict the effects of STV here. Because the votes are so dispersed, a lot of ballots are going to wind up exhausted as lower-vote-count stories are eliminated. At least one slate nominee would be chosen, and I would guess not more than 2-3, but I'm definitely guessing here.

#5 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 07:15 AM:

Okay, so let's have a discussion. If you're commenting on a particular proposal, please use the option number (or numbers) so we can all more easily follow what you're talking about.

Please think about this for a while before commenting. Considered opinions are more valuable than snap judgments. And while this can be a theoretical discussion, opinions from non-experts are valuable and wanted.

Someone somewhere else should take up the discussion of whether it's better to do nothing, and whether changing the electorate is a good idea or not. And, as long as we're tinkering under the Hugos' hood, maybe we should talk about updating the IRV process of the final election as well. When there are such discussions in progress, please post links here so we can join it if we want to.

#6 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 07:17 AM:

Stating my preference....

I don't want to make voters rank nominees. I want nominating to be as easy as possible -- "It's okay that you haven't read everything, just send us names of works you think are worthy" -- and ranking makes nominating harder. I don't want to make anything public in real-time to encourage more campaigning and more strategic voting. I think adding a third round of any kind would make the election too hard to administer. And I think that simpler systems are more likely to make it through the business meeting.

My suggestion: Voters should be allowed to nominate up to four candidates (x=4), that the number of total nominees should be five (z=5) or more using the Next Two Rule, and that the nominees should be chosen using RAV with exponential weights: Option 3c-3.

Making x less than y ensures that a single slate can never dominate the nominees, while the specific choices of x=4 and y=5 make the new system as close as possible to the old system. The Next Two Rule allows nominees that just barely lose, something that I think is a goodness in any case. And while RAV is definitely harder to explain than PAV, it's easier to administer, Plus, there's no change for the voters: they just nominate works they feel deserving.

I am partially opposed to Option 4: Banning Slates. I can see how to detect slates, both in the received votes and by watching the Internet for campaigning. But I don't like hurting authors who are put on slates against their will, and have no way to verify if someone is on a slate willingly or not. And there are too many edge cases that worry me. On the other hand, it's a clean solution to the problem.

I am strongly opposed to Option 5: Making the Voting Tallies Public. It does not make the distributed and unorganized general electorate more powerful. Instead, it makes slates more powerful by giving them more information. And this is to the extent it's actually useful. Already most people vote at the last minute. And any smart slate would do the same. This is exactly the sort of change we should not make; it focuses on the details of this year's tactics rather than the broader problem.

No proposal is going to be immune from strategic voting, and no proposal other than banning voting blocs outright is going to eliminate their effectiveness. Our goal should be to agree on a voting system that reduces the influence of voting blocs, even as it is accepts their power.

#7 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:18 AM:

Thoughts and Recommendations

Most of the options listed here would be at least somewhat effective at keeping a slate from completely dominating the Hugo ballot. Here are my thoughts based on playing with the (faux) ballot data:

Option 1: Fewer nominations per voter. This would make it much more difficult for a slate to completely dominate the ballot. On the other hand, by reducing the number of nominees a non-slate voter can list, it may actually increase a slate's ability to get x candidates onto the ballot. At best, we'd be guaranteed a ballot with a couple non-slate candidates (and once multiple slates start showing up, non-slate candidates are going to have a very tough time).

Several of the voting methods in Option 3 actually allow a voter to list a theoretically unlimited number of candidates and can yield a better result in consequence. I think going in that direction or keeping the number at five are better choices for us.

Option 2: Nominate more candidates. This effectively moves more of the decision process to the second election, which is less vulnerable to slates. If multiple slates appear, it would be very easy to end up with a ballot in which every option is from a slate. I also have concerns that the length of the Hugo ballot would mean a big increase in voters who hadn't read most of the nominated work, or a decrease in voters as people find the list too overwhelming.

Option 3a: SAV. Having run the simulations and looked at some of the literature, I'm not enthusiastic about this method. It doesn't do enough to stop slates, and it makes the strategy of listing only one candidate too powerful.

Option 3b: PAV. This system was reasonably effective at maintaining representation for non-slate voters. However, I can't tell you with certainty who wins for some of the examples, because my computer doesn't have the processing power to examine all the possible candidate sets. One paper offers a proof that the problem is NP-Hard (though I'm not enough of an expert in the area to validate the proof). To quote my CS professor, "If you don't know what NP-Hard means, it means really hard. It's bad. NP-Hard is bad."

One option would be to use PAV after dropping all candidates under a certain threshold or above a certain number from the ballot. This makes computability less of an issue, but further complicates an already difficult-to-explain system.

Option 3c: RAV. This system was reasonably effective at maintaining representation for non-slate voters. D'Hondt weights were both the least effective in the area and have some undesirable theoretical properties. Sainte-Lague weights controlled slates most effectively, but exponential weights were close and might be easier to explain ("every time you get a nominee, your vote counts half as much" versus "after the first nominee, it counts 1/3, then 1/5, then 1/7, and so on..."). While the system increases the workload of computing the nominees, I found it easier than any of the other Option 3s.

Option 3d: STV. This is the system I was least able to experiment with, due to its dependence on ranked ballots. It would considerably dilute the power of a slate after it gets its first nominee, but it would increase the load on nominators by requiring a ranked list. It is used for Academy Award nominations so there's some history of its use for this sort of purpose, but I don't think they release their underlying data pretty much ever. (Also, they have the resources to contract Price, Waterhouse to do the actual computations. We probably don't.)

Option 4: Banning Slates. I can see two ways to do this: lay out in exacting detail what a slate is (which means that people will immediately game their way around the definition), or leave identifying slates up to the Hugo administrators, which is a responsibility I suspect most concoms do not want and are not prepared for. I'm all for a statement in principle that slates are inconsistent with the goals of the Hugo Awards, but enforcement is a thornier problem.

Option 5: Public Voting Tallies. In general, this would make tactical voting easier since some information about the state of the election would be public. Encouraging tactical voting means we'd have less information about people's actual opinion, but we might get a better outcome than in the slate situation where only one group votes tactically. That leaves me two primary areas of concern. First, the method is very vulnerable to bad actors. Voting for one candidate then switching their votes to another could confuse the picture of where actual preference are (and potentially allow them to gauge the difference in votes between candidates). A DDoS attack on the servers near the deadline could also cause havoc. (There'd be some danger of the servers going down as everyone logged in to cast their final votes anyway.) Finally, it may not be practical to assemble a list of all possible candidates in advance of launching the site (and a partial list would be a de facto slate). This means that, as mentioned in the previous thread, the administrators will need to curate submissions to identify identical ones and maintain the list. This becomes a more burdensome responsibility when it needs to be done in real time, or even weekly.

Of these options, then, I would recommend RAV with Sainte-Lague or exponential weights as the change most likely to limit the power of voting slates without too many negative side effects, with STV as a runner-up.

One paragraph of non-voting-system stuff: no change to the voting system will eliminate the power of slates entirely. In my opinion, a rules change barring slate voting is unenforceable and would enmesh future Hugo administrators in a bog of controversy. Social pressure against slates (e.g. non-slate voters declining to nominate or vote for slated works, a statement that slate voting undermines the premises of the Hugo) might help, especially in combination with a voting system change that limits the power of slate. These voting system changes would tend toward producing a more diverse group of nominees, which ought to make the more reasonable SPs happy (assuming they actually exist).

While I have some experience working with voting systems, I am hardly an expert and you shouldn't take my word as authoritative. If you see any errors in any of this work, please correct me. I'm happy to share the spreadsheet I've used to run these simulations; I'm also happy to simulate other situations, especially if you send me a set of ballot data. (Generating "plausible" ballot data is the most time-consuming part.)

I look forward to a lively discussion of the different options! Thank you for listening.

#8 ::: John A Arkansawyer ::: (view all by) ::: April 13, 2015, 07:51 AM:

Cheradenine @ 7: "Social pressure against slates...might help, especially in combination with a voting system change that limits the power of slate."

"once multiple slates start showing up, non-slate candidates are going to have a very tough time"

And in those two statements, you have made the case for acting with some urgency. For better or for worse, social pressure can prevent change much more easily than it can reverse it.

I started to present a case where there were more slates than there were nomination slots--the Urban Fantasy slate, the Young Adult slate, the Best-Selling Authors slate, the MilSF slate, the Social Justice slate, the Slipstream slate, the Alternate History slate--when I realized that if things got to that point, there was probably no good solution and no going back.

I don't want to run that experiment.

#9 ::: J Thomas ::: (view all by) ::: April 13, 2015, 08:09 AM:

Looking at the comments in this topic so far, I can imagine the following situation arising:

Someone looks at the votes for nominations, and they announce:

"These three candidates got the 2nd, 3rd, and 4th most votes. But none of them are on the list of nominations. Why not?"

And then someone else will have the responsibility to explain the voting system in a way that the general public will understand.

If it stays controversial in 2017, when the new system comes in, it's possible that Fox News might spend a few minutes on it, and somebody involved with the voting might have 15 seconds or 30 seconds to explain why the voting system is OK.

Maybe the best way to deal with this would be to keep the votes secret until nobody cares.

#10 ::: nathanbp ::: (view all by) ::: April 13, 2015, 08:45 AM:

Bruce, #6: If we're using RAV with exponential weights (Option 3c-3), which is my preference as well, then if someone's 5th vote counts it will be as 1/16 of a vote, so I feel like there's not much of a point to limiting people to 4 nominations per category instead of leaving it at 5.

Also, looking at the 2013 data, the Next Two rule does not seem like a good idea. Since most categories have no sharp cliff of votes but instead decrease gradually, in almost every category it picks all 15 works listed to be nominees (and would probably continue on to almost the end of the list). Using the 60% threshold as proposed towards the end of that paper would have helped in some categories, but not all of them.

On option 5, what if the current nomination status was released just once during voting, at say one month before voting closes? Release a list of the top 15 in each category in alphabetical order without vote counts. I feel like this would help to concentrate votes without giving out too much information that could be used strategically or being subject to gaming.

#11 ::: Vivien ::: (view all by) ::: April 13, 2015, 08:47 AM:

I mostly agree with Bruce Schneier :

RAV works as well as anything we could try. Difficult but not impossible to explain, especially to a community that is using something as complex as IRV for the final election. Testable on previous data.

Exponential coefficients are the easiest to explain to someone who doesn't know what a suite is.

Next two rules : interesting, and remarkably polemic-free.

Nominate 4, 5 on the final list : well, it is not useless, but that is maybe the extra-rule that will get your average WSFS member to tell you to leave him alone with your damned rule changing.

#12 ::: Vivien ::: (view all by) ::: April 13, 2015, 08:48 AM:

Ank thank you Cheradenine for the simulations, they are great !

#13 ::: Chris Lawson ::: (view all by) ::: April 13, 2015, 09:10 AM:

Cheradenine -- good work. I'd love to see someone with better programming skills and a regular website take this even further and put up an online simulator of the different voting options using the 2013 and 2014 nomination data to compare how each system would work out.

Bruce S. -- re: banning slates: if we can detect slates statistically, I suspect we can also estimate the proportion of a book's nominations due to the slate, in which case it is possible to ban a slate without hurting an author by removing only that proportion of nominations deemed to be slate-dependant.

#14 ::: Galen Charlton ::: (view all by) ::: April 13, 2015, 09:21 AM:

John A Arkansawyer @ 8: I started to present a case where there were more slates than there were nomination slots--the Urban Fantasy slate, the Young Adult slate, the Best-Selling Authors slate, the MilSF slate, the Social Justice slate, the Slipstream slate, the Alternate History slate--when I realized that if things got to that point, there was probably no good solution and no going back.

I share your hope that we don't end up with this state of affairs, but it occurs to me that if we do, there may be a small upside. The appearance of (say) a YA slate could be interpreted as a signal of a desire for a new Hugo Award category. That suggests to me an orthogonal sort of change: accommodating slates that exist to promote sub-genres by encouraging each Worldcon to use its privilege to name a special award category. This would allow us "try out" potential permanent categories and, in the long run, give more opportunities for folks who feel that their favorite subgenre is underrepresented to nominate.

As I suspect that this comment is almost, but not quite, off-topic for this particular post, perhaps any follow-ups should go in the "Clean Living" thread.

#15 ::: Kimiko ::: (view all by) ::: April 13, 2015, 09:24 AM:

I do not know what the best solution is, but the most straightforward implementation looks like the one Bruce outlined above:
My suggestion: Voters should be allowed to nominate up to four candidates (x=4), that the number of total nominees should be five (z=5) or more using the Next Two Rule, and that the nominees should be chosen using RAV with exponential weights: Option 3c-3.

Making x less than y ensures that a single slate can never dominate the nominees, while the specific choices of x=4 and y=5 make the new system as close as possible to the old system. The Next Two Rule allows nominees that just barely lose, something that I think is a goodness in any case. And while RAV is definitely harder to explain than PAV, it's easier to administer, Plus, there's no change for the voters: they just nominate works they feel deserving

There's also something elegant about using exponential weights. I think implementing such a thoughtful system would combine nicely with an anti-slate social norm. "Look at this cool thing we built so people won't wreck our awards. Isn't it shiny? Let's not try to break it, because that would be uncool."
I'm going to go back through the examples given and see if I can understand how anti-slate voting interacts with it.

#16 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 09:30 AM:

nathanbp @10:

"Bruce, #6: If we're using RAV with exponential weights (Option 3c-3), which is my preference as well, then if someone's 5th vote counts it will be as 1/16 of a vote, so I feel like there's not much of a point to limiting people to 4 nominations per category instead of leaving it at 5."

Makes sense. I agree.

"Also, looking at the 2013 data, the Next Two rule does not seem like a good idea. Since most categories have no sharp cliff of votes but instead decrease gradually, in almost every category it picks all 15 works listed to be nominees (and would probably continue on to almost the end of the list). Using the 60% threshold as proposed towards the end of that paper would have helped in some categories, but not all of them."

Interesting. Maybe we should just expand nominations to 6 or 7.

"On option 5, what if the current nomination status was released just once during voting, at say one month before voting closes? Release a list of the top 15 in each category in alphabetical order without vote counts. I feel like this would help to concentrate votes without giving out too much information that could be used strategically or being subject to gaming."

I really dislike this. It's too much responding to the tactics, and not looking at the broad picture. The risk of unforseen consequences are great.

#17 ::: Martin (not a glyptodont) ::: (view all by) ::: April 13, 2015, 09:33 AM:

I am also in favor of 3c3, plus an "honor pledge" that you haven't voted a slate. Don't include any enforcement details - if good faith + social pressure is not enough to stop Hugo slates, Hugo slates are probably a given from now on and folks should be laying infrastructure for their Hugo Political Party rather than trying to save the old way.

More testing of various methods vs. multiple (at least 2 and 3) slates is warranted. I would be surprised if there was much hope for independent voters under any system in the 3 slate arena.

As for hope of implementation, I think the following approach is important - the anti-slate movement cannot be a specifically anti-Sad-Puppy movement. They to want to stop a shadowy cabal - and anti-slate is also anti-shadowy cabal. Rabid Puppies are not going to help us here, but if we do it right the Sad ones might. Since the whole point is to return the Hugos to good will and general consensus, working with everyone who is even minimally work-with-able is sort of required.

#18 ::: Reimer Behrends ::: (view all by) ::: April 13, 2015, 09:36 AM:

I note that if there is a voting bloc that has more than five times as many members as the otherwise most popular candidate in a given category receives votes, they will always be able to get five nominations in that category under all current option 3 proposals (by splitting their votes evenly between their five preferred candidates and casting only one vote per ballot); smaller voting blocs can unavoidably still obtain 1-4 nominations. While it is difficult to coordinate voting behavior to obtain such an even split, there are several ways in which it can be approximated with relatively little overhead. The primary benefit of slate voting over this approach currently is reduced recruitment effort and that it is cheaper; you get essentially five votes for the price of one.

This is why I rather like what Joshua Kronengold calls "Single Divisible Vote" (which is essentially a variant of unranked STV), because it eliminates that incentive by making each ballot is worth one vote; to get the current benefit of slate voting, you'd have to cast five ballots.

#19 ::: Randolph ::: (view all by) ::: April 13, 2015, 09:45 AM:

Has anyone considered returning to a mailed-in paper ballot for nominations? That would keep out the internet trolls, at the cost of some serious inconvenience. Such a ballot could still be gamed, but it would be a lot more difficult.

#20 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 10:05 AM:

Vivien @12, Chris @13: You're welcome! It was very interesting to look at what happened in more depth (even if it did kind of eat my weekend). Chris, I'd be delighted if someone wanted to take the website idea and run with it (my programming skills are sadly limited).

I do really want to track down some genuine data and see how the system performs. My hunch is that in the absence of a slate RAV is highly likely to yield the same set of nominations as the historical set, but I'd like more to go on than the hunch. I know someone in the other thread thought they could get their hands on the 1984 ballot information - I await with bated breath.

Reimer @18 - In 2013, at least, the leading nominee in each category had at least 14% of the vote. If a voting bloc controls more than 5 times as many voters as that, they're already pretty much in control of the election.

I can take a look at an SDV simulation if that would be helpful. (Not while I'm at work, though!)

#21 ::: J Thomas ::: (view all by) ::: April 13, 2015, 10:20 AM:

Given the goal to discourage slates, it might be best to actually discourage slates. When you discourage something that sort of correlates with slates, you can get unexpected side effects even while you don't always hit the slates you want to.

So here is an approach to do that.

We filter the ballots.

For each voting category with five slots, eliminate any ballots which are identical. Any ballots which are completely unique -- no one else voted for any of these nominations -- get no further filtering.

Ballots which are identical except for one item, get counted as 1/5 votes.

Ballots which are identical except for two items get counted as 2/5 votes.

Ballots which are identical except for three items get counted as 3/5 votes.

Ballots which agree on one item count as 4/5 votes.

This will eliminate any value from slate voting. People may however try to do strategic nominating by selecting one nominee they want to push, and add four other nominees they hope that no one else would think of nominating.

It would be a courtesy for the automated filtering system to send each nominator a message telling them how much their vote counted.

#22 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 10:28 AM:

J Thomas @21: I think this is likely to cause considerably more collateral damage to non-slate nominators than something like RAV. Keep in mind that a significant portion of nominators don't list five items, which significantly increases the odds of their ballots being partially or completely canceled out.

#23 ::: J Thomas ::: (view all by) ::: April 13, 2015, 10:35 AM:

Keep in mind that a significant portion of nominators don't list five items, which significantly increases the odds of their ballots being partially or completely canceled out.

Tell them to list five items.

Or count blanks as misses, so that someone who lists only one nomination can expect it to count as 80% of a vote.

#24 ::: Seth Gordon ::: (view all by) ::: April 13, 2015, 10:43 AM:

Many thanks to Bruce and to Cheradenine for applying your math-fu to this cause. I also want to co-sign Martin’s remark above that a voting method that resists slate voting will also resist shadowy cabals, and therefore is a Good Thing, whether or not you think there is a shadowy cabal that needs resisting.

Option 1: I would rather preserve as many of the existing rules as possible (it seems to make both good political and engineering sense), so let’s leave this alone unless we absolutely have to.

Option 2: I’m not keen on giving Hugo voters even more stuff to read (it increases the temptation to skip the reading and vote based on shallow considerations, which is of course one of the things we are trying to prevent). But see my response to option 4 below.

Option 3: I am torn between 3a, which seems easiest to explain, and 3c, which seems most slate-resistant.

Option 4: I don’t like the idea of banning slates by fiat, because I can see future Hugos descend into meta-controversies over whether such-and-such a group of candidates is or is not a slate. However, I think it might be worth giving each Worldcon the discretion to add up to three(?) nominees to each category if it sees irregularities in the nominating process, including but not limited to slate voting. (This can be proposed as a rule change independent of Option 3.) Voters who think these special nominees did not deserve such honor can, of course, express their disapproval by ranking them below you-know-who.

Option 5: This is an anti-gaming tactic which can, itself, be gamed aggressively, so I am against it.

PS: I am intrigued by the Majority Judgment system that Jameson Quinn proposes here. It’s too radical a change for the Hugos themselves, but perhaps another large convention (DragonCon or ComicCon) might be inspired to try it out, or a Worldcon could use it to elect a “Special Award of Merit” independent of the Hugos.

#25 ::: nathanbp ::: (view all by) ::: April 13, 2015, 10:51 AM:

Bruce, #16: "Interesting. Maybe we should just expand nominations to 6 or 7."

Looking at the nomination counts they mostly seem to gradually trail off, so I'm not sure that any algorithm would be able to pick a reasonable stopping point.

Reimer Behrends, #18: I'm not sure what you're trying to say? Under SDV any group with 5x as many members as the next most popular work gets all 5 of their picks on the ballot too, and without having to guess what the final totals will be. Under RAV if they try the 5 way split they can end up with no works on the final ballot. If they vote for all works under RAV they'll get 3 of the final slots.

Example (I'm simplifying and assuming no cross work support for QRSTV):
101 votes for ABCDE - The Slate
20 votes for Q
19 votes for R
18 votes for S
17 votes for T
16 votes for V

Without the slate QRSTV are nominated.
With the slate under SDV, ABCDE each get 20.2 votes and are nominated (similar result from STV).
With the slate under RAV, ABC win with 101, 50.5, 25.25 votes each, then Q and R get the remaining 2 nominations.

J Thomas, #21: I agree that the listed systems don't do a lot to punitively punish slates, instead just limiting them to something like their appropriate power given the number of voters they command. I think if you want to do something like that the simplest system is Bruce's #1 on the old post (once one thing on a ballot wins, it doesn't count for the rest of the process).

#26 ::: TomB ::: (view all by) ::: April 13, 2015, 11:11 AM:

Option 4: Banning Slates:

We know that slates are bad because they disenfranchise non-slate nominations.

Banning slates will discourage the promotion of slates, encourage voters to nominate their own personal preferences, and invalidate arguments that slates are allowed within the rules.

Enforcing the rule is complicated. The simplest solution would be to enforce it by using one or more of the other options. You might ask then, why bother banning slates? Because the other options can at best only mitigate slates and do nothing to prevent them. It's like trying to stop spam without having any rule that spam is not allowed.

#27 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 11:16 AM:

Seth Gordon @24:

...a voting method that resists slate voting will also resist shadowy cabals, and therefore is a Good Thing, whether or not you think there is a shadowy cabal that needs resisting.
There are no shadowy cabals. There never have been. This is a fact. Facts matter.

#28 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 11:20 AM:

If one set of works has 5x as many votes as any others, it should win. If a work that is that massively popular compared to the others *fails* to get on the ballot, that's a problem.

#29 ::: Steve Halter ::: (view all by) ::: April 13, 2015, 11:26 AM:

3c-3 seems to fit the various goals the best. Setting x=5 (number of nominations) seems simplest and is not much different than x=4 under 3c-3.

I don't really like a precise rule banning slates as it would seem to leave room for too much gaming--positive or negative. Some sort of non-binding statement that voters should read things and vote for what they personally like could be useful in the voting guide.

I would be against making the current tallies public (Option 5).

#30 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 11:26 AM:

So, let's turn these scenarios on their head: they're all assuming that the slate is from a committed minority of voters. Let's say that the vast majority of voters vote for a slate. The slaters have a 2:1 majority over the non-slate voters. I would argue that in this case, the slate should just win.

How does the slate fare under the various schemes listed here if this is the case? Assuming that it does not succeed in getting all of its nominees in, how much of a majority does a slate need to have before it gets all of its members elected under the various 3.x schemes?

#31 ::: Pfusand ::: (view all by) ::: April 13, 2015, 11:30 AM:

"Maybe we should just expand nominations to 6 or 7."

This would be fine for short stories, but a real killer for novels.

#32 ::: J Thomas ::: (view all by) ::: April 13, 2015, 11:42 AM:

Option 5: Public Voting Tallies. In general, this would make tactical voting easier since some information about the state of the election would be public. Encouraging tactical voting means we'd have less information about people's actual opinion, but we might get a better outcome than in the slate situation where only one group votes tactically.

I want to stress a basic point: To deal with slates etc, we can find a way for the good people to come to agreement -- not a bad way where they just play politics, but a good way where they notice which of the things they like are popular enough to win.

Or we can look for ways to manipulate the vote or the vote-counting so that the bad people who have voted together to embarrass us do not have their votes counted.

There is nothing morally wrong with the first approach. But the good guys might not in fact reach agreement, and then they could lose. It might be more reliable to find a way to count the votes so the bad guys cannot win even if the good guys don't vote for competitive candidates.

That leaves me two primary areas of concern. First, the method is very vulnerable to bad actors. Voting for one candidate then switching their votes to another could confuse the picture of where actual preference are (and potentially allow them to gauge the difference in votes between candidates).

If it's done by, say, acceptance voting then there's a limit to the bad guys' ability to game the system. They can vote for somebody they don't like to see how it changes the numbers, but they can't unvote afterward.

A DDoS attack on the servers near the deadline could also cause havoc. (There'd be some danger of the servers going down as everyone logged in to cast their final votes anyway.)

Could you possibly accept, say, email ballots sent to a prominent mailserver? You get a timestamp even if you don't get the actual ballots until later. It would be harder to DDoS them than you. There might be other ways to palliate the problem.

Finally, it may not be practical to assemble a list of all possible candidates in advance of launching the site (and a partial list would be a de facto slate). This means that, as mentioned in the previous thread, the administrators will need to curate submissions to identify identical ones and maintain the list. This becomes a more burdensome responsibility when it needs to be done in real time, or even weekly.

How is that handled now? Do the administrators have a list of all possible candidates in case somebody nominates one of them? Do they have a truncated list and nominations have to come from it?

If some nominator or kibitzer suspects that two nominations are the same thing but misspelled or something, won't they tend to announce it? News often spreads by the fastest available channel....

I don't see what's different about making nominations between the old system and this proposal.

#33 ::: Clay Shentrup ::: (view all by) ::: April 13, 2015, 11:43 AM:

The best option is Proportional Score Voting, aka Reweighted Range Voting.
http://scorevoting.net/RRV.html

It's simpler than STV and superior in the sense that it satisfies a proportionality theorem. It can be tabulated using a Google spreadsheet and a few simple formulas.

The simplest form Proportional Score Voting is Proportional
Approval Voting, described in your "Option 3c". Here's a video where I demonstrate how to tabulate this method in Google Spreadsheets.
https://www.youtube.com/watch?v=_jS7b-0PV9E

> It should be fair.

This is a basically meaningless desideratum frequently referenced by people not versed in social choice theory. Suppose we can select option X, which will give Bob and Alice utilities of 5. Or we can select option Y, which will give Bob a 6 and Alice a 7. Do you really want to choose X (thereby making Bob and Alice BOTH less happy) in the name of "fairness"?

Answer: NO. A rational organism's goal is to maximize its expected utility, which means maximizing the net utility of the group. You do NOT want to maximize "fairness", because it is a nonsensical/meaningless concept in the first place.

Clay Shentrup
Co-founder, Th Cntr fr lctn Scnc

#34 ::: TomB ::: (view all by) ::: April 13, 2015, 11:44 AM:

Bruce Schneier @6: "Already most people vote at the last minute. And any smart slate would do the same."

I proposed a fix for that in #569 on the original thread.

To prevent sniping, extend the deadline until the moving average of the nomination rate goes below a threshold.
In practice, the deadline will always be extended at least one day because of normal last-minute nominating. If there is a controversy, the various parties will try to get more people to nominate. The nominations will remain open until all parties are exhausted and the nomination rate goes down.
To prevent endless procrastination, charge a late fee to nominate after the initial deadline.
With the fix, Option 5 would still turn the nomination process into a tactical GOTV process, but at least it would not be broken.

#35 ::: Douglas Henke ::: (view all by) ::: April 13, 2015, 11:44 AM:

The endgame of Option 5 is that "vote this slate" becomes "download and run our software agent to automatically vote our slate during the last possible tens of milliseconds". See also eBay sniping software.

This means that, while the nomination process is in progress, the block voters have perfect information about the behavior of the honest voters, while the honest voters have zero information about the behavior of the block voters. This is a pessimal outcome.

Option 4 comes in two flavors. Either the definition of a slate is objective (4a) or subjective (4b).

4b immediately fails the "be fair", "look fair" and "be easy to administer" requirements. If the test for something being a slate is to look like one in the personal opinion of one or more people, then anyone supporting something declared to be a slate will -- probably, rightly -- cry foul.

4a requires a specific objective test. It seems as though any such test would itself be game-able, would risk false positives, or both. In any case, I would suggest that anyone proposing 4a need also propose a specific test in order for their proposal to be seriously considered.

(Aside: My comment 424 in the previous thread suggested a test for 4a: multiple ballots highly correlated across multiple nomination categories. This fails the "easy to explain" requirement -- do you want to stand up in front of the WSFS business meeting in Spokanistan and start talking about eigenvectors?)

#36 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 11:46 AM:

Thoughts

Option 1: I agree with nathanbsp@10 that if we're using RAV, especially exponential RAV, there's no point in limiting the number of nominees per ballot. And allowing more than 5 could help promote more overlap between non-slate voters in "long tail" categories such as "short story". This would help more nominees pass the 5% threshold. Also, since it means more non-slate votes overall, it increases the number of slate votes that would be required to get a given number of slate-based nominees. (This is still treating slate- and non-slate- votes fairly; it's just that permitting more nominations naturally does different things for those two cases).

Option 2: I like the basic idea of Bruce's suggestion of an adaptive number of nominees, but the next-2 rule¹ is clearly not right in this case. That rule was designed for a different ballot format, and on this data, it could easily go crazy and include all the nominees.

There are two goals for having an adaptive rule.

First, it's good to deal with a "long tail" situation. Imagine a category where the top nominee is only on 10% of the nominating ballots. Then, there's no way to limit it to 5 nominations without ignoring over 50% of the ballots. In order to deal with this, if you were using RAV, you could have a rule about not quitting until the total remaining ballot weight is below some proportion p — say, 60%-70% — of the original number of ballots. That is another way of saying that at least 2(1-p) of the ballots have at least one nominee they supported. So, if you set the threshold at 60%, you're adding nominees until you've listened to at least 80% of voters; at 70%, you're listening to at least 60%.

The second goal for an adaptive rule is to limit cases of "almost made it". So you don't want the between the lowest nominated work (call this "work n") and the biggest non-nominated work (call this "work n+1") to be "small" in some sense. One reasonable rule would be that this gap should have to be bigger than the gap between work n+1 and work n+2, or else you set n

#37 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:48 AM:

Fred @30: If 60% of voters vote for a slate and assuming that other voters are not voting a counter-slate, the likely outcomes are:

Options 1-2: The slate gets a number of candidates nominated equal to the number they can vote for.

Option 3a: Probably elects all of the slate candidates.

Option 3b: Probably elects all of the slate candidates.

Option 3c: With d'Hondt weights, probably elects all of the slate candidates. With Sainte-Lague weights or exponential weights, probably elects 3-4 of the slate candidates.

Option 3d: Probably elects 3 of the slate candidates.

Under 3c or 3d, the 3 slate candidates who are most popular with non-slate voters will be nominated, and presumably one of them will win the election. Under 3c, it is possible that the 1st choice of the slate voters won't make the final election if it is very unpopular outside of the slate, though this is not a likely scenario. If the slate voters have a strong 1st choice, they might be better off *not* voting in lockstep on the remaining candidates.

#38 ::: J Thomas ::: (view all by) ::: April 13, 2015, 11:50 AM:

There are no shadowy cabals. There never have been. This is a fact. Facts matter.

People might wonder how you know that. If you aren't a member of a shadowy cabal, how can you be sure whether it exists or not.

I want to assure you about this. I am a member of a couple of shadowy cabals and so I am in a position to affirm that there are no shadowy cabals and there never have been. I know.

#39 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:52 AM:

Clay @33: I'm not sure how voters can meaningfully assign scores in a nomination election when they don't have access to the entire field to compare it to, so range voting strikes me as problematic here. Simple approval ("yes, I want this item to be on the ballot") strikes me as preferable here.

#40 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 11:53 AM:

Sorry, my previous post got borked by a stray less than sign. Here it is again:

Thoughts

Option 1: I agree with nathanbsp@10 that if we're using RAV, especially exponential RAV, there's no point in limiting the number of nominees per ballot. And allowing more than 5 could help promote more overlap between non-slate voters in "long tail" categories such as "short story". This would help more nominees pass the 5% threshold. Also, since it means more non-slate votes overall, it increases the number of slate votes that would be required to get a given number of slate-based nominees. (This is still treating slate- and non-slate- votes fairly; it's just that permitting more nominations naturally does different things for those two cases).

Option 2: I like the basic idea of Bruce's suggestion of an adaptive number of nominees, but the next-2 rule¹ is clearly not right in this case. That rule was designed for a different ballot format, and on this data, it could easily go crazy and include all the nominees.

There are two goals for having an adaptive rule.

First, it's good to deal with a "long tail" situation. Imagine a category where the top nominee is only on 10% of the nominating ballots. Then, there's no way to limit it to 5 nominations without ignoring over 50% of the ballots. In order to deal with this, if you were using RAV, you could have a rule about not quitting until the total remaining ballot weight is below some proportion p — say, 60%-70% — of the original number of ballots. That is another way of saying that at least 2(1-p) of the ballots have at least one nominee they supported. So, if you set the threshold at 60%, you're adding nominees until you've listened to at least 80% of voters; at 70%, you're listening to at least 60%.

The second goal for an adaptive rule is to limit cases of "almost made it". So you don't want the between the lowest nominated work (call this "work n") and the biggest non-nominated work (call this "work n+1") to be "small" in some sense. One reasonable rule would be that this gap should have to be bigger than the gap between work n+1 and work n+2, or else you set n to n+1 and try again.

Let's test these rules looking at "Novelette" from 2013. Since we have no reason to believe there was too much slate voting that year, let's assume that the correlations between any two works are essentially zero, so that RAV is basically not a factor:

Best Novelette
Total ballots: 616
A: 89
B: 62
C: 61
D: 54
E: 55
F: 38
G: 37
H: 36
I: 35
J: 31
K: 30
L: 28
M: 22
N: 22
O: 20
P: 20

I've copied Cheradenine and put this in a spreadsheet to get the following results:

Depending on the overlaps, the first rule ("make sure you listen to around 60% of the electorate, by continuing until the total remaining ballot weight is under 70%") would choose 9 nominees (up to H). The second rule ("Start with 5 nominees, then try to avoid an almost-made-it") would have actually happily stopped at 5 (up to E). (The possible stopping points are E, I, L, and N). The combination of both rules (1 then 2) would have chosen 10 nominees (up to I). If you use a slightly weaker form of rule 1, where you require a total weight under 75% (that is, "try to listen to at least half"), that would suggest stopping at F, which rounds up to I if you add rule 2 (because of the unusual near-5-way-tie).

Hmm... 10 works sounds like a bit much. So, how about using rule 2, but comparing the gaps (work 5, work n+1) and (work n+1, work n+2). This would mean rule 2 would rarely add nominees beyond the 6th. So combining this with rule 1 at 75% threshold would give 6 nominees that year, with the 6th a marginal inclusion to ensure that at least half of the voters had a say. Work G could complain that it was almost as good as F, but the response would be that F was lucky to get in in the first place, so G can't really argue it "almost deserved" a spot by comparing itself to F.

So I'd suggest that combined adaptive rule: make sure at least 50% of the ballots have at least one nominee, then use the second version of rule 2 above.

OK... that's long enough for one comment, so I'll comment on options 3, 4, and 5 separately.

¹ Actually, both of the authors for the Next Two Rule paper are on the advisory board of my organization, the Center for Election Science. (I hope it's OK to mention the name of the organization. Normally, I'd say the website there; we usually refer to the organization by its website "nickname". But in the last thread I was slapped down for using the website too much. If admins think it's inappropriate to even name the organization, go ahead and disemvowel the name; it will be still basically readable, and I'll get the hint).

#41 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:00 PM:

@31: Generally, the rules I proposed would leave the novel category at around 5.6 average nominees, while something like Short Story might average as much as 7 or 8. That's because voting on novels generally has less of a "long tail". So the rules naturally achieve what you want here.

#42 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 12:02 PM:

If looking at a rule to add more nominees to the ballot, looking at the 2013 short story category may be worthwhile. The vote distribution was

A: 107
B: 38
C: 34
D: 30*
E: 28
F: 28
G: 28*
H: 28*
I: 24
I: 24*
J: 23
K: 21
L: 20
M: 19
N: 19
O: 19
P: 19

Assuming no correlation, we'd need 9 entries to get to 50% of the ballots. But D, G, H, I are likely to overlap because they're by the same author, so we might need to go past 9 to get there.

(This assumes, though, that we don't retain the 5% rule. If the 5% rule is retained, we end up with the 3 nominees they actually had.)

#43 ::: Reimer Behrends ::: (view all by) ::: April 13, 2015, 12:10 PM:

Cheradenine @ 20: Yes, I know (except for categories where the vote is really splintered and few candidates even get 5%). Note, however, that's only for controlling ALL the nominations. It takes fewer votes to control 2-4.

I've also done some quick simulations, SDV seems to be doing reasonably okay in some typical setups, though my tests have been hardly exhaustive.

nathanbp @ 25: I'm saying that this is the point where it becomes pointless trying to make the system even more anti-slate. Moreover, organized voting blocks may then shift strategy away from slate voting towards a different approach that e.g. RAV may be poorly suited to deal with. RAV with exponential or Sainte-Laguë weights really penalizes the last votes on the slate, at which point it becomes more efficient to allocate votes differently rather than just voting a slate. At this point, it becomes more difficult to analyze how well RAV works vs. voting blocs that are flexible enough to adjust their strategies (because they won't bother voting five-nominee slates but pick a different approach). In short, penalties for slates that are too severe may prove ineffective, while also hurting normal voters.

#44 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:13 PM:

@33: Clay, I got this.

One of the leading proposals is reweighted approval voting, which is like reweighted score voting except it's simpler for the nominators. And you're right that "fair" can be ill-defined and some possible definitions are bad, but I don't think anybody here is suggesting anything equivalent to the particular bad definition you give, so you're straw-manning.

The Center for Election Science (again, usually I'd use the web site nickname) is already in this thread, making its points. You're welcome to join the discussion, but helicopter comments are not helpful.

#45 ::: Reimer Behrends ::: (view all by) ::: April 13, 2015, 12:16 PM:

nathanbp @25: To expand on my previous comment, in your example, the slate voters could switch to 1/5th voting for A, B, C, D, E each, securing 20+ votes per candidate, rather than voting the slate, thus working around RAV.

#46 ::: albatross ::: (view all by) ::: April 13, 2015, 12:26 PM:

RAV seems like a nice option, but I am a little concerned about the strategic voting aspects. Suppose I want to nominate A, B, and C, but I expect A to be heavily nominated--I have an incentive to leave A off my nomination, so my nominations for lesser-known works B and C get more weight.

I don't have much intuition for how much this matters--nor whether it matters more in some categories than others. It's not something we can detemine from simulations using previous years' data, without some additional assumptions.

This is the cost of not ranking the nominations, right?

#47 ::: J Thomas ::: (view all by) ::: April 13, 2015, 12:33 PM:

The endgame of Option 5 is that "vote this slate" becomes "download and run our software agent to automatically vote our slate during the last possible tens of milliseconds".

This means that, while the nomination process is in progress, the block voters have perfect information about the behavior of the honest voters, while the honest voters have zero information about the behavior of the block voters. This is a pessimal outcome.

If the block voters are a minority, and they get to watch a majority agree on other winners while they haven't gotten started yet, then they lose.

Unless they think the majority would vote to stop them and not vote with them, they do better to look like a viable choice early on.

If the good guys find choices they can stand behind, a secret group of bad guys cannot compete against them. Unless there are a whole lot of bad guys. (If the Puppies show up next year with 3000 ballots they'll probably win regardless.)

What gives the bad guys their power is that they agree, while the good guys do not agree.

#48 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 12:35 PM:

Reimer @43: The research on RAV indicates that there is no clearcut "best" strategy, and that even given knowledge of the other ballots, determining the best strategy to use to attain a given result is computationally difficult (see this paper). While the method is not strategy-free, the most reliable (and simplest!) strategy is generally to vote for precisely the works that you want to make the final ballot.

Note that RAV is not designed to "punish slates" per se; its intention is to ensure diverse representation in a multi-candidate election. (And I'm not pushing it from any vested interest - I was only vaguely familiar with the system before this conversation started.)

@45: Basically what this point comes down to is that if everyone only nominates one candidate, the system boils back down to approval, which is the case for any of these systems. I think that's unlikely behavior, and in any case, it doesn't really support an argument for any particular system since they all behave the same way.

#49 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:37 PM:
@42: Assuming no correlation,

You mean, "assuming no overlap". In my numbers above I assumed no correlation. (Here's the difference: if two works have 50% coverage each, then with no overlap they'd combine to 100% coverage, but with no correlation they'd combine to 75% coverage.)

It doesn't make a big difference; there would have been 11 short story nominees before 50% saturation (75% RAV threshold) if you presume that all stories are uncorrelated. Interestingly, you'd also get 11 nominees if you assume D, G, H, and I are maximally correlated (in which case, the only one of those that wins is D).

#50 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 12:41 PM:

@49: Thanks for catching that. I changed methodology mid-sentence and didn't spot it on review.

#51 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:42 PM:

@48: That is a pretty good explanation of strategy under RAV. There's just one thing I'd add. If, as @46 suggests, you like A, B, and C, and expect A to win without your vote, then the simple (imperfect, but generally good) strategic rule of thumb is to vote for A if and only if you prefer it to all other candidates. So there would be some people leaving off a popular work if they have lesser-known works they like better, but it's not generally a good strategy to leave off a popular work if it is actually your favorite. That latter strategy can work in some rare cases, but it's generally not worth the risk.

#52 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 12:45 PM:

Clay Shentrup @33:

1. We don't do .sigs here. Please leave that off.

2. We tend to look askance at commenters whose very first comment contains a link to another site, especially if it's connected to a business they run.

3. This thread continues the conversation started in the previous thread. "It should be fair" is one of the stated requirements of the voting systems we're discussing. It is not your place to abruptly show up and announce that we're having a different conversation, one whose terms you propose to define.

We are not talking about social choice theory. We are certainly not talking about whether a rational organism's goal is to maximize its expected utility.

The turf you're standing on belongs to science fiction fandom. We're talking about how our worldcons conduct the voting for our Hugo Awards. If you want to join us in respectfully discussing that subject, you're welcome to stay. If not, I'm sure you'll be much happier elsewhere.

#53 ::: J Thomas ::: (view all by) ::: April 13, 2015, 12:50 PM:

@33 Clay Shentrup

> It should be fair.

This is a basically meaningless desideratum frequently referenced by people not versed in social choice theory. Suppose we can select option X, which will give Bob and Alice utilities of 5. Or we can select option Y, which will give Bob a 6 and Alice a 7. Do you really want to choose X (thereby making Bob and Alice BOTH less happy) in the name of "fairness"?

Sometimes. It depends.

If Alice feels strongly that she does not want Bob to benefit *more*, Bob might agree to a more even result that's worse for everybody. This happens in real life. There's a debate going on in the bigger world about equality. I will parody it:

"In today's world we are all better off. I have an extra hundred billion dollars, and you have a better computer than you could otherwise have. Because of the Iron Laws of Economics, if anything happened that made me worse off, you would be worse off too. So let's all be happy with our lives, and if you just work harder and smarter you can be rich like me."

"It isn't fair and I'm not putting up with it."

People bring meaning to concepts. You might find no meaning in fairness, but others create meaning for it.

Is this off-topic?

#54 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 12:51 PM:

Jameson @44, Th Cntr fr lctn Scnc is not here. Jameson Quinn and Clay Shentrup are here.

#55 ::: albatross ::: (view all by) ::: April 13, 2015, 12:54 PM:

J Thomas:

I am curious how much non-slate voters agree in their nominations. My best guess is that people mostly tend to read clusters of related books--one person may read mostly MilSF, another may read mostly alternative history, still another may prefer urban fantasy. I'd expect that to lead to correlated nominations. Also, there are probably correlations between favorite authors (based partly on categories, but also on when the authors started publishing, political or social ideology, style of writing, hardness of their SF, series vs standalone works, etc.)

This matters because anything that decreases the weights of slates is likely to also decrease the weight of these correlated nominations. Suppose we have RAV with exponential weights, and we have a year where there are two really strong space opera choices. It would be easy for the less initially popular one to get cut out of the nominations by being in the shadow of the other one. That might be good or bad in a given year, but it's a potentially important change in how the nominations will work.

#56 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:55 PM:

I took a look at the 2014 short story nominee list — the puppy bête noire — and I think RAV might have "helped" there too. I mean, I actually liked "Dinosaur" and "Selkie" and "Water That Falls"; but these are three Escape Pod/ Podcastle stories that are all arguably more experimental from a literary point of view, less hard-science, and more "politically correct" in their themes than the "average" story in the genre would be (if such a thing really existed). So you don't have to posit any shadowy cabal to imagine that the voting for those three stories was correlated. Which means that, with RAV, "Selkie" (which is actually my favorite of the three, but whatever) probably wouldn't have made it. And then maybe there would have been less impetus for the puppies to go ape this year...

#57 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 01:04 PM:

@52: Clay is a co-founder of the Center for Election Science, and we respect him as such; but he is not a current board member, and has never been the executive director, so it's not exactly a "business he runs".

@54: Point taken. I thought my comment was just saying "welcome to join the conversation, but we don't need the talking points".

#58 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 01:06 PM:

Cheradenine @ 37:

Considering that you and Bruce have stated your support for a system that would knock out works with absolute majority support, I would be more comfortable if you just added a proviso that a work that appears on an absolute majority of ballots always makes it to the final ballot. If a single bloc manages to produce an absolute majority of the votes, then so be it, they win.

This is not far-fetched when we consider that a competing bloc to the Puppies might emerge and we end up with a two-party system. If one party dominates the votes to the point of having an absolute majority, they should reap the rewards.

#59 ::: J Thomas ::: (view all by) ::: April 13, 2015, 01:06 PM:

@55 albatross

I am curious how much non-slate voters agree in their nominations. My best guess is that people mostly tend to read clusters of related books--one person may read mostly MilSF, another may read mostly alternative history, still another may prefer urban fantasy. I'd expect that to lead to correlated nominations. Also, there are probably correlations between favorite authors (based partly on categories, but also on when the authors started publishing, political or social ideology, style of writing, hardness of their SF, series vs standalone works, etc.)

Yes. But the difference between people who vote for the same nominations because those are the ones they like, versus people who do it because they have an evil plan to dominate the nominations, is subtle.

You can't tell the difference from the ballots, you have to get stuff from their websites or listserves or emails where they are plotting. Or maybe you can just tell.

When the purpose is to reward diversity and independence, then when people vote in lockstep it doesn't matter whether their intentions are evil. We should punish them the same.

So if a bunch of people vote for the same 5 space operas or the same 5 feminist stories, they must be treated like evil slates. We will get more diversity if we give their opinions less weight when they agree.

Suppose we have RAV with exponential weights, and we have a year where there are two really strong space opera choices. It would be easy for the less initially popular one to get cut out of the nominations by being in the shadow of the other one. That might be good or bad in a given year, but it's a potentially important change in how the nominations will work.

Sure, but it only works that way if the same people vote for both. If the people who like one space opera hate the other one, then the weaker one will get its full chance to be nominated.

It's the usual strategic rule -- if you want nominee X to win, don't also vote for its closest competitors.

#60 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:07 PM:

@56: It actually looks to me like "Water hat falls" (the eventual winner!) had the fewest nominations of the 3, so if one of them was hurt, it would probably be that one. It's hard to say without full ballot info, though.

#61 ::: dh ::: (view all by) ::: April 13, 2015, 01:08 PM:

When the purpose is to reward diversity and independence, then when people vote in lockstep it doesn't matter whether their intentions are evil. We should punish them the same.

So if a bunch of people vote for the same 5 space operas or the same 5 feminist stories, they must be treated like evil slates. We will get more diversity if we give their opinions less weight when they agree.

The problem is that punishing slate voters gives enemies of a particular work or author or ideology a new weapon to fight with.

#62 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:11 PM:

Fred @58: We may have to agree to disagree here. I actually don't think it's appropriate to allow one group to completely domination the nomination pool. If 55% of people prefer SF and 45% prefer fantasy, is a list of 5 SF nominees most representative? I would tend to say no.

(OK, i would actually say that if we're in a situation where 5 works are all receiving an absolute majority of nominations, the Hugo nomination system is irretrievably broken anyway.)

#63 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 01:12 PM:

J Thomas @53, fairness and the perception of fairness must be on-topic. They're right there in Bruce Schneier's initial post.

I've helped administer TAFF, which is also a large idiosyncratic fannish institution that holds elections. Fairness and the perception of fairness are both essential in fannish systems. Of the two, losing the perception of fairness is the blunder that'll kill you faster.

#64 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:13 PM:

dh @51: Is a Hugo ballot consisting of five space operas ideal?

#65 ::: AD_FV ::: (view all by) ::: April 13, 2015, 01:28 PM:

In support of option 3d, the single transferable vote: Of the choices presented above, only a change in the voting system will have the desired impact. The primary flaw with the current Hugo nominating process is that it uses winner-take-all block voting, under which a cohesive plurality of the electorate can determine the winners, leaving the rest unrepresented.

The choice of particular voting system will always involve some compromise, but option 3d, the single transferable vote, provides an accurate and fair representation of the electorate while avoiding many of the pitfalls and compromises inherent in other systems.

Approval voting systems violate the later-no-harm principle, as a voter approving of more than one candidate may be harming the chances of a more preferred candidate, incentivizing voters to vote tactically rather than express their true preferences. More on concerns about approval voting and the comparative merits of STV can be found here.

Use of STV with rankings ensures a proportional outcome, eliminates the spoiler effect, and is immune to tactical voting, allowing voters to express their true preferences. It will produce a proportional and representative outcome regardless of the existence of slates or organized factions. It is currently used to nominate films for Best Picture at the Academy Awards, and is the natural counterpart to the instant runoff system used to select the Hugo award's ultimate winner.

The experiences of cities that have adopted ranked choice voting for single and multi-winner elections, like San Francisco, Oakland, Minneapolis, and Cambridge, MA, shows that voters are very likely to make use of the ability to rank candidates and understand and adapt to a ranked system. The same would certainly be true of Hugo award voters.

#66 ::: Seth Gordon ::: (view all by) ::: April 13, 2015, 01:32 PM:

Even in the absence of deliberate slates to defend against, I don’t think it would be an awful thing if a final Hugo ballot ended up with the most popular military SF novel of the year vs. the most popular feminist SF novel of the year vs. the most popular steampunk novel of the year and so on. OK, it would disappoint the author of the second-most-popular military SF novel of the year, but if I understand the voting algorithms correctly, the rocket would end up going to someone who not only appealed to one particular constituency, but was highly regarded in other constituencies as well.

#67 ::: J Thomas ::: (view all by) ::: April 13, 2015, 01:32 PM:

@63 Teresa Nielsen Hayden

Fairness and the perception of fairness are both essential in fannish systems. Of the two, losing the perception of fairness is the blunder that'll kill you faster.

I tend to agree. When you lose the perception of fairness, people who lack a personal relationship with the guys on top may start to vote with their feet etc.

If it's objectively unfair but people don't notice, then they suffer whatever consequences come from that. Since I don't know how to tell what's objectively unfair or what consequences come from it, I'm more likely to notice the perception of unfairness and write off the actuality of unfairness as bad luck.

#68 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 01:33 PM:

Jameson, use "politically correct" like that again and you will lose your vowels. So will anyone else who uses it. I am not kidding.

The casual political heterodoxy of the SF community has to be experienced to be fully appreciated. We will cheerfully hold intelligent, respectful political conversations where the participants include monarchists (early medieval model), extropians, neoplatonists, moderate liberals, Labour, LibDems, Parti Québécois, EU Greens, beleaguered US centrists, and six different strains of libertarian.

Anyone who thinks "pltcl crrctnss" is a significant literary criterion in our field is short on information, or has been fed bad information.

#69 ::: albatross ::: (view all by) ::: April 13, 2015, 01:38 PM:

Re nominating strategy:

ISTM that part of what we're wanting here is for it to be sensible behavior for someone to say "Of the new books I've read this year, I'd say A, B, and C were plausibly Hugo-worthy" and then nominate A, B, and C. Since most people probably haven't read everything worthy of consideration for an award, the nomination is an attempt to get lots of people to propose things they think might be worthy. But that does clash a bit with RAV, because now the same nominator should also think "Hmmm, I think A, B, and C are plausibly Hugo-worthy, but A is a heavily-marketed work by a popular author, so maybe I should just nominate B and C."

Re fairness:

In political elections, it's hard to opt out of the results if you think they're crooked or fixed. But most other things are voluntary. If you conclude that the federal government has lost legitimacy, you still have to pay your taxes and the FBI can still arrest you. If you conclude that an organization you belong to has lost legitimacy, you can just stop paying dues and attending meetings--enough of that, and the organization will fall apart.

#70 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 01:40 PM:

Cheradenine @ 62: It looks like your preferred system would solve the biggest issue of this year, that a tiny minority managed to get all the nominees in many categories. But it still wouldn't, by itself, stop that tiny minority from getting 1-3 nominees in every category, right? So as long as they can continue to get nominees, there's little incentive for that minority to stop blocing. As long as everyone else is not organized they can still get a win, even if it's not as big as this year's win.

I would then expect a large group of people to organize in response, and to form a significantly bigger bloc.

But guess what? Even if the puppies are a tiny minority bloc up against a much bigger bloc, they *still* get at least one nominee on each category in the ballot, because now under 3c the bigger bloc eats its children and the top minority bloc candidate can squeak in.

I think you should give more consideration to what happens after people absorb the lessons of this year. Assume that blocs are here to stay. Do we want to ensure that each bloc gets at least one nominee per category, or not?

#71 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 01:42 PM:

@68: I understand and take your warning. In my (somewhat irrelevant) defense, I was using the term in explicit scare quotes, to stand for what I believe a "puppy" would say. As I said, I personally thought that all three of the works in question were above average, and that "Selkie" was outstanding.

#72 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:44 PM:

@65: Apparently everyone will now drop by this site to make the case for their favorite system without reading the threads or considering the details of the situation in question.

STV is a good system (imo) with flaws like all voting systems. A number of your statements are hyperbolic or incorrect (STV neither eliminates the spoiler effect nor is immune to tactical voting, per the Gibbard-Satterthwaite Theorem.) It also cannot guarantee a proportionate outcome, particularly in a case where not all voters rank all nominees - which, as noted repeatedly in this discussion so far, is impossible for the Hugo nomination election.

#73 ::: albatross ::: (view all by) ::: April 13, 2015, 01:47 PM:

Re number of nominees:

It seems like there's a natural limit to how many nominations can be accepted, based on how much time people will have to read the nominated works. I wonder if it would make sense to allow some rule to expand the nominations but only within a narrow range--say, the default is five, but if the next highest nominee is close enough in some sense you can go up to six or even seven nominees.

#74 ::: Andrew M ::: (view all by) ::: April 13, 2015, 01:50 PM:

When the purpose is to reward diversity and independence, then when people vote in lockstep it doesn't matter whether their intentions are evil. We should punish them the same.

I think there is a serious issue here which it would help to be clear about. There seem to be two views at work in this debate. On one, the Hugos have worked pretty well up to now, and the new practice of slate voting has disrupted this; the aim is to restore the Hugos to something like their historic way of working. On the other, the aim is to improve on the Hugos as they have been; there is a feeling that they are not sufficiently diverse, and a modification of the system would make them more so.

Are the Hugos diverse? Well, I think there is more than one kind of diversity, and one kind may be the enemy of another. Clearly, Hugo nominees are not all the same kind of work - they can be incredibly different. But they don't reflect the full range of the field. There seem to be two factors which tend to make a work a Hugo nominee; one, which I mentioned in an earlier thread, is that they have, or at least might be imagined to have, cross-group appeal, rather than being in the core of a specific subgenre. The other, which someone else mentioned, is that they have a kind of uniqueness, rather than just being typical of their author. I think that these are good qualities for nominees to have; they help to pick out the most distinctive and significant work of the year; they mean that the final ballot does not consist of five works each of which is loved by 20% of the voters and hated by the other 80% [this is a rhetorical exaggeration], and that the winner, though not everyone's favourite, is not just the 'least hated' but has fairly wide support.

I think some people are assuming that if a lot of people vote for the same five works (not as part of an organised slate), this will be because they are all similar works - as dh says, five space operas or five feminist works. But I think it's quite likely that a fair number of people may vote for the same five works because they want to reward diversity and independence - because those works, diverse in nature, are the ones that stand out as significant. None of us knows what three works were knocked off the Novel ballot by the puppies, but I think we could name six or eight works and say with some confidence that the three missing works were among them; and a lot of ballots will have made their picks from among those works. I'm afraid that if the voting system positively rewards difference, we will end up with a duller set of nominees - the epic fantasy nominee, the urban fantasy nominee, the MilSF nominee and so on.

One other thing to bear in mind - I think this harmonises with some things that Brad Templeton has been saying - is the effect of the award as a recommendation. The voters are not the only beneficiaries of the process; we are sending a message to the wider world, about the most significant things in SFF. From the voters' point of view, it may be fair that clumped preferences should have less weight, so as to give some representation to more people. But if we are sending a message to the wider world, I think we should be telling them about the works which have the most support, not leaving things out because those who like them like a lot of the same other things.

#75 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 01:51 PM:

Here's an example of what might happen once voting gets heavily bloc-ed up:

infernokrusher bloc: 65% of the vote
grimdark bloc: 15% of the vote
puppies bloc: 10% of the vote
non-bloc: 10% of the vote

If we use 3c/exponential weighting, I think infernokrusher gets 3 spots on each category, and grimdark and puppies each get 1. Is that what we want?

#76 ::: JonW ::: (view all by) ::: April 13, 2015, 01:53 PM:

3c3 looks appealing to me (as it does to a bunch of folks here), but I don’t pretend to say that from a position of technical sophistication.
A word on banning slates: Douglas @#35 explains the problem here, so I’ll just speak to the weaker suggestion of an honor pledge that one hasn’t “voted a slate,” per #17. That’s still a problem unless we can adequately define “voting a slate,” from the perspective of the individual nominator, so that nominators know what it is they’re pledging that they haven’t done. But I don’t think we want to tell Some Jane With A Blog that it’s against the rules for her to publish a list of the five novels she was most excited about this year, or to tell Some Joe Who Reads A Blog that it’s against the rules for him to be inspired to read those novels and really like them and decide to nominate them. The best we can really do with an honor pledge is to make people pledge only to nominate works they’ve actually read or viewed (which wouldn’t do much, because folks inclined not to comply wouldn’t be deterred). The beauty of 3c3 it that it would adequately address the ways that slate-driven nominations tend to break a FPTP nominations system, with minimal downsides, without telling people that certain sorts of enthusiasms and voting behavior are forbidden.

#77 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:59 PM:

Fred Bush @70: If there are competing blocs, I do think that each of them being represented on the ballot, and popular non-bloc works having a shot, is a better outcome than the larger bloc running the table (as will happen under the current system). In part this is because I see the ideal outcome of this election as providing Hugo voters a range of choices which is as representative as possible of the range of Hugo voters as a whole. So "avoid one group completely controlling the ballot contents by collusion" was a high priority in making my recommendations.

I think that STV would provide a slightly higher chance of a smaller bloc being shut out entirely, but due to the dispersion of Hugo nominations I'm not sure it would work. In a situation like the short story nominations this year, the bloc would win because no story would ever accumulate enough votes to exceed the bloc's votes or reach the quota. Balloting behavior might change in response to the system change, but I don't know if that would help here.

Voting system tweaks can keep blocs from taking over the ballot, but I don't think they can provide a strong disincentive to bloc voting. If we want to make blocs less powerful than single voters, a change like Option 4 would be necessary. (And as I noted above, I don't think enforcing that is viable.)

#78 ::: John A Arkansawyer ::: (view all by) ::: April 13, 2015, 02:06 PM:

Fred Bush @ 75: I can haz moar infernokrusher pleez?

JonW @ 76: The honor pledge against having voted a slate and the honor pledge that one has read the works which one is nominating have two good qualities in common:

1) They appeal to the better angels of people's nature.

2) They are imperfect but work well enough.

That last is so important. A perfect solution (should there be one) is gameable or authoritarian, in perception surely and in fact probably. Imperfection is a feature, not a bug.

#79 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 02:11 PM:

AD_FV@65: Sigh.

It is simply untrue that STV "is immune to tactical voting". The Gibbard-Satterthwaite theorem shows that no voting system is. (Dictatorships can be, but that's not voting, is it?)

In particular, all proportional systems, STV included, can in some cases reward a voter for leaving off a candidate that they prefer, but that can win without their help.

"Later no harm" is not, in my opinion and that of many voting theorists, a desirable characteristic in a voting system. It is incompatible with other, more-important criteria such as participation, consistency, monotonicity, Condorcet, and favorite-betrayal-proofness. There is some arguable benefit to a post-hoc, strategic later-no-harm-like criterion, such as a continuous median system would evince. But not to full out LNH.

...

I'm really sorry to have to do this, but I'm going to have to air some dirty laundry.

The "FV" in AD_FV refers to "FairVote", an election reform advocacy organization. (I also recognize the initials before the underscore.) FairVote promotes IRV and STV. FairVote is older than my organization, the Center for Election Science (CES). The CES was founded a few years ago to promote voting systems that are in line with "recent" (since the 80s) advances in voting theory. Prior to founding the CES, most of the people involved had attempted to convince FairVote to reduce its emphasis on IRV and embrace newer methods. Of course, I'm not going to claim that "my side" never overstated our case, or that we did everything right. But the outright refusal of FairVote to listen to us, and their continued use of what we view as outright falsehoods in promoting IRV (similar to the falsehoods about STV in AD_FV's comment; STV is a good system, and I'd support it here, but there's no need to lie about it), led us to be disenchanted with FairVote, and was part of the impetus for founding the CES.

I have been upfront about my own affiliation in this and the prior thread. And I honestly believe that there's not much "ulterior motive" in my suggestions here. RAV is not a system that the CES strongly promotes, unlike the case with STV and FairVote.

So, now you know. You can discount both my and AD_FV's statements accordingly.

I do NOT intend to dox anyone here, and I believe that I have not done so. If any moderators disagree, I encourage them to delete everything after the ellipsis. If any of this is over the line, then it would still be so after disemvowelment or rot13.

#80 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 02:12 PM:

Andrew M @74: That's a very sound point. I'm not quite convinced that uniqueness and cross-group appeal have typified the Hugo nominees over the past few years, but that's a subjective assessment. I think that even if a lot of people are selecting from the same list of 8-10 books, the effect size is likely to be pretty small. If we could agree on some parameters for what we think the distribution might look like, I'm happy to sim it. (Though again, real data would be the Holy Grail here.)

#81 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 02:18 PM:

For what it's worth, as far as I know, Cheradenine is unaffiliated with either CES or FairVote. More importantly, their comments seem knowledgeable yet entirely unbiased. My intention is that my own comments should be unbiased, too, of course, but that's not for me to judge.

#82 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 02:24 PM:

(As weak evidence for the assertion that I'm not biased by my CES membership: note that when my fellow-CES-member Clay came by, his proposal was different from mine.)

#83 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 02:36 PM:

J Thomas @67, my only disagreement is that there are no people on the top. The worldcon is run by fans, not pros, and every worldcom committee is different.

#84 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 02:45 PM:

Jameson @71: I understand; but you know, sooner or later someone's going to think you mean it. I'd rather keep that mess from happening than try to clean it up after the fact.

In any event, it really was intended as a general warning, and everyone should understand it as such. (Unless they're Bruce Schneier. If he wants to use it, he can.)

#85 ::: rcade ::: (view all by) ::: April 13, 2015, 02:50 PM:

Cheradinine @ 7: Option 4: Banning Slates. I can see two ways to do this: lay out in exacting detail what a slate is (which means that people will immediately game their way around the definition), or leave identifying slates up to the Hugo administrators, which is a responsibility I suspect most concoms do not want and are not prepared for.

I think there's a third way: Worldcon can adopt language to discourage slates without specifying any enforcement, such as something like what I proposed earlier: "3.7.4: Members should not vote for nominations by copying any slate of nominees suggested by others, but instead should make their own individual choices for what they believe are the best works."

Without enforcement, the purpose of such language would be to take away the argument that slates are within the rules. Any Worldcon member who launched one would be in direct contravention of the WSFS Constitution, which fans would no doubt remind that person.

To the people who favor "do nothing" as an option, I'd like them to consider at least finding some means of expressing the idea that Worldcon strongly opposes bloc voting.

#86 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 02:54 PM:

Jameson again, @79: Thank you for the explanation. I think we could all tell something was going on. A brief explanation is less distracting than mysterious scuffling noises ongoing behind the arras.

Also, what you did there wasn't doxing. You were giving us relevant identifications of some participants. We'd have had a lot more trouble sorting that out on our own. Not that we couldn't have done it, if we'd felt motivated; but it's better to not get distracted.

Now let's all go back to talking about the mechanics of voting.

#87 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 02:54 PM:

rcade@86:

"3.7.4: Members should not vote for nominations by copying any slate of nominees suggested by others, but instead should make their own individual choices for what they believe are the best works."

I think we should do this no matter what method we eventually decide to use. It does no harm, might do some good, and certainly expresses what the Hugos are supposed to be about.

Kilo

#88 ::: Joshua Kronengold ::: (view all by) ::: April 13, 2015, 02:54 PM:

As should be obvious, I support RAV (which was my first thought wrt slates, although I didn't know it had a name), or SDV (which I came up with while cogitating on an easier to explain and similar outcome equivalent of RAV). There are corner cases where they have different outcomes, but they are both fundamentally purportional representation systems that try to avoid a dedicated plurality controlling multiple slots in nomination. (I'd like to see how SDV does with competing slates -- my instinct is that the larger slate will tend to knock out the smaller one (if everyone votes party-line; if they don't then one smaller slate work will likely make it)--and then still not get everything it wants on the ballot.

Some considerations:

How do we ease the impact on administrators of nominees declining nomination.

How do we coherently display the results of the election in a way that's comprehensible to the average fan?

My guess -- since both loser-first and winner-first systems (which is the central difference between RAV and SDV; you could even have SDV act more like high-weight RAV systems by having ballots give, say, 1/2**(N-1) or some other calculated divisor to their candidates rather than 1/N) have unstable results -- that is, that elminating one candidate before election can result in substantial cascatding changes to the results, that any adoption of a similar system would also need to discuss how refusal would happen. The choices would be:

1. When a nominee declines, offer the slot (if any; it's possible that there are more than 5 nominees at that point) to the 6th place candidate.

2. When a nominee declines, re-run the calcualtion as if they were not present (eliminating them from all ballots), then offer the slot to the highest ranking candidate who is not already on the ballot.

3. Only ask people one at a time, so the results can be recalculated in full if people decline.

3 can obviously be dismissed as far too great a burden on Hugo administrators.

Of the remaining two, the first is by far easiest to explain. But I'd have to prefer the second -- because it would be most likely to maintain the goal of proportional representation, rather than potentially eliminating a bunch of people's second pick on the ground that we were taking their first pick. It might sometimes add a page to the newsletter (or just extend the web page, since at least last year we didn't bother with paper vote breakdowns IIRC) to summarize the runnoff elections, but that's worth not disenfranchizing people.

Regarding the second question, we've had a bit of attention paid to it, but I'd guess for RAV we'd want:

Round 1:
Candidate 1 selected (XXX votes)

Remaining top 15 candidates:
Candidate 2: XXXX votes at full value, YYYY votes with one successful candidate
Candidate 3: XXXX votes at full value, YYYY votes with one successful candidate
...
Candidate 15: XXXX votes at full value, YYYY votes with one successful candidate

Candidate 2 selected as winner.

And then similar breakdowns after the selection of candidates 2, 3, and 4 (none needed after 5, of course).

For SDV, I'd probably instead start after all below threshold candidates are eliminated, eg:

(after 400 candidates are eliminated for not having enough nominations to make the ballot):

Candidate 1: 50 votes in 200 ballots
Candidate 2: 35 votes in 89 ballots
etc.

I think SDV would require more elision or more space to handle, since it goes through many more steps prior to selecting winners (ie, eliminating candidates one at a time rather than 5 steps of approving them).

Question re RAV: The assumption seems to be that rank with RAV still matters, right--just only in tie situations? If not, you end up with slates being able to break it by exactly lining up their votes so the tie cannot be broken. This was the other reason I came up with SDV -- because it seemed to produce decent results without having to change the balloting system. Mind, ties are only a theoretical issue--we won't know until August who well SP/RP managed voting discipline (my guess, given that the louder parts say they didn't vote for the whole slate, is that they had at best 80%-90% discipline--enough to dominate the nominations but not enough that ties were that much of an issue). But it's good to have reasonable behavior on ties.

So difficulty in brevity explaining runoffs for SDV is a substantial point against it relative to RAV, whereas being able to cleanly avoid having rank matter at all (rather than just in ties) is a point in SDV's favor.

#89 ::: J Thomas ::: (view all by) ::: April 13, 2015, 02:58 PM:

@64 Cheradenine

Is a Hugo ballot consisting of five space operas ideal?

It depends.

Imagine that the nominations were done with approval voting, and the top 5 space operas got 55%, 54%, 53%, 52%, 51%.

It was just a space opera year and a lot of people really liked them that year.

Meanwhile the top candidates in other genres were

10% steampunk
9% cyberpunk
8% SF Regency Romance
7% knights and unicorns
6% Piers Anthony
5% military SF.

We would get a more diverse nomination to take one from each genre, but probably the space opera would win the election. If we had 5 space operas we'd find out which one of them won.

On the other hand it's a different situation when it's 55% of the nominators who like space opera, versus every single nominator voting for 2 or 3 space operas and other things too.

If everybody likes space operas then maybe that's what they want. But if it's only a small majority maybe we should give the minorities some of what they want too. But then on the third hand, if the minorities are all over the map and don't agree about anything, how do we decide which of them to throw some nominations to?

So my own preference is to let the voters see what's going on and make choices themselves. Maybe they will know better what they want than we know for them. On the other hand, future voters might have bad intentions, while we know that we don't have bad intentions now. If we make mistakes they will be honest mistakes because we can't predict what will happen, but the more power we give to future voters the more chance they will misuse it. On the third hand, we really don't know what they'll need and we are likely to make mistakes now.

#90 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 03:02 PM:

Joshua @89: What am I missing? It seems like SDV would potentially run into ties as well if we hit an elimination step with multiple tied candidates.

I was thinking on the way in about dealing with RAV ties. I'd be tempted to break them by which candidate appears on the highest number of ballots; if they're tied there too, then things get messier. (If the tie is for the last space, we can just take both candidates. The issue is if the tie is for second place, but picking one of the tied candidates would knock the other one down in the order.)

#91 ::: BillDoor ::: (view all by) ::: April 13, 2015, 03:23 PM:

No nominations.

When you register you get a (paper) ballot. Hand it in at the convention (appropriate mail-in arrangements, just like IRL elections). (Look to Oregon for a model)

Enter author & title (or equivalent identification)

Every vote is a write-in, including No Award.

Last day of of the con, see who's won.

#92 ::: Andrew M ::: (view all by) ::: April 13, 2015, 03:25 PM:

Cheradenine@80:

I'm not quite convinced that uniqueness and cross-group appeal have typified the Hugo nominees over the past few years, but that's a subjective assessment.

Well, I think it's a tendency; it's clearly not universal. There are a couple of writers of non-unique books who do well; and as for subgenres, borders shift. (I think you could say that GRRM, for instance, when he started out, was breaking the borders of his subgenre, epic fantasy, but now a new subgenre has formed around him.) I think it does explain some things, though. You do get occasional complaints - not only from puppies - that the Hugo nominees are all the same, and while that's clearly not true, this can help to explain why they don't cover everything.

#93 ::: Buddha Buck ::: (view all by) ::: April 13, 2015, 03:27 PM:

BillDoor @92:

I don't believe that particular method is one of the ones our hosts are entertaining in this thread. You can see the specific changes we are discussing listed at the top of the page.

I see you've never posted here before. Welcome. Do you write poetry?

#94 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 03:30 PM:

JThomas @89, if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, the people of fandom have spoken.

The Hugos reall are about what the Hugo voters like. This worthy principle keeps us out of fights to the death over the difference between SF and fantasy, or whether technothrillers are SF, or whether "YA" on the package makes a book ineligible.

And don't assume the space opera would win. I learned long ago to distrust statements like "Of course (Title) is going to win." Not only does (OtherTitle) win too often to be ignored, but when the voting breakouts are released after the ceremony, they may show that it had the lead all along; i.e., was a genuinely popular choice.

#95 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 03:33 PM:

JThomas @89, if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, fandom has spoken.

The Hugos really are about what the Hugo voters like. This worthy principle keeps us out of fights to the death over the difference between SF and fantasy, or whether technothrillers are SF, or whether "YA" on the package makes a book ineligible.

And don't assume the space opera wins. Not only do less obvious works win too often to be ignored, but when the voting breakouts are released after the ceremony, they may show that the winner had the lead all along; i.e., was a genuinely popular choice.

#96 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 03:40 PM:

Andrew M @92: Do the people who think the Hugo nominees are all the same actually read them? I am having real trouble with this concept. Where on the map of subgenres does Among Others belong?

#97 ::: J Thomas ::: (view all by) ::: April 13, 2015, 03:56 PM:

Here is my current concern with RAV.

In the nomination I vote for A, B, C, D, E. I think A is the best and I really care about it, but if A doesn't win I want B. C is pretty good. D is OK too. E, meh, I can't think of a better fifth choice.

But then it turns out E wins handily. My other four choices get recalculated as half votes. And then D wins. My other three choices get recalculated as quarter votes. And none of them win.

I didn't care that much about D or E in the first place, but the result of putting them on the ballot was that my vote for A got turned into a quarter of a vote.

Given my druthers, I'd rather be able to cast a full vote for A, half a vote for B, a quarter vote for C, etc and then if A loses I give B a full vote, C a half vote, etc.

I guess. Maybe I'd rather give A and B equal weights, it depends.

It's all so complicated.

I can see the reasoning. After I win something I don't deserve to have as much influence. Give somebody else a chance.

IRV lets you influence *just one*, but you get to pick which one. You can only vote your second choice if your first choice loses. You don't get penalized if your fifth choice wins before your first choice has lost. IRV has its problems, but maybe some of that is that it was a front-runner for so long that people have had a lot of time to figure out the problems. These newer approaches look good because we haven't had as long to knock them down.

#98 ::: J Thomas ::: (view all by) ::: April 13, 2015, 04:08 PM:

@95 Teresa Nielsen Hayden

if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, the people of fandom have spoken.

Yes, that's how I think it ought to be.

But as I understand it, by some of the new rules we're discussing, if a majority of fans each nominate the same five space operas, then we pass the one with the most votes and we reduce their votes for the second one, and reduce the votes more for the third one, and with enough reduction in votes some others that got fewer votes will get on the ballot.

Maybe space operas that got fewer votes but that got votes from *other people*.

Say that the best space opera got 52% of the votes, and the second-best got 51%, and the overlap was 3%. Then they both get nominated.

But if everybody who voted for the second-best also voted for the best one, then it doesn't get 51% of the votes. It gets 25.5% of the votes. Anything that gets 26% will beat it.

It isn't about how many votes it got. It's about who voted for it.

if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, the people of fandom have spoken.

With this proposed rule it isn't about whether it gets the most votes. It has to get the most voters who didn't also vote for something else. It's complicated.

#99 ::: J Thomas ::: (view all by) ::: April 13, 2015, 04:12 PM:

@96 Teresa Nielsen Hayden

Do the people who think the Hugo nominees are all the same actually read them?

Some years ago I read about a Chinese convenience store in San Francisco that got robbed by a white man. The clerk could not give the police a description. She said they all look alike.

It's like that.

#100 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 04:41 PM:

Something to think about...

Whatever system we choose, it needs to be easily explained to the Hypothetical Average Hugo Voter (tm). As an astrophysicist, I've got a pretty good background in statistics, so while the voting theory aspect is new to me, I think I'm understanding fairly well. I'm not sure we can say that about our HAHV. You only have to look at all the confusion surrounding the "no award" process this year to see the problem -- and the "no award" process is a much, much simpler process than most of, for example, the option 3 variants.

I'm not saying we should avoid complex voting systems. But whichever system we choose, I think we need to come up with a simple "mission statement" for that particular system, along with some examples that illustrate why that system fulfills the mission.

As a completely made-up template (and not one that is explicitly intended to match any particular system), a sample mission statement might read something like, "Ideally, Hugo voters should have one favorite work that they want to win, and failing that, some backup candidates that they wouldn't mind winning. As a result, your first choice is weighted the most, and if your first choice ends up being successfully nominated, then your other choices will be counted less -- in essence, you've had your turn to nominate, now let someone else propose something."

Is it possible to define a similar mission statement for all of the options under consideration? If not (though I personally think we can), I think think we should seriously consider not using such a system. If the HAHV can't intuitively see how the process works, that's going to tend to engender mistrust in the system, which undermines everything we're trying to do here.

Is anyone up to the challenge of writing a mission statement for each option now? It might help focus our discussions.

Kilo

#101 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 04:51 PM:

@86: Thanks for clarifying.

One more bit of dirty laundry; the last, I promise.

FairVote and CES agree on a number of things. We agree that plurality is a horrible system, with distortionary and pathological effects wherever it is used. We agree that proportional representation is a good idea in most multi-winner contexts. We agree that you also need some form of single-winner reform, since not all elections are or could be multi-winner. We agree that no single-winner system is ideal in all respects; there are tradeoffs involved, depending on what characteristics you value.

We even agree that arguing about which single-winner system to propose is counterproductive and makes us all look like the Judean People's Front ("splitters!"). Unfortunately, we have not yet found a way to agree on how to stop arguing.

I'm going to try to present a fair guess at what I think 4 hypothetical people would say. You can probably guess who they're supposed to stand for, except the last one. There is no hidden meaning in the names besides the most obvious pun.

IrishWhiskey_CES: The things that matter about voting systems are their strategic properties and their outcomes in realistic scenarios, as well as their simplicity, expressiveness, and auditability. There are a number of good systems; good single-winner systems include approval, Condorcet, majority judgment, score, and SODA, while good multi-winner systems include BTV, PAL, RAV, RRV, SAV, and STV. Unfortunately, IRV has enough problems with outcomes, simplicity, and auditability to make it worse than other single-winner systems. If we want to stop arguing, we should settle on the simplest proposal, and the one which has the fewest downsides. That means approval voting is the first step.

Dirt_CES: The main thing that matters about a voting system is its outcome from a utilitarian perspective. It is possible to measure this numerically using monte-carlo simulations (yielding numbers called "voter satisfaction efficiency" or "bayesian regret"), and if you do enough such simulations, questions of strategy become moot. The best single-winner system is score voting, and approval voting is an acceptable as a first step towards that. By analogy, RRV is probably the best multi-winner system. IRV is basically a step in the wrong direction.

SUM_FV: The main thing that matters about a voting system is whether it can be passed as a reform. This in turn depends mainly on three things: how much of a track record it has in political use, how little strategy it encourages from a coordinated/elite perspective, and how well you sell its advantages. For multi-winner systems, STV has the longest track record. IRV has the most track record of any single-winner reform. It also is a good stepping stone to STV, and its later-no-harm characteristic means that there are few incentives for candidates to encourage dishonest strategy. Approval voting, since it is not IRV, is a threat and must be fought against.

Nonaligned_Electorama_mailing_list_theorist: I have my favorite single-winner system X for reasons Y but I could live with approval as a first step.

#102 ::: felice ::: (view all by) ::: April 13, 2015, 04:52 PM:

J Thomas @97: "In the nomination I vote for A, B, C, D, E. I think A is the best and I really care about it, but if A doesn't win I want B. C is pretty good. D is OK too. E, meh, I can't think of a better fifth choice. But then it turns out E wins handily. My other four choices get recalculated as half votes. And then D wins. My other three choices get recalculated as quarter votes. And none of them win."

Which is why a system that takes rank into account is a good thing. People already rank their nominations, because there's no way not to do so; I really don't think telling them that the order matters is going to discourage participation. In the nomination instructions, just say something like "Does it matter what order I list them in? It can make a difference, but how many other people list the same works is a much bigger factor; if you're not sure, don't worry about it."

#103 ::: Joshua Kronengold ::: (view all by) ::: April 13, 2015, 05:10 PM:

#90: Cheradenine: Re ties, because SDV eliminates weaker candidates first, it has the luxury of considering the remaining field when deciding how to handle ties, and thus still end up being more or less fair to the voters.

The approach I used for my initial draft was that there's a threshold of the maximuim number of nominees (as well as the desired number). If removing all tied candidates would not put the number of nominees below the desired number, or if the current number of candidates is above the maximum number, then all tied candidates are eliminated as part of a single elimination step. If, on the other hand, this is not true (if the current number of candidates is within the minimum and removing all tied candidates would drop us below the desired number), then the runnoffs stop with a slightly higher than usual number of nominees.

This does act as a weak anti-slate measure (since a slate that manages to enforce strict voting discipline risks being below the threshold of nomination and having the entire slate eliminated rather than the entire slate accepted and added to the ballot), but I think manages to handle ties in most cases (obviously, ones where you end up with an empty or no-candidate balllot aside) reasonably well and fairly.

#104 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 05:12 PM:

J Thomas @97: Under RAV, if you feel "meh" about a candidate, you probably shouldn't list it. Actually that's true about the current system as well - E could end up keeping one of the ones you felt better about off the ballot by one vote. (If you look at the nomination counts, it's not unusual for 5th and 6th place to be this close!)

STV does give you more control over where your votes get placed, but possibly a lot less transfer. If A gets nominated but isn't way over the minimum votes, your ballot is basically done, whereas with RAV, it would count as 1/2 a vote for your remaining nomination choices. So there are scenarios where your "bang for your voting buck" ends up higher with either system. STV is probably more appealing for nominators with a strong favorite, RAV for ones without one. I don't know which group is larger.

It would be interesting to look at the ballots for a particular election and see what proportion named none of the nominees, 1 of the nominees, 2 of the nominees, 3 of the nominees, and so forth, to give us a sense of how often this will happen. "Interesting" might be an understatement. "Vital" might be better. :)

#98: I think your comment "It isn't about how many votes it got. It's about who voted for it." is actually a very cogent statement of how proportional voting systems work. (I might add "just" before the "about".) Note that this is true of STV as well, and, to an extent, of the IRV system that's used in the final round. If this idea makes you balk, then I think your best option is to switch to single nontransferable vote (basically allowing one vote per nominator - which will switch the focus back to campaigning for individual works). You could also keep the current system and try to effectively ban slates (tough in a system where they're so powerful), or alter the electorate.

#105 ::: felice ::: (view all by) ::: April 13, 2015, 05:19 PM:

Bruce Schneier @16:
"nathanbp @10:
"On option 5, what if the current nomination status was released just once during voting, at say one month before voting closes? Release a list of the top 15 in each category in alphabetical order without vote counts. I feel like this would help to concentrate votes without giving out too much information that could be used strategically or being subject to gaming."

I really dislike this. It's too much responding to the tactics, and not looking at the broad picture. The risk of unforseen consequences are great."

I think it addresses a bigger, longer term problem, not just the Puppy tactics. Most nominations are always for works that have no chance of getting on the final ballot; that means they're essentially wasted. Publishing the current frontrunners allows a lot more people to give their opinions on the works that do have a chance - as well as letting people who nominated unpopular works change their nominations, it would encourage participation from people who currently don't nominate at all.

This variant of Option 5 isn't particularly vulnerable to DDoS, sniping isn't a problem, it doesn't require much extra work for the admins (they'd have to check for duplicates under variant/typo names before releasing the top 15, but that leaves less to do before announcing the final ballot), and I don't see how the information it provides is of any use to the Puppies. Can anyone think of any examples of what "unforeseen" consequences there could be?

Do you object to a three stage voting system with a longlist between nominations and final ballot for any reason other than the extra time and work involved? A one-off publication of the top 15 half way through the nomination period serves essentially the same purpose, while being far simpler to implement.

#106 ::: JonW ::: (view all by) ::: April 13, 2015, 05:44 PM:

Felice@#105 -- Without regard to whether this is a good plan, I think strategy for the business meeting has to be "fix one thing at a time" -- it's exponentially harder to convince the business meeting to make *multiple* changes in the voting system all at once. So this proposal makes sense for the next business meeting only if it would be adequate all by itself to address the problem slates present (without also switching the nomination-tallying method to RAV). But I don't think it would be.

#107 ::: J Thomas ::: (view all by) ::: April 13, 2015, 06:01 PM:

@104 Cheradenine

It looks to me like with proportional voting, basicly it's "When you vote for more than one, you give full support to the one that's easiest to win. The more help a nominee needs, the less you can give it."

For this particular case I currently prefer straight approval voting. "When you vote for more than one, you don't choose between them. You are voting for all the ones you vote for, and basicly voting against all the ones you don't vote for."

The more you vote for, the less choosing you do. When you vote for just one, there are (N-1)!/(N-6)! different ways to lose. When you vote for two, there are (N-2)!/(N-7)! ways to completely lose. Fewer. (I may have the formula wrong, I don't feel like thinking it out right now. I think it's qualitatively right.)

There could be some choices you think would be fine Hugo candidates that you don't vote for because you don't want to dilute your vote. But if you find out that the ones you did vote for can't win, you have nothing to lose by voting for other good choices too.

You only choose between the ones you vote for and the ones you don't vote for. So if you only care about voting against a slate, one possibility is to vote for every nomination that has a chance, except those on the slate. That maximises your chance that things not on the slate get nominated, but it gives you no choice at all about which items not on the slate get nominated.

#108 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 06:12 PM:

Kilo @100: Good point. I'm pretty confident that given time and thought, I can write a cogent explanation of HOW a given method works, but that's distinct from WHY it works.

A first try:
SAV/PAV: "Pick the slate that will make the average voter as satisfied as possible with the results."

RAV: "Ballots count more if none of their nominees have been selected yet." (In this case, I'm kind of reversing the mechanical description. An equivalent way to model RAV with exponential weights would be "Whenever you select a nominee, double the votes on every ballot that didn't select that nominee," but this would be more work in practice.)

Joshua @103: Thanks, that was indeed what I was missing. It does mean that a slate with perfect discipline and no support outside the slate would either get all its candidates in or none of them - but I suspect we're as likely to have to deal with unicorns interfering with the election. I definitely will do some SDV simming and see what happens.

Felice @105: I fear that in a dispersed election, giving the top 15 at the halfway point will be misleading if anything. Looking at past ballots, the difference between the vote counts for 5th place and 16th place, with (probably) less than half the vote in, would quite likely be less than ten votes. But if most late voters choose one of the top 15, it will skew the election towards those that got a few more early votes. This would increase the power of campaigning and slates.

So I'm still pretty nervous about it. When I think of votes that run over a period of time and release the results at some point during that period of time, I think of those Internet polls that no one credits with any particular accuracy. It might well drive up interest in the Hugo Awards, but probably at the cost of a lot of added politicking, campaigning, and a skewed result.

#109 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 06:25 PM:

@107 JThomas: Just for clarity of language, you probably mean "RAV" for "proportional." (Proportional voting systems include STV, SAV, and PAV, and the dynamic you describe isn't true of the others.)

I understand your preference for the straightforwardness of approval, but under straight multi-candidate approval voting, the largest voting bloc is always bound to control all nominations (there's a reason it's also called bloc voting). The continuous voting system you suggest is basically an attempt to form the largest bloc dynamically over time - in other words, it's essentially a caucus. I don't think inviting everyone to an online caucus will end particularly well, but at this point I'm going to start repeating my arguments against it, so I'll stop.

#110 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 06:33 PM:

@104: STV is probably more appealing for nominators with a strong favorite, RAV for ones without one. I don't know which group is larger.

Here's how I'd put it:

STV is better for you if:

- it's easy for you to put your nominations in order of preference.
- you think that for two works with the same amount of overall appeal, the one whose appeal is concentrated within a subgenre appeal is usually better than the one with crossover appeal.

RAV is better for you if:

- you'd find it simpler just put "the good stuff this year" in a pile, without deciding which of them is better than another
- you think works with crossover appeal tend to be better than those with specialized appeal, assuming the total support is equal.

(Why do I say that about crossover appeal? Because STV can have a problem with premature elimination; something can be eliminated if it's nobody's first choice, even if it's everyone's second choice.)

#111 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 06:40 PM:

Cheradeine@109:

The continuous voting system you suggest is basically an attempt to form the largest bloc dynamically over time

I'm not wedded to continuous voting (though I do find it intriguing), but your comment struck me as odd. Isn't that, essentially, what we are trying to do in selecting a Hugo winner, no matter which system we choose? I've seen this proposed as an objection several times, but I'm still confused -- it seems to me it's a strength.

I agree there are unknowns that haven't been considered, but I don't think anyone has really considered what those unknowns might be. The attraction of the option 3-class systems is that you can mathematically model how an election will go, so we can get some determinate feel for how effective the system is. I'm wondering if we might be biased towards this type of system as a result. That may be a good thing, but I think we should, in good conscious, examine any biases that we might have. Just because something has a bias doesn't mean it's not biased in a direction that the Hugo voting community as a whole doesn't value and approve of, naturally.

Kilo

#112 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 06:53 PM:

Jameson@110:

Is there a system that can be summarized as "you'd find it simpler just put 'the good stuff this year' in a pile, without deciding which of them is better than another" without regard to cross-over or subgenre appeal? From my point of view, the whole purpose of nominations is just to "put the good stuff" out there and let the actual voting process decide which is the best. I can see arguments against either system if it favors subgenre or cross-genre appeal. That may be unavoidable, however, hence my question.

I find myself wondering how each system aligns to a given "philosophy" of what the Hugo award actually is. As someone (Teresa, I think) pointed out, to date no one has ever had to define what a Hugo was intended to be -- it was simply whatever was the result of the Hugo process. I'm reminded of the same debate in my own field over the definition of a planet. It was never a problem that we lacked a definition until we started discovering objects that didn't clearly fit in any definition. I'm wondering if the different voting systems (in addition to the "mission statement" I mentioned previously) also have a "value statement" that runs something like, "This is what we believe the Hugo is, and this is how this voting system realizes that."

Again, as a completely hypothetical template, a value statement might read, "The Hugo is awarded for the work which appealed the most to the entire fandom community, regardless of subgenre interest." (As an aside, given what we know about people's loyalty to subgenres, I'm not sure that was or ever could be the case, but that's my own opinion.) An equally valid value statement might be something like, "The Hugo recognizes the diversity of subgenres in the SF field, and awards its prize to the work that garners the most support, whether that majority comes from a dedicated group of subgenre fans or from overall broad appeal."

I'm just thinking that we may never be able to choose among the options presented until we all agree on what we're trying to accomplish.

Kilo

#113 ::: J Thomas ::: (view all by) ::: April 13, 2015, 07:00 PM:

@109 Cheradenine

I understand your preference for the straightforwardness of approval, but under straight multi-candidate approval voting, the largest voting bloc is always bound to control all nominations (there's a reason it's also called bloc voting).

The way I say that is "The nominations with the most votes tend to win". Somehow it sounds worse when you say it.

Thank you, that's a clear argument!

You could be right. My view is that when we specifically try to change the rules to exclude the Sad Puppies, and we judge how well the changes work by how well they would have excluded the Sad Puppies given historical data, we will have some difficulty explaining to journalists that we are not doing it to exclude the Sad Puppies. We can say that we aren't trying to break the Sad Puppy slate, we are trying to break all the slates that people might hypothetically try to form.

When we take away fractions of every Sad Puppy vote so their votes are worth less, because they deserve it, we may have some difficulty explaining to outsiders how this is fair.

If instead we *can* dynamically agree on nomination candidates that get more nominating votes than the SP slate, it's much easier to argue that it's fair. They win now because most of us waste most of our votes on things that cannot win. That's how they do it. If we can overcome this flaw in our own voting we can beat them without jiggering with the rules to take away fractions of their votes.

Since it *looks* unfair to jigger their votes, and maybe it *is* unfair, naturally I want an alternative that can work. The better that fans can come to agreement on some sort of consensus, the harder it is for outsiders (or for that matter insiders) to take over.

#114 ::: Cassy B. ::: (view all by) ::: April 13, 2015, 07:23 PM:

May I make a modest request? I've been seeing the term "bad guys" used in these threads, and I'd really rather we didn't use such loaded language. "Block voters" or "slate voters" is what we're trying to deal with, after all; I seriously doubt that most Sad Puppies are actually bad guys. Misled, perhaps. Misinformed, very likely. (A secret cabal? REALLY?) But not actually bad. (I reserve judgment on Rabid Puppies....)

We're all SF fen. Maybe we don't all read the same thing, but that's ok. Let's leave the demonization to the Rabid contingent.

Cassy

#115 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:51 PM:

Kilo @111: Sorry for poor communication. I was trying to explain why the power of blocs in an approval voting system is consistent with the ability of a negotiated set of candidates to defeat a smaller bloc, and then struck by the fact that what we were proposing was similar to a caucus. It wasn't intended as an argument against the proposal.

My issues with continuous voting basically boil down to: (1) implementation difficulties, (2) increased ability for bad actors to wreak havoc, and (3) that it basically embraces making the Hugo award a political process with campaigning, negotiation, arguments about who needs to compromise to push so-and-so out of the top 5, etc. Which is certainly a way that we could go, but it seems to be a prospect that many members of the community regard as undesirable and distasteful.

J Thomas @113: I probably should have called the slates in my example elections something else. I have strong personal feelings about the Puppies but my goal in looking at the voting system is less, "How do we stop the Puppies?" than, "The current voting system is vulnerable to a small but organized bloc controlling the entire ballot, how can we reduce this vulnerability?" Essentially, in my view, the bloc has revealed an exploit in the current voting system, and now we have reason to fix the exploit to stop them or any other group from taking advantage of it. Otherwise, I think the voting structure will lead us to warring blocs, because once enough voters are in blocs the price of not being in one is irrelevance.

Essentially, my goal coincides with the Sad Puppies' stated goal of aiming for a diverse ballot representing a cross-section of fan opinion. I happen to believe that ballot will end up looking a lot more like the pre-2014 ballots than the 2015 ballot, but voters would have the chance to prove me wrong.

#116 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 08:01 PM:

@112:

I think that it's probably a rube's game to try to state all the mathematical differences between STV and RAV in philosophical terms. Any words I use are going to be loaded. Even if we had full historical data and could give examples of the differences, how would we agree on which answer was better? And even if we could, what actually matters is next year's election, not last year's. As a statistician, I can't even think about trying to decide on that basis without alarms in my brain shouting "OVERFITTING" and "POORLY-DEFINED ESTIMAND" and "MISSING DATA".

So I'm sorry I even brought up the "broad vs genre appeal" thing. And I'm definitely not going to say a word about the options for hybrid STV-RAV systems, or for tuning the STV quota, or for Bucklin Transferrable Vote, all of which could work but you absolutely didn't hear that from me.

To stay within the bounds of sanity, I think that a choice between STV and RAV should be made based on ballot format. STV is more expressive. Some people like that. On the other hand, that could mean more work. It could also mean more strategic opportunities ("masking" strategy... yuck). That's really the choice.

When it comes to SDV vs RAV: they're really similar. Probably too similar to be worth arguing about. But if the motivation for SDV was to avoid the possibility of ties in RAV, I can say that that's misguided; in theory, ties in SDV between two works with fewer than 5 votes each could snowball and end up affecting which totally-unrelated works win.

#117 ::: J Thomas ::: (view all by) ::: April 13, 2015, 08:57 PM:

@115 Cheradenine

"The current voting system is vulnerable to a small but organized bloc controlling the entire ballot, how can we reduce this vulnerability?" Essentially, in my view, the bloc has revealed an exploit in the current voting system, and now we have reason to fix the exploit to stop them or any other group from taking advantage of it.

I fully agree. The existing system is broken.

Essentially, my goal coincides with the Sad Puppies' stated goal of aiming for a diverse ballot representing a cross-section of fan opinion.

My view is that what's broken is that normal fans do not produce a lot of votes for winners at the nomination stage. That leaves an opening to be exploited by anybody who can get a lot of votes.

I have trouble seeing how we can explain that nominees that get the votes should be discarded in favor of other nominees that get less votes. I can see it when I think about abstract voting systems and abstract goals. But when it gets specific....

Here's a Hugo award nominee, and it has 250 votes. It will not be on the final ballot but will be replaced by another nominee that has 72 votes. I think about how I would explain that to a Fox News guy who wants a 15 second soundbite for his "Corruption in the Hugos" exposee, and my mouth gets dry and my forehead starts sweating.

Would you volunteer to be on call to field that interview if it ever happens?

#118 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 09:07 PM:

J Thomas @117: Sure! Though I hope I would get an assist from whoever's in charge of explaining the final result in the case that the candidate with the most first-place votes loses...

In terms of presentation and PR, though, this does seem like an advantage of a system like STV or SDV, where votes are always getting added to candidates instead of subtracted from them. (Or of reformulating RAV as "double the votes of any ballots who didn't get nominated," but that could produce eyebrow-raising numbers as well. "First we chose X with 100 votes, then Y with 160 votes, and now the third choice is Z with 180 votes...")

#119 ::: Cat ::: (view all by) ::: April 13, 2015, 09:24 PM:

Okay, based on what I've seen so far I'm liking RAV with exponential weighting.

It's not really possible for a slate to *hurt* a despised work, as dh brings up at one point, because the slate still contributes votes to that work, just fewer of them. Unless the slate choses a despised work that has such wide support that it wins before anything else on the slate, in which case all the slate's votes for wanted work just got cut in half, so the slate can go right ahead and throw me in the briar patch; next problem.

The more I think about it, the more I like the idea of more voters getting to have at least something they like on the ballot, which seems to be the point and product of RAV.

One thing does occur to me though. Has anyone modeled how RAV would behave in the presence of two slates? Because we already have two. So if we ended up in some hypothetical future year with the Redshirts Slate and the Puppetmistress Slate, it would be nice to know how the system would behave, or fail.

#120 ::: Emily H. ::: (view all by) ::: April 13, 2015, 09:25 PM:

J Thomas:

I just don't see it as a problem that "normal fans do not produce a lot of votes for winners at the nomination stage." It's just a natural consequence of having a big, diverse field, of which nobody can read more than a small chunk every year. Any way to increase coordination on the part of voters seems to encourage voting for what you haven't read but your friends say is good, or for what you only mildly liked but stands a decent chance of winning, and I think that's more broken than the status quo.

I can't even theoretically manage to believe that Fox News might take an interest in the Hugo Awards even as a "making fun of political correctness" talking poing, but if people can understand that North Dakota has two senators and California has two senators, then they can understand setting up a voting system in a way that gives a louder voice to some than to others.

#121 ::: Edmund Schweppe ::: (view all by) ::: April 13, 2015, 09:28 PM:

Cheradenine @115:

Essentially, in my view, the bloc has revealed an exploit in the current voting system, and now we have reason to fix the exploit to stop them or any other group from taking advantage of it.
This actually ties nicely into the question I've been trying to figure out how to ask for the last hour. I had an idea that could prevent blocs hijacking the nomination process. Details are at entry 96 of the "Clean Living" thread; the quick summary is for the Worldcon Committee to create a section 3.3.17 special category of Best Bloc Nominee (BBN) and use their powers under sections 3.2.7 and 3.2.8 to move bloc-nominated works into the BBN special category, where they won't interfere with the rest of the nominating process.

However, the Hugo admins for that Worldcon would need a good, transparent, mechanical way to figure out which ballots represented bloc nominations, in order to figure out which works to reassign to BBN. The bloc nominators will howl in protest, but the rest of fandom would probably accept it - if that good, transparent, mechanical way exists and the Hugo admins use it.

So, my question: is there a good, transparent, mechanical way to detect (a) that bloc voting is occurring and (b) which ballots are voting for which blocs? And, if so, what is it?

#122 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:36 PM:

3b, 3c, and 3d will all work. Pick whichever is easiest to compute and most comprehensible to the voters.

I've studied election systems for decades. These options (3b, 3c, 3d) are all variants of proportional representation.

Proportional representation is *always* the correct method of choosing a parliament, a committee, or a list of nominees, for very good reasons -- you're trying to get diversity on the committee or list of nominees which represents the diversity of the electorate.

Any proportional representation system prevents bloc voting from working.

If you're trying to get a single winner, or deciding something like the location of next year's convention, you will do better with straight approval voting.

Approval voting also prevents bloc voting from working, but only works with single-winner results.

Every election system is subject to tactical voting, but not all of them are subject to *bloc* voting -- these are systems which are not subject to bloc voting. (Tactical voting in proportional representation systems is much subtler, and quite risky, since it involves voting your second choice and not your first choice, and so risks letting your first choice lose out entirely.)

#123 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 09:42 PM:

JThomas @98: Every one of these proposed changes necessitated by the introduction of slates to the Hugo process is less democratic than the system that preceded it.

J Thomas @99:

@96 Teresa Nielsen Hayden

Do the people who think the Hugo nominees are all the same actually read them?

Some years ago I read about a Chinese convenience store in San Francisco that got robbed by a white man. The clerk could not give the police a description. She said they all look alike.

It's like that.

The hell is it like that.

If they care so little about science fiction that they can't tell one Hugo nominee from another, why are they even voting?

#124 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:45 PM:

FWIW, many serious election math theorists concluded years ago that reweighted approval voting (RAV, 3c) was the best theoretical method for selecting a committee (and thus, a slate of nominees). You can look into the papers on this if you like.

People in the field agree even more strongly that approval voting is the most democratic way to go for single-winner elections. The major problem with approval voting, which has been used successfully by many organizations, is that it keeps getting repealed by self-interested organizing committees in favor of First Past The Post... while the voters aren't paying attention.

#125 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:48 PM:

"One thing does occur to me though. Has anyone modeled how RAV would behave in the presence of two slates?"

Yes. There are papers on this. It's been studied in the context of modeling US legislative elections should we adopt RAV (assume a starting position of two political parties).

RAV gets the third parties some seats in the legislature.

#126 ::: J Thomas ::: (view all by) ::: April 13, 2015, 09:54 PM:

@118 Cheradenine

"Would you volunteer to be on call to field that interview if it ever happens?"

Sure!

OK, try a quick trial run. The cameras are pointed at you, and the microphones, and the bright lights. It's hot.

The interviewer starts his lead-in. "As we have all heard, for the majority of nominations for this prestigious award, the nominations with the most votes were thrown out and replaced by other nominations with a quarter -- or less than a quarter -- as many votes. For example, Jameson's Genociders, a story about patriotic soldiers winning a war on a desert world, was replaced by Mothra Faker, where a transsexual hobbit repeatedly tricks the sexists who try to get him pregnant. But the people responsible for this -- caught red-handed -- admit the facts but claim that what they did was the right thing to do and not corrupt at all. Here is Cheradenine to tell us their excuse."

What do you say in three or four short sentences?

#127 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:58 PM:

J Thojmas wrote: "Here is my current concern with RAV.

In the nomination I vote for A, B, C, D, E. I think A is the best and I really care about it, but if A doesn't win I want B. C is pretty good. D is OK too. E, meh, I can't think of a better fifth choice.

But then it turns out E wins handily. My other four choices get recalculated as half votes. And then D wins. My other three choices get recalculated as quarter votes. And none of them win.

I didn't care that much about D or E in the first place, but the result of putting them on the ballot was that my vote for A got turned into a quarter of a vote. "

J Thomas: in this situation you should have voted for A, B, and C. If you don't really care about D and E, don't vote for them.

In approval voting systems (where you vote up or down on each story), there is a tactical question of where to make the cutoff: what's "good enough" to be worth spending your vote on.

Deal with it. Every system has tactical considerations. The fact is that approval voting systems are more democratic, and less "game-able", than any other voting system ever proposed. This is confirmed both by mathematical theory and by practical tests.

I'm a bit of a partisan on this; I switched a club to running all its meetings on approval voting a decade or so back, and it saved SO MUCH TROUBLE. Every other voting system featured people complaining that the winners weren't "really" the most popular.

With single-winner approval voting, you can prove mathematically that the winning option is the one the largest number of people approved of.

With reweighted approval voting, using the correct weights, you can prove mathematically that the list of "winners" (nominees in this case) is the list with the largest number of voters approving of at least one winner.

If your goal for your organization is to have lots of members who feel that they have been represented, this makes approval voting variants obviously the best. (If that isn't your goal, well, democracy and voting may not be right for you.)

#128 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 09:59 PM:

Nathanael@124 (and others):
FWIW, many serious election math theorists concluded years ago that reweighted approval voting (RAV, 3c) was the best theoretical method for selecting a committee (and thus, a slate of nominees). You can look into the papers on this if you like.

This kind of gets to the heart of my concern expressed in #100: I think very few fans are willing to take "trust me, I'm an expert" as an acceptable reason to support a particular voting system. It may, in fact, be true that it does work, and works well, but so far I've only seen "appeals to authority" when what we need is a coherent, easy-to-understand explanation that (most importantly) explains why the system is, in fact, fair. As others have said, if you lose that appearance of fairness, then all is lost no matter you do.

The discussion of "how do you explain why the person with the most votes lost" gave me a cold shiver. Consider that there will be those who will delight in making whatever system that is proposed look bad, and certain people have shown that they aren't above misrepresenting what has been said to make their points. Now imagine that you've just made a statement similar to the quote above. I predict that saying, "Well, yes, but..." isn't going to cut it with a large portion of the HAHV's.

I'd almost go so far as to say that a system which seems fair may be the single most important of Bruce's criteria.

Please keep in mind that I'm not saying any of the option 3 systems are not fair, but I'm beginning to see that it's critical that the system be explained in such a way that the HAHV's can -manifestly- see that it is fair.

Kilo

#129 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 10:09 PM:

Cat @119: I tried a two-slate model as one of the examples I ran in post #3. Each slate got some nominations, as well as one or two popular non-slate candidates. (The counts depended on the popularity of the slate choices and how many slate voters there were.) This matches the behavior Nathanael cites in #125.

Nathanael @124: Do you have some citations? I'm certainly not up to date on the literature in any comprehensive way - I'm a teacher, not a researcher, and this is a small fraction of what I teach - but almost all the mathematical analysis I've read has given phrases like "most democratic" a fairly wide berth, because it's loaded and difficult to define.

#130 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:13 PM:

The usual explanation for proportional representation in legislatures goes like this:

40% of people vote Democratic. 40% vote Republican. 10% vote Green. 10% vote Libertarian. So the legislature should be 40% Democratic, 40% Republican, 10% Green, 10% Libertarian. We need a system which does this.

The same argument can be used for selecting a list of *nominees*. If 10% of the nominating population support colonialist military SF and 10% support gender-bending SF and 20% support mass-market explosion-oriented SF and 10% support wordplay-heavy literary SF, the list of nominees should represent these tastes in proportion to their presence in the electorate.

Explaining *why* a system like RAV achieves proportionality requires a little math. No getting around it. It's simple math, but there's nothing to be done for people who refuse to read a couple of pages of it.

Approval voting is much, much, much simpler to explain -- the most popular candidate always wins -- but you'd have to run a very different system in order to use it (you don't really need nominees with an approval voting system).

#131 ::: nathanbp ::: (view all by) ::: April 13, 2015, 10:16 PM:

RAV doesn't really seem that hard to explain. "Each time a work on your ballot is nominated, it counts for half as much for picking the rest of the nominees." Maybe I'm missing something, but I think that covers the gist of it accurately?

#132 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 10:17 PM:

Nathanael, the moderators are discussing your most recent comment. This may take a while.

#133 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:27 PM:

Upon looking again for the first time in a decade, I see that most of my academic citations for this stuff have dropped off the internet (sigh). Election math is an old field. I suppose I'm not surprised that people are reinventing the wheel.

There are some cites. Straight-up approval voting has its own advocacy group, which has some links:

http://www.electology.org/

This isn't the first such group, but the last one, which had a lot of academic links, gave up and dropped off the internet...

Range voting has its own advocacy group: it's mathematically equivalent to approval voting, but may or may not be an easier sell (the Olympics uses it):
http://www.rangevoting.org/

And some other links.

http://www.tursiops.cc/idhop/av/

Reweighted range voting (using the very specific weights needed to guarantee proportionality):
http://www.rangevoting.org/RRVr.html

I spent a lot of time studying this stuff two decades ago, first in the context of governmental elections, and then in the context of running committee meetings, where I got some practical experience. (My god, approval voting simplifies parliamentary procedure with regard to amendments massively -- throw out all the possible amended versions at once and have everyone vote up-or-down on each of them. Most popular version wins.)

The conclusions haven't changed in two decades. For most purposes which people might want to use a voting system for, you want either Approval Voting or Range Voting (single-winner) or Reweighted Approval (or Range) Voting (multi-winner). Multi-winner STV works reasonably well too. (Single-winner IRV doesn't, by the way.)

If you're actually *trying* to make bloc voting successful at suppressing candidates with wide popularity (and I'm sure some people are), then you might try a different voting system.

The real problem is that most people don't give a single thought to whether the voting system actually does what it's supposed to. This is a worse problem in the US than in the UK, where voting systems remain a hot topic of discussion.

Anyway, upon Googling, apparently the Oscars use reweighted range voting to determine the Best Visual Effects nominations. Maybe the message is getting through slowly.
http://www.goldderby.com/news/7980/oscars-boyhood-birdman-selma-imitation-game-entertainment-13579086-story.html

#134 ::: David Dyer-Bennet ::: (view all by) ::: April 13, 2015, 10:36 PM:

I don't think STV and IRV and the like work well for sparse long-tail things like...Hugo nominations. (I'm slightly surprised at how well some of the STV tests seem to have gone; but we don't have real data, and I wouldn't be surprised if real data is more diverse and thinner than the test data we've used.) Minneapolis is a ranked-choice voting city now, which is good, but it's a LOT different from Hugo nominating. Hugo voters of course are already intimately familiar with the procedure since it's used for the final ballot since time immemorial (i.e. before I got involved).

Possibly some statement against promulgating slate ballots might be useful. I'm utterly certain that any attempt to solve the problem by simply forbidding them will have horrific outcomes, though, probably in the very first year. Secrecy is one counter-measure, gaming the rules is usually possible, *and* it's a hammer that can be waved against legitimate attempts to stir up interest in Hugo nominating. It would be a huge disaster. I wonder if anybody vaguely competent would consent to be Hugo administrator under such rules? And then a less-competent admin would make things worse. Etc.

At some point a group stops being a "voting block" and becomes "a solid majority of the members" :-) . Reasonable democratic rules can't protect us from giving a Hugo award to a work that's the favorite of every single Worldcon member (of course, that degree of popularity is also not really a worry; then again even half easily suffices). And one of our better protections is to get more of our existing members involved in nominating. I don't see rules-changes relevant to that, but maybe some publicity dealing with the "I haven't read enough to nominate" idea?

#135 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:39 PM:

Teresa -- no worries from me about moderation, do as you like. I just found myself here and went "what, reinventing the wheel on voting systems again?" and sighed. Sometimes it feels like human knowledge does not advance over time.

The problem of having a list of nominees which reflect the diversity of the voters (rather than ending up with a "bloc slate") is known as the "representative committee" problem in the literature. Any proportional system will do. I happen to think that the range/approval variants are easier to vote, easier to count and provide less weird gaming-the-system opportunities, but Single Transferrable Vote with multiple winners has worked just fine in many countries, so I'm not going to complain about it.

#136 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 10:39 PM:

Nathanael @130: Fandom is not going to Balkanize the Hugos according to your schema of what's important in literature. You need to recognize that your map of subgenres and categories is not an objective map of the world. It's your personal map of your personal reading. The rest of us are unlikely to adopt it.

The Hugos are about what the voters love. When you vote in the Hugos, vote for what you love. Take the risk of loving it for its own sake, whether or not you think it has a chance of winning. Let other voters do the same.

#137 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 10:42 PM:

Me @136: Because believe it or not, that's actually how it has worked until now.

#138 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:50 PM:

Mr. Dyer-Bennett -- you might be surprised how sparse and long-tail most municipal elections actually are. :-P I've seen municipal elections with a large number of candidates (more than 12) and a widely scattered vote. From a distressingly small number of voters.

Any proportional voting system seems to work fairly well for selecting a "representative committee", regardless of how sparse and long-tail the voting is. The issue of "more people have heard of this candidate than that one" remains unavoidable, of course.

STV with multiple winners (at least three) is proportional; IRV with a single winner isn't, and has some problematic mathematical properties which tend to reduce it to first-past-the-post, but can actually make it worse in some cases.

The difference between STV and IRV in practice is stark. It can best be demonstrated by the Australian Parliament's lower house (single-winner IRV, behaves exactly like first-past-the-post, two-party system, winning party isn't always the most popular party) and its upper house (multi-winner STV, has proportional representatives from many parties, including both the Greens and "Palmer United" which is basically backers of coal mining).

I do not think of STV and IRV as the same voting system, because the results are so different.

#139 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 10:50 PM:

Nathanael@130:

Explaining *why* a system like RAV achieves proportionality requires a little math. No getting around it. It's simple math, but there's nothing to be done for people who refuse to read a couple of pages of it.

Well, and I say this with much respect for your obvious expertise, but I think being dismissive of a (potentially) large portion of fandom in this way is exactly what we must avoid at all costs.

I can certainly see working out the merits of various systems mathematically (as we are mostly doing here) and then coming up with a popular way to explain it after the fact. I'm also okay with (as I suggested above) coming up with a "mission statement" that we want to achieve and finding a system that best accommodates it. What we cannot (again, in my opinion) do is to say to fandom, "If you can't be bothered to figure out why this is a manifestly fair system on your own, then tough for you."

Ultimately, I think we can all agree that the Hugo -is- the fan's award, and I feel our primary responsibility is to make sure that the fan's continue to feel it is their award.

Kilo

#140 ::: Cat ::: (view all by) ::: April 13, 2015, 10:56 PM:

Cheradenine @129 Oops! Sorry I missed that, and thanks for pointing that out; I'll look at that again.

As for explaining RAV so that everyone thinks it's fair, I don't think that will be possible. For some people, no system is fair until they win all the marbles and own the field. Some people, for example, were very angry about the final Hugo voting process last year, because a person can have the most 1st place votes and still end up in last place if everyone else puts them low enough on the ballot.

You need to be able to explain RAV well enough that a person without a specific axe to grind thinks it's fair. I don't think Fox News will ever be in that category. But the people we need to please are the fans.

For them I think "the thing with the most votes becomes the first nominee. Once one of your picks becomes a nominee, your other votes count for half. We count the votes and half votes up. The thing with the next most votes becomes the second nominee. If two of your picks have become nominees your other votes count for a quarter, if only one, they count for half, if none, they still count full. We count the votes and half and quarter votes again. The thing with the next most votes becomes the third nominee... and so on. The end result of this is that as many people as possible have at least one thing they liked on the ballot."

One of the questions that Joshua brought up I haven't seen answered yet. Sometimes nominees decline the nomination. Is the ballot recalculated from there? If it is, could that lead to some second nominee that did make the ballot the first time failing to make the ballot the second time? Because that's the kind of thing that could produce hard feelings, I think.

#141 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:59 PM:

Teresa: *shrug* I'm not sure how you got that response from my comment.

I'm pretty sure you have specific tastes which tend towards particular genres or subgenres or styles. I've never met anyone who didn't. Some people have a more conscious understanding of their own tastes than others. Most people fall into groups regarding their tastes.

Award winners for well-run awards tend to appeal to people who have a wide variety of different tastes. (That's often what makes them so special.)

Creating a good list of *nominees*, however, is quite another matter. The goal of a pre-published nominees list is, if I remember correctly, to get people to read and consider stuff which they might not ordinarily have read. To do this, you *deliberately* want to get stories put on the list from people who tend towards different tastes.

Because you have to assume that before the nominees list comes out, a lot of people will simply not have looked at the stuff which wasn't to their 'typical' taste.

If this exposure wasn't the goal, you could skip the entire nominee procedures and just let people vote straight up for anything for the final vote.

So if you want the nominee list to work this way... you should design the system for nominee selection to *make* it work this way. You can call this paranoid, but it's really the "security mindset" that Bruce Schneier describes: assume that someone's going to try to break your system and preemptively design a system which doesn't break.

#142 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:05 PM:

J Thomas @126: OK, I'd give this more thought (and have more data at hand) if it were a realistic possibility, but, "The current system is designed to represent as wide a group of fans as possible on the final ballot. The final ballot includes choices from 52% of the ballots that were submitted. Using the system you're suggesting, only 24% of the ballots would have any representation."

And then if they ask if it isn't true that the system was just changed, say, "Yes, the concern was raised that the old system didn't guarantee a wide range of nominees."

This won't sway Fox News, but the best you can do in this scenario is try to give a clear and accurate answer. (I should note that this took me a while because it took a lot of gyrations to find a scenario where this could even happen to estimate what the percentages would be: the slate has to get 2 votes, Jameson's Genociders has to be listed on slate ballots and absolutely no others, and at most two non-slate books can get more than 5-6% of the total vote (and they can't share any votes with . It's basically a perfect storm, the equivalent of a book with 6.26% of the first-place votes winning the IRV round, which is also conceivably possible. But Fox could still generate outrage over a win by a work with 281 votes over one with 283, if they cared enough to.)

Kilo @128: I genuinely am trying not to argue from authority here. I don't consider myself an expert, for one thing. You see above my stab at arguing that it's "fair," but I'm honestly a little uncomfortable even using the word. There are a lot of ways to define fairness, and no voting system can meet all of them.

Nathanael @127: I'm uncertain that you can contribute much to this conversation by telling people to deal with it and making statements like, "With single-winner approval voting, you can prove mathematically that the winning option is the one the largest number of people approved of." Well, yes, because that's how approval voting is defined. I might as well say that with Condorcet voting, you can prove mathematically that a Condorcet candidate will always win. It doesn't add anything meaningful to the discussion if everyone pops in and rattles off the fairness criteria their chosen method satisfies.

#143 ::: Brad from Sunnyvale ::: (view all by) ::: April 13, 2015, 11:08 PM:

Bruce asked me last night to do some commentary on this posting, but I had too much work to do. And still do. But I have written up something fairly detailed on the philosophy around the Hugos and why I have come to a different opinion from most here.

Second Musings on the Hugos

For the TL;DR crowd, I outline what I think the purpose of a Hugo voting system is -- "to aggregate the opinion of the members, as accurately as possible, on what the best work of the year is, if it were the case that most of them actually read/evaluated most of the leading contenders." The nomination process helps find those contenders to give people a chance to look at them.

In looking at all the voting systems, I see too many flaws. Aside from complexity, they all add strategy. There are too many times when a member, armed with only a little information on likely popular choices, is best guided to deliberately leave certain works they actually loved off their ballot to improve the chances of others. I find this unacceptable.

The second thing I find unacceptable is that all these systems still put at least one, and as many as 3 slate candidates on a ballot from a group that is a minority (like 10%) of members. If we strive for the goal of an accurate measurement of true fan opinion, this is not right. Sure, you stop them from getting 5. But why is one OK if it needed collusion to get it on there?


In the end, I conclude something different. The only defence against attack by humans is not a set of rules, but a system of justice. Pit humans against humans. The humans on our side, the side of justice, will have some rules, some transparency, some accountability, but they will also have discretion so they can solve not just the current problem, but anything else the attackers throw at the system.

I know many will not like some of the examples I give of how to do this, but I want to focus on that principle. Only human judgement will foil clever attackers. Attempt to foil them by committee with voting system complexity will both fail, and also cause side effects.

#144 ::: David Dyer-Bennet ::: (view all by) ::: April 13, 2015, 11:08 PM:

My memories of nominating, and discussing nominating, definitely agree with people talking about diversity within ballots. My actual experience is that while each of us prefers some sub-genres of SF, we are able to recognize outstanding works in most of them. (Note, *most*; at least some of us despise some sub-genres enough to be unable to recognize outstanding works there.)

Thus for example, long ago, lots of *the same people* thought it was clear that both The Dispossessed and The Mote in God's Eye belonged on the ballot that year. They were also new works from major authors, so very broadly read, which always helps.

So I don't think that, absent slates, the probable distribution of real nominations is one pile with all space opera and one pile with all feminist fantasy, but a much more mixed arrangement.

#145 ::: Tammy Coxen ::: (view all by) ::: April 13, 2015, 11:08 PM:

I think the best way to handle declined nominations is just to offer it to the next person on the list. The way the process works is that the Hugo Administrators reach out to the nominees and ask if they'd like to accept or decline. If they are unable to reach someone, the assumption is acceptance. I don't know what happens if they receive a decline late, after the finalists have already been announced, but you definitely wouldn't want to recompute the entire finalist list after it's been made public. So it seems to me that just going to the subsequent nominee is the best choice.

#146 ::: P J Evans ::: (view all by) ::: April 13, 2015, 11:11 PM:

134
I'm hoping to have real not-exactly-live data next week, maybe the week after.

#147 ::: TomB ::: (view all by) ::: April 13, 2015, 11:13 PM:

I would like the Hugo nominations process to be better at representing diversity than the current FPTP process. Both the diversity in the candidates, and also the diversity of the community making the nominations. Of course it has to minimize the effects of slates, because slates are terrible for diversity, but it shouldn't just be an anti-slate measure, it should be pro-diversity.

This is another way of saying that Option 3 looks good. I don't know which of the variations would be best. They all have an aspect of "okay you got one of your choices, now let the other fans have a chance."

#148 ::: Brad from Sunnyvale ::: (view all by) ::: April 13, 2015, 11:18 PM:

Ok, odd, my link in #143 is missing even though I double checked it. Let's try again.

Second Musings on the Hugos

OK, looks like you really need the quotes! See #143 for TL;DR summary.

#149 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:27 PM:

Brad @143: I respect your opinion here (though I think it's a bit tangential to a thread dedicated to "exclusively focused on voting systems and their relative merits"), but I do want to challenge the idea that more complicated systems necessarily add more strategy. Determining whether leaving some candidates off your ballot is strategically beneficial or not in RAV, for example, is a very hard problem even if you have exact knowledge of how everyone else voted; in general, you will get the best chance of an optimal result by voting the precise candidates you want to see on the ballot. (This is actually not true of the current system. Reference is here, but it's technical.)

P.J. Evans @146: I'm trying really hard not to drool on my keyboard. Mmmm... real data. (I suppose "live data" is not the way to go when the data is considerably older than most of my students.)

Cat @140: Declines are a bit tricky. My instinct would be to say "rerun the process, but only to the point that it selects a person who wasn't in the original nomination group, and offer it to that person." I'd have to look at this more and/or check the literature to see if it leads to any problematic behavior.

#150 ::: AnnieY ::: (view all by) ::: April 13, 2015, 11:30 PM:

This will be long but I had been thinking...

No matter what new system is chosen, it will be gameable. Not as easy as the current one but there is no system that cannot be gamed provided enough will. And the puppies can be a lot of things but I would not call them stupid and I would not be surprised if they hire someone to help them figure out whatever system the Hugos come up with.

So the most logical answer seems to be to stop the system from being gameable by making it impossible for anyone to predict what system will be used this year. I don't mean announce it after the nominations close (that obviously will not work) but tie it to a number that cannot be predicted. Number of valid ballots for novel. Or for short stories. Or for fan artist. If need be vote for which category to be used on each Worldcon for the next nomination cycle.

Then build a table that defines which algorithm is to be used based on the interval the number goes into. Make the intervals 10 wide, 50 of them (so 0-10, 10-20 and so on up to 490-500) and if there are more than 500 ballots divide by 500 and use the remainder.

And just to make sure that noone calls fault, build the table on every Worldcon, if need be with double blind draw in two separate rooms - one of them choosing the order of the intervals, the other one choosing the order of the algorithms. If we have 5 algorigthms, each takes 10 spots so this is the 50 elements that you can draw out of a hat/ball/whatever.

Once the table is built, just publish it on next Worldcon site or on the federation site and forget about it. Until it is time for the next nominations counting.

Yes - this eliminated ANY strategic voting but.. technically that is the idea - trying to vote strategically puts you in a path that leads to slates and decision for other people (most people won't go that far but...). This also will close all the rumors on a cabal that decides anything - noone can predict how many valid ballots there will be in a category in a year - unless if someone decides to say that the Hugo administrators are part of a cabal or something - and noone from any side had done that so far (and they would not). And even if numbers leak, noone can predict when the last 10 fans from somewhere will decide to vote - I added a nomination for a movie on my ballot ~3 minutes before closing time - before that this category was empty for me.

The main issue will be that the people that count will need to know all the systems but there is enough people in this thread that can help with that I suspect and that should not be such a problem.

Under these rules, a slate can end up with all works on the ballot as this year but can also go against an algorithm specifically built against slates. Somehow I do not see the puppies running the chance. And if they do, well - at least we had done all we could to try to stop the gaming.

#151 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 11:43 PM:

Comments will be shutting down at midnight. We'll be back tomorrow morning.

#152 ::: Brad from Sunnyvale ::: (view all by) ::: April 13, 2015, 11:49 PM:

#149: I disagree. The reason Option 4 above is my current choice is because there are too many flaws in the proposed systems, which is what we are discussing. This thread and the prior have outlined them, and so I think Option 4 (and grander versions of it) are the only remaining approach that works towards the goals.

I think RAV is one of the easiest ones to be strategic on, but for almost all the proposals above, which "punish" your ballot for having named a popular winner. The strategy is simple -- if you think there's one or two obvious choices, don't vote for them. They will get on the ballot without you, and your ballot will weigh more strongly to promote your less obvious selections.

Yes, if lots of people do this, then we get a strange counter-result, which would be even worse, which is one reason why you don't want your ballot system to have a strategy. But more than that, the existence of strategy makes the system inherently more complex, because you must not just understand the system, you must understand the strategy.

So I view that any system which seriously increases the effectiveness of strategy fails the complexity test, as well as the most important test of all -- does it make members express their true desires? If people don't vote what they really feel, the ballot is useless.

Approval worked in the past because it has really minimal strategy. You can leave off a popular choice, but the only thing that buys you is the ability to name another item on your ballot. Risk hurting one of your top choices to give a boost to your #6 choice -- not a very good deal, and that's why there is not much strategy to the current system, and why we liked it.

It was our last, best hope for peace. It failed.

I'll tell you now, if we use RAV or similar, then I'm going to take any choices I have which showed their popularity by winning the Locus poll, or doing well in other awards that came out earlier, or any other thing I can find that's a good predictor, and I am not putting it on my ballot.

I don't think so many will do it as to take it off the ballot, so I won't cause any harm. Though when the final tally is published, it will not have done as well as it would have in an honest and independent poll.

That's why we like STV for the final ballot, with all its flaws. There is no strategy. It's a much more important feature than people are imagining.

#153 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 11:57 PM:

A question for the experts:

I'm going to try to take my own advice and lay out an explanation first, then propose a voting system around it. This particular one almost has to be a variation of one of the option 3 systems, but I'm uncertain which one. Can you point me to it?

Here's a proposed system:

Everyone is told that at the end of the day they will be able to nominate no more than one nominee per category (and possibly none, if there is insufficient support for all of your preferred candidates). The goal is to give you the best chance of having one of your preferred candidates make the ballot, and preferably, one that you ranked highest on your list.


The process is that they list their top five nominees, in order of preference. Important: Even though the nominators have listed their nominations in order of preference, that is only to determine when they are "finished" having a say in voting. A vote for a work anywhere on the list counts the same. If, after tallying all of the nominations, their first choice wins, none of the rest of their nominations in that category are counted. Poof. Their nomination ballot no longer exists for that category.

For the next round, we eliminate the first nominee's name from all nomination ballots. We go through -all- of the remaining nomination ballots and select the nominee who gets the most nominations. If your first choice candidate was already chosen, then you have no say in this round -- you already got your candidate. If your first choice candidate did not win, however, then your nomination ballot is counted, exactly as if this were the first round (though you may only have four nominees on your ballot if one of your lower-ranked nominees came in first in the last round). Again, a vote for a name anywhere on your ballot is a vote for that work to be nominated. If your first choice has now been selected, again, you're done. If not, you get to participate in the next round of nominating. This continues for a total of exactly five rounds, selecting the five nominees for the Hugo award that will actually be voted on.

I -think- this is similar option 3d, but it's not exactly as Bruce describes it; I claim it's a little bit simpler to understand. It may not work as well mathematically (I'm not qualified to judge, but I'm hoping someone here is), but it seems to me that the "perceived fairness" rating is high (whatever that may mean). There is almost certainly some tweaking that would need to be done, hence my appeal to the experienced ones here.

A slate will almost certainly get one nominee on the ballot -- but only one. If they have a large enough representation, I don't see that as a problem -- there is something to be said for giving a majority a say, yes? I'm probably not seeing all the ways to game the system, and I suspect I'm a little more tolerant of "tactical voting" than some here are, so long as it leads to a genuine consensus, so I may be blind to it. I'm immediately curious what happens if there are two (or more) slates. Is this similar to the case we were discussing where basically if there are five slates, then they will each get to choose one of the final nominees? If so, can you explain why for me?

Additionally, what if a truly well-organized slate says, "We want 20% of our nominators to list this as their first choice, 20% to list it as their second choice, 20% to list it as their third choice, and so on." Does this break the process? The way to fix this might be to change the system so that if -any- of your five candidates makes the ballot, then your nomination ballot vanishes. I think this, too, is true to the original proposed philosophy.

Can one of the experts help me out? Again, my goal here is to try to explain what I mean by coming up with an explanation that everyone can intuitively understand and then designing a system around it. I have no problem using the "pure form" option 3 variations, so long as we can meet this criteria, but right now, I don't see that any of them really do.

Thanks,
Kilo

#154 ::: Cheradenine ::: (view all by) ::: April 14, 2015, 01:18 PM:

Kilo @153: At first I thought this was a slight variant of STV, but after some consideration I realized that, assuming informed voters, this system works out as equivalent to single non-transferable vote (everyone gets exactly one vote and we take the 5 works with the most votes) – i.e. the current system with each voter reduced to one nomination. The reason is that there’s no motivation to list anything in slots 2-5, because all that having a work there can do is get your ballot tossed out – it’s never going to help that work get elected. So optimal strategy is to only list one nomination.

In general I think we’re better off using a system that has been proposed and studied in the past, because there will be existing research on its implications, advantages and disadvantages, vulnerabilities, etc. (That said I do like SDV a lot more than I did at first – which brings me to...)

Joshua @88: I ran two sims using SDV, one using the 2013 nominations with bullet vote for #6 case, and one using the slate of 150 case. In both cases, the results agreed with RAV. I didn’t simulate more elections because SDV is considerably more work than RAV (since if there are N candidates above the cutoff, you need to go through N-5 iterations instead of only 5), though this could be eased by program support.
I’m warming up to this method. It yields a similar outcome to RAV with d’Hondt weightings, but I think its fairness is more intuitive. (“Everyone gets one vote split among the works on their ballot, and as we eliminate the works with the least votes, the vote gets redistributed to the choices left on the ballot. So everyone continues to have the same amount of say until all their nominees are eliminated from the ballot.”) The reported vote counts only increase, so dealing with the investigative reporter is a bit easier.
I would really like, though, to find some literature on this method, or find an actual voting systems expert to analyze it. Using an untried system can yield unexpected surprises.
Incidentally, it’s interesting to see how different the results look for slate and non-slate candidates. Non-slate candidates tend to slowly and steady gain votes as other candidates are eliminated, whereas slate candidates would stay at a fixed number of votes and then jump up when one of the slated candidates was eliminated. (This might be less apparent with real data, where there might be more correlations between votes for non-slate nominees.)

Brad @152: In general, all voting systems have flaws. The flaws in bloc approval are just flaws we’re more used to accepting, but that doesn’t make them less significant.

“The strategy is simple -- if you think there's one or two obvious choices, don't vote for them.” I’m glad you said this, because it prompted me to think through the details of why RAV is more strategy-resistant than straight AV (despite appearances), instead of relying on a theorem in a paper. Here’s the basic issue: being among the top few vote-getters in a proportional voting system is not an absolute guarantee that a work will be nominated.

Suppose that your top five choices are A, B, C, D, and E (in any order), with F as your sixth choice. You are certain that a lot of people are going to put A on their ballot.

Case One: You are sure that A is going to get far more nominations than any other candidate.
In this case, it’s safe to leave A off your ballot. Under AV, it’s probably better strategy to list BCDEF, on the theory that you’d like a final ballot with A, F, and three things you hate better than one with A and four things you hate. Under RAV, listing either BCDEF or BCDE could make sense, depending on how strongly you feel about F.

Case Two: You are sure that A will be among the top 2 to 4 vote-getters or so.
Under AV, it’s still good strategy to vote BDCEF. A doesn’t need your vote to make the ballot, and as with the previous scenario, you might prefer that A and F get nominated over a scenario where A is your only choice that gets nominated.

Under RAV, voting BCDEF or BCDE can help you, but it can also hurt you, because the order in which candidates are selected matters. If your withholding a vote for A means that G gets nominated before A, it’s possible that adjusting weights for G’s selection could hurt the standing of A and cause it to drop out of the nominee list. In fact, ballot configurations exist where you voting for ABCDE causes both A and B to be nominated, but voting for BCDE or BCDEF causes none of your choices to be nominated.

This scenario has low probability, but the scenario where you having 1 vote for B instead of 0.5 vote determines whether B gets nominated also has low probability. And in the absence of complete information about how everyone else is voting, you cannot determine which scenario is more likely. Therefore, there is not a clear advantage to voting against your true preferences.

One issue here, though, is that the strategy of withholding votes from more popular works can appear to be best in the absence of a more sophisticated analysis. We would need to get the word out that it’s not the case, but some people would probably still follow their instincts to leave off their perceived frontrunners.

(This leaves aside that AV also has a dominant strategy of “join the largest bloc,” as we’ve seen.)

On a side note, the statement that there is no strategy with STV/IRV (in your last sentence) is incorrect. It’s more strategy-free than first-past-the-post (or bloc approval), but there’s a significant spoiler effect still present (where it can be tactically better to list your second choice first if your second choice is likely to perform better against your last choice). More detail is here. It also has some vulnerability to a tactic called “push-over.”

All: just so you know, I’m in classes all day today. I wrote this in the early morning and am hoping to steal a minute to get it posted after the thread reopens, but if people have questions about any of this, I probably won’t be able to read and reply until tonight.

#155 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 01:25 PM:

nathanael@124, 125: I absolutely agree with your conclusions. But I think you’re overstating the consensus in the field a bit. And I’d like to see your references.

… I did this for AD_FV, so I’ll do it for nathanael to be fair. Nathanael seems to be knowledgeable about voting theory and seems to be basically on “my team”. Election theory is a small world — I recognized Clay and AD_FV immediately, and I think would have strongly suspected I knew who they were even if their names hadn’t been a giveaway. However, I don’t recognize nathanael by name or style. And again, I agree with him, but personally I’d tend to qualify my statements a bit more.

“How would you explain how the thing with the most votes lost”: that would never happen. The thing with the second-most votes might lose. In that case, I’d say something like what Cheradenine@142 suggests.

Brad@152:I think RAV is one of the easiest ones to be strategic on,

Actually, that’s pretty much the opposite of the truth. That is to say: there are various ways you could precisely define “easiest” and “be strategic” in that sentence, but for the definitions I find most natural, I believe you could prove that RAV is one of the hardest to be strategic on, especially in the kind of situations I think would be common in Hugo voting (where overwhelmingly dominant winners are rare). Certainly it’s harder to use strategy in RAV than in STV.

I’m not saying that the strategy you’re suggesting here would never work; something like that kind of strategy is possible in any PR system I know of, and I bet you could rigorously prove that it’s possible in any deterministic, Droop-proportional system with fixed numbers of seats and finite candidates.

But if your preferences are ABCDE in that order, and you think C is going to be nominated without your vote, under RAV you might decide to vote AB, or you might decide to vote ABCDE, but voting ABDE is a very risky strategy; I think it would be attempted rarely and successful basically never. If everybody uses the strategy where they vote AB in that situation, I think that leads to a pretty good chance of a good slate of nominees; perhaps even better by some diversity criteria than if everybody votes ABCDE. And in the long run, people would learn that their prior “certainty” that C will be nominated is often unwarranted, so plenty of people will give up on such finicky strategies and just vote ABCDE.

@153: Under that system, a slate could take 4 nominees in each category by just telling all their members to find random obscure works to put first.

And yes, I’m sure you can come up with a way to fix that pathology. But you’ll only create some other pathology. Patching problems is not a good way to design voting systems. And I trust a system whose problems I understand much more than one which seems too perfect. I think I understand the issues with RAV, and they’re minor (as I argued just above with Brad).

In a larger sense, I appreciate what you’re trying to do. You want a system that is easy to explain in terms of simple, easy-to-grasp principles. The trouble is, well-designed voting systems involve finding elegant solutions to various trade-offs between competing values. (In fact, you could say that that’s what a good voting system is; a way for groups of people to find a good trade-off among their different values). If you look at a voting system “from only one side” — in terms of only one basic principle — it’s always going to look a bit flawed.

I’m a grad student in statistics. When teaching statistics to undergrads, this kind of problem is common. There’s a lot of times when some result is based on more than one abstract principle, and it’s really hard to “get” it intuitively because without some practice, your naive intuition can’t see it from enough sides at once.

So what’s the kind of practice that gets you there? Concrete examples. In the case of voting theory, voting scenarios. I could certainly make a set of simple examples that shows how RAV works and why it’s a good thing that it works that way.

But there’s a danger with that. No voting system is perfect, and if you want to make some system look bad, it’s always possible to create a scenario that does so. So from the point of view of a non-expert, there’s always going to be competing experts with competing systems and scenarios and arguments. Each person has to decide for themselves how much of that to carefully decipher and analyze, and how much to tune out. For most people, they’ll start tuning out pretty quickly. So if they want to have an opinion, it’s going to be based on whom they trust and whom they don’t.

As an expert, how do I get people to trust me? I have some ways, but I’m not going to pretend I have it all figured out. I’m upfront about my biases. I try not to overstate my case, but I also try to avoid making overly-technical qualified statements that sound like fine print. I try to say things in a way that I think a non-expert could understand, but that would also make sense to an expert; I’m sure I often err on the side of the latter. I try to give people good heuristics for detecting fake experts; for instance, if somebody tells you their voting system is perfect, or they that this one precise scenario shows that some other voting system is bad, don’t trust them. Look for simple, realistic scenarios, along with some analysis of how they could vary and/or go wrong.

So… anyway, as yet another “nothing up my sleeves” gesture, I’m going to invite others to create simple scenarios to show off RAV. I imagine it would look at the first three winners in simple elections with

#156 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 01:47 PM:

Grr, another borked comment.

Here's another try:

nathanael@124, 125: I absolutely agree with your conclusions. But I think you’re overstating the consensus in the field a bit. And I’d like to see your references.

… I did this for AD_FV, so I’ll do it for nathanael to be fair. Nathanael seems to be knowledgeable about voting theory and seems to be basically on “my team”. Election theory is a small world — I recognized Clay and AD_FV immediately, and I think would have strongly suspected I knew who they were even if their names hadn’t been a giveaway. However, I don’t recognize nathanael by name or style. And again, I agree with him, but personally I’d tend to qualify my statements a bit more.

“How would you explain how the thing with the most votes lost”: that would never happen. The thing with the second-most votes might lose. In that case, I’d say something like what Cheradenine@142 suggests.

Brad@152:I think RAV is one of the easiest ones to be strategic on,

Actually, that’s pretty much the opposite of the truth. That is to say: there are various ways you could precisely define “easiest” and “be strategic” in that sentence, but for the definitions I find most natural, I believe you could prove that RAV is one of the hardest to be strategic on, especially in the kind of situations I think would be common in Hugo voting (where overwhelmingly dominant winners are rare). Certainly it’s harder to use strategy in RAV than in STV.

I’m not saying that the strategy you’re suggesting here would never work; something like that kind of strategy is possible in any PR system I know of, and I bet you could rigorously prove that it’s possible in any deterministic, Droop-proportional system with fixed numbers of seats and finite candidates.

But if your preferences are ABCDE in that order, and you think C is going to be nominated without your vote, under RAV you might decide to vote AB, or you might decide to vote ABCDE, but voting ABDE is a very risky strategy; I think it would be attempted rarely and successful basically never. If everybody uses the strategy where they vote AB in that situation, I think that leads to a pretty good chance of a good slate of nominees; perhaps even better by some diversity criteria than if everybody votes ABCDE. And in the long run, people would learn that their prior “certainty” that C will be nominated is often unwarranted, so plenty of people will give up on such finicky strategies and just vote ABCDE.

@153: Under that system, a slate could take 4 nominees in each category by just telling all their members to find random obscure works to put first.

And yes, I’m sure you can come up with a way to fix that pathology. But you’ll only create some other pathology. Patching problems is not a good way to design voting systems. And I trust a system whose problems I understand much more than one which seems too perfect. I think I understand the issues with RAV, and they’re minor (as I argued just above with Brad).

In a larger sense, I appreciate what you’re trying to do. You want a system that is easy to explain in terms of simple, easy-to-grasp principles. The trouble is, well-designed voting systems involve finding elegant solutions to various trade-offs between competing values. (In fact, you could say that that’s what a good voting system is; a way for groups of people to find a good trade-off among their different values). If you look at a voting system “from only one side” — in terms of only one basic principle — it’s always going to look a bit flawed.

I’m a grad student in statistics. When teaching statistics to undergrads, this kind of problem is common. There’s a lot of times when some result is based on more than one abstract principle, and it’s really hard to “get” it intuitively because without some practice, your naive intuition can’t see it from enough sides at once.

So what’s the kind of practice that gets you there? Concrete examples. In the case of voting theory, voting scenarios. I could certainly make a set of simple examples that shows how RAV works and why it’s a good thing that it works that way.

But there’s a danger with that. No voting system is perfect, and if you want to make some system look bad, it’s always possible to create a scenario that does so. So from the point of view of a non-expert, there’s always going to be competing experts with competing systems and scenarios and arguments. Each person has to decide for themselves how much of that to carefully decipher and analyze, and how much to tune out. For most people, they’ll start tuning out pretty quickly. So if they want to have an opinion, it’s going to be based on whom they trust and whom they don’t.

As an expert, how do I get people to trust me? I have some ways, but I’m not going to pretend I have it all figured out. I’m upfront about my biases. I try not to overstate my case, but I also try to avoid making overly-technical qualified statements that sound like fine print. I try to say things in a way that I think a non-expert could understand, but that would also make sense to an expert; I’m sure I often err on the side of the latter. I try to give people good heuristics for detecting fake experts; for instance, if somebody tells you their voting system is perfect, or they that this one precise scenario shows that some other voting system is bad, don’t trust them. Look for simple, realistic scenarios, along with some analysis of how they could vary and/or go wrong.

So… anyway, as yet another “nothing up my sleeves” gesture, I’m going to invite others to create simple scenarios to show off RAV. I imagine it would look at the first three winners in simple elections with <10 voters and candidates, starting with one where there are three works with separate, equal support, and progressing to cases where the supporters of two of the works explicitly ally against the third.

If nobody else makes scenarios, I'll do it later.

#157 ::: J Thomas ::: (view all by) ::: April 14, 2015, 02:14 PM:

@142 Cheradenine

"The current system is designed to represent as wide a group of fans as possible on the final ballot. The final ballot includes choices from 52% of the ballots that were submitted. Using the system you're suggesting, only 24% of the ballots would have any representation."

Thank you! I doubt I could make a better response even spending significant time on it.

The central point is that the nominations are not an election. People who think of them as an election will want the ones with the most votes to win. But they are intended to do something else.

I don't have it exactly straight what they're supposed to do, which leads me to propose doing without separate nominations. It used to be, we didn't have to define what it was for, we could just do it and get an adequate result. For a long time there weren't so many subgenres, they weren't big enough to credibly argue for more than one or two Hugo-level works in one, fans tended to read everything, etc. SF and fandom have gotten bigger and many fans have less catholic tastes, and this is the result.

We want diversity in Hugo nominations. We don't want the same people making all the nominations unless they suggest enough diversity. We don't want too many nominees from the same subgenre unless that's what the consensus is among the fans. Something like that.

We want a variety of nominations that represent SF. They should all be high quality, and should give a sense of what fandom is about. Yes!

Ideally, when two works that are too similar get nominated, one of the authors will decide that his own work is inferior and will decline so that we can get more diversity in the nominations. Of course we can't expect that, but that's the spirit we're looking for.

The nominations are not an election to pick the best. They are an attempt to choose a diverse group of great SF, so that each member of the group is worthy to represent SF and fandom.

It follows that the following election is not an election to choose the best SF novel etc of the year. The nominations are intended to provide a diverse group of high quality work, not to choose the five best. So the election among them does not imply that the winner is the best.

I started off not knowing what the nominations were for, and now I don't know what the election is for. I guess that's progress.

#158 ::: J Thomas ::: (view all by) ::: April 14, 2015, 02:29 PM:

@153 Keith "Kilo" Watt

Here's a proposed system:

Everyone is told that at the end of the day they will be able to nominate no more than one nominee per category (and possibly none, if there is insufficient support for all of your preferred candidates). The goal is to give you the best chance of having one of your preferred candidates make the ballot, and preferably, one that you ranked highest on your list.

I found your explanation a little bit unclear. I'll repeat it in my own words:

Each voter lists up to 5 choices in order of preference.

Then the votes are tallied by acceptance voting. Each of your 5 choices counts at this stage. A winner is found.

If the winner is your first choice, your ballot counts this time but is then discarded. If you listed it #2 through #5, your vote is not discarded. If none of your choices won, of course your vote is not discarded.

Do the same thing again, ignoring the known winner. A second winner is found. If this winner is your current top choice, your ballot is discarded after the vote. Otherwise not.

Repeat this until there are five winners.
So you get up to five votes. When your best remaining choice is selected, you're done.

The advantage I see for this approach is that you don't lose your vote if your fifth choice is selected early. Regardless what happens with the votes you care least about, your vote is not done until you have won the one you most care about, that can win.

A slate will almost certainly get one nominee on the ballot -- but only one.

If I understand you, no. If the slate voters arrange the same 5 names in random order, when the first one wins about a fifth of the ballots will be lost. Then if they win a second time, another fifth will be lost. If they lose the second time and win the third time, they lose the second fifth of their votes then and are down to 60%. They get as many nominations as their slowly-dwindling ballots happen to get.

If they were devious, they might try to have all of them vote in the same order, but have two voters who vote only for the last, two who vote for the last two, etc. Then their last choice wins and they lose two votes, their next-to-last choice wins and they lose two votes, etc. But it takes a lot of organization to make it go smooth.

I'm immediately curious what happens if there are two (or more) slates.

Say there are two slates, one with 20% of the votes and the other with 15%. Both with votes in random order. Assume that nothing else has more than 10%. Then the first slate wins one, and is cut down to 16%. It wins again and is cut down to 12%. The second slate wins one and is cut down to 12%. One of them wins and is cut down to 8% or 9%. The other wins one. If there was a sixth round then the first non-slate choice would win.

#159 ::: P J Evans ::: (view all by) ::: April 14, 2015, 03:11 PM:

Ideally, when two works that are too similar get nominated, one of the authors will decide that his own work is inferior and will decline so that we can get more diversity in the nominations.

Why do you think this should happen? (Unpack your assumptions about why you think people should be doing it your way: we can't actually read minds.)
Nominations are for works, unless otherwise specified. It's quite possible for someone to have read one for those two works but not the other, and they'd nominate the one they read.

#160 ::: J Thomas ::: (view all by) ::: April 14, 2015, 03:29 PM:

I got inspired by a misreadng of Keith "Kilo" Watt's proposal and made another.

I call it Reverse Single Acceptance Voting. It goes like this:

The purpose of the Hugo nominations is to present a variety of excellent works that are worthy to represent Worldcon, fandom, and SF. There is no guarantee that the best works will be nominated.

A nominator nominates as many titles as he wants, maybe up to 5, maybe more – I'm not interested in that limit, do it how you want. Acceptance voting, he needs to think each of them is worthy to represent Worldcon etc.

The voting program adds up all the votes for all the nominations. It then selects the nomination with the fifth most votes and adds it to the final nomination list. Then all ballots that include a vote for that title are discarded.

The program adds up all the votes for the remaining nominations. It selects the nomination with the fourth most votes and adds it to the final nomination list. All ballots that include a vote for that title are discarded.

Etc.

When explaining to voters, here's what it means. “You basicly get to vote once. Your vote will go to the one of your nominations that needs it the most to win.”

Side effects: The one that gets the most votes may not win. It starts out with the most votes, but then maybe it loses some when the fifth place winner is chosen, and maybe it loses more when the fourth place winner is chosen, and maybe it doesn't have the second-most when we pick the second place winner, or the most when we pick the first-place winner. This is OK. We have made diverse choices that represent us well. Each of them got pretty many votes that didn't overlap too much. When the most popular choice overlaps too much with other good ones, we're better off with them instead.

Slates: A big slate will get one win. A smaller slate may get one win. Slates might as well limit themselves to one candidate each because if they do two they can't predict how their vote will be split. If a slate nominates only one candidate, and has the most votes at the beginning, it will still have the most votes at the end when we select the first place. Meanwhile a smaller slate might be in fifth place when we look for the fifth place, fourth place when we look for the fourth place, etc. You just don't know.

Strategic voting: It does you no good to avoid voting for a popular work so your vote will count for a less-popular one. If your less-popular vote can win in fifth place or fourth place or whatever, it will. If it can't win, it won't. You might however consider whether you want the vote for a less-popular work to have the chance to interfere with your vote for the popular one. If it does win, your popular vote is lost. Only vote for more than one if you are willing to support whichever one needs the support more.

I am sure that this approach is better than the version which is otherwise just like it, but which chooses the most popular one first. For that, imagine the following extreme case:

Terry Pratchett's last posthumous book gets 80% of the vote. 20% of the ballots remain.

Something really good gets 40% of the 20% to get second place. 12% of the ballots remain.

Something else good gets 25% of the 12%, leaving 9%.

Something you probably wouldn't expect gets 44% of the 9%, leaving 5%.

That last 5% of the voters, the ones who don't like Terry Pratchett or any of the other stuff a lot of people thought was good, may be extremely diverse. If there are five votes for Brick Bronson the Bulletproof Commando and only four for Little Black Sambo at the Lesbian Orgy then the Hugos have dodged a bullet.

If we're going to dilute people's votes when they've already won something, then there's something to be said for STV which can minimize the votes lost, or maximize them, or anywhere in between. It is, however, complex and hard to explain. With my proposal your vote is completely gone after it wins something, but the number of votes lost at each step is minimized. We get five nominations based on uncorrelated votes, and each of them will tend to be a reasonably large number of votes.

It's simple to say what you're doing, and simple to do it. Not quite intuitive, unfortunately. If the numbers come out wrong you can get a bad result, but that's true of any system.

I expect it would be hard to game because your less-popular choices affect the chances of the popular ones instead of vice versa, and it's harder to guess just how unpopular your less-popular choices are.

#161 ::: Cheradenine ::: (view all by) ::: April 14, 2015, 03:30 PM:

Kilo @153: Ugh, I managed to miss the two times you said that all votes on the ballot counted and draw the conclusion that only first-place votes were counted. If I'm understanding it correctly this time (I swear I read it multiple times!), the problem is that there's no actual advantage to you of ranking your favorite work #1. It doesn't make that work any more likely to win than if you put it #2; it just creates a condition when your ballot stops counting. In a lot of cases, you'd be better off to rank the words you just scrawled on your napkin (i.e. something no one else will nominate) #1, and your favorites in spots #2-#5. (If everyone does this, it's equivalent to the current system with 4 nominations per person instead of 5.)

Sorry to misread you the first time! (Back to class now.)

#162 ::: Andrew M ::: (view all by) ::: April 14, 2015, 03:57 PM:

Teresa@96: You may well ask. I suppose Among Others belongs to the 'interesting and distinctive work' subgenre.

Cheradenine@80: Well, it may be that you wouldn't have much of a problem if votes were distributed over 8-10 works; but then in a thin year it might be fewer; Patrick noted near the start of this discussion that this was arguably a very strong year, and so votes might be more divided than they often are.

But I do think there is a general problem with carrying over assumptions from political voting. There, it's reasonable that voters will tend to clump around particular kinds of candidate, and so approval voting does a good job of counteracting the effects of clumping, and giving more voters representation. Some comments here seem to assume that the same is true in the Hugos, and the point is to resolve conflicts between fans of different kinds of work. But in fact voters themselves may well be seeking diversity, in which case a model derived from politics won't fit them. It's a rare voter who decides to vote for one conservative, one liberal, one socialist, one libertarian and one green, in the interests of diversity; it's not at all surprising that someone should vote for one work of epic-ish fantasy, one space opera, one piece of near-future SF, one this-worldly fantasy and one alternate history, or the like.

#163 ::: J Thomas ::: (view all by) ::: April 14, 2015, 04:05 PM:

"Ideally, when two works that are too similar get nominated, one of the authors will decide that his own work is inferior and will decline so that we can get more diversity in the nominations."

Why do you think this should happen? (Unpack your assumptions about why you think people should be doing it your way: we can't actually read minds.)

Nominations are for works, unless otherwise specified. It's quite possible for someone to have read one for those two works but not the other, and they'd nominate the one they read.

OK, I'll try to write in more detail.

Originally I thought of the nominations as being like an election. In an election, I want the ones with the most votes to win. When you throw away some votes but not others, when you say that some votes count as one vote each while other votes count as two votes or three votes, it sounds like corruption.

But the Hugo nominations are not an election with five winners. The final Hugo election is an election, and we want to everybody who votes to have read all the alternatives. We cannot possibly expect the nominators to have read all the alternatives. The various things that make the final election an election, are not present in the nominations. For example, it's perfectly OK in the nominations for the entry with the second-most votes and the third-most votes etc not to be chosen. In an election that would be vote-stealing. But the nominations are something else.

I looked at what people here said the nominations are. Every winner ought to be high-quality. And they should be diverse. They should get votes from diverse nominators -- we don't want the same small majority or small-but-focused bloc to nominate all the winners. And it would be rare that we would want them to all be space operas or all be sword-and-sorcery or all be hard as ten-point steel.

We aren't choosing them in isolation, the five individual best nominations we can find. We want a diverse group of excellent works. Things that people can use as a reading list. A canon, maybe.

I got the idea that the purpose of the nominations is to create a short list of excellent SF that represents the field. "This is what Worldcon is about." "This is what fandom is about." "This is what SF is about."

So we nominated Rite of Passage and Protector, but not the same year. Dune and Courtship Rite -- different years. A Civil Campaign and To Say Nothing of the Dog -- different years.

We want a diverse showcase of what SF has to offer, and if two excellent works are too similar in one year, probably one of them will lose out.

It is not a contest to pick the best five. If it was that, then the ones that got the most votes would be the ones that ought to win. If Planetbusters gets 300 votes you don't replace it with something that got 80 votes, just because you already accepted Starsmashers and Moonwhompers.

We are choosing a diverse collection of excellent work, and to do that we want work that is chosen by a diverse collection of nominators.

Are my assumptions clearer now? I could be wrong about what "we" want, but this is the explanation I see that seems to fit best so far.

#164 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 04:06 PM:

J Thomas: that would be crazy. Since it's basically impossible to predict what might end up in 5th place and suck up your vote, a strategic voter would be highly motivated to bullet vote. And then it reduces to plurality — the worst system.

If you want something like this, you use SDV, except transfer overvotes. So:

Vote for any number
while more than 5 candidates remain:
Divide each vote into n "shards" for each of the remaining candidates on that ballot
For i=1 to 4
redistribute any excess that the 1 through i'th top candidates have above the amount the (i+1)th candidate has
Eliminate the lowest candidate

It's SDV, but doesn't waste any "overvotes" on strong candidates. In other words, your ballot will tend to give more of its weight to the candidates that need it more to win.

Not hard to understand conceptually, but it would be utterly impossible to carry out without a computer. But it would be pretty easy to see if the computer were "cheating" somehow.

#165 ::: Derek Balsam ::: (view all by) ::: April 14, 2015, 04:06 PM:

Re: option 2

The minimum number of nominees is three, not five, though five is the usual number. See the WSFS constitution:

> 3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.

#166 ::: felice ::: (view all by) ::: April 14, 2015, 04:14 PM:

Cat @140: "One of the questions that Joshua brought up I haven't seen answered yet. Sometimes nominees decline the nomination. Is the ballot recalculated from there? If it is, could that lead to some second nominee that did make the ballot the first time failing to make the ballot the second time? Because that's the kind of thing that could produce hard feelings, I think."

If the calculation is rerun as Joshua suggests, it's only used to find the highest ranked nominee not in the current top 4; whether or not the current top 4 are still all in the recalculated top 5 is ignored. It's not a perfect solution - if the new winner would have pushed one or more of the others off the ballot, then it means the nominators who supported the works that would have been pushed are getting a disproportionately high number of their preferred works shortlisted - but it's better than throwing away the nominations of everyone who supported the declined work, and withdrawing a shortlisting because of recalculation really isn't an acceptable option.

If someone known to be likely to decline ends up on the ballot, ask them first before contacting any of the other nominees! Then if necessary the entire ballot can be recalculated, because none of the others on the original shortlist have been contacted yet.

#167 ::: J Thomas ::: (view all by) ::: April 14, 2015, 04:43 PM:

@164 Jameson Quinn

J Thomas: that would be crazy. Since it's basically impossible to predict what might end up in 5th place and suck up your vote, a strategic voter would be highly motivated to bullet vote. And then it reduces to plurality — the worst system.

Jameson Quinn, I don't see it that way, but that does not mean you must be wrong. On the other hand you probably haven't thought about it as much as I have. On the third hand, you have more experience with voting systems in general than I do. On the fourth hand, maybe we find out what it does better by testing.

Here's what I think: If there is only one work you care about winning, then of course you should only vote for that one. This is true in any voting system.

If you care equally about five different works, then it makes sense to vote for all five of them. You don't know which of them your vote will be cast for. But you know that it is the one which can win, that has the least support.

If it turns out that the one you think is a long shot actually has no hope, then you have lost nothing by voting for it. (Unless you are limited to five slots and there was another fifth choice you wanted.)

If it turns out that your vote helps to put your long-shot choice into fifth place, then your vote made a difference!

You made more difference bringing that into fifth place, than you would if your vote helped put something else into first place that would otherwise be in second place.

But you don't lose anything by voting for the popular one, if it is in fact popular, because it will have no effect unless your long shot loses. And it just might turn out that the one you thought was popular wasn't that popular, and your vote might be the one that lets it take fifth place.

So my strategy is "Vote for the ones you think ought to win." Your vote will benefit one of them (provided any of them make it). It will help the one that can win, that has the least number of votes.

#168 ::: Brad from Sunnyvale ::: (view all by) ::: April 14, 2015, 04:51 PM:

Hmm, is it a requirement the nominees be diverse? For that, you want a jury, I suspect.

When I say that strategy for some of these ballots is "easy" I mean it's not hard to figure out what your strategic ballot would be. You take the clear leader and leave it off your ballot (or perhaps 2.) That's also the easy strategy in AV.

The difference is, with AV, removing the leader from your ballot buys you very little. It buys you some support for your 6th place choice. That's such a minor win that most people will not be strategic -- which is good.

With some of the systems proposed here, including RAV, leaving the popular choice off your ballot buys you much more. It buys you improved support for all your other choices and in particular, it buys you improved support for what may be your top choices, not your 6th choice.

This is a really, really big difference. So big a difference that I think it's one of kind, not degree. Because if you are going strategic, you balance the cost (the risk that one of your choices which you think is a shoo-in doesn't make the ballot because you didn't support it) against the benefit.

With AV, the cost is low but the benefit is so low that I doubt many people even think of it. With RAV, you may argue the risk is slightly higher, but the benefit is so much more that many would consider it.

Now, I don't think for a moment that the majority of voters will go strategic. But I think a large enough fraction would that it compromises the accuracy of the nomination ballot.

All of this logic is complex, but the instructions to the voter who wants to go strategic are simple:

"Are there choices you are confident are shoo-ins? Leave them off your ballot, it will help your other top choices a lot, and even let you add some extras. Don't worry, only about (small percentage) of people do this, so you won't hurt the shoo-in."

Easy to do and understand but bad.

Some of the other methods are worse. In the approaches that give more weight to each candidate if you list fewer than the maximum, this is even stronger. "Are your top two choices a shoo-in and a more obscure work? Definitely nominate only the more obscure work, you will be doubling your support for it."

And as I've pointed out, all of the above approaches in option 1 2 and 3 still mean 1 or 2 slate candidates get on in each category. We've been shocked by a slate sweeping a category, but I think 2 out of 5 being slate choices is also quite bad, if not at the level of calamity.

#169 ::: albatross ::: (view all by) ::: April 14, 2015, 04:52 PM:

I think all these schemes (RAV, STV, AV) give you an ordered list, assuming you don't have any ties. So it seems like the natural way to handle declined nominations with a minimum of hard feelings and confusion is something like:

a. Run the vote-counting process to get the top 10 or more candidates.

b. Announce the top 5.

c. If someone declines, go to the next person on the list.

Is there some reason this isn't the right way to do things? The ballots were cast with the belief that this nominee was in the running, so it seems like the other ways to handle this situation mean that people cast ballots that now have to be treated in ways the voters never expected--like ignoring or removing some nominee from all the ballots and rerunning the election.

Of course a bigger problem is that when someone refuses a nomination, you've probably already called the other nominees. If the vote counting scheme could conceivably change who the other winners are based on one candidate being removed (so that taking Alice off the ballot knocks Bob from 2nd place to 7th place, and thus keeps him from getting a nomination), you've set up a situation where someone is going to feel, pretty understandably, like he got screwed out of a Hugo nomination. (And so will people who nominated him.)

#170 ::: felice ::: (view all by) ::: April 14, 2015, 05:09 PM:

albatross @169: "If someone declines, go to the next person on the list.
Is there some reason this isn't the right way to do things?"

Yes. It depends on the counting method (it's not a problem under the current system), but generally, it means discarding (at least part of) the nominations of everyone who supported the candidate who declined nomination - potentially a very large percentage of nominators. That means everyone who didn't like the declined nomination gets disproportionate influence. Rerunning the calculation allows their second choices to be counted instead.

#171 ::: albatross ::: (view all by) ::: April 14, 2015, 05:14 PM:

Assume that 2016 is the year of the Hugo Slates. Let's assume there are a half dozen big slates and several smaller ones, some using SP-like ideology, some trying to ensure that their favored category, which never seems to win a best novel, gets in this year, others trying to campaign for a particular writer they think has been unfairly overlooked.

My intuition here is that in this case, the slates will largely dominate the nomination almost no matter what we do, because bloc-voting gives you more of a voice, and who doesn't want more of a voice? It would be easy for the slates to capture a majority of the nominators--not any one slate, but one of the top half-dozen.

RAV and Ron Rivest's original scheme from the previous thread make it hard for a single slate to dominate. But suppose you have six slates, each with a substantial fraction of people, each voting for more-or-less disctinct things. The SPs get Manly Men In Space With Blasters on the ballot, the SJPs[1] get Ass Kicking Hermaphrodite Subverts the Patriarchy on the ballot, the SWVPs[2] get Sexy Sparkly Vampires Fall In Love with Misunderstood Tween on the ballot, the TCPs[3] get Atlanta Nights on the ballot, etc.

Each slate recommends only voting for their category of works--perhaps single nominations, perhaps simply "vote for our kind of work, especially this one, certainly not one of those evil selections by the wrong kinds of puppy."

We *still* don't get everyone voting their conscience. The slate voting still seems likely to dominate the individual votes, even with mechanisms to weaken any single slate. The SPs can't run the table anymore, but any slate that coordinates will be more likely to get their nominees onto the list of finalists.

[1] Social Justice Puppies
[2] Sexy Wereworf and Vampire Puppies
[3] Tor Conspiracy Puppies

#172 ::: Buddha Buck ::: (view all by) ::: April 14, 2015, 05:15 PM:

albatross @169:

I think treating the selection method as a way to get a fixed ordered list of all nominees and picking off the top until there are 5 accepted nominees is about the only way to go.

Worse than someone going "if the balloting had been rerun without Tom, who declined, I would have been nominated" is the unenviable phone call of "Um, yeah, you know how we said you were the #2 nominee? Well, the #5 guy dropped out, and re we re-ran the nomination software without him, and, um, you came in 6th..."

#173 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 05:21 PM:

@167: Sorry, as a person who's spent thousands of hours writing about, analyzing, testing, and programming voting systems, I'm pretty sure that your proposal doesn't work. I can think of several reasons why not. I don't think it's worth our time to go through the reasons one by one.

But I do like the basic idea. If you avoid the pathologies, it's actually quite similar to SDV in its philosophy. SDV is not my favorite proposal, but in light of your arguments for yours, it's actually growing on me. In my last message, I proposed a complex modification to SDV to make it more like what you want. I now have a simpler one:

Divvy up the votes, a la SDV.
Of the two works with the smallest amount of divvied votes, eliminate the one which is on fewer ballots.
Repeat until there are 6 candidates left
Take the top 5.


... This system is actually a tiny bit friendlier to slates than vanilla SDV; but on the other hand, there's less of an incentive to "leave off popular candidates" strategy. It's also, maybe, easier to explain philosophically than RAV. I still would choose RAV as my first choice, but the above is good. For a name, how about Fractional Approval with Popularity Eliminations (FAPE)?

#174 ::: nathanbp ::: (view all by) ::: April 14, 2015, 05:22 PM:

Brad from Sunnyvale @168: On diversity, you're right that selecting a diverse group of nominees (from different genres of SF&F) is not the goal of changing the voting system. Instead, it's to select a system that will take into account the votes of as diverse a group of voters as possible. I think in this discussion at times we've used the first as shorthand when we actually meant the second.

albatross @169: I see a couple choices for picking the next nominee when a nominee refuses a nomination.
1) Offer the nomination to the 6th (etc.) winner of the election.
2) Cross the nominee that refused off everyone's ballots and rerun the election.
2a) The new nominees are the top 5 returned (which may in extreme cases not match any of the old nominees)
2b) The new nominee is the top result from the new election that was not previously nominated

Under the current Hugo voting system, (1), (2a), and (2b) are all identical. Under most of the systems we've been discussing, they are not. For example, under (1), if the top candidate from a slate refused the nomination, that could (depending on the exact system in use), result in all of that slate's voters having no voice in who is nominated. No matter what you think about slates, I don't think that's really fair. (2a) seems the most fair to the voters to me, but could result in very awkward phone calls withdrawing someone's Hugo nomination. Under RAV, I believe (2a) and (2b) are identical, and if nominees are phoned in order would never result in a nomination being taken away.

#175 ::: J Thomas ::: (view all by) ::: April 14, 2015, 05:30 PM:

@W172 Buddha Buck

Worse than someone going "if the balloting had been rerun without Tom, who declined, I would have been nominated" is the unenviable phone call of "Um, yeah, you know how we said you were the #2 nominee? Well, the #5 guy dropped out, and re we re-ran the nomination software without him, and, um, you came in 6th..."

Don't call the people and say "You came in #2.". Call them and ask, "If you got onto the final ballot at the Hugo, would you accept?".

Far more work but better if you can handle it, call them earlier and say "You got a preliminary nomination for the Hugo, would you accept it if you got onto the final ballot?" If they say no then you can take them off the list and warn voters not to nominate them because they won't accept.

#176 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 05:52 PM:

me@173: actually, it's simpler to just say "repeat until there are 5 candidates left".

albatross@171: Say the competing slates had 30%, 20%, and 10%, leaving 40% non-slate voters. If there was a widely popular work that year that wasn't on any of the slates, it could still easily beat out the 10% slate. It is also possible that the non-slate voters would be frozen out.

But I think most fans would rather not vote a slate. So I suspect it would be more like 20%, 15%, and 8%. At that point, there's nothing really "slatey" about the 8% slate; they might as well be just similarly-minded voters, which has existed since forever.

#177 ::: felice ::: (view all by) ::: April 14, 2015, 05:56 PM:

Can we call publishing the unranked top 15 once halfway through the nomination period Option 5b?

Cheradenine @108: "Felice @105: I fear that in a dispersed election, giving the top 15 at the halfway point will be misleading if anything. Looking at past ballots, the difference between the vote counts for 5th place and 16th place, with (probably) less than half the vote in, would quite likely be less than ten votes. But if most late voters choose one of the top 15, it will skew the election towards those that got a few more early votes. This would increase the power of campaigning and slates."

I'd expect more people to nominate before the halfway point in order to get their first choices published in the top 15; it's a new deadline that would motivate people to nominate earlier than they do now. And if the difference is less than ten nominations, it's less than ten nominations; Hugo winners have been decided by less than 10 votes before. Most people who care about the Hugos don't want to support slates; and a Puppy-style slate that gets in the top 15 will just encourage people to nominate the other ten candidates. And that's before you take into account any other measures to reduce the impact of slates, such as the alternative counting methods most of this thread is discussing. Campaigning might help get a work in the top 15, but if it doesn't have genuine broad support, it's unlikely to make the final five, and increasing the total number of nominations contributing to the final ballot strongly counters the effects of slates and campaigns.

Cheradenine @115: "My issues with continuous voting basically boil down to: (1) implementation difficulties, (2) increased ability for bad actors to wreak havoc, and (3) that it basically embraces making the Hugo award a political process with campaigning, negotiation, arguments about who needs to compromise to push so-and-so out of the top 5, etc."

Most of that isn't an issue with Option 5b.

Emily H. @120: "Any way to increase coordination on the part of voters seems to encourage voting for what you haven't read but your friends say is good, or for what you only mildly liked but stands a decent chance of winning, and I think that's more broken than the status quo."

More broken than letting the Puppies sweep entire categories? You're right about the issues with increasing coordination, but it's also getting more people's opinions taken into account, which I think is a good thing even if they're not as informed as would be ideal. The problems you mention already exist to some extent in the final voting - not everyone reads/watches every nominee in every category they vote for. And knowing the midpoint top 15 does give people a month or so to investigate which candidates are deserving of their nominations - reading some that look of interest, checking the opinions of reviewers (not just friends).

#178 ::: J Thomas ::: (view all by) ::: April 14, 2015, 05:58 PM:

@171 albatross

Assume that 2016 is the year of the Hugo Slates. Let's assume there are a half dozen big slates and several smaller ones

As I understand it, the SPs this year were something like 15% of the vote. If we have six separate slates that are each 15% of the vote, that's 90%. They will dominate the elections because there will be nothing left.

#179 ::: felice ::: (view all by) ::: April 14, 2015, 06:12 PM:

J Thomas @126: "The interviewer starts his lead-in. "As we have all heard, for the majority of nominations for this prestigious award, the nominations with the most votes were thrown out and replaced by other nominations with a quarter -- or less than a quarter -- as many votes. For example, Jameson's Genociders, a story about patriotic soldiers winning a war on a desert world, was replaced by Mothra Faker, where a transsexual hobbit repeatedly tricks the sexists who try to get him pregnant. But the people responsible for this -- caught red-handed -- admit the facts but claim that what they did was the right thing to do and not corrupt at all. Here is Cheradenine to tell us their excuse."

How about "No, Jameson's Genociders didn't make the ballot, but all the same people also nominated Samson's Slaughterers, and that did get a place. Only 10% of the electorate nominated those works; once they got Samson's Slaughterers on the ballot, the other 90% got their turn, and Mothra Faker was the most widely supported amongst the 90%. You're not suggesting that 90% of voters should be ignored, are you?"

#180 ::: J Thomas ::: (view all by) ::: April 14, 2015, 06:16 PM:

@168 Brad from Sunnyvale

Hmm, is it a requirement the nominees be diverse? For that, you want a jury, I suspect.

I don't know what the requirements are. I've been trying to reverse-engineer the requirements from what people say they want and the voting systems they think encourage that.

They say they don't want slates. But voting systems can't tell what a slate is, they can only give less weight to ballots that are correlated. The more you and I agree about what to vote for, the less our votes count.

There are many fascinating ways to do that. If it's straight correlation that's the issue, then if you and I both vote A B C D E while Felice and Volts vote F G H I J both pairs get punished for agreeing. But if instead I vote A B C I J and Felice votes F G H D E then the four of us get punished less.

On the other hand, if A wins they could punish you and me for voting A, when the time comes to count up B C D and E.

I try to imagine this in a traditional context. "Brad, your Presidential candidate lost, so you get two votes in the Senatorial election. Meanwhile J, it appears your Senatorial candidate has lost so you get two votes for the congressman." People would scream bloody hell.

#181 ::: Brad from Sunnyvale ::: (view all by) ::: April 14, 2015, 07:08 PM:

I think the chaotic results when a candidate withdraws -- which could result in retracting a prior nomination -- are another sign of how complex these systems are, and thus how they fail the criteria I would set for them. (This is not fixed by saying you could just take the 6th place winner from the first run. The chaos is still present.)

It is possible to detect slates. With algorithms, but even better with algorithms and human oversight and jurisprudence. This is part of what leads me to option 4. Collusion should be banned. Any action (which is distinct from any pure speech like plain advocacy, which compromises the independence and honesty of the ballots is forbidden, and the committee should be able to correct it if they possibly can.

Yes, even though that could lead to mistakes and abuses. They would be fewer than the problems we are heading for.

#182 ::: J Thomas ::: (view all by) ::: April 14, 2015, 07:10 PM:

@173 Jameson Quinn

Sorry, as a person who's spent thousands of hours writing about, analyzing, testing, and programming voting systems, I'm pretty sure that your proposal doesn't work.

I notice that I do not respect your attempt to pull rank. Perhaps others might. When you give us useful ideas, and explanations that make sense to us, that's a valuable contribution.

But your field has changed a lot in the last X years and there's every reason to think it will change just as much in the next X years. You are trying to balance contradictory goals, and fashions change about which goals are worth balancing. You are trying to assist us when our statement of the problem we want to solve is basicly incoherent. So there are various reasons why your unsupported opinion may not fit my needs.

But I do like the basic idea. If you avoid the pathologies, it's actually quite similar to SDV in its philosophy. SDV is not my favorite proposal, but in light of your arguments for yours, it's actually growing on me.

Good! I'm glad our discussion is not useless to you.

Divvy up the votes, a la SDV.
Of the two works with the smallest amount of divvied votes, eliminate the one which is on fewer ballots.
Repeat until there are 6 candidates left
Take the top 5.

Interesting! If we did that without SDV it would be plain acceptance voting. In your hybrid system you first choose the two lowest when ballots that include some winners are devalued, and then you discard the one of the two lowest that has undevalued votes from the fewest people.

Earlier I presented a voting plan that would tend to stop slates. I didn't think to name it, and that looks important. I think I should call it Independent Voting.

The idea is, your ballot contains five nominees that will be handled with acceptance voting. But before the votes are counted, we adjust them according to how much they correlate with other ballots.

If your ballot has exactly the same five nominees as someone else's ballot, they both get a weight of zero.

If your ballot has four nominees that are the same and one different compared to someone else's ballot, they both get a weight of 0.2.

For three the same it's 0.4, for two the same 0.6, and for one the same 0.8. If your ballot is absolutely unique and nobody else voted for any of the same things, it gets a weight of 1.0.

We add up the votes with these weights and take the top five nominees.

Obviously, if you make up nominees that do not exist hoping to increase your weighting, your ballot should be disqualified.

With this system nobody in his right mind would attempt slate voting. That problem is solved. We reward independent voting, not going with crowds and cabals and slates.

I am not being sarcastic. Not entirely.

#183 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 08:42 PM:

@182: Notice that I "pulled rank" only after having cited a specific problem that you discounted.

Your voting system could fail to elect a candidate that was voted for by all but 1 voter. Obviously, that's an extreme, but less-extreme cases of an "obvious winner" who didn't get nominated would sour people on the system. Furthermore, all voters who prefer one specific work — anybody with a "favorite author" — will have a dominant strategy of voting for only that one. Anybody with wide-ranging tastes will contribute their judgment to helping only the least-popular of their choices. The way to get your work nominated would be to have fans who don't really read much. It's a bad system. Yes, there's the germ of a good idea there.

Experience with voting systems doesn't give you an infallible nose for a good system. SDV, for instance, is growing on me. But once in a while, you can smell a stink. I've learned that from having invented my share of stinker voting systems. It's no shame.

#184 ::: Keith "Kilo" Watt ::: (view all by) ::: April 14, 2015, 09:36 PM:

Jameson@183:

I'm sorry, but I don't see your counter-examples as being evidence that JT's proposal is bad at all. Maybe I just need more details.

Your first objection is that "a system could fail to elect a candidate that was voted for by all but one voter." Isn't that possible only if all the other ballots are identical? In other words, in the case that all the other ballots are slate votes? Isn't that precisely the effect we're looking for? Even though I'm not a specialist, I am a scientist -- my statistics background is enough for me to feel fairly certain that such a situation is impossible due solely to random chance.

Secondly, you say, "all voters who prefer one specific work — anybody with a "favorite author" — will have a dominant strategy of voting for only that one." Well... yes. Again, isn't that exactly the effect we want to achieve? If a fan likes a work, he/she should nominate it. If the fan likes -only- that work, that's within his/her right as well. It's still not a slate, and it's not preventing anyone else from getting their nominations on the ballot.

Finally, you conclude that it's a bad system because "you can smell a stink". Well, as others (and I) have pointed out, "Because I said so" isn't really going to fly with Hugo voters, so I really think we should cease using this type of argument for or against a system.

I'm beginning to suspect that an optimal voting system for a political election (which is what our experts have mostly studied, as I understand it) may not, in fact be the optimal system for the Hugos. This leads to an issue for this thread, however, that the moderators may need to rule on.

Teresa/Bruce: I think JT's proposal may have merit in that it is a minimal change to the existing Hugo system that -may- have no other effects (plus or minus) than to remove the influence of slates. It also may have other consequences that affect the nominations, but not in ways that we as Hugo voters care about. However, I'm not sure it fits as one of Bruce's options. Is it still kosher to discuss in this topic, or would you prefer that we stay focused on the options Bruce has presented? I can definitely see the need not to let the discussion get too far from the original intent, else we won't get anywhere, but I do think there is merit in the idea that is worth exploring. So before going further, I thought it best to check in with you -- I don't want to be guilty of transgression! :)

Thanks,
Kilo

#185 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 10:18 PM:

Consider the following election:

(num voters:works supported; ? stands for various works supported by fewer than 5 voters each)

4:?????
1:F????
5:FA???
15:EA???
20:DA???
25:CA???
30:BA???

The system elects E,D,C,B,F. A is not elected, despite being supported by an overwhelming 95% of the voters. None of the voters are voting more than 2 in common with any other, and of those 2, 1 is always A, which is just that good.

Yes, this is not realistic. But scale A down to a reasonable number — say, 50% — and increase F's support to 12% or so, and it becomes so.

As to strategy: you don't want all people who prefer P>Q>R>S>T to be voting for P only.

As to information: this system throws away ballots after they have one candidate elected. That's more radical a punishment than needed for proportionality. Aside from encouraging strategy, this also decreases both statistical validity and consensus validity.

As I said, the way to get your work nominated would be to have fans who don't read much, or whose taste is wildly different from the norm. If they haven't read as many of the other good works, or if they don't like what others like, they won't have their ballots discounted for supporting something else.

So that's 4 objections. This is not a good system. I understand that "pulling rank" made me look like an arrogant ass, and I'm sorry. I just don't know how else I can convey the difference between "interesting, but it may have a problem with X" and "no, that system is a non-starter". It's not that I don't like the idea of "use your vote where it's needed the most"; I found it worth rescuing, with two separate proposals that would accomplish the same thing. But starting a sequential procedure from the middle, that is not going to work.

#186 ::: J Thomas ::: (view all by) ::: April 14, 2015, 10:20 PM:

@183 Jameson Quinn

Notice that I "pulled rank" only after having cited a specific problem that you discounted.

You suggested a possible problem that I don't see is much of a problem. If you like five works enough to nominate them for the Hugo, it may turn out that some of them cannot win. And it may turn out that some of them are shoo-ins. If the one given your vote on is the one that barely makes it, rather than one that has a very good chance without you or one that cannot win, why would you complain about that?

Your voting system could fail to elect a candidate that was voted for by all but 1 voter. Obviously, that's an extreme, but less-extreme cases of an "obvious winner" who didn't get nominated would sour people on the system.

It depends on what people want. What I understand the purpose is supposed to be, is that we get a list of five excellent works that are supported by very different people, not all by the same clique. We want diversity.

If that is the goal, we can't do better than five works that each have 20% of the votes with no overlap. That is maximum diversity. If each of them had a unique 19% of the vote, but 1% of voters voted for all five of them, that would be less diversity. If 20% of the voters voted for all five of them, that would be a slate and a travesty.

So, say we have five works that each have a unique 20% of the vote, the best case, and then there is a sixth work that has 100% of the vote. Every ballot contains one of the magic five, and also this one. For diversity, this one is no better than the fifth one of the others.

Imagine this less-extreme case: We have selected four winners, each with about an uncorrelated 20% of the vote. At the last step, the 1st place, there are two candidates that both get all 20% of the remaining votes. One of them doesn't have anything else, and the other also has another 20% because it is on a slate with a previous winner. Does that make it better or worse?

Furthermore, all voters who prefer one specific work — anybody with a "favorite author" — will have a dominant strategy of voting for only that one.

That's true under almost any voting system, unless you forbid them to cast a lone vote. If they only care about this one work winning, how would it help them to vote for anything else? If they thought the rules favored a devious strategy of voting for something else too to help their one choice win, that would not make those rules look better to me.

Anybody with wide-ranging tastes will contribute their judgment to helping only the least-popular of their choices.

They don't necessarily know which one is the least popular. Their best strategy is to vote for the five they like best, if they like five enough to want them to win the Hugo. The one that is really the least popular -- but that can win with their support -- gets their vote. All their other choices either have no chance, or have a better chance than that one. I get the impression you don't understand how this voting system works. (It's possible I misunderstand it myself.)

Or maybe you think this person should have more than one vote? Maybe so. With some voting systems, you could be the deciding vote for all five winning nominations. If you made five different choices, all the outcomes would be different. With this system you can determine at most one. I'm not sure which is better.

The way to get your work nominated would be to have fans who don't really read much.

With any voting system, the surest way to get votes is to have a lot of readers who like your work and don't read anything else.

It's a bad system.

It may be, but you have not yet presented your case. So far you have one valid observation which is a feature and not a bug.

You are within your rights to consider it a bug, since the criteria we are working toward has so far been left totally unclear except by me.

And while I think I've been somewhat clear there hasn't been much feedback to say whether I have it right. The main feedback I've noticed is that I thought there was a sentiment that we shouldn't have too many Hugo nominees from the same sub-genre, and some people have said they think that's fine. (I like it better that way too, since there's no obvious way for the voting system to encourage people to nominate from multiple sub-genres.)

You have a second observation that each voter really only gets to vote for one nominee. Many other voting systems have that too, and I don't know whether it's a feature or a bug.

With this system, like many others, a slate can win one nomination if they have enough votes. Then they lose their votes and can't win any more nominations. It reduces the power of slates as a side effect of creating diversity among the winners, which is what various people have said they wanted.

You might possibly find a better system to achieve this goal. Or you might find a goal you think is better, that another system satisfies.

Given the goal that I think we're trying to meet, it will not improve diversity to contaminate some of the vote counts with fractional votes left over from other winners. But if you find some additional goal that justifies the reduced diversity, then you can do better at the combined goals.

#187 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 10:26 PM:

Oh, by the way, I agree completely that "an optimal voting system for a political election (which is what our experts have mostly studied, as I understand it) may not, in fact be the optimal system for the Hugos." I certainly wouldn't suggest my modified-SDV idea for a political election, for instance. Or the best system here may be something else unheard-of. But if a system is bad enough, it is bad for both politics and Hugos.

This is the last I'll comment on J Thomas's proposals. At least on that one point, I've obviously blown whatever goodwill I'd accumulated in this thread, so further argument from me isn't going to accomplish anything. I'd still be happy to answer questions but as I said when I first arrived, I'm not a Worldcon member so this is really y'all's decision not mine, and I fully respect that.

#188 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 10:40 PM:

Hmm... My "I won't comment again" obviously crossed with J Thomas's response. I am not going to go point-by-point, but his comment does raise another scenario:

25: A???
24: B???
23: C???
22: D???
21: E???
20: EF???

Your system will elect F,D,C,B,A. E will be punished because it was "on a slate" with F (that is, liked by an overlapping set of people), even though it had more non-slate than "slate" votes, and absolutely nobody voted F over E.

Unlike the previous scenario I gave, I find this one entirely plausible, more or less as-is.

#189 ::: heresiarch ::: (view all by) ::: April 14, 2015, 10:44 PM:

albatross @ 55: "I am curious how much non-slate voters agree in their nominations. My best guess is that people mostly tend to read clusters of related books--one person may read mostly MilSF, another may read mostly alternative history, still another may prefer urban fantasy."

I agree, though my suspicion is that people define their clusters by criteria far more idiosyncratic than recognizable sub-genres.

"This matters because anything that decreases the weights of slates is likely to also decrease the weight of these correlated nominations."

I tend to see it as a good thing. Partially because, as others have mentioned, proportionality in a nominees list is useful in the same way as proportionality in a committee or parliament is useful: if you are trying to bring multiple groups to a common decision, they all need to be (and feel) represented. I also think it is useful on a nominator-by-nominator scale: insofar as it encourages people to nominate widely across their reading rather than within their favored genre, that feels like a good thing. What do I think deserves a Hugo that isn't the sort of thing a lot of people will think of/is outside my normal reading cluster feel like good questions.

Cassy B. @ 114: "May I make a modest request? I've been seeing the term "bad guys" used in these threads, and I'd really rather we didn't use such loaded language."

I whole-heartedly second this. I think a fairly fundamental assumption of the current project is that Sad Puppies are only one iteration of the larger problem, which is that the Hugo nomination is prone to capture by dedicated and organized minorities. Such a minority could echo my politics down to the smallest detail, I could love each and every one of them like my own child, and if they didn't have more than 20% of the vote they shouldn't get more than one nominee out of five. Talking about this in terms of thwarting the puppies makes this project seem a) petty and vindictive and b) distracts from coming up with robust, long-term solutions.

#190 ::: J Thomas ::: (view all by) ::: April 14, 2015, 10:45 PM:

@185 Jameson Quinn

Sorry, I was writing and didn't see your response.

Consider the following election:

Good! I wrote about basicly the same thing, I think your example is written far more clearly.

The goal I understand so far is diversity. We want nominees to win that are supported by different nominators, not by the same nominator. (It's OK if some of the same nominators support more than one winning candidate, but it's important that they each have diverse support.)

What is the additional goal that the Hugo nomination system should achieve? I can easily imagine it should fit some other goal too. Can we clearly specify what that goal should be?

As to strategy: you don't want all people who prefer P>Q>R>S>T to be voting for P only.

If they think T is good enough to be a Hugo finalist, why not vote for T also? If enough of the people who vote for P also vote for T that it endangers P's chances, maybe T is more popular than it looks.

As to information: this system throws away ballots after they have one candidate elected. That's more radical a punishment than needed for proportionality. Aside from encouraging strategy, this also decreases both statistical validity and consensus validity.

If our main goal is to get nominees with ballots from diverse sources, we do not increase consensus validity by choosing winners because they correlate with other winners. If this is the goal, then any statistical validity or consensus validity we get by depending on the same ballots for multiple winners, is bogus.

I understand that "pulling rank" made me look like an arrogant ass, and I'm sorry.

It's a natural thing to do, faced with a system that looks awful and a proposer who refuses to understand. I don't blame you for trying it. I just don't want it to work. ;-)

It's not that I don't like the idea of "use your vote where it's needed the most"; I found it worth rescuing, with two separate proposals that would accomplish the same thing.

Thank you! Every tool in the toolbox is potentially useful.

But starting a sequential procedure from the middle, that is not going to work.

I don't know yet whether it can work. It's much simpler than other methods I've seen to do similar things. With luck, it will create five independent groups of ballots that are about the same size. The alternative of using this method starting at the beginning is *bad*. First you throw away the biggest group of votes, then you throw away the second biggest. By the time you get to the fifth group it's likely to be tiny.

#191 ::: nathanbp ::: (view all by) ::: April 14, 2015, 10:46 PM:

J Thomas, I don't think your Independence Voting system does what you think it does. At best I believe it performs about as well as any of 3a-d, and at worst it probably has other holes. I don't think it's reasonable to expect everyone to fill out all 5 slots with valid nominees or to disqualify them if they fail to do so, so any slate can continue on, just with reduced numbers of nominees. However, their power is not reduced as much as you might have thought. First, every ballot that matters shares at least one work with another ballot, so everyone's votes go down to being valued at 0.8. Most ballots likely share 2 works with another ballot (the 2nd-7th placed nominees for best novel totaled at 54.3% last year, so roughly speaking they were likely on at least half of the ballots that also had the 1st place nominee), bringing the average nominating power down further. A slate performing relatively as well as it did this year with 2 or 3 works listed per category would probably still have gotten all of their nominees in.

In my mind it also fails the fairness test, although this is somewhat subjective. Why should someone's vote for a popular work count less just because they nominated the same less popular works as someone else? In effect it's reducing the power of the supporters of works 7-15 to decide who is nominated. In the strategic voting sense, it disincentivizes putting more than 1 or 2 works on the ballot because if you happen to match up with someone else then your vote won't count for as much. At least with RAV or STV when your vote's value is reduced in later rounds it's because something you liked actually got nominated.

As to your other proposed system, I think the way I would describe it is arbitrary. Why pick the 5th, 4th, 3rd, 2nd, 1st candidates at each step instead of the 1st each time (same as Bruce's #1 in the old thread)? Or the 3rd each time? I think that any system which has a demonstrated example of not picking the most popular candidate is not a good choice and one that would also be difficult to describe as fair or likely to lead to outcomes that would be viewed as fair.

#192 ::: J Thomas ::: (view all by) ::: April 14, 2015, 11:05 PM:

Jameson Quinn

This is the last I'll comment on J Thomas's proposals. At least on that one point, I've obviously blown whatever goodwill I'd accumulated in this thread, so further argument from me isn't going to accomplish anything.

I welcome your recent substantive comments, and you are welcome to continue if you want to. If it's just too painful dealing with my failure to understand, I'll have to accept that.

25: A???
24: B???
23: C???
22: D???
21: E???
20: EF???

Count the votes.

41: E
25: A
24: B
23: C
22: D
20: F

The 5th place winner is D.
The 4th place winner is C.
The 3rd place winner is B.
The 2nd place winner is A.
The 1st place winner is E.

A similar version:

5: A???
24: B???
23: C???
22: D???
21: E???
20: F???
20: FA??

40: F
25: A
24: B
23: C
22: D
21: E

D takes 5th place
C takes 4th place
B takes 3rd place
A takes 2nd place

21: E
20: F

E takes 1st place.

F had 40 votes and did not place.

But we did get five nominees that had diverse votes.

#193 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 11:22 PM:

@192: Sorry, you're right. Here's the scenario I meant:

25: A???
24: B???
23: C???
21: D???
20: E???
19: EF???
2: F???

#194 ::: Keith "Kilo" Watt ::: (view all by) ::: April 14, 2015, 11:23 PM:

(I haven't heard from the mods on my previous question as to whether it's okay to discuss an option that is not one of the original five, so please feel free to delete this if it's a problem.)

@185:

I would like to consider just the "slate weighting" that JT proposed. As a first cut, this is a minimalist change to the Hugo nominating rules. The original nominating rules stand with the addition that:

3.8.8 It is the intent of the Hugo nomination process that Worldcon members will only nominate works that they personally have read and enjoyed, and will not allow their nominations to be co-opted by any other individual or group.

3.8.9 Any nominating ballot which has five nominations, all five of which are duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.0.
Any nominating ballot which has four nominations, all four of which are duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.2.
Any nominating ballot which has three nominations, all three of which are duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.4.
Any nominating ballot which has two nominations, both of which are duplicated on any other ballot shall have each nomination on that ballot weighted by a factor of 0.6.
Any nominating ballot which has one nomination which alone is duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.8.
The purpose of this rule is to discourage nominating that violates the spirit of 3.8.8


Okay, let's apply the system to your hypothetical ballots:

Consider the following election:

(num voters:works supported; ? stands for various works supported by fewer than 5 voters each)

4:?????
1:F????
5:FA???
15:EA???
20:DA???
25:CA???
30:BA???

Group 1's ballots are weighted at 1.0.
All other groups' ballots are weighted at 0.8.

Remember, the goal is to discourage -slates-, not a given work appearing on multiple ballots. There is only one duplicate for each of the other groups. Group 2 duplicates work F with Group 3. Groups 3, 4, 5, 6, and 7 all duplicate work A. Why doesn't Group 3 get penalized twice, then? That's not what we're proposing. We are only weighting copies of slates, not works that happen to be held in common by many voters -- the whole -point- is to find works that are held in common, yet not through artificial means.

So, tallying the votes:

Work A: 95 nominations * 0.8 = 76 nominations
Work B: 30 nominations * 0.8 = 24 nominations
Work C: 25 nominations * 0.8 = 20 nominations
Work D: 20 nominations * 0.8 = 16 nominations
Work E: 15 nominations * 0.8 = 12 nominations
Work F: 5 nominations * 0.8 + 1 nomination * 0.8 = 4.8 nominations

Final ballot is therefore ABCDE. I'm not sure I see how this is a problem.


As to strategy: you don't want all people who prefer P>Q>R>S>T to be voting for P only.

Why not?


As to information: this system throws away ballots after they have one candidate elected. That's more radical a punishment than needed for proportionality. Aside from encouraging strategy, this also decreases both statistical validity and consensus validity.

I think you may be misunderstanding the system. Even so, if it's simpler to understand, then a "more radical punishment than needed" is fine. In my opinion, it's a worthwhile trade-off.


As I said, the way to get your work nominated would be to have fans who don't read much, or whose taste is wildly different from the norm. If they haven't read as many of the other good works, or if they don't like what others like, they won't have their ballots discounted for supporting something else.

Sorry, but I guess I'm not understanding this at all. I'm probably just not parsing it correctly, however.


So that's 4 objections. This is not a good system.

It is not a system you -value-. Your objections are not hard and fast objective facts, they are value statements. We, as Hugo voters, may agree with all of your value statements. Or we may agree with none of them. That's a debate worth having, and one I'd still like to see the community undertake. I think that question is arguably more important than that of specific voting systems.


I just don't know how else I can convey the difference between "interesting, but it may have a problem with X" and "no, that system is a non-starter".

I think the problem is not that you're an "arrogant ass" (feel free to disemvowel as required!), but the fact that -you- have decided it's a non-starter. I don't believe anyone here has the power or right to declare that, save only Teresa and Bruce, as it's their playground, as I understand it.

Kilo

#195 ::: J Thomas ::: (view all by) ::: April 14, 2015, 11:23 PM:

@!191 nathanbp

J Thomas, I don't think your Independence Voting system does what you think it does.

Wow, I didn't think anyone would actually take it seriously. It fits some criteria for what we say we want, but....

If you vote for just one, that's worthless unless other people also vote for it. So your weight is .8 on one vote.

If you vote for two, also worthless unless other people also vote for them, your weight is .6 on two votes for a combined voting power of 1.2

If you vote for three with a weight of .4, you also have a power of 3*.4=1.2

If you vote for four with a weight of .2, you have a power of 4*.2=0.8

If you vote for five and anybody else votes for the same five your power is zero.

It destroys naive slates. Of course real slates would vote for three like anybody else who had three candidates they liked.

It was a joke but the idea of applying different weights to ballots depending on how similar other ballots are, and/or how many are similar, has a whole lot of possibilities provided we can get away from the idea that it is supposed to be a fair election.

Why should someone's vote for a popular work count less just because they nominated the same less popular works as someone else?

Almost everything we do with voting systems to reduce the power of slates, is doing this.

That isn't true for allowing 4 nominations per ballot and six winners. That means even a winning slate leaves a couple of slots for others.

But in general, we are looking for ways to make slate ballots count less because they nominated the same works as other slate ballots.

Why should someone's vote for a popular work count less just because they nominated the same less popular works as someone else?

No, I take it back. Often they make the *second* and later votes count less because they nominated the same popular work. They let the first one slip through and then clamp down on later votes, which in the order of things tend to be the less popular ones.

#196 ::: heresiarch ::: (view all by) ::: April 14, 2015, 11:25 PM:

Single Distributed Voting and Reweighted Approval Voting approach the same problem, ensuring a widely representative ballot, from two directions: SDV says if one of your choices has been eliminated then we shall increase the weight of your remaining choices; and RAV says if one of your choices has been selected then we shall decrease the weight of your remaining choices. They both have the same strategic problem: voters who do not want to "waste" their full-power vote on a less-favored selection have a reason to reduce their total number of nominations (bullet voting).

Both SDV and RAV have a maximum vote power: for SDV it is conveniently one, and for RAV it is either 1.79 (St.-Laguë; 1 + 1/3 + etc.) or 1.94 (D'Hondt; 1 + 1/2 + etc.). Hugo-Classic has a maximum vote power of 5. It's possible to exert less effect on the election than this, but no more. Under RAV, what voters are doing when they strategically vote for a single candidate is sacrificing the .79-.94 extra vote in favor of being absolutely certain where the remaining one will go. (SDV entails no sacrifice: this implies that SDV favors bullet voting more than RAV.)

However, the rate at which votes are altered as candidates are selected/eliminated is open to adjustment. What if SDV, rather than redistributing the entire vote according to the number of candidates remaining, was redistributed at a rate less than 100% and according to the number of candidates eliminated? So, for example, if one of your five nominees was eliminated, the other four would go from 1/5 each to 1/4.5, rather than 1/4? And if you only listed four to begin with, they would still only start with 1/4.5? What if RAV votes were reweighted at double St.-Laguë--2/3, 2/5, etc. as winners are removed, rather than 1/3 and 1/5? Under this kind of proposal, bullet voting under RAV would mean sacrificing 1.57/2.57ths of your maximum voting power, all on the hopes of one boost. A voter would have to be very committed to that single work to give up being able to help out four other works one and a half times as much.

#197 ::: nathanbp ::: (view all by) ::: April 14, 2015, 11:36 PM:

J Thomas@195:
Wow, I didn't think anyone would actually take it seriously.

I'll confess I got confused about which of your proposals Jameson was responding to and then typed all that up before realizing I was talking about the wrong one.

No, I take it back. Often they make the *second* and later votes count less because they nominated the same popular work. They let the first one slip through and then clamp down on later votes, which in the order of things tend to be the less popular ones.

I think this is important. Decreasing the value of the remainder of someone's ballot (to half, or even to zero) seems fair to me because they got something.

#198 ::: J Thomas ::: (view all by) ::: April 14, 2015, 11:47 PM:

25: A???
24: B???
23: C???
21: D???
20: E???
19: EF???
2: F???

39: E
25: A
24: B
23: C
21: D
21: F

A tie! I hadn't thought of that. As a first guess, I ought to say that in case of ties we should pick the one that has less overlap with other tentative winners, and also pick the one that has more disqualified ballots due to previously confirmed winners. I mostly don't want disqualified ballots to matter, but when it's to break a tie, why not?

But that gives the wrong answer for the example. Let's say that F wins fifth place.

25: A
24: B
23: C
21: D
20: E

The next winners are D, C, B, A.

E went from 1st place to fifth and never won. It had the most votes.

If we had chosen A B C D E instead of A B C D F it would have been better. E got more votes than F, and none of the shared ones were shared with other winners. Only with F.

So here is an example where RSAV creates a good selection of uncorrelated nominees, but not the best selection. I don't know whether it's possible to fix it in a way that that solves this and all other problems, or not.

#199 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 01:41 AM:

Regarding ties:

Is that a bad thing? We're just trying to get a list of nominations, not select a winner. It ultimately doesn't even matter how items on the final ballot are ranked. Is there problem with just saying:

3.8.10 In the event of a tie for the fifth nominated slot, all works that are part of the tie will added to the ballot.

So if we get a tie for fifth, we have six nominees for that category. I don't see that as a big deal...

Kilo

#200 ::: Brad from Sunnyvale ::: (view all by) ::: April 15, 2015, 01:47 AM:

This thread should be giving us some idea of how well the voters and business meetings would understand these systems. (Though to be fair, I think with effort, simpler explanations can be made.)

Once again, these systems aren't coming close to matching the non-strategic qualities of the simple system we used that was spoiled by the slates. Not only is there effectively zero reason to be strategic on the approval ballot, people are even openly non-strategic.

For example, I have often, in a Hugo nominating category, only listed two, three or even just one work. And I know I am not alone. I am "wasting" my "power" and entirely not caring. It's just that I don't follow that category super closely, but I know the people I am nominating were good enough by my standard that year.

Many of the above systems will give this "only care modestly" ballot greater weight, as though all ballots were written by people who have 20 things they want to nominate and have to carefully pick the best 5. Now the hidden message will be "nominate fewer, your choices will do better."

Yuck.

#201 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 02:03 AM:

A hasty P.S. to my #194:

Groups 3, 4, 5, 6, and 7 all duplicate work A.

... whoops, I've got the weighting wrong. Each of group 3, for example, is obviously a two-work "slate", so should be weighted at 0.6, not 0.8. I was only comparing them to other slates, forgetting that they themselves are a slate.

Let me try the tally again:

4:?????
1:F????
5:FA???
15:EA???
20:DA???
25:CA???
30:BA???

Group 1's ballots are all weighted at 1.0 (no duplicates on their nomination ballots).
Group 2's ballot is weighted at 0.8 because it duplicates one work from Group 3.
Group 3's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 4, 5, 6, and 7 (so 0.8 weighting), and a different work with Group 2. (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights.
Group 4's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 5, 6, and 7 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights.
Group 5's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 4, 6, and 7 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights
Group 6's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 4, 5, and 7 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights
Group 7's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 4, 5, and 6 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights

Work A: 95 nominations * 0.6 = 57 nominations
Work B: 30 nominations * 0.6 = 18 nominations
Work C: 25 nominations * 0.6 = 15 nominations
Work D: 20 nominations * 0.6 = 12 nominations
Work E: 15 nominations * 0.6 = 9 nominations
Work F: 5 nominations * 0.6 + 1 nomination * 0.8 = 3.8 nominations

Winners are ABCDE as before, but it's important that I get the example weighting right.


If something like this could work without having to select "rounds of winners", it might be closest to what the system we have now while still discouraging slates.

But maybe not -- I'll try to come up with counter-examples where a slate can still control the nomination list under this change. I'd be interested in someone else doing the same, since I likely can't see the forest for the trees.

Kilo

#202 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 02:05 AM:

Brad@200:

Now the hidden message will be "nominate fewer, your choices will do better."

Yuck.

Why yuck? It seems to me that if someone nominates fewer choices, then they feel more strongly about those choices they do make. Isn't that the behavior we want to see?

Kilo

#203 ::: heresiarch ::: (view all by) ::: April 15, 2015, 02:38 AM:

Brad from Sunnyvale @ 200: "Once again, these systems aren't coming close to matching the non-strategic qualities of the simple system we used that was spoiled by the slates."

You write as though the previous system lacked strategy, when the very fact that it was spoiled by the formation of a slate shows that that is not true. The current nomination system always had a dominant strategy--it just went blissfully unexploited.

"For example, I have often, in a Hugo nominating category, only listed two, three or even just one work. And I know I am not alone. I am "wasting" my "power" and entirely not caring. "

The goal of discussing "power" and "strategy" is precisely to protect this kind of voting. Now that gaming the nomination system has happened, its bare existence is forcing Hugo voters to agonize over a series of unpleasant strategic decisions (form a slate, form anti-slates, boycott slates, give slated works a fair read) rather than doing what they ideally ought to be doing, which is, as abi eloquently describes, thinking deeply on what works have moved them enough to be worth nominating.

It is by thinking seriously about how various voting schemes can be gamed that the impact of those strategies can be minimized, leaving the best choice, both morally and also strategically, to simply vote for what you think deserves your vote. That's the point.

#204 ::: felice ::: (view all by) ::: April 15, 2015, 02:41 AM:

Keith "Kilo" Watt @202:"Why yuck? It seems to me that if someone nominates fewer choices, then they feel more strongly about those choices they do make. Isn't that the behavior we want to see?"

It's the exact opposite of what we need to counteract slates. The fewer nominations in total, the fewer needed to get on the ballot, and the fewer slate nominators needed to swamp it. If total nominations were low enough, a slate could win against any counting method by dividing their supporters into multiple sub-slates (potentially, five slates of one work in each category).

#205 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 03:01 AM:

felice@204:

I think I see what you're saying: Fewer nominations total means a slate needs to gather fewer nomination votes to run the nominations, correct? I can see that, though to a certain extent that's true even if with lots of nominations, if they are widely scattered among different works.

But is your objection truly the case for all systems? It seems to me that the slate-weighted system solves that problem. If I nominate a single work that I really love, the odds are my vote will count with a 0.8 weight. But a slate is likely to get a 0.0 weight regardless of what I do personally.

I do see that one could argue that the system encourages me to choose only one work to support. I don't think that's a bad thing, particularly since you still have the option of nominating other works if you want, though at the increased risk that your weighting may go down. But because the odds of randomly matching someone else's nomination ballot go way down the more works you nominate, the actual chance of your weighting going down also decreases. If, on the other hand, you have intentionally matched your nomination ballot to someone else for ask five slots, then the odds of you getting a weight of 0.0 are 100%. So, even in the case you suggest, the system seems to function okay, doesn't it?

Kilo

#206 ::: felice ::: (view all by) ::: April 15, 2015, 04:45 AM:

Keith "Kilo" Watt @205: "It seems to me that the slate-weighted system solves that problem."

I'm not quite clear on how the slate-weighted system works (what's the weighting on a nominating ballot with 5 works, of which 4 are duplicates and one unique?), but in any case there are ways for a slate to avoid a zero weighting (ie instruct supporters to nominate a random subset of the slate instead of the whole thing). The power of the slate is reduced, but the power of non-slate nominations is also reduced, both because there are fewer of them (more people only nominating one work), and non-collusion overlap in ballots (the distribution of nominations isn't random, there's some correlation based on tastes). I'm not convinced it would help against slates at all, and I think the side-effects are problematic.

#207 ::: J Thomas ::: (view all by) ::: April 15, 2015, 05:25 AM:

@201 Keith "Kilo" Watt

I intended that Independent model at most to focus discussion. It stops 5-item slates cold. A 4-item slate would suffer an 80% penalty -- 300 votes would amount to 60 votes. A 3-item slate would have a 60% penalty -- 300 votes would amount to 120 votes. A 2-item slate would have only a 40% penalty, for those two items 300 votes would amount to 240 votes which is not so bad. And if a bunch of people agreed to push just one item? They would have no more penalty than anybody else does who votes for a single item that might win.

Meanwhile, regular voters would suffer random effects. If you accidentally vote for the same five works as somebody else, both your votes are lost. Etc.

This is actually kind of representative of the voting systems we're looking at. We're looking for ways to punish people for voting alike, so that slates will have less effect.

So for example, if a ballot has power 5, each slate ballot can affect five nominations. If we reduce that to power 1 then a slate ballot can affect only one nomination. We've taken away 4/5 of its power. That doesn't affect regular votes nearly as much because it's far more likely that a regular vote would mostly be wasted anyway. If you vote for 5 nominations and 3 of them are things that cannot win, you only lose half your power. With RAV you lose hardly anything, while the slate still loses around 3/5.

These are palliatives. Our central problem is that most of our nominations are wasted. Slates win because currently they each have five (5) votes that they do not waste. We're looking for rules to make them waste most of their votes too.

If you look at the 2013 Hugo statistics,
http://www.thehugoawards.org/content/pdf/2013HugoStatistics.pdf starting page 19
for novel there were 1113 ballots, and there were 717 votes for nominees that won. At least 396 ballots had no winners at all.

Out of more than 5500 opportunities to vote, 717 of them were for winners. Another 785 were for the rest of the top 15. That's about a quarter of them, total. Three quarter of the votes were for nominees that -- in hindsight -- could not have won.

Slates do not have this problem unless they are small pathetic slates. They never vote for works that nobody else does. Because they have each other.

There's a limit to how much we can fix this by adjusting the voting procedure to take away votes from slates.

#208 ::: J Thomas ::: (view all by) ::: April 15, 2015, 07:13 AM:

I think that Felice's proposal goes some way toward fixing this.

Announce the top 15 nominations halfway through, and let people who have not voted for them vote for them if they want to, during the second half.

This is almost the same as a run-off vote. One difference is that your earlier ballot still counts. You don't lose your vote if you fail to vote again. To the extent that the vote is online, it is not much extra work for the administrators.

If you want a sense of everybody's honest feelings, after the election they can post the half-time voting results that show how many votes every nomination got. Or just the top 15 with their votes, if that's what you care about.

But the more people who choose to vote for nominees that can win, the less influence slates can have.

So, the first half of the nominations is about expressing your honest feelings. You have an incentive to vote in the first half to improve the chance that your nominees get into the top fifteen. As far as I'm concerned you can express your honest feelings in the second half too. Nominate new things that can't win. Vote for nominees that aren't in the top 15, it's like a write-in vote but it isn't completely impossible to make a difference.

The second half can be about winning. With acceptance voting, you might as well vote for all the nominees you like. If you think it over, and you find yourself thinking "I'd hate it if B won and A didn't", then probably you shouldn't vote for B. If you'd be satisfied with either of them winning, then vote for both.

I suggest we let nominators vote for as many nominees as they want, at least in the second half. So if you are particularly angry at some slates, you can vote for everything but those slates. You do nothing to decide who wins, except to vote against them. Or you might want to vote for only one nominee. Or none. Whatever you choose.

In 2013 the top novel got about 18% of the ballots (maybe 4% of the votes) and the 2nd got around 12%. A slate with about 140 votes could have gotten 4 nominations. The highest novella got 103 votes, the same slate would get everything.

Now, a quick sanity check. In the very best case, imagine that 1113 nominators each chose 5 out of the top 15. That's 5565 votes. About half in the top 5, around 2800. A slate would only have to be about 4 times as big to get the same result. In the best case. About 560 nominators on top of the 1100 that are already there.

Best Novella had 587 nominators instead of 1113. Whatever the rules, a slate half as big would hit it.

Small awards are inherently slateable. A slate that's big enough to affect a big award will roll right over a small award. Period. We can look for a way to get a lot more honest nominators, or get them to vote more for winners, or get rid of the award, or find some clever way to nerf slates, or all four. But whatever we do has to work *really well* or we have to write off the small awards.

Bottom line: Helping more nominators nominate winners is another palliative. In the best case it could increase the number of votes for serious competitors against slates, by about 4 times. If we also use a voting system that reduces voting power to 1 (which reduces slates by about 80%) that will reduce votes for winners by about 80%. The two approaches do not work together.

#209 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 07:27 AM:

@207: "There's a limit to how much we can fix this by adjusting the voting procedure to take away votes from slates."

Yes. Voting theorists study limits like this.

I can't find it right now, but there's a graph of proportional systems, with "average satisfaction"—how much the average voter likes the average candidate—on one axis, and "representativeness"—a measure of how evenly-distributed satisfaction is across voters—on the other. You can see a clear pareto front on that graph; that is, some systems do well on one dimension but poorly on the other, some do OK on both, and then there are the poorly-designed systems that do badly on both. But there is no magic "perfect" system that does perfectly on both.

You could make that kind of graph for any two competing goals that you could define and measure in a monte carlo simulation. For instance, the power of "slate" strategy versus the power of "leave off popular candidates" strategy. On this latter graph, the current system would have a nice low power for the latter, but a crazy high power for the former. The STV-like systems which don't transfer overvotes would err in the other direction. RAV, SDV, and FAPE would be on the pareto front, representing reasonable compromises, with RAV giving more weight to avoiding slate strategy, FAPE giving more weight to avoiding "leave off popular" strategy, and SDV somewhere in the middle.

(Remember, FAPE is the system where you divvy up votes, then successively eliminate whichever of the bottom two has support from fewer voters.)

I'd recommend choosing a system with the following characteristics:

1. Resolvable without NP-complete calculations. That rules out SAV and PAV.
2. Uses an approval-like ballot, as with the current system. That rules out standard STV.
5. Proportional. That rules out the current system.
4. On or near the two pareto fronts I discussed above. I suspect, but cannot currently cite or prove, that that rules out STV-without-ranking, which I suspect is dominated by SDV.

That leaves RAV, SDV, and FAPE. So: if you want to encourage people to vote for more candidates, without giving too much power to slates, choose FAPE; if you want to punish slates as much as possible, without causing collateral damage among groups with common tastes who don't organize slates, choose RAV-exponential or, even further in that direction, RAV-Saint-Lagüe; and if you want a compromise between those "reasonable extremes", choose SDV.

Note that if the goal is to "beat" slates, it's better to do it "naturally", by encouraging non-slate voters to vote broadly and thus find common ground, than to do it "artificially", by having the voting system deliberately deweight slate votes. This argues that, in terms of the tradeoff above, you'd rather err on the FAPE side of things.

On that basis, I'd say FAPE is the best choice. But my personal self-interest is in having you choose a previously-known system which would have wider application in other use cases, that is, RAV. Really, I'd be happy with any of the three.

It is likely that a system somebody here just invented is not going to be on the pareto front, unless they at least took care to make it proportional. You don't have to be a super-expert to invent a good voting system, but you do have to have some awareness of the pitfalls and be careful about things like proportionality.

#210 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:20 AM:

The other argument in favor of SDV and FAPE over RAV is that, based on this thread and reactions like GRRM's, "dividing up the votes" seems to be easier to explain and to seem fairer to people than "reweighting the votes".

For instance: above, people are talking about RAV in terms of the sum of your ballot's voting weight over the rounds. But actually, that's not really a meaningful concept. Consider:

10:A
10:B
10:C
Elect 3. Winners: ABC

All votes have equal weight. Now:

20:AB
10:C
Elect 3. Winners: ABC

The first two groups have added a second preference, but the result is unchanged. Would you say that the first two groups have 50% more power? I wouldn't. Mathematically, the things that matter are how many of the nominees you supported, and how much power your vote has to choose the next nominee, and the ratio between those numbers. Adding a ballot's power across rounds doesn't really make sense.

Consider the extreme case. You have RAV with D'Hondt weightings, and you're electing 100 nominees. 99% of voters choose slate A1-A100, and 1% choose B. The winners are A1-A99 and B. The sum of 1/n from 1 to 99 is 5.177. Who would say that the A voters have 5.177 times the voting power of the B voters here? That's just silly; you could say they have 99 times the power, or the same proportional power. but 5.177 corresponds to nothing.

But I understand that it feels right to add up the voting weights. And that's because RAV is pretty unintuitive. I think my "cookie grab" metaphor helps understand it, but it's still tough.

So, SDV and FAPE avoid that problem. It's easy to understand dividing up a vote; it's treating the vote as a fixed resource, and our intuition knows how to handle that.

I still think RAV is a great system. But basically I'm beginning to talk myself around to thinking that FAPE is better.

#211 ::: J Thomas ::: (view all by) ::: April 15, 2015, 08:30 AM:

@209 Jameson Quinn

I can't find it right now, but there's a graph of proportional systems, with "average satisfaction"—how much the average voter likes the average candidate—on one axis, and "representativeness"—a measure of how evenly-distributed satisfaction is across voters—on the other. You can see a clear pareto front on that graph; that is, some systems do well on one dimension but poorly on the other, some do OK on both, and then there are the poorly-designed systems that do badly on both. But there is no magic "perfect" system that does perfectly on both.

I expect that many of us would prefer a graph with "average satisfaction of non-slate nominators" on one axis, and "representativeness for non-slate nominators" on the other. The preference would be that the slate nominators quit in utter disgust that their tactic failed.

But we don't have a certain way to identify slate voters from the voting data. "A large number of identical ballots with nominees nobody else votes for" is a slate, but they can disguise that pretty easily.

We can define "probably a slate" as "a significant number of ballots that are too correlated" and look for ways to punish probable slates. I'm not sure whether that requires NP-complete calculations or not.

(NP-complete calculations are OK for elections that don't get too big. The problem is only that they don't scale well.)

#212 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:32 AM:

As for the idea of publicizing a "long list" of the 15 frontrunners a month or so before nominations close, and allowing voters to redo their nomination ballot on that basis if they wish: it's not a bad idea. It would help with the problem where too many voters supported zero winners because the votes are too dispersed. But it wouldn't solve the problem of slates, and could even potentially make it worse. Personally, I'd suggest making one change at a time; so right now, this wouldn't be the most important change.

#213 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:42 AM:

@211: Any system built to ignore slate voters, as in the graph you suggest, would have collateral damage in groups of voters who just innocently had similar tastes. In general, insofar as there is "good taste" and "bad taste", and voters with "good taste" are roughly identifying some objective quality measure in works while voters with "bad taste" are just randomly voting based on what the weather was like when they read the book, any such system would tend to decrease the power of the "good taste" voters. This is true even if there are several conflicting versions of "good taste"; the truly random "bad taste" voters would still have more than their share of power.

So, I strongly feel that a good voting system gives slate voters their proportional share. It should be no more than that; but also no less than that.

#214 ::: Joe in Australia ::: (view all by) ::: April 15, 2015, 08:42 AM:

We're looking for a gate-keeping system, not a voting one: the actual voting system is pretty robust against slates.

The existing nomination system is broken because it lets attackers exclude other works. All we need from the nomination system is a list that includes works that are worth voting on. Having extra works on the ballot is only a failure if it creates too much work for the scrutineers; a solution to this problem that creates more work anyway is no solution at all!

For those reasons I support Option 2 (change the number of winners of the nomination election). I see no reason to change the number of nominations a member can make or to change the ranking of nominees.

It's better to err on the side of inclusivity to ensure that we have enough "genuine" (non-slate) nominees. For this reason I think we should have overlapping mechanisms for determining the length of the list.

My preferred mechanisms are:
1) Ask the members - it's an additional vote, but a very quick one. Take the median response, because it is hard or impossible to manipulate;
2) Leave it up to the ConCom. If attackers control the ConCom we have worse problems;
3) Set it by a formula - I don't care much for this because any fixed formula will be a target for attackers.

I'd be very happy with the first two options (no fewer than five nominees, plus as many more as the members elect in a followup vote, plus as many more as the ConCom elect). But any of these should work well and should not be too burdensome..

#215 ::: Cassy B. ::: (view all by) ::: April 15, 2015, 09:25 AM:

I know this is a naive question.... but some of those systems above are talking about (if I understand correctly) comparing *every* ballot to *every other ballot* in order to determine weighting.

Is this feasible in terms of computational power? Are we talking about running a script on a laptop, or about hiring the time of a supercomputer? I honestly have no idea.

That said, as a non-theorist who has been struggling to keep up with the proposals, I have one other question; in the case of the 80/60/40/20/0 weighting, does one of the identical ballots get full weighting, or does (in the impossible situation of every ballot being perfectly identical) every iteration get thrown out? That is, in an election with 101 ballots, and 100 have five identical nominees, and 1 has one (different) nominee... do we end up with one item on the ballot, because every single instance of the 100 are thrown out? or does one ballot survive of the 100, and we have six items on the ballot due to the tie? (Yes, I know, impossible hypothetical, but I'm trying to understand how this works.)

On the gripping hand, if I'm understanding the 80/60/40/20/0 system at all correctly, I rather like it. It's simple and easy to explain. And since it compares only within categories, the fact that EVERYONE liked The Avengers won't matter for Best Short Story...

#216 ::: P J Evans ::: (view all by) ::: April 15, 2015, 09:47 AM:

214
A motion has been proposed to change the number of nominations that can be made from five to four. (I know there are other changes also being proposed, but my brain isn't retrieving.)

#217 ::: J Thomas ::: (view all by) ::: April 15, 2015, 09:54 AM:

@213 Jameson Quinn

We don't have to worry about increasing the share of voters who vote at random. They have essentially no power because there are so many random things to vote on that they cannot be in the top 15 except by rare accident.

What I call a "probable slate" would show up basicly by cluster analysis. A bunch of ballots that are very similar to each other, and not very similar to anything else. We could punish it more the bigger it was, which would reduce the problem of accidental clusters.

Punishing cliques that all like the same things, and rewarding diversity.

I am getting a better idea of the criteria -- probably we want to reward diversity and also to a lesser extent reward things that get a lot of votes whether they are diverse votes or not. More later, I have a minor crisis to attend to.

#218 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 09:58 AM:

@215: You are correct; in principle, SAV and PAV involve comparing every possible set of 5 possible winners to every other. In practice, there are ways to home in on the "reasonably good" sets. So the exact computation should be feasible with high probability, and even if it becomes prohibitive to find the optimal winner set, a nearly-optimal one would definitely be easy to find. Still, you might reasonably count that as a downside to those two systems. But I haven't seen anyone in this thread supporting those systems so I think they're pretty much dead.

The 80/60/40/20/0 idea does not have good theoretical properties. At all. It can "punish" anybody who just happens to agree with somebody else. Weighting votes at 0 is a very bad idea; for instance, it means that if you have some way of knowing how somebody else voted, you can cancel out their vote entirely by just casting an identical one. It's a real mess.

If it were up to me, we'd be focusing any voting system discussion ("option 3") on RAV, SDV, and FAPE. I'd support whichever of these is seen to have the best chance of passing in the business meeting. If additional information, proofs, or simulations would help that passage, I'd be willing to help make those.

Ideas on options 1, 2, 4, and 5 are also welcome of course.

#219 ::: J Thomas ::: (view all by) ::: April 15, 2015, 09:58 AM:

@212 Jameson Quinn

As for the idea of publicizing a "long list" of the 15 frontrunners a month or so before nominations close, and allowing voters to redo their nomination ballot on that basis if they wish: it's not a bad idea. It would help with the problem where too many voters supported zero winners because the votes are too dispersed. But it wouldn't solve the problem of slates, and could even potentially make it worse. Personally, I'd suggest making one change at a time; so right now, this wouldn't be the most important change.

I doubt the membership will have a lot of tolerance for many repeated changes. We need to change to one system that works, and we will make a second change if the first system demonstrably fails. We will think about how to change it more after an extended discussion about whose fault it was and how we will never listen to them again about anything.

Not that this is how it ought to be done, but it's par for the course, right?

#220 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 10:08 AM:

@217: If you think of it in terms of "cluster analysis", there are various clusters, each of which has a "strength" (number of voters) and a "cohesiveness" (aka precision or inverse variance). Statistical tools can try to detect clusters. But such tools are fundamentally subject to both false positives and false negatives; the latter especially so, because any actual organized slate will presumably try to avoid being detected. Because of that, it is my firm belief as both a statistician and a voting theorist that looking to punish slates by more than a proportional system would (for instance, by more than highly anti-slate standard proportional mechanisms such as FAPE, RAV with Saint-Lagüe reweighting, or STV with a Hare quota) will do more harm than good. Explicit late voters will dodge it, so you'll only end up punishing innocents.

#221 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 10:13 AM:

In particular: cluster analysis in statistics is not an easy thing, but it gets easier the more items there are in a cluster. In this case, that would mean that unless you used some really advanced math, you'd probably end up punishing "large groups of voters with naturally similar tastes" more than "nasssty sssslate voterses".

#222 ::: Mark Wonsil ::: (view all by) ::: April 15, 2015, 11:08 AM:

I posted this on Bruce's site as well but adding it here for those who didn't go there. The process of determining rules for elections is covered quite well in a book called, "Choosing in Groups: Analytical Politics Revisited" by Michael and Kevin Munger. You can find it on Amazon if you wish or listen to a podcast about the book here: http://www.econtalk.org/archives/2015/02/michael_munger_1.html One may have the "perfect" voting system but a powerful few can make elections look very "democratic". It's an interesting read and this discussion is fascinating in light of the book.

#223 ::: Brad from Sunnyvale ::: (view all by) ::: April 15, 2015, 11:55 AM:

Yuck because this is not supposed to be a war. It is not supposed to be an election, with power and influence and clever strategies. That's why we're in crisis, because people treated it that way.

It's supposed to really be a survey, to learn from fans what they thought the great works were of the year that should be contenders for the Hugo.

For that, we want their true opinions, their independent opinions. I'm OK if they rank them (Hugo voters are used to that) because that's a true opinion. But when it becomes better for you to not give your true opinion because you can do better for "your" candidates by lying, I say "yuck,"

You are not in a competition with other Hugo nominators to see who can get more of their choices onto the ballot. You're trying to help the awards figure out good nominees.

#224 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 11:57 AM:

Regarding "pulling rank", I have an analogy.

Say the Holy Paper Airplane has just crashed and burned, so the faithful organize a contest to make a replacement for it. One or two experts and one or two hundred amateurs enter the contest. Chances are still good that the winner will be one of the amateur planes; expertise is worth something, but building good paper airplanes isn't that hard, and some of the amateurs will be naturally talented.

But say one of the amateurs says "the Holy Plane crashed because it had a tendency to turn left. We should avoid that problem by making a normal paper airplane and then cutting off the right wing." The expert will reply, "That won't work," and she will be right.

Designing voting systems takes a light touch and a balance of conflicting principles. If you make something designed only to avoid the last disaster, you'll only get a different disaster. Not all good systems look like each other, but there are subsets of bad systems that are bad for the same reason, and it's possible to learn to recognize those.

#225 ::: nathanbp ::: (view all by) ::: April 15, 2015, 12:29 PM:

Brad from Sunnyvale @223: I'm pretty sure it's not possible to have a voting system without the possibility for clever strategies at edge cases. Even in the current system, big fans of a work on the edge of nomination are potentially costing that work the nomination if they list other works in the same group on their ballot. For example, last year Parasite was nominated for Best Novel by only 2 votes over The Shining Girls. It's not likely, but possible that 2 people may have listed both works but would have preferred The Shining Girls over Parasite.

All the systems without ranking assume that the people voting will be equally happy with whichever of the works they vote for being nominated, which is obviously not true.

#226 ::: kimiko ::: (view all by) ::: April 15, 2015, 12:30 PM:

214,
All we need from the nomination system is a list that includes works that are worth voting on. Having extra works on the ballot is only a failure if it creates too much work for the scrutineers; a solution to this problem that creates more work anyway is no solution at all!

Okay, what's the largest number of works we could have in a given category, that a reasonably able person could read and evaluate? More for short stories than novels, yes?

#227 ::: P J Evans ::: (view all by) ::: April 15, 2015, 12:37 PM:

225
Traditionally, we haven't had rankings on nominations because we're creating a list of the stuff that people like and think is good enough for a rocket. If we have to rank them at that stage, why bother with a final ballot?

Short: make a case for ranking at the nomination stage.
Because so far, I haven't heard one that doesn't come out sounding like 'we don't want anyone to get a lot of nominations, and we want as many things nominated as possible, because diversity is more important than what people actually like'.

#228 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 12:46 PM:

Brad@223:

It's supposed to really be a survey, to learn from fans what they thought the great works were of the year that should be contenders for the Hugo.

I agree with you -- strongly, in fact. The purest and simplest form of nomination, and one that directly achieves this goal, is to simply have everyone nominate one work. Everything makes ballot, we all vote, and we're done. I still think this is a good idea, the simplest idea, and immune to almost all of the problems we've been discussing. The objection seems to be that no one can possibly read the huge numbers of works that would be suggested, and I think that's probably true. I still claim it's open to debate as to whether that might be the lesser evil. What a great reading list that would be!

But we all seem to be holding conflicting aims at the same time:

1) We want a wide and diverse field of nominations so that every fan's opinions are equally valid and considered. Every fan is voting for works he or she personally enjoyed.

2) We want to constrict the potential nominations to a small subset so that people can sample works they didn't necessarily read themselves previously. Their personal choices may need to be sacrificed to the choices of larger groups of people that share similar choices.

3) Larger groups of people who share similar choices should not be able to force the personal choices of others to not be considered.

Do you see what I'm getting at?

Kilo

#229 ::: Brad from Sunnyvale ::: (view all by) ::: April 15, 2015, 12:48 PM:

#225 nathanbp -- while I would hope we didn't have so many "nominate everything by my favourite writer" sorts, I will presume we do.

Your case just asks for ranking. There is no strategy available to the SMcG fans there that anybody would risk applying. Nominating just one of your 2 favourites is foolish unless you are sure about the other, and that is not the case here. Not nominating other favourites can work, but if they are other favourites, why do you want to hurt them?

It's not that there are no strategies at all to Approval. It's just that they all fairly ineffective, with low gains for the risk they provide, and hard to be sure will work. That is about as good as you can get in a strategy free system.

I am sure there were 2 fans like you describe, but how would they have done their ballots differently without foreknowledge? Or even with the foreknowledge that "it's probably gong to be close and only one will make it," which is actually an unrealistic amount of foreknowledge with a secret ballot.

However, it does not take a lot of strategy to affect the close races. Even if only 1% of nominators do strategy, it is going to affect races on a fairly regular basis, and the more strategy there is to do, the more this will happen. The battle for 5th and 6th -- possibly a career changing event for some writers -- is often quite close.

#230 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 12:52 PM:

A P.S. to my 228:

As a variation of the simple nomination system (everyone nominates one work, everything makes the ballot), one interesting thing would be to report all nominations on the ballot, but also include how many people nominated that work, strictly as informational purposes. A person who does not have time to read the entire list (and who would?) could decide for themselves how many people need to recommend a work before he or she will consider reading it. And even then, if something with fewer nominations still catches his eye, there nothing stopping him from reading it too. The advantage of this simple system is that it puts the power to decide what to read and vote on in the hands of the individual Hugo voter. No one can stop them from reading and/or voting on anything they like.

Kilo

#231 ::: Steve Halter ::: (view all by) ::: April 15, 2015, 01:01 PM:

Keith "Kilo" Watt@230:I kind of like that. Here's everything* that was nominated with some information (or not?).

*culled for invalid nominations.

#232 ::: kimiko ::: (view all by) ::: April 15, 2015, 01:05 PM:

Keith "Kilo" Watt@230, Steve Halter, 231,
That sounds like a pretty attractive proposal to me too. I have some doubts about it getting adopted, but I really like it.

#233 ::: nathanbp ::: (view all by) ::: April 15, 2015, 01:10 PM:

P J Evans @227: I don't feel strongly that the nomination ballot should be ranked (my preference is for RAV, 3c-3). Having said that, as I see it the case for ranking the nomination ballot is if people have strong preferences within their list on which work they'd like to see nominated, instead of being equally happy with any of them being nominated.

Brad from Sunnyvale @230: The point I was perhaps failing to make was in response to your comment @223 about the proposed systems:
But when it becomes better for you to not give your true opinion because you can do better for "your" candidates by lying, I say "yuck,"
In all systems, with perfect foresight of all the other voters, there are going to be cases where it's better not to give your true opinion. In a good system, including I believe all of 3a-d above (and the current system), without perfect foresight, voting your true opinions should be the best you can do. So just because there are hypothetical examples that make you say "yuck" does not mean the system would produce those results in practice.

#234 ::: Joshua Kronengold ::: (view all by) ::: April 15, 2015, 01:39 PM:

I am fond of the no system nomination system myself (i.e. take N nominations from each person; report the results minus invalid nominees and let people vote for what they want), but there are several reasons it's entirely impractical:

1. The quality of the Hugos is directly predicated not just on high nomination quality, but also on people generally watching/reading everything in a category before voting. That's not possible for this system, and the difficulty of convergence would be a real hit. Also, the Hugo Packet (which really, really helps with this) would more or less be demolished by this scheme.

2. It's already a big job to eliminate invalid nominees only culling from the top. Doing the same but listing everyone would be a collosal job, and we simply don't have the manpower to do it reliably.

3. It's important to give nominees the chance to decline nominations. This system wouldn't allow for this, which is a big no.

4. Except in years like this one, being a Hugo nominee is a big deal; this would pretty much put paid to this idea.

So no. And also no.

Jameson: SDV seems viable as an alternative tiebreaker for SDV that is kinder to works that tie without a big slate -- while still being quite harsh (presuming that you use the original tiebreaker for otherwise unbreakable ties) for constructed ties due to coordination. Since FAPE is really a slight modifier to vanilla SDV for extraordinary situations, I suggest referring to the combination as SDV/FAPE.

I don't think overvotes as you described are a good idea with SDV -- adding that complexity throws away the biggest advanvage that SDV (or SDV/FAPE) has over other proportional schemes -- the ease of understanding by the average voter/WSFS member. I could see a similar scheme that assigned votes after elimination not evenly among your votes, but instead to the weakest thing you selected -- so after elimination your vote would be divided 1/5 / 1/5 / 1/5 2/5. But I don't think that gives enough advantage for the simplicity cost.

I -could- see an overvote scheme in RAV. There, you could compare the winner to the nearest contender. On a tie, you use straight weighing, but otherwise you adjust that weighing based on the proportional number of ovtervotes. I'm not sure what the right spillover scheme would be -- the simplest one would just be to multiply the weight by votes/nearest competitor, maxing out at 1. So if you vote for something that's twice as popular as anything else, you don't lose any votes at all (giving you no real incentive to avoid voting for the popular thing you really like; the more votes there are for it relative to anything else, the less your other votes will be affected.

This would argue going for a harsher rather than a kinder weight (powers of 2 is good here), since the overvote system will act as a counterweight.

#235 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 02:09 PM:

@234: the "FA" in FAPE is basically synonymous with the "SD" in SDV. So if you want a hybrid name, it would be SDV/PE. So mote it be.

The overvotes thing was my first attempt at getting the "don't waste your vote unless it's needed" idea inspired by J Thomas. SDV/PE was a later, simpler, better attempt to get the same effect. So forget about the overvote thing.

An overvote scheme in RAV would be basically STV with approval ballots, fractional assignment, and a flexible quota. For instance, you could run RAV once to find the 6th-place total as a rough quota, then run unranked-STV using that quota. Or you could keep doing it over and over again using the 6th place total as a new quota each time until it converged (which it would). I think this system would have good properties — comparable to SDV/FAPE — but it's really hard to explain.

#236 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 02:13 PM:

This last suggestion I made in @235 is a bit like a second-price auction, only it's a "proportional sixth-price auction". There may be a way to figure out the result using systems of linear equations rather than running until convergence. And while nice linear convergence would indeed be the norm, there could be edge cases where it stuttered between two candidates. If that happened, you could take whichever of them led to the lowest quota at the end.

#237 ::: Tammy Coxen ::: (view all by) ::: April 15, 2015, 02:45 PM:

Any proposal that "everything makes the ballot" is not paying attention to the number of works/people that are nominated. In 2014 the lowest numbers of unique entries in a category were 100 (semi-prozine) and 124 (editor-long form) and the highest numbers were 728 (short story) and 587 (novel). With the rest of the categories being scattered in between.

Which is also a question for folks speaking to computational requirements - has the huge number of entries been considered?

#238 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 02:52 PM:

Joshua @234: Ooh, I like the idea of RAV with overvote, though I don't know if it makes it too complicated for the masses. It reduces the apparent incentive to omit favorites from the ballot. (As I've mentioned, that isn't as good a strategy as it appears, but my conversation with Brad convinced me I'd need to offer each Hugo voter a 4-hour class on game and probability theory to convince them of that, and that's probably just a bit impractical.)

I'm interesting in simming SDV, RAV, and their tweaked versions some more, but I'm inclined to wait until we have the 1984 Hugo nominations data. Admittedly it may not be representative of the distribution of votes 30 years later, but it's probably closer than my random ballots are.

#239 ::: P J Evans ::: (view all by) ::: April 15, 2015, 02:53 PM:

237
I doubt it.

#240 ::: Tammy Coxen ::: (view all by) ::: April 15, 2015, 02:55 PM:

Sorry, I said "in 2014" above - that data is for the current Hugos, which recognize works from 2014, sorry I was unclear.

#241 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 03:01 PM:

Tammy @237: Thanks for that info! I knew they'd be large, but not exactly how large. That's one of the reasons I wrote off PAV, which with 728 candidates would run until the heat death of the universe. (I'm kidding. Mostly.)

RAV scales up very nicely because you only need to iterate 5 times, and the number of computations you need varies linearly with the number of candidates and voters.

SDV has more computational complexity, but with a fixed number of voters it should scale up as the square of the number of candidates, which isn't unreasonable. (Some implementations of SDV might begin by discarding all ballot entries appearing less than N times. In addition to reducing computations, that might reduce the pain of Hugo administrators, who otherwise might spend a lot of time figuring out if two similar entries appearing on one ballot each represented two votes for one thing or one vote each for two things.)

#242 ::: Tammy Coxen ::: (view all by) ::: April 15, 2015, 03:03 PM:

I was really excited about this thread and all the options. But as the conversation has evolved, I'm afraid that all this effort is for naught. The inability of any of the proportional voting systems to elegantly handle a declined nomination is going to make it dead in the water in the business meeting, I think. I think you'd be hard pressed to find a Hugo Administrator who'd support something that required re-running all the data whenever someone declined. What if a decline was received an hour before the scheduled announcement? What if, like now, works were deemed ineligible and removed after the ballot went public?

If past and future Hugo administrators at the business meeting say it's not workable, it will not pass, no matter how good of a system it is.

#243 ::: P J Evans ::: (view all by) ::: April 15, 2015, 03:04 PM:

241
Generally, the voters are smart enough to only list something on the nomination ballot once. Generally....

(I want to get hold of the data because the Hugo site only has the list of nominees, not the numbers.)

#244 ::: Buddha Buck ::: (view all by) ::: April 15, 2015, 03:04 PM:

Tammy @ 237:

The most computationally complex proposal that I saw was (at first analysis) O(n^5), meaning that the time requirements scaled at approximately the fifth power to the number of nominees. That was the class of proposals which attempted to find the set of 5 nominees which maximized "happiness". With 587, n^5 is in the trillions, so it's probably infeasible to use a naive approach, but there may be not-so-naive approaches which would work better.

REV, which seems to be one of the favored proposals, appears to be doable in O(n^2), or O(n^3) at the worst. For n=587, that's relatively trivial.

None of the methods appear to have extensive memory requirements.

#245 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 03:11 PM:

Joshua@234:

At least two of these are indications of what you value about the Hugo process, and I think they are good ones. Some people may also value them, some people may not. What I want to be clear about is that they are not, in and of themselves, objectively good or bad. It's painfully apparent that none of these systems are ideal. Trade-offs are going to have to be made. We have to decide what trade-offs we want to make, though, of course. Some may agree with you and value (for example) the Hugo voter packet over a simpler system. Some may not. It may even be that -most- people value the packet more, but I don't think that's been established (certainly I don't, but I have no way of establishing whether I'm in a majority or not).

To address your #1, I strongly disagree that the quality of the Hugos has anything to do with people watching/reading everything in the category. There's room for disagreement, of course. As mentioned above, the Hugo packet is an excellent freebie that I would be sad to see go, but if that were the cost to pay for a better system, I personally would pay it. It may be that not many people would.

Regarding #2 and #3, this is manpower issue, and is certainly not insurmountable. In fact, I'd put my money where my mouth is and volunteer. Alone. Yes, it's a big job, but I've handled bigger ones. So, I don't think we can automatically declare that this is a "no" based solely on these criteria.

And finally, regarding #4, this is again a value statement, and one I happen to agree with. Saying something is "Hugo nominated" used to mean something, and I agree it would not under this system. That would be a cost, and maybe more than most fans would want to pay. For me, even recognizing that it is a cost, it's one I'd gladly pay, but again, I may be in a minority.

I guess what I'm trying to point out is that every single system we are discussing, without exception, involves a cost. The real question is what costs are the fans willing to pay? If we want a system with no cost, then we should give up, and simply make no changes at all. There are those on this board and others who are proposing exactly that, and I can see why. We're just going to go in circles until we do.

===

Tammy@#237: This is a really excellent point. And if the number of nominations is too high, is it reasonable to set a threshold cut off as we do now? For example, if a work doesn't get at least, say 5% of the nominations then it is not put on the ballot? Everything else would be kept as originally proposed.

I recognize and accept the argument that "patching" a new system could be seen to be worse than using a known and well-studied system. I get that. But it is looking to me like none of the well-researched systems are turning out to be very popular. They were new once as well, so for me, I'm not intimidated by trying to find an alternative.

Just my thoughts,
Kilo

#246 ::: felice ::: (view all by) ::: April 15, 2015, 03:53 PM:

Jameson Quinn @ 209: "2. Uses an approval-like ballot, as with the current system. That rules out standard STV."

Nominations use a write-in ballot - that's inherently ranked, so STV (ranked with redistribution of overvote) would not need any change to the ballot at all. And I don't believe knowing that ranking mattered would discourage nominations. With STV, you can rank an obscure work over a popular one, to maximise the obscure work's chances without taking nominations away from the popular work if/when the obscure one is eliminated; that encourages nominating a full ballot rather than just your first choice, which is a good thing.


Jameson Quinn @212: "As for the idea of publicizing a "long list" of the 15 frontrunners a month or so before nominations close, and allowing voters to redo their nomination ballot on that basis if they wish: it's not a bad idea. It would help with the problem where too many voters supported zero winners because the votes are too dispersed. But it wouldn't solve the problem of slates, and could even potentially make it worse."

How could it make it worse, and why wouldn't it solve the problem? It could drastically reduce dispersal of nominations, pushing up the number of nominations received by non-slate works, and thereby increasing the number of supporters a slate needs to overwhelm legitimate voters. No reasonable system is going to help if the slate supporters become a majority, and I think this stands the best chance of keeping slate works off the final ballot entirely rather than merely reducing them to one or two places (and through the obviously fair method of other works getting more nominations - nothing for Fox News to complain about there :)


P J Evans @216: "A motion has been proposed to change the number of nominations that can be made from five to four."

Which is trivial for the Slate to overcome with a simple randomiser to assign Puppies with 4 of the 5 slate works each. The slate power is slightly reduced, but there are less non-slate nominations too, which will at least partially cancel out.


P J Evans @227: "Traditionally, we haven't had rankings on nominations because we're creating a list of the stuff that people like and think is good enough for a rocket. If we have to rank them at that stage, why bother with a final ballot?"

Because more people vote on the final ballot, and the nominators might change their minds after reading shortlisted works they were previously unfamiliar with. Ranking removes the incentive to leave good-but-not-your-first-choice works off your nomination ballot, and we want as many genuine nominations as possible to counter slates.

#247 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 03:57 PM:

Andrew M@162: "It's a rare voter who decides to vote for one conservative, one liberal, one socialist, one libertarian and one green, in the interests of diversity"
Actually, I do vote somewhat like that in primaries. I'm one of Those Annoying Libertarians, and California has a "top two open primary" system where you can vote for anybody in the primary and only the top two get into the general election, which usually means a Democrat and a Republican, but sometimes two D or two R. Since I don't want Republicans getting elected, and the Democrats will be getting my vote in the fall unless somebody gets lucky, I vote for minor parties and independents in the primary; last election that included a Green and an Occupy activist in the various districts I'm in, as well as a Libertarian and the potential second-spot Democrats. But it's a hopelessly gameable system that's designed to keep third parties from annoying Democrats and prevent the traditional open-primary problems of one big party voting for unelectable candidates from the other big party.

#248 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 03:59 PM:

Tammy @242: Assuming the computer implementation is reasonable (and most of these would need to be implemented via computer - RAV is almost reasonable, but anything on the complexity level of STV isn't), the new results could be generated in a minute. It would also be an easy programming problem to, when the nominee list is generated, generate the "6th choice" for each possible withdrawal. (In the absence of slates, these will often all be the same.) So re-running would only be necessary if multiple candidates declined in one category.

Alternatively, you could figure out the 6th, 7th, etc. place candidates when you run the original election. They could yield a somewhat skewed slate in terms of being proportional, but probably no worse than a non-proportional method would yield.

Kilo @245: If the ballot has a large number of nominees and most people haven't read most of them, voting blocs become much more powerful in the finals. Since many ballots will "run out" of votes, non-bloc works may never accumulate enough votes to defeat the bloc works.

You can certainly attempt to develop a new system from scratch, but keep in mind that this is a field of study that's been active since at least the 13th century (Ramon Llull's work) and heavily researched over the last 50 years in a wide variety of settings. The Academy Awards commissioned studies when they adopted STV. The paper I cited on SAV, PAV, and RAV notes that there's great interest in them in AI research, as a way of forming a consensus based on the different results from different modules or sensors.

If I want to bake my first loaf of bread, I can use a recipe, or I can throw random ingredients into a pan, see what happens, and then try again. If I iterate the second process enough, I might get bread eventually, but I'll produce a whole lot of inedible junk en route. If I met a baker and asked them to explain what was wrong with each set of ingredients, it would be understandable if eventually they said, "Why don't you try one of these reliable bread recipes?"

I also have to admit that I'm not quite getting the conclusion "none of the well-researched systems are turning out to be very popular" from what I see in this thread, which is "the same three people are loudly and repeatedly dissatisfied with the ones that have been presented." (And we by no means have exhausted "the well-researched systems." We've been discussing "the ones that people with knowledge of this area of study and those who participated in the last thread thought were most appropriate to this situation.")

I have tried to avoid appeals to authority, I'm happy to explain things, but when I take an hour to write up a summary of what I know from the existing research and the response is basically, "Well, what do those experts know?" there doesn't seem to be much point in continuing that conversation.

#249 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 03:59 PM:

Tammy@242: Interesting point. The simplest answer to that is to have the rule for replacing declined nominations be to use Joshua@88's #1: "When a nominee declines, offer the slot... to the 6th place candidate".

How crazy is it to do that? That is to say, how likely is that to give a different nominee than you would have gotten by re-running the process with the declined candidate removed from all ballots? Not particularly likely for any of the systems we're considering. But if you want to compare the systems, it's relatively more likely with RAV, in the middle with SDV, and relatively less likely with SDV-PE. That is, SDV-PE has the least "problem" with ties. That's because in SDV-PE, being on a lot of ballots can tend to protect a candidate from elimination, and that doesn't depend on which other candidates are in the election at all.

So, that's another argument in favor of SDV-PE, which is continuing to grow on me.

#250 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 04:10 PM:

nathanbp@174 - Huh? Your choices for what to do if somebody withdraws (1: Replace them with nominee#6,etc. vs. 2/2a/2b: Remove them and rerun the nominating-election) would produce radically different results.
If the slate voters uniformly picked their top five candidates, and dominated, then #6 is the first non-slate candidate, etc., as happened in Best Novel this year. But if you rerun the election, providing information about the top N choices and a week or two for "All of fandom [to be] plunged into war", then probably the slate loses.

If you allow this to happen more than once, and don't restrict new nominations, the Puppies can probably keep all of fandom in war for a month or more; even if you only allow one round of reelection, the Committee to RE-Elect the Puppies can still cause few weeks of chaos, cackle their Bwahahah, and invite the Reavers in to help.

#251 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 04:16 PM:

Minor followup to me@250 - For Best Pro Artist, since one of the Puppies was deemed ineligible, nominee#6 was chosen, and was also a puppy (rabid but not sad), so there's also a case for what happens when slates don't vote quite uniformly, or there's more than one slate, or whatever.

#252 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 04:34 PM:

felice @246: I think Jameson means "ballot" in a voting-theory sense of "what information are we requesting from voters," not "what does the ballot physically look like?"

There's some research suggesting that approval ballots lead to higher participation levels than ranked ballots, but (as far as I know - this is more on the poli sci side of voting theory, and I'm more on the math side) it isn't conclusive. We'd probably have to try it and see how participation rates are effected (though I suspect they'll be up for the next few years regardless of rule changes or their lack!).

I do note that STV is not quite as strategy-free as you suggest. You can safely rank an obscure work over a more popular one if you are *sure* the obscure work won't compete with the popular one for a nomination, but if there's any possibility they might compete, ranking the obscure work higher can cause both to fail to be nominated.

A danger in releasing votes and then allowing everyone to change the vote is that bad actors (and by this, I don't mean just "slate voters" but "people intending to gum up the process") get more opportunities to game the system. If they enter one set of votes en masse before the data release and then change them all, they can potentially affect voter behavior. (Or they can claim they changed them all and then not do so, etc.)

Bill Stewart @251: In this context, "rerun the election" means "take the existing ballots, remove the candidate who declined, and generate a set of winners again," not physically having another election.

#253 ::: nathanbp ::: (view all by) ::: April 15, 2015, 04:35 PM:

Bill Stewart @250: Sorry, by re-run the election I mean take the existing set of ballots, cross off the nominee who refused, and then recalculate the results under whatever voting system is being used. I did not mean to suggest that everyone be given a chance to vote again, that would indeed not work at all.

Jameson Quinn @249: Could you point me to which post(s) you described SDV-PE in? I seem to have missed it and don't see exactly which system you're referring to.

#254 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 05:05 PM:

@253: When I first proposed SDV-PE (single divisible vote with popularity eliminations), I called it FAPE. The rules are:

(Optionally: First, clean things up by eliminating all candidates with fewer than N votes, where N is, say, two thirds the number of votes for that the weakest nominee last year had. This shouldn't change the result, but makes things simpler, and involves less drudge-work cleaning up irrelevant data.)
Then:
While there are more than 5 candidates:
Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
Find the two candidates with the smallest pile of fractional votes.
Eliminate whichever of those two is on the fewest ballots.

Here are the benefits of this system, as I see them:
-This is easy to program, and in my opinion it's easier to intuitively explain or grasp than RAV.
- Of the proportional systems that have been proposed here, it does the least to encourage the "don't nominate popular works" strategy (both in a strict strategic sense, and, I'd guess, in an intuitive sense of "how likely is it that voters will try that strategy in real life"). This means that it would encourage the most "broad" voting of any of the proposals here, which would help fight slates by encouraging more nomination votes overall, and thus a higher threshold for slates to be winners.
-It is the most stable under declined nominations; while no proportional system can be perfectly stable in that case, with SDV-PE, eliminating a candidate before the election is (as) unlikely (as possible) to change the relative ranks of other candidates.

In other words, I think it's a good system for the Hugos. (I do not think it would be a particularly good system for a political election, by the way.)

#255 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 05:12 PM:

If you think of each ballot as a pizza of a unique flavor, then the elimination step of SDV-PE is: of the two candidates with the least pizza, eliminate the one with the fewest flavors.

#256 ::: felice ::: (view all by) ::: April 15, 2015, 05:17 PM:

Cheradenine @ 252: "felice @246: I think Jameson means "ballot" in a voting-theory sense of "what information are we requesting from voters," not "what does the ballot physically look like?""

The distinction is pretty ambiguous. We currently ask voters to write down up to five works in order (because there's no way to avoid ordering them), but tell them we won't take the order into account. Is using the order information instead of discarding it really a change in the information being requested?

"There's some research suggesting that approval ballots lead to higher participation levels than ranked ballots, but (as far as I know - this is more on the poli sci side of voting theory, and I'm more on the math side) it isn't conclusive."

That could well be the case in selecting from a shortlist; ticking your preferred option is certainly less work than putting numbers next to several options. Does it apply to write-in ballots, though? Where the information requested is the same in either case, and how many other people nominate the same candidates is much more important than the order?

"I do note that STV is not quite as strategy-free as you suggest. You can safely rank an obscure work over a more popular one if you are *sure* the obscure work won't compete with the popular one for a nomination, but if there's any possibility they might compete, ranking the obscure work higher can cause both to fail to be nominated."

Good point. Does #173 help with this? "Of the two works with the [fewest first place nominations], eliminate the one which is on fewer ballots" rather than just eliminating the one with the fewest first places nominations each round.

"A danger in releasing votes and then allowing everyone to change the vote is that bad actors (and by this, I don't mean just "slate voters" but "people intending to gum up the process") get more opportunities to game the system. If they enter one set of votes en masse before the data release and then change them all, they can potentially affect voter behavior. (Or they can claim they changed them all and then not do so, etc.)"

Bad actors are always going to be a minority, or there's no way to save the awards. If they get a slate in the top 15, it can be ignored and people can choose from the remaining 10. What else can bad actors do? Nominate works they think genuine voters will like, to improve those works' chances over other works we like? That hardly seems like much of a win for them. Claims of changed nominations can safely be ignored, whether they're true or not; we don't care what bad actors nominate, we just want to get more genuine nominations. If bad actors have sufficient power to take over the entire top 15, then they've done us a service by warning us, and a counter-slate can be prepared that ignores the official top 15 (counter-slates aren't good, but they're certainly the lesser evil in this case).

#257 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 05:28 PM:

Cheradenine@248:

Oh, whoops, profound apologies. I didn't mean to imply "Well, what do those experts know?" at all. I'm truly sorry I gave that impression. My observation was that we don't seem to be any closer to coming to a consensus about any of the option 3 systems (at least not that I see -- we seem to keep proposing and discarding or modifying several different systems). That is entirely different from saying none of them are popular, so I definitely shouldn't have phrased it that way. Your work in particular has taught me more about voting systems than I've learned in a lifetime so far, so I very much appreciate that. Please don't stop!

So let me ask plainly, is there a consensus forming? Can someone lay out the pluses and minuses (there have to be both in any system, of course -- not a bad thing)? It might be good to have a summary of where we are at this point, do you think?

If the ballot has a large number of nominees and most people haven't read most of them, voting blocs become much more powerful in the finals.

Do they? Okay, that doesn't seem intuitively obvious to me, but I think I can see it. Let's see if I can reason it out , and let me know where I've got a misconception. If there are a large number of nominations, then there might also be a large number of works that serve to dilute the concentration of votes in the finals, correct? And so, those votes are distributed so widely that a voting bloc (as opposed to a nomination bloc) can get more than the 50% of the total number of votes, right? I think I see that, but if that's not right, then my next statement isn't going to valid: Wouldn't that mean that they legitimately have more than half of the total electorate anyway? And so actually deserve to win? Under the voting rules, if they don't get quite 50%, then the last place candidate is eliminated and those votes are added to the second-place choice. Does this still allow them to run the election? If so, isn't that possible under the current system as well?


If I met a baker and asked them to explain what was wrong with each set of ingredients, it would be understandable if eventually they said, "Why don't you try one of these reliable bread recipes?"

A very adept analogy, and I do agree with what you're saying. I suppose my feeling was that no one was willing to even consider another system because it was not well-studied. I've never thought that was a path to innovation, but as you point out, the field is 800 years old, so maybe there is literally nothing new under the sun to really even consider. I think that's the point you're trying to make, right? In that case, if I'm wrong that no consensus is forming, then maybe we could move on to a way to "sell" the consensus candidate system.

I can definitely see why you wouldn't want to waste your time with an amateur system when there are perfectly good and well-researched systems out there. If any of those systems are viable (which, of course, means we have to define what viable means) for the Hugos, then I'll strongly support them. I'm not seeing that the experts here agree on that yet, but maybe I'm missing that part of the conversation. Could it be that we (and mostly I mean you folks, as the experienced ones) just haven't settled into a choice yet, and I'm not being patient enough? :)

I honestly don't mean to offend anyone or disregard their work. Yours in particular has been very helpful, and I've never seen you resort to saying "trust me!" I do appreciate that.

Again, apologies,
Kilo

#258 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 05:42 PM:

Jameson#254:
A quick comprehension check (mine, not yours ;-) )...


While there are more than 5 candidates:
1. Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
2. Find the two candidates with the smallest pile of fractional votes.
3. Eliminate whichever of those two is on the fewest ballots.

So the single divisible vote part is step 1, correct? And the popularity elimination part is steps 2-3?

I'm trying to figure out the interaction between the two parts as I go back through and try to follow your scenarios for each part.

Thanks,
Kilo

#259 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 06:08 PM:

Kilo @259: My apologies in turn; I think my frustration with a couple different phenomena and people focused inappropriately on your last comment, and you got a rant that only partly related to you. I'm sorry for that, and I appreciate your apology and your willingness to engage.

With a large number of nominees in an STV election, the danger is that ballots will be exhausted (that is, all choices on them will be eliminated) before the end of the election. So if the bloc only has 15% of the vote, and the order of elimination takes out any candidate that appears on more than 15% of the ballots, the bloc can end up with the win despite having a minority of votes. They were a minority of the electorate, but now the electorate has shrunk so much that they have a majority of the remainder.

(One way to deal with this would be to declare no winner if the winner doesn't have a majority, but I suspect that might mean that the short story award, for example, is never given again.)

It's very helpful to hear your view of the discussion, because my sense was that what we've done is narrow down the map of voting systems to a fairly small corner of territory (essentially, proportional systems that reduce slate power by either reducing the weight of ballots that have already gotten nominees, or increasing the weight of ballots that haven't gotten nominees yet or had candidates eliminated) and are now tinkering around with the details. But it makes perfect sense that if you don't know the landscape, there's no easy way to distinguish between "tinkering with the details" and starting over from scratch.

At this point, I think the most helpful step will be to get some actual ballot data so we can run some better-informed simulations and talk about how the results vary with each method, and which set of results is most desirable (as well as other factors like difficulty of implementing and explaining each system). I'm certainly happy to answer questions and talk over systems in the interim, but doing a full analysis of the pros and cons of a new system is pretty time-consuming. It may be more productive to discuss a set of goals and how we could modify an existing system to better achieve them than to start with a new set of rules and figure out their implications.

#260 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 06:09 PM:

That should, of course, be Kilo @257. Does the numbering sometimes change, or am I just being especially bad tracking numbers over the last few days? (Rather embarrassing in my line of work.)

#261 ::: P J Evans ::: (view all by) ::: April 15, 2015, 06:11 PM:

260
If the gnomes take out a duplicate post, it messes with the numbers. But they'll usually say something.

#262 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 06:18 PM:

Keith@258: Yes, that's correct.

#263 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 06:27 PM:

@257, @259: I agree with Cheradenine. Also, note that there are now several "custom made" voting systems under discussion. These may have slightly improved characteristics, but really I think that the (non-drive-by) experts in this thread generally agree that even if we had to just take one of the better "off-the-shelf" voting methods (say, RAV with Saint-Lägue weights), it would work pretty well in general, and in particular adequately resolve the slate crisis. So yeah, we're tinkering at this point.

#264 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 06:29 PM:

Cheradenine@252, nathanbp@253 - Thanks for the clarification. That makes a lot more sense (and yeah, with the current Hugo nominating system, that should produce the same results, while under at least some of the various voting systems that have been proposed, you might need to do a whole recalculation and get a different result.

#265 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 06:39 PM:

Cheradenine@259

It may be more productive to discuss a set of goals and how we could modify an existing system to better achieve them than to start with a new set of rules and figure out their implications.

I wholeheartedly agree with this. It's sort of the point I've been trying to make in some of the other threads in the discussion. Until we all agree on what we value in a system, it would be rather difficult to choose any system, particularly since my read on the discussion is that many of our goals are contradictory.

It seems to me no system is perfect and all will have costs/trade-offs. Is this a fair statement of voting theory? (My engineering background biases me to see all systems that way, of course, so I may have blinders there.)

Kilo

#266 ::: P J Evans ::: (view all by) ::: April 15, 2015, 06:46 PM:

265
Stuff I've read in the past on voting systems (Scientific American had at least one article on them) says that every system has at least one flaw.

#267 ::: nathanbp ::: (view all by) ::: April 15, 2015, 07:32 PM:

Kilo @265: There is actually a proof that no voting system can be perfect (under a defined set of conditions). See Wikipedia's article on Arrow's impossibility theorem.

#268 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 07:42 PM:

PJ and Nathan, thanks, I appreciate it.

A quick request:

I've been reading up on voting systems and the math behind it in hopes that I can communicate more easily in the discussion. Are there a few particular canonical references that I should be looking at? I'm an astrophysicist, college professor, and former NASA engineer, so consider me "intelligent but inexperienced," to use the famous quote. :) My statistics background is pretty good (I also have a grad degree in education research), but not at the specialist level. Any suggestions at an appropriate level would be appreciated!

Thanks in advance!
Kilo

#269 ::: J Thomas ::: (view all by) ::: April 15, 2015, 07:52 PM:

@248 Cheradenine

I have tried to avoid appeals to authority, I'm happy to explain things, but when I take an hour to write up a summary of what I know from the existing research and the response is basically, "Well, what do those experts know?" there doesn't seem to be much point in continuing that conversation.

I welcome your explanations. I notice that the Hugo goals are poorly defined but sound like they might be pretty unusual. Existing methods might or might not be a good fit to them. Of course, new methods might or might not be a good fit also. At least with well-studied methods you have a better idea what you're getting, whether or not it's what you want.

I want to look closely at Hugo goals and see how much of a consensus we can get about those.

#270 ::: J Thomas ::: (view all by) ::: April 15, 2015, 08:01 PM:

@254 Jameson Quinn

When I first proposed SDV-PE (single divisible vote with popularity eliminations), I called it FAPE. The rules are:

(Optionally: First, clean things up by eliminating all candidates with fewer than N votes, where N is, say, two thirds the number of votes for that the weakest nominee last year had.
This shouldn't change the result, but makes things simpler, and involves less drudge-work cleaning up irrelevant data.)
Then:
While there are more than 5 candidates:
Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
Find the two candidates with the smallest pile of fractional votes.
Eliminate whichever of those two is on the fewest ballots.

That's beautiful! It deserves careful simulation to look for flaws, but it *looks* great.


#271 ::: conrad6 ::: (view all by) ::: April 15, 2015, 08:07 PM:

Nn-sltns t vry rl prblm!

Frst prblm - thr hs bn scrt slt snc th 1990s (SMF/SJW/Shvr cnsprcy(1950s fr y jvnls)).

Scnd prblm - y hv pkd th slpng br t mny tms, nd thr s nw nn-scrt slt. Dl wth t - thr vstrs ctlly by bks (s lng s thy dn't hv th dprctd Hg/Nbl tx stmp).

Thrd prblm - why d y dny nclsvnss t fns? Wld y ccpt $40.00 pll tx t vt n lcl lctn? ntnl lctn?

Frth prblm - Why th ht fr n pn slt? nlk SMF ths ws LWYS t n th pn n wb st.

S th vtng lgrthm nd dt r nt th prblm. nlss sbrnd lk mny lctrnc vtng mchns.

Th prblm s nt th bllt r th mthd - t's tht y ddn't gt th rslt tht y wntd.

#272 ::: P J Evans ::: (view all by) ::: April 15, 2015, 08:18 PM:

271
Y'all own it, entirely, because y'all invented the first problem.

We are not responsible for your actions or your lack of money. (Many of us are spending our grocery money to have a say in the mess y'all created.)

#273 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:56 PM:

@268: I can't think of anything that's perfect for this situation. "Gaming the Vote" is a good read, but it doesn't spend too much time on the technical side, and focuses on single-winner systems. "Choosing in Groups: Analytical Politics Revisited" (as discussed above) isn't bad, but it's also a bit skewed to single-winner issues and spatial models (which are pretty wrong for a Hugo context). There's a UN publication on proportional representation systems, but it's for politics, and only really talks about systems used by existing political democracies. Hmm... looking through my library of papers, I'd say "Divisor-Based Biproportional Apportionment in Electoral Systems: A Real-Life Benchmark Study" is not a bad one as a technical intro to PR.

#274 ::: Kimiko ::: (view all by) ::: April 15, 2015, 09:09 PM:

269,
On Hugo goals:
picture a 2x2 grid:
Good & Known ........................ Not-so-good & known
Good & Unknown .....................Not-so-good & unknown

For the nomination round, we want to select items from the first column, and we are particularly interested in the Good & Unknown, since the Good & Known can take care of themselves. We don't want to over select from Good & Known, since we want the final vote to be about ranking, not the nomination.

Our current problem is how to prevent noise from drowning out the signal.

#275 ::: conrad6 ::: (view all by) ::: April 15, 2015, 09:17 PM:

@ P J ?? r y sppsd t b frm Wst Vrgn? lbm/ Wst L? Pls lrn t tlk.

y'll??? WTF tht mn? Y sm t b prtndr lk ----- 'll stp xcrtng y s wn't gt bnnd, bt y r rlly stpd.

#276 ::: P J Evans ::: (view all by) ::: April 15, 2015, 09:23 PM:

277
Try California and four years in Texas.
Also try reading before you make any more remarks about others, because y'all sound like a troll to me.

#277 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 09:25 PM:

"Y'all" is a great word. Everybody understands it, and allows English to join most languages in distinguishing singular and plural in the second person.

@270: I'm glad you like it. Your proposal was the inspiration.

#278 ::: Lori Coulson ::: (view all by) ::: April 15, 2015, 09:33 PM:

conrad6 -- you are nekulturny, comrade.

And if there is anyone lacking in intelligence in that exchange, you'll see them in your mirror.

#279 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 09:33 PM:

I was born in the great thriving metropolis of Bells, Tennessee, population 635 and a couple of dogs. It was years before I knew there even was a second person pronoun other than y'all. :)

Kilo

#280 ::: P J Evans ::: (view all by) ::: April 15, 2015, 09:43 PM:

281
It's my understanding that the plural, in Texas anyway, is 'all y'all'. (I don't generally like their political views, but most of the people I met were fine, and I miss them.)

My grandfather was born in, or near, Plummer's Landing, KY, population almost enough to count.

#281 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 09:57 PM:

Back on topic: Jameson, I also love the SDV/PE proposal -- the simplicity of it is marvelous. As I mentioned, I've been working through the examples that have been given, and it's growing on me even more. That's in no small part to your effective and clear statement of how the system works, so hat's off to you for that.

I'm also interested in the simulations that Cheradenine is running; I had actually started coding something myself when I realized that had already been taken care of. It'd be educational to see how it performs in various other situations, particularly since that would give scenarios that could be put out to further explain the system (as well as look for problems, of course).

Also, thanks for the recommendation; I'll see if I can find that reference.

Regards,
Kilo

#282 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 09:59 PM:

felice @256:
"Is using the order information instead of discarding it really a change in the information being requested?"

This is not a super scientific answer, but based on the number of "Top 5"/"Top 10" Tumblr memes I see where people say "In no particular order, because I couldn't possibly rank them," I think it's not that numbering items is by itself more work, but that assigning meaningful rankings to a set of 5 items involves a higher cognitive load.

Whether this would have a significant effect on Hugo nomination rates is harder to say. I got the impression in the last thread that people thought it would be an issue, I guess?

Good point. Does #173 help with this? "Of the two works with the [fewest first place nominations], eliminate the one which is on fewer ballots" rather than just eliminating the one with the fewest first places nominations each round.

I think that if something similar is applied to STV, it does reduce though not eliminate the spoiler effect. I've worked more with IRV than STV, though, so my intuition for the one-winner case may not match the behavior with multiple candidates.

Bad actors are always going to be a minority, or there's no way to save the awards. If they get a slate in the top 15, it can be ignored and people can choose from the remaining 10. What else can bad actors do?

Having thought more about it, a single slate is probably best off not even to release the slate before the revelation of nomination counts. The more slate works on the ballot, the more likely other people's votes will clump on the non-slate might actually benefit most from the list of 15 being a bunch of varied, highly appealing works to split anti-slate votes.

If there are 3+ slates and they vote early, then there's a danger that all 15 works will be from slates.

With this suggestion, the early nominations sort of serve as a nonbinding poll of what people are planning to nominate, which could serve as a crowdsourced recommended reading list. I don't know how many people would realistically have time to read a bunch of works during the 2nd half of the nomination period, but they could, or could just let their ballots stand.

I do think this approach avoids many of the flaws of either open continuous voting or releasing nominations at the halfway point but not allowing changes. Ultimately how well it would work comes down to predicting how voters would respond, which is a guess. I'm not confident that by itself it would be a sufficient anti-slate tactic, but if it were the only reform we could get through, I'd at least be interested to see what happened.

#283 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 10:04 PM:

@274: Hmm..that article seems to be behind a paywall. I'll have to see if we get Management Science on campus.

#284 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 10:31 PM:

Kilo @268: I don't know of a great overview, alas; most of my reading has been piecemeal. I really like the overview of the "classic" mathematical approach to voting in Taylor and Pacelli's Mathematics and Politics, but it's a math book, so it spends very little time on questions of implementation, and it doesn't cover multiple-winner situations like the nomination round. It would give you an idea of how mathematicians have tended to evaluate voting systems, though. (It also has good coverage of other problems in the field like apportionment, fair division, and yes-no voting systems. The general rule of thumb is that there's almost always a list of desirable qualities and a proof that no system can have them all.)

@271: Ah, I long for those bygone days of yesteryear, where people who posted without reading the thread were here to beat the drum for their favorite voting system. Now we've gone from people with "voting system" Google alerts to people with "Sad Puppies" Google alerts. (For clarity, Jameson, I don't mean you, I mean the folks who showed up once to tell us why their chosen system was Superior In All Ways and then disappeared again.)

Kilo @283: If you have any programming knowledge at all, you can probably code something much better than what I'm doing, which for SDV is basically having Excel calculate the weights, then me manually entering which candidate gets eliminated, then having Excel recalculate the weights. I'm happy to share my ersatz data if you'd rather not generate your own ersatz data.

@265, 269, 276 RE Goals: I'm operating on the assumption - which is admittedly partly reverse engineered from the current final voting system - that the overall goal of the award is something like "recognize the SFF work in the category most acclaimed by WorldCon members." ("Most acclaimed" instead of "most popular" because a book that 60% put 1st will win over a book that 100% like, and I avoided "best" because, well, measurement difficulties.)

In that case, it seems like a reasonable goal for the nomination procedure would be, "Produce a set of works which includes the books likely to be most acclaimed by WorldCon members, but is not so large that WorldCon members cannot reasonably compare them." (Note that the set of books might also include books that are NOT likely to be most acclaimed by WorldCon members. Screening those out entirely would be really hard.)

My assumption (which could be wrong) is that we can approach closest to this goal with a procedure that best balances the following 3 ideas:
1) Select works which are popular (on more nomination forms).
2) Select a group of works that together represent as large a set of nominators as possible (to reduce the chance that the entire set of nominees is acclaimed by a minority of the voting group).
3) Don't select too large a group of works.

Of course, there's the word "best" again... I think we've seen proposals that would prioritize #1 and #2 by disregarding #3, and ones that would prioritize #2 and #3 by disregarding #1. (The current system prioritizes #1 and #3 by disregarding #2, which is how we got here.) SDV and RAV variants are basically attempts to balance #1 and #2.

Does this make sense? If so, what do people think the relative importance of these ideas is? Or are they barking up the wrong tree in achieving the goal of the Hugo nomination system? Or do I have THAT goal, or the one of the Hugos entirely, wrong?

Whew.

@254, 270: OK, you've twisted my arm, I'll go simulate SDV-PE now. :)

#285 ::: P J Evans ::: (view all by) ::: April 15, 2015, 10:39 PM:

286
Repeating your crap argument does not make it stronger.
You start from a false premise (that there's a conspiracy), and you won't get any answer that will be good. (Ever hear of GIGO? This is GIGO.)

#286 ::: felice ::: (view all by) ::: April 15, 2015, 10:57 PM:

Cheradenine @284: "This is not a super scientific answer, but based on the number of "Top 5"/"Top 10" Tumblr memes I see where people say "In no particular order, because I couldn't possibly rank them," I think it's not that numbering items is by itself more work, but that assigning meaningful rankings to a set of 5 items involves a higher cognitive load."

True, but "if you don't have strong preferences, just write them down in the order you think of them" is I think a pretty easy and reasonable way to avoid the cognitive load in this scenario. I'd expect people would feel less pressure to rank them accurately with a write-in ballot than assigning numbers to an existing list, because in the former case the ordering is secondary to the main goal of specifying which works you're nominating, while in the latter the ordering is the sole focus. Alternatively, put a "ranked in order of preference?" checkbox on the form, and assign random orders when the checkbox isn't ticked.

"Having thought more about it, a single slate is probably best off not even to release the slate before the revelation of nomination counts. The more slate works on the ballot, the more likely other people's votes will clump on the non-slate might actually benefit most from the list of 15 being a bunch of varied, highly appealing works to split anti-slate votes."

Yep; though hopefully they wouldn't benefit enough, since even if nominations are pretty evenly divided between 15 works, the total number of nominations for the top 5 should still be much higher than under the current system.

"If there are 3+ slates and they vote early, then there's a danger that all 15 works will be from slates."

That would require three times as many bad actors as we have now, which seems unlikely. And of course there's the question of how the top 15 are counted - if our preferred proportional option (whichever that ends up being) is used, then that makes it much harder for slates to dominate this stage.

"With this suggestion, the early nominations sort of serve as a nonbinding poll of what people are planning to nominate, which could serve as a crowdsourced recommended reading list. I don't know how many people would realistically have time to read a bunch of works during the 2nd half of the nomination period, but they could, or could just let their ballots stand.

In some cases, people will have read works in the top 15 but not originally nominated them; if they agree that those works are good enough to be on the final ballot, and their original choices aren't in the running, it would be reasonable for them to change their nominations accordingly. Other people will not have nominated at all previously, but see something they like on the top 15, and nominate it. Others might read one or two of the most appealing works from the top 15 and nominate them if they like them. The nomination stage is about "this work is good enough to be in the running for an award", not "this work was better than anything else published last year", because nobody can possibly know if the latter is the case. Reading the entire top 15 isn't necessary.

"I do think this approach avoids many of the flaws of either open continuous voting or releasing nominations at the halfway point but not allowing changes. Ultimately how well it would work comes down to predicting how voters would respond, which is a guess. I'm not confident that by itself it would be a sufficient anti-slate tactic, but if it were the only reform we could get through, I'd at least be interested to see what happened."

One key point - as far as I can see, it's not against the current rules, so it could in theory be done next year. Would it be possible for the Business Meeting to vote to make Option 5b an explicit requirement for future awards, and recommend that MidAmeriCon II voluntarily carry it out before the change is ratified? As well as hopefully protecting next year's awards from Puppies, this would give the system a trial run before ratification, allowing for a more informed vote at next year's Business Meeting.

#287 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 10:59 PM:

On the two data sets I ran with SDV earlier, I got the same nominees with SDV-PE. However, in the situation where all the people who voted for #6 voted only for #6, it was a lot closer. In the situation with 150 slate ballots, the eliminated slate candidates stuck around just a bit longer.

Overall, I think SDV-PE reduces the incentive to vote for only one candidate that's present in SDV. This helped the slate a little bit since they were all voting for 5 candidates in this model, but the effect wasn't pronounced. I'd tentatively say that makes SDV-PE a bit more strategy-proof than SDV. I'd want to run a lot more sims, but first I need to make my spreadsheet more sophisticated so I'm not repeatedly searching a column of numbers for the two smallest, then looking to another column to find which corresponding number is smaller...

#288 ::: David Dyer-Bennet ::: (view all by) ::: April 15, 2015, 10:59 PM:

That's not an argument; it's simply contradiction.

#289 ::: P J Evans ::: (view all by) ::: April 15, 2015, 11:01 PM:

289
It should be possible to run both methods in parallel, to get good comparisons, assuming software is available for it.

#290 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 11:03 PM:

Cheradenine@287:

If you have any programming knowledge at all, you can probably code something much better than what I'm doing, which for SDV is basically having Excel calculate the weights, then me manually entering which candidate gets eliminated, then having Excel recalculate the weights. I'm happy to share my ersatz data if you'd rather not generate your own ersatz data.

That would be outstanding, particularly the dataset. Having the Excel sheet would be helpful too, if only so that I can check my implementation. It's exam week for my students, so I'll be grading all weekend, but this would make an interesting break as I have time.

I've been reading "Voting Systems" by Paul E. Johnson, and found it to be an interesting introduction, but without any referents, I've got no idea if it's a good source or not. I did recognize immediately that there were papers put out by some organizations that had an obvious axe to grind.


Regarding goals:

I like your first cut. I might add a #2a to be something like, "No minority should be able to absolutely prevent any other set of nominators from having their acclaimed works considered for nomination." Which is essentially what you're saying in #2, I realize, but there is a slightly different focus. I would contend that a nominating ballot that was not diverse would be okay if no one tried to propose anything different. Not likely to happen, I realize, but the possibility should exist. I'm not for forcing diversity if there is none, if that ever could possibly become the case. For me personally, my problem is not so much that the slates got their works on the ballot, it's that their method prevented other works from even having the opportunity to be on the ballot. Does that make sense?

Kilo

#291 ::: nathanbp ::: (view all by) ::: April 15, 2015, 11:03 PM:

My preference is still for RAV over SDV or SDV-PE because I think RAV does a better job of reducing the power of slates.

For example, consider this case:
A - 50 Votes
B - 39 Votes
C - 35 Votes
D - 30 Votes
E - 25 Votes
QRSTU - 160 Votes (the slate)

In SDV the winners are A with 50 votes, then QRST with 40 votes each. In RAV, the winners are Q with 160 votes, R (80), A (50), S (40), B (39). Effectively, SDV does the work of figuring out the optimum size for the slate for free instead of making the organizers of the slate guess ahead of time.

I don't think my example is that different from potential situations in the less popular categories and more spread out categories.

#292 ::: Martin Schafer ::: (view all by) ::: April 15, 2015, 11:32 PM:

In trying to come up with a mission statement for the nominating process, I think about a universe of reactions that the group of people who care about the Hugos might have.

"Those are the best works in the genre"

"Those are all worthy works"

"Some of the things I like best are on the list"

"Some of the things that I like are on the list"

"Nothing on the list embarrasses me"

Given the imperfect world we live in I think having as many people as possible react with 3 or at least 4 is the most achievable goal that will keep the Hugo's going on much as they have.

My personal preference is for the RAV 3c with exponential as I think it privileges those reactions.

Something of a side note: I get really uncomfortable when people use sub genre as a stand in for faction in the nomination selection process. I think the taste clusters are much more complicated than that. Nearly everyone reads more than one sub genre and will have only a few if any that they can't appreciate an exceptional work in. The things that make a story beyond the pale for me is a strong reaction of "people don't work like that" or "the world doesn't work like that."

#293 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 11:37 PM:

Kilo @293: Let me know where I should send them. I might take a day to make my Excel coding somewhat less embarrassing.

I skimmed Johnson's piece and it looks like a decent introduction to the original main thrust of voting theory. It covers more or less the same systems as are covered in a liberal arts math class, but covered in more depth (particularly the different Condorcet methods) and with a more rigorous mathematical approach. He doesn't touch on approval methods, strategic voting, or fairness criteria that aren't the four covered by Arrow's, and he doesn't have a lot of detail on STV. Occasionally his terminology seems a little nonstandard (I've never heard it called Arrow's Possibility Theorem instead of his Impossibility Theorem), but I don't think in a way that would cause you major trouble. I do think the Taylor & Pacelli book gives a clearer presentation, but it's also not available on the Internet.

This is all based on quickly flipping through it, so it's possible I missed a giant mistake, but it looked OK.

#294 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 15, 2015, 11:45 PM:

I hate to put the brakes on such a good conversation, but I've got some acts of corporeal charity to take care of before I crash for the night. I'll be shutting down comments in 5-10 minutes. See you all tomorrow --

#295 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 09:20 AM:

@294: Yes, you've hit on the key difference between RAV and SDV. However, I think the issue is more subtle than what you've said.

In voting theory, when you're comparing systems XYZ and PDQ, and superficially XYZ seems more hoopy, it's often the case that those froods who are voting will strategically overreact, making PDQ come out more hoopy in the end. For example, look at Condorcet and IRV. If voters from a one-dimensional continuum vote honestly, IRV is subject to center squeeze, prematurely eliminating a centrist and electing a more extreme candidate; while Condorcet would elect the centrist. But with strategic voting, IRV voters may learn to strategically vote for the "lesser evil" centrist to prevent this from happening, leading to a system that is actually more biased towards centrist parties than Condorcet.

I think that could happen in this case, too. Let's take your example, slightly modified for extra realism:

45 Votes: AXX
9 Votes: ABX
36 Votes BXX
35 Votes: C (no X because C is a bit of an upstart)
30 Votes: DXX
25 Votes: EX
106 Votes: XXX
7 Votes: QXX
5 Votes: RXX
3 Votes: SXX
QRSTU - 160 Votes (the slate)

The changes I made make no difference to either of the outcomes you worked out. I just added a bit of overlap, and added 105-120 ineffective voters.

My point is that SDV and SDV-PE are designed not to incentivize much "don't vote for popular candidates" strategy. So, of the 320 non-slate voters in this scenario, if just 10% made one more approval and 5% made 2 more approvals, that would be 64 extra approvals. If 1/8 of those are for B, then B gets 8 more approvals — easily enough to win in both SDV and SDV-PE.

So I think that in practice, the outcome of RAV, SDV, and SDV-PE would probably be the same in scenarios like this. But that's not to say that they're identical. With SDV and SDV-PE, the outcome would be more variable dependent on how voters acted. If they reacted to the presence of a slate by approving more works, they could easily keep the slate to just 3 slots, as with RAV, and possibly even keep it to just 2. (In my estimation, that would take about 110-160 extra approvals; not a trivial amount, but certainly achievable).

So the questions for comparing the systems are:

1. Do we want a more off-the-shelf method (RAV), or something custom-build (SDV or SDV-PE)?
2. Which is more important: the risk that SDV or SDV-PE might not be able to guard against a slate "rounding up" its slots, or the opportunity that these systems could help people could come together to ensure a slate "rounded down"?
3. Which is easier to explain? (Probably argues for SDV, SDV-PE, and RAV in that order)
4. Is it better for a system to be more flexible based on what the voters do (SDV-PE), or should it try to be more foolproof and "protect them from themselves" (RAV)?

#296 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 09:47 AM:

Morning all!

Jameson@298:

Good questions! In my opinion:

1. I don't think (obviously) custom-build is a problem, so long as we have tested it to the greatest extent we can (simulations/proofs, etc.).

2. I'm not 100% sure I'm parsing "rounding up" vs "rounding down", but I think this is related to the goal 2a I proposed to Cheradenine last night: It's important that a system not allow a slate to prevent works from being considered. It is not important at all (again, in my opinion) that a slate's suggestions be kept off the ballot. Does that relate to this one?

3. Probably the single most important criteria, in my opinion. If it doesn't get buy-in from fandom and (perhaps more importantly) the business meeting, then it's all for nothing anyway.

4. I'm never in favor of protecting someone from themselves. If it's the "fans' award", then whatever the fans want is by definition the proper course.

Kilo

#297 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 09:52 AM:

Cheradenine@296:

I'll see if the work you recommend is in the library on campus; thanks for the recommendation.

It occurs to me that posting an email address here is probably a singularly bad idea. I'm not sure how to get that to you. Teresa/Abi: I presume you have both mine and Cheradenine's email addresses; can you possibly send my address to her directly?

Many thanks,
Kilo

#298 ::: Carrie S. ::: (view all by) ::: April 16, 2015, 10:31 AM:

conrad6 @#271: Not that you'll read it (or care if you did), but consider: the whole point of this discussion is to nix the power of slates. So your hypothetical "SMOF Slate", insofar as it would be represented here, is going to a great deal of time and effort to kill its own effect.

Now I suppose that makes sense in the universe in which They plan to simply start ignoring the voting altogether and just giving the Hugo to whoever They like best (surely the simplest way to get what They want, since They have such a lock on the process!), but it seems like a lot of energy to waste on mаскировка.

#299 ::: J Thomas ::: (view all by) ::: April 16, 2015, 10:50 AM:

Here is a first draft of a somewhat-formal description of what the Hugo nomination might be for. Anyone who wants to crib from it toward a document to be sent to the business meeting or whatever is welcome to.


The traditional Hugo nominations are broken. To fix them we must think carefully about what we need them to do.

Here is a possible general statement:

The Hugo nominations are a survey, to learn from fans what they think are the great works of the year that should be contenders for the Hugo award.

They should produce a final ballot which will be used to vote for a single winner. This ballot is a short list of excellent SF or SF-related items or people, chosen by a large number of Worldcon members. The nominations are a survey, not an election. They do not need to be (and cannot be) “fair” to a particular work.


Detailed goals:

1. All Hugo voters should have an equal opportunity to suggest great works.

2. Hugo voters should each be able to suggest multiple great works, up to some maximum.

3. Other things equal, works suggested by more fans should be chosen over works suggested by fewer fans.

4. Other things equal, two works suggested by different fans should be chosen over two works suggested by the same fans. Other things equal, between two candidate final ballots, the one that includes suggestions from more fans should be chosen.

5. To the extent possible, the nominating system should not encourage “strategic” suggestions. The preference is that a fan should think “What are the most excellent works I have experienced” and not “How can I game the system to make sure the ones I most want will win”.

6. If a nominee declines at the last minute, it should not cause undue disruption of the process.


Discussion of detailed goals:

2. We get various complications when single fans suggest multiple works, which would go away if they could suggest only one. But we learn more about the great works of the year when each fan can suggest more of them.

3. We cannot hope to be “fair” to works, and discussion of what would be fair to fans is complex and nuanced. But it makes sense that a work which is suggested by more fans is preferable to one which is suggested by fewer fans, other things equal.

4. Similarly, other things equal it is preferable to have two works that together are suggested by more fans, than two works that together get the same number of suggestions, but the suggestions come from fewer fans. Similarly for groups of three works, and so on.

3 & 4. Goals 3 and 4 can conflict with each other. We might sometimes prefer two works that get fewer total suggestions, because they have more total fans suggesting them. Somehow we must choose how much weight to give these different goals when they conflict. It might occasionally happen that the work that is suggested by the largest number of fans is not chosen. This is because all the chosen works put together got suggested by more fans, compared to the total fans for any combination of works that includes the one with the most. Etc.

5. The goals of the nomination system are best met if each fan nominates the most excellent works he knows. But fans may have other goals. A fan may want one particular work to be chosen, even though he thinks others are also excellent. If he wants one to win, why would he suggest any others? Or he may feel that a particular excellent work is so popular that it will be suggested by sufficient people that he can suggest something else in its place. Or many other possibilities. We must accept that fans may have goals that oppose our goal of getting an accurate survey. But we should attempt to design a voting system which encourages people to suggest the works they think are great. It is always possible to game the system, but one criteria for a good system is that it should not make it obvious in detail how to game the system.


This document is provided without copyright. Anyone can use or modify it however they want, provided they do not restrict the right of other people to use or modify the original.

#300 ::: J Thomas ::: (view all by) ::: April 16, 2015, 11:04 AM:

@301 Carrie S.

It would be possible to hold an attitude similar to Conrad6 and be logically consistent.

First, given the chaotic nominating system, small groups could maneuver to get their choices onto the final ballot, crowding out everything else. The groups could tend to win and still be small enough to be deniable.

By creating an undeniable threat, he helps to get us to fix the problem he sees. If there was no problem before, still it is a potential problem that is worth fixing. Presumably the small secret groups would be against any fix but they can't very well say so. We, trying to fix it, become his allies or his dupes.

It can fit together. The previous conspiracies are still deniable but not disproven. Ideas similar to his can fit together.

#301 ::: J Thomas ::: (view all by) ::: April 16, 2015, 11:06 AM:

Brad from Sunnyvale had a complaint:

For example, I have often, in a Hugo nominating category, only listed two, three or even just one work. And I know I am not alone. I am "wasting" my "power" and entirely not caring. It's just that I don't follow that category super closely, but I know the people I am nominating were good enough by my standard that year.

Many of the above systems will give this "only care modestly" ballot greater weight, as though all ballots were written by people who have 20 things they want to nominate and have to carefully pick the best 5. Now the hidden message will be "nominate fewer, your choices will do better."

Applying that to this:

While there are more than 5 candidates:
1. Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
2. Find the two candidates with the smallest pile of fractional votes.
3. Eliminate whichever of those two is on the fewest ballots.

If a single vote counts as m=1, then it counts as m votes in the early stages and other votes only catch up after all the other votes on their ballots have been eliminated. That might be a powerful advantage.

If instead, a single vote counts 1/m and never increases because there are no other votes on the ballot to be eliminated, that might be a powerful disadvantage. But someone who wants to game that can list his single choice and then include four garbage works that he thinks no one else would consider nominating. They will be eliminated quickly, giving his real vote weight 1.

Is this an issue? If so, is there any way to fix it?

I tend to think it isn't. The only way for a candidate to lose is to be eliminated for having one of the fewest adjusted votes and the fewest total voters. If it has a chance at being in the top five, most of its voters' votes will have been eliminated before its own turn comes. Giving it a full vote early probably won't matter -- if it's in danger of being eliminated early, it probably doesn't have a chance later.

#302 ::: Lydy Nickerson ::: (view all by) ::: April 16, 2015, 11:24 AM:

On the topic of being easy to explain and fair: The truth is that the muse of math and I don't get on. Simple arithmetic and I are on fighting terms. 4s and 7s are confusing, and the numeral 8 and I haven't been on speaking terms in years. I genuinely can't evaluate the mathematics of the various voting systems. What I can evaluate are the people who recommend a change. I do this partly by reputation, partly by reading the posts looking for obvious self-serving bullshit, and partly by noting the response of people I know and trust to people I don't know and trust. If Bruce Schneier and Kevin Standlee assure me that the new nomination system is fair, and will yield a slate which is diverse but reflects the informed interest of Worlcon members, I'm likely to be in favor of it, although I am incapable of understanding it.

I am also looking for a system where I can follow the instructions. As per above, I don't and can't really evaluate the exact reasons why one does a certain thing. But I do need to know if it is important that I rank below No Award, or some other arcane thing. The instructions need to be reasonably understandable. I'm capable of following instructions that are somewhat complex, but I do need sufficient information to be able to figure out how to deal with my own complicated preferences in order to get a result close to what I want to achieve.

My best guess is that I'm not alone. It is important that the math make sense to anyone who can follow it. But reputation is also going to be important, here. This is probably true for the SP as well. They are likely to refuse to accept any explanation from anyone they don't trust. Do they have any good statisticians on their side? Of course, if all they really want to do is break the Hugos, then no amount of analysis will matter.

#303 ::: J Thomas ::: (view all by) ::: April 16, 2015, 12:23 PM:

@305 Lydy Nickerson

I am also looking for a system where I can follow the instructions. .... The instructions need to be reasonably understandable.

OK! If you aren't interested in how to game the system or how to stop somebody else from gaming the system, then it's simple. Here's the most important part.

Nominate the five you think are best. If you don't think there are five that are good enough, then nominate four or as many as you choose. Your vote will help the ones you nominate.

That's it. That's all.

Now here are some complications you can think about if you want to.

We have two different ways we can arrange counting. With one of them, you just list all the choices you think are good enough. The other way, you list them in order. The one you most want to win first, then the second, then the third, etc.

If it's the first way, then here's something you might choose to think about. Say that you think Fires of Europa is the best, and you think Feldspar's Luck is good enough. Imagine that you nominate both of them, and then it turns out that Feldspar wins and Fires does not. If that happens and you're ready to get all upset about it, maybe it's better that you not nominate Feldspar. Because it might happen. You can put as much calculation as you like into this. If you're pretty sure that Feldspar isn't that popular, not like it can win when Fires doesn't, then you might think you can afford to nominate it anyway. Or if you wouldn't be *that* upset. If you could just yell a little "How could they *do* that, what were they thinking?" and get over it, you might figure it's still worth doing.

If your attitude is that you'll just nominate the ones you think are good enough and let whatever happens happen, then you don't need to think about this at all.

If they say to list them in order, then do that. If you can't decide between two which you want more, then it doesn't matter which comes first.

Finally, No Award. This is for voting. You can vote to not have any award made.

The special issue since the votes count from best to least, is that things you don't list you don't vote for, but things you do list below No Award may get voted on if No Award loses. Basicly, if No Award is not that popular but they are, then voting them below No Award means your votes for them might count. You're saying that if No Award won't win, you'd like them to win.

If that's too confusing then just don't list anything below No Award. Then you're saying that if the things you want don't win, and No Award doesn't win, then you don't really care which of the others wins. That isn't a terrible stand to take.

I can't think of anything else you need.

If you want to play games to win more than your share, or stop other people from playing those games, then it's endlessly complicated. But if you just want to do the right thing yourself then the complications dissolve away.

#304 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 12:53 PM:

OK, read all the comments so far, now commenting.

Bruce @6: I want nominating to be as easy as possible -- "It's okay that you haven't read everything, just send us names of works you think are worthy"

Gah! I want nominating to be easy as well, but I think suggesting that people nominate things they haven't read is more pro-slate than anti-slate. (And, for the record, I'm a Sad Puppy saying that. I can't speak for anyone else involved with SP3, but I didn't nominate anything I hadn't read.)

I've seen a lot of discussion of various option #3s, but very little discussion of frobbing option #2. I think playing with option #2 offers one of the best chances to reduce and/or eliminate the effect of slates by simply eliminating the 5 candidate limit for the final vote, and putting everyone who gets 5% of the vote on the final ballot. If the Gonosypherpalitic Juvenile Canid Slate votes in absolute lockstep, all 300 of them, then yes, their 5 candidates will be on the final ballot. But since they only comprise 14% of the total population (historically speaking, 2100-ish nominators), there could be as many as 17 other candidates.

It makes the pool fairly large, but doesn't leave it quite as wide open as the "no nomination system at all" or "everything that gets nominated at all is on the final ballot" proposals back around the #230's.

I suppose it's still somewhat subject to things which came close getting swept off if there are lots and lots of competing slates, but not nearly the possibility of the effect of this year's nomination round.

#305 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 12:55 PM:

@302: That looks good. A few notes:

2: I'd suggest removing "up to some maximum". Yes, I understand that Worldcon members may be reluctant to remove the maximum, and that's not the end of the world. But really, most if not all of the systems under discussion work better without a maximum. In any case, a detail like that certainly shouldn't be in a general statement of principles like this one.

Add:

7: The voting system should make as much sense as possible to as many voters as possible. Complicated mathematics should be kept to the minimum necessary, and if possible that necessity should have a good intuitive explanation.

and then in "Discussion":

2. Well said.

5. "It is always possible to game the system" is a little too pessimistic. Rather: "No system is impossible to game." That is, there may be some limited circumstances when it is indeed impossible to game a given system.

.....


So, let's see how the systems under consideration measure up on those goals. Here's my subjective rating each of RAV, SDV, and SDV-PE from 0(worst) to 10(best) on each of those criteria.

1 (equal opportunity) All three get a 10.
2 (multiple works) RAV 9, SDV 10, SDV-PE 10
3 (more over fewer) RAV 9, SDV 9.5, SDV-PE 10
4 (diversity) RAV 10, SDV 9.5, SDV-PE 9
5 (strategy resistance) RAV 8, SDV 9, SDV-PE 10
6 (declined nominations) Depends on the rules, but assuming you build a list of the top 10 and then when somebody declines, just pull from the list: RAV 8, SDV 10, SDV-PE 9 or 10 (???)
7. (understandable and seems fair) RAV 3, SDV 6, SDV-PE 5


#306 ::: P J Evans ::: (view all by) ::: April 16, 2015, 12:58 PM:

307
Generally, people aren't going to be nominating anything they haven't read or seen. How else can you decide if it's worth a rocket?

If you think only one item is that good, that's all you need to put down. (It isn't really a contest for how many you read or saw.)

#307 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 01:04 PM:

Actually, I think I just realised that I might be confused about how the 5% rule works.

Presuming everyone who sends in a nomination ballot picks five nominations for Best novel, and there are 2000 total ballots, is that "5% of 2000" or "5% of 10,000"?

If it's the former, my above proposal might end up with 100 nominees on the final ballot. Which is, um, a lot.

If it's the latter, if the Canids are a block of 200 out of 2000, and no-one else votes for any of the works on their slate, I don't think any of their slate picks make the final ballot, because each of their picks only have 2% of the final vote.

#308 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 01:09 PM:

P J @309: Sure. I only put 4 things on my ballot this year for Best Novel for that very reason. I just got a frisson of twitch when I read that comment by Bruce, so... *shrug*

#309 ::: P J Evans ::: (view all by) ::: April 16, 2015, 01:11 PM:

310
It's by category. So 5% of 2000.
This rule matters a lot more in the categories with fewer nominees - not usually in novel and dramatic presentation.
At the Hugo Award site, some years have the numbers posted, so you can see the cutoff points.

#310 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 01:12 PM:

@299: "I'm not 100% sure I'm parsing "rounding up" vs "rounding down"....

In order to explain, let's use some numbers. Say that in a certain category, 15% of votes are ineffective long tail "undervotes" — that is, only voting for things that almost nobody else mentioned; and 10% are ineffective "overvotes" — that is, only voting for things that could have won anyway even without their votes, or things that have no chance, but nothing in the middle. So that leaves 75% effective votes; 1/5 of that, or 15%, should be about right to get one nomination slot.

Now imagine that a slate has 33% of the voters. So it "deserves" about 2.2 slots. Of course, it can't get 2.2, so there's a choice between 2 or 3. I think most people here would agree we'd rather "round down" and give it 2 slots, than "round up" and give it 3.

RAV would tend to "round down" in this case until it had about 40%, so that's a good system. SDV and SDV-PE would tend to "round up". However! SDV and SDV-PE might tend to encourage "undervotes" less than RAV would, so it's I think it's probable that if you account for that, they would get the same answer as RAV, and also get it "for better reasons" (in a way where it's easier to demonstrate that it satisfies more of the voters).

And those numbers were my guesses of what might happen in a category like "Best Novel". In something more diffuse, like "Short Story", under- and overvotes could make up almost 50% of the votes. In that case, RAV would probably give the slate 3 slots, while something like SDV-PE could conceivably give it 2 ("rounding down), 3, or 4 ("rounding up") slots.

So is the chance of keeping the slate to 2 worth the risk of it getting 4? That's the basic question here. And it hinges on how people would respond to the strategic incentives particular to a given system — something we can't possibly know for sure ahead of time, even if we had all the old election data.

I don't think this is the most important difference between RAV and the two SDV options. But it is a difference.

#311 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 01:15 PM:

Note that numbers are probably going to jump next year anyway, so the idea that slates will be motivated enough to grab 4 slots even in a proportional system may be far-fetched.

#312 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 01:36 PM:

@313: Thanks, that makes sense. My personal preference would be for SDV-PE, then, specifically because there is more uncertainty in how the nomination would go. It seems to me that uncertainty of this type (in general) makes the system somewhat less easy to game, since -- as you point out -- neither we, nor a potential slate, can possibly know what voter behavior is going to be ahead of time.

Kilo

#313 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 01:49 PM:

#312: No, I mean, a ballot has five slots on it for Best Novel. If all 2000 ballots have all five slots filled, that's 10,000 "votes" cast for nominations for Best Novel, even if they were only cast by 2000 people.

#314 ::: P J Evans ::: (view all by) ::: April 16, 2015, 01:54 PM:

That's why I mentioned that for some years, at the Hugo site, you can see the actual numbers with the cutoff points.

#315 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 02:07 PM:

#317: Ah. Yes, good call. I'll go do some math.

#316 ::: Mary Aileen ::: (view all by) ::: April 16, 2015, 02:29 PM:

perlhaqr (307): Bruce @6: I want nominating to be as easy as possible -- "It's okay that you haven't read everything, just send us names of works you think are worthy"

Gah! I want nominating to be easy as well, but I think suggesting that people nominate things they haven't read

I'm pretty sure Bruce was saying "nominate worthy things from the subset that you *have* read", not "it's okay to nominate things you haven't read".

#317 ::: J Thomas ::: (view all by) ::: April 16, 2015, 02:43 PM:

@308 Jameson Quinn

Thank you for the comments. I have incorporated them in my copy, I'll wait in case there are more rather than repost it.

While I was looking at ways to say that, I found criteria to judge the result of a nomination system. Not a way to find out how to pick the winners, but a way to rate how well it's been done.

How to score a nominating result:

1. Count each ballot that includes at least one of the five winners.

2. Count each ballot that includes more than one winner.

3. Add #1 plus a factor n times #2, divided by the number of ballots.

4. Take the number of votes for the fifth nomination divided by the number of votes for the fourth nomination. Multiply this fraction times the fraction from #3.

As a first guess I would set n = 0.1.

The first part is a sort of "dissatisfaction" score. The more ballots that have no winners, the more dissatisfaction.

The second is an "antidiversity" score. It's better to have one ballot get more winning votes than to not have more winning votes, but it's much better to have a ballot that gets its first winning vote.

The third is an "antievenness" score. It's better not to have any of the winners with too little support.

It should be clear where this is heading. If 20% of the voters are a slate that wins everything, that gives a score of .22

If 5 groups of voters, each 20% of the total, get one winner each with no overlap, that's a score of 1.0.

If 5 groups of voters get one winner each and two of them overlap, so that the vote comes out 20%, 20%, 20%, 30%, 30%, the score is 1.02. Times 2/3.

If 5 groups of voters get one winner each and two of them overlap a different way, so that the vote comeds out 20%, 20%, 20%, 20%, 20% and 10% of ballots have no winner, the score is 0.92. Times half.

This doesn't match up perfectly with "average satisfaction" versus "representativeness". But that's OK. A system that caters too much to slates can't do well because it doesn't spread the votes out well enough, it doesn't promote diversity.

But diversity isn't the only goal, we also care about getting more votes even when they aren't diverse.

Fraction of ballots with at least one win will not correlate strongly with fraction of ballots with more than one win.

I like this way to decide how good a result is, but I am not certain what value the n parameter should have.

Ideally knowing the desired outcome would lead to a voting procedure that would produce that result, but I don't see it this time.

#318 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 02:51 PM:

J Thomas @302: This draft looks good to me. A few thoughts:

I think I would reword the statement "This is a survey, not an election," to clarify what distinction it's intended to make, since different people will have different associations with each word. To me, technically, the process of gathering and analyzing the data is a survey; once we're using a method to determine a set of choices based on that data, we're having an election (or, if we want to be super formal, "applying a social choice function"). I wonder if something like "This is intended to be a representative process, not a political one" captures what you meant? But you're a better judge of that than I.

I might trim down #4 by deleting the second sentence (which I think repeats the thrust of #3, unless I'm missing a distinction?) and reword #5 to something like, "The voting system should minimize the incentive for a voter to think strategically, instead of casting an honest vote for the works they consider excellent."

#319 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 03:14 PM:

J Thomas #320: "4. Take the number of votes for the fifth nomination divided by the number of votes for the fourth nomination. Multiply this fraction times the fraction from #3."

I think this rule introduces some odd behavior into the metric. Say we have the following distribution of ballots (assuming bullet voting for simplicity):

A 20%, B 20%, C 20%, D 17%, E 12%, F 11%

The score for ABCDE is .89 + 0 (no one has more than one winner) multiplied by (12/17) = 0.628.

The score for ABCEF is .83 + 0 multiplied by (11/12) = 0.7608.

So according to the metric, ABCEF is a better set of candidates than ABCDE, which doesn't seem right.

If we disregard #4, this is quite similar to the metric used in PAV. In PAV, a voter has a satisfaction score determined by how many of their nominees made the ballot: 1 for 1 nominee, 1 + 1/2 for 2 nominees, 1 + 1/2 + 1/3 for 3 nominees, ... This proposal is equivalent to those scores being 1, 1 + n, 1 + n, 1 + n, ... The fact that getting more than 2 of a ballot's nominees doesn't increase satisfaction at all could lead to some weird behavior in edge cases (say, if 60% of the ballots all agree on 3 candidates); 1, 1 + n, 1 + n + m, ... where m is less than n.

The most straightforward way to design a system around a metric like this is the PAV approach: compute this score for all possible winning sets, and take the one with the highest score. As with PAV, though, this runs into computational difficulties.

We could also use it as you suggest to analyze different choices of voting systems, though we'd need to tune the metric first.

#320 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 03:45 PM:

If you're looking for a measure of evenness, one way is to use squares of fractions. For instance: divide each vote among the winners, a la SDV; then take the sum of the squares of the amount of votes for each winner.

However, as Cheradenine (and Clay, earlier) pointed out, this kind of measure will sometimes favor a slate of winners that seems objectively worse than another from point of view of simply maximizing happiness. For instance:

10: AB
10: AC

This measure prefers BC to AB or AC, even though all the voters like A.

#321 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 03:48 PM:

Aak... I got the math very wrong in my last comment. Please ignore.

#322 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 03:58 PM:

I think if you just turn everything I just said completely backwards, it is correct. Take the negative square of the number of people not voting for each candidate.

#323 ::: J Thomas ::: (view all by) ::: April 16, 2015, 04:24 PM:

@322 Cheradenine
"4. Take the number of votes for the fifth nomination divided by the number of votes for the fourth nomination. Multiply this fraction times the fraction from #3."

I think this rule introduces some odd behavior into the metric.

Oops! I guess so! I don't know how that slipped by, I meant the fifth divided by the first. The fractional difference of the biggest and the smallest.

Maybe it would be better to use the one with the least votes divided by the total number of ballots. The point is we want the smallest to be big, not that we give it a pass if the largest is small too.

If we have four nice big groups and then the fifth is a tiny scattering of votes for a Gor novel, the four extra-good groups don't make up for the bad one.

#324 ::: albatross ::: (view all by) ::: April 16, 2015, 04:41 PM:

Jameson Quinn:

I was thinking more about SDV-PE, and tried to hack something up in Python to make sure I understand it. I noticed a couple places where I wasn't sure what to do:

First: Am I right in thinking that as we eliminate candidates from the list of possible candidates, we also remove those candidates from all the ballots? That is, if I have a ballot

[Alice, Bob, Carol]

then in this round, Alice, Bob, and Carol each get 1/3 vote from my ballot. At the end of this round, I eliminate Carol. It seems like I should now change that ballot for the next round to

[Alice, Bob]

for the next round. Is that right?

Second: When we're deciding which candidates are up for possible elimination, we can have ties for lowest or second-lowest fractional vote total.

Suppose at the end of a round, I have these ballots:

3: ADE
9: E
1: AB
1: C
1: BC
1: BE
8: A

We thus get these fractional vote totals:

A 9.5 votes
B 1.5 votes
C 1.5 votes
D 1.0 votes
E 10.5 votes

Your description is that we want to choose the lowest two fractional vote totals in this round, and decide which one to eliminate by seeing which is on fewer ballots. But in this case, we'd have to make an arbitrary choice between Bob and Carol to keep it down to two potential candidates to eliminate.

The only sensible way I can see to handle this case is to put Bob, Carol, and Dave all three up for possible elimination.

Third: When considering candidates for elimination, we could have ties for lowest number of ballots. (Again, I think this will happen a lot early on, when there are lots of candidates with few votes.)

That is, suppose we followed my idea above and now are considering Bob, Carol, and Dave for elimination in this round. And suppose when we look at it we see

Bob: 3 ballots
Carol: 2 ballots
Dave: 3 ballots

Now, we have a tie between Bob and Dave. In this case, they have different fractional vote totals, but that's not always going to be true--they could be tied on everything.

The simplest way to deal with that is to just eliminate all the tied candidates. But that does leave the possibility that one round of the election could eliminate all the remaining candidates. That's just what will happen if we have really dispersed votes (like everyone votes for 3 candidates, and they're all different candidates).

It's always going to be possible to tie in this elimination step, and have more than one candidate that is equally worthy of elimination.

#325 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 04:59 PM:

Fifth divided by first actually leads to some even funkier behavior:

A 60%
B 10%
C 9%
D 8%
E 7%
F 6%

ABCDE gets a rating of .93 X (7/60) = 0.1085.
BCDEF gets a rating of .40 X (6/10) = 0.24.

Using fith place votes/total ballots would fix this pathology, but I think it may weight too heavily toward taking the five top vote-getters, even if some or all of them are on a slate. Consider:

20% vote ABCDE (the slate)
40% vote F
15% vote G

ABCDF would have a score of 0.64 X (.20) = .128.
ABCFG would have a maximum possible score (assuming no overlap between F and G voters) of 0.79 X (.15) = 0.1185.

So a 15% increase in the number of fans represented is overpowered by the fact that the non-slate work only has 3/4 as many votes as the slate works.

I don't think it's unreasonable to have a factor that reflects "we'd like the individual works to have larger numbers of votes," but I don't think it can be anywhere near this strong.

#326 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 05:10 PM:

albatross @327: I had a few ties in the vote count come up while simming SDV-PE (partly because I have a bunch of unnamed candidates with 50 votes) and broke them by going back to the candidate with the least fractional votes. It's certainly possible to have a tie in both of these; in that case, I would lean toward a protocol of

1. If eliminating all the tied candidates leaves 5 or more candidates, eliminate all the tied candidates.
2. If eliminating all the tied candidates leaves 4 or fewer candidates, expand the nomination pool to include all the tied candidates (which is the current practice).

This procedure isn't perfect; #1 in particular could lead to a different outcome than eliminating one tied candidate at random. The alternative would be to develop more tiebreakers.

In the other case, I think your solution of "compare all 3 candidates and eliminate the lowest" is definitely the best. So, if two or more candidates tie for the lowest fractional votes, consider them all and eliminate the one on the fewest ballots. If two or more candidates tie for second lowest, consider the lowest and all the tied candidates, and eliminate the one on the fewest ballots. (And if a tie exists in the ballot count, follow the procedure above.)

#327 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 05:17 PM:

@327

Eliminations (Alice etc.): Yes, you're right.

Ties for lowest two: consider all three for elimination, but eliminate just one.

Ties for elimination: (Your example is not tied for last, so it's easy, but I know what you mean.) Various possibilities. Flipping coins, using some other attribute of the ballot to decide. One possibility would be to eliminate both. This would make slates which didn't get votes from non-slate voters extra-risky, and actually probably wouldn't affect any other candidates (since if they don't have significant overlap, both will probably be eliminated soon anyway).

#328 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 05:20 PM:

Jameson @330: I think a nondeterministic method is preferable to flipping coins here. I don't want the Hugo administrators to have to get their coin flips notarized. :)

#329 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 05:21 PM:

A deterministic method. Apparently it's State Things Backward Day.

#330 ::: CHip ::: (view all by) ::: April 16, 2015, 05:43 PM:

J Thomas @ 269: I don't think the Hugos ever had a goal as such; the object was to perfect the process of picking a winner. (As noted previously, some people offer education as an excuse for the Retro Hugos; I disagree that this is even an effect, let alone a justification -- but the RH are a separate and self-limiting case.) Your attempt at a mission statement is interesting, but I wonder where it will fit in; the WSFS Business Meeting usually deals with much more focused/quantitative matters. Possibly you should put this up in a wider forum?

piling on to conrad6 @ 271: If you want to convince us of anything, you might try getting your most checkable fact correct. Shaver was a 1940s phenomenon.

PJ @ 312: This rule matters a lot more in the categories with fewer nominees. I think you've got that backwards; the 5% rule has historically taken effect in the shorter fiction categories, where there are many more nominees. Long DP will never have many nominees (because a long-form DP costs a lot to bring out) and is very unlikely to get trimmed by the 5% rule.

ddb @ 291: <grins>

#331 ::: P J Evans ::: (view all by) ::: April 16, 2015, 05:49 PM:

333
Getting it backwards day, indeed: that was what I meant. Me braining not good today.

#332 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 06:00 PM:

OK. So, looking at the math at thehugoawards.com for past nomination rounds, it looks like the percentage is calculated as #votes / #ballots.

Which means that my "take everything with 5% or more" leads to a potential situation where there are 100 items on the final ballot, which is probably untenable.

OK, so... what if that got changed? #votes for a work / #votes cast total? Of course, then you might not get anything that gets 5% of the vote, so you might want to change the 5% rule as well, to something more like "everything between the highest percentage of the vote * $X and the work which got the highest percentage of the vote", where $X is .5 or .75 or something determined by looking at actual vote totals, and figuring out what fraction there should be based on how many nominees one ends up with from various years, and with a floor value so that you never have fewer than $Y works on the final ballot.

It also has the side-effect of causing slates to either self-throttle or drown themselves. A slate could vote for one work in lockstep, and do fairly well for it, because their votes only increase the vote total by 200 as well. But a slate of five nominees, voted in lockstep, only increases the vote count for each novel by 200, but increases the total vote count by 1000, effectively reducing the voting power of the slate votes.

#333 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 06:08 PM:

CHip@333: I agree the goals of the Hugo have not been stated, historically. That's because there was never a need to. But now a situation has arisen where it's become clear that we actually need to define what we're talking about.

I'm an astronomer; the same situation famously came up with Pluto. We realized we didn't actually have a definition of a planet -- but we never really needed one before. Once Eris was discovered, we realized we needed to be precise in what we were talking about.

The definition that eventually passed is ridiculous, but I digress...

Kilo

#334 ::: Brad from Sunnyvale ::: (view all by) ::: April 16, 2015, 06:10 PM:

Several people have said to me "every system has strategy" -- this misses the point which is that some systems have much more strategic voting incentive in them than others. And I think the systems proposed above all have too much.

Even single nomination has more strategy than 5 nomination approval that we do now. And single nomination can still be dominated by slates.

I still don't see anything that comes even close to the plain approval 5 ballot that we do now when it comes to the influence of strategy. I know this because people very rarely talk about strategy for it, because there almost inn't one.

Only option 4 fights slates without increasing the complexity and strategy in the nomination process.

#335 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 06:18 PM:

Brad @337: Please stop making authoritative-sounding statements about strategy and gameability. You're basically talking about game theory and probability, which are well-studied fields, and human instincts for probabilities in complex systems are terrible.

#336 ::: J Thomas ::: (view all by) ::: April 16, 2015, 06:45 PM:

@323 Jameson Quinn

If you're looking for a measure of evenness, one way is to use squares of fractions. For instance: divide each vote among the winners, a la SDV; then take the sum of the squares of the amount of votes for each winner.

However, as Cheradenine (and Clay, earlier) pointed out, this kind of measure will sometimes favor a slate of winners that seems objectively worse than another from point of view of simply maximizing happiness.

That is inevitable.

If you start out with criterion A, and then you let some other criterion B influence your results, you will no longer maximize A. It will look worse to people who think you should just maximize A.

Of course, it's sometimes possible to fall between two stools. To get a compromise that's worse than either extreme.

#337 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 06:46 PM:

Brad@337: I agree that option 4 is simpler, and honestly to be preferred (since it alone -directly- addresses the problem). Personally, I'd love to see it happen. I just don't see any way to make it work, apart from giving a human committee the power to make the call, as you suggested on your blog. My problem with that is that this effectively proves what the puppies have been saying: "There's a committee which gets to approve which works can be nominated." You and I know this committee wouldn't exclude something because of content or who wrote it, but it would certainly look that way -- or at the very least, could be taken out of context to look that way.

Then of course there's the problem of how to unambiguously define and detect slates. I've got the same problem detecting cheating when I give an exam. The more similar two wrong answers are, the less likely it is to have occurred by chance, but can I -prove- it was cheating? It's surprisingly hard. I'm not saying it can't be done, but after reading your blog, I thought very hard about it and couldn't figure out a way beyond "trust the committee's judgement." I just don't think I could accept that, and I'm fairly certain the puppies and their ilk couldn't either.

If there were enough nomination slots, then the odds of having (say) ten identical nomination ballots just by chance is very low -- so it's probably a slate. Another example: What if a large group pushes three works out of a possible five, is that a slate? What if they divide into two groups and each pushes three different works? I'm not sure how the committee would be able to throw that out. I dunno, I'd be interested if there were way, though.

Kilo

#338 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 06:48 PM:

Hmm, number swapping mania... I guess Brad's message is now #334..

K

#339 ::: J Thomas ::: (view all by) ::: April 16, 2015, 06:52 PM:

@332 perlhaqr

But a slate of five nominees, voted in lockstep, only increases the vote count for each novel by 200, but increases the total vote count by 1000, effectively reducing the voting power of the slate votes.

I may not be following what you're saying here.

It looks to me like it increases the total vote count by 1000 for everybody else too, but increases their individual vote counts not at all.

#340 ::: J Thomas ::: (view all by) ::: April 16, 2015, 07:03 PM:

@325 ::: Cheradenine

I don't think it's unreasonable to have a factor that reflects "we'd like the individual works to have larger numbers of votes," but I don't think it can be anywhere near this strong.

Yes. Three different rules leave me wondering just how to bolt them together.

I think these are the parameters I want. The winners should have more votes. They should have votes from more fans. And the least of them should have votes from more fans too.

#341 ::: William Hay ::: (view all by) ::: April 16, 2015, 07:16 PM:

The works you nominate will greatly depend on which books you happen to have read. The books you have read will in turn depend on accidents of your personal history and acquaintances. Why not admit this and use chance to make the final decision as to nominees?

For each work nominated place one token into a tombola for each person who nominated it. Mix well. Draw tokens one at a time until you have five unique nominees for the final ballot.

Should be roughly proportionate over time, won't overweight slate candidates, will encourage diversity and discourage a tyranny of the majority by occasionally throwing up something that wouldn't have a chance in more deterministic systems.

#342 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 07:18 PM:

J Thomas @340: I think adding a small weight for "ballots with 3 works nominated," "4 works nominated," etc. may do enough to weight the metric toward avoiding works that appear on a small number of ballots. If we have separate elements in the rating representing "more votes" and "least with more votes," what it basically ends up doing is increasing the weight on "more votes." I think we'd be better served to increase that weight directly (I actually think .1 is probably too low for n).

#343 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 07:22 PM:

William Hay @341: Random ballot came up for discussion in the first thread, and I'm not sure how much is to be gained from reopening that discussion.

#344 ::: nathanbp ::: (view all by) ::: April 16, 2015, 07:27 PM:

Jameson Quinn@305
7. (understandable and seems fair) RAV 3, SDV 6, SDV-PE 5

I'm curious what you (or others) think is difficult to understand about RAV? Or are you knocking it down for seeming less fair than SDV? To me, RAV seems among the easiest to explain and understand of all the proposed systems.

#345 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 07:39 PM:

@344: Oh. I was really trying to guess what other people would think. Basing it on my own understanding is a bad idea, because I think about voting systems all day.

So, what do other people think? Which of the three is easiest to understand?

#346 ::: Brad from Sunnyvale ::: (view all by) ::: April 16, 2015, 08:18 PM:

#335 -- what is the statement I have made from authority, and what analysis shows you it is false?

Having a human based system of justice does not require exactness, though it is a virtue. The world has a lot of experience with such systems, some better than others. If humans are to exercise judgement, we know things we want -- transparency, accountability and a few others.

I am not entirely comfortable with the idea. I am led to it by finding too many flaws in the systems being proposed when viewed through the lens of what I believe is the goal of the Hugo system.

If you dislike the term "survey" I offer the term "measurement." Fans have opinions. There is such a thing as "The works which are the most supported, according to metric X, by the fans" It is something that can be measured -- if fans will express their true (and independent) opinions to us. Any factor, such as collusion, or voting strategy compromises how truly fans tell those views.

#347 ::: J Thomas ::: (view all by) ::: April 17, 2015, 10:15 AM:

I found a better way to say what I want for nominations, which may turn out to be similar to what a lot of fans would want.

What we want is for the nominations to provide a good set of results. We want to have five very good nominations. We want diversity, meaning a lot of fans have input. When one faction mostly gets its way, that's bad even when the total votes for their choices is more than the total votes for a more even set of nominations.

If at all possible, we want to wind up with five great choices. If the nominations result in one great choice and four mediocre ones, that is not so good. Four great choices and one mediocre one is not that good. We want five great choices. It's more important that the fifth choice be great than that the first choice be the best.

Here's a picture of the result I want:

You have six piles of ballots. Five of the piles are ballots for the five winners. The sixth pile is the pile of ballots that didn't win anything.

We want that sixth pile to be small. And we want the smallest of the five winning piles to be large.

If a slate is big enough to get a pile, then it gets one pile.

Five piles of winners and we want the smallest of those piles to be pretty big.

Imagine that there's a different way to arrange the piles where one of the piles is giant and the others are small, and the nominee in the giant pile is not in any of the winning five piles the first way. There is a big winner that we did not choose the first time. Would that be a better way to pile them up? Not necessarily.

Imagine this situation: One book got marketed very well and every fan in the world has read it. 20% of them liked it. A second book got read by only 10% of fans and they all liked it. Then in the nominations say the first gets 200 votes and the second gets 100. If we nominate the first one, everybody has already read it and four fifths of them didn't like it. If we nominate the second one, a bunch of people will read it and like it. I say in that particular situation the second book is the better one to nominate. Because in the nominations we can't expect everybody to have read everything. It's more important to have five nominations where each of them has a lot of fans that like it, than to have the one that the most people like.

I don't know exactly how much emphasis to put on making the pile of losers small, or how much to put on making the fifth-largest pile of winners large. That's two different goals that might conflict. And after we decide how to weight those, I don't know how to divide things up to get the best result.

But that's the result I want.

Oh, one more thing. If there are two different ways to pile up the ballots that get the same score, then I'd want to count all the second winning votes on winning ballots, and break the tie that way.

#348 ::: J Thomas ::: (view all by) ::: April 17, 2015, 10:18 AM:

@347

Why is this a good way to do it?

When I make suggestions to people, I don't say “Here's the suggestion I most want you to do, and here's the suggestion I second-most want you to do.” I just make suggestions. My suggestions are not a ranked vote. They are not even an approval vote. The nominations are not votes for best SF whatever of the year. They are suggestions for what to put on the short list.

The point of the nominations is to offer five great things that are all worthy. We want each of them to be suggested by a lot of fans, because when there are too few fans suggesting it then it might not be that great.

We want to accept suggestions by varied fans, we don't want all the choices to come from the same ones. Fans are likely to read everything on the short list, and we want that list to represent the diversity of fandom, for fans to sample.

#349 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 10:30 AM:

@346: I believe Cheradenine is using "authoritative-sounding" to mean "overconfident", not "based on argument from authority". So we're talking about @334 (which used to be @337). In particular:

Several people have said to me "every system has strategy" -- this misses the point which is that some systems have much more strategic voting incentive in them than others.

Correct, though I don't see anyone denying the point you say is missed.

And I think the systems proposed above all have too much.

This is your opinion, and you have a right to it.

Even single nomination has more strategy than 5 nomination approval that we do now. And single nomination can still be dominated by slates.

These are "authoritative-sounding" statements, and they are unsupported and in conflict with both the common voting-theoretic and common-sense definitions of the terms you're using. There may be some definitions under which they are correct, but you should at a very least be more careful.

I still don't see anything that comes even close to the plain approval 5 ballot that we do now when it comes to the influence of strategy. I know this because people very rarely talk about strategy for it, because there almost inn't one.

Are you kidding? Slates are strategy. That's the only reason we're having this discussion in the first place. Do you think we would be here if each puppy had independently nominated their 5 favorite works in each category?

Only option 4 fights slates without increasing the complexity and strategy in the nomination process.

Option 4 is not without complexity. And as regards strategy, you are either wrong, or at least have failed to explain under what definitions you could be considered right.

Strategy is anything which considers how other people are likely to vote. Say my preferences are A-Z in that order. If I always vote for A-E, or always vote for half of the candidates (A-M), or always vote for candidates I consider above average, then that is not strategic. But if I vote ABCD because E is too popular, or worse, EAIHD because I want to make sure TONSR lose, then that is strategic. Strategy can be honest or dishonest; it can favor or disfavor candidates perceived to be popular; and it can lead to a high-strategy equilibrium or a low-strategy equilibrium. The current system strongly favors dishonest strategy which helps popular candidates and leads to a high-strategy equilibrium (a two-party system). The proposed changes weakly (in some cases, very weakly) favor honest strategy which helps relatively-unknown candidates and leads to a low-strategy equilibrium. I really don't see how that could be considered a step down in terms of strategy, though I'm willing to hear any arguments you have.

#350 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 10:39 AM:

@347: This very much aligns with the SDV-PE process. SDV-PE is basically a bunch of steps where in each step you:

Divide everything into piles
Take the smallest two candidate piles
Eliminate the one of them which leads to the "didn't get a nominee" pile growing the least.¹

This is not guaranteed to be optimal by your definition, but it is pretty much exactly a greedy algorithm for approaching that, and greedy algorithms tend to do pretty well even when they're not optimal.


¹This is only approximately what the elimination does. You could define the elimination so it more strictly led to this outcome, but I don't think that would actually improve this system, for at least 3 reasons. Basically, it would be harder to understand; would lead to winners with less-broad support; and would encourage "avoid popular candidate" strategy more. I could state all of those with more mathematical precision if needed.

#351 ::: J Thomas ::: (view all by) ::: April 17, 2015, 10:54 AM:

@346 Brad from Sunnyvale

Having a human based system of justice does not require exactness, though it is a virtue.

Depending on the judgement of a large committee might not work well. For example, it might tend not to do anything until a lot of people get worked up, and then tend to do something drastic and counter-productive. But it might do some good and it might be that it will happen regardless.

Independent of that, we still need a way to measure the Hugo nominations. That's what this topic is about.

If you dislike the term "survey" I offer the term "measurement."

I like that. When we talk about voting it seems to imply obligations to the winners which I don't think we are obliged to honor, certainly not in the nominations.

Fans have opinions. There is such a thing as "The works which are the most supported, according to metric X, by the fans" It is something that can be measured -- if fans will express their true (and independent) opinions to us. Any factor, such as collusion, or voting strategy compromises how truly fans tell those views.

Your goal is that the fans should provide their true opinions about which five X are the best.

Sometimes something that looks to you like strategy might be an honest opinion. And regardless, other people have their opinions about what they want the Hugos to do.

For example, if you believe that there was only one X this year worthy of a Hugo, then the obvious way to express that opinion is to nominate only that one and then if it's on the final ballot to vote for that one. That's honest.

But if you believe that there were thirty X that were good enough for the Hugo but you want your one to win for some reason (like you want the author to survive as a professional writer and write more stuff you like) then you might also only nominate the one.

No way to tell which way a fan thinks unless you ask him and believe his answer.

Either way, he votes for just one candidate. We want to include as many ballots as we reasonably can, we want a nomination system that gives at least one win to a lot of fans. Either we give him that one, or his ballot is a loser. He has given us this implicit ultimatum. Any voting system has to handle it somehow.

If a bunch of people agree about which X they think are best, why shouldn't they vote that way? Most fans don't agree that much. But if some of them do, should we try to stop them from agreeing?

If they are shills who don't care about SF but just vote to help out their friend or patron, that's bad. But if they deserve to vote, then they get to vote their way. The most a voting system can do is to find some fair way to give them the same limited influence others have.

#352 ::: P J Evans ::: (view all by) ::: April 17, 2015, 11:28 AM:

351
Either way, he votes for just one candidate. We want to include as many ballots as we reasonably can, we want a nomination system that gives at least one win to a lot of fans. Either we give him that one, or his ballot is a loser. He has given us this implicit ultimatum. Any voting system has to handle it somehow.

I don't think that's going to fly very well at all.

#353 ::: TomB ::: (view all by) ::: April 17, 2015, 11:39 AM:

J Thomas @347: "We want five great choices. It's more important that the fifth choice be great than that the first choice be the best."

I agree that the relative ranking of the nominees doesn't matter. Due to the humongous size of the field, it is not possible to ensure that the five very best works are nominated, much less that they are nominated in order. One just hopes that the nominees are representative of the best of the year, and that the very best managed to get in as one of them.

What I want is for the actual best of the year to somehow get onto the ballot as one of the choices. (And a pony.)

"One book got marketed very well and every fan in the world has read it. 20% of them liked it. A second book got read by only 10% of fans and they all liked it. Then in the nominations say the first gets 200 votes and the second gets 100. "

Imagine that the second book was ready by 20% of fans and they all loved it. Then in the nominations it gets 200 votes, the same as the first book. There is no way to tell, just by counting votes, the difference between the widely marketed clunker and the neglected gem.

#354 ::: albatross ::: (view all by) ::: April 17, 2015, 11:43 AM:

Brad:

You've proposed giving the people running the award process the power to intervene in some way when they see evidence of slate voting. I wonder wheher any of the people who've run these processes would like that responsibility. It sounds like a nightmare to me.

My guess is that, say, disqualifying a bunch of nominations because they came from a slate, or disqualifying a nominated work because it was mostly nominated by a slate, would cause a lot of conflict and discord--probably many times what we're having now. It would certainly bolster the narrative of "see how our kind of works are robbed of their deserved awards?" It wouldn't be too shocking to see lawsuits arising from that, and at the least, I'd expect years of ongoing bad blood. This is especially true if next year's slate is explicitly not a slate (but rather another Greek of the same name)--you can imagine language that would make a reasonable-sounding case that you weren't doing a slate. (And you can't really forbid people making recommendations or suggesting a list of works you think are Hugo-worthy.)

There may be administrator actions that wouldn't kick off an endless conflict--letting the administratior add a work or two to the list on his own authority might work. But it looks like a bucket of snakes to me.

#355 ::: J Thomas ::: (view all by) ::: April 17, 2015, 12:12 PM:

@350 Jameson Quinn

@347: This very much aligns with the SDV-PE process.

Yes, kind of. What I want here is a simple obvious way to describe the goal.

Without that, somebody is going to say "Wait a minute, this candidate here had more votes than #5 and didn't get onto the ballot. This is wrong, bogus, corrupt."

I want to make an intuitive explanation why the five with the most votes don't have to be the winners. Otherwise, we have to either let a slate have most of the winners or else explain why the slate's votes are disqualified.

SDV-PE does not choose what I want but it chooses something vaguely similar, and it will often give a pretty good result. I would like to find something that does better, of course.

Also I will try to find a simple explanation for what SDV-PE does, to make a clear pitch for it.

#356 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 12:23 PM:

J Thomas @355: I think I'm having a little trouble perceiving what you're looking for that isn't covered by the goal of "balance selecting the works with the most nominations with selecting the works which together represent the most nominators' preferences." Does that capture it, or am I missing a nuance?

(Incidentally, in an SDV system, I'd be inclined to refer to the fractional votes at each stage as "votes" (or "points," if you really want to avoid election vocabulary) and the number of ballots on which a candidates is listed as "nominations" or "ballot appearances.")

#357 ::: J Thomas ::: (view all by) ::: April 17, 2015, 12:53 PM:

@353 TomB

"One book got marketed very well and every fan in the world has read it. 20% of them liked it. A second book got read by only 10% of fans and they all liked it. Then in the nominations say the first gets 200 votes and the second gets 100."

Imagine that the second book was ready by 20% of fans and they all loved it. Then in the nominations it gets 200 votes, the same as the first book. There is no way to tell, just by counting votes, the difference between the widely marketed clunker and the neglected gem.

Yes, exactly!

I can think of a way that would tend to palliate that, if the side effects weren't bad. If people could make negative nominations, then the things that a lot of people read and didn't like could show up. Things that people hadn't read would tend not to get negative nominations.

However, consider something that everybody has read and 80% didn't like. It might get 20% positive votes and 20% negative votes. (Because people tend not to think to make negative votes.)

Meanwhile something that only 10% of people have read that 20% of the readers liked and 80% disliked, might get 2% positive nominations and 0 negative nominations because the people who read it and didn't like it wouldn't imagine that it would get nominated at all. 2% won't get it on the final ballot, but there could be variations in there that go bad.

To know what to mark down you'd need to know what was nominated. It's hard to make just one change, one leads to another.

Also, people might play politics. Mark down things they think are good because they want their own favorites to win. If even 10% mark down everything they think is popular, it will be hard to find winners.

I don't know whether it's worth trying to fix this.

But if we don't, then we have to accept that the nominees with the most votes may not at all be the best.

So I think it isn't bad to ignore which have the most votes. Pick five that have the most varied votes, that together do the best job of getting something from each ballot, where each one got pretty many votes. That might not include the ones with the most votes. (If the ones with the most votes come from a slate, it won't include more than one of them.) But it will be a good survey of the ballots.

#358 ::: J Thomas ::: (view all by) ::: April 17, 2015, 01:16 PM:

@356 Cheradenine

J Thomas @355: I think I'm having a little trouble perceiving what you're looking for that isn't covered by the goal of "balance selecting the works with the most nominations with selecting the works which together represent the most nominators' preferences." Does that capture it, or am I missing a nuance?

I want to balance selecting the works which together represent the most nominator's preferences, with dividing the selected works' ballots into discrete groups while maximizing the size of the smallest group.

So other things equal, it's better for the stack of ballots that don't include a single winner to be 24% of the total than 25%.

And other things equal, if we can divide the winning ballots into 17% for A, 16% for B, 15% for C, 14% for D, 13% for E, that's better than if the best we can do is 20% for A, 16% for B, 16% for B, 15% for C, 12% for D, 12% for E.

Given ballots that include both A and B, and both A and E etc we have some leeway in how to divide them up.

The point here is that the winner with the least votes can carry its own weight. It is on a significant number of ballots. It isn't just something we stuck in at the end to recover the largest number of losing ballots after we maximized the first four.

#359 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 01:30 PM:

Cheradenine@356:

(Incidentally, in an SDV system, I'd be inclined to refer to the fractional votes at each stage as "votes" (or "points," if you really want to avoid election vocabulary) and the number of ballots on which a candidates is listed as "nominations" or "ballot appearances.")

I like calling them "points" a great deal, as I think we are pre-conditioned to thinking of "one person, one vote". It makes a person wonder, "How can I have a fractional vote?" We're writers (and readers), so I think we all have an intuitive understanding that words matter and often have a context beyond their actual meaning, so must be chosen carefully. One of the great things about physics is that I get to define a quantity and it means exactly what I want it to, regardless of what my reader wants it to mean. :) As a writer, I don't get that luxury!


JT@357 (and others):

I like the goal of (ideally) seeing nominators get -one- of their choices,but not necessarily their first choice, on the ballot as a good thing. In fact, I'd prefer that choices not even be ranked at all, even though we know each voter isn't going to like all of the works equally. It just seems to me that ranked choices still has too strong a connotation of a "winner", which isn't supposed to be the point at this stage. If there was no expectation that your nominations were ranked, I think most of fandom would accept it as just "part of the rules". There will always be the, "Well, this one isn't bad and probably deserves to be nominated, but I really want this other one to win!" But I'm all for letting the individual decide what constitutes "nomination worthy". Because, ultimately, I think that's what the nomination process is supposed to be -- not an election, really (though while there may not be a winner, there are obviously losers), but instead a threshold (set by the individual reader) as "above this, it's worth being considered, below this it's not". Of course, from a logistical standpoint, we have to decrease the number of nominees on the final ballot, so we choose a system that will narrow the field while still staying true to that principle. I've been going through all of the SDV-PE examples, as well as looking through some of the literature on SDV itself, and I'm starting to become convinced that SDV-PE may just meet those goals. I really think it's worth testing as a candidate system, and maybe even drafting some language to present it.

Kilo

#360 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 01:34 PM:

@356: In @347, there are three principles, which I interpret as:

1. Lots of average votes for each nominee.
2. Lots of ballots "satisfied"; few voters get 0 nominees.
3. No nominees with few votes.

Your question, if I understand it, is: why isn't 3 redundant with 1 and 2? So here's a scenario:

20: AB
5: AI
19: CD
5: CI
18: EF
5: FI
15: GH
7: GI
5: JH
1: J

Possible winner sets:
ACEGJ: 100 total votes from 100 total voters; worst winner J with 6 votes
BDFHI: 99 total votes from 99 total voters; worst winner F with 18 votes.
ACEGI: 116 total votes from 94 total voters; worst winner I with 20 votes.

I think that you could make an argument that BDFHI might be the best answer in this scenario. But according to principles 1 and 2 alone, it is dominated.

#361 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 01:39 PM:

J Thomas @358: I worry that that metric, as described, winds up actually disadvantaging the works with the most nominations, unless it's only the vote totals for E that are relevant. And, in that case, any criterion that focuses on the votes for a single nominee (the fifth in this case) is likely to lead to some distorted behavior where tweaks to the status of the fifth nominee have outsized effects on any measurement.

#362 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 01:45 PM:

In order to get SDV-PE to get the "right" answer in the scenario I just gave, you'd have to make the JH voters be a bigger pile and switch them to voting BDFHJ. It would also help if you switched some GI voters to IJ.

There's no way RAV can get the "right" answer in any scenario like this; there's no way to recover from the inevitable misstep on the very first nomination.

#363 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 02:01 PM:

Jameson @360: I see what you mean; a strict reliance on 1 and 2 would rank ACEGJ as better than BDFHI, and it probably shouldn't be. Nice example.

Isn't ACFHI probably superior to any of the listed options, though? (114 total votes from 99 total voters, worst winner H with 20 votes.)

#364 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 02:18 PM:

@363: You're right about ACFHI being better. You could probably tweak the example until it wasn't. But anyway, the main point of the example was comparing ACFGJ against BDEHI. (I mixed up E and F earlier.)

#365 ::: Brad from Sunnyvale ::: (view all by) ::: April 17, 2015, 02:28 PM:

A few notes on various:

I am using "strategy" to refer to individual strategy, as opposed to collusion. Slates are collusion, your votes are not independent.

There are many factors which might lead a nominator to not just offer their true opinion. I hope to find the system which minimizes the incentive to do that, and that is what I mean by choosing the system with the least strategy.

I do think Hugo admins would definitely not want to yield the power to intervene. I hope they would not want to yield it! But I think they would yield it this year if they had it; in general I hope this would be rare.

At the same time, a system of human justice offers a deterrent to attack. There is no point to attempting a slate if you know that if it is detected, it will be nullified as best as possible, and perhaps (though for liability reasons this may not happen) you will be identified as having tried to cheat, and your result will be negative.

As to why I say single-nominee has more strategy, this is because with multiple entries (both in nomination and final ballot) we reduce the extent to which you must "sacrifice" some choices to promote others.

#366 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 03:14 PM:

Brad@365:

I am using "strategy" to refer to individual strategy, as opposed to collusion. Slates are collusion, your votes are not independent.

OK, that helps me understand where you're coming from. But...

There are many factors which might lead a nominator to not just offer their true opinion. I hope to find the system which minimizes the incentive to do that, and that is what I mean by choosing the system with the least strategy.

...do you see how this is a different definition of strategy than the one you just gave? Slate voters are "not just offering their true opinion".

Furthermore, do you see how slates can happen through only individual strategy, without any collusion? Imagine the candidates were letters; my preferences are alphabetical, but I think that other votes will be approximately frequency-based (ETAOINSHRDLU...). Do you see how it behooves me to vote EAIHD rather than ABCDE? And how if there's a group of voters like me, we can end up all voting EAIHD without having to collude?

Finally, do you see how this kind of strategy is more problematic than the kind that can happen (much less, and much more riskily) under RAV or SDV-PE? If it's strategic to vote for popular candidates you kinda like ("lesser evils"), that becomes a self-reinforcing equilibrium. But if it's strategic not to vote for them, then strategy becomes less effective the more people use it, so it should tend to burn itself out naturally.

I do think Hugo admins would definitely not want to yield the power to intervene. I hope they would not want to yield it! But I think they would yield it this year if they had it; in general I hope this would be rare.

At the same time, a system of human justice offers a deterrent to attack. There is no point to attempting a slate if you know that if it is detected, it will be nullified as best as possible, and perhaps (though for liability reasons this may not happen) you will be identified as having tried to cheat, and your result will be negative.

It's better if enforcement is not a "nuclear option". When it is, the enforcers might be tempted to let minor violations slide; and if, as in this case, violations are a self-reinforcing cycle, that's a recipe for disaster.

As to why I say single-nominee has more strategy, this is because with multiple entries (both in nomination and final ballot) we reduce the extent to which you must "sacrifice" some choices to promote others.

I'm sorry, I don't understand what you're saying here.

#367 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 03:18 PM:

Brad @365: Any time you're adjusting your vote based on what other people are doing, your votes aren't independent, whether it's conforming to a slate or avoiding voting for the leading contender.

Block plurality (the current system) has a known serious vulnerability to slates (see here - yes, it's a Wikipedia link, the primary literature is too technical), but is also subject to being "gamed" even in a low-information environment. For example, it offers the strategic choice of nominating one candidate ("bullet voting") vs. nominating more than one. This isn't a decision offered with single-nominee. Saying the system where you have to choose one candidate to vote for involves more strategy than a system where you can make a tactical choice as to whether to choose one candidate or not is pretty much nonsensical.

(That Wikipedia entry is a little painful. "A coalition has substantial incentive to nominate a full slate of candidates, as otherwise supporting voters may cast some of their remaining votes for opposing candidates." Yeah, I think we just got to see how that works.)

#368 ::: perlhaqr ::: (view all by) ::: April 17, 2015, 04:02 PM:

J Thomas @339: "It looks to me like it increases the total vote count by 1000 for everybody else too, but increases their individual vote counts not at all."

Well, yes, that's true from a purely numeric sense, but not an accurate reflection of the effect.

Imagine you have a voting block of 200 Poor-Digestion Foxes, and they all nominate the same five books.

"Book One", by A Writer
"Book Two", by B Lister
"Book Three", by C Monster
"Book Four", by D Best
"Book Five", by E Pluribus

So, as it stands now, they vote that way, the vote count for each novel goes up by 200, and the divisor goes up by 200.

Under my theory, the vote count for each novel goes up by 200, but the divisor goes up by 1000.

Now, you've got a different pool of 1800 Trufen, of whom, 200 of them all happen to nominate "Book Six", by F Miner, and purely by happenstance, none of their other suggestions overlap in any way. (Ok, unrealistic, there just aren't that many books published in a given year.)

...

Actually, y'know, I think you're right that this doesn't accomplish what I thought it did. Bugger. :-/ Sorry. Nevermind.

#369 ::: albatross ::: (view all by) ::: April 17, 2015, 04:02 PM:

Short-term effect of SP/RP actions: Messing up the Hugos for a few years.

Long-term effect: SFF fans become major force in driving reform of election systems, having become fascinated en masse with election theory.

#370 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 04:38 PM:

@369: That's my devious plan. I'm even part of an organization devoted to the latter outcome. But I'm not supposed to post the name of that organization here, even legalistically endorsing common themes of lookouts on ... oh, forget it.

Our triumph is inevitable! Rot13 is futile!

#371 ::: J Thomas ::: (view all by) ::: April 17, 2015, 04:53 PM:

@361 Cheradenine

J Thomas @358: I worry that that metric, as described, winds up actually disadvantaging the works with the most nominations, unless it's only the vote totals for E that are relevant. And, in that case, any criterion that focuses on the votes for a single nominee (the fifth in this case) is likely to lead to some distorted behavior where tweaks to the status of the fifth nominee have outsized effects on any measurement.

I can't be sure that won't happen. I had a vague idea about a scheme to actually achieve the goal, and I hope it is easier to raise the minimum than to try to reduce the variance.

If I rate outcomes by how even all five are, that might involve a lot of computation.

I can imagine the works with the most nominations might often be disadvantaged, because sometimes they will tend to have the most overlap with other nominations, and so that might make them the most disposable. In a group of five works, the one that has the fewest unique ballots -- ballots that have none of the other four votes on them -- is the obvious choice to replace with something else. When the work with the most votes also has the most unique votes it's no problem.

I'd want some simulation to get a feel for what to expect, and it will take me some time to ramp up.

#372 ::: felice ::: (view all by) ::: April 17, 2015, 05:28 PM:

J Thomas @351: "Either way, he votes for just one candidate. We want to include as many ballots as we reasonably can, we want a nomination system that gives at least one win to a lot of fans. Either we give him that one, or his ballot is a loser. He has given us this implicit ultimatum. Any voting system has to handle it somehow."

We really want to encourage people to nominate as many works as possible. With fewer works nominated, there's likely to be less overlap between ballots, fewer nominations for the best works, and more power for slates. I'd recommend that in "making the pile of losers small" we count the number of nominations in the pile, not the number of ballots. So, for example, a pile of 10 losing ballots each nominating 5 works is worse than a pile of 45 ballots each nominating 1 work. That could theoretically advantage relatively small slates by ensuring they get one work on the ballot when they'd otherwise get none, if lots of genuine voters nominate single works anyway, but I think it's more likely to increase the number of non-slate nominations. And slates aren't the only form of manipulation; this is helpful against attacks like the Scientologist bulk nominations for Hubbard in '87.

#373 ::: J Thomas ::: (view all by) ::: April 17, 2015, 05:49 PM:

@372 Felice

We really want to encourage people to nominate as many works as possible.

I see no workable way to make people nominate more works when they don't want to. A higher limit on the number of nominations they can make would let them nominate more if they wanted to. And if a slate mostly just gets one of it nominations no matter how many it makes, that particular downside of allowing more is blocked.

With fewer works nominated, there's likely to be less overlap between ballots, fewer nominations for the best works, and more power for slates.

Yes, agreed.

I'd recommend that in "making the pile of losers small" we count the number of nominations in the pile, not the number of ballots. So, for example, a pile of 10 losing ballots each nominating 5 works is worse than a pile of 45 ballots each nominating 1 work.

Interesting! That's easy to game, though. Just put down the name you want, and then put down four bad names -- awful writing that can't possibly win. Then your ballot counts just as much as one with 5 sincere nominations.

That could theoretically advantage relatively small slates by ensuring they get one work on the ballot when they'd otherwise get none, if lots of genuine voters nominate single works anyway, but I think it's more likely to increase the number of non-slate nominations. And slates aren't the only form of manipulation; this is helpful against attacks like the Scientologist bulk nominations for Hubbard in '87.

I think it takes simulation or something to get a feel for what we're talking about. Say there are eight slates and each of them has 6% of the nominations. They can't all win. But if we throw them all out then that throws out almost half the ballots. If it reaches that point there may not be any adequate answers.

#374 ::: felice ::: (view all by) ::: April 17, 2015, 06:04 PM:

J Thomas @373: "Interesting! That's easy to game, though. Just put down the name you want, and then put down four bad names -- awful writing that can't possibly win. Then your ballot counts just as much as one with 5 sincere nominations."

True, but at least it makes you think about it; and you have to be sure that other people aren't writing down any of the same four bad names (otherwise there'd be a high chance of ending up with Terry Goodkind on the ballot!), so they have to be obscure bad names; and we can throw out ballots that nominate non-existent works. It would be easier and safer for most people to just nominate other works they do like, and since extra nominations that can't win do no harm, even a small proportion of voters nominating extra works would be a win.

"I think it takes simulation or something to get a feel for what we're talking about. Say there are eight slates and each of them has 6% of the nominations. They can't all win. But if we throw them all out then that throws out almost half the ballots. If it reaches that point there may not be any adequate answers."

If slates make up 48% of nominations, we're screwed anyway.

#375 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 07:43 PM:

@372: Good point. I think that all three proportional systems we're talking about now are better than the "average" proportional system in terms of not motivating a bullet voting strategy; and SDV-PE is the best of the lot. Frankly, I can't imagine a better proportional system in that regard. So I think they'd all do pretty well. It is also worth lifting the limitation of 5 votes per ballot, and even explicitly suggesting to voters that more than 5 votes is good.

#376 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 08:30 PM:

Jameson@375:
It is also worth lifting the limitation of 5 votes per ballot, and even explicitly suggesting to voters that more than 5 votes is good.

I can see why more than 5 nominations better meets the goals we're developing, but I'm not 100% certain I see all the effects of that with SDV-PE. Say I have 10 nominations on my list. Each nomination now gets fewer "points" (to use Cheradenine's proposed terminology) for each work listed (which seems reasonable and fair). Eventually, as nominations are eliminated, I think I can see that you're going to end up in the same place (at the slimmed down list of nominations), but getting to that point might be quite different, correct? Is SDV-PE going to be independent of what happens in the middle rounds of elimination? It's not clear to me that it will -- though it's also not clear to me that it would matter if it wasn't. Can you or Cheradenine give me some guidance as to how that would play out (i.e., the same, different and it matters, different and it doesn't matter, etc.)?

Thanks,
K

#377 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 09:01 PM:

@376: Almost certainly the same, except that with more votes overall, the top works would tend to have more votes too, so that slates working with a fixed number of voters would be less effective.

The other difference is that some works which are more "everyone likes it but few really love it" would do slightly better relative to others. I suspect this is a minor effect.

In theory, it is possible for the eliminations to work out differently aside from the above changes, but in practice, I very much doubt anything else would change. Most things would be in about the same place relative to everything else (in both fraction of total points and in number of voters), and so end up eliminated at much the same time.

#378 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 09:26 PM:

@377

Okay, that's the impression I was getting too, but I couldn't convince myself it was right. That's a good thing.

So we could conceivably say in the new system, "Nominate as many works as you like, the more the merrier!" I can see that actually as an attraction to voters, both because it seems more inclusive and because it removes some of the pressure of how many works you should be putting forward. If they can put up as few or as many as they wish, you've got a system that further leaves the choice to the voters -- something I strongly approve of for a "fans' award". One could argue that might even be a strength over the current system.

Of course, since every nomination is a write-in, I can see the extreme cases getting to be a tedious headache for those who have to verify that the write-in nominations are valid. But maybe if that step wasn't applied until further along into the rounds (when the field has been narrowed), it wouldn't be such an issue.

Kilo

#379 ::: felice ::: (view all by) ::: April 17, 2015, 09:58 PM:

felice @286: "as far as I can see, it's not against the current rules, so it could in theory be done next year. Would it be possible for the Business Meeting to vote to make Option 5b an explicit requirement for future awards, and recommend that MidAmeriCon II voluntarily carry it out before the change is ratified? As well as hopefully protecting next year's awards from Puppies, this would give the system a trial run before ratification, allowing for a more informed vote at next year's Business Meeting."

Does anyone have any thoughts on this?

#380 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 10:16 PM:

Felice@379:

As I understand it, a given Worldcon can do any kind of sample election they want, but the -actual- Hugo has to be awarded based on the current constitution, which can't be changed for two years at least. That said, if a system were approved at the business meeting, I think it would be a logical step to test it out at the next Worldcon, even if the results were non-binding and unofficial. That would apply to any system, not just option 5. That said, I strongly suspect no one would undertake the hassle of a test unless a specific system had been formally approved.

Kilo

#381 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 10:45 PM:

Felice @379, Kilo @380: I think the reference is to releasing the names of the top 15 nominees at the halfway point of the nomination period. I suspect the answer is political - would the WorldCon meeting support it? Would it reduce the chance of other reforms passing the meeting? I'll defer those to people with more WorldCon experience than my zero. Trying something - even if I'm not sure it will help - to fight slates in the "gap year" when we can't reform the rules system certainly has some appeal.

#382 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 11:02 PM:

Oh I see, I did misunderstand. So basically the idea is to see if a counter slate is needed or desired, yes?

Hmm, not sure how I feel about counter slates even as a temporary meyasure. It just seems hypocritical. I think for me, anything that moves towards political parties is a bad thing. I realize option 5 doesn't require that, but I suspect it would inevitably lead that way based on what we've discussed here, so I guess I would oppose it.

I'm dreaming of course, but I would like even the puppies to buy into whatever new system was proposed. If it can be shown to be fair to everyone, it would lay their arguments to rest, but ultimately, their actual argument is that "our stuff isn't on the ballot" and nothing else will satisfy no matter what is done. VD had already gone on record as saying if any no award takes any category (i.e., if he doesn't win), he'll make sure that category never sees another Hugo. So I'm not sure there's any reasoning with that. I would hope the sad puppies are more sincere in their beliefs, but who knows.

If I had the time and energy to deal with the inevitable abuse, I'd really like to post on Brad T's site and ask. But I suppose I can guess how that would go.

Kilo

#383 ::: felice ::: (view all by) ::: April 17, 2015, 11:15 PM:

Cheradenine @381: "Felice @379, Kilo @380: I think the reference is to releasing the names of the top 15 nominees at the halfway point of the nomination period."

Yep; the top 15s from the actual nominations, not a separate sample election. And allowing people to update their nominations at any point during the nomination period, in the same way they've been able to update their final votes during the voting period in the past.

http://www.thehugoawards.org/the-voting-system/ says "The full details of the nominations are not released until after the final ballot. This is to prevent the nomination numbers influencing voters in the final ballot. " - Option 5b doesn't involve releasing actual numbers, just a pseudo-longlist, so it shouldn't influence voting on the final ballot (though it will influence what ends up on the final ballot - that being the whole point).

("Pseudo-longlist" because works don't have to appear on that list to get on the final ballot; there could be a surge in nominations for something else due to "What? I can't believe X didn't make the top 15! I'd better nominate it myself")

Keith "Kilo" Watt @382: "Oh I see, I did misunderstand. So basically the idea is to see if a counter slate is needed or desired, yes?"

Not exactly; the idea is to get more nominations for the works that have a chance of winning, so a greater number of people have a say in the makeup of the final ballot. This is particularly important for categories like short story where nominations are spread out so much that we can get as few as three works crossing the 5% threshold to make the ballot. It helps against slates (irrespective of whether the slates put in their nominations before or after the publication of top 15s), but that's not the only advantage.

#384 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 11:30 PM:

Okay I got you now. That's basically what I (and JT, I think) were supporting a while back, I believe. I've become convinced that it doesn't mesh with what most people feel the Hugos should be (which, in my mind, is a key factor), but as a temporary measure? I might be able to see that. I had pretty much written off the next two Hugos anyway, so maybe it's not so bad as a short term thing.

The only problem I see is that, the way I read the constitution, while there may not be anything against posting the top 15, I think that the ability to -change- your nomination would have to be a change to the constitution (which doesn't help).

Kilo

#385 ::: felice ::: (view all by) ::: April 18, 2015, 12:06 AM:

Keith "Kilo" Watt @384: "Okay I got you now. That's basically what I (and JT, I think) were supporting a while back, I believe. I've become convinced that it doesn't mesh with what most people feel the Hugos should be (which, in my mind, is a key factor), but as a temporary measure? I might be able to see that. I had pretty much written off the next two Hugos anyway, so maybe it's not so bad as a short term thing."

It's only next year that's the problem (all going well at the Business Meetings, anyway); changes can be voted for at Sasquan this year, ratified next year, and in place for the 2017 Hugos. Do most people really feel the Hugos should let lots of nominations be wasted on works that have no chance of making the final ballot? 15 is a fairly arbitrary number - is there some other way of determining how many works to list at the halfway point that would work better for you? Eg the larger of everything nominated by at least 5% of voters in the category and the top 15 works nominated by at least 5 people? (That would be a theoretical upper limit of about 100, but probably much lower in practice since a large percentage of first-half nominations would be for works below the threshold or for works supported by significantly more than 5% of voters).

"The only problem I see is that, the way I read the constitution, while there may not be anything against posting the top 15, I think that the ability to -change- your nomination would have to be a change to the constitution"

No, they just don't count as submitted till nominations close; the website stores your current choices, and whatever's in the database at the deadline is what's submitted. There's nothing in the constitution about changing votes on the final ballot either, and that's not a problem, so it shouldn't be an issue for nominations either.

BTW, who here is actually able to attend the next two Business Meetings? Getting to the WorldCons is well out of my budget, unfortunately.

#386 ::: Brad from Sunnyvale ::: (view all by) ::: April 18, 2015, 03:58 AM:

Yes, just to note: For some reason my brain typed "yield" a few times when I meant "wield." It's been a stressful week. (Those who are comic book fans will know why.)

To further clarify terms, when I say independent, I don't mean independent of information out in the world, I mean independent of collusion with other nominators. A "slate" is not a slate unless there is some sort of agreement between two or more people to both vote it to increase its chances. (And yes, the person proposing a slate is not strictly colluding, it is the person who follows through who is colluding, the former person is colluding only if they know of the other. But there is still collusion by the latter in any event.)

This is the thing we want to get rid of, and trying to get rid of accidental slates is one of the problems we face, as well as strategic voting.

My general feeling is:

a) Only a modest percentage of fans will be strategic in their choices.
b) Those who are will do it based on fairly strong signals. However, those come often enough, there are many years when most fans will correctly identify a shoo-in for that year's nomination. (In certain categories you can often predict 3 or 4 of the 5.)

In some cases it will be "that semiprozine is always nominated." In other cases it will be "that book won 3 awards and the Locus poll." Either way, that's a real thing that happens, where you can get a high degree of certainty. Knowing the likely order of nominations is not.

With those givens, examine some simple situations:

1) The shoo-in is your favourite, and a less popular work is your 2nd choice.
2) The less popular work is your 1st choice, the shoo-in is your 2nd.

Consider each of the above systems. In basic approval the course is pretty simple. Name them both.

But in several of the above systems, the correct strategy, if you are trying to "win," is to name only the less popular work. Not always, but too often.

There are strategies in all the systems. What concerns me is how easy they are to understand (they will be done more, and blogs will advocate for them,) how much risk they present, and what reward they offer. Sadly, a number of the proposed approaches do poorly here -- fairly simple plans offer good reward for little risk.


#387 ::: J Thomas ::: (view all by) ::: April 18, 2015, 05:04 AM:

Hmm, not sure how I feel about counter slates even as a temporary meyasure.

I don't see it as a slate.

I'll use the same numbers that I've almost memorized:

In 2013, for novels, there were 1113 nominating ballots. There were a total of 717 votes for winners, and 785 votes for the 10 losers with the most votes. That's 1502 votes that had some kind of chance to win. If I read it right, there were 3837 votes cast. 2335 of the votes cast could not possibly have won. 60%. (There were an average of around 3.5 votes per ballot.) Brad of Sunnyvale is right that in the current system there is no advantage for nominating only once (except that there are four fewer votes for competing works). All of the votes not cast were wasted.

Obviously, people who vote a slate have no votes wasted, they have 5 votes for their candidates.

If the people who have already voted for the ones they thought were the best were to then vote for the ones that could win that they wanted, I don't see how that could be a bad thing.

Suppose the top 15 were announced, and that consisted of 5 from a slate that nobody but the slate wants, and 10 others. Voting for 5 out of 10 is not a slate. Voting for as many as you want out of 10 is not a slate.

If the slate waited until the last minute to make their nominations, then the top 15 would be 15 others and the vote would be diluted more. Doing this would imply that the slate did not expect others to join them. A slate which is among the top 15 could get votes from others. A slate that tries not to do that, is admitting they don't think others would want to vote with them.

#388 ::: J Thomas ::: (view all by) ::: April 18, 2015, 06:15 AM:

@386 Brad from Sunnyvale

1) The shoo-in is your favourite, and a less popular work is your 2nd choice.
2) The less popular work is your 1st choice, the shoo-in is your 2nd.

Consider each of the above systems. In basic approval the course is pretty simple. Name them both.

In basic approval, if you would hate for your second choice to win while your first choice loses, maybe you should vote only for the first choice. A vote for the second choice increases the chance of the outcome you hate.

But in several of the above systems, the correct strategy, if you are trying to "win," is to name only the less popular work. Not always, but too often.

If you like six things but you are only allowed to nominate five, then it might make sense to leave off the choice which is most likely to be nominated anyway. However, only five can win. If you are nominating your last choice -- the one you wouldn't nominate if you were expressing your true feelings -- because you think something you like better doesn't need your vote, what is going on here? Only five can win. You think your sixth choice won't win without your vote, so you are giving it your vote so it has a chance to displace something you rank higher?

Maybe you think your sixth-best-choice can't win, but you want it to be listed as something nominated for? Something in the top 15, maybe? Well, OK.

I'll try again. You think the first choice is so popular it will be nominated regardless. If you vote for it, you will reduce the chance your second choice will also be nominated. So you only vote the second. How does that work?

Oh! Proportionate voting. As long as you have two choices that haven't been eliminated, your vote counts less, for example half. The more popular choice will not be eliminated while the less-popular choice is still there. So you never get more than half-a-vote for the less popular choice.

Yes, there are voting systems like that. Let's not use them.

I'll look at SDV-PE which is the main one I'm interested in at the moment.

Barring ties, if a particular nominee is among the top five for votes, it will win any challenge to be removed. And barring ties, if it is among the top four with the system of proportional voting, it will never be considered for removal.

So apart from special cases coming from ties, the top four will be the ones with the best proportional votes, and the fifth will be the one with the best absolute vote among the remainder.

So if you are sure that one of your choices will have the most votes even without your vote, you can let that one win its place in the fifth slot. By voting for only the other one, you improve its chance to get into one of the four other slots.

That only works when you are sure it has the most votes, though. And mostly when there is only one other you care about. If you have four other nominations then they each start out with 1/5 vote, and leaving off the best would only bring that up to 1/4. But the last one, the one with the best chance, would still only get half a vote.

I could be wrong, but I'm afraid this is one of the ones where that strategy works. Provided you are sure that one of your choices will be the one that gets the most votes, and won't come in second. (As a picky point, any of the four that win on proportional vote could have a higher absolute vote and not get in the way.)

#389 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 08:20 AM:

@386: Let's take the example I used before. Your preferences are ABCDE...etc, your expected outcome without your vote is approximately ETAOINSHRDLU...etc. You are inclined to vote strategically.

Under the current system, your logic would be like this:
A? Of course, it's my favorite, and has a chance to win.
B, C? No, don't want to waste one of my 5 precious votes on something with no chance.
D? OK. It probably won't win but I'll give it a shot.
E? Well, maybe. I mean, it isn't bad, but it can probably win without my vote. Depends on how strategic I'm being, but let's say I don't vote for it.
F, G? See B,C.
H, I? Yes, I hope this wins, and beats the hated O, N, and S.
J, K? See B,C.
L? Yes.
Final ballot: ADHIL (extreme strategy) or ADEHI (reasonable strategy)

Under SDV-PE, here's your logic:
A? I expect this can win without me, but I'm not certain. So I'd probably vote for it. But if I'm highly strategic and feeling lucky, I guess I could risk leaving it off.
B-D: Why not? B and C will be eliminated early but that's no skin off my back for voting for them; D has a bit of a chance.
E: Probably not.
F-I: Why not?
J-L: Pretty much a toss-up.
So, final ballot BCDFGHI (extreme strategy) or ABCDFGHI (moderate strategy)

By any reasonable measure, the current-system ballot is more dishonest than the SDV-PE one. In particular, it includes various kinds of strategy (leaving off candidates for being too weak and for being too strong), while the SDV-PE only leaves off those that are too strong, and even that, probably only if the voter doesn't like them very strongly.

---

@387: Everything you say here is true. Just to play devil's advocate for a moment, though: announcing a "long list" early would mean that early voters would have extra influence. Is that fair?

---

@388: You've explained it pretty well. But you misspeak at one point: "Barring ties, if a particular nominee is among the top five for votes, it will win any challenge to be removed." No. If it is the top for votes among those which are not among the top four for proportional support, it will win any challenge. A slate could easily have all of the top 5 for votes, and still end up with only 1 or 2 slots. (It's clear from the later stuff that you actually understand this, but I just wanted to correct it so others wouldn't read this and get a wrong understanding.)

#390 ::: J Thomas ::: (view all by) ::: April 18, 2015, 09:04 AM:

@389 Jameson Quinn

Your preferences are ABCDE...etc, your expected outcome without your vote is approximately ETAOINSHRDLU.

It's plausible you might know your preferences are ABCD....etc. But the expected outcome without your vote is likely ET? or ETA? or possibly ETAO? or in utterly extreme and unlikely cases ETAOI?

If you can with any certainty predict the fifth spot in the Hugos, you probably have skills that can make a whole lot of money.

With uncertainty your strategy would probably be to leave off E and maybe leave off A. BCDFG would be plausible. Maybe BCDFI if you think I has a chance but it's kind of iffy.

On the other hand, a lot of people prefer to win over having the maximal chance to make a difference. To be reasonably sure you would have one winner, you might do ABCDF or ABCDI. But you might not be sure A will win either. You could lose every single time! The safest is to do ABCDE. Then you're reasonably sure E will win, you're helping out A which might not, and BCD deserve their chance. You could do ABCEI on the assumption that I has a chance while you can't predict D's chance. But if you vote for I which you don't like that much, and later you find out that D almost made it and I didn't, you'll feel bad.

It's just hard to tell. If you wander around blogs etc people will try to persuade you that their favorites are ahead so you'll vote for them. What do you really know? It's a fog.

If you vote for your real favorites you at least know that you did right by them.

#391 ::: J Thomas ::: (view all by) ::: April 18, 2015, 09:14 AM:

@387: Everything you say here is true. Just to play devil's advocate for a moment, though: announcing a "long list" early would mean that early voters would have extra influence. Is that fair?

It depends. If you vote early you're flying blind. You *might* vote for things that do well and so your vote influences people. You might vote for things that do badly and your vote has no influence at all.

Meanwhile the people who vote late get to influence which of the top items win, if they choose to vote that way.

I would like for each voter to get at least two ballots. Vote early, and then vote for different things late. That could be equivalent to letting each voter do acceptance-voting for up to 10 things. Five early and five late. So everybody who bothers to vote early gets influence, and everybody who bothers to vote late gets the other kind of influence.

But it's easy to stretch the rules enough to announce early leaders halfway through. Allowing people to split their ballot and vote on two different occasions might be a bigger stretch. They might be more likely to think they have to vote on a rule change and not do it for 2 years.

#392 ::: J Thomas ::: (view all by) ::: April 18, 2015, 09:20 AM:

"Barring ties, if a particular nominee is among the top five for votes, it will win any challenge to be removed."

I apologize, that was bad proofreading.

The first four slots will be filled by the top four according to SDV. The fifth slot will be filled by the one with the most absolute votes among all the others.

#393 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 09:49 AM:

"If you can with any certainty predict the fifth spot in the Hugos, you probably have skills that can make a whole lot of money."

I think it's actually plausible that you could divide the pool into about 4 tiers, such as the scrabble-point-based division:
E
TAOINSRLU
GD
(everything else)

You'd have no idea if T had a better chance than L, but you could be pretty sure that three-point-and-up scrabble letters like B and C had no chance at all. (And you'd be wrong in some regards — for instance, H has a better chance than you think it does — but there's no way to know where you're wrong.)

Based on that division, the current-system strategic vote is ADGIL or ADILN, and the SDV-PE strategic vote is ABCDFGIJKL. It's clear that the former are more dishonestly strategic.

#394 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 09:55 AM:

Oops, I left out H. The SDV-PE strategic vote is ABCDFGHIJKL

#395 ::: Keith "Kilo" Watt ::: (view all by) ::: April 18, 2015, 10:57 AM:

JT@390:

It's just hard to tell. If you wander around blogs etc people will try to persuade you that their favorites are ahead so you'll vote for them. What do you really know? It's a fog.

I think you've just hit on one of the prime qualifications a system should have to be considered "strategy-resistant". I think what matters is not if there exists a strategy, given perfect knowledge of the order of potential winners. What matters is that the uncertainties in any strategy are so large, that you may as well vote your preference.

Kilo

#396 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 10:59 AM:

So there's been a lot of discussion of option 3, a bit of 4 and 5, and general skepticism to option 1. But that still leaves option 2. It's actually pretty easy to incorporate option 2 into SDV-PE. Just say:

1. No option will be eliminated if it has over X% of voters supporting it and no other non-eliminated candidate, where X is the lowest percent of voters supporting any nominating candidate in that category in the previous year (and thus, X is never less than 5%).

2. For the purposes of elimination, any ties are both eliminated at once, unless that would take the number of candidates below 5, in which case neither is eliminated and the process stops. In this case, any totals separated by under 3 ballots are considered to be ties.

Both of these rules could expand the number of nominations. But neither of them would be likely to be a factor in "best novel", where the burden of reading an extra work would be highest. In general, they might expand the "Short Story" nomination list to 6 or 7 — still quite manageable. And that would be an additional bit of insurance against slates, since the extra nominations would likely be non-slate works.

#397 ::: Keith "Kilo" Watt ::: (view all by) ::: April 18, 2015, 11:38 AM:

Jameson@396:

I see what you're saying, but it seems to me that it adds complexity for not a lot of benefit, doesn't it? The argument could be made that the more changes to the current system that we propose (e.g., now there are more than five nominees on the final ballot), the harder it will be to pass the business meeting. The idea of "nominate 4 and have 6 winners" is apparently gaining traction elsewhere in the blogosphere, apparently, mostly because of it's simplicity, I think. I don't really think that solves the problem, since the ballot would still be dominated by a couple of slates.

I do like the opposite idea, though, of letting voters nominate as many works as they like, with the understanding that doing so means each work gets fewer of their "points" (1/N). That seems an intuitive trade-off that most anyone can understand and agree with.

#398 ::: Kevin Standlee ::: (view all by) ::: April 18, 2015, 01:23 PM:

As Chairman of this year's WSFS Business Meeting, I'm trying to avoid proposing new business or taking a stand on any specific proposal, but I did recently propose something that is not a constitutional change in that it's legal under the current system, but unprecedented and not in keeping with existing tradition. It doesn't necessarily change the finalists, but it does shed some light on the process:

  1. Instead of announcing the five finalists in each category, announce fifteen "semi-finalists" in random order with no indication of who has the most nominations. (For the actual number of semi-finalists, pick a number greater than five; I use fifteen here because there is an existing rule about the top fifteen nominees, that's all.) The Administrator doesn't contact in advance the semi-finalists and doesn't necessarily do exhaustive eligibility checking. Announce that the finalists will be drawn from this list of semi-finalists and that the final ballot will be announced on a specified date.
  2. The Administrator starts contacting the semi-finalists, saying, "If you are selected as a finalist, will you accept your position on the shortlist? If you do not decline by date X, you are assumed to have accepted. This decision cannot be undone after that date." BTW, inasmuch as some nominees turn out to be difficult to contact, having the semi-final list in public helps this step, even as it increases the number of nominees the Administrator has to contact, because you can ask in public for help finding a nominee.
  3. During this same period and in parallel with it, the Admin does eligibility checking. This also is helped by the "crowdsourced" nature of the semi-finalists, because you can bet that people will turn up prior publication and the like during the process.
  4. At the specified date, the Admin announces the five finalists in each category, based on the top 5 semi-finalists who (a) did not decline and (b) appear to be eligible. Note that this means that finalist simply cannot break a news embargo because they don't know whether or not they made the shortlist until it is announced.
  5. The final ballot then continues as usual.

As I said, I don't think this process violates the existing WSFS Constitution, but it is clearly not the traditional process. When I proposed this to a group of people with considerable experience (the SMOFS e-mail list, the secret of which is that it's not secret), the reactions included:

  • The extra workload on the administrator (255 potential finalists instead of 85 including the Campbell) far outweighs the strain of contacting the 85 current finalists and determining eligibility during a very short period between the end of nominations and the finalist announcement. (This from a recent Administrator who has had to do the detail work himself, which I admit that I never had to do during my turn in the barrel.)
  • Such a change, while legal, is so unprecedented that it should not be done without explicit constitutional sanction. This is the "no elephants in mouseholes" principle, in that WSFS rules aren't made in a vacuum but in the context of the traditional way of administering the election.

While I cannot easily dismiss either of the above two arguments, I also think that my proposed procedure would deal with a couple of the known logistical flaws in our existing process that are not part of the issue over how five-highest-pluralities can be dominated by an organized minority bloc.

#399 ::: J Thomas ::: (view all by) ::: April 18, 2015, 01:43 PM:

@398 Kevin Standlee

I like your proposal, for what that's worth.

The extra workload on the administrator (255 potential finalists instead of 85 including the Campbell) far outweighs....

Given the nonsecret and crowdsourcing side, it might not in fact be that much extra workload. It might turn out easier. So if you could get permission for the administrator to do it or not, his choice, then the business meeting would not have to decide about this question of fact. Of course, if the administrator in question thinks he wouldn't want to do it, it probably isn't worth considering further for this year at all. If he does want to do it, that's pretty strong in its favor.

Such a change, while legal, is so unprecedented....

I can't argue with that.

#400 ::: Brad from Sunnyvale ::: (view all by) ::: April 18, 2015, 02:15 PM:

By the way, I do think STV is the least strategic of the proposals. STV (and its cousins) are known to be the least strategic systems, and that's why they are used in the final ballot. I have not had time to study the multi-winner version -- if a work has 40% of the 1st place votes, how is it decided to pick which half of those votes which get eliminated? Randomly, I presume? Or perhaps with one of the happiness algorithms?


Anyway, I am afraid I must not be communicating my view on how strategy gets applied very well. Several keep citing strategies for approval. They exist, but are very weak. People just don't care that much about their 6th place choice and how much they help it. That's by definition -- it's their 6th place choice. Because they don't care that much, it reduces their tolerance for putting any risk on higher placed choices. The most I can ever see is somebody not nominating a 4th or 5th place shoo-in to get their 6th on their ballot.

This is not true of some of the proposals, where you can affect how much you support your 1st place choice. Again, by definition, that is the one you care about. Because you know your intuition about shoo-ins is uncertain, you will only use it to help your #1 or #2 choices in most cases.

So it doesn't matter if you lay out a scenario which shows some mild strategies for Approval, or STV or any other. The key is, how often will we get the combination of factors which pushes a voter to the strategy. I believe it is necessary that:

a) They have strong confidence in their prediction of the shoo-in. (This happens reasonably often in novels, though not every year, and very often in certain categories.)

b) The work they wish to help with strategy is one of their top choices.

There are extreme situations where this would modify your plain approval ballot -- that's not what I think we need to discuss. What is important is, "how likely is it that the reward exceeds the risk, and does so clearly in a member's mind?" That's how much strategy will be applied.

And again, one point I have made which I am surprised I don't see any comment on. I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so. Yes, there is a backup system -- Vox Day lost to No Award in 2014, but that hardly discouraged him in 2015. However, it did keep Ken Liu off the ballot. (Though he is on it now, as a translator, in 2015.) (It may not be fair, but I feel less pain for Seanan McGuire who may also have been nudged off.)


Another note -- while not nearly as important as winning, there are people who pay attention to the total nominee counts published after the awards. Writers want to know how they did. Fans look for up and coming writers. These systems tweak these numbers as well. Only honest and independent fan opinion gives valid numbers there. This year people will look with special interest, in fact, for the "real" nominees.

Kevin#398. I suspect promoters of slates will take more care in future about which works are eligible, and you won't see a repeat of 2015.

#401 ::: nathanbp ::: (view all by) ::: April 18, 2015, 02:48 PM:

Brad@400:
Excess votes in STV are typically divided according to the proportion of voters' second place choices. For example, if the quota is 300, 500 people vote for work A, and their second place choices are evenly split between B, C, D, and E, each of B, C, D, and E gets 50 ((500-300)*(500/125)) votes for the next round. This does mean that in a widely divided field you can get very small fractional votes being added.

And again, one point I have made which I am surprised I don't see any comment on. I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so.
I'm pretty sure there's no voting system that prevents slates from getting at least 1 nominating slot. Our discussions have mostly been focused on preventing a slate from getting all 5. I don't believe there's any way that would work and be perceived as fair to eliminate slates entirely.

Anyway, I am afraid I must not be communicating my view on how strategy gets applied very well. Several keep citing strategies for approval. They exist, but are very weak. People just don't care that much about their 6th place choice and how much they help it. That's by definition -- it's their 6th place choice. Because they don't care that much, it reduces their tolerance for putting any risk on higher placed choices. The most I can ever see is somebody not nominating a 4th or 5th place shoo-in to get their 6th on their ballot.
Not nominating your 4th or 5th place shoo-in to increase the chances that one of your other top 6 choices is nominated seems very similar to the amount of strategy there is in RAV or SDV. I guess for approval this only shows up if you have 6 top choices to choose between, whereas for RAV and SDV it shows up even if you have less than 6 top choices.

#402 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 03:59 PM:

@400:

"STV (and its cousins) are known to be the least strategic systems"

By whom? On what planet? This is really, really not true. There's a whole literature on how many seats have been strategically swayed by vote-management strategy in STV elections. And if by "(and its cousins)" you mean IRV, then I really don't even.

In particular, in the Hugo, a "shielding" strategy in STV would be common, and could lead to very pathological results. The idea is that (in my alphabet example) you'd vote QA rather than A so that if A has a quota to start out with it doesn't use up any of your vote. This strategy could even lead to a popular candidate getting eliminated prematurely.

Then later you say:

"a) They have strong confidence in their prediction of the shoo-in. (This happens reasonably often in novels, though not every year, and very often in certain categories.)

b) The work they wish to help with strategy is one of their top choices."

You're right, these are indeed the conditions for "strategy" to be worthwhile in RAV or SDV-PE. But if, as you say, people don't care much beyond their first few preferences, in that case, the alphabet voter will vote ABCD. This is "strategic" in the sense that they have chosen their cutoff based on their expectations of what others will do; but it is still a perfectly honest ballot. If everybody voted like this, it's possible (though unlikely) that E would fail to be nominated; but in that case, it would be in part precisely because E had already won enough acclaim, sales, and/or prizes and didn't really need a Hugo. Why, then, is it a bad thing if, through perfectly honest voting, the Hugo voters decide to focus on other worthy works?

#403 ::: J Thomas ::: (view all by) ::: April 18, 2015, 04:42 PM:

I found this mildly amusing.

I was putting together a potential nominating system that depended on arranging ballots in patterns. To do that I was using some set theory, I'd take piles of votes and do intersections and unions etc. I noticed that a few votes somehow disappeared, and I couldn't tell where they were going.

It turned out that with that programming language, when you convert a list to a set it throws away all duplicates. A set has no two the same. So by accident -- just because of the rules of math, it replaced every slate (and accidental slate) with a single vote.

I think I see a way to work around that, but for the moment I'll keep it. It's simpler to play with. And I like it.

#404 ::: felice ::: (view all by) ::: April 18, 2015, 05:22 PM:

Kevin Standlee @398: "Such a change, while legal, is so unprecedented that it should not be done without explicit constitutional sanction."

Does that mean it shouldn't be implemented till after it's ratified and formally added to the constitution, is is the approval of one Business Meeting sufficient?

"my proposed procedure would deal with a couple of the known logistical flaws in our existing process that are not part of the issue over how five-highest-pluralities can be dominated by an organized minority bloc."

Possibly not the best time to bring it up, then? It seems likely that the minority bloc issue will generate quite sufficient business to fill up this year's meeting all by itself...

#405 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 05:32 PM:

I didn't mean to imply that STV was the highest-strategy system in general. But I do believe that it would lead to more dishonest strategy than any of the other voting system options under discussion here (except the status quo). You could make an argument the other way. But "Everyone knows STV is low-strategy" is not an auspicious beginning for such an argument.

#406 ::: felice ::: (view all by) ::: April 18, 2015, 05:45 PM:

Jameson Quinn @389: "@387: Everything you say here is true. Just to play devil's advocate for a moment, though: announcing a "long list" early would mean that early voters would have extra influence. Is that fair?"

If everyone knows early voters will have extra influence, and everyone can choose to be an early voter if they want that influence, then yes, perfectly fair.


J Thomas @391: "I would like for each voter to get at least two ballots. Vote early, and then vote for different things late. That could be equivalent to letting each voter do acceptance-voting for up to 10 things. Five early and five late. So everybody who bothers to vote early gets influence, and everybody who bothers to vote late gets the other kind of influence."

That's an unnecessary complication. Just do it the same way final ballot votes are already done; you can edit the content of your form any time you like, and only what's in the database at closing time counts. (Paper form nominations don't have that option, but is there anyone who still has to use paper forms?) No rule change required, and it really should be implemented irrespective of whether or not Option 5b is approved. Let people input their current favourites as soon as nominations open, and edit them if they read anything new during the nomination period that they think's better than their original choices.


Jameson Quinn @393: "Based on that division, the current-system strategic vote is ADGIL or ADILN, and the SDV-PE strategic vote is ABCDFGIJKL. It's clear that the former are more dishonestly strategic."

How much of that is counting system, and how much removing the limit on number of nominees? Would the SDV-PE strategic vote be ABCDF if it was restricted to five choices? (And I have reservations about allowing more than five works per nominator)

#407 ::: J Thomas ::: (view all by) ::: April 18, 2015, 06:24 PM:

@406 felice

J Thomas @391: "I would like for each voter to get at least two ballots. Vote early, and then vote for different things late. That could be equivalent to letting each voter do acceptance-voting for up to 10 things. Five early and five late. So everybody who bothers to vote early gets influence, and everybody who bothers to vote late gets the other kind of influence."

That's an unnecessary complication. Just do it the same way final ballot votes are already done; you can edit the content of your form any time you like, and only what's in the database at closing time counts.

I did not realize that's how it was already.

There were some drawbacks to that which I think are minimized if the extended list is revealed only once. With a similar proposal it might be possible to game the system, to vote for something else to trick people and then change later.

But if the leading nominees are revealed only once, a cabal would need tremendous communication among themselves to even track what they were doing. Complicated plans would fail. And they couldn't trick people very much. Things are either on the list or off the list. Vote for something that otherwise wouldn't be on the list, and then take away the votes later. Big deal. To do that they risk their own stuff not getting on.

It looks at worst harmless.

People who want an accurate look at what people want, would do better to look at the voting before the reveal. Afterward it's contaminated by what other people want. We can save the old opinions, so that's potentially available.

OK, I don't see any harm in letting people unvote things they've already voted, instead of just add new votes. All the things people complained about it before were connected with other parts of the proposal that are gone.

It looks good to me.

#408 ::: TomB ::: (view all by) ::: April 18, 2015, 06:39 PM:

Here are my thoughts so far on the proposed voting systems:

Option 3a: Satisfaction Approval Voting (SAV)

The way "satisfaction" is defined under SAV doesn't match how I feel when I am nominating for the Hugos. If one of my nominations wins, I am thrilled. Under SAV if I made 4 nominations in that category, I am supposed to feel only 25% satisfied. But my definition of satisfaction is not my winning percentage, it is the quality and diversity of the winners. In some categories such as Short Story, I have been thrilled with the final ballot even though none of them were what I nominated.

SAV rewards voters for making fewer nominations per category, because they receive a higher satisfaction score per nominee. I don't think that is a good thing. In the Hugos, voters who make more nominations in a category tend to be more knowledgable about that category (with the glaring exception of the slate voters).

SAV weights additional winners on a ballot just as strongly the first, so its anti-slate effect is weak.

In summary, SAV is not a good fit for the Hugo nominations. It maximizes an arbitrary definition of satisfaction that not what I (and I think other Worldcon members) actually want. It has a negative effect on knowledgeable voters, and it is not very effective against slate voting.

#409 ::: Brad from Sunnyvale ::: (view all by) ::: April 18, 2015, 06:39 PM:

#402 -- Sorry, the planet I was referring to where STV is low in strategy is this thread, and the world of Hugos in general. And I was referring to the broad class of preferential ballots. I don't contend they are the best in a broad sense, but one of their attributes when doing comparison is that preferential ballots reduce the value of strategy in a number of problems.

But I have not done much study of STV for multi-winner, but I am sure like other IRVs it has highly chaotic results from time to time. Chaotic results can engender strategy, though they tend to be "hard to predict" situations which in turn discourage strategy, if you don't know whether your strategy will help or hurt.

I think the classic situation I would describe as likely to cause too much strategic voting is the relatively common case where your first choice is not a shoo-in, and you feel it "needs your support" while other works you like somewhat less need it less.

I believe the goal is that you state your true opinion, which is to list the works you think best qualify for the award, and not to treat it as a contest where things need your support.

I can think of a few recent situations where this was true for me. Works like Super Sad True Love Story, The Quantum Thief, and in DP-Short, a few episodes of Black Mirror (including last year's White Christmas.) I even toyed in these cases with being strategic on Approval voting, but did not. In 2012, it actually would have worked on Approval -- leaving off Leviathan Wakes from my ballot would have given TQT a nomination, but I listed them both.

In RAV and other systems that reduce the effect if other choices when one wins, I would have considered leaving Embassytown offf the ballot to give more to these works of lesser known writers. That would not have made a difference. But a system should not encourage me to do it.

But let's bring this to a close. While I do think the proposed solutions are not adequate, their increase of strategy is just one of their flaws. They also downgrade honest and independent correlated opinion, and they don't keep all the slates off the ballot.

There are ways to keep all the slates off the ballot, such as option 4, which would not work all the time but generally should work. The slates are coordinated but not ultra-coordinated, which is to say I don't they they would easily collude a complex set of ballots to stay under certain types of algorithmic and human radar. Rather, I think the slate supporters simply see the virtue of voting the slate, knowing that they assure a set of choices they generally like a slot, vs. wasting their time with their own particular choices. In their internal survey, the puppies did not have much agreement.

If a "slate" only wishes to try for one nominee, it can succeed against most approaches here, except systems which inherently make everybody pool resources, colluding or not. This would include the open ballot proposals, they can stop even a single collusion candidate. A version of the open ballot, namely a "real runoff" as opposed to the instant runoff, would also destroy the slates.

In "real runoff" where you see the results of the first round and vote again, everybody changes their vote to not waste support on the works that don't have a chance, and support concentrates on your best alternative. IRV also does this, but doesn't give you a chance to step back and see the slate and decide on a more radical reallocation of your votes. In real-runoff, you would get political actions, such as withdrawing support for a work you liked because you see it's in a slate, and giving more support to a work that you otherwise only liked too mildly to rank, but which you would rather see there instead of "those bastards."

Of course, any system which reveals intermediate results is going to be more political.

#410 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 06:40 PM:

@406: "(And I have reservations about allowing more than five works per nominator)"

Why?

#411 ::: P J Evans ::: (view all by) ::: April 18, 2015, 06:54 PM:

410
Why do you think more is better in nominations?

#412 ::: TomB ::: (view all by) ::: April 18, 2015, 07:18 PM:

Option 3b: Proportional Approval Voting (PAV)

PAV fixes most of my concerns with SAV. Its definition of satisfaction more closely matches how I (and I think most people) feel: "Hey, one of my nominees made it!". It does not discriminate against knowledgeable nominators who fill a category. It is more effective against slates because it gives a lower weight to each additional winner on the same ballot.

The downside of PAV is computing it. If I understand the math correctly, given n total nominees for m, roughly n^m combinations need to be considered. (Actually: n * (n -1 ) * (n - 2) * ... * (n - m + 1)).

The upside is the certainty that all possible combinations have been evaluated. Other techniques such as Sequential PAV (aka RAV) are easier because they avoid evaluating all possible combinations. In theory, PAV is better. I don't know if it is enough better in practice to justify doing all that work. My guess is that it isn't worth it.

#413 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 07:19 PM:

@411: Because only with more non-slate votes, enough so that good non-slate candidates can get around 20%, can we ensure that slates don't get more than their fair share. Slates can always maximize their overlap, the only way to have that much honest overlap is if the average voter votes more than the current approx 3.5, and that means increasing the ceiling, because there will always be some who vote for few.

#414 ::: felice ::: (view all by) ::: April 18, 2015, 07:35 PM:

Jameson Quinn @410 & 413: "@411: Because only with more non-slate votes, enough so that good non-slate candidates can get around 20%, can we ensure that slates don't get more than their fair share. Slates can always maximize their overlap, the only way to have that much honest overlap is if the average voter votes more than the current approx 3.5, and that means increasing the ceiling, because there will always be some who vote for few."

A slate can nominate 20 works in each category as easily as 5; what would that do to the results? If the current average number of nominations is only 3.5, it seems likely that only a small proportion of genuine voters would nominate more than 5 if they could. This gives more power to a minority, rather than getting more people's opinions taken into account. I think Option 5b is a much more effective way of getting more nominations for good non-slate candidates.

#415 ::: TomB ::: (view all by) ::: April 18, 2015, 07:35 PM:

Option 3c: Reweighted Approval Voting (RAV)
Option 3d: Single Transferable Vote (STV)

This will be brief because I have dress up and go out now.

RAV is basically a better PAV. It looks good, especially with exponential weights because it would be very effective against slate voting and give more voters a chance to get at least one of their choices on the final ballot.

STV might be even better because the vote reallocation means voters are not punished for listing obviously great shoo-in candidates.

#416 ::: Cheradenine ::: (view all by) ::: April 18, 2015, 08:11 PM:

Felice @414: Slates don't benefit in the same way because their individual nominees don't pick up as many more votes from non-slate voters. They might have 20 nominees with 150 nominations apiece, but non-slate candidates would be more likely to pass 150. And under either RAV or SDV, the slate candidates are going to get winnowed down in the same fashion whether there are 5 or 20 of them.

A ballot with more than 5 nominations is not so much more powerful as more flexible. You're expanding the group of candidates you can help win, at the expense of selectivity among that group.

That said, my suspicion is that most people would list 5 or fewer, with a small portion of people listing 6-7 and a very small group listing any more than that. So while the voting system has better theoretical properties with no limit on the number of candidates one can list, I don't think that in practice it will make a huge difference. I'd favor increasing the maximum number of candidates listable, but it's not a hill I'd be inclined to die on.

#417 ::: Cheradenine ::: (view all by) ::: April 18, 2015, 08:13 PM:

It occurs to me that in combination with 5b, expanding the number of candidates would let someone add more popular candidates to their ballot, without sacrificing their original choices. I'm not sure that would have a big effect on results, but it might be psychologically preferable.

#418 ::: Cheradenine ::: (view all by) ::: April 18, 2015, 08:16 PM:

TomB @415: It kind of depends on how dramatic the number of votes for the shoo-in candidate is. If it's less than twice the quota, you'll get more vote transfer with RAV than with STV; if it's higher, the reverse.

#419 ::: TomB ::: (view all by) ::: April 19, 2015, 04:02 AM:

Back from the symphony. It was awesome, but I did manage to think some more about Hugo nomination voting procedures.

Option 3c: Reweighted Approval Voting (RAV)

The downside is it penalizes voting for obviously great shoo-in candidates. This is a dramatic change from the current FPTP system which rewards consensus candidates.

Option 3d: Single Transferable Vote (STV)

The "simplified version here", referenced in the main post, looks to me like RAV with a weight factor of 0 (or 1/∞). That makes it super effective against slates, but it maximizes the problem RAV has with penalizing votes for consensus winners.

The standard version of STV has the nice effect that if everyone votes for the same thing, it wins by a huge margin and there are a lot of leftover votes that are redistributed to the remaining candidates. Therefore there is a reward for backing a consensus winner. But if a candidate just barely wins, there are no leftover votes and there is no reward for backing it. This biases the system towards consensus and against slate candidates.

The downside of STV seems to be that it is only weakly anti-slate, unless the number of "votes" you get is less than the number of nominations you are allowed to make. In which case, try explaining that to the voters.

#420 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 05:11 AM:

@409:

First off: I still disagree with you, but I do want to thank you for backing off from saying that STV is "known to be" less strategic. Now that you're not saying that, what you're saying is reasonable, if in my opinion not quite right.

So, we're basically talking about the situation where you love A and kinda like the shoo-in E. You're right that in this situation, SDV-PE and RAV do encourage voting a bit more strategically than in STV. But the actual outcome is almost certainly the same, and if not, arguably better.

To see why, let's look at how that situation could play out in SDV-PE and STV.

SDV-PE: you're right that a strategic voter might be inclined to leave E off their ballot, voting something like ABCD or even ABCDFGHI. What are the possible outcomes?

1. A and E win: No harm no foul.
2. E wins, A doesn't: Unsuccessful strategy, not a concern.
3. E loses, A wins: If E was such a shoo-in, this seems unlikely. If it does happen, it's because E's support was lower than people thought. And if that's the case, is this outcome such a bad thing? If E's support seemed higher than it actually was, that's probably because it has high sales or has won other awards. In that case, giving A a chance in the sun is worthwhile.
4. E and A lose: again, if this is what the honest ballots lead to, why is this bad?

The only ways strategy can be harmful here, in my opinion, have to do with I:
4b. E and A lose, but I wins, because the voter voted ABCDFGHI: Sure, this is arguably a problem with strategy. It's also extremely unlikely.
1b. E and A win, but I, who could have also won, loses, because the voter voted just ABCD. In this case, whichever other candidate replaced I probably deserved it more. And again, it's very unlikely.

How would STV have come out? Generally speaking, you're right, the voter has no strategic reason to vote ABCD rather than ABCDE. So let's say they vote the latter. What happens if:
1. SDV-PE would have elected A and E: STV probably gives the same result. But in fact, there is some chance that STV eliminates either A or E due to weak first-choice support, even though they clearly have strong support when considering the ballot as a whole. The voter's ballot does nothing to help prevent E from being eliminated here.
SDV
1b. SDV-PE could have elected I with help from this ballot, but didn't. In STV, this ballot is never going to make it to I, even if the voter thought to vote ABCDEFGHI, so STV seems to be doing worse than SDV-PE if anything for this situation.
2. SDV-PE elected E but not A: STV gives same result.
3. SDV-PE elected A but not E: STV probably gives same result. Yes, this ballot included E, where it wouldn't have with SDV; but the vote was used up on A. In theory it is possible that the transfers from A put E over the top; in practice, it seems very unlikely. And again, arguably electing E in this scenario isn't even better than not doing so.
4. SDV-PE elected neither: STV does the same.
4b. SDV-PE elected I but not A or E: STV would probably do the same, though possibly not if the quota was set using a formula that didn't account for the long tail.

So yes, you're right: the voter could plausibly vote "more strategically" under SDV-PE. But if you actually look at the outcome of the election, it is very likely the same as under STV. When the outcomes differ, the SDV-PE outcome is either obviously or arguably a better one. And the SDV-PE "strategic" ballot is probably the semi-honest ABCD; personally, I think that semi-honest "strategy" is barely strategy at all, and certainly not worth worrying about.

The only way this could actually be a problem is if voters under SDV-PE started to recklessly over-apply this strategy, to the point of bullet voting. I find that highly unlikely.

Note that I've focused on SDV-PE here, but the same arguments all apply to RAV. The only difference is that RAV encourages this strategy a bit more strongly than SDV-PE does; but the outcomes of the strategy are still probably the same as the unstrategic STV result, and if not, arguably better. I might be convinced that misguidedly-strategic bullet voting could become significantly more of a problem under RAV than under SDV-PE, but frankly I still doubt it.

#421 ::: TomB ::: (view all by) ::: April 19, 2015, 05:16 AM:

Here is another option. It is an attempt to create a RAV/STV hybrid that has the best characteristics of each. I am not at all an expert at this stuff, so please excuse me if it is reinventing the wheel.

Option 3c-?: Popularity Reweighted Approval Voting (PRAV) — This system first nominates the candidate with the most votes. Then, all ballots featuring that candidate are “reweighted” so that votes on them are worth proportionally less. The weight is nc/n where nc is the number of votes for candidate c and n is the total number of ballots. (Or it could be the total number of non-blank ballots, or the total number of votes cast.)

For example, let's say the first candidate with the most votes is "The Dispossessed" with 1500 votes out of 2000 total. Therefore it is nominated. All the ballots that listed "The Dispossessed" are updated and a weight adjustment of 1500/2000 = .75 is applied to the remaining candidates listed on those ballots. The next candidate with the most votes is "Atlanta Nights" which was pushed by a slate of 400 voters. It gets nominated. All the ballots that listed "Atlanta Nights" are updated and a weight adjustment of 400/2000 = .2 is applied to the remaining candidates listed on those ballots. At this point there are four possible groups of ballots:

- 120 did not list D or AN: Weight is a pristine 1.0
- 1480 listed D: Weight = .75
- 380 listed AN: Weight = .2
- 20 listed both LHoD and AN: Weight = .15

Because the weight adjustment is based on the popularity of the work, slates that push unpopular works get adjusted downwards faster than RAV with exponential weighting, so they are unlikely to succeed at nominating more than one. Because the penalty is less for nominating popular consensus candidates, people can simply vote for what they prefer and don't have to vote strategically.

Computationally I think this is even simpler than regular RAV, because the same weight is applied to all the ballots that list the winning nominee. With regular RAV, the weight has to be determined for each individual ballot.

#422 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 05:24 AM:

@409: Aside from the technical arguments I just made above, I understand that you'd rather focus on options 4 and 5. I personally think 4 (outlawing slates) wouldn't solve the problem; the fallout from disqualifying ballots would be too high especially when both sides can make reasonable arguments that the other side is getting away with it, so the slates would be ever more of a problem. So while I don't oppose mild forms of option 4, I don't think a solution should hinge on it. As for 5 (publishing preliminary results such as a "long list"), it may help, but I still think that it's unwise to rely on this alone without also fixing the voting system (option 3).

Cheers. I apologize if the "what planet" comment was too harsh. When you're not claiming that your side is "known" to be right, your arguments are definitely a help to this conversation; I may disagree with you, but you are clearly thoughtful, reasonable, and sincere.

#423 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 05:32 AM:

@TomB: Interesting, but I think it has various problems.

Basically, you're trying to ensure that the "reweighting" factor depends on the popularity of the work. That is how unranked-STV already works. (Note that people here are using "STV" mostly to refer to ranked-STV, but unranked-STV has been referred to various times in this thread.) Unranked-STV, unlike your proposed system, is guaranteed to preserve proportionality. I suspect your system would deweight too harshly in practice, so that it's intended result of not discouraging votes for a popular candidate would not work, or even come out worse than RAV. Note also that SDV-PE was designed with much the same goal, though with different mechanisms.

#424 ::: J Thomas ::: (view all by) ::: April 19, 2015, 06:12 AM:

@420 Jameson Quinn

3. E loses, A wins: If E was such a shoo-in, this seems unlikely. If it does happen, it's because E's support was lower than people thought. And if that's the case, is this outcome such a bad thing? If E's support seemed higher than it actually was, that's probably because it has high sales or has won other awards. In that case, giving A a chance in the sun is worthwhile.

Is this one a concern?

People who want E think it will win without them, so they don't vote for it. If too many of them use this strategy then E loses even though the strategic voters wanted it to win.

They all thought they could depend on somebody else to do it. They were wrong.

But this is a stupid strategy. If what you want most is ABCDE and you instead vote ABCDF because you think E will win without your vote? You are voting for your sixth choice, hoping it will displace one of your five best choices. That doesn't make sense.

How about this one:

You vote ABCD because you think that if you vote E it makes it more likely that one of ABCD will lose. You want D to win more than you want E to win. The chance that your vote will help E is not worth the chance that it will hurt D (and C and B and A).

Imagine that the vote counting happened entirely in public and you were watching. You voted ABCDE, honestly. You watch while B loses, your second best choice. You gave 1/16 of a vote to help B. Then you watch C lose. You gave C 1/8 of a vote. Then you watch D lose. You gave D 1/4 of a vote. Then A loses. You gave A half a vote. And finally E wins, with your full vote.

Meanwhile your friend is standing beside you. He gave 1/8 of a vote for B, 1/4 of a vote for C, 1/2 of a vote for D, and a full vote for A. Nothing he voted for won, but he might easily feel better about his votes than you do.

You both might feel like you'd rather have the choice to give 1 vote for A, 1/2 vote for B, 1/4 vote for C, 1/8 vote for D, and 1/16 vote for E.

Or maybe 7/8 vote for a, 5/8 vote for B, 3/16 vote for C, 3/16 vote for D, and 1/16 vote for E.

But you don't get to choose. Some sort of mathematical process assigns your vote, and it gives the biggest part to the things that are already the most popular.

I would much prefer that we could honestly say "Your vote goes to the one on your list that *can* win, that needs it the most."

Or get away from the voting concept. "Your ballot will help to provide a well-rounded collection of works where the worst of them was suggested by many fans and the winning suggestions as a whole came from many fans."

If you just plain can't predict how your ballot will fit into the whole thing, then you might as well suggest a full slate of the ones you think are best.

#425 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 09:46 AM:

@424: I understand your point, but you get the details a bit wrong. Here's a corrected version, using SDV. Imagine that the text letter frequency times 4 (because the average voter approves 4 letters) are the raw totals, and the proportional results are as the dictionary letter frequency (second table), except that A has an unexpectedly weak showing — the 6th highest proportional total, below R and N. Thus, clearly there's a slate for THE.

Imagine that the vote counting happened entirely in public and you were watching. You voted ABCDE, honestly. You watch while B, your second best choice, comes up against H, and is eliminated by a crushing margin in raw ballots. You gave 1/5 of a vote to help B, but it's clear that even if you'd given more, H would have beaten it easily. Then you watch C lose to D, even though C had more proportional votes. Even if you'd not voted for D, D would easily have beaten C in raw totals. You could have given C 1/3 of a proportional vote over D, but the fact that that could have helped put C even further ahead of D wouldn't have mattered; in the contest that mattered, the elimination, C would have had your full support. After D squeaks past C and L on raw votes, you watch it lose to O. You gave D 1/3 of a vote in the proportional contest, and full support in the actual elimination. Then, A loses to T. You gave A half a vote, which helped put it ahead of T in proportions, but T had a higher raw vote, which you couldn't have changed even if you'd left E off. The final winners are EISR (proportionally) and T (total ballots). E was never in danger, and didn't really need your vote.

Meanwhile, your friend left E off. Thus, their vote counted 1/4 for B, (1/3 on both sides for C and D), 1/2 for D, and fully for A, in their respective proportional counts. But in none of those cases would changing the proportional count made one whit of difference to how the letter was eliminated, because the key in all cases was the margin. So, you're not upset by your non-strategic vote.

However, your strategic friend says, "Look! The dastardly THERS slate got four of their candidates on the ballot, even though they had only just over 30% of the vote, so they only deserved 1! If the RISEN voters had been as strategic as ABCDE voters like me, maybe we could have made it so the last elimination was between T and E, and so only one of them would have been on the final ballot! (That is, as long as there weren't too many ERIS voters sowing chaos.) I don't regret my strategy one bit!"

(Your friend is, of course, wrong about THERS being a slate. The real slate is just THE, which only got 2 of its candidates on the ballot. But that's still out of proportion to THE's numbers, so the rest of what she says is correct.)

Personally, I think that most Hugo voters would be more like you than like your friend. And even if they were like your friend, the result is only that they (for instance) leave E off and vote ABCD. And in that case, unstrategic STV would have gotten the same result anyway.

....

I would much prefer that we could honestly say "Your vote goes to the one on your list that *can* win, that needs it the most."

But with SDV-PE, we can say that. In the pairwise elimination comparisons, which is what actually counts, your vote does go to each of the works on your list when they need it the most (unless they are up against another work you supported).

#426 ::: J Thomas ::: (view all by) ::: April 19, 2015, 10:28 AM:

@400 Brad from Sunnyvale

And again, one point I have made which I am surprised I don't see any comment on. I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so.

There are few possible voting systems that can prevent this. Particularly if the colluders realize how the voting system works and respond to it.

The first obvious approach would be to throw out every pair of ballots that have all five votes the same. This will not affect regular voters at all, it isn't one in ten thousand that's identical to another. But the slaters would of course retreat to using four.

It would be hard to tell the difference between a slate that all voted for a single work, versus a bunch of people who independently voted for that work.

It would be hard to tell the difference between a slate that dishonestly voted for a single work because their group identity made them do it, versus a bunch of people who believed that single work was the one that deserved the Hugo.

And the argument could be made that some slates deserve a single win. If 10% of the voters want a particular work, nominated, there might not be five other works that get 10%. For 2013, the novel got 10.6%, the novella got 12.4%, but various awards did worse, plus the same slaters can hit smaller awards harder.

If we're going to go after 10% of the voters because of who they are, that's definitely politics.

If we find a voting system that gives us results we like better than what we have, it would make sense to change. It won't entirely solve the problem of slates.

What do you think the goals should be for the nominating survey? Whatever specific goals we choose will suggest an ideal voting system (which might be too cumbersome to use in reality) and we can rate other voting systems by how well they approach the goals.

#427 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 01:05 PM:

Brad@400:

I think the core of your point is: "I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so." If I understand you correctly, you're not objecting to a faction with 20% of the vote getting one slot, because in a situation where the non-slate voters are highly diverse, there's no voting system that could p prevent that. Nor are you objecting to the puppies becoming voters in the first place; while stricter membership qualifications may be desirable, this is not the thread for that discussion. What you object to, and rightly so, is that in an environment where even the second- and third-place non-slate candidates get around 10-15% at best, a second-rate slate candidate can elbow its way onto the ballot with well under 20% support, just by coordinating votes; and with 30%, a slate can hope for 2 or even 3 slots.

I agree, this is bad. Very bad.

There are several things that could be done to fight this. For instance, option 5 — announcing a preliminary "long list" — could help. But focusing only on voting systems, what's the best we can do?

If that's the question, I can see how you've convinced yourself that the answer is STV. You're worried about an honest ABCDE voter voting ABCD, because that leads to fewer non-slate votes, which lowers the hurdle for the slate voters. And you've figured that that kind of strategy is (generally) viable in systems like SDV-PE, but not in STV. All of that is correct.

But there's one thing you're missing. If there are enough voters who honestly put E fifth, so that their leaving off E would drop E out of the winning set and allow a slate candidate in, then STV won't elect E, even if voters honestly rank her fifth! The whole reason STV doesn't encourage a "leave popular candidates off of the bottom of your ballot" strategy is that it doesn't even see the bottom of your ballot until the higher choices have all been eliminated. So a few extra honest votes for E down at the bottom of people's ballots will not protect E from getting eliminated early on.

So that means that if SDV-PE only causes slightly more "truncation" strategy than STV, then the former will actually be better at fighting slates by finding the honest overlap between non-slate voters. In fact, even if truncation is used whenever it is strategically appropriate, SDV-PE will be as good on average as STV in this regard. It's only if truncation is significantly over-used, that SDV-PE becomes worse than STV.

Is there any reason to think that would be the case? I think, absolutely not! Consider:

- Adding further candidates to a ballot is more of a bother with STV, as you have to make sure to put everything in the right order, rather than just throwing it all out there. So STV could in principle actually lead to less honest overlap, as well as being less able to take advantage of the overlap that exists.

- Because truncation doesn't help with the pairwise comparisons, only with the points, SDV-PE only encourages truncation if you think that you can somehow get a rival eliminated before it comes up against one of your preferred candidates. Pretty much the only way that can happen is if the rival is a slate candidate, and your preferred candidate gets enough extra points to jump past another candidate on the same slate and into the top 4. It's pretty tough to imagine that an extra fraction of a point from your ballot is going to be enough to do that. So probably, truncation will be pretty rare in SDV-PE.

In other words: I think that if the goal is to take advantage of non-slate overlap to help defeat slates, SDV-PE is the best system we've discussed, RAV is pretty close behind, and I'd be very surprised if there were anything significantly better than these.

#428 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 03:30 PM:

My first problem with strategy is it makes the member not say the truth on their ballot, and leaves us the hope that the strategy is not likely to neither backfire (giving us a ballot without a choice that's actually popular) nor work (giving us a ballot where a work with rabid fans displaces a work supported by more, but less strategic fans.)

I think the only way to do that is the minimize the advantage of strategy. If you can make a case that it has little chance of affecting the result, you get a feedback loop, because people are less likely to do it. We can all paint stories of how one strategy attempt might work, another might backfire etc.

The best thing to do would be to get real sets of ballots and run tests of different systems on them, and also different systems in the presence of certain strategies operated by some fraction of the voters. Doing this a lot would give some hard evidence.

As to Option 4 -- I have not outlined here just how you would design a rule against slates, and the rules under which that would be judged. Bruce has requested not to do so at length here. It is not something hugo admins would relish, but then again, they don't relish what's happening now. I think there may be other systems of justice we can just import -- it probably isn't practical to invent a whole fannish one.

I do think the strongest effect of rule 4 might well be deterrence. If there is such a rule, it's hard to do a slate. It's probably not going to work, and in addition, you have to ask all your supporters to waste their ballots (and their membership fees, if that's the only reason they joined the con) on something that won't work. At "best" you can use it to claim victimhood, which they already do.

The systems above are perhaps best described as anti-sweep, not so much generally anti-collusion. They will prevent sweeps. It is a good thing to prevent sweeps, of course, but is it enough?

Anti-sweep does offer a deterrence against a classic slate. The slate might just say, "Here are the 2 slate candidates, be sure to nominate them, and if it doesn't hurt them, add your own."

Now I do agree that getting only a single candidate on is also a harder sell to the conspirators, but more rewarding than just getting grumbling points.

Of course, one of the first things I thought about was to see if you could get self-selection out of the process entirely. If nomination were done by randomly selecting 1,000-2,000 fans from a pool of 30,000 or so you could probably make it very difficult for a conspiracy to be that many of the sample. Though if they got to be 5% or more of the sample space (and thus the sample) they start having the ability to have an effect.

Could we get a sample of 30,000? If the sample were "Anybody who has ever attended any major convention, ever, for whom we still have the valid e-mail?" (Yeah, that has issues, like people who attended under different names getting 2 samples, and the fact that people will think it's unfair that only the first John Smith to respond gets to nominate, since it's way too expensive to try to tell them apart.)

I wonder what the sample size is for "People who have in the past, received a hugo voting PIN at any time?" Another alternative -- a smaller space -- is people who have ever voted in a past Hugos.

No supporting memberships clearly, that's highly self-selected. Sorry to those who never attend but regularly support. You are great fans but you are the definition of self-selected.

#429 ::: felice ::: (view all by) ::: April 19, 2015, 04:16 PM:

Brad from Sunnyvale @428: "If nomination were done by randomly selecting 1,000-2,000 fans"

What proportion of fans actually read a lot of what's published as soon as it comes out? I for one usually wait till things are out in mass market paperback, and a lot of my reading is catching up with older work I haven't read yet. And as for shorter fiction, the voter packet is virtually all I read some years. I think I'm a good voter, but I'd be a lousy nominator (in the absence of 5b). Self-selection of nominators isn't a bad thing.

#430 ::: JonW ::: (view all by) ::: April 19, 2015, 04:31 PM:

#428 - If Bruce has asked us not to discuss the design of a rule against slates, I've missed that. Bruce has asked us not to discuss options other than the five set out above, but a rule banning slates is one of those options. In my view, no version of a rule banning slates would be workable. It would either unduly trammel the ability of fans to communicate with each other (as in the rule "no fan may use the Web to bring SF works to the attention of other fans during Hugo nominating season"), or would be easy to evade (as in the rule "nobody may explicitly urge others to nominate particular works"), or would give too much discretion to administrators. If you have a version of a rule against slates that you think actually would work, you should suggest it.

#431 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 05:15 PM:

Ah. It depends what you mean by a rule. Unlike the mathematical formulations above, which are precise, rules are written in English and are expected to have interpretations, and humans make judgments. The rules have a spirit, as well as a letter, and the adjudicators can be permitted to enforce that spirit. And also told to do it very rarely.

You can also try to define the rules strictly. That permits loopholes. People find bugs in the rules and exploit them. This particular "bug" wasn't secret, everybody knew you could collude and take over the Hugos, it just wasn't felt anybody would be so slimy as to do so. That approach failed.

Here are a couple of quick possibles ways to structure the fannish inquisition.

a) A simple court of say 5, elected in some proportional system by fans. If puppies are 20% of fans they probably get a member. The court might be required to be unanimous, 4-1 or just 3-2.

b) A larger delegate group, perhaps of 50 or more, and a supermajority needed.

It might be controversial, but it could be the right structure is to give the group very broad powers, with everything made public at the end. Their duty: "If something has gone really wrong, fix it." Possibly as vague as that.

Possibly not as vague as that too. You could name specific wrongs -- collusion, voting by non-natural persons (already not allowed), buying memberships for others with an expectation of how they will vote. The more specific you are, the more possible it is a loophole can be found, but that might be OK -- it's not like we don't know what most of the loopholes are.

Now I know there will be debate about one issue. What's the status of holding a fancy semi-closed party at the worldcon and giving gifts and food to some members and not others to promote publications? There definitely are writers and publication houses that spend a lot of effort and money to build themselves up and become well known and liked in the community, and it does give them an advantage in voting, I would say. Not that much of one, and I doubt these parties are done with that motive -- but marketing does work, we are more likely to read and nominate books that have been well promoted to us, though one hopes most of us are not swayed in the final ballot.

I think we can come to a reasonable conclusion about whether that's OK, and it might be a different conclusion from whether people agreeing to vote the same to game the system is OK.

My primary role -- speech vs. action -- is blurry here, since giving away free drinks is an action, and saying on your blog, "here's my slate, I ask my supporters to vote for it" is speech.

(What's not speech is the supporter agreeing to that and voting the slate.)

But even so, I think it can be worked out. I would hope the fannish inquisition would be called upon almost never, and only would awaken if things are clearly very wrong. Perhaps it could not even meet unless 66% of fans asked it to meet.

In other words, one should not expect the fannish inquisition.


Unless you are planning something nasty, as it should deter you.

#432 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 05:27 PM:

Ok, writing off the cuff, let me outline a more specific system.

a) At any time, from before nominations to after the awards, any fan may propose a problem they see happening on the con web site.

b) Fans may, with their hugo pins, indicate they wish to see an inquisition into the matter proposed.

c) If any matter gets the support of X% of fans within a certain time frame, an inquisition is called on the matter. (How the members of the fannish inquisition are chosen is another issue. They might be pre-elected, or only elected then.) Two proposals may overtly declared to overlap -- only the one with greatest support is acted on.

d) The inquisition, of 5 members, may by 4-1 vote or better do just about anything related to the problem for which they were summoned. After discharging their duties, members are not eligible for another inquisition for some amount of time (a couple of years at least.)

e) Fans may invoke another inquisition (with different members) to appeal. But this should be extremely rare.

Now, getting 50% of fans would be a pretty major event. I don't believe 50% of the members of a worldcon have ever nominated or voted or attended a single event or any other fanac. 50% might even be too high, though it conveys a democratic fairness than any minority number would not.

I do suspect, that in this year, there would have been a call for an inquisition even before nominations closed but it would not have broken 50%. However, I think it might have after nominations were disclosed. I think it would have met, and corrected the ballot to remove the slate votes in a fair way, and the Hugos would have gone on. I think the puppies would have called for an appeal, and not gotten 50% for it.

Modest bit of work to code this, but not that large.

#433 ::: felice ::: (view all by) ::: April 19, 2015, 05:49 PM:

Brad from Sunnyvale @431: "Perhaps it could not even meet unless 66% of fans asked it to meet."

As a simpler alternative that doesn't require election of delegates, how about empowering the Hugo subcommittee to take actions not otherwise specified in the constitution, if 66% of voters endorse their proposed action? (With a "quorum" of 50% of the WorldCon membership?) The Hugo subcommittee could potentially be aware of a problem before everyone else, and in case like this, propose a special anti-slate measure before announcing the shortlist, describing the nature of the problem without specifying which works are involved. Other people could of course draw their attention to a problem if there's an issue that they aren't in the best place to spot first.

#434 ::: Keith "Kilo" Watt ::: (view all by) ::: April 19, 2015, 05:59 PM:

Brad@431:
Here are a couple of quick possibles ways to structure the fannish inquisition.

Heh, okay, that one got a chuckle out of me... Nice one. And the best thing about this type of rule is that No One Will Expect it!!!

Kilo

#435 ::: Joe in Australia ::: (view all by) ::: April 19, 2015, 10:25 PM:

I think "punishing" slates is too risky: the consequences of getting this wrong would be awful and delegitimise the vote. Also, how do you deal with non-malicious things that *resemble* slates, like nominations of local authors when the Worldcon is held outside North America? The problem of slates is that they exclude non-slate works; if we can't block slates the solution must be to give voters a more useful and genuine choice by including additional works.

I propose keeping the present nomination system, but having a final ballot that can be adjusted in length when necessary. The problem of reading all these nominated works will only be an issue when slates have distorted the results: the ballot will otherwise be at or around its present length. I suppose if that's a problem for anyone they can just ignore the works that were artificially thrust forward.

I favour the present nomination system because it's less burdensome for electoral officers: a ballot that's twice as long would take twice as long to count, but this can be done gradually as votes are received. In contrast, you can't start calculating some re-weighted voting systems (e.g., STV) until all votes are received. If people are impatient (if!) this means a bunch of people all working at the same time, which is harder to organise. Re-weighted voting systems are also trickier and more error-prone than simply enumerating the number of nominations for each work separately.

I think the easiest way to determine the length of the final ballot is to show people the ranked nominations and have an additional vote to determine the number of works they want on the final ballot. If you take the median response then the answer is effectively impossible to manipulate.

I understand that some people think that revealing the nominations will bias the voting process. I don't know whether we should make a big deal of this problem, but I would happily settle for alternatives like leaving it to the ConCom's discretion or using a formula to determine the length of the ballot. My only caveat is that any formula can be attacked: a smart attacker would design their slate(s) to minimise the number of alternative works that get included.

#436 ::: felice ::: (view all by) ::: April 19, 2015, 11:23 PM:

Another advantage to 5b - spotting ineligible works early, and giving potential nominees the opportunity to decline early, reducing the chance of the administrators having to deal with last-minute or after-announcement withdrawals. If a work in the top 15s is withdrawn for whatever reason, recalculate the results (by the same method that the final ballot would be recalculated with in the event of a late withdrawal) and publish the revised top 15s. Recalculation should only use the original half-way point data, not anything more recent, to prevent gaming the system.

#437 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 11:48 PM:

There is some merit to what you suggest -- allowing the Hugo committee to, if and only if they rule there has been attempted manipulation of the nominations, to expand the number of nominations so that exactly 5 nominations ruled as unlikely to be due to manipulation are on the ballot. They would not disclose this until after the ceremony. But if there are more than 6 it will be obvious what they did, but you would not learn from them which works they ruled against.

This works because the online community will certainly have a good idea of what the slates are, because it's very hard to run a slate in secret. As such, many fans may decide that they can not bother to read the works they regard as from slates but still be acting fairly, and so the burden on the fan to read the whole ballot is not expanded by this expansion.

The only negative is that the slate voters do get to call themselves Hugo Nominees, and if you get multiple slates it could get a bit crazy. Sadly, even though it would not do much good for them, I could see the puppies (and others) continue to run slates in order to provide publicity (good or bad) for their works and cause, and to be able to call themselves Hugo finalists.

However, it has a lot to recommended it because the committee's actions can not truly harm anybody, not with a preferential ballot. Having extra nominees is only harmful because it puts a burden on the honest fan who wants to read them all and be fair in judgment. It is "harmful" in that you might lose your Hugo to a work that, once fans saw it on the ballot, they realized was better than yours. That's not harm at all.

As for things that "resemble" slates, I don't see that as too likely. A slate is a large conspiracy. You can't keep it secret. It will almost always involve public declaration of the slate, and if it's done in private, all it takes is one person who got it to make it public or forward it to the Hugo admins.

Most of my proposals for a fannish inquisition have had it do all the work, most of the proposals here have involved complex rule changes. The right answer may well be a combination -- minor rule changes that give the Hugo admins the tools to fix the problem with minimal harm.

At least until the attackers come up with something that was not anticipated by those rules -- thus my idea of 50% of fans being able to invoke a Fannish Inquisition that can fix almost anything.

#438 ::: J Thomas ::: (view all by) ::: April 20, 2015, 07:05 AM:

A slate is a large conspiracy. You can't keep it secret. It will almost always involve public declaration of the slate, and if it's done in private, all it takes is one person who got it to make it public or forward it to the Hugo admins.

Can that be gamed?

Somebody sends the Hugo committee a document they claim somebody sent them, with two names they recommend voting for. They say this list was sent to a lot of people. Both names show up on a lot of early ballots, two popular works that appeal to a certain sort of person, the sort of social undesirable that might want to do a slate. It looks like one or both of them might place on the final ballot.

Maybe two people send them the same document.

So they don't accuse anybody of anything, they instead add one or two slots to the short list.

Adding slots doesn't help a slate very much, if it only had one or two on the short list in the first place. Say you started with two slate nominations and three others. Instead you have two slate nominations and five others. The slate still gets all its votes for either the one item the slate has agreed to vote for, or for two. Everybody else (who presumably dislike the slate nominations) have five others to split their votes among instead of three.

It isn't a big help to a slate, but maybe enough that members of an actual slate might want to reveal it to get that result.

The good side is the slate doesn't get crowd out other works from the short list. But you might have other ways to prevent that.

Would anybody try to game it the other way? Create the illusion of a slate to get more things on the ballot? That doesn't seem plausible to me at first thought. It's a big effort that could possibly backfire, and the result is you get 2 extra chances to be nominated, but there's no reason to think the one you want would come in 6th or 7th. The votes are split among more choices which could help or hurt a particular works' chance of winning. I don't see how it does much good to fool the Hugo Committee into believing in fake slates.

#439 ::: Brad from Sunnyvale ::: (view all by) ::: April 20, 2015, 10:57 AM:

If the slate is not obvious, I doubt the committee would act. And they would be on the lookout for fakes.

Now, if there are more than 5 slate candidates, the committee need only put 10 (the top 5 slates and the top 5 non slates) but I hope that would not happen.

#440 ::: JonW ::: (view all by) ::: April 20, 2015, 11:00 AM:

Brad @# 431 & 432,

As I understand you, you’re answering the “what would be forbidden” under a rule against slates and “what would Worldcon be authorized to do in response” questions with “anything the Inquisition thinks is Really Bad” and “anything the Inquisition wants.” This strikes me as a terrible idea.
First, if we’re giving Worldcon authority to act arbitrarily against certain of its members just because folks think (on a non-rule-constrained, I-know-it-when-I-see-it basis) that the members have done Bad Stuff, then those guys will complain that they’ve been treated arbitrarily, and they’ll be right. If we want people to feel that they’re being treated fairly, we need to tell them the rules in advance rather than telling them that there are no rules, but they’re at risk whenever the Inquisition decides after the fact that they’re done wrong.
Second, it’s only human nature that the Inquisition will be inclined to cut slack to well-respected members of the community, and disinclined to do so for jerks like the Puppies. So we’ll be moving from a system that at least has the virtue of rule-bound fairness (something the vote-tallying proposals now on the table would retain) to one that would actually provide basis for the Puppies’ complaints about the in-group excluding and arbitrarily impeding outsiders.
Finally, if it’s unclear what sort of stuff will get Inquisitory attention, a lot of people will be inclined to steer clear of those lines. But that’s not an unambiguously good thing, because what they’ll be inclined to steer clear of is promoting books, which is an ill-defined subset of talking about books – and talking about books is a feature, not a bug. So if people have reason to worry about being Inquisitioned for recommending books for Hugo consideration, we want to be really thoughtful about what we're chilling them from doing.

#441 ::: JonW ::: (view all by) ::: April 20, 2015, 11:02 AM:

Brad @# 431 & 432,

As I understand you, you’re answering the “what would be forbidden” under a rule against slates and “what would Worldcon be authorized to do in response” questions with “anything the Inquisition thinks is Really Bad” and “anything the Inquisition wants.” This strikes me as a terrible idea.
First, if we’re giving Worldcon authority to act arbitrarily against certain of its members just because folks think (on a non-rule-constrained, I-know-it-when-I-see-it basis) that the members have done Bad Stuff, then those guys will complain that they’ve been treated arbitrarily, and they’ll be right. If we want people to feel that they’re being treated fairly, we need to tell them the rules in advance rather than telling them that there are no rules but they’re at risk whenever the Inquisition decides after the fact that they’re done wrong.
Second, it’s only human nature that the Inquisition will be inclined to cut slack to well-respected members of the community, and disinclined to do so for jerks like the Puppies. So we’ll be moving from a system that at least has the virtue of rule-bound fairness (something the vote-tallying proposals now on the table would retain) to one that would actually provide basis for the Puppies’ complaints about the in-group excluding and arbitrarily impeding outsiders.
Finally, if it’s unclear what sort of stuff will get Inquisitory attention, a lot of people will be inclined to steer clear of those lines. But that’s not an unambiguously good thing, because what they’ll be inclined to steer clear of is promoting books, which is an ill-defined subset of talking about books – and talking about books is a feature, not a bug. So if people have reason to worry about being Inquisitioned for recommending books for Hugo consideration, we want to be really thoughtful about what we're chilling them from doing.

#442 ::: JonW ::: (view all by) ::: April 20, 2015, 11:03 AM:

Damn. Internal Server Error strikes again. Sorry.

#443 ::: J Thomas ::: (view all by) ::: April 20, 2015, 12:04 PM:

@441 JonW

Everything you say here is reasonablem but let me briefly take the devil's advocate position.

When we are at war we must assume that any rules we impose on ourselves will be used by the enemy in their attempts to hurt us. We are the good guys and they are the bad guys, and we must not have any rules of war against bad guys.

Since we are the good guys and they are the bad guys, anything we do to them is justified while nothing they do to us is justified.

They will complain that we are treating them arbitrarily, but they will do that regardless so what rewards do we get to do otherwise?

Anyway, if you are in a position of authority and you treat your friends no better than your enemies, soon you will have no friends.

Taking the bigger picture, there are two sides here, Sad Puppies and Social Justice Warriors. Neither wants to tolerate the other. They want a war. If you try to come between them and make their war less likely, both of them will consider you an enemy. But if you take one side and push for war, the others on your side will like you a lot and the other side will develop a grudging respect for you.

The more the hate builds up the better your position. Some day in the distant future your side may win and then you will get fading appreciation for old victories, but while it rages, the more it rages the more important you will be.

The SP side has done something outrageous to get their enemies arrayed against them. Now it's our turn to do something outrageous to help their side get stronger, so it can turn into a bigger and more glorious conflict.

Why would you get in the way of all that? Is this the hill you want to die on?

#444 ::: JonW ::: (view all by) ::: April 20, 2015, 12:14 PM:

J Thomas @#443,
I don’t think you actually believe any of that, and I think you and I can agree that none of the sentiments expressed in your post are a path down which we want to go.

#445 ::: albatross ::: (view all by) ::: April 20, 2015, 12:15 PM:

Let me be the first to announce my definitely-not-a-slate list of SFF works which I heartily recommend reading and nominating if you find them worthwhile. With exactly the same words on my part, this will or won't be a slate depending on:

a. What various nominators do with my suggestions
b. What you started out expecting of me w.r.t. slate voting
c. How much you dislike my choices or politics or whatever else.

I think the natural points to get to w.r.t. an explicit rule against slates are either:

a. Nobody has a slate, because everyone carefully puts the right not-a-slate wording in their webpages. The SPs and RPs and various other Ps now have hearty reading recommendations rather than a slate. Nothing changes.

b. Deciding who has a slate turns on your judgment of who's basically up to no good and who's not. The resulting judgment calls are probably almost impossible to do without pissing off a big chunk of fandom, and they will inevitably be based partly on the prior opinions of the people making the judgments. People on the other end of those judgments will basically never be convinced they weren't just attempts to fix the rules to prevent the wrong kind of works being nominated.

#446 ::: J Thomas ::: (view all by) ::: April 20, 2015, 12:25 PM:

@444 JonW

Agreed, I was taking a devil's advocate position.

However, I think that what I said is definitely a big influence in a lot of people's back-brains.

There isn't a lot of wiggle-room between trying to live in peace and trying to start a war. That middle ground is real unstabl.

Holding to that ground, I say that I don't want a minority slate to get all or most of the Hugo nominations just because it's better-organized. On the other hand if it's 10%-20% of the ballots, it probably does deserve one win. If we decide it can't have any because it's a slate, that is going to seem way unfair to 10% to 20% of the voters and all their friends.

A combination of a better voting system and a mid-time announcement of the ones that have a chance, is probably enough to get that result on average. It is a move against the SPs, but it is not an intemperate move against them -- they got more than they deserve and we scale it back to what they deserve.

I don't think we need more at this point.

#447 ::: Keith "Kilo" Watt ::: (view all by) ::: April 20, 2015, 11:32 PM:

Hi all -

I've just finished coding a simulator for the SDV-PE nomination system. The code will read in a text file of works (called, creatively enough, "works.txt"), one per line. My sample just has the alphabet:
A
B
C
D
E
F
G
H

...etc., but you can use actual titles if you are using historical Hugo data. It's not necessary that all of the works actually get any nominations (the code will eliminate them in the first round, since they won't get any points).

The code will then read if a comma-delimited text file of nomination ballots (again, stretching my creativity here, called "ballots.txt"). Right now I'm assuming there are five nominations per ballot, but the code can (in theory) handle any number of nominations up to whatever maximum we decide. Any blanks are just listed as "none" (any invalid nominations -- i.e., nominations not appearing on the works list -- will automatically be changed to "none", so in practice it shouldn't matter what the blanks are called).

So, for example, I ran the following:

A,B,C,D,E
A,B,C,D,E
A,B,C,D,E
A,B,C,D,E
A,B,C,D,E
A,F,G,H,I
B,G,H,I,J
C,H,I,J,K
D,I,J,K,none
E,J,K,none,none
K,none,none,none,none

This is obviously an extreme case with half the ballots coming from a slate. Interestingly, the slate works ended up getting the least number of points, but since they all had six nominations, they were all tied for elimination in what would end up as the last round. Since removing them all would put the ballot at less than five works, the process was stopped and the remaining works were put on the final ballot (we had decided, you'll recall, to eliminate all ties, unless that would reduce the final ballot to less than five works). The winners were:

A,B,C,D,E,I,K

If any one of the slate votes had not gotten an "external" nomination, they would have been eliminated.

So! I need your help. I need to test the code. Are there any situations that you would like me to run? Just send me the ballots in the format above (I'll assume you're using A->Z for the works unless you say otherwise). I'll run the code and post the results (including the round-by-round results, if desired), and maybe our experts can see if the results look reasonable. In particular, Cheradenine, if we could run it against the cases you've already tested, that would give us some confidence that the code works. If it looks good, we can in theory thoroughly investigate the properties of SDV-PE with some hard numbers. Here's a chance to test any situation that's been bothering you!

There are undoubtedly bugs (there always are!), but hopefully with some test data we can squash them quickly.

Thanks in advance for the help!

Kilo

#448 ::: P J Evans ::: (view all by) ::: April 21, 2015, 12:02 AM:

447
I did something similar, but I had the file tell it how many nominees it would be reading in that category (so that input file had category name, nominee count, list of nominees - I believe in letting the computer do as much of the work as possible).

#449 ::: Keith "Kilo" Watt ::: (view all by) ::: April 21, 2015, 12:20 AM:

PJ@448:

I agree, let the computer do the work! The code reads in the files line-by-line, resizing the dynamic array the data is stored in as needed. The total number of entries is just the length of the array. I don't see any reason why the code as written couldn't handle any number of nominations, I just haven't tested it. I love it if you could post your input data so I can see if I get the same thing you do.

I just added a formatted "results.txt" file that contains the round-by-round results, as well.

Kilo

#450 ::: J Thomas ::: (view all by) ::: April 21, 2015, 01:20 AM:

Here's a preliminary attempt at a routine to generate ballots, in python 2.75.

It assumes that the ballots fit a modified zero-truncated poisson distribution.

Here is my reasoning: Assume that votes are not correlated, that just because a fan likes one thing has no influence over whether he also likes something else.

Then according to how popular things are, there is some probability he will pick work #1, and if he doesn't do that, there is some probability he will pick work #2. If he doesn't pick either of the first two there is some probability he will pick work #3, and so on. But at some point it gets too ridiculous and if he hasn't picked one of the top N he'll leave a line blank rather than pick the N+1st one.

If those probabilities were all the same, it would be a zero-truncated poisson. But I figure as we go from more to less popular, the probabilities should get lower. The less popular something is, the less likely it will be what he picks if he doesn't pick anything *more* popular.

So I modeled that, and found parameters that seemed to vaguely fit the 2013 novel nominations. It isn't done right because I thought about independent votes and then actually worked on ballots with up to 5 votes. But it sort of vaguely fits the shape of the curve.

It isn't bad for 20 minutes effort.

If you use this to generate random ballots that reflect the votes that aren't slates, you'll get something that's kind of like the real ones and you can have as many of them as you like while you look for unexpected results and aberrations.

You can't use this to estimate probabilities of things happening with real Hugo nominations, but if something happens pretty often with this, it's likely to be fairly common in reality too.

http://notes.io/YJ8

#451 ::: Jameson Quinn ::: (view all by) ::: April 21, 2015, 07:53 AM:

@450 What you're looking for is an Indian Buffet Process. I have an idea for a heirarchical buffet that is to a buffet as a buffet is to a Chinese Restaurant, but it doesn't fit in this margin.

#452 ::: David Harmon ::: (view all by) ::: April 21, 2015, 08:11 AM:

Popping in late and responding to the last few dozen messages:

The basic problem with trying to set up the "fannish inquisition", is legitimizing it. ISTM that merely getting 50% of "fans" (that is, Worldcon registrants) -- even 30%, perhaps 20% -- to respond promptly to a call for action, is itself implausible. There are times when we're tempted to think of "The Internet" as a single big room, where we can yell and get everyone's attention. But it's nothing of the sort -- not only does a broadcast depend on an assortment of relays and pass-alongs, but not everyone is "on the net" all that regularly, and they won't necessarily be watching the Hugo hotline for developments. E-mail itself is an increasingly secondary medium (thanks, spammers), and even roping in other channels... well, a lot of folks still live basically all of their lives out in "meatspace", only occasionally poking their noses into the 'nets.

On a separate line, any committee is necessarily composed of humans, and thus any attempt to judge the motivation of voters will be subject to bias. Accordingly, the committee and the system itself really need to be limited to reponding to behavior, as opposed to motivation -- and preferably, limited to behavior within the system. That's going to lead to a lot of "limited restraints", where bad actors can still get something, but only so much.

As a simple example, (and this may well have been covered before, apologies if so) the previously-suggested rule that one person can only win nomination in 1 or 2 slots (with corporate entities also blocking their officers and vice versa) would by itself have squelched a fair chunk of the problems in this year's ballot. Not all of them, but part, and that's something. The problem of one author's multiple works splitting the vote, might be handled by allowing "Best Short Story", and perhaps other "Best <works>", to pool same-type works by a common author, automatically shifting from a "single-work" to a "personal/plural" award.

#453 ::: J Thomas ::: (view all by) ::: April 21, 2015, 09:50 AM:

@451 Jameson Quinn

What you're looking for is an Indian Buffet Process. I have an idea for a heirarchical buffet that is to a buffet as a buffet is to a Chinese Restaurant, but it doesn't fit in this margin.

Model first, categorize models later.

What I have is quick enough at our sample sizes that I don't need to use theory to find something more efficient. It can stand in for a better model until I need a better model.

The flaw I found in my model is that if the average nominator has a 1/6 chance to choose the most popular work on his first vote, and he doesn't, then the chance he will choose the next-most-popular vote is less, let's call it 1/8. But if he does choose the most popular on his first chance, then the chance he picks the 2nd-most-popular work on his second vote goes up to 1/6, and if he doesn't choose that then the chance for the third-most-popular goes up to 1/8, etc.

I had no theoretical basis for choosing how fast the 1/6 1/8 etc rate goes down, and I had already adjusted it to look pretty good by eyeball before I noticed the flaw which I then did not fix.

It occurs to me that it might be faster to fix it than to explain why I didn't, but then again it might not. If something went wrong and I spent 2 days solving some fascinating problem that doesn't help me achieve my goals, I'd be 2 days behind.

I figure, get it good enough to use and use it for the next step. If I then find a bad assumption that means I can't possibly do what I wanted to, or if the results come out completely boring, I haven't wasted as much time getting set up. If it's worth getting right (and particularly if it gets funding), then go back and make sure about all the details.

#454 ::: Jameson Quinn ::: (view all by) ::: April 21, 2015, 12:30 PM:

Don't reinvent the wheel, at least not before you've looked at prior wheels: https://cocosci.berkeley.edu/tom/papers/indianbuffet.pdf

#455 ::: J Thomas ::: (view all by) ::: April 21, 2015, 12:49 PM:

When you're doing simple preliminary work, use whatever works. The real world is likely not to exactly fit prior work, and it might be worth shoe-horning it into place -- later.

#456 ::: Steve Halter ::: (view all by) ::: April 22, 2015, 04:11 PM:

Jameson Quinn@454:Interesting paper--thanks. I'll have to mull that a bit.

#457 ::: Keith "Kilo" Watt ::: (view all by) ::: April 22, 2015, 10:38 PM:

Okay gang, we seem to have settled down to a nomination system, since discussion seems to have died off. I'd like to propose a draft language to be presented at the business meeting. It includes FAQ's and commentary, so it's a bit long -- I'll post it in the next message. One thing: I suggest we change the name of the system to "Single Divisible Vote with Least Popular Eliminated" or SDV-LPE. I do understand "Popularity Elimination" is intended to mean the same thing, but on first reading it sounds like the -most- popular work is being eliminated. I think the clearer we make things, the easier it will go.

Please keep in mind this is very rough draft with items pulled from this thread and the original thread, then edited into a cohesive whole. Hopefully it will serve as a start for discussion.

Thanks,
Kilo

#458 ::: Keith "Kilo" Watt ::: (view all by) ::: April 22, 2015, 10:45 PM:

A Proposed Revision to the Hugo Award Nomination System

The Worldcon Constitution shall be modified as follows:

3.3.1 The guiding principles of the Hugo Award are as follows:

3.3.1.1 The Hugo Award should reflect the opinion of Worldcon members that the winners are the most-enjoyed works of the previous year.

3.3.1.2 The Hugo nominations are a survey, to learn from fans what they think are the great works of the year that should be contenders for the Hugo award.

3.3.1.2 All Hugo voters should have an equal opportunity to suggest great works. No minority should be able to absolutely prevent any other set of nominators from having their acclaimed works considered for nomination.


3.7.4 It is the intent of the Hugo Award nomination process that Worldcon members will only nominate works that they personally have read and enjoyed, and will not allow their nominations to be co-opted by any other individual or group.


3.8.1 Worldcon members may nominate any number of works for each category, up to the limit specified in 3.7.1. Each Worldcon member gets one nomination “vote”, which is divided equally among all nominations on his or her ballot. For example, if only one work is nominated in the category, that nomination gets a full “vote”. If two works are nominated in a category, each nomination gets 1/2 of a vote. If three works are nominated in a category, each nomination gets 1/3 of a vote, etc.

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the two works getting the least total number of votes will be compared. The work which appears on the fewest number of nominations ballots shall be eliminated.

3.8.1.2 The eliminated work is removed from all ballots. For the next round, votes are assigned to each work as in 3.8.1. Note that if a ballot contained, for example, five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now get 1/4 of a vote for the current round.

3.8.1.3 The least popular works are eliminated until only five works remain in the category.

3.8.1.4 In the event that more than two or more works are tied for the lowest number of votes received in 3.8.1.1, all tied works will be compared to see which appears on the fewest number of ballots.

3.8.1.5 In the event that two or more works are tied for appearing on the fewest number of ballots, all the tied works will be eliminated, unless eliminating all the tied works would bring the total number of works on the final ballot to be less than five. In this case, all the tied works will be retained, and the final ballot will be extended to include more than five works, as required.


Commentary:
1. The system is very simple and intuitive. It has two major components:
a. The least popular work is eliminated each round.
b. Nominators may decide how strongly they feel a given work or works should be on the final ballot. If there is one work they feel very strongly about, they can give that work their entire nomination vote. If there are a number of works they feel are Hugo-quality, they can nominate as many as they like, while keeping in mind that because they don’t feel as strongly about any one of them, they are spreading their one nomination vote among the works they have nominated.

2. Nominator instructions are equally simple:
a. Nominate whichever works you feel are Hugo worthy, up to the maximum permitted.
b. You have one nomination “vote” for each category, however, you may choose to divide your nomination vote among more works by simply listing more works on your nomination ballot. Your nomination vote will be split equally among all the works you recommend.
c. It is not necessary to nominate the entire maximum number of works.
d. If you feel very strongly that a particular work should be nominated, you can choose to put the entire weight of your entire nomination vote behind that work. If you have several works you feel are worthy, you can choose to recommend each one slightly less strongly.

3. The guiding principles in recommending this system are:
a. Hugo voters should each be able to suggest multiple great works, up to the maximum indicated in 3.7.1.
b. The works selected for the final ballot should be works which are popular among Worldcon members.
c. The nomination system should select a group of works that together represent as large a set of nominators as possible.
d. The nomination system should not select too large a group of works.
e. Other things equal, works suggested by more fans should be chosen over works suggested by fewer fans.
f. Other things equal, two works suggested by different fans should be chosen over two works suggested by the same fans. Other things equal, between two candidate final ballots, the one that includes suggestions from more fans should be chosen.
g. To the extent possible, the nominating system should not encourage “strategic” suggestions. The preference is that a fan should think “What are the most excellent works I have experienced” and not “How can I game the system to make sure the ones I most want will win”.
h. If a nominee declines at the last minute, it should not cause undue disruption of the process.

4. A wide variety of nomination systems were considered, drawing from the experience of experts in the field of election theory as well as from experienced members of fandom. The system chosen here, formally called “Single Divisible Vote with Least Popular Eliminated” or SDV-LPE, was found to be the system which best fits all of the desired properties listed above. The system is both mathematically and intuitively rigorous and robust.

FAQ’s
1. How does this system eliminate slate or bloc voting?

It doesn't, exactly, nor should a work be automatically eliminated just because it appears on a slate. On the other hand, any slate which nominates a full set of five works will find that each of its nominations only count 1/5 as much. With a large enough support behind the slate (five times as much), the slate may still sweep a category; however, if that many voters support the slate, they arguably deserve to win, and no fair and unbiased system of nomination will prevent that. The answer is, simply, to increase the general pool of voters. Regardless, with SDV-LPE, slates will never receive a disproportionate share of the final ballot, as occurred in the 2015 Hugos.

2. What if there are multiple slates (slate wars, “parties”, etc.)?

As with a single slate, the more works that anyone nominates, the less their votes count for each work. The end result is that even multiple slates are unable to sweep the nominations.

3. What happens if a genuinely popular work is nominated by a group of unrelated people?

If it is genuinely popular, the system will still select that work for the final ballot.

4. What happens if a genuinely popular work also appears on a slate?

Even if it is on a slate, if the work garners support from individuals – particularly if they list it as their only nomination, or with just a few nominations – then the system will select that work for the final ballot.

5. Isn't it true that any voting system can be gamed (or strategized, etc.)?

Yes, there is a theorem which proves that all voting systems must have inherent flaws. The objective is to choose a system whose flaws are not in an area of concern to the electorate.

6. What are SDV-LPE’s flaws?

A significant one is that ties may occur somewhat more easily than with the current system. The result of that is that it is possible that a category could have as many as 7 or 8 works on the final ballot. This still isn't common, but it is more likely than it is now. This is considered to be a worthwhile trade-off for SDV-LPE’s benefits.

7. What are SDV-LPE’s benefits?

Simply put, it reduces the power of bloc voting without eliminating the chance that works appearing on slates will make it to the final ballot. Conversely, it makes it very difficult for slates to prevent non-slate works from appearing on the ballot.

8. Couldn't slates just recommend a single work for a candidate, and it will automatically appear on the final ballot?

Yes, that is certainly a viable possibility – it’s also completely fair. It does not force any other works off of the ballot, and the final Hugo winner is determined by the same voting process we have always had. Just appearing on the ballot isn't a guarantee of winning a ballot. However, if a large section of fandom strongly believes that a work deserves a Hugo nomination, then it should, in fact, be represented on the ballot.

9. What happens with a large field with no stand-out favorites when a slate votes?

Even in this case, we were unable to find a simulation in which no non-slate works appeared on the final ballot. Slate works did receive a larger proportion of nomination slots than they did otherwise, however, again, this could be considered a fair and valid result. If there was no general favorite, then voters really had no collective preference.

10. Isn't this system too complicated for the average voter to understand?

No, it’s actually quite simple and straightforward, both in terms of voter instructions and in how the system operates. Essentially, the total number of votes for each work are totaled (and this will usually be the sum of factional votes). These votes are used to determine our two candidates for elimination, since the voters felt the least strongly about them. We then look at the number times the two works appeared on any ballot. The work that appeared the least number of times must be the least popular of the two, so is eliminated. This process continues until the five finalists remain.

11. I think we should just increase the number of nomination slots on the final ballot to (for example 6), and decrease the number of slots a voter can vote for to a smaller number (for example, 4). Wouldn't that be simpler and easier?

Unfortunately, this simply means that the largest slate will receive four of the nominations and the next largest will receive the remaining two. It doesn't solve the problem of forcing works off the ballot that had a chance to win the final election.

12. I think we should set up a committee to handle these situations as they occur. The committee would be empowered to add nomination slots or throw out slate-influenced ballots as required.

This could work. The problem is that now you have a small group of people who serve as literal gatekeepers to the Hugo Awards. In spite of the word on the Internet, this has never been the case in the past. Establishing it now means that those groups who believed it existed in the past will now be correct. Ultimately, human judgement is fallible. The fairness of a committee’s decisions will forever be subject to opinion. The end result is that the prestige of the Hugo Awards will forever be tarnished.

13. I think we should use [insert other mathematical voting system].

We considered essentially every type of voting system currently in the literature, guided by two experts in the field. Some of these systems do in fact have positive properties that speak for them. None of them were as simple or as intuitive as SDV-LPE, yet SDV-LPE meets all of the stated goals for a Hugo nomination system.

14. Won’t SDV-LPE be complicated to code and implement?

Actually, no. One of our non-experts coded a full simulator for the system in a matter of days. A full web-based app would not be much more difficult to handle.

15. Wasn't this system just designed by Social Justice Warriors to block the Good Stuff?

It is true that much of the discussion for this system occurred on Teresa and Patrick Nielsen Hayden’s “Making Light” discussion board, and it is also true that groups such as the Sad Puppies and the Rabid Puppies consider TNH and PNH to be The Enemy, and therefore completely biased and not to be trusted. TNH and PNH had absolutely no input in the discussions, however. Those of us who worked on the system were very clear that our goal was not to keep the Sad/Rabid Puppies off of the Hugo ballot and that any system which specifically targets any type of work is inherently wrong and unfair. One of the members of the group is a retired US Naval officer, a combat veteran, a certified Navy marksman, a Christian, and considers Robert Heinlein to be the greatest science fiction author who has ever lived. In short, he is exactly the Puppies’ demographic. But any slate, of any sort, be it a Sad Puppy or a Happy Kitten of Social Justice, breaks the Hugo Award because a small percentage of voters can effectively prevent any other work from appearing on the final ballot. This is a major flaw in the Hugo nomination system, and it is a flaw that must be fixed if the integrity of the award is to be maintained. Politics should play no role whatsoever in whether a work is Hugo-worthy or not.

#459 ::: Keith "Kilo" Watt ::: (view all by) ::: April 22, 2015, 10:49 PM:

Okay, it seems to have formatted reasonably well. Please let me know what you think, particularly errors and misconceptions I have (if I have them after all the research I've been doing, chances are other non-experts will have them, too). We may need more simulations of different cases, as well.

Looking forward to hearing your thoughts!

Kilo

#460 ::: David Wallace ::: (view all by) ::: April 22, 2015, 11:35 PM:

Keith@458: A couple of thoughts. First, I thought the conclusion above was to call the fractional "votes" points rather than votes, so a ballot for ABCDE would award 0.2 points to each work.

Second, I have a question about the SDV-LPE system: suppose there are 26 works being nominated, A-Z, and a slate of 200 voters is going for VWXYZ. All the other voters are going for various combinations of works from A-U, but no single work is selected by more than 190 voters (and in fact, most of the works are pretty close to being mentioned on 190 ballots). As I understand the way the system works, that means in any paired elimination from 3.8.1.1, any work from V-Z will beat any work from A-U. So it looks to me like the slate will sweep (in spite of not getting lots of points in the later rounds) unless two slate works face off against each other in one or more of the paired eliminations. Can someone simulate this a few times to see how many slate works survive?

#461 ::: David Wallace ::: (view all by) ::: April 22, 2015, 11:44 PM:

Followup to my @460: Actually, under my exact scenario, the slate has a pretty good chance of being eliminated entirely due to the tie provision of 3.8.1.5. So instead, assume that the slate gives a different number of votes between 195-205 to each of its 5 works, and simulate that case.

#462 ::: felice ::: (view all by) ::: April 23, 2015, 01:14 AM:

Keith "Kilo" Watt @458: "3.3.1 The guiding principles of the Hugo Award are as follows:"

I'd suggest moving that section to the commentary, rather than including it in the rules. It's not a substantive change, and I suspect is more likely to cause debate than aid the passage of the motion. Ditto 3.7.4.

Section 3.7.1 will have to be changed; it currently says "up to five (5) equally weighted nominations in every category".

David Wallace @460: "in any paired elimination from 3.8.1.1, any work from V-Z will beat any work from A-U. So it looks to me like the slate will sweep (in spite of not getting lots of points in the later rounds)"

Only the fifth place work gets compared for elimination. As works from A-U get eliminated, the votes for the remaining works go up (since their supporters' votes are less divided), potentially as high as 190, while the vote for each work on the slate stays around 40 (200 divided 5 ways), so the first four slots should go to the most popular works from A-U. For the final slot, the most popular slate work will win the elimination comparison and get on the final ballot.

#463 ::: David Dyer-Bennet ::: (view all by) ::: April 23, 2015, 01:43 AM:

FAQ #15: we can't reasonably claim "TNH and PNH had absolutely no input in the discussions, however." Teresa, for example, has commented on this very post.

More broadly, I'm not comfortable backing away from my friends quite so hard. Shouldn't the argument be more that this system is better, and if anything *helps* the puppies stated goal (diversity), but is less gameable?

#464 ::: Soon Lee ::: (view all by) ::: April 23, 2015, 02:49 AM:

Removing "TNH and PNH had absolutely no input in the discussions, however." doesn't change the point of FAQ#15.

Would it be useful to include URLs to point to this & the previous discussion? Explicit pointers to those interested to see how the sausage was made?

#465 ::: J Thomas ::: (view all by) ::: April 23, 2015, 06:37 AM:

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the two works getting the least total number of votes will be compared.

As I understand it, the rule is confusing and this simple statement is not adequate -- it makes the later explanations harder to understand.

Here are the situations to deal with:

1. The lowest three point scores are 21, 21, 24.
We compare the nominations with 21 and 21.

2. The lowest three point scores are 21, 22, 24.
We compare the nominations with 21 and 22.

3. 21, 21, 21.

We compare all three.

4. 21, 24, 24.

We compare all three.

It isn't the two works with the least number of points, because sometimes there are more.

It isn't the works with the two lowest numbers of points, because when there are two or more works tied at the lowest number then the 2nd-lowest is spared.

When there is a tie at the lowest number of points, that's all that's compared.

When there is not a tie at the lowest number but there is a tie at the second-lowest number, then we take more.

If you try to say it too simply then it hides the reality.

I don't particularly like the following:

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, at least two of the works with the lowest total number of points are compared. The rule is, take the two with the lowest number of points, and add more if and only if they are tied with the second-lowest nominee.

There ought to be a better way to say that.

#466 ::: J Thomas ::: (view all by) ::: April 23, 2015, 07:38 AM:

I am fundamentally dissatisfied with SDV-LPE but I don't have a better alternative to propose.

SDV-LPE does not work as effectively against slates as I'd like. In my simulations with reasonable estimates of the spread of other voting, with 1000 votes spread over hundreds of other nominees and 200 votes for the slate, I average about 1.7 slate wins out of five. I would prefer an average closer to 5/6, their fraction of voters.

I don't have much spare time and no uninterrupted time, so my explorations have been slow. I have the idea that something which actively tries to spread the winning votes among ballots would do better. But I haven't finished even working out the true result to measure voting routines against.

I want to grade a winning combination of nominations by first if you try to divide the ballots into five piles, each containing votes for a different nominee, how big is the smallest pile? and second, how small is the pile of votes that don't have a single winning nomination?

The first part looks pretty easy. When you have votes in a more-or-less power function that kind of fits some Hugo data, it's usually easy to bring the lowest pile up close to the average (or its maximum) and have no others lower. We're basicly solving five linear simultaneous equations in ten variables, with some restrictions on those variables. When two of the piles are from a slate then it's hard to reduce the number of ballots left out. So something that achieved that result would probably work well.

There are only 3003 combinations to choose 5 out of 15. It should be possible to just do all of them and compare, pretty quickly. But I haven't done it.

#467 ::: David Harmon ::: (view all by) ::: April 23, 2015, 08:17 AM:

Typo watch: 3.8.1.4 In the event that more than two or more works

#468 ::: Mercy ::: (view all by) ::: April 23, 2015, 08:33 AM:

I have a question that I didn't see an answer to in either the proposal language or the discussion (although searching on an ipad is less than optimal, so my apologies if I missed it).

What happens under this proposed change if someone nominates something in the wrong category? Is it simply discarded? Is it treated as it is in the current rules, moved if there's a free spot in the appropriate category?

#469 ::: Mercy ::: (view all by) ::: April 23, 2015, 08:40 AM:

Er. I realized after I previewed and hit post that I was a little unclear. I meant, if someone's nominating ballot has a work in the wrong category, right now it gets moved if they left a blank spot in the appropriate category. There's no mention of this case (which I'm guessing happens not that rarely with the shorter fiction categories) in this proposal.

#470 ::: nathanbp ::: (view all by) ::: April 23, 2015, 08:47 AM:

One thing that's not listed is there is the proposed system under SDV-LPE for picking the replacement nominee when one of the nominees refuses.

#471 ::: Keith "Kilo" Watt ::: (view all by) ::: April 23, 2015, 09:48 AM:

JT@465:

One quick clarification: The way I've coded it, in the situations you list, we compare all three in cases 1, 3, and 4. Case 2 has two with the lowest point totals, so is the "standard result". That's not to say it can't be coded some other way, but as you point out, I don't see any inherent reason to muddy the explanation with anything more complicated.

Maybe the proper way to say it is that it's the works that have the two lowest point totals.

All: Please keep the comments coming -- as we come to a group consensus, we may end up with a proposal we can use!

Kilo

#472 ::: dotless ı ::: (view all by) ::: April 23, 2015, 10:11 AM:

Keith "Kilo" Watt@458:

For FAQ#13 ("I think we should use X"): It's probably worth pointing out here that there's difference in requirements between a good system for choosing, say, a representative committee from multiple political factions, and the requirements for choosing Hugo finalists, especially given the diffuse distribution of favorite books among most of the membership. It seems to have taken a while for the discussion here to make that difference explicit, and I doubt that the difference will immediately come to mind if someone in the business meeting has a favorite voting scheme.

For FAQ#10 ("Is it too hard to understand?"): Should this section point out that the final Hugo vote already uses a system of eliminating the lowest-ranked contenders and rescoring ballots until there's a final result? That may make it seem less strange. The different rules for elimination and rescoring are because of the differences in requirements between choosing a single winner and choosing a good set of finalists.

#473 ::: Stewart ::: (view all by) ::: April 23, 2015, 10:19 AM:

As a matter of wording 3.3.1.1 and 3.3.1.2 are mildly inconsistent - enjoyable and great (best) as applied to books are imperfectly correlated concepts.

#474 ::: Steve Halter ::: (view all by) ::: April 23, 2015, 10:20 AM:

Here is an attempt to reword/reorder 3.8.1.1 through 3.8.1.5 to smooth them out and make tie handling more immediately apparent:

3.8.1.1 The final Award ballots shall list in each category the five eligible nominees as determined by successive rounds of elimination. In each round, the two works with the least total number of votes will be compared. Of those works, the work that appears on the fewest number of ballots will be selected for elimination from all ballots with ties handled as described in 3.8.1.2 and 3.8.1.3.

3.8.1.2 In the event that more than two or more works are tied for the lowest total number of votes received in 3.8.1.1, the work that appears on the fewest number of ballots will be selected for elimination.

3.8.1.3 In the event that two or more works are tied for appearing on the fewest number of ballots, all the tied works will be eliminated. If eliminating all of the tied works would bring the total number of works on the final ballot to be less than five, all the tied works will be retained, and the final ballot will be extended to include more than five works, as required.

3.8.1.4 For the next round, votes are assigned to each work as in 3.8.1. Note that if a ballot contained, for example, five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now get 1/4 of a vote for the current round.

3.8.1.5 Works are eliminated until only five works remain in the category.

#475 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 10:20 AM:

That's great work! A few comments:

1. Maximum votes per ballot: I understand the desire not to change things about the current system if the changes aren't essential. But one of the key advantages of SDV-LPE over other proportional voting systems is that it usually doesn't discourage voters from voting inclusively for more than 5 candidates. That's important because it can help achieve more overlap in non-slate ballots, which helps give inclusive, high-quality results. So if it were me, I'd definitely lift the limit on votes per ballot.

2. In the same spirit, I think commentaries 1b and 2d encourage "narrow" voting too much. I'd say :

b. Nominators may decide how strongly they feel a given work or works should be on the final ballot. If there is one work they feel very strongly about, they can give that work their entire nomination vote. If there are a number of works they feel are Hugo-quality, they can nominate as many as they like. There is an inherent tradeoff here, which no voting system can entirely avoid. Voting for many works, especially if they are popular, risks spreading your one nomination vote among the works they have nominated. However, voting for few works, or only unpopular works, risks having all of the works you voted for eliminated. Also, note that no matter how many other works you voted for, your vote still has full power to protect all your choices from being eliminated by a less-popular work.

2. Nominator instructions are equally simple:
a. Nominate whichever works you feel are Hugo worthy, up to the maximum permitted.
b. You have one nomination “vote” for each category, however, you may choose to divide your nomination vote among more works by simply listing more works on your nomination ballot. Your nomination vote will be split equally among all the works you recommend, but will still count fully to prevent all of them from being eliminated by a less-popular work.
c. It is not necessary to vote for any partcular number of works, only whichever ones you honestly believe may deserve a Hugo. Idividually, it is best to vote for enough of your favorites so that one or two of them have a good chance of becoming nominees. Collectively, the system can give the best slate of nominees if the average voter votes for around 6 or 7 candidates, so that the ballots carry plenty of information about which works are broadly popular.
d. If you feel very strongly that a particular work should be nominated, you can choose to put the entire weight of your entire nomination vote behind that work. If you have several works you feel are worthy, you can choose to recommend all of them. Voting for more cannot decrease the chances that at least one of your choices will be nominated, but it can in some rare cases decrease the chances for particular work you favor, as splitting your vote might cause that work to fall out of the set of 4 candidates which are never considered for elimination.

#476 ::: J Thomas ::: (view all by) ::: April 23, 2015, 10:24 AM:

@471 Keith "Kilo" Watt

One quick clarification: The way I've coded it, in the situations you list, we compare all three in cases 1, 3, and 4.

Interesting! I saw no hint of that in your explanation.

The two works with the lowest points. If more than two are tied for lowest, take them all.

20, 20, 23 the two with 20 are tied. The one with 23 is not tied with them.

20, 20, 23, 23 take all four.

If this is the way you want to do it, then it gets extra-easy to say. Are you sure this is what you want? The way I assumed you meant is just as easy to program. I don't know which gives better results, if there is a difference.

Here's a possible way to write what you want.

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the works getting the two (different) lowest levels of points will be compared. The work which appears on the fewest ballots shall be eliminated.


FAQ 16: How does that nomination thing go again?

First, count up the points. Say that one ballot has (A B C). Then you give 1/3 point to each of A B and C. Another ballot has (A D E F). You give 1/4 point to A D E and F. A now has 7/12 point while B has 1/3 point. Count all the ballots this way.

When you have finished adding points, you look for the two lowest levels of points among the nominees. Say that D G J and K have 21.3, 22.5, 21.3, and 22.5 respectively, and everything else has more than 22.5. Then choose D G J and K to test for elimination.

Find the total number of ballots that list D G J and K. Suppose that the numbers are 78, 89, 82, and 78, respectively. Then D and K have the lowest number of ballots, and those two are eliminated.

Repeat this process. If at some step there are fewer than five nominees remaining, restore the nominee(s) removed last time.

#477 ::: dotless ı ::: (view all by) ::: April 23, 2015, 10:39 AM:

Jameson Quinn@475: if it were me, I'd definitely lift the limit on votes per ballot.

Two reasons not to do that:

Political: Anything that makes the balloting look less like the current system might incline people to avoid the proposal as being too innovative/experimental.

Administrative: Kevin Standlee has already expressed concern about the current administrative load of determining which of the permutations of author and title refer to the same work, giving an actual example of someone kept off the final ballot due to a failure in the current system. More nomination slots means more load, which means a greater chance of failure.

#478 ::: cyllan ::: (view all by) ::: April 23, 2015, 10:42 AM:

Speaking as someone who has not been following this particular thread in any detail, I have to say that I found the language, the FAQs and the accompanying explanations quite clear. It's probably not a perfect system -- and I think some of the latter tweaks have been useful -- but it comes across cleanly. The proffered reason and 'how it works' makes sense. Give me a bit to ponder it just to make sure I understand it, and I can easily see that this is a solution that I'd be willing to support.

#479 ::: Keith "Kilo" Watt ::: (view all by) ::: April 23, 2015, 11:23 AM:

All: I was thinking about it on the way into work, and I think that the way JT suggests handling ties might actually be better. Basically, the only difference is that if there is a tie in the lowest point total, then we don't consider the second lowest point works for elimination in that round. The advantage I see to doing it this way is that you may have fewer situations where you also get ties in the number of nominations, so you will also have fewer situations where the final ballot gets extended to more than 5 works. I don't think these cases will come up often, but as JT says, it's really no harder to code, so it makes sense to do it this way. I can also think of some (essentially statistically impossible) situations where you might get some other issues.

Can the experts weigh in on this? Are there problems with just considering the lowest point total works if they are tied?

I'm keeping a list of the other suggestions, so as the group agrees on which should be incorporated, I will add that to the draft.

Thanks again,
Kilo

#480 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 11:35 AM:

dotless@477: I understand what you're saying about the political concerns, and can't really speak to that. I'm just saying that it would be better from a theoretical point of view without a limit.

As to the administrative concerns: again, the people with experience running these elections are of course the true experts on that, but I'd suggest that there are ways to make sure this isn't a problem. Basically, it's worth including an initial "blanket elimination round". Here's how I'd structure that to make it simple and robust:

1. Make a quick rough tally of the number of ballots that vote for each candidate. Don't worry too much about disambiguating the tough cases for now. Also, for this initial rough tally, do NOT count any votes on any ballot after the 7th vote per category. (This last condition is merely to prevent a slate from gaming this process by voting for 16 candidates. Since it prevents that strategy, it is very unlikely to have any impact in practice.)

2. Sort the candidates in order of number of votes.

3. Find the biggest gap in votes between two successive candidates in positions 15-21, breaking tied gap sizes by choosing the one closer to the 15th candidate. For instance, if the tallies 15-21 were 75, 73, 73, 69, 66, 62, 61 the gap in question would be 73 to 69; that is, 17th and 18th place.

4. So, the "number to beat" would be fixed at 73. Go back and disambiguate more carefully, but do not worry about cases where disambiguation could not bring a candidate above 73 or affect a total already above that level.

5. Once you've finished step 4 properly, eliminate any candidates with fewer than the "number to beat".

6. Proceed with SDV-LPE.

Theoretically, it is not impossible that the end result of the process would hinge on whether you set the initial mass-elimination "cutoff" at 15 or 20 candidates. But in practice, it is astronomically unlikely that that would happen; I would literally bet my house that it would never happen.

#481 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 11:36 AM:

dotless@477: I understand what you're saying about the political concerns, and can't really speak to that. I'm just saying that it would be better from a theoretical point of view without a limit.

As to the administrative concerns: again, the people with experience running these elections are of course the true experts on that, but I'd suggest that there are ways to make sure this isn't a problem. Basically, it's worth including an initial "blanket elimination round". Here's how I'd structure that to make it simple and robust:

1. Make a quick rough tally of the number of ballots that vote for each candidate. Don't worry too much about disambiguating the tough cases for now. Also, for this initial rough tally, do NOT count any votes on any ballot after the 7th vote per category. (This last condition is merely to prevent a slate from gaming this process by voting for 16 candidates. Since it prevents that strategy, it is very unlikely to have any impact in practice.)

2. Sort the candidates in order of number of votes.

3. Find the biggest gap in votes between two successive candidates in positions 15-21, breaking tied gap sizes by choosing the one closer to the 15th candidate. For instance, if the tallies 15-21 were 75, 73, 73, 69, 66, 62, 61 the gap in question would be 73 to 69; that is, 17th and 18th place.

4. So, the "number to beat" would be fixed at 73. Go back and disambiguate more carefully, but do not worry about cases where disambiguation could not bring a candidate above 73 or affect a total already above that level.

5. Once you've finished step 4 properly, eliminate any candidates with fewer than the "number to beat".

6. Proceed with SDV-LPE.

Theoretically, it is not impossible that the end result of the process would hinge on whether you set the initial mass-elimination "cutoff" at 15 or 20 candidates. But in practice, it is astronomically unlikely that that would happen; I would literally bet my house that it would never happen.

#482 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 12:30 PM:

@466:

SDV-LPE does not work as effectively against slates as I'd like. In my simulations with reasonable estimates of the spread of other voting, with 1000 votes spread over hundreds of other nominees and 200 votes for the slate, I average about 1.7 slate wins out of five. I would prefer an average closer to 5/6, their fraction of voters.

Yes, those are about the same numbers I would have guessed.

In defense of how SDV-LPE is working here: consider that, among the 1000 non-slate voters, there are probably at least 400 who, purely by chance, are voting for only no-hope "long tail" candidates which no voting system would elect. So from that point of view, it's a matter of 200 slate voters vs. 500-600 non-slate voters who support any viable candidate. In that case, 1.7 slate candidates is not too far from the "correct" proportional answer.

In my opinion, the best way to change that is to encourage the 400-500 "nonviable" voters to vote for more candidates overall, so that they have a better chance of including something viable. I think that if the average votes per ballot were higher - 5 or 6 or 7 instead of 3 or 4 - chances are that in this kind of scenario there would be only 2-3 hundred "nonviable" voters, so that SDV-LPE (like any other reasonable proportional system) could reasonably keep the slate to no more than 1 slot. And I think that SDV-LPE, of the proportional systems I've considered, has the best chance of encouraging that kind of "broad" voting (certainly, more so than STV, in my opinion).

I realize this isn't a perfect solution to the issue you're raising, but I think it's the best one that's available.

#483 ::: J Thomas ::: (view all by) ::: April 23, 2015, 12:49 PM:

@480 Jameson Quinn

Find the biggest gap in votes between two successive candidates in positions 15-21, breaking tied gap sizes by choosing the one closer to the 15th candidate. For instance, if the tallies 15-21 were 75, 73, 73, 69, 66, 62, 61 the gap in question would be 73 to 69; that is, 17th and 18th place.

The 5% rule which is currently in the WSFS constitution, will tend to cut it off at somewhere in the range 3 to 18.

Would you suggest lobbying to get the 5% rule removed? As you point out, nominations beyond the top 15 are unlikely to make a difference, and it could be argued that nominations that get less than 5% should not make a difference.

#484 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 01:03 PM:

@483: Personally, I think that the 5% threshold is unnecessary, because it's redundant with the No Award option in the final round. But speaking as a voting theorist, there's nothing wrong with a little redundancy. If you decide to keep the 5% rule, the initial absolute cutoff should be calculated in the way I suggested, or at 5% of the votes in that category, whichever is more.

#485 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 01:34 PM:

@479: Either way is fine from the point of view of theory. It's just a matter of what's easier to explain, pass, and program.

#486 ::: J Thomas ::: (view all by) ::: April 23, 2015, 01:35 PM:

In defense of how SDV-LPE is working here: consider that, among the 1000 non-slate voters, there are probably at least 400 who, purely by chance, are In defense of how SDV-LPE is working here: consider that, among the 1000 non-slate voters, there are probably at least 400 who, purely by chance, are voting for only no-hope "long tail" candidates which no voting system would elect. So from that point of view, it's a matter of 200 slate voters vs. 500-600 non-slate voters who support any viable candidate. In that case, 1.7 slate candidates is not too far from the "correct" proportional answer.voting for only no-hope "long tail" candidates which no voting system would elect. So from that point of view, it's a matter of 200 slate voters vs. 500-600 non-slate voters who support any viable candidate. In that case, 1.7 slate candidates is not too far from the "correct" proportional answer.

I think it's defensible, it just isn't the result I want. If we want a representative result that reflects the voting population, then it's justifiable to give a slate that's in the 10% to 25% range one spot to represent them, and rather than give them a second slot give it to something else that represents a reasonably large group of voters provided there is a good enough alternative.

I'm reasonably sure my approach would do that, but it looks like it might be hard to sell. If we had the 2015 situation over again it would under-represent the Sad Puppies if we think of the nominations as a one-man/five-votes thing. It would appear to fit their claim of being persecuted. Anybody else who thinks it should directly go by the number of votes might also dislike it.

After all, if the important thing is to nominate the best so the final vote can be for the best, then we wouldn't want to admit that the best one might be thrown off the ballot because it didn't fit together well enough with the other good choices. If we want the best, then we have to include the one that got 17% of the vote and not replace it with something else that got only 12% of the vote. ;)

SDV-LPE has that criticism too, but it sounds more like a fair vote and also it will keep slates from winning more than 2 nominations except in categories that don't get many votes. Anyway, if the best one gets onto the ballot along with slate nominations, then the best one will presumably win.

#487 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 02:09 PM:

@458:

13. I think we should use [insert other mathematical voting system].

We considered essentially every type of voting system currently in the literature, guided by two experts in the field. Some of these systems do in fact have positive properties that speak for them. None of them were as simple or as intuitive as SDV-LPE, yet SDV-LPE meets all of the stated goals for a Hugo nomination system.

Two responses to this.

1. We haven't come close to considering "every type of voting system currently in the literature". You should qualify that somehow, such as "every _applicable_ type".

2. I presume that the "two experts" are Cheradenine and me. If so, I'm glad to have been of assistance. My goals in participating here were primarily, to help a community I care about (though I'm not a member) to use systems that will help you make good decisions; and secondarily, to help show the expertise of my organization, Electology.xyz¹ (The Center for Election Science).

As a 501(c)3 organization, we've already been officially recognized as close consultants for the Webby awards, and have played unofficial consultant roles with other awards/hall of fame processes. We view our role as non-partisan, merely helping organizations to be more responsive to their members or voters, and helping to avoid results that make people unhappy. I hope that I've lived up to that goal here; while I've at times expressed my own personal opinion on some matters, I've been clear when I was doing so.

So frankly, I'd be grateful if this FAQ referred to me as something like "Jameson Quinn, from the voting reform organization Electology.xyz¹." I also understand if that is not in the offing. Obviously, even if it were a possibility, it would be only fair if Cheradenine also agreed to be referred to specifically by whatever name or handle they prefer.

¹ Of course, the xyz is actually the first three letters of "organization", but I want to be clear that I'm not trying to break the rules by posting a url, it's just what we call ourselves. (The board has made a decision to make the short name more prominent in the future; though many of our materials still focus more on the older, longer name.)

#488 ::: Cheradenine ::: (view all by) ::: April 23, 2015, 03:02 PM:

Sorry to be so quiet lately, I've been sick as a dog and all the energy I could muster went to work.

If I'm on of the two experts referred to - honestly, I'm hesitant to describe myself as an expert of any sort on this stuff. I'm just a person with a degree in a completely different area of math who happens to have developed an interest in voting theory. I feel like I've learned far more than I've taught during this discussion and exploration.

On the subject of ballot files: I do have the ones I used to sim and can send them to you, Kilo, if that would be helpful. I think I have probably found your work email, but don't want to use it without your OK.

#489 ::: David Wallace ::: (view all by) ::: April 23, 2015, 03:37 PM:

felice@462: No, I get that. The question is how many of the "other" works (A-U) make it above the 40 point threshold before getting eliminated. If the answer is 5 or more in my @460 scenario, I think the slate gets eliminated completely due to rule 3.8.1.5 on ties. If the answer is 4 or fewer, then I think the whole slate gets on the ballot, together with those other works.

In my modified scenario @461, I think the slate work with the highest number of votes always gets on, and I'm not sure what happens to the rest. Basically, I think the effect of 3.8.1.5 in its current form is "if there is a single work with a plurality of ballot mentions, that work is always on the final ballot; if two or more works are tied with the largest number of ballot mentions, usually either all of them will get on or none of them will (with a possible exception where all but one of them are eliminated in a single round due to being tied with each other.)"

It's the "all of them" possibility that concerns me a bit, although I suspect the probability of it happening in most reasonable scenarios is pretty low. We're probably not going to get a perfectly disciplined slate where nobody else votes for any of the slate works, which is basically what you would need for an n-way plurality tie.

#490 ::: felice ::: (view all by) ::: April 23, 2015, 03:57 PM:

Jameson Quinn @475: "I think commentaries 1b and 2d encourage "narrow" voting too much. I'd say :

b. Nominators may decide how strongly they feel a given work or works should be on the final ballot. If there is one work they feel very strongly about, they can give that work their entire nomination vote. If there are a number of works they feel are Hugo-quality, they can nominate as many as they like. There is an inherent tradeoff here, which no voting system can entirely avoid. Voting for many works, especially if they are popular, risks spreading your one nomination vote among the works they have nominated. However, voting for few works, or only unpopular works, risks having all of the works you voted for eliminated. Also, note that no matter how many other works you voted for, your vote still has full power to protect all your choices from being eliminated by a less-popular work."

I think we also need to highlight that as works on your ballot get eliminated, the other works you nominated get that share of your vote back, so when there's only one left, it gets your entire nomination vote, just as if you'd only nominated one work in the first place.


Jameson Quinn @482: "In my opinion, the best way to change that is to encourage the 400-500 "nonviable" voters to vote for more candidates overall"

Option 5b should be very effective in encouraging more votes for viable candidates, and reducing the impact of the 5% threshold. It should be a separate motion, though, not lumped in with SDV-LPE.


David Wallace @489: "It's the "all of them" possibility that concerns me a bit, although I suspect the probability of it happening in most reasonable scenarios is pretty low. We're probably not going to get a perfectly disciplined slate where nobody else votes for any of the slate works, which is basically what you would need for an n-way plurality tie."

I'd suggest that we amend the rules so that if there are more than three works tied for 5th place, then we eliminate all of them and only put four works on the ballot, giving us a ballot size ranging from 4 to 7.

#491 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 03:58 PM:

@489, @461: Generally, there are a couple of possibilities for this scenario (5 slate works have 195-205 votes, highest non-slate work has 190 votes). If the 4th-place non-slate vote has more than 102 divisible votes (when there are 4 non-slate and 2 slate candidates remaining), then the last elimination round will be between the top two slate candidates; one will be eliminated, leaving 4 non-slate and 1 slate winners. Otherwise, the 4th-place non-slate candidate will face a slate candidate in the last elimination round, and so the result will be 3 non-slate and 2 slate candidates. Generally speaking, if the second-place non-slate candidate has over around 150, it's about an even bet that the 4th-place will be over 100; with the first place at 190, I'd actually bet that the 4th place would be under 100 in over half the cases, but not too much more than half.

#492 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 04:00 PM:

@490: I agree with you on all three counts. Good points/ideas.

#493 ::: Keith "Kilo" Watt ::: (view all by) ::: April 23, 2015, 04:21 PM:

David, I will recode the sim to handle ties as JT suggests and run some other situations. I'm thinking it may satisfy your concerns in essentially all likely cases.

Cheradenine, feel free to send that data to my work address if you have it. I'd appreciate it! I'll let you know if out comes through.

Jameson/Cheradenine: I hate to say it, but there may be risk in using personal information with the proposal. I'd like to think not, since as someone else pointed out, this really does meet the Puppies' stated goals. Still... Ultimately, it's up to you guys.

Kilo

#494 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 04:40 PM:

@493: Speaking for myself: I think I understand what you're talking about. Obviously, it's only the most rabid of the rabid that we're thinking of here. Without trying to minimize the issue, I believe their bark is worse than their bite, especially when it comes to somebody like me, who, to put it delicately, tends to land close to the top of the kind of totem pole those people care about. So in other words: bring it on.

(As for my organization: there's no such thing as bad publicity.)

I obviously don't know about Cheradenine, but they've comported themselves admirably in every respect, and while it might not be impossible to dig up a real identity on them, in their case I don't see why anyone would maliciously try. But obviously that's up to them.

#495 ::: J Thomas ::: (view all by) ::: April 23, 2015, 08:37 PM:

Here is a way to say it for the other approach, that I believe is correct though not fully intuitive.

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the works with the lowest total number of points are compared, and the one(s) present on the fewest number of ballots is eliminated.

The rules are: Compare at least two with the lowest number of points. If two works have the same number of points, they must either both be compared or both be spared this round. Compare the smallest number of works that fit these rules.

FAQ 16: How does that nomination thing go again?

First, count up the points. Say that one ballot has (A B C). Then you give 1/3 point to each of A B and C. Another ballot has (A D E F). You give 1/4 point to A D E and F. A now has 7/12 point while B has 1/3 point. Count all the ballots this way.

When you have finished adding points, you look for the nominees with the least points. Say that D G J and K have 21.33, 22.5, 22.5, and 22.5 respectively, and everything else has more than 22.5. Then choose D G J and K to test for elimination. We accept D because it is the lowest. We include G because we must have two. We accept J and K because their points are the same as G and we must accept G.

Find the total number of ballots that list D G J and K. Suppose that the numbers are 78, 89, 82, and 78, respectively. Then D and K have the lowest number of ballots, and those two are eliminated.

(Note that on the next round, the ballot (A D E F) will give 1/3 point to each of A E and F instead of 1/4 point, because D is no longer counted. Each time we eliminate a nominee, the points increase for other nominees on the same ballots. So we must recompute the points each time.)

Repeat this process until there are five works remaining.

If there are two or three works tied for fifth place, include them so that there are six or seven final nominees. If there are more than three tied for fifth place, discard them all, leaving four final nominees.

#496 ::: David Wallace ::: (view all by) ::: April 23, 2015, 08:57 PM:

@490, @495: I'm not sure that fifth place should be considered special. You could also go directly to a five-way tie for 2nd, 3rd, or 4th (something like my scenario @460 with 1, 2, or 3 non-slate works meeting the survival threshold above the slate). It's even mathematically possible to get something like a 10-way tie for 2nd, which would give the organizers a nasty choice between an 11 item final ballot or declaring the top choice the winner without a final vote. But it would take truly impressive levels of coordination or coincidence to pull that last off on behalf of one or two slates, so I doubt it's a possibility we need to worry about too much.

#497 ::: felice ::: (view all by) ::: April 23, 2015, 10:06 PM:

David Wallace @496: "@490, @495: I'm not sure that fifth place should be considered special. You could also go directly to a five-way tie for 2nd, 3rd, or 4th"

How about this?

"3.8.1.5 In the event that two or more works are tied for appearing on the fewest number of ballots, all the tied works will be eliminated, unless eliminating all the tied works would bring the total number of works on the final ballot to be less than five. In this case, the final ballot may be extended to include up to seven works as required. If that is not sufficient to include all tied works, but eliminating all tied works would leave four works on the ballot, then all tied works will be eliminated. If eliminating all tied works would leave fewer than four works on the ballot, tied works will be randomly selected for elimination until the ballot size is reduced to seven."

Random selection isn't ideal, but if multiple works are tied on both number of ballots and sum of fractional votes, it seems the only fair option. We can't allow an unlimited number of tied works on the final ballot; it's theoretically possible to have a hundreds-of-ways tie for a spot on the ballot (albeit statistically improbable).

#498 ::: J Thomas ::: (view all by) ::: April 24, 2015, 05:26 AM:

I'm not sure that fifth place should be considered special.

Ties are rare. If there's a two-way tie above the fifth place in the final vote, you can still keep it to five by throwing out the fifth place. If there's a three-way tie above the fourth place you can throw out the fourth and fifth. If there's a tie in fifth place you're stuck. You can throw them both out or leave them both in or flip a coin.

Fifth place is special for this kind of nominating, because it's usually fifth place that's at stake.

But then, ties are rare and we don't really need to make a special rule for extremely rare cases. I got carried away.

If we want to make unresolved cases rarer, when we want to resolve ties we could use a third rule -- count the ballots that include one of the tied votes, and for each of them count the number of other winners on it. Add up the other winners, and break the tie by eliminating the one(s) that have the most overlap with other winners.

#499 ::: David Wallace ::: (view all by) ::: April 24, 2015, 04:44 PM:

J Thomas@498: Ties may be rare historically, but they become a lot more likely in the presence of slate voting, which is why we are having this discussion in the first place. If you have idealized slate voting (every slate voter votes for all the works on the slate, no one else votes for any of them), then a five-way tie in some position becomes a certainty.

That ideal model may still be rare in practice, because slates won't normally vote with perfect discipline, and some other voters may vote for one or two slate works either because they liked them or just to mess with the slate, but it's certainly a lot more likely to have a multi-way tie in the presence of a slate than if everyone is just voting their individual preferences. I think it's reasonable to ask how a voting system designed to combat slate voting can cope with ideal slate voting, and tweak the voting system if that model reveals a problem.

#500 ::: Joshua Kronengold ::: (view all by) ::: April 24, 2015, 06:20 PM:

I wouldn't include the 5% threshold in a proposal. If the meeting doesn't like its removal, it can get struck via amendment, but over the past few years it's done far more harm than good, as Best Short Story got spread thinner and thinner.

It ends up being about 5th place, because if there's a 5 way tie for 2nd, we encounter it when we run the elections for 5th. So that's ok. In general, the Hugos are better with a "long tail" candidate or three on the ballot than with only 3 candidates.

While I don't disagree that increasing the number of nominees per ballot is potentially desirable, I am very resistant to doing so with this measure. There are potentially complications with doing so in terms of the load on administrators, and on the design of nomination forms, and I think it would distract from the base point, so I've left it out of the following draft.

The usual form for this kind of thing is something like:

(note: ML only allows italics and strong -- but WSFS constitutional ammendments should use strong (bold) for the title, underlines for new text, and strikeout for removed text. I've used [strikeout] pseudotags for strikeout and <em> instead of underlines.

Short Title: Change Nominations to Single Divisible Vote, Least Popular Elimination

Moved, to amend section 3.8 (Tallying of Nominations) as follows:

Section 3.8: Tallying of Nominations.

[strikeout]3.8.1: Except as provided below, the final Award ballots shall list in each category the five eligible nominees receiving the most nominations. If there is a tie including fifth place, all the tied eligible nominees shall be listed.[/strikeout]

3.8.2: The Worldcon Committee shall determine the eligibility of nominees and assignment to the proper category of works nominated in more than one category.

3.8.3: Any nominations for “No Award” shall be disregarded.

3.8.4: If a nominee appears on a nomination ballot more than once in any one category, only one nomination shall be counted in that category.

[strikeout]3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.[/strikeout]

3.8.6: The Committee shall move a nomination from another category to the work’s default category only if the member has made fewer than five (5) nominations in the default category.

3.8.7: If a work receives a nomination in its default category, and if the Committee relocates the work under its authority under subsection 3.2.9 or 3.2.10, the Committee shall count the nomination even if the member already has made five (5) nominations in the more-appropriate category.


3.8.8 The final Award ballots shall list in each category the five eligible nominees as determined by successive rounds of elimination, where each nominator gets a single vote, divided equally among their nominations.


3.8.8.1 In each round, the two works with the least total number of (fractional) votes will be compared. Of those works, the works (at least 2) that appear on the fewest number of ballots will be selected for elimination.


3.8.8.2 In the event that more than two or more works are tied for the lowest total number of votes, the work that appears on the fewest number of ballots will be selected for elimination. If that elimination would reduce the number of ballots to fewer than 4, then instead none of the works should be eliminated and all remaining works appear on the final ballot. (In the unlikely event that more than 8 works would make the final ballot, the administrators should use their judgement).


3.8.8.3 For the next round, votes are reassigned to each work as in 3.8.8. (Example: If a ballot contained five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now gets a 1/4 of a vote)

Submitted by: (insert members here, ideally including at least one attending member who will be at all the relevant WSFS meetings. A proposing member gets the privlege of speaking first to the pro side of the proposal)

Commentary: As has been shown by numerous recent Hugo ballots, but most explicitly by the 2015 ballot, the current Hugo nomination system can be too easily dominated by the largest single faction members nominating in that category. This proposal would replace the nomination tallying system with a system designed to produce proportional representation among nominators, so that every sizable group generally gets to elect a nominee, but none get to set the terms of the entire ballot.

#501 ::: Avram ::: (view all by) ::: April 24, 2015, 06:43 PM:

We actually do support the strike tag, along with a bunch of others that aren’t listed. You can use b instead of strong if you want, as well as cite or i instead of em. And tt and code are supported (but for not kbd for some reason), and sup, sub, and small. And a few block-level tags that can wind up looking really ugly in combination with the automatic paragraph formatting.

#502 ::: David Harmon ::: (view all by) ::: April 24, 2015, 06:46 PM:

Joshua Kronengold #500: Re: tags, it's <strike>, not "strikeout". <s> doesn't seem to work, though.

#503 ::: J Thomas ::: (view all by) ::: April 24, 2015, 09:00 PM:

I was all along confused by 3.8.4, 3.8.6, and 3.8.7, but ignored it because it looked like it wasn't what we were doing. What are they doing here?

Are they saying that if they notice two blocks of votes that turn out to be for the same nominee, that they discard one of them instead of merging them? That when they move a work to a more appropriate category they discard no more than five votes for it in the "default" category?

If we're going to write about it, should we find out what it means and write it clearer? Or is it clear to other people?

About 3.8.8.2, the ways we can get in trouble are:

1. a 3 or more way tie when there are 6 left gives us too few left if they are all discarded. So we should include them all and have six nominees.

2. A 4 or more way tie when there are 7 left gives us too few left if they are all discarded. We should include them all hand have seven nominees.

3. A 5 or more way tie when there are 8 left gives us too few if they are all discarded. We should include them all and have eight nominees.

4. A 6 or more way tie when there are 9 left gives us too few if they are all discarded, and too many if they are all kept.

But one slate can't give us a 6-way tie even if they coordinate perfectly. For a 6-way tie we pretty much need for a slate to give us the same number of votes for each member which accidentally matches up with the number of votes for one non-slate choice.

My preference is to go ahead and discard them even if there are only three nominees left. This sort of tie looks very unlikely without a slate and pretty unlikely with a slate. Why go to extra trouble to keep 3 or 4 or 5 slate members on the ballot?

So if the number of final entries can be 3 to 8, then it's trouble when there's:

1. a 4-way tie when there are 6 left.
2. a 5-way tie when there are 7 left.
3. a 6-way tie when there are 8 left.

I hope these will be rare.

#504 ::: felice ::: (view all by) ::: April 24, 2015, 10:56 PM:

J Thomas @503: "I was all along confused by 3.8.4, 3.8.6, and 3.8.7, but ignored it because it looked like it wasn't what we were doing. What are they doing here?

Are they saying that if they notice two blocks of votes that turn out to be for the same nominee, that they discard one of them instead of merging them? That when they move a work to a more appropriate category they discard no more than five votes for it in the "default" category?"

3.8.4 says if a work is listed multiple times on the same ballot it's only counted once for that ballot, eg if someone nominates AAABC, it gets counted as ABC instead. Though for SDV-LPE, this rule isn't actually necessary, and we could let people weight their ballots by giving their preferred works a bigger share of their vote.

3.8.6 means that if eg someone nominates Avengers: Age of Ultron for best short story, it will be counted as a nomination in the best dramatic presentation (long form) instead, unless they've nominated five other movies in bdp(lf), in which case the nomination in the wrong category is discarded. The second half of this is also not needed for SDV-LPE with no limits, though it should stay if we keep the five works per category limit for nominations.

3.8.7 appears to be broken in the current rules - I think it should be referring to subsections 3.2.7 and 3.2.8. But it means if someone nominates eg Avengers for bdp(lf), but the administrators move it to bdp (short form) instead, the bdp(lf) nomination counts even if they've also nominated five TV episodes. Not necessary if we drop the five work limit. (And technically Avengers probably isn't eligible to be moved to bdp(sf), I'm not sure on the runtime)

#505 ::: Kevin Standlee ::: (view all by) ::: April 24, 2015, 11:33 PM:

Joshua Kronengold @500:

I wouldn't include the 5% threshold in a proposal. If the meeting doesn't like its removal, it can get struck via amendment,...
There has already been such a proposal submitted ("The 5% Solution") and it will be on the agenda. I suggest that any other proposals avoid touching that section so that it can be considered in isolation.

(This time, speaking officially from Business Meeting; my life is going to be difficult enough as it is sorting out overlapping and conflicting proposals.)

#506 ::: Chris ::: (view all by) ::: April 24, 2015, 11:51 PM:

"3.8.8.1 In each round, the two works with the least total number of (fractional) votes will be compared. Of those works, the works (at least 2) that appear on the fewest number of ballots will be selected for elimination."

I think this needs some editing. It looks like two or more of two works will be eliminated, and that can't be right.

#507 ::: P J Evans ::: (view all by) ::: April 25, 2015, 12:14 AM:

504
Under current rules, I believe that A would only be counted once under those circumstances. (It would be looked at very closely, because it looks like someone is trying to pack the ballot box. Which is what we don't want.)

#508 ::: J Thomas ::: (view all by) ::: April 25, 2015, 01:30 AM:

3.8.4 says if a work is listed multiple times on the same ballot it's only counted once for that ballot, eg if someone nominates AAABC, it gets counted as ABC instead. Though for SDV-LPE, this rule isn't actually necessary, and we could let people weight their ballots by giving their preferred works a bigger share of their vote.

Thank you! Now it makes sense.

3.8.7 appears to be broken in the current rules - I think it should be referring to subsections 3.2.7 and 3.2.8. But it means if someone nominates eg Avengers for bdp(lf), but the administrators move it to bdp (short form) instead, the bdp(lf) nomination counts even if they've also nominated five TV episodes.

I think 3.8.7 is written backward. If you nominate a work in the wrong category, they shouldn't move it to the right category if you already have 5 nominations in the right category.

But they should move it even if it's the sixth nomination in the wrong category. Don't throw it away for being the sixth nomination in the wrong category.

I think the wording of those was clumsy, but most of the reason I misunderstood it was my own misreading. I probably see what they mean just fine once you pointed it out.

#509 ::: felice ::: (view all by) ::: April 25, 2015, 01:57 AM:

J Thomas @508: "I think 3.8.7 is written backward. If you nominate a work in the wrong category, they shouldn't move it to the right category if you already have 5 nominations in the right category.

But they should move it even if it's the sixth nomination in the wrong category. Don't throw it away for being the sixth nomination in the wrong category."

Under SDV-LPE, yes, there's no need to throw away a sixth nomination. But under the current rules (and the text for 3.8.6 and 3.8.7 in #500 is the current version, not a proposed change), you're only allowed five nominations per category, and you shouldn't be allowed to get around that by deliberately or accidentally nominating extra works in the wrong category. But you shouldn't be penalised if you nominated correctly, but the administrators changed the work's category.

#510 ::: J Thomas ::: (view all by) ::: April 25, 2015, 07:28 AM:

@509 Felice

This is an academic question and it probably doesn't matter what we think. But I can see it either way.

Yes, it makes sense not to penalize somebody who acted in good faith. Also, it makes sense not to penalise them if they have room in the category you're moving them too, even if they don't have room in the category you're moving them from.

One reason I'm inclined to think it's backward is that it specifies *less than* five in the category it comes fron when it's legal at five or less there. But less than five in the category it's going to would leave it at five or less in the new category.

On the other hand, why would they specify about ballots that have six nominations with one in the wrong category? Why would that ever happen? Maybe for the same reason we are looking into five-way ties?

Sorry to clutter the discussion with this. We can ignore it, it doesn't affect what we're looking at. But thank you for explaining it.

#511 ::: J Thomas ::: (view all by) ::: April 25, 2015, 08:27 AM:

I'm ready to propose an alternative to SDV-LPE.

I know it works now.

It would be tedious to do by hand, but it's quick by computer.

Here is what it does that I want:

The Hugo nominations are a survey of what voters think deserves a Hugo. This method chooses according to two criteria.

One is that we want the largest number of voters to be represented. We don't want one minority to determine too many of the winners.

But imagine that we did that by first choosing the work with the largest number of votes, and remove those ballots from further consideration. Choose the work with the largest number of remaining votes.

By the time we get to the fifth choice, we have a much smaller number of ballots, and they are likely to be more random. The fifth winner could easily be something that has only a few votes.

So the second criterion is that each one of the five in the combination should have pretty many votes too.

We want five that cover the field about as well as five can, and each of them has a following.

The method

Take every combination of five nominees that has a plausible chance to win. (With the 5% rule there are not a large number of likely plausible winners, almost certainly 18 or less. So there are fewer than 20,000 combinations.)

For each combination, count how many ballots are saved, that is, they include a vote among the five.

For each combination, arrange the saved ballots into five groups labeled by the five nominees. Each ballot in a nominee's group includes a vote for that nominee. Do this so that the smallest group is as large as it can be. (I have a method to do this which works. It is not tuned to be efficient but still is fast enough not to need to be more efficient.)

If M is the number of ballots saved, and N is the size of the smallest group, the combination's score is N + aM where a is a tunable parameter and will probably be less than 0.5.

The winning combination has the largest score. In case of a tie, choose the tied combination that has the fewest lost votes.

If there is still a tie, choose the tied combination that has the largest second-smallest group.

Discussion

This may be NP-hard for number of winners. I haven't really checked. That is, if there were 30 winners and we had to check every combination of thirty, it may take too long. But there are only five winners and it's fine for that.


We can tune it to decide the relative importance of the two criteria. With some tunings, large slates will get one nominee but no more unless the nonslate competition is utterly pathetic.

If we put all the emphasis on M, the number of ballots saved, a slate will never get more than one nomination but some of the other winners may have only a few votes.

#512 ::: Jameson Quinn ::: (view all by) ::: April 25, 2015, 11:49 AM:

@584: That system clearly accomplishes its goals with honest votes. It's also the kind of thinking I've learned to distrust. In particular, I've learned that when you optimize too hard in one direction — in this case, trying to make sure that a slate has a hard time getting more than one nomination — there can be a kind of "strategic rebound" in an opposing direction — in this case, a strategic incentive for bullet voting. It may be that, with actual Hugo voters, bullet voting won't be a problem with your system. But to me, it is clearly strategically advantageous, far more so than under SDV-LPE.

To be honest, I think I'm a little bit biased against your system. It reminds me too much of too many other attempts to devise a voting system that optimise the result for honest votes, and which all too often end up creating a strategic nightmare. Looking rationally, I think your system has strategic problems, but certainly not at nightmare levels. But it still rubs me wrong on an aesthetic level, and there's part of me that's still saying "there has to be a catch, or even, more of a catch than you've noticed yet".

When I try to ignore my aesthetic biases, it's not a bad system. I still think that I'd rather choose SDV-LPE, because the better strategic incentives are worth the slightly weaker resistance to slates. And I also think that the discussion is far enough along that bringing up a new option isn't really what's needed.

#513 ::: felice ::: (view all by) ::: April 25, 2015, 04:49 PM:

J Thomas @510: "One reason I'm inclined to think it's backward is that it specifies *less than* five in the category it comes fron when it's legal at five or less there. But less than five in the category it's going to would leave it at five or less in the new category."

Ah, I think I see what you mean. 3.8.7 is talking about works being moved, not nominations being moved (see 3.2.7 and 3.2.8 in http://www.wsfs.org/bm/const-2014.html), and "the more-appropriate category" means the destination category, not the default category; if the new category wasn't more appropriate than the default, the admins wouldn't have moved the work. It's saying that if the admins move a work, it keeps all its nominations regardless.


J Thomas @511: "For each combination, arrange the saved ballots into five groups labeled by the five nominees. Each ballot in a nominee's group includes a vote for that nominee"

But not necessarily all the ballots with a vote for that nominee, even in the smallest pile. Interesting, but the tunable parameter is a bit of a red flag, and it seems harder to understand the process than SDV-LPE; does it produce significantly better results?

#514 ::: J Thomas ::: (view all by) ::: April 25, 2015, 05:10 PM:

@512 Jameson Quinn

You could be right. I haven't studied it much yet, beyond confirming that my coding is correct. I wanted to announce it as soon as I could since I'm so late.

In almost any system, if what you care about is getting one nomination to win, additional votes for anything else will only hurt because anything which helps another nominee is good for the competition and not for your single nominee.

That's true for SDV-LPE. Your half-vote for your preferred nominee does less good than a full vote until the second nominee loses. There's the chance that you preferred nominee will lose first. The times to vote for two are when you don't care which one wins, or when you hope they can both win with a half-vote from you for each of them.

But if what you care about is that your preferred nominee wins, it's silly to vote for a second one also, hoping that the second one will lose in time to give your full vote for the one you want.

I will present some reasoning about this, although to my way of thinking simulation trumps reasoning, if done right.

If you vote for one, you contribute to every combination of 5 nominees that includes that nominee. Any combination that doesn't include that nominee cannot save your vote. Your vote adds to your nominee's pile and to no other, which could be good or bad.

If you vote for two, you contribute to every combination of 5 nominees that includes those two nominees. It becomes easier to include your vote. It contributes somewhat more to combinations that include both of your nominees because your ballot can contribute to whichever of the two piles of votes that's smaller.

So if everybody voted for one, all the voting system could do would be to pick the 5 nominees that have the largest number of votes. There is nothing else to do in that case. But if enough people vote for two, then there's increasing flexibility.

Say that for one case we pile up votes for A B C D E as follows:

A 194
B 172
C 168
D 132
E 112

But there are 12 AE votes in A. We could move those votes to E.

A 182
B 172
C 168
D 132
E 124

And there are 10 BE votes in B. Move those to E too.

A 182
B 162
C 168
D 132
E 134

There are 24 CD votes in C so we move 18.

A 182
B 160
C 150
D 150
E 134

There are 10 DE votes in D so we move 7.

A 182
B 160
C 150
D 143
E 143

When you vote for two you like, your vote can increase the diversity score for some of the combinations that include those two.

So if you like both of them, it's good. If you only like A then I'm not sure you do better to also vote for E.

If there are 15 different nominees that have a chance, and you only vote A, you are improving the 1001 combinations that include A and not the 2002 combinations that don't contain A. You improve them more when A is the fifth choice.

If you vote A and E then you improve the 1716 combinations that include one or the other of them, and you do more for the 284 combinations that include both. So you are also improving the 717 combinations that include E but not A.

I'm not clear what strategies would work, but at first look I'd guess that if you only want one, you do better to also vote for others that look promising but not too promising. It looks very complicated to figure out, and also it may heavily depend on guessing the odds, which tends to be hard to do particularly when the result depends on the number of cross-votes for the fifth candidate.

I think it has a lot of potential, sad that I didn't get it working until now. It takes a lot of careful testing to find problems. We haven't done enough of that for SDV-LPE, much less an alternative.

"Mother Nature always sides with the hidden flaw."

#515 ::: Keith "Kilo" Watt ::: (view all by) ::: April 25, 2015, 05:35 PM:

JT@514:

We haven't done enough of that for SDV-LPE, much less an alternative.

I'm still trying to think my way through your proposal, but FYI, Cheradenine and I have been running a multitude of sims, and so far everything is looking very reasonable with SDV-LPE. We haven't yet found a case that breaks it when using variations of the 2013 Hugo data (in fact the baseline test gave the actual result of the 2013 nomination list). That's not to say those cases aren't out there, though, so I'm hoping more people will propose cases that might be troublesome so we can test them.

BTW, since suggestions have died down a bit, I'll post a summary tomorrow so that maybe we can nail down some actual proposal language. I'm not opposed to doing compare and contrast tests with the system you're describing though. My only initial complaint so far is that it's not intuitively obvious how it works philosophically yet, but as we've said before, there's value in describing the system first, then developing an explanation for it.

Kilo

#516 ::: J Thomas ::: (view all by) ::: April 25, 2015, 05:43 PM:

@513 Felice

J Thomas @511: "For each combination, arrange the saved ballots into five groups labeled by the five nominees. Each ballot in a nominee's group includes a vote for that nominee"

But not necessarily all the ballots with a vote for that nominee, even in the smallest pile.

Here's how that can happen. Say the two smallest piles are tied with 60 D votes and 60 E votes. And there is one DE vote.

If the DE vote is put in D then the smallest is E and E doesn't have all its votes. If the DE vote is put into E then the smallest is D and D doesn't have all its votes.

It takes a 5th place tie to get that. If there are more D votes, you can put the DE vote into E and if necessary get another D vote from some other pile.

Interesting, but the tunable parameter is a bit of a red flag, and it seems harder to understand the process than SDV-LPE; does it produce significantly better results?

It was intended to get better results. It hasn't been tested enough yet.

It was intended to get different results. Whenever you have more than one goal, you must choose among them. If you have a simple system that hides that choice, it has hidden the choice in the simplicity, it has made that choice without you noticing. Remember the STV systems that had different weights for the transferred parts of the vote? They made that choice explicitly, to get the result they wanted. SDV-LPE makes it seem reasonable that when you split a vote it's one vote that you're splitting so each fractional vote should be 1/n. But it's a choice. If we wanted to encourage people to vote for multiple candidates, we could make the fractional votes bigger so that they win something by voting for extras.

Two goals. If we count only the total number of votes, then the second nominee on a slate gets nothing. But the fifth winner might be from another small slate.

If we count only the spread then a slate of 200 votes will be run as one nomination with 200 votes that are all unique, and two nominations with 100 votes each, and 3 nominations with 67, 66, 66, and so on.

But I'm talking about it when I should be actually running simulations and finding out whatever I find out.

#517 ::: Cheradenine ::: (view all by) ::: April 26, 2015, 12:32 AM:

The scoring system in #511 has the potential to lead to some very weird rankings of choices of nominees.

Consider the following set of 100 ballots:

1x A
1x B
1x C
1x D
1x E
19x AF
19x BF
19x CF
19x DF
19x EF

F appears on 95% of the ballots; A, B, C, D, and E each appear on 20% of them. BUT:

ABCDE represents all ballots and can be divided in 5 equal piles of 20, giving a score of 100 + 20a.

ABCDF (likewise ABCEF, etc.) represents 99 ballots and can be divided into 4 piles of 20 and 1 of 19, giving a score of 99 + 19a, which is a lower score than ABCDE's regardless of the value of a.

I don't think a system that potentially rejects a work appearing on 95% of the ballots in favor of works which each appear on 20% of the ballots is viable.

#518 ::: Brad from Sunnyvale ::: (view all by) ::: April 26, 2015, 12:37 AM:

Long delay, working in Shanghai, but...

There are arguments for and against a lot of power for a fannish inquisition or special committee rule.

A high threshold for triggering of the power (like support of 50% of all members, something I think would be very hard to get) means it's very hard to abuse the power or even use it unless the problem is very, very clear to fans. One could learn and revise the threshold over time. The goal, as I said, is to not expect (to need) the fannish inquisition.

With a high threshold, you can give it more leeway. More leeway means more ability to fix whatever problem the attackers can dream up. Less leeway means that attackers can find loopholes -- as is an issue with purely algorithmic rules.

The amount of power can be part of the process calling for the inquisition. Which is to say, the proposal for an FI would state what it is charged to do, and 50% of fans (or whatever) would need to say yes to that. It would be easier to get 50% support for a very specific action like "Remove works with too much cluster voting from the ballot this year" or "increase nominations this year" and harder for one that authorizes doing whatever it takes.

The chief weapon is fear and surprise. Which is to say the attacker knows that whatever they do, they might get it undone. This stops the attack from happening in the first place, or makes it more mild. (The more mild the attack is, the less likely fans would approve a fannish inquisition.)

It is an interesting question. This year's ballot, with some categories swept by the slates, would probably get an inquisition. Last year's, with just a copule of slate candidates, might well not.

In effect this is a use of some direct democracy as a fallback. Permanent rule changes still need ratification and will be slow, as planned. Short term fixes can be fast, but need overwhelming support.

No change to voting, very minimal strategy, no explaining of voting system -- though there is explaining of the system by which the inquisition is activated. After which, their chief weapon can be a ruthless efficiency, and an almost fanatical devotion to fandom.

#519 ::: Joshua Kronengold ::: (view all by) ::: April 26, 2015, 01:32 AM:

Avram, thanks for the info -- I tested with <s> (which is what I usually use) but not <strike>.

I don't like JT's newer proposal. It's gameable in non-obvious ways (probably) and feels too clever to me. And SDV-LPE seems to do the job nicely for expected results and, hugley importantly, is very comprehensible both in how it operates and why it works, which is a key component to acceptance. Also, my programmer's heart rebels at NP-hard solutions even if their N is theoretically (I'm not convinced) low enough that NP-Hard isn't non-computational. Also, the lack of ease of understanding makes it much harder to vet the process.

#513: Exactly. Obviously, if we relaxed restrictions on the number of works we could drop this restriction. The idea of course is that if the will of the nominators is such that a work gets moved (this is the usual reason that the Admins move a work), then people who voted it in the "right" category don't lose the their votes--but you still don't get to nominate 6 things by dropping a nomination in the wrong category when the rules say you can only nominate 5 of them.

Re duplicates: I think avoiding the gamability of being able to list a work but give another work most of your votes, while cool on a gamer level, is a very good idea.

#506: yeah, I put the "or more" in the wrong part of the clause. Best fix that.

Kevin@505: I'll accept your preferences on this -- other Chairs have requested that measures modifying the same section be collapsed (and I thought that one was a mistake at the time even though the measures had similar but not identical purposes), but in this case the nitpicking and flyspecking committee can clean up the mess if there is one (it won't be a huge deal, if so).

Also, they really called the measure "The 5% solution?" Calling back to "The Seven Percent Solution?" I'm afraid I only know one song from that show, but it seems apropos.

TTTO: "I Never Do Anything Twice/The Madam's Song"

When I was just a young fan,
I don't recall the date,
We made a rule for our premier award,
Even if it made the cut, a nomination met its fate,
If one in twenty didn't think it scored

At first it proved a good rule,
Avoided the long tails,
But later, when the field ballooned in size,
If everbody fails,
To read all the same tales,
Where the ballot's concerned there's too much for the prize,

Then, yes, the genre was small,
Now, though, you can't read it all,
Then, tastes were more concentrated,
The best stories rated,
And found themselves slated

We must this rule amend,
At this point, it's hard to defend,
Think, you'll see it makes sense,
To not restrict works by percent.

....

Anyway, here's another draft:

Short Title: Change Nominations to Single Divisible Vote, Least Popular Elimination

Moved, to amend section 3.8 (Tallying of Nominations) as follows:

Section 3.8: Tallying of Nominations.

3.8.1: Except as provided below, the final Award ballots shall list in each category the five eligible nominees receiving the most nominations. If there is a tie including fifth place, all the tied eligible nominees shall be listed.

3.8.2: The Worldcon Committee shall determine the eligibility of nominees and assignment to the proper category of works nominated in more than one category.

3.8.3: Any nominations for “No Award” shall be disregarded.

3.8.4: If a nominee appears on a nomination ballot more than once in any one category, only one nomination shall be counted in that category.

3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.

3.8.6: The Committee shall move a nomination from another category to the work’s default category only if the member has made fewer than five (5) nominations in the default category.

3.8.7: If a work receives a nomination in its default category, and if the Committee relocates the work under its authority under subsection 3.2.9 or 3.2.10, the Committee shall count the nomination even if the member already has made five (5) nominations in the more-appropriate category.


3.8.8 The final Award ballots shall list in each category the five eligible nominees as determined by successive rounds of elimination, where each nominator gets a single vote, divided equally among their nominations.


3.8.8.1 In each round, the two works (or more, in the case of a tie) with the least total number of (fractional) votes will be compared. Of those works, all but one will be selected for elimination, removing the works that appear on the fewest number of ballots.

3.8.8.2 If (due to a tie) elimination would reduce the number of ballots to fewer than 4, then instead none of the works should be eliminated and all remaining works appear on the final ballot. (In the unlikely event that more than 8 works would make the final ballot, the administrators should use their judgement).

3.8.8.3 For the next round, votes are reassigned to each work as in 3.8.8. (Example: If a ballot contained five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now gets a 1/4 of a vote)

Submitted by: (insert members here, ideally including at least one attending member who will be at all the relevant WSFS meetings. A proposing member gets the privlege of speaking first to the pro side of the proposal)

Commentary: As has been shown by numerous recent Hugo ballots, but most explicitly by the 2015 ballot, the current Hugo nomination system can be too easily dominated by the largest single faction members nominating in that category. This proposal would replace the nomination tallying system with a system designed to produce proportional representation among nominators, so that every sizable group generally gets to elect a nominee, but none get to set the terms of the entire ballot.

#520 ::: J Thomas ::: (view all by) ::: April 26, 2015, 02:15 AM:

@517 Cheradinine

Yes, you can create cases that look like not what you'd expect. But this example actually does fit the goals for the surveying system.

It's better to represent 100% of the votes than 95% of them. That is one of the goals. It is not a stated goal to include the one with the most votes, though that may happen as a side effect of achieving the other goals. The one with the most votes is a strong candidate for helping to provide the most votes total -- particularly when there are many votes for just F -- and a strong candidate for helping to provide a balanced combination -- particularly when there are many votes for F combined with other members of that combination.

Consider the following example:

7 votes for A
7 votes for B
7 votes for C
7 votes for D
7 votes for E
5 votes for FG
4 votes for GH
4 votes for HI
4 votes for IF

With SDV-LPE that's

Absolute votes:
A-E 7 votes F-G 9 votes H-I 8 votes

1st round:
A 7
B 7
C 7
D 7
E 7
F 4.5
G 4.5
H 4
I 4

H and I tie and are removed.

2nd round:
A 7
B 7
C 7
D 7
E 7
F 6.5
G 6.5

F and G tie and are removed.

The winners are ABCDE, because all of the candidates with more votes were eliminated.

But this has a good side. Consider

7 votes for A
7 votes for B
7 votes for C
7 votes for D
7 votes for E
34 votes for FGHIJ

The first round, we get
A 7
B 7
C 7
D 7
E 7
F 6.8
G 6.8
H 6.8
I 6.8
J 6.8

There is a 5-way tie for the slate, and they all lose. Every nominee that got 34 votes has lost and every nominee that got 7 votes has won.

However, 35 of the people who cast have a winner and only 34 of them have no winners.

By some ways of looking at it, this is the right outcome. It just doesn't seem intuitive when you think in terms of voting systems where the one with the most votes is supposed to win.

#521 ::: J Thomas ::: (view all by) ::: April 26, 2015, 02:43 AM:

@519 Joshua Kronengold

3.8.8.1 In each round, the two works (or more, in the case of a tie) with the least total number of (fractional) votes will be compared. Of those works, all but one will be selected for elimination, removing the works that appear on the fewest number of ballots.

This needs changes for technical reasons.

In each round, the two works (or more, in the case of a tie) with the least total number of (fractional) votes will be compared.

Suppose the lowest fractional votes came out

3.75
3.5
3.5

Then there is a tie for the lowest, but no need to include the third. That one isn't so bad, and you could include the third one if you wanted to.

Of those works, all but one will be selected for elimination, removing the works that appear on the fewest number of ballots.

Say that you have two to compare, and they each have 19 votes. You will want to remove both unless as you say in the next section it's the last round and that removes too many. Not so bad, the next section sort of implies there can be ties where you'd remove both.

Say you have three to compare, and the votes are 19, 19, 17. Obviously you will remove the one with the low vote. Will you also remove the two that tied? If you leave them in and go another round, you may get a different result after that one is gone. They may not be tied any more, they may not even have the lowest fractional votes.

Similarly when the absolute votes are 36, 18, 9. Do you remove the lowest or all but the highest?

So I think there are things that are worth spelling out here, which the current wording does not spell out.

I'm not sure which way it ought to go. I tend to think that when the bottom two are tied for fractional votes there is no need to add more, and when the bottom two tie for absolute votes they should both usually be eliminated. IMHO.

I think that when there are more than one that don't have the lowest absolute vote, just remove the ones that do have the lowest absolute vote. Again IMHO.

#522 ::: J Thomas ::: (view all by) ::: April 26, 2015, 04:16 AM:

@519 Joshua Kronengold

I don't like JT's newer proposal. It's gameable in non-obvious ways (probably) and feels too clever to me.

I believe that to game it, you need to know things about the other votes that would be very hard to know. But it's new, and I don't know everything.

And SDV-LPE seems to do the job nicely for expected results and, hugley importantly, is very comprehensible both in how it operates and why it works, which is a key component to acceptance.

Would you say that this new system has hardly any chance of being accepted? I have the impression that you know a lot about the Business Meeting, and I don't. If it's too hard a sell, then I should either give up, or put some of my limited spare time into simple descriptions what it's about hoping to get it more palatable. (As well as some into checking that it performs as promised.) How much time is left? If there isn't enough time, or if people are just too impatient, I should drop it.

Also, my programmer's heart rebels at NP-hard solutions even if their N is theoretically (I'm not convinced) low enough that NP-Hard isn't non-computational.

That prejudice has occasionally been very good to me. "Don't do that, at N=35 the universe will end before you can get an answer." "We can sell it at N=20 right now. If somebody offers us a lot of money to do it for N=35 we can tell them to look elsewhere." ;-)

Also, the lack of ease of understanding makes it much harder to vet the process.

See if this makes sense:

Pick a possible solution, with five winners.

We want to include as many ballots as possible. So we count the number of ballots that have at least one of these five. (This is easy.)

We also want each of the five nominees to make a significant contribution to the whole. We don't want a combination where four of them mostly cover the field and the fifth one adds in five offball ballots which is enough to tip the combination into first place. So find a way to split up the ballots so that each nominee gets its own pile, and make the smallest pile be pretty big. Obviously the smallest pile can't be any bigger than the total number of votes for that nominee. (That lets us rule out some nominees early on, and rule out more as we get better solutions.)

There might be some combinatorics method that gives a super-easy solution for this, but a super-easy solution is not needed.

Start out with five piles, with the ballots that include more than one of the five distributed any which way among their choices. Take the total number of ballots in the piles and divide it by 5 to get the average. Find the smallest pile and the largest pile. If there are ballots in the largest pile which can go into the smallest pile, put them there but no more than will increase the smallest pile to average or reduce the biggest pile to average. Do that with all the other piles.

Find the new smallest pile and do it again. If you can't bring the smallest pile up to average, then pick a large pile and look for ways to send ballots to another small pile and send just as many from the small pile to the smallest. When there's no path left to send ballots from any large pile to the smallest using any number of intermediaries, then you're done. It doesn't take as many steps as you'd think. The size of the smallest pile gives the second number.

The intention is to get diversity, inclusiveness, and strength. We want to get input from as many different fans as we can, while we pick five works that are each of them strong.

A strong slate can expect to get one nomination. To compete for a second nomination, they have two nominees each with half the votes, and the second contributes nothing to total ballots included. A nonslate nominee may do better by contributing ballots that none of the other four have, even if it has less than half the number of votes that the slate does.

#523 ::: felice ::: (view all by) ::: April 26, 2015, 04:22 AM:

J Thomas @520: "H and I tie and are removed. F and G tie and are removed."

That looks like a good reason for not ever eliminating two works in one round. I'd suggest in the event of a tie on both fractional votes and number of ballots, that doesn't involve fifth place, randomly select one of the tied works to eliminate. Random selection isn't great, but it's better than denying both works the opportunity to increase their fractional vote totals. In this example, I think ABCFH, ABCGI, CDEFI, etc (34 or 38 satisfied) would all be better results than ABCDE (35 satisfied), or AFGHI (24 satisfied). Random elimination of ties produces a better balance between including the most popular works and satisfying as many people as possible.

#524 ::: Joshua Kronengold ::: (view all by) ::: April 26, 2015, 05:06 AM:

Re J Thomas@521: I've shown up to a bunch of BMs, and submitted at least one successful amendment and a measure that made it through passage (eviscerated, but with one important piece left unchanged), but like any political process it's hard to predict unless you've already polled the room and therefore have a decent baseline.

So while my instinct is that your "most respresentation" proposal would see a speedy death, I'm not about to claim any oracular authority. I do really hate that it specifically fails at proportional representation -- as Cheradine said, a proposal that if 95% of the con liked 7 specific works and the remaining 5% divided into roughly 4 groups that liked four other works, would ignore the remaiing 6 works and instead pick other tiny market segments is very much the opposite of what we want. You want a rough image of the field, not an attempt to leave as few people out. Proportionally is more important than diversity, and your proposal fails significantly on that score.

The deadline for new proposals is August 6. We have some time yet, although having stuff out and available for review will give more of a chance for technical fixes and for people to get used to the ideas.

Also, I'm not convinced that there will be numbers for any revision to nomination despite the proven brokenness. Splitting our efforts among different proposals will not help, particularly if there's one that enough people like enough.

Re ties and elimination, I don't like randomness. It's hard to predict and impossible to vet in these conditions. Frankly, if we're not just going to throw out all candidates in the case of a tie (and I do see some good reasons to do so), I'd rather in this one case make nomination order matter and break ties by how high in people's ballots they were placed -- and failing that, by order of nominator number, just to have an entirely reproducable process.

The right fix IMO to my wording re ties is to say "may" in the case of a tie--and rely on common sense that only if you have a three or more way tie for lowest -- or if you have a tie for second lowest -- do you include more than one item in the contest. This is a constitution, not a set of game rules, and some things can be left to sense and judgement.

#525 ::: J Thomas ::: (view all by) ::: April 26, 2015, 05:43 AM:

@523 felice

If we keep moving the goalposts we're never going to score.

SDV-LPE was not designed to satisfy as many people as possible. It was designed to put some emphasis on fractional votes (so that voters can vote for multiple nominees without getting too much power), and also to put some emphasis on popularity, on how many votes a nominee gets.

If you want to satisfy as many people as possible, then put that into the criteria the voting system should satisfy.

It's easy to make up pathological cases that demonstrate a voting system failing to meet criteria it was not designed to meet.

The original problem was that the existing nominating system was wide open to abuse by slates. SDV-LPE is not designed to interfere with slates, but it does interfere with slates by reducing the effect of ballots with multiple candidates. Since slates have to have multiple candidates, their effect is reduced.

It isn't designed to make the nominees with the most votes win, so sometimes they'll win and sometimes not. It isn't designed to maximize the number of satisfied voters, so sometimes it will do that and sometimes not.

If we decide what we want, we can probably get a voting system which will do that. But if the agreed goals are opposed or perpendicular, we may have to accept some trade-offs.

Jameson Quinn in #209 above had an explanation about two important criteria.

http://nielsenhayden.com/makinglight/archives/016206.html#4089097

#526 ::: J Thomas ::: (view all by) ::: April 26, 2015, 06:07 AM:

@524 Joshua Kronengold

I do really hate that it specifically fails at proportional representation -- as Cheradine said, a proposal that if 95% of the con liked 7 specific works and the remaining 5% divided into roughly 4 groups that liked four other works, would ignore the remaiing 6 works and instead pick other tiny market segments is very much the opposite of what we want. You want a rough image of the field, not an attempt to leave as few people out. Proportionally is more important than diversity, and your proposal fails significantly on that score.

I don't expect it to do that, though if you can find an example where it would, we can look at how likely that is.

If we take one popular work and fill in the gaps with four tiny ones, the smallest of them will not have many votes and so we cannot make a pile with a lot of votes for it. That means a low score.

If we instead have five popular works we can split up their votes so the smallest of them has a large pile of votes. You have postulated that a big fraction of the ballots will include at least one of them, too. So they ought to do fine unless they're a slate.

Also, I'm not convinced that there will be numbers for any revision to nomination despite the proven brokenness. Splitting our efforts among different proposals will not help, particularly if there's one that enough people like enough.

I am not convinced that we have reached a consensus about what it is a nominating system ought to do. Not to the point of choosing one. I haven't seen a traditional voting system as good as SDV-LPE for our needs. If it's that or something traditional, I'll support that. I'm not convinced that we should settle on one, yet.

We don't have it clear what we want or how we'd know when we found it.

The right fix IMO to my wording re ties is to say "may" in the case of a tie--and rely on common sense that only if you have a three or more way tie for lowest -- or if you have a tie for second lowest -- do you include more than one item in the contest.

Here you're talking about when to have 3 or more items to compare with SDV-LPE? Yes, that looks good. You can specify it or you can assume that whoever programs it will be sensible.

I'd rather in this one case make nomination order matter and break ties by how high in people's ballots they were placed -- and failing that, by order of nominator number, just to have an entirely reproducable process.

I don't like it to be random either. If you're willing to leave the other thing to committee judgement, you could do that with this one too.

#527 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:13 AM:

J Thomas: I think your system is interesting, and, as I've said, it's a surprisingly good system for one that is neither proportional nor as tradeoff-conscious as I'd like. But I think you should drop it. Several people have responded skeptically, and you've responded with further rationalizations.

-I stated, and still believe, that it does significantly more to encourage bullet voting than SDVLPE. You responded that SDVLPE can also encourage bullet voting. While true, that's not really a counter argument. Note that bullet voting makes the long tail problem tougher and the task for slates and other strategic voters easier.

-Others have said it's harder to understand.

-Others have said it seems less likely to pass the BM.

-It is not Droop proportional unless a is infinity.

-It is probably NP hard (though feasible in practice )

In defending your idea, you've made several statements about SDV-LPE that I'd disagree with.

I don't think that your system's upsides (mostly: clearer design goals) are enough to overcome all those objections.

#528 ::: J Thomas ::: (view all by) ::: April 26, 2015, 10:38 AM:

But I think you should drop it. Several people have responded skeptically, and you've responded with further rationalizations.

I might drop it, but I'm not ready to yet. It's clear that I have not explained it adequately, since people respond with claims about it that are not true.

I stated, and still believe, that it does significantly more to encourage bullet voting than SDVLPE.

How would you like that to be tested?

You responded that SDVLPE can also encourage bullet voting. While true, that's not really a counter argument.

Agreed. If encouraging bullet voting is a big drawback, how would we measure which one encourages bullet voting more?

Others have said it's harder to understand.

Yes, and I'm just beginning to learn how to describe it. I have the idea that my central description could be very clear. You make six piles of ballots, one for each of five nominees and one that has none of them. We want the smallest of the five to be large (showing that all of them have strong followings) and the pile that is not represented should be small.

Two goals, and they are right there in the picture.

Others have said it seems less likely to pass the BM.

That's a concern, but I've only just started trying to make it clear. If I can get it very clear, maybe it will look good. I have a suspicion that they wouldn't like it because it's new and different. But then, the same applies to SDVLPE. In the short run either one looks more likely than not to be rejected. I think that the runoff idea would do a lot to resolve the slate problem, and either of these would contribute more. But the runoff by itself would work better than SDVLPE by itself plus there is not much to explain. They have to do something, and if they do the minimum it will probably be that.

More than push this one alternative, I want to get it straight what criteria a nominating system should meet. If we get the criteria clear we can probably get a voting system that will meet them.

It is not Droop proportional unless a is infinity.

Should it be? Why should it be? Droop sets a quota for the number of votes a nominee should have to pass. This one maximises the number of votes the lowest candidate has, subject to also satisficing number of total votes represented.

It is probably NP hard (though feasible in practice

This is a red herring. I should not have mentioned it. I don't even know that it is NP hard, I only imagined it might be. If it is NP hard maybe somebody else should take the time to show that it is.

In defending your idea, you've made several statements about SDV-LPE that I'd disagree with.

These were not defending my idea, they were attacks on SDVLPE. My intention was to show that this sort of handwaving attack did not show much. It would probably be good though to show where I have gone wrong about SDVLPE. If my claims are wrong I will apologize. If they are right then maybe they can be palliated, or they may turn out to be right but not that important.

I don't think that your system's upsides (mostly: clearer design goals) are enough to overcome all those objections.

You may be right. If everybody's too impatient to consider it then I'll just slink away.

I think the design goals are central, and I want to get those clear, and so far mostly no one has cooperated on that. If nobody's interested in doing that then I'll slink away on that one too.


#529 ::: J Thomas ::: (view all by) ::: April 26, 2015, 11:12 AM:

What Goals Should A Nominating System Have?

Candidate goals:

Slates should tend to lose.

The nominees with the most votes should win.

The system should not encourage bullet voting or any other voting strategy.

It should be Droop-proportional whether it is a proportional voting system or not. (Is that too snarky?)

Every winner should be pretty popular. (This is a variation on "biggest vote is best vote".)

It should be politics-free and should not encourage bloc voting. People should always vote for what they personally think is best regardless of whether they think it can win. (Did I mention that it should discourage slates?)

If we had a way to estimate voter satisfaction (like how satisfied they are to have one win, or two wins, up to five wins), then we would want to maximize total voter satisfaction.

If we had a way to estimate voter satisfaction, then we would want to minimize the variance in voter satisfaction. It's better for everybody to have one win than for a small group to have 5 wins, even if the total satisfaction comes out bigger when the small group has a giant victory over everybody else. (This is another way to say we want to discourage slates. Maybe we should remove the slate goal and just use this one?)

We want a way to deal with ties that is not arbitrary. We never want to have a tie where one is accepted and the other is not, and we can't explain why it happened that way.

Have I left out a goal?

#530 ::: Cassy B. ::: (view all by) ::: April 26, 2015, 11:38 AM:

J Thomas @529, we want Joe and Jane Average Fan to understand the process at least enough to trust that it's fair, even if they don't get the nuts-and-bolts of it.

#531 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 11:57 AM:
More than push this one alternative, I want to get it straight what criteria a nominating system should meet. If we get the criteria clear we can probably get a voting system that will meet them.

And that... is why you fail. In this world, there is no "do or do not", there is only "try".

That is to say: if you explicitly maximize non-strategic criteria in a voting system, you generally make it easy for strategic voters to take advantage of that maximization, and often for them to outright subvert it.

SDV-LPE is designed with essentially the same goals as your system. I know, because I designed it. The difference is, I was also keeping an eye on minimizing strategic exploitability, and on computability, and on ease of explanation (not only in terms of answering the question "how does this work", but also the question "why did X lose?").

If you want to help us make sure we're not overlooking the best system, you need to look at plausible scenarios with an open mind, not tell more stories about why your system is best. I'm satisfied in my own mind that it's not really better. And yes, it's not entirely fair of me that I'm not willing to put in the work of explaining all the scenarios I've considered in my head, but as long as it's you against everyone else, and as long as you seem to be in defensive mode, I'm not sure that doing that explaining would have enough chance of changing many minds. I'm sorry, I know that must seem infuriating. If you can convince somebody else that your system is probably the best option, I'd put more work into explaining what I think about it.

#532 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 12:25 PM:

David@461: I ran the sim you requested, and the results were A,B,C,D,V where V was the most popular slate choice. A-D were the most popular non-slate choices. So, in this case, I think SDV-LPE did pretty well.

All: I'm going through the list and organizing the modifications that have been suggested. Once we've reached concensus, I'll type it it in a similar format to what Joshua suggests.

Thanks,
Kilo

#533 ::: J Thomas ::: (view all by) ::: April 26, 2015, 12:55 PM:

@531 Jameson Quinn

That is to say: if you explicitly maximize non-strategic criteria in a voting system, you generally make it easy for strategic voters to take advantage of that maximization, and often for them to outright subvert it.

If you can clearly say what it means for strategic voters to subvert the system, then we can design a system with thought to minimizing that.

Until then, we'll still just be floundering.

For example, people talk like it's a bad thing when voters make only one choice. But for a voter who only cares about his one choice winning, that's the right thing for him to do. So we want a voting system that won't give him inappropriate power when he does that.

But we probably don't want a system that discourages him from doing it, because we don't want him to load up on choices he doesn't really want so he can get special benefits by avoiding bullet voting.

Would it be fair to say that in general we don't want to give particular voters undue power?

Slate voters get undue power in the existing system because by working together they can get most of the wins even when they're only 20% or less of the voters.

Bullet voters get undue power in some systems because their single votes count too much.

Specify what is undue power, and we're ready to roll.

If you want to help us make sure we're not overlooking the best system, you need to look at plausible scenarios with an open mind, not tell more stories about why your system is best.

Hey, my system *is definitely the best at the two criteria it was designed to balance. That's built in, the same way that acceptance voting is the best when you define the goal as acceptance voting.

Now people are adding an ill-defined collection of other criteria. That's fine, it probably isn't the best at criteria it wasn't designed to handle. Let's define the criteria we care about, and then look at how to achieve them.

And yes, it's not entirely fair of me that I'm not willing to put in the work of explaining all the scenarios I've considered in my head

No, it's not fair at all. Not that I want to prove that this particular system will be best at all the random criteria people come up with. But we need to set up the specifications we want to build a system for, not just come in with random new requirements after we have one we don't like.

Does this remind you of the bad old days of software development? Management says they want something new. It should use genetic algorithms -- no, make that simulated annealing -- to help investors pick stocks. Oh, and quaternions. It should have a database of the stocks they're considering, and a spreadsheet they can use to calculate with, and an internet browser they can use to look up stuff, and a word processor they can make notes with, and something like PowerPoint so they can present their results to other people. It should have enough encryption that they'll feel like their data is safe. Never mind the details, just do it.

So you come up with a demo and ask them if that's what they wanted, and they say "No, the simulated annealing interface is too hard for investors to understand, and the spreadsheet looks too much like Excel but the wordprocessor doesn't look enough like Word. And the borders ought to be green and yellow." You point out that the user can set the borders to whatever colors he wants. "They need to be green and yellow."

So you make another demo, and another, and eventually you have a demo that they think they like. "OK, that's it. We'll ship that." And you know it's time to dust off your CV...

There's no reason to expect an ad hoc group to be organized, but of course until we agree about the specs we'll keep flopping around.

You might persuade people without testing that your pet system is good, but if they accept it and then they use it at WorldCon and don't like it, your name will be on it.

#534 ::: J Thomas ::: (view all by) ::: April 26, 2015, 01:01 PM:

What Goals Should A Nominating System Have?

Candidate goals:

It should be easy to explain, so that average fans will trust that it's fair even if they don't understand the details.

Slates should tend to lose.

The nominees with the most votes should win.

The system should not encourage bullet voting or any other voting strategy.

There is a minimum number of votes that every winner must get.

Every winner should be pretty popular. (This is a variation on "biggest vote is best vote" and also on "Every winner must be so big".)

It should be politics-free and should not encourage bloc voting. People should always vote for what they personally think is best regardless of whether they think it can win. (Did I mention that it should discourage slates?)

If we had a way to estimate voter satisfaction (like how satisfied they are to have one win, or two wins, up to five wins), then we would want to maximize total voter satisfaction.

If we had a way to estimate voter satisfaction, then we would want to minimize the variance in voter satisfaction. It's better for everybody to have one win than for a small group to have 5 wins, even if the total satisfaction comes out bigger when the small group has a giant victory over everybody else. (This is another way to say we want to discourage slates. Maybe we should remove the slate goal and just use this one?)

We want a way to deal with ties that is not arbitrary. We never want to have a tie where one is accepted and the other is not, and we can't explain why it happened that way.

Have I left out a goal?

#535 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 01:38 PM:
Does this remind you of the bad old days of software development?

No. As far as I can tell, nobody's asking you to add bells and whistles, because people seem to be satisfied with SDV-LPE.

I understand you're not satisfied, because it's not clear to you why we would want the thing that SDV-LPE is explicitly maximizing, and you think that should be clear. But if you want others to be dissatisfied, show them why SDV-LPE doesn't meet worthwhile goals in some realistic situation. In this case, unlike the case of your software-design memories, I think the burden of proof really should be on you.

As for your list of goals: it's pretty complete. However, I think there's one you missed. You mention not encouraging bullet voting "or any other strategy". I also think the system should make broad voting (listing relatively many candidates) easy and strategically decent, insofar as that's possible without encouraging slates or "dummy" votes for unserious candidates.

My experience tells me that your urge to explicitly maximize is pretty much at it's limit with the system you've designed; and that by trying to explicitly pursue yet more goals, you're going to find the pitfalls multiplying. But I could be wrong; as I said, your proposal is already better than I would have expected likely to come from your design attitude.

I honestly wish you luck. I do know that if you do succeed at making a better system than SDV-LPE and convincing people you've done so, it's not going to be because I've carefully tried to shoot it down and you've responded in detail, but because you yourself have carefully tried to shoot it down.

#536 ::: P J Evans ::: (view all by) ::: April 26, 2015, 01:41 PM:

534
That sounds reasonable to me.

#537 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 02:17 PM:

Okay, as promised, here are the discussion points that have been made regarding SDV-LPE so far. Please forgive the inevitable typos, as this is consolidated from my raw notes.


1. Felice@462: Move the guiding principles of the Hugo Award (3.3.1) to the commentary rather than the proposal itself.
2. Felice@462: Move the “no slates allowed rule” (3.7.4) to the commentary rather than the proposal itself.
3. Felice@462: Modify the current 3.7.1 to fix the phrase “up to five (5) equally weighted nominations in every category”
4. David Dyer-Bennet@463: We can’t claim “TNH and PNH had absolutely no input in the discussions, however.” Nor should we distance ourselves from TNH/PNH. [Kilo’s comment: I contend that their input into SDV-LPE itself has been minimal to non-existent, but I realize that’s not the same thing as no input in the discussions.]
5. Soon Lee@464: Include URLs to the discussion in the commentary section so that people can see how the system was developed
6. J Thomas@465 (see also Steve Halter@474):If there is a tie for the lowest-scoring works, then don’t consider the second-lowest work(s) for elimination. [Kilo’s comment: This is already implemented in the current SDV-LPE code, but a better description needs to be written.]
7. David Harmon@467: Typo in 3.8.1.4 In the event that more than two or more works
8. Kilo: Typo -- FAQ#10 should say “fractional” not “factional”
9. Mercy@468: What happens if something is nominated in the wrong category?
10. nathanbp@470: What do we do if a nominee withdraws?
11. dotless i@472: In FAQ#13 point out the difference between a good system for choosing representatives from various political factions and choosing a final Hugo nomination list, given the diffuse distribution of favorite works.
12. dotless i@472: In FAQ#10 point out that the Hugo voting system already uses a system of eliminating lowest-ranked contenders and rescoring ballots until there is a winner.
13. Stewart@473: 3.3.1.1 and 3.3.1.2 are inconsistent: enjoyable and great are imperfectly correlated in literature
14. Jameson Quinn@475 (see also dotless i@477): Expand number of works that can be nominated beyond five
15. Jameson Quinn@475: Soften the language of commentaries 1b and 2d so that voters aren’t encouraged to only nominate one work [Kilo’s comment: I don’t yet see that as a bad thing in the sims I’ve run, but there are likely several scenarios I haven’t tried.]
16. Jameson Quinn@480: Have an initial “blanket elimination round” to eliminate works that have no chance of winning
17. Jameson Quinn@485: Remove the 5% rule
18. Jameson Quinn@487 In FAQ#13, change “every type of voting system” to “every applicable type of voting system”
19. Jameson Quinn@487: Identify Jameson’s organization in the FAQ
20. Felice@490: Highlight that as works on your ballot get eliminated, the remaining works on your ballot count more; if there is only one of your choices remaining, it gets your full vote.
21. felice@490: If there are more than three works tied for fifth place, eliminate all of them, giving a range of 4 to 7 on the ballot.
22. J Thomas@495: New FAQ#16 explaining nomination process
23. felice@497 (see also@523): Random elimination when ties occur
24. J Thomas@498: Eliminate number of nomination ties by eliminating work that overlaps the most with other winners [Kilo’s comment: I’m not entirely sure I understand this, so may have summarized it incorrectly.]
25. Joshua Kronengold@500 (see also @519): Reformatted proposal with strikeouts [Kilo’s comment: I haven’t completed gone through each section to see if anything new has been added.]
26. Joshua Kronengold@500: At least one of the proposal writers should present it at the business meeting. [Kilo’s comment: I hadn’t planned on going to Worldcon this year, but I can, particularly since I can run any sims on the fly that might be requested. It would be great if one of our experts could go as well…]
27. J Thomas@503: Discard ties, even if that leave only three works on the ballot
28. felice@504 (see also P J Evans@507): Under current rules, if someone nominates AAABC, it is counted as ABC; how should SDV-LPE count this? [Kilo’s comment: My thoughts are to change as little of the existing system as possible, to ease passage at the business meeting.]
29. felice@504: Under current rules, works in the wrong category are moved to the correct category unless that would cause it to extend beyond the 5-work limit; how should SDV-LPE count this? [Kilo’s comment: Again, better to change only what is absolutely needed, IMO.]
30. felice@504 (see also J Thomas@508): 3.8.7 in the current rules is broken. [Kilo’s comment: If so, that should probably be brought up as a second proposal.]
31. Kevin Standlee@505: Avoid touching the 5% rule to prevent issue with other proposals.
32. Joshua Kronengold@524: Deadline for proposals is August 6.
33. Joshua Kronengold@524: Opposed to randomness in any form
34. Joshua Kronengold@524: Change wording to “may” remove ties and leave it to [someone] to judge when it is required [Kilo’s comment: I’m still pretty much opposed to any human intervention, just because the thought that it exists is what led to this in the first place.]
35. J Thomas@529 (see also@534): Goals for a nomination system
36. Cassy B@530: Nomination system should be easily understandable at a general level by average fans
37. Jameson Quinn@535: Broad voting should be easily strategic without encouraging slates or votes for unserious candidates.

Please note, I have only included commentary on SDV-LPE. JT’s new system is worth looking at as well, but I haven’t considered that here.

Can we start narrowing down the list of changes that we think are important? I do think that the less we change the current system (and, as some have mentioned, point out how it is similar to the current voting system), the easier it will be to garner support in the business meeting.

Secondly, who here is planning or willing to go to Worldcon to present the proposal?

Thanks,
Kilo

#538 ::: J Thomas ::: (view all by) ::: April 26, 2015, 02:28 PM:

revised.

Candidate goals:

1. It should be easy to understand, so that average fans will trust that it's fair even if they don't understand the details.

2.
-------------
The nominees with the most votes should win.

There is a minimum number of votes that every winner must get.

Every winner should be pretty popular.

If we had a way to estimate voter satisfaction (like how satisfied they are to have one win, or two wins, up to five wins), then we would want to maximize total voter satisfaction. We don't have a reliable way to measure voter satisfaction, and this theoretical statement basicly says that the nominees with the most votes should win (and if the ballots are rank-ordered then the higher ranks should count more).

These four are different ways to address similar issues. Basically, we don't want nominees to win even in fifth place unless they at least get "enough" votes.
------------

3. If we had a way to estimate voter satisfaction, then we would want to minimize the variance in voter satisfaction. It's better for many nominators to have one win each than for a small group to have 5 wins, even if the total satisfaction is bigger when the small group has a giant victory over everybody else.

This basicly says that votes should count more when they're spread out among ballots. Better to have more ballots with winners than fewer ballots with winners.

4. The system should not encourage voting strategy. Specific voting strategies we have thought of include bullet voting, slate voting, and adding unserious nominations to improve the chance that your real nomination will win (shotgun voting?). But while we don't want to reward adding extra nominations to ballots, we also want to do nothing to punish additional nominations. We want nominators who think in terms of strategy to feel they have little to lose by adding additional nominees that they think deserve the Hugo.

5. We want a way to deal with ties that is not arbitrary. We never want to have a tie where one is accepted and the other is not, unless we can show a good reason for it.

Have I left out a goal?

#539 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 02:59 PM:

@537: Wow, good job.

An attempt at categorization:

Minor uncontroversial fixes:

3. Felice@462: Modify the current 3.7.1 to fix the phrase “up to five (5) equally weighted nominations in every category”
7. David Harmon@467: Typo in 3.8.1.4 In the event that more than two or more works
8. Kilo: Typo -- FAQ#10 should say “fractional” not “factional”
25. Joshua Kronengold@500 (see also @519): Reformatted proposal with strikeouts [Kilo’s comment: I haven’t completed gone through each section to see if anything new has been added.]
18. Jameson Quinn@487 In FAQ#13, change “every type of voting system” to “every applicable type of voting system”

Handling special cases:


6. J Thomas@465 (see also Steve Halter@474):If there is a tie for the lowest-scoring works, then don’t consider the second-lowest work(s) for elimination. [Kilo’s comment: This is already implemented in the current SDV-LPE code, but a better description needs to be written.]
9. Mercy@468: What happens if something is nominated in the wrong category?
10. nathanbp@470: What do we do if a nominee withdraws? [JQ's comment: this is important. I'd suggest the simplest answer is just to add in the last to be eliminated, LOFI style, as long as the list is under 5 works and any absolute threshold is passed.]
21. felice@490: If there are more than three works tied for fifth place, eliminate all of them, giving a range of 4 to 7 on the ballot.
23. felice@497 (see also@523): Random elimination when ties occur
24. J Thomas@498: Eliminate number of nomination ties by eliminating work that overlaps the most with other winners [Kilo’s comment: I’m not entirely sure I understand this, so may have summarized it incorrectly.]
27. J Thomas@503: Discard ties, even if that leave only three works on the ballot
28. felice@504 (see also P J Evans@507): Under current rules, if someone nominates AAABC, it is counted as ABC; how should SDV-LPE count this? [Kilo’s comment: My thoughts are to change as little of the existing system as possible, to ease passage at the business meeting.]
34. Joshua Kronengold@524: Change wording to “may” remove ties and leave it to [someone] to judge when it is required [Kilo’s comment: I’m still pretty much opposed to any human intervention, just because the thought that it exists is what led to this in the first place.]

Simplifying administrative burden:

16. Jameson Quinn@480: Have an initial “blanket elimination round” to eliminate works that have no chance of winning


FAQ and explanatory improvements:


4. David Dyer-Bennet@463: We can’t claim “TNH and PNH had absolutely no input in the discussions, however.” Nor should we distance ourselves from TNH/PNH. [Kilo’s comment: I contend that their input into SDV-LPE itself has been minimal to non-existent, but I realize that’s not the same thing as no input in the discussions.]
5. Soon Lee@464: Include URLs to the discussion in the commentary section so that people can see how the system was developed
11. dotless i@472: In FAQ#13 point out the difference between a good system for choosing representatives from various political factions and choosing a final Hugo nomination list, given the diffuse distribution of favorite works.
12. dotless i@472: In FAQ#10 point out that the Hugo voting system already uses a system of eliminating lowest-ranked contenders and rescoring ballots until there is a winner.
15. Jameson Quinn@475: Soften the language of commentaries 1b and 2d so that voters aren’t encouraged to only nominate one work [Kilo’s comment: I don’t yet see that as a bad thing in the sims I’ve run, but there are likely several scenarios I haven’t tried.]
19. Jameson Quinn@487: Identify Jameson’s organization in the FAQ
22. J Thomas@495: New FAQ#16 explaining nomination process
20. Felice@490: Highlight that as works on your ballot get eliminated, the remaining works on your ballot count more; if there is only one of your choices remaining, it gets your full vote.


Philosophy and goals

1. Felice@462: Move the guiding principles of the Hugo Award (3.3.1) to the commentary rather than the proposal itself.
2. Felice@462: Move the “no slates allowed rule” (3.7.4) to the commentary rather than the proposal itself.
13. Stewart@473: 3.3.1.1 and 3.3.1.2 are inconsistent: enjoyable and great are imperfectly correlated in literature
33. Joshua Kronengold@524: Opposed to randomness in any form
35. J Thomas@529 (see also@534): Goals for a nomination system
36. Cassy B@530: Nomination system should be easily understandable at a general level by average fans
37. Jameson Quinn@535: Broad voting should be easily strategic without encouraging slates or votes for unserious candidates.

Points of order

26. Joshua Kronengold@500: At least one of the proposal writers should present it at the business meeting. [Kilo’s comment: I hadn’t planned on going to Worldcon this year, but I can, particularly since I can run any sims on the fly that might be requested. It would be great if one of our experts could go as well…]
32. Joshua Kronengold@524: Deadline for proposals is August 6.

Related changes to existing rules:

14. Jameson Quinn@475 (see also dotless i@477): Expand number of works that can be nominated beyond five
17. Jameson Quinn@485: Remove the 5% rule
29. felice@504: Under current rules, works in the wrong category are moved to the correct category unless that would cause it to extend beyond the 5-work limit; how should SDV-LPE count this? [Kilo’s comment: Again, better to change only what is absolutely needed, IMO.]
30. felice@504 (see also J Thomas@508): 3.8.7 in the current rules is broken. [Kilo’s comment: If so, that should probably be brought up as a second proposal.]
31. Kevin Standlee@505: Avoid touching the 5% rule to prevent issue with other proposals.


...

Looking at this, there's one thing I've been thinking about, that I see I probably haven't said out loud in relation to the specific proposal. I think it's worthwhile including a rule that expands the number of nominations in cases of near-ties. For instance: "3.8.1.6 When there are only 6 candidates remaining, works will be considered 'tied for appearing on the fewest number of ballots' if the number of ballots they appear on differs by 2 or less."

#540 ::: J Thomas ::: (view all by) ::: April 26, 2015, 03:07 PM:

@537 Keith "Kilo" Watt

24. J Thomas@498: Eliminate number of nomination ties by eliminating work that overlaps the most with other winners [Kilo’s comment: I’m not entirely sure I understand this, so may have summarized it incorrectly.]

Here's the idea. We're having various problems with ties. Sometimes we would do better if we can avoid ties, particularly when the result of a tie is that we have too few or else too many final nominees.

One possible way to break a tie is to look back at the fractional votes. They might have had the same fractional vote or maybe not. If the fractional votes are different we could discard the one with the lower fractional vote and then go on to the next round.

I was thinking that the votes could be the same and the fractional votes the same and something else related be different, but now I think that isn't possible. If A and B both have fractional vote 2.5 but A has A A AC while B has B B BCDE BCDE they will have different total votes. It's rare that you could have A AC AC AC and B B BCDE BCDE and have both the votes and the fractional votes come out the same. It just wouldn't help enough to distinguish between those.

But OK, we can benefit from other ways to choose between ties. Here's one way -- for each ballot, count the total number of votes on that ballot including votes for nominees that have already lost.

So if A had A A A A A while B had BCDEF BDFHI BCGJM BEKLN BFILM then I think we could consider B the winner. I'd like to subtly reward extra votes. But of course, this subtly rewards extra votes for dummy candidates that have no chance. So it isn't that good.

If the top seven are currently ABCDEFGH, and A and B are tied for elimination, maybe we could reward the one whose ballots also have more of CDEFGH. It's a way to reward nominating additional serious candidates. One side effect of that is that we'd eliminate the one that has less effect on the other nominees for the next round. I don't know whether that's good or bad.

It might seem like a somewhat arbitrary rule for breaking ties. But an explicit rule whose side-effects are known might sometimes be better than the alternatives of making it random, leaving it up to individual judgement, or treating both the same.

#541 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:13 PM:

Jameson@539: Thanks for the organization; my notes were simply in chronological order.

All: To my thinking, here are the major questions that need to be answered, roughly in order of importance:

1. How do we handle ties, in both points received and number of nominations received?

2. How do we handle refused nominations?

3. How do we handle works placed in the wrong category?

4. Should we expand the maximum number of nominations?

5. Should we change or eliminate the 5% rule?

6. Should we include a definition of what the Hugo nomination system (including an explicit "no slates" rule) in this proposal, or should that be proposed separately?


Most of the rest come under language and FAQ clarifications, so we could deal with them separately in successive drafts after these more fundamental questions have been answered.

Kilo

#542 ::: P J Evans ::: (view all by) ::: April 26, 2015, 03:21 PM:

540
people should not be putting the same item more than once in a category - that's a Bad Thing. It's going to be looked at ballot-box stuffing. (It's not likely to happen, but don't encourage it by letting people use it to get their favorite more than one try at being selected!) Just say 'duplicate nominees on a ballot won't be counted'.

#543 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:22 PM:

And now, my opinions:

1. As long as it's -simple- and doesn't lead to odd cases, I'm open to almost anything (other than randomness -- I don't think that will be well-received by the Hugo voters). I think some minor variation of JT's tie system for SDV-LPE (if there is a tie in the lowest points, don't consider the second-lowest points works for elimination) is the best way to go. I would support a simple system with some rare odd results over a complicated system that -might- be bulletproof. Whatever the group decides, I'm pretty sure I can code it.

2. The code reports the results of each round of elimination. Just put the work(s) that were eliminated in the previous round back in. If there's interest in seeing a sample output, just let me know and I'll post it.

For the rest, my thoughts are to leave the current rules unchanged -for this proposal-. None of those issues are essential to SDV-LPE -- it will work either way. I suggest that if we want those things changed, they be done as a separate proposal. My guiding principle is that the fewer changes to the current rules are made, the more likely it is to pass the business meeting as well as be accepted by fandom.

Kilo

#544 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:25 PM:

JT@540:

So if A had A A A A A while B had BCDEF BDFHI BCGJM BEKLN BFILM then I think we could consider B the winner. I'd like to subtly reward extra votes.

Okay, I think I see what you're saying, but I actually feel just the opposite way. To my mind, if someone felt so strongly about a work that it was the only one they nominated, then that should be weighted more heavily (in this case) than someone who ranked it equally with 4 other works.

K

#545 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:32 PM:

PJ@542:
(It's not likely to happen, but don't encourage it by letting people use it to get their favorite more than one try at being selected!) Just say 'duplicate nominees on a ballot won't be counted'.

Just a point of clarification, listing the same work multiple times won't give them more than one try. If a work is eliminated, then it will be wiped from all instances in which it appears on a ballot. What it does do is let them give more of their vote to one work than another. For example, with ABC, each work gets 1/3 of a point. With AAABC, work A gets 3/5, work B gets 1/5 and work C gets 1/5. While that might be considered a feature instead of a bug, I still don't really like it. I really don't want to encourage any sort of games with the nomination process, even ones that don't necessarily hurt anything.

Kilo

#546 ::: P J Evans ::: (view all by) ::: April 26, 2015, 03:38 PM:

545
That's why I don't like it - and in the past, duplicates like that (when they've occurred) haven't been counted because it's gaming the system in ways that are wrong. It doesn't fly with me.

#547 ::: J Thomas ::: (view all by) ::: April 26, 2015, 04:04 PM:

@544 Keith "Kilo" Watt

"So if A had A A A A A while B had BCDEF BDFHI BCGJM BEKLN BFILM then I think we could consider B the winner. I'd like to subtly reward extra votes."

Okay, I think I see what you're saying, but I actually feel just the opposite way. To my mind, if someone felt so strongly about a work that it was the only one they nominated, then that should be weighted more heavily (in this case) than someone who ranked it equally with 4 other works.

If we want to use this as a criterion for breaking ties, it can work either way. We just need an explanation that sounds good.

I would prefer to count the votes for surviving nominees instead of all votes, because those are more serious contenders than the ones that are already eliminated.

Another possibility is to go back and look at the points. A lot of the time the points will not be the same.

When it's the bottom two that are not tied on points, but are tied on votes, that's easy. When it's three or more that are not all tied on points, likewise. It's only when you get two or more tied for last place on points and then tied on votes too, where eliminating them both (all) results in too few winners but leaving them both (all) results in too many winners, that it's an issue. So that obvious, sensible rule would take care of most of the troublesome cases.

I kind of like eliminating all the tied nominees unless it causes a problem, because that would be more likely to hurt slates than anything else. But that's only a preference.

Also, when it's two items from a slate that are tied, probably every rule we might use to pick one from the other will fail. They'll be as similar as clones in a vat, we could count the letters in their names or put them in alphabetical order or something.

#548 ::: J Thomas ::: (view all by) ::: April 26, 2015, 04:21 PM:

5. Should we change or eliminate the 5% rule?

That could and probably would be a separate proposal. The voting system can live with the 5% rule or do without it.

In theory, I think the 5% rule probably should be changed. Currently small categories have trouble getting 5 candidates that qualify, maybe because they don't get a lot of votes but they do have a lot of potential nominees. But I have some vague theoretical reasons to think that for the Hugo nominations, larger samples may have bigger variance than smaller samples. It might turn out in the future as the number of voters goes up to 3000, 5,000 etc, it might get harder to have 15 nominees or even 5 that each get 5% even for big categories.

I'm not ready to stand behind that prediction, but *maybe* the 5% rule simply won't keep working as well as it has so far. We probably don't need it.

An intermediate runoff among the top 15 would probably give you nominees with more than 5%.

The 5% rule expresses a worthy goal -- we want winners that are not tiny minorities. But the rule itself may not be an adequate way to get the result the rule is supposed to enforce. If we do have other ways that do reliably get the result we want, then we don't need the rule to require them to get that result.

#549 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 04:33 PM:

JT@547

I would prefer to count the votes for surviving nominees instead of all votes, because those are more serious contenders than the ones that are already eliminated.

This is actually how it works now. Basically each round starts completely from scratch, except that at least one work has been removed from both the ballots and the list of potential works that have been nominated. It has to be this way, since we have to recalculate the points for each work every round (because a ballot that originally gave a work 1/5 of a point now may be giving it 1/4 of a point, etc.). Once a work is eliminated, it has no further effect on the process.

Kilo

#550 ::: felice ::: (view all by) ::: April 26, 2015, 04:50 PM:

J Thomas @525: "It's easy to make up pathological cases that demonstrate a voting system failing to meet criteria it was not designed to meet."

I don't think this is anything to do with criteria; rather, certain rare ties can break the intended functioning of SDV-LPE.


Keith "Kilo" Watt @537: "My thoughts are to change as little of the existing system as possible, to ease passage at the business meeting."

Sensible. So we're sticking with the limit of five nominations per category?


P J Evans @546: "545 That's why I don't like it - and in the past, duplicates like that (when they've occurred) haven't been counted because it's gaming the system in ways that are wrong. It doesn't fly with me."

Under the current counting system, yes, it would be gaming the system if it wasn't prohibited. But under SDV-LPE, there's nothing wrong with it at all, and it's not reasonable to criticise it for what effects it used to have. But as Kilo says, change as little as possible, so we should stick with the current rule on duplicates within a ballot.

#551 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 04:52 PM:

J Thomas? Drop it.

The next few times you post, I'd like to see you concentrate on discussing other commenters' opinions.

Ignore this at your peril.

#552 ::: P J Evans ::: (view all by) ::: April 26, 2015, 04:57 PM:

550
The one year I had my fingers actually in that pie, it wasn't a problem. Most people aren't likely to try it. (And really, it isn't a winning move.)

#553 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 05:36 PM:

Can we have a round of applause here for Joshua Kronengold @519?

Because even the best threads are better if there's poetry.

#554 ::: J Thomas ::: (view all by) ::: April 26, 2015, 06:09 PM:

@551 Teresa Nielsen Hayden

The next few times you post, I'd like to see you concentrate on discussing other commenters' opinions.

Drop which? One alternate voting proposal? The attempt to get a consensus about criteria that voting proposals should meet? Responses to Keith Watt's open questions about how to set up a particular proposal?

Please be more specific. I want to honor your opinion about this, and I don't see what you mean.

#555 ::: felice ::: (view all by) ::: April 26, 2015, 06:14 PM:

Joshua Kronengold @519: "When I was just a young fan"

*applauds*

(Would have done so when originally posted, but I was unfamiliar with the tune so didn't fully appreciate the filk)

#556 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 06:39 PM:

J Thomas @554: Huh. I didn't realize that the concept might be unfamiliar to you. Take a time out from the thread while I think about this.

#557 ::: felice ::: (view all by) ::: April 26, 2015, 07:03 PM:

Jameson Quinn @539: "When there are only 6 candidates remaining, works will be considered 'tied for appearing on the fewest number of ballots' if the number of ballots they appear on differs by 2 or less."

Ties are bad; we don't want to introduce more of them. Sucks to be 6th by two points, but what if 7th, 8th, and 9th are only one point behind 6th? If we let 6th in, they've got even more cause for complaint. But the near-ties issue isn't specific to SDV-LPE; if you think it's a good idea, you can propose it separately.


J Thomas @547: "when it's two items from a slate that are tied, probably every rule we might use to pick one from the other will fail. They'll be as similar as clones in a vat, we could count the letters in their names or put them in alphabetical order or something."

How about all else being equal, eliminating the last one to appear in the database? So whoever nominated first wins, and if they nominated all the tied works, the one(s) they listed first win. Technically breaks the "equally weighted" bit in 3.7.1, and theoretically encourages nominating as early as possible, but I don't think in practice anyone's going to care enough to affect behaviour.

#558 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:03 PM:

@556: In JT's defense: he really was carrying on at least three threads, and was only being stubborn in one and self-centered in at most two. I'm not going to tell you you're being too harsh; it's your blog. But I think you could at least be more specific.

#559 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:12 PM:

And I've no intention of putting myself above anyone here. 33% stubborn and 67% self-centered would probably be better than my usual averages.

#560 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 08:18 PM:

Jameson: That could be a useful distinction. In what part of the discussion would you say he's being most helpful and responsive?

#561 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:31 PM:

For instance, @547 was very much a responsive part of the general conversation, with no hint of going off on any hobby horses.

#562 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:34 PM:

@557: You're probably right. I think that it would be possible to make a good rule to minimize "almost made it" cases, but it's not related to the main problem here, and so it should probably be a separate proposal.

#563 ::: Kevin Standlee ::: (view all by) ::: April 26, 2015, 11:21 PM:

J Thomas @548:

Should we change or eliminate the 5% rule?
It's already on the agenda under the title "The Five Percent Solution" (that was my suggestion when the maker was looking for a short title). I'd really prefer that proposal be debated separately from other proposals. Remember that people need to understand what they're voting on. If they're confused, they'll probably vote it down on principle. (I know I would.) Loading too many subjects into a single proposal confuses people.

#564 ::: Joshua Kronengold ::: (view all by) ::: April 27, 2015, 03:02 AM:

Felice: This youtube is a good rendition of the base Sondheim tune. It's a saucy song, but not dirty: https://www.youtube.com/watch?v=4mf33LcIe1A

Theresa: Thanks! *bows*

I think JT has been a reasonably good citizen recently except regarding his new pet proposal.

I like a first tiebreak of score.

I -prefer- lowest sum of placement for 2nd tiebreak, because despite now making placement matter, it still doesn't matter much, and it's an actually meaningful tiebreak.

For third tiebreak, I favor something entirely unambiguous, since as we've determined, SDV produces much fairer results if ties are broken than if all tied works are eliminated. We've exhausted meaningful tiebreaks at that point, so I'd favor "the item appearing first on a ballot going chronologically" (so the item appearing first on the earliest ballot where any tied item appears).

I do think we should keep the rule that when we get a tie that includes 5th place, we should keep all tied items. This means we're using our slightly terrible (except for #1) tiebreak only in cases where we -have- to break a tie. But otherwise, we do have to break the tie if we want to keep the election proportional.

I originally proposed eliminating all tied votes because it hurts slates more than other things -- but it would have unintended consequences, and frankly we don't -need- to hurt slates that much, and randomly, while also punishing non-slate candidates for being randomly involved in a tie. So I repent of that notion.

#565 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 06:53 AM:

I do think the "eliminate all members of a ballots tie" works well. It naturally leads to allowing 6 or more nominees. And I think it would literally never hurt a non-slate candidate. If you save one of the two using a tiebreaker, the only way that candidate avoids elimination in the very next round is if it had a significantly elevated overlap with the work it tied with. Thus, both the tie and the ability to be helped by a tiebreaker are evidence of slates at work; each weak evidence on its own, but together reasonably strong.

#566 ::: J Thomas ::: (view all by) ::: April 27, 2015, 07:20 AM:

@558 Jameson Quinn

Thank you. I feel like I owe you one.

@565

I do think the "eliminate all members of a ballots tie" works well. It naturally leads to allowing 6 or more nominees.

This is a feature or a bug, according to opinion.

If you save one of the two using a tiebreaker, the only way that candidate avoids elimination in the very next round is if it had a significantly elevated overlap with the work it tied with.

This is a picky point, but not necessarily. You could have had two contestants both with low points due to many votes on the ballots, and the next one up has a lot of bullet votes and a low absolute vote count so it loses next time. If you disagree I could write up a scenario for it, unless I'm wrong. But I don't see that there's a lot riding on it.

Thus, both the tie and the ability to be helped by a tiebreaker are evidence of slates at work; each weak evidence on its own, but together reasonably strong.

I tend to agree with you on this. We can find examples where it isn't so, but if there's a slate it will usually be the slate that creates this situation.

Sometimes we need to break ties, particularly if they result in too few or too many final nominees. We have candidate methods to do that now, and I like Joshua Kronengold's method assuming I understand it.

So the other question is which ties are we better off breaking, and which ones should we eliminate all the tied ones?

I agree with you that it's usually better to eliminate both, but we could get by without that. What data would inform us about which way is better in the usual cases where it isn't clear whether we need to break the tie?

#567 ::: J Thomas ::: (view all by) ::: April 27, 2015, 07:34 AM:

@564 Joshua Kronengold

I like a first tiebreak of score.

I like that one. We've already calculated it and we already think it's important. Sometimes it will be the same, particularly for slates.

>o/I -prefer- lowest sum of placement for 2nd tiebreak, because despite now making placement matter, it still doesn't matter much, and it's an actually meaningful tiebreak.

Are you saying, number the order of votes on each ballot from 5 to 1, and give 5 points for each vote where it's in first place down to 1 point when it's in last place, and the higher sum wins?

That looks good to me! So if two nominees on the same slate are competing, the one that comes first will win. That's probably what the people who did the slate wanted, and it will almost always break a tie. They could possibly have an even number of ballots with the nominees exactly balanced, but it would not be easy.

For third tiebreak, I favor something entirely unambiguous, since as we've determined, SDV produces much fairer results if ties are broken than if all tied works are eliminated. We've exhausted meaningful tiebreaks at that point, so I'd favor "the item appearing first on a ballot going chronologically" (so the item appearing first on the earliest ballot where any tied item appears).

That one will always break a tie, so with these three methods any tie can be broken. Good.

I do think we should keep the rule that when we get a tie that includes 5th place, we should keep all tied items. This means we're using our slightly terrible (except for #1) tiebreak only in cases where we -have- to break a tie.

Here it sounds like you're saying to use it only to break ties we have to break. I'm assuming those are when the number of final nominees is otherwise less than 5 or bigger than 6.

I originally proposed eliminating all tied votes because it hurts slates more than other things -- but it would have unintended consequences, and frankly we don't -need- to hurt slates that much, and randomly, while also punishing non-slate candidates for being randomly involved in a tie. So I repent of that notion.

This looks more ambiguous, it could imply that you want to break all ties unless they result in six final nominees. If that's your meaning, I want to suggest we try to get data about its likely effects before we decide. We can have unintended consequences either way.

#568 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 27, 2015, 10:47 AM:

Okay, this is good.

#569 ::: Joshua Kronengold ::: (view all by) ::: April 27, 2015, 12:15 PM:

My concern with my "kill them all" tiebreak is that while it punishes slates, it also punishes highly correlated nominations (like, say, fandoms for particular TV shows) that happen to tie.

I'd hate to simply ignore a very popular show just because its fans happened to manage a true tie between two episodes.

Unlike a slate, a fandom (which is the type of constituency that has heretofore dominated categories) is voting entirely honestly based on things that it's pretty much guarunteed everone involved has watched. But they will (not unreasonably) discuss what the best episodes are, and thus can act slate-like in their category without breaking any of the customs of the Hugos (or intendeding to make things difficult).

#570 ::: Joshua Kronengold ::: (view all by) ::: April 27, 2015, 12:21 PM:

Oh, sorry for the double:

J Thomas@567: I think you have it backwards. It's necessary to break a tie if we're not making the final selection, since otherwise we have to eliminate all tied candidates, and as above that can have undesired consequences of disenfranchizing some of the electorate. So in that case I favor a bad but consistent rule (with some good rules mixed in early) over no tie break.

On the other hand, if all remaining tied candidates are tied for a set of places including 5th (and we don't have so many candidates that the ballot would be too large -- I think 7 or 8 is probably an upper limit here, not 6), we don't have to break a tie--we can instead stop eliminating. So in that case I favor stopping on a tie rather than breaking it.

There's no good advantage to the voters on ending up with fewer than 5 nominees unless there really -aren't- 5 nominees with any popular support. So I don't see any reason to go below 5 except in very odd situations.

#571 ::: Jon Lennox ::: (view all by) ::: April 27, 2015, 12:31 PM:

@519 Joshua Kronengold, and others making proposals:

A proposal also needs to update section 3.11.4 of the WSFS constitution, to describe how the nomination results should be published.

The practical (as opposed to legalese) part of this requirement is that we'll need to figure out how the steps and outcome of the SDV-LPE process (or whatever) should be presented in the statistics packet, in a way that's comprehensible to readers. (Separately, we should figure out how much of this presentation should be legislated in the constitution.)

#572 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:03 PM:

@569: "But they will (not unreasonably) discuss what the best episodes are, and thus can act slate-like in their category without breaking any of the customs of the Hugos (or intendeding to make things difficult)."

I understand that this behavior is not unreasonable, but it's also something that a fandom can easily avoid if they realize it risks causing simulelimination. And I think there's nothing wrong, and arguably something right, with discouraging it like that. For instance, even if one whole fandom voted a set of 2 episodes, they could still probably prevent a tie by making sure that each of them voted for something outside their common interest, so that the SDV votes for the two would be unlikely to be sequential.

Really, ties are very unlikely to exist in the absence of explicit strategic coordination, and even less likely to matter.

#573 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:06 PM:

@571: With the bulk elimination round at the start, a round-by round account of vote totals (along with a single list of ballot totals) would be pretty easy to publish. Without the bulk elimination round, it would be pretty tedious.

#574 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:11 PM:

@570: I agree, there's no reason that the voting system should go below 5 nominees, except because of the threshold. Even if there were a 5-way tie for 5th place, you could simply publish that fact. Many final round voters probably wouldn't need to read all of the 5 tied works to figure out where to put them; that is, the community could probably reach some informal consensus on 1 or 2 of the tied works to consider as "the real nominee(s)", and ignore the rest.

#575 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:12 PM:

(not that evaluating 9 nominees is impossible if it occasionally happens in categories besides Best Novel)

#576 ::: J Thomas ::: (view all by) ::: April 27, 2015, 01:16 PM:

@573 Jameson Quinn

@571: With the bulk elimination round at the start, a round-by round account of vote totals (along with a single list of ballot totals) would be pretty easy to publish. Without the bulk elimination round, it would be pretty tedious.

I recognize there are potential problems with this, but it might be possible to put up the software and the anonymised ballots into a public software repository. Let anybody who's interested run the code and see what it did.

It can be as tedious as anybody wants to look at.

That might put too much knowledge into the hands of the public. But it wouldn't have to be a bad thing.

#577 ::: J Thomas ::: (view all by) ::: April 27, 2015, 03:53 PM:

@573 Jameson Quinn

With the bulk elimination round at the start, a round-by round account of vote totals (along with a single list of ballot totals) would be pretty easy to publish. Without the bulk elimination round, it would be pretty tedious.

If you never eliminate more than one nominee per round, then:

Take the ones with the top five votes. Calculate their point scores. Look at the lowest of them.

Any nominee whose total votes is less than that point score cannot possibly win. Sooner or later it must compete with one of those five. (Or something that has beaten one of those five, which can't happen.)

If its point score was high enough it might possibly avoid competing with them until some of them have done each other in, and that would give it a chance to win. But their point scores are already higher than its can ever be. So it must compete and lose.

That provides some mass elimination, if the rules wind up working that way.

Is my logic right?

#578 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 05:35 PM:

Here's a sample output from the SDV-LPE code. This was run using the baseline data from the 2013 Hugo ballots. The ballots were randomly generated with frequencies that match the reported frequencies from the 2013 Best Novel category. This output could easily be made public to anyone who was interested.

- Works 1-15 are the top works in order of number of nominations received. (#15 received 55 votes)

- Works 16-19 are unknown works receiving 54,53,52 nominations each.

- Works 20-24 are unknown works receiving 50 nominations each.

- Works 25 is slate candidate #1 (2 nominations); work 26 is slate candidate #2 (1 nomination) -- note we increased these to actual slates in other scenarios.

- The remaining works are for various minor works that received few or no nominations.

Here are the results:

Results of Round: 1
Work(s) Eliminated:
28 Points: 0 Nominations: 0
29 Points: 0 Nominations: 0
30 Points: 0 Nominations: 0
31 Points: 0 Nominations: 0
32 Points: 0 Nominations: 0

Results of Round: 2
Work(s) Eliminated:
27 Points: 0.25 Nominations: 1

Results of Round: 3
Work(s) Eliminated:
26 Points: 0.583333333333333 Nominations: 2

Results of Round: 4
Work(s) Eliminated:
23 Points: 21.1666666666667 Nominations: 50

Results of Round: 5
Work(s) Eliminated:
20 Points: 22.4166666666667 Nominations: 50

Results of Round: 6
Work(s) Eliminated:
24 Points: 23.5833333333333 Nominations: 50

Results of Round: 7
Work(s) Eliminated:
21 Points: 24.9833333333333 Nominations: 50

Results of Round: 8
Work(s) Eliminated:
19 Points: 22.25 Nominations: 52

Results of Round: 9
Work(s) Eliminated:
22 Points: 26.2833333333333 Nominations: 50

Results of Round: 10
Work(s) Eliminated:
25 Points: 27.0666666666667 Nominations: 50

Results of Round: 11
Work(s) Eliminated:
16 Points: 27.2833333333333 Nominations: 55

Results of Round: 12
Work(s) Eliminated:
17 Points: 30.2 Nominations: 54

Results of Round: 13
Work(s) Eliminated:
18 Points: 30.9166666666667 Nominations: 53

Results of Round: 14
Work(s) Eliminated:
15 Points: 31.8666666666667 Nominations: 56

Results of Round: 15
Work(s) Eliminated:
14 Points: 34.4166666666667 Nominations: 58

Results of Round: 16
Work(s) Eliminated:
13 Points: 32.95 Nominations: 61

Results of Round: 17
Work(s) Eliminated:
12 Points: 40.6666666666667 Nominations: 62

Results of Round: 18
Work(s) Eliminated:
10 Points: 39.25 Nominations: 69

Results of Round: 19
Work(s) Eliminated:
11 Points: 44.5 Nominations: 68

Results of Round: 20
Work(s) Eliminated:
9 Points: 45.5 Nominations: 74

Results of Round: 21
Work(s) Eliminated:
8 Points: 62.9166666666667 Nominations: 90

Results of Round: 22
Work(s) Eliminated:
7 Points: 68 Nominations: 91

Results of Round: 23
Work(s) Eliminated:
6 Points: 69.6666666666667 Nominations: 101

Final Hugo Ballot:
1 Points: 152.083333333333 Nominations: 193
2 Points: 106.75 Nominations: 138
3 Points: 102.666666666667 Nominations: 135
4 Points: 100.583333333333 Nominations: 133
5 Points: 94.9166666666667 Nominations: 118

#579 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 05:39 PM:

JT@577:
Any nominee whose total votes is less than that point score cannot possibly win.

I'm not sure that's true, if I understand what you're saying. It's entirely possible -- likely, even -- for the points a work receives to go up in the next the round under SDV-LPE. As a work is eliminated from a ballot, the remaining works get more of that ballot's vote. Maybe I'm just misunderstanding, though.

Kilo

#580 ::: felice ::: (view all by) ::: April 27, 2015, 06:05 PM:

Jameson Quinn @572: "I understand that this behavior is not unreasonable, but it's also something that a fandom can easily avoid if they realize it risks causing simulelimination. And I think there's nothing wrong, and arguably something right, with discouraging it like that. For instance, even if one whole fandom voted a set of 2 episodes, they could still probably prevent a tie by making sure that each of them voted for something outside their common interest, so that the SDV votes for the two would be unlikely to be sequential. "

We really, really don't want nominators to have to worry about that sort of thing! And I don't think your suggestion is very practical - if you're requiring them to nominate things other than their top two preferences, the extras have to be both diverse (it obviously doesn't help if they all nominate the same extras), and popular enough to not all get eliminated before the potential tie is reached, but not so popular that supporting them is likely to help them push the two really popular works off the ballot.

Ties aren't that likely with slates, because bloc voters don't appear to all stick exactly to the slate (and some works may have supporters who aren't following the slate). Ties are unlikely but perfectly possible through coincidental genuine nominations of popular works (eg there's likely to be broad consensus on the best two Doctor Who episodes of the year without any deliberate coordination, which increases the probability of an accidental tie). There's no meaningful advantage to not breaking ties, and potential real harm, on occasion.

#581 ::: P J Evans ::: (view all by) ::: April 27, 2015, 06:32 PM:

580
My imperfect memory says that of the nearly 1500 final ballots in 1984, more than 1200 had 'Return of the Jedi' ranked. I can't remember how many nominations it got (if I ever knew).

Still waiting on Frisbie: he's baking tapes for one last read, which is 10 hours of baking (under controlled conditions) per batch of tapes. And some of the stuff in on 8-inch floppies, and he can't yet get at that computer.

#582 ::: dotless ı ::: (view all by) ::: April 27, 2015, 08:38 PM:

Keith "Kilo" Watt@579: I read the suggestion as comparing the number of nominations for each work against the fifth-best initial point totals. The final point total for any work can't exceed its number of nominations, so that should be a safe initial filter.

I can't tell how useful this would be without more insight into actual ballot distributions. In the scenario you just posted, for example, the fifth-lowest initial point total could have been as little as 23.6 (118/5), which doesn't eliminate very many works.

It's been a long day, so I may be misanalyzing this.

#583 ::: J Thomas ::: (view all by) ::: April 27, 2015, 09:06 PM:

@579 Keith "Kilo" Watt

"Any nominee whose total votes is less than that point score cannot possibly win."

I'm not sure that's true, if I understand what you're saying. It's entirely possible -- likely, even -- for the points a work receives to go up in the next the round under SDV-LPE. As a work is eliminated from a ballot, the remaining works get more of that ballot's vote.

Maybe you will find a flaw in the logic.

We have five nominees whose minimum score is, say, 20.

A 31
B 29
C 26
D 22
E 20

We have a sixth nominee whose total votes add up to less than the minimum of the five scores.

votes for F = 19.

F can never have a score higher than 19, because when its votes are completely unsplit it gets one vote per ballot.

There are two ways for F to win. One is to have the most votes. The other is to have a score so high that it does not get challenged.

F can never have the most votes, because there are five that already have scores higher than F has votes. So all five must have at least as many votes as they have scores, more than F.

Maybe some of them will be eliminated? Maybe, but their scores are higher than F's score can ever be, so F will be challenged and eliminated before any of them. Even if four of them have such high scores that they are never challenged, the fifth has more votes than F and will eliminate F. If something else has a high score and the result is that D and E face a challenge and E is eliminated, that will happen only after F is eliminated because F's score is lower than E's and lower than D's.

F cannot avoid challenge, and cannot win challenge. So F cannot win.

I think this generalizes. At any time, if you look at the five highest scores, any nominee whose total number of votes is less than the fifth highest score cannot win.

Is this wrong?

#584 ::: felice ::: (view all by) ::: April 27, 2015, 09:15 PM:

dotless ı @582: "I can't tell how useful this would be without more insight into actual ballot distributions. In the scenario you just posted, for example, the fifth-lowest initial point total could have been as little as 23.6 (118/5), which doesn't eliminate very many works."

It eliminates every work nominated by fewer than 24 people, which I would expect to take out most of the candidates in a typical set of Hugo nominations; there'd probably be plenty of works nominated by just one person, for example.

#585 ::: Chris ::: (view all by) ::: April 27, 2015, 10:01 PM:

J Thomas @583:
That looks like a sound argument to me, provided that "total votes" means the total number of ballots the nominee is on. You'll probably want to clarify that if this becomes part of the formal proposal

#586 ::: dotless ı ::: (view all by) ::: April 27, 2015, 10:04 PM:

felice@584: I'm just being cautious because I don't have a good intuition about the actual distribution of the votes. Scanning through all the 2013 nomination stats I see only one category (Best Editor, Short Form) where any of the top 15 listed would definitely be eliminated by that rule.

#587 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 10:09 PM:

JT@583:
F can never have a score higher than 19, because when its votes are completely unsplit it gets one vote per ballot.

Unless I'm totally messed up, this isn't true. Unless you mean total nominations and not total points?

It's virtually guaranteed that F will have more than 19 points in a later round. If it appeared on a ballot with, say, G then that ballot only gave it half a point. Once G is eliminated, that ballot will give it a full point and F's total goes to 19.5, doesn't it?

Kilo

#588 ::: felice ::: (view all by) ::: April 27, 2015, 10:17 PM:

dotless ı @586: "Scanning through all the 2013 nomination