Back to previous post: On voting systems: a guest post from Bruce Schneier

Go to Making Light's front page.

Forward to next post: The 2015 Hugo finalists, v. 2.0

Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)

April 13, 2015

Discussing Specific Changes to the Hugo Nomination Election: Another Guest Post By Bruce Schneier
Posted by Bruce Schneier at 07:11 AM *

This is a continuation of an earlier thread (now closed) about the Hugo nomination process and how it might be modified. It is not a discussion of either 1) whether or not the nomination process should be changed, or 2) any potential changes in the rules about who is allowed to nominate. It is exclusively focused on voting systems and their relative merits, given what happened in the 2015 Hugo nomination process. Whatever we choose should have these properties:

  • It should be fair.
  • It should be perceived as fair.
  • It should be relatively easy to explain, both to the voters and at the WSFS business meeting.
  • It should be relatively easy to administer.
  • It should encourage people to nominate.
  • It shouldn’t result in too many nominees for the electorate to reasonably read and rank by the Worldcon.
  • It should be resilient to some degree against strategic voting: i.e., minority voting blocs.

In the earlier thread, we identified several different ways to change the voting system. I am going to number them, so we can more easily discuss and compare their properties and suitability.

Option 1: Change the number of candidates a person can nominate. Right now, that number is 5. It can be made less — or more — than 5. Call this parameter x.

Option 2: Change the number of winners of the nomination election. Right now that number is 5, with the possibility of more in the case of a tie. We can make this larger (or smaller, I suppose). We can either fix this number at a single value, or make it variable based on various characteristics of the votes (several ways of doing this is are here, here, here, and here). Call this parameter z.

Option 3: Change the mechanism by which the winners are selected. Right now it is a simple first-past-the-post system, in which the nominees that get the most votes win. There are other ways to choose a winner, some more resilient to bloc voting than others. Here, we have several possibilities:

Option 3a: A Satisfaction Approval Voting (SAV) system (see here, here, and here). In this system, the “satisfaction” of each voter with each possible set of winners is computed based on how many of their nominations win). The winners are chosen to maximize the total satisfaction of all voters. SAV computes each voter’s satisfaction as the fraction of their nominees who are elected.

Option 3b: A Proportional Approval Voting (PAV) system (see here). This system is like SAV, except each voter’s satisfaction is computed with the first nominee elected giving +1 satisfaction, the second nominee +1/2, the third nominee +1/3, and so on - so a voter who had three of their nominees chosen would have a satisfaction of 1 + 1/2 + 1/3 = 1 5/6.

Option 3c: A Reweighted Approval Voting (RAV) — also called Sequential Proportional Approval Voting — system (see here, here, here, and here). This system first nominates the candidate with the most votes. Then, all ballots featuring that candidate are “reweighted” so that votes on them are worth proportionally less. The votes are tallied again with the new weights, and then this process is repeated until all nominees are selected. Multiple values of weights has been proposed; these include (Option 3c-1) d’Hondt (1, 1/2, 1/3, 1/4, …), (Option 3c-2) Saint-Lague (1, 1/3, 1/5, 1/7, …), and (Option 3c-3) exponential (1, 1/2, 1/4, 1/8, 1/16…) as the weights for ballots with 0, 1, 2, 3, … candidates already nominated.

Option 3d: A Single Transferable Vote (STV) system (see here and here, and a simplified version here) probably without — but possibly with — ranking. (Illustration with ranking here.) Votes are “divvied up” among the candidates, according to what the ballots say. Candidates are nominated one by one, based on which one has the most votes. Each time a candidate gets nominated, it “uses up” a certain number of votes, which is taken proportionally from the votes it currently holds. If a voter has elected a pre-determined number of candidates, it is discarded. (Call this parameter y.) If no candidate has enough votes, the one with fewest votes is eliminated. Remaining votes are redistributed after each election or elimination. Variants: with or without ranking; and with or without transfers of “extra” votes.

Option 4: Banning Slates. We create a rule outlawing slates. We’d have to figure out how to define a slate, and how to enforce the rule, but it could be done.

Option 5: Making the Voting Tallies Public Throughout the Process. The voting administrator makes public the current state of the vote throughout the nominating process.

These are not mutually exclusive, of course, although we can choose only one Option 3.

Other systems people have mentioned are Cumulative Voting, Single Divisible Vote, Random Ballot, Condorcet Proportional Representation, anti-votes, and several different ways of adding a third voting/approval round. These have not garnered very much support (for good reasons, I think), and I don’t think these are worth further consideration.

Again, there are many important issues relating to Hugo voting that are not part of this discussion, but should be discussed elsewhere, including: 1) whether to do something at all, 2) whether to change the electorate, either by making voting easier, making it harder, or turning either the electorate into some sort of preselected jury, 3) changing the voting mechanism of the final election in addition to the nominating election. This discussion assumes that whoever decides these things wants something to be done about the nomination system. We’re here to figure out what should be done in that case.

(Thank you to Cheradenine and Jameson Quinn for helping with this summary post.)

Comments on Discussing Specific Changes to the Hugo Nomination Election: Another Guest Post By Bruce Schneier:
#1 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:13 AM:

Example for Options 3a-d:

Assume that 16 ballots are cast, including votes for 10 different works (labeled A through J), and that 3 works are being nominated. In principle, voters in any of these systems can put forth for as many works as they want, but in this example, no one has listed more than 3 works. There is a slate in play, consisting of ABC.

#1: ABC
#2: ABC
#3: ABC
#4: ABC
#5: ABF
#6: AC
#7: AE
#8: BD
#9: DFG
#10: DG
#11: DIJ
#12: EH
#13: EI
#14: EJ
#15: FGH
#16: H

The raw vote counts are A 7, B 6, C 5, D 4, E 4, F 3, G 3, H 3, I 2, J 2. Under the current system, the nominated works would be A, B, and C (i.e. the slate).

Option 3a: Satisfaction Approval Voting (SAV).

We compute a satisfaction score for each voter and each possible group of nominees by looking at the fraction of a voter's choices that were nominated. For example, if B, C, and G were nominated, voter #1 would have a satisfaction score of 2/3 because 2 out of their 3 nominees were nominated. We then add up the scores for each possible set of nominees. For BCG, this yields a satisfaction score (going in the same order as the table) of 2/3 + 2/3 + 2/3 + 2/3 + 1/3 + 1/2 + 0 + 1/2 + 1/3 + 1/2 +0 + 0 + 0 + 0 + 1/3 + 0 = 5.17.

There are 120 possible sets of nominees. For this group, the highest satisfaction score is 6.83 for ABE - so A, B, and E would be nominated. (ABC and ABH are close at 6.67.)

In practice, the winner can also be found by finding a satisfaction score for each candidate, and nominating the 3 candidates with the highest satisfaction score. This yields the same result and requires less computation.

Option 3b: Proportional Approval Voting (PAV).

Again, we compute a satisfaction score, but this time it is 1 if there's one nominee from the ballot, 1 + 1/2 if there are two, and 1 + 1/2 + 1/3 if there are three. The idea is that the more candidates a ballot provides, the less heavily we will weight that person getting more candidates on the ballot. Using this method, if B, C, and G were nominated, voter #1 would have a satisfaction score of 1 + 1/2. We then add up the scores for each possible set of nominees; here, for BCG they are 1.5 + 1.5 + 1.5 + 1.5 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 = 12.

Again, there are 120 sets of nominees to examine. The highest satisfaction score is 14.5 for ADE - so A, D, and E would be nominated. (ABD, ABE, and ADH are close at 14.)

There is no known more efficient way to compute the winner, which means this method is computationally difficult for high numbers of nominees.

Option 3c: Reweighted Approval Voting (RAV).

I'm using the exponential weights (1, 1/2, 1/4, ...) for this example.

We start by nominating the candidate who appears on the most ballots, in this case A. The ballots with A on them (#1-7) are now worth 1/2 the votes for their remaining works.

The new vote counts are B 3.5, C 2.5, D 4, E 3.5, F 2.5, G 3, H 3, I 2, J 2. D now has the most votes and is nominated, and ballots containing D (#8-11) are now worth 1/2 the votes. (If any ballots contained both A and D, they would now count 1/4.)

The vote counts now stand at B 3, C 2.5, E 3.5, F 2, G 2, H 3, I 1.5, J 1.5. E has the most votes left and is our third nominee. ADE is our final ballot. (If we continued another round, B would be the fourth.)

Option 3d: Single Transferable Vote (STV).

I assume that each voter's choices are ranked in the order given above (so slate voters rank A 1st, B 2nd, C 3rd).

The quota (number of votes to guarantee election) is 16/(3 + 1) + 1 = 5.

The first-place vote count is A 7, B 1, C 0, D 3, E 3, F 1, G 0, H 1, I 0, J 0. A has more votes than the quota, so it is nominated. There are 2 surplus votes over the quota, so these are evenly divided over the second choices of A's supporters. This means B receives 10/7 of a vote, C receives 2/7 of a vote, and E receives 2/7 of a vote.

New vote counts are B 2.43, C 0.29, D 3, E 3.29, F 1, G 0, H 1, I 0, J 0. No one has the quota, so we eliminate the candidate(s) with the fewest first-place votes. G, I, and J are eliminated; C is eliminated and their vote does not transfer (since the AC ballot has no candidates left on it); F and H are eliminated and their votes do not transfer (since their ballots are likewise exhausted).

We stand at B 2.43, D 3, E 3.29. No one has the quota, so B is eliminated and the remaining two candidates (D and E) are nominated. The final list of nominees is ADE.

#2 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 07:13 AM:

I think we are going to need to do some modeling of the various systems under different scenarios. Here is some data to start with.

Let's take 2013 is a "normal" year. The voting bloc had just started, and they weren't much of a force. The data is here. Two examples:

Best Novel
Total Ballots: 1113
A: 193
B: 138
C: 135
D: 133
E: 118
---Above six on ballot---
F: 101
G: 91
H 90
I: 74
J: 69
K 68
L: 62
M: 61
N: 58
O: 56
P: 55

Best Novelette
Total ballots: 616
A: 89
B: 62
C: 61
D: 54
E: 55
F: 38
---Above six on ballot---
G: 37
H: 36
I: 35
J: 31
K: 30
L: 28
M: 22
N: 22
O: 20
P: 20

This is what we know about those same two categories in 2015:

Best Novel
Total Ballots: 1827, 587 entries total
A: 387
B: ?
C: ?
D: ?
E: 256
---Above five on ballot---
---Three are bloc entries; two are not---
F: ?
... ?

Best Novelette
Total Ballots 1031, 314 entries total
A: 267
B: ?
C: ?
D: ?
E: ?
F: 165
---Above six on ballot---
---All six are bloc entries---
G: ?
... ?

We also know something about the size of the voting bloc.

#3 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:14 AM:

I generated sample ballot data using the nomination totals from the 2013 Best Novel nominations, and assuming that anything that didn't get enough votes to appear on the list wasn't in contention. (They were left as "minor" votes on the ballots because they affect the outcome of SAV, though not the others.) I used the following assumptions:

* 40% of nominators listed 5 novels, 15% listed 4, 15% listed 3, 15% listed 2, 15% listed 1
* Votes were distributed randomly (no correlation aside from coincidence)

I have no idea how accurate the first assumption is, and I'm sure that votes aren't actually randomly distributed. In the absence of real data, I tried to go with a premise that would involve relatively few biasing assumptions.

I then looked at the following four scenarios. I've numbered the candidates based on the number of ballots they appeared on; so #1 would correspond to Redshirts with 193 first-place votes.

Using the randomized ballots with 2013 vote totals: The same five candidates are nominated. #3 and #4 swapped orders using RAV, but they were only 2 votes apart so that's not very significant.

Using another set of randomized ballots, but assuming that all the supporters of book #6 listed only book #6 on their ballots: Where the current method would still nominate #1-#5, All three methods selected #1, #2, #3, #4, and #6 - so with any of them, there are potential advantages to having your book listed by itself. Using SAV, it generated such a high satisfaction score that it would still have made the ballot with only half the votes. Using PAV and RAV, it was much closer - losing 7 votes would have dropped it off the RAV ballot, and losing 14 would drop it off PAV.

Using the randomized ballots, but adding 150 "puppy" voters who voted for books #2 and #14, as well as three other books that basically no one else voted for: (Apparently this is an alternate universe where the Puppies are all Seanan McGuire fans.) Since only #1 got over 150 votes, the current method would have nominated #1, #2, #14, and two of the three "Puppy only specials." All three of SAV, PAV, and RAV would nominate #1, #2, #3, #4, and #14 - so the slate does add #14 onto the ballot in place of #5, but all 3 methods handle the slate much more robustly than the current method.

Using the randomized ballots, but adding 300 "puppy" voters and 250 "anti-puppy" voters who vote for #12 and 4 works with negligible other support: The current system would nominate #2, #12, #14, and two puppy specials - so one work from the smaller slate and four from the larger. SAV nominates #1, #2, #12, #14 and a puppy special - so the top non-puppy work makes the ballot, though it's close to losing to another puppy special. I think PAV nominates #1, #2, #4, #12, and #14 - so two puppy works (one of which would have made it anyway), one anti-puppy work, and two non-slate works. The result of RAV depends on the weights used. With Sainte-Lague weights, it nominates the same candidates as PAV #1, #2, #4, #12, and #14. With d'Hondt or exponential weights, #4 gets booted in favor of an anti-puppy candidate, so each slate gets 2 nominees and the remaining nominee is the most popular non-slate candidate.

As above, but the "anti-puppies" vote a slate closer to the overall preference, slating #1, #4, #5, #7, and #12: The current system would nominate #1, #2, #4, #5, and #14 as the five candidates with the most votes. SAV nominates #1, #2, #4, #5, and #7 - so the slightly smaller slate that's more popular with non-slate voters now gets 4 nominations. PAV and RAV (using any system of weights) nominate #1, #2, #4, #5, and #14 - so in this case they actually agree with the popular vote totals.

As above with two slates, but 1/3 of the non-slate voters remove any slated work from their ballots: This assumes they don't replace them with non-slated works, which is probably unrealistic but makes the computation manageable. Very much to my surprise, the results are completely unchanged from the case above.

Predicting the winner using STV is tricky because we really have no information about what ranked ballots would look like. The quota would be 187 for the first two situations, so #1 would immediately be nominated, its 6 excess votes would be transferred, and then we would have to start eliminating the candidates with less support and transferring their votes. In the second case, whether #6 got nominated would depend on whether enough ballots got exhausted to put 101 votes in the top 5 vote counts.

For the 150-puppy case, the quota would increase to 212. Book #2 might meet this if a large number of puppies listed it first. Assuming that was the case, the 76 excess votes would be distributed among book #2's other supporters. This would probably leave the puppy group with too few votes to get another book on the ballot, though #14 would be an outside possibility.

With 300 puppies, the quota becomes 237. As long as the puppies are in reasonable agreement on a first choice, it would clear the quota even with no non-puppy support. Depending on vote distribution, enough votes might well transfer to nominate another puppy candidate, but more than that would still be fairly unlikely.

Note that STV is likely to nominate the puppies' favorite candidate, whereas the other methods favor slate candidates with the greatest non-slate support.

Please take this analysis for STV with an extra-large grain of salt, since I don't have a good enough idea of what the ballots would look like to convincingly simulate the election using this method.

#4 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:15 AM:

I did a smaller set of experiments based on the novelette category to see what happens when the nominations are more dispersed. There were a total of 616 votes cast; the first-place story received 89 votes, and the fifth-place story received 45. I distributed votes randomly, assuming in this case that equal numbers of nominators listed 1, 2, 3, 4, or 5 works.

Using the randomized ballots with 2013 vote totals: SAV nominated the actual #1, #2, #3, #4 and #8 - it received 36 votes, but they wound up heavily concentrated on ballots which didn't list #1-#4, which pushed it just ahead of #5 for the last spot. PAV and RAV (for all weights) matched the actual nominations of #1, #2, #3, #4, and #5.

Adding 100 "puppy" voters who voted for five stories that picked up 0-18 non-slate votes each Under the current system, the puppies would completely sweep the ballot (as happened this year). SAV, PAV, and RAV with D'Hondt or exponential weights all nominate #1, #2, #3, and 2 puppy picks. RAV with Sainte-Lague weights nominates #1, #2, #3, #4, and the top puppy pick.

Adding 200 "puppy" voters as above: Under the current system, this is a dominant puppy win. SAV would also yield a sweep for the puppies. PAV, and RAV with D'Hondt weights, yield a ballot with #1, #2 and/or #3 (they tie for the last spot), and three puppy candidates. RAV with exponential or Sainte-Lague weights nominates #1, #2, #3, and two pieces of puppy chow.

It's hard to predict the effects of STV here. Because the votes are so dispersed, a lot of ballots are going to wind up exhausted as lower-vote-count stories are eliminated. At least one slate nominee would be chosen, and I would guess not more than 2-3, but I'm definitely guessing here.

#5 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 07:15 AM:

Okay, so let's have a discussion. If you're commenting on a particular proposal, please use the option number (or numbers) so we can all more easily follow what you're talking about.

Please think about this for a while before commenting. Considered opinions are more valuable than snap judgments. And while this can be a theoretical discussion, opinions from non-experts are valuable and wanted.

Someone somewhere else should take up the discussion of whether it's better to do nothing, and whether changing the electorate is a good idea or not. And, as long as we're tinkering under the Hugos' hood, maybe we should talk about updating the IRV process of the final election as well. When there are such discussions in progress, please post links here so we can join it if we want to.

#6 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 07:17 AM:

Stating my preference....

I don't want to make voters rank nominees. I want nominating to be as easy as possible -- "It's okay that you haven't read everything, just send us names of works you think are worthy" -- and ranking makes nominating harder. I don't want to make anything public in real-time to encourage more campaigning and more strategic voting. I think adding a third round of any kind would make the election too hard to administer. And I think that simpler systems are more likely to make it through the business meeting.

My suggestion: Voters should be allowed to nominate up to four candidates (x=4), that the number of total nominees should be five (z=5) or more using the Next Two Rule, and that the nominees should be chosen using RAV with exponential weights: Option 3c-3.

Making x less than y ensures that a single slate can never dominate the nominees, while the specific choices of x=4 and y=5 make the new system as close as possible to the old system. The Next Two Rule allows nominees that just barely lose, something that I think is a goodness in any case. And while RAV is definitely harder to explain than PAV, it's easier to administer, Plus, there's no change for the voters: they just nominate works they feel deserving.

I am partially opposed to Option 4: Banning Slates. I can see how to detect slates, both in the received votes and by watching the Internet for campaigning. But I don't like hurting authors who are put on slates against their will, and have no way to verify if someone is on a slate willingly or not. And there are too many edge cases that worry me. On the other hand, it's a clean solution to the problem.

I am strongly opposed to Option 5: Making the Voting Tallies Public. It does not make the distributed and unorganized general electorate more powerful. Instead, it makes slates more powerful by giving them more information. And this is to the extent it's actually useful. Already most people vote at the last minute. And any smart slate would do the same. This is exactly the sort of change we should not make; it focuses on the details of this year's tactics rather than the broader problem.

No proposal is going to be immune from strategic voting, and no proposal other than banning voting blocs outright is going to eliminate their effectiveness. Our goal should be to agree on a voting system that reduces the influence of voting blocs, even as it is accepts their power.

#7 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:18 AM:

Thoughts and Recommendations

Most of the options listed here would be at least somewhat effective at keeping a slate from completely dominating the Hugo ballot. Here are my thoughts based on playing with the (faux) ballot data:

Option 1: Fewer nominations per voter. This would make it much more difficult for a slate to completely dominate the ballot. On the other hand, by reducing the number of nominees a non-slate voter can list, it may actually increase a slate's ability to get x candidates onto the ballot. At best, we'd be guaranteed a ballot with a couple non-slate candidates (and once multiple slates start showing up, non-slate candidates are going to have a very tough time).

Several of the voting methods in Option 3 actually allow a voter to list a theoretically unlimited number of candidates and can yield a better result in consequence. I think going in that direction or keeping the number at five are better choices for us.

Option 2: Nominate more candidates. This effectively moves more of the decision process to the second election, which is less vulnerable to slates. If multiple slates appear, it would be very easy to end up with a ballot in which every option is from a slate. I also have concerns that the length of the Hugo ballot would mean a big increase in voters who hadn't read most of the nominated work, or a decrease in voters as people find the list too overwhelming.

Option 3a: SAV. Having run the simulations and looked at some of the literature, I'm not enthusiastic about this method. It doesn't do enough to stop slates, and it makes the strategy of listing only one candidate too powerful.

Option 3b: PAV. This system was reasonably effective at maintaining representation for non-slate voters. However, I can't tell you with certainty who wins for some of the examples, because my computer doesn't have the processing power to examine all the possible candidate sets. One paper offers a proof that the problem is NP-Hard (though I'm not enough of an expert in the area to validate the proof). To quote my CS professor, "If you don't know what NP-Hard means, it means really hard. It's bad. NP-Hard is bad."

One option would be to use PAV after dropping all candidates under a certain threshold or above a certain number from the ballot. This makes computability less of an issue, but further complicates an already difficult-to-explain system.

Option 3c: RAV. This system was reasonably effective at maintaining representation for non-slate voters. D'Hondt weights were both the least effective in the area and have some undesirable theoretical properties. Sainte-Lague weights controlled slates most effectively, but exponential weights were close and might be easier to explain ("every time you get a nominee, your vote counts half as much" versus "after the first nominee, it counts 1/3, then 1/5, then 1/7, and so on..."). While the system increases the workload of computing the nominees, I found it easier than any of the other Option 3s.

Option 3d: STV. This is the system I was least able to experiment with, due to its dependence on ranked ballots. It would considerably dilute the power of a slate after it gets its first nominee, but it would increase the load on nominators by requiring a ranked list. It is used for Academy Award nominations so there's some history of its use for this sort of purpose, but I don't think they release their underlying data pretty much ever. (Also, they have the resources to contract Price, Waterhouse to do the actual computations. We probably don't.)

Option 4: Banning Slates. I can see two ways to do this: lay out in exacting detail what a slate is (which means that people will immediately game their way around the definition), or leave identifying slates up to the Hugo administrators, which is a responsibility I suspect most concoms do not want and are not prepared for. I'm all for a statement in principle that slates are inconsistent with the goals of the Hugo Awards, but enforcement is a thornier problem.

Option 5: Public Voting Tallies. In general, this would make tactical voting easier since some information about the state of the election would be public. Encouraging tactical voting means we'd have less information about people's actual opinion, but we might get a better outcome than in the slate situation where only one group votes tactically. That leaves me two primary areas of concern. First, the method is very vulnerable to bad actors. Voting for one candidate then switching their votes to another could confuse the picture of where actual preference are (and potentially allow them to gauge the difference in votes between candidates). A DDoS attack on the servers near the deadline could also cause havoc. (There'd be some danger of the servers going down as everyone logged in to cast their final votes anyway.) Finally, it may not be practical to assemble a list of all possible candidates in advance of launching the site (and a partial list would be a de facto slate). This means that, as mentioned in the previous thread, the administrators will need to curate submissions to identify identical ones and maintain the list. This becomes a more burdensome responsibility when it needs to be done in real time, or even weekly.

Of these options, then, I would recommend RAV with Sainte-Lague or exponential weights as the change most likely to limit the power of voting slates without too many negative side effects, with STV as a runner-up.

One paragraph of non-voting-system stuff: no change to the voting system will eliminate the power of slates entirely. In my opinion, a rules change barring slate voting is unenforceable and would enmesh future Hugo administrators in a bog of controversy. Social pressure against slates (e.g. non-slate voters declining to nominate or vote for slated works, a statement that slate voting undermines the premises of the Hugo) might help, especially in combination with a voting system change that limits the power of slate. These voting system changes would tend toward producing a more diverse group of nominees, which ought to make the more reasonable SPs happy (assuming they actually exist).

While I have some experience working with voting systems, I am hardly an expert and you shouldn't take my word as authoritative. If you see any errors in any of this work, please correct me. I'm happy to share the spreadsheet I've used to run these simulations; I'm also happy to simulate other situations, especially if you send me a set of ballot data. (Generating "plausible" ballot data is the most time-consuming part.)

I look forward to a lively discussion of the different options! Thank you for listening.

#8 ::: John A Arkansawyer ::: (view all by) ::: April 13, 2015, 07:51 AM:

Cheradenine @ 7: "Social pressure against slates...might help, especially in combination with a voting system change that limits the power of slate."

"once multiple slates start showing up, non-slate candidates are going to have a very tough time"

And in those two statements, you have made the case for acting with some urgency. For better or for worse, social pressure can prevent change much more easily than it can reverse it.

I started to present a case where there were more slates than there were nomination slots--the Urban Fantasy slate, the Young Adult slate, the Best-Selling Authors slate, the MilSF slate, the Social Justice slate, the Slipstream slate, the Alternate History slate--when I realized that if things got to that point, there was probably no good solution and no going back.

I don't want to run that experiment.

#9 ::: J Thomas ::: (view all by) ::: April 13, 2015, 08:09 AM:

Looking at the comments in this topic so far, I can imagine the following situation arising:

Someone looks at the votes for nominations, and they announce:

"These three candidates got the 2nd, 3rd, and 4th most votes. But none of them are on the list of nominations. Why not?"

And then someone else will have the responsibility to explain the voting system in a way that the general public will understand.

If it stays controversial in 2017, when the new system comes in, it's possible that Fox News might spend a few minutes on it, and somebody involved with the voting might have 15 seconds or 30 seconds to explain why the voting system is OK.

Maybe the best way to deal with this would be to keep the votes secret until nobody cares.

#10 ::: nathanbp ::: (view all by) ::: April 13, 2015, 08:45 AM:

Bruce, #6: If we're using RAV with exponential weights (Option 3c-3), which is my preference as well, then if someone's 5th vote counts it will be as 1/16 of a vote, so I feel like there's not much of a point to limiting people to 4 nominations per category instead of leaving it at 5.

Also, looking at the 2013 data, the Next Two rule does not seem like a good idea. Since most categories have no sharp cliff of votes but instead decrease gradually, in almost every category it picks all 15 works listed to be nominees (and would probably continue on to almost the end of the list). Using the 60% threshold as proposed towards the end of that paper would have helped in some categories, but not all of them.

On option 5, what if the current nomination status was released just once during voting, at say one month before voting closes? Release a list of the top 15 in each category in alphabetical order without vote counts. I feel like this would help to concentrate votes without giving out too much information that could be used strategically or being subject to gaming.

#11 ::: Vivien ::: (view all by) ::: April 13, 2015, 08:47 AM:

I mostly agree with Bruce Schneier :

RAV works as well as anything we could try. Difficult but not impossible to explain, especially to a community that is using something as complex as IRV for the final election. Testable on previous data.

Exponential coefficients are the easiest to explain to someone who doesn't know what a suite is.

Next two rules : interesting, and remarkably polemic-free.

Nominate 4, 5 on the final list : well, it is not useless, but that is maybe the extra-rule that will get your average WSFS member to tell you to leave him alone with your damned rule changing.

#12 ::: Vivien ::: (view all by) ::: April 13, 2015, 08:48 AM:

Ank thank you Cheradenine for the simulations, they are great !

#13 ::: Chris Lawson ::: (view all by) ::: April 13, 2015, 09:10 AM:

Cheradenine -- good work. I'd love to see someone with better programming skills and a regular website take this even further and put up an online simulator of the different voting options using the 2013 and 2014 nomination data to compare how each system would work out.

Bruce S. -- re: banning slates: if we can detect slates statistically, I suspect we can also estimate the proportion of a book's nominations due to the slate, in which case it is possible to ban a slate without hurting an author by removing only that proportion of nominations deemed to be slate-dependant.

#14 ::: Galen Charlton ::: (view all by) ::: April 13, 2015, 09:21 AM:

John A Arkansawyer @ 8: I started to present a case where there were more slates than there were nomination slots--the Urban Fantasy slate, the Young Adult slate, the Best-Selling Authors slate, the MilSF slate, the Social Justice slate, the Slipstream slate, the Alternate History slate--when I realized that if things got to that point, there was probably no good solution and no going back.

I share your hope that we don't end up with this state of affairs, but it occurs to me that if we do, there may be a small upside. The appearance of (say) a YA slate could be interpreted as a signal of a desire for a new Hugo Award category. That suggests to me an orthogonal sort of change: accommodating slates that exist to promote sub-genres by encouraging each Worldcon to use its privilege to name a special award category. This would allow us "try out" potential permanent categories and, in the long run, give more opportunities for folks who feel that their favorite subgenre is underrepresented to nominate.

As I suspect that this comment is almost, but not quite, off-topic for this particular post, perhaps any follow-ups should go in the "Clean Living" thread.

#15 ::: Kimiko ::: (view all by) ::: April 13, 2015, 09:24 AM:

I do not know what the best solution is, but the most straightforward implementation looks like the one Bruce outlined above:
My suggestion: Voters should be allowed to nominate up to four candidates (x=4), that the number of total nominees should be five (z=5) or more using the Next Two Rule, and that the nominees should be chosen using RAV with exponential weights: Option 3c-3.

Making x less than y ensures that a single slate can never dominate the nominees, while the specific choices of x=4 and y=5 make the new system as close as possible to the old system. The Next Two Rule allows nominees that just barely lose, something that I think is a goodness in any case. And while RAV is definitely harder to explain than PAV, it's easier to administer, Plus, there's no change for the voters: they just nominate works they feel deserving

There's also something elegant about using exponential weights. I think implementing such a thoughtful system would combine nicely with an anti-slate social norm. "Look at this cool thing we built so people won't wreck our awards. Isn't it shiny? Let's not try to break it, because that would be uncool."
I'm going to go back through the examples given and see if I can understand how anti-slate voting interacts with it.

#16 ::: Bruce Schneier ::: (view all by) ::: April 13, 2015, 09:30 AM:

nathanbp @10:

"Bruce, #6: If we're using RAV with exponential weights (Option 3c-3), which is my preference as well, then if someone's 5th vote counts it will be as 1/16 of a vote, so I feel like there's not much of a point to limiting people to 4 nominations per category instead of leaving it at 5."

Makes sense. I agree.

"Also, looking at the 2013 data, the Next Two rule does not seem like a good idea. Since most categories have no sharp cliff of votes but instead decrease gradually, in almost every category it picks all 15 works listed to be nominees (and would probably continue on to almost the end of the list). Using the 60% threshold as proposed towards the end of that paper would have helped in some categories, but not all of them."

Interesting. Maybe we should just expand nominations to 6 or 7.

"On option 5, what if the current nomination status was released just once during voting, at say one month before voting closes? Release a list of the top 15 in each category in alphabetical order without vote counts. I feel like this would help to concentrate votes without giving out too much information that could be used strategically or being subject to gaming."

I really dislike this. It's too much responding to the tactics, and not looking at the broad picture. The risk of unforseen consequences are great.

#17 ::: Martin (not a glyptodont) ::: (view all by) ::: April 13, 2015, 09:33 AM:

I am also in favor of 3c3, plus an "honor pledge" that you haven't voted a slate. Don't include any enforcement details - if good faith + social pressure is not enough to stop Hugo slates, Hugo slates are probably a given from now on and folks should be laying infrastructure for their Hugo Political Party rather than trying to save the old way.

More testing of various methods vs. multiple (at least 2 and 3) slates is warranted. I would be surprised if there was much hope for independent voters under any system in the 3 slate arena.

As for hope of implementation, I think the following approach is important - the anti-slate movement cannot be a specifically anti-Sad-Puppy movement. They to want to stop a shadowy cabal - and anti-slate is also anti-shadowy cabal. Rabid Puppies are not going to help us here, but if we do it right the Sad ones might. Since the whole point is to return the Hugos to good will and general consensus, working with everyone who is even minimally work-with-able is sort of required.

#18 ::: Reimer Behrends ::: (view all by) ::: April 13, 2015, 09:36 AM:

I note that if there is a voting bloc that has more than five times as many members as the otherwise most popular candidate in a given category receives votes, they will always be able to get five nominations in that category under all current option 3 proposals (by splitting their votes evenly between their five preferred candidates and casting only one vote per ballot); smaller voting blocs can unavoidably still obtain 1-4 nominations. While it is difficult to coordinate voting behavior to obtain such an even split, there are several ways in which it can be approximated with relatively little overhead. The primary benefit of slate voting over this approach currently is reduced recruitment effort and that it is cheaper; you get essentially five votes for the price of one.

This is why I rather like what Joshua Kronengold calls "Single Divisible Vote" (which is essentially a variant of unranked STV), because it eliminates that incentive by making each ballot is worth one vote; to get the current benefit of slate voting, you'd have to cast five ballots.

#19 ::: Randolph ::: (view all by) ::: April 13, 2015, 09:45 AM:

Has anyone considered returning to a mailed-in paper ballot for nominations? That would keep out the internet trolls, at the cost of some serious inconvenience. Such a ballot could still be gamed, but it would be a lot more difficult.

#20 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 10:05 AM:

Vivien @12, Chris @13: You're welcome! It was very interesting to look at what happened in more depth (even if it did kind of eat my weekend). Chris, I'd be delighted if someone wanted to take the website idea and run with it (my programming skills are sadly limited).

I do really want to track down some genuine data and see how the system performs. My hunch is that in the absence of a slate RAV is highly likely to yield the same set of nominations as the historical set, but I'd like more to go on than the hunch. I know someone in the other thread thought they could get their hands on the 1984 ballot information - I await with bated breath.

Reimer @18 - In 2013, at least, the leading nominee in each category had at least 14% of the vote. If a voting bloc controls more than 5 times as many voters as that, they're already pretty much in control of the election.

I can take a look at an SDV simulation if that would be helpful. (Not while I'm at work, though!)

#21 ::: J Thomas ::: (view all by) ::: April 13, 2015, 10:20 AM:

Given the goal to discourage slates, it might be best to actually discourage slates. When you discourage something that sort of correlates with slates, you can get unexpected side effects even while you don't always hit the slates you want to.

So here is an approach to do that.

We filter the ballots.

For each voting category with five slots, eliminate any ballots which are identical. Any ballots which are completely unique -- no one else voted for any of these nominations -- get no further filtering.

Ballots which are identical except for one item, get counted as 1/5 votes.

Ballots which are identical except for two items get counted as 2/5 votes.

Ballots which are identical except for three items get counted as 3/5 votes.

Ballots which agree on one item count as 4/5 votes.

This will eliminate any value from slate voting. People may however try to do strategic nominating by selecting one nominee they want to push, and add four other nominees they hope that no one else would think of nominating.

It would be a courtesy for the automated filtering system to send each nominator a message telling them how much their vote counted.

#22 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 10:28 AM:

J Thomas @21: I think this is likely to cause considerably more collateral damage to non-slate nominators than something like RAV. Keep in mind that a significant portion of nominators don't list five items, which significantly increases the odds of their ballots being partially or completely canceled out.

#23 ::: J Thomas ::: (view all by) ::: April 13, 2015, 10:35 AM:

Keep in mind that a significant portion of nominators don't list five items, which significantly increases the odds of their ballots being partially or completely canceled out.

Tell them to list five items.

Or count blanks as misses, so that someone who lists only one nomination can expect it to count as 80% of a vote.

#24 ::: Seth Gordon ::: (view all by) ::: April 13, 2015, 10:43 AM:

Many thanks to Bruce and to Cheradenine for applying your math-fu to this cause. I also want to co-sign Martin’s remark above that a voting method that resists slate voting will also resist shadowy cabals, and therefore is a Good Thing, whether or not you think there is a shadowy cabal that needs resisting.

Option 1: I would rather preserve as many of the existing rules as possible (it seems to make both good political and engineering sense), so let’s leave this alone unless we absolutely have to.

Option 2: I’m not keen on giving Hugo voters even more stuff to read (it increases the temptation to skip the reading and vote based on shallow considerations, which is of course one of the things we are trying to prevent). But see my response to option 4 below.

Option 3: I am torn between 3a, which seems easiest to explain, and 3c, which seems most slate-resistant.

Option 4: I don’t like the idea of banning slates by fiat, because I can see future Hugos descend into meta-controversies over whether such-and-such a group of candidates is or is not a slate. However, I think it might be worth giving each Worldcon the discretion to add up to three(?) nominees to each category if it sees irregularities in the nominating process, including but not limited to slate voting. (This can be proposed as a rule change independent of Option 3.) Voters who think these special nominees did not deserve such honor can, of course, express their disapproval by ranking them below you-know-who.

Option 5: This is an anti-gaming tactic which can, itself, be gamed aggressively, so I am against it.

PS: I am intrigued by the Majority Judgment system that Jameson Quinn proposes here. It’s too radical a change for the Hugos themselves, but perhaps another large convention (DragonCon or ComicCon) might be inspired to try it out, or a Worldcon could use it to elect a “Special Award of Merit” independent of the Hugos.

#25 ::: nathanbp ::: (view all by) ::: April 13, 2015, 10:51 AM:

Bruce, #16: "Interesting. Maybe we should just expand nominations to 6 or 7."

Looking at the nomination counts they mostly seem to gradually trail off, so I'm not sure that any algorithm would be able to pick a reasonable stopping point.

Reimer Behrends, #18: I'm not sure what you're trying to say? Under SDV any group with 5x as many members as the next most popular work gets all 5 of their picks on the ballot too, and without having to guess what the final totals will be. Under RAV if they try the 5 way split they can end up with no works on the final ballot. If they vote for all works under RAV they'll get 3 of the final slots.

Example (I'm simplifying and assuming no cross work support for QRSTV):
101 votes for ABCDE - The Slate
20 votes for Q
19 votes for R
18 votes for S
17 votes for T
16 votes for V

Without the slate QRSTV are nominated.
With the slate under SDV, ABCDE each get 20.2 votes and are nominated (similar result from STV).
With the slate under RAV, ABC win with 101, 50.5, 25.25 votes each, then Q and R get the remaining 2 nominations.

J Thomas, #21: I agree that the listed systems don't do a lot to punitively punish slates, instead just limiting them to something like their appropriate power given the number of voters they command. I think if you want to do something like that the simplest system is Bruce's #1 on the old post (once one thing on a ballot wins, it doesn't count for the rest of the process).

#26 ::: TomB ::: (view all by) ::: April 13, 2015, 11:11 AM:

Option 4: Banning Slates:

We know that slates are bad because they disenfranchise non-slate nominations.

Banning slates will discourage the promotion of slates, encourage voters to nominate their own personal preferences, and invalidate arguments that slates are allowed within the rules.

Enforcing the rule is complicated. The simplest solution would be to enforce it by using one or more of the other options. You might ask then, why bother banning slates? Because the other options can at best only mitigate slates and do nothing to prevent them. It's like trying to stop spam without having any rule that spam is not allowed.

#27 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 11:16 AM:

Seth Gordon @24:

...a voting method that resists slate voting will also resist shadowy cabals, and therefore is a Good Thing, whether or not you think there is a shadowy cabal that needs resisting.
There are no shadowy cabals. There never have been. This is a fact. Facts matter.

#28 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 11:20 AM:

If one set of works has 5x as many votes as any others, it should win. If a work that is that massively popular compared to the others *fails* to get on the ballot, that's a problem.

#29 ::: Steve Halter ::: (view all by) ::: April 13, 2015, 11:26 AM:

3c-3 seems to fit the various goals the best. Setting x=5 (number of nominations) seems simplest and is not much different than x=4 under 3c-3.

I don't really like a precise rule banning slates as it would seem to leave room for too much gaming--positive or negative. Some sort of non-binding statement that voters should read things and vote for what they personally like could be useful in the voting guide.

I would be against making the current tallies public (Option 5).

#30 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 11:26 AM:

So, let's turn these scenarios on their head: they're all assuming that the slate is from a committed minority of voters. Let's say that the vast majority of voters vote for a slate. The slaters have a 2:1 majority over the non-slate voters. I would argue that in this case, the slate should just win.

How does the slate fare under the various schemes listed here if this is the case? Assuming that it does not succeed in getting all of its nominees in, how much of a majority does a slate need to have before it gets all of its members elected under the various 3.x schemes?

#31 ::: Pfusand ::: (view all by) ::: April 13, 2015, 11:30 AM:

"Maybe we should just expand nominations to 6 or 7."

This would be fine for short stories, but a real killer for novels.

#32 ::: J Thomas ::: (view all by) ::: April 13, 2015, 11:42 AM:

Option 5: Public Voting Tallies. In general, this would make tactical voting easier since some information about the state of the election would be public. Encouraging tactical voting means we'd have less information about people's actual opinion, but we might get a better outcome than in the slate situation where only one group votes tactically.

I want to stress a basic point: To deal with slates etc, we can find a way for the good people to come to agreement -- not a bad way where they just play politics, but a good way where they notice which of the things they like are popular enough to win.

Or we can look for ways to manipulate the vote or the vote-counting so that the bad people who have voted together to embarrass us do not have their votes counted.

There is nothing morally wrong with the first approach. But the good guys might not in fact reach agreement, and then they could lose. It might be more reliable to find a way to count the votes so the bad guys cannot win even if the good guys don't vote for competitive candidates.

That leaves me two primary areas of concern. First, the method is very vulnerable to bad actors. Voting for one candidate then switching their votes to another could confuse the picture of where actual preference are (and potentially allow them to gauge the difference in votes between candidates).

If it's done by, say, acceptance voting then there's a limit to the bad guys' ability to game the system. They can vote for somebody they don't like to see how it changes the numbers, but they can't unvote afterward.

A DDoS attack on the servers near the deadline could also cause havoc. (There'd be some danger of the servers going down as everyone logged in to cast their final votes anyway.)

Could you possibly accept, say, email ballots sent to a prominent mailserver? You get a timestamp even if you don't get the actual ballots until later. It would be harder to DDoS them than you. There might be other ways to palliate the problem.

Finally, it may not be practical to assemble a list of all possible candidates in advance of launching the site (and a partial list would be a de facto slate). This means that, as mentioned in the previous thread, the administrators will need to curate submissions to identify identical ones and maintain the list. This becomes a more burdensome responsibility when it needs to be done in real time, or even weekly.

How is that handled now? Do the administrators have a list of all possible candidates in case somebody nominates one of them? Do they have a truncated list and nominations have to come from it?

If some nominator or kibitzer suspects that two nominations are the same thing but misspelled or something, won't they tend to announce it? News often spreads by the fastest available channel....

I don't see what's different about making nominations between the old system and this proposal.

#33 ::: Clay Shentrup ::: (view all by) ::: April 13, 2015, 11:43 AM:

The best option is Proportional Score Voting, aka Reweighted Range Voting.
http://scorevoting.net/RRV.html

It's simpler than STV and superior in the sense that it satisfies a proportionality theorem. It can be tabulated using a Google spreadsheet and a few simple formulas.

The simplest form Proportional Score Voting is Proportional
Approval Voting, described in your "Option 3c". Here's a video where I demonstrate how to tabulate this method in Google Spreadsheets.
https://www.youtube.com/watch?v=_jS7b-0PV9E

> It should be fair.

This is a basically meaningless desideratum frequently referenced by people not versed in social choice theory. Suppose we can select option X, which will give Bob and Alice utilities of 5. Or we can select option Y, which will give Bob a 6 and Alice a 7. Do you really want to choose X (thereby making Bob and Alice BOTH less happy) in the name of "fairness"?

Answer: NO. A rational organism's goal is to maximize its expected utility, which means maximizing the net utility of the group. You do NOT want to maximize "fairness", because it is a nonsensical/meaningless concept in the first place.

Clay Shentrup
Co-founder, Th Cntr fr lctn Scnc

#34 ::: TomB ::: (view all by) ::: April 13, 2015, 11:44 AM:

Bruce Schneier @6: "Already most people vote at the last minute. And any smart slate would do the same."

I proposed a fix for that in #569 on the original thread.

To prevent sniping, extend the deadline until the moving average of the nomination rate goes below a threshold.
In practice, the deadline will always be extended at least one day because of normal last-minute nominating. If there is a controversy, the various parties will try to get more people to nominate. The nominations will remain open until all parties are exhausted and the nomination rate goes down.
To prevent endless procrastination, charge a late fee to nominate after the initial deadline.
With the fix, Option 5 would still turn the nomination process into a tactical GOTV process, but at least it would not be broken.

#35 ::: Douglas Henke ::: (view all by) ::: April 13, 2015, 11:44 AM:

The endgame of Option 5 is that "vote this slate" becomes "download and run our software agent to automatically vote our slate during the last possible tens of milliseconds". See also eBay sniping software.

This means that, while the nomination process is in progress, the block voters have perfect information about the behavior of the honest voters, while the honest voters have zero information about the behavior of the block voters. This is a pessimal outcome.

Option 4 comes in two flavors. Either the definition of a slate is objective (4a) or subjective (4b).

4b immediately fails the "be fair", "look fair" and "be easy to administer" requirements. If the test for something being a slate is to look like one in the personal opinion of one or more people, then anyone supporting something declared to be a slate will -- probably, rightly -- cry foul.

4a requires a specific objective test. It seems as though any such test would itself be game-able, would risk false positives, or both. In any case, I would suggest that anyone proposing 4a need also propose a specific test in order for their proposal to be seriously considered.

(Aside: My comment 424 in the previous thread suggested a test for 4a: multiple ballots highly correlated across multiple nomination categories. This fails the "easy to explain" requirement -- do you want to stand up in front of the WSFS business meeting in Spokanistan and start talking about eigenvectors?)

#36 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 11:46 AM:

Thoughts

Option 1: I agree with nathanbsp@10 that if we're using RAV, especially exponential RAV, there's no point in limiting the number of nominees per ballot. And allowing more than 5 could help promote more overlap between non-slate voters in "long tail" categories such as "short story". This would help more nominees pass the 5% threshold. Also, since it means more non-slate votes overall, it increases the number of slate votes that would be required to get a given number of slate-based nominees. (This is still treating slate- and non-slate- votes fairly; it's just that permitting more nominations naturally does different things for those two cases).

Option 2: I like the basic idea of Bruce's suggestion of an adaptive number of nominees, but the next-2 rule¹ is clearly not right in this case. That rule was designed for a different ballot format, and on this data, it could easily go crazy and include all the nominees.

There are two goals for having an adaptive rule.

First, it's good to deal with a "long tail" situation. Imagine a category where the top nominee is only on 10% of the nominating ballots. Then, there's no way to limit it to 5 nominations without ignoring over 50% of the ballots. In order to deal with this, if you were using RAV, you could have a rule about not quitting until the total remaining ballot weight is below some proportion p — say, 60%-70% — of the original number of ballots. That is another way of saying that at least 2(1-p) of the ballots have at least one nominee they supported. So, if you set the threshold at 60%, you're adding nominees until you've listened to at least 80% of voters; at 70%, you're listening to at least 60%.

The second goal for an adaptive rule is to limit cases of "almost made it". So you don't want the between the lowest nominated work (call this "work n") and the biggest non-nominated work (call this "work n+1") to be "small" in some sense. One reasonable rule would be that this gap should have to be bigger than the gap between work n+1 and work n+2, or else you set n

#37 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:48 AM:

Fred @30: If 60% of voters vote for a slate and assuming that other voters are not voting a counter-slate, the likely outcomes are:

Options 1-2: The slate gets a number of candidates nominated equal to the number they can vote for.

Option 3a: Probably elects all of the slate candidates.

Option 3b: Probably elects all of the slate candidates.

Option 3c: With d'Hondt weights, probably elects all of the slate candidates. With Sainte-Lague weights or exponential weights, probably elects 3-4 of the slate candidates.

Option 3d: Probably elects 3 of the slate candidates.

Under 3c or 3d, the 3 slate candidates who are most popular with non-slate voters will be nominated, and presumably one of them will win the election. Under 3c, it is possible that the 1st choice of the slate voters won't make the final election if it is very unpopular outside of the slate, though this is not a likely scenario. If the slate voters have a strong 1st choice, they might be better off *not* voting in lockstep on the remaining candidates.

#38 ::: J Thomas ::: (view all by) ::: April 13, 2015, 11:50 AM:

There are no shadowy cabals. There never have been. This is a fact. Facts matter.

People might wonder how you know that. If you aren't a member of a shadowy cabal, how can you be sure whether it exists or not.

I want to assure you about this. I am a member of a couple of shadowy cabals and so I am in a position to affirm that there are no shadowy cabals and there never have been. I know.

#39 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:52 AM:

Clay @33: I'm not sure how voters can meaningfully assign scores in a nomination election when they don't have access to the entire field to compare it to, so range voting strikes me as problematic here. Simple approval ("yes, I want this item to be on the ballot") strikes me as preferable here.

#40 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 11:53 AM:

Sorry, my previous post got borked by a stray less than sign. Here it is again:

Thoughts

Option 1: I agree with nathanbsp@10 that if we're using RAV, especially exponential RAV, there's no point in limiting the number of nominees per ballot. And allowing more than 5 could help promote more overlap between non-slate voters in "long tail" categories such as "short story". This would help more nominees pass the 5% threshold. Also, since it means more non-slate votes overall, it increases the number of slate votes that would be required to get a given number of slate-based nominees. (This is still treating slate- and non-slate- votes fairly; it's just that permitting more nominations naturally does different things for those two cases).

Option 2: I like the basic idea of Bruce's suggestion of an adaptive number of nominees, but the next-2 rule¹ is clearly not right in this case. That rule was designed for a different ballot format, and on this data, it could easily go crazy and include all the nominees.

There are two goals for having an adaptive rule.

First, it's good to deal with a "long tail" situation. Imagine a category where the top nominee is only on 10% of the nominating ballots. Then, there's no way to limit it to 5 nominations without ignoring over 50% of the ballots. In order to deal with this, if you were using RAV, you could have a rule about not quitting until the total remaining ballot weight is below some proportion p — say, 60%-70% — of the original number of ballots. That is another way of saying that at least 2(1-p) of the ballots have at least one nominee they supported. So, if you set the threshold at 60%, you're adding nominees until you've listened to at least 80% of voters; at 70%, you're listening to at least 60%.

The second goal for an adaptive rule is to limit cases of "almost made it". So you don't want the between the lowest nominated work (call this "work n") and the biggest non-nominated work (call this "work n+1") to be "small" in some sense. One reasonable rule would be that this gap should have to be bigger than the gap between work n+1 and work n+2, or else you set n to n+1 and try again.

Let's test these rules looking at "Novelette" from 2013. Since we have no reason to believe there was too much slate voting that year, let's assume that the correlations between any two works are essentially zero, so that RAV is basically not a factor:

Best Novelette
Total ballots: 616
A: 89
B: 62
C: 61
D: 54
E: 55
F: 38
G: 37
H: 36
I: 35
J: 31
K: 30
L: 28
M: 22
N: 22
O: 20
P: 20

I've copied Cheradenine and put this in a spreadsheet to get the following results:

Depending on the overlaps, the first rule ("make sure you listen to around 60% of the electorate, by continuing until the total remaining ballot weight is under 70%") would choose 9 nominees (up to H). The second rule ("Start with 5 nominees, then try to avoid an almost-made-it") would have actually happily stopped at 5 (up to E). (The possible stopping points are E, I, L, and N). The combination of both rules (1 then 2) would have chosen 10 nominees (up to I). If you use a slightly weaker form of rule 1, where you require a total weight under 75% (that is, "try to listen to at least half"), that would suggest stopping at F, which rounds up to I if you add rule 2 (because of the unusual near-5-way-tie).

Hmm... 10 works sounds like a bit much. So, how about using rule 2, but comparing the gaps (work 5, work n+1) and (work n+1, work n+2). This would mean rule 2 would rarely add nominees beyond the 6th. So combining this with rule 1 at 75% threshold would give 6 nominees that year, with the 6th a marginal inclusion to ensure that at least half of the voters had a say. Work G could complain that it was almost as good as F, but the response would be that F was lucky to get in in the first place, so G can't really argue it "almost deserved" a spot by comparing itself to F.

So I'd suggest that combined adaptive rule: make sure at least 50% of the ballots have at least one nominee, then use the second version of rule 2 above.

OK... that's long enough for one comment, so I'll comment on options 3, 4, and 5 separately.

¹ Actually, both of the authors for the Next Two Rule paper are on the advisory board of my organization, the Center for Election Science. (I hope it's OK to mention the name of the organization. Normally, I'd say the website there; we usually refer to the organization by its website "nickname". But in the last thread I was slapped down for using the website too much. If admins think it's inappropriate to even name the organization, go ahead and disemvowel the name; it will be still basically readable, and I'll get the hint).

#41 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:00 PM:

@31: Generally, the rules I proposed would leave the novel category at around 5.6 average nominees, while something like Short Story might average as much as 7 or 8. That's because voting on novels generally has less of a "long tail". So the rules naturally achieve what you want here.

#42 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 12:02 PM:

If looking at a rule to add more nominees to the ballot, looking at the 2013 short story category may be worthwhile. The vote distribution was

A: 107
B: 38
C: 34
D: 30*
E: 28
F: 28
G: 28*
H: 28*
I: 24
I: 24*
J: 23
K: 21
L: 20
M: 19
N: 19
O: 19
P: 19

Assuming no correlation, we'd need 9 entries to get to 50% of the ballots. But D, G, H, I are likely to overlap because they're by the same author, so we might need to go past 9 to get there.

(This assumes, though, that we don't retain the 5% rule. If the 5% rule is retained, we end up with the 3 nominees they actually had.)

#43 ::: Reimer Behrends ::: (view all by) ::: April 13, 2015, 12:10 PM:

Cheradenine @ 20: Yes, I know (except for categories where the vote is really splintered and few candidates even get 5%). Note, however, that's only for controlling ALL the nominations. It takes fewer votes to control 2-4.

I've also done some quick simulations, SDV seems to be doing reasonably okay in some typical setups, though my tests have been hardly exhaustive.

nathanbp @ 25: I'm saying that this is the point where it becomes pointless trying to make the system even more anti-slate. Moreover, organized voting blocks may then shift strategy away from slate voting towards a different approach that e.g. RAV may be poorly suited to deal with. RAV with exponential or Sainte-Laguë weights really penalizes the last votes on the slate, at which point it becomes more efficient to allocate votes differently rather than just voting a slate. At this point, it becomes more difficult to analyze how well RAV works vs. voting blocs that are flexible enough to adjust their strategies (because they won't bother voting five-nominee slates but pick a different approach). In short, penalties for slates that are too severe may prove ineffective, while also hurting normal voters.

#44 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:13 PM:

@33: Clay, I got this.

One of the leading proposals is reweighted approval voting, which is like reweighted score voting except it's simpler for the nominators. And you're right that "fair" can be ill-defined and some possible definitions are bad, but I don't think anybody here is suggesting anything equivalent to the particular bad definition you give, so you're straw-manning.

The Center for Election Science (again, usually I'd use the web site nickname) is already in this thread, making its points. You're welcome to join the discussion, but helicopter comments are not helpful.

#45 ::: Reimer Behrends ::: (view all by) ::: April 13, 2015, 12:16 PM:

nathanbp @25: To expand on my previous comment, in your example, the slate voters could switch to 1/5th voting for A, B, C, D, E each, securing 20+ votes per candidate, rather than voting the slate, thus working around RAV.

#46 ::: albatross ::: (view all by) ::: April 13, 2015, 12:26 PM:

RAV seems like a nice option, but I am a little concerned about the strategic voting aspects. Suppose I want to nominate A, B, and C, but I expect A to be heavily nominated--I have an incentive to leave A off my nomination, so my nominations for lesser-known works B and C get more weight.

I don't have much intuition for how much this matters--nor whether it matters more in some categories than others. It's not something we can detemine from simulations using previous years' data, without some additional assumptions.

This is the cost of not ranking the nominations, right?

#47 ::: J Thomas ::: (view all by) ::: April 13, 2015, 12:33 PM:

The endgame of Option 5 is that "vote this slate" becomes "download and run our software agent to automatically vote our slate during the last possible tens of milliseconds".

This means that, while the nomination process is in progress, the block voters have perfect information about the behavior of the honest voters, while the honest voters have zero information about the behavior of the block voters. This is a pessimal outcome.

If the block voters are a minority, and they get to watch a majority agree on other winners while they haven't gotten started yet, then they lose.

Unless they think the majority would vote to stop them and not vote with them, they do better to look like a viable choice early on.

If the good guys find choices they can stand behind, a secret group of bad guys cannot compete against them. Unless there are a whole lot of bad guys. (If the Puppies show up next year with 3000 ballots they'll probably win regardless.)

What gives the bad guys their power is that they agree, while the good guys do not agree.

#48 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 12:35 PM:

Reimer @43: The research on RAV indicates that there is no clearcut "best" strategy, and that even given knowledge of the other ballots, determining the best strategy to use to attain a given result is computationally difficult (see this paper). While the method is not strategy-free, the most reliable (and simplest!) strategy is generally to vote for precisely the works that you want to make the final ballot.

Note that RAV is not designed to "punish slates" per se; its intention is to ensure diverse representation in a multi-candidate election. (And I'm not pushing it from any vested interest - I was only vaguely familiar with the system before this conversation started.)

@45: Basically what this point comes down to is that if everyone only nominates one candidate, the system boils back down to approval, which is the case for any of these systems. I think that's unlikely behavior, and in any case, it doesn't really support an argument for any particular system since they all behave the same way.

#49 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:37 PM:
@42: Assuming no correlation,

You mean, "assuming no overlap". In my numbers above I assumed no correlation. (Here's the difference: if two works have 50% coverage each, then with no overlap they'd combine to 100% coverage, but with no correlation they'd combine to 75% coverage.)

It doesn't make a big difference; there would have been 11 short story nominees before 50% saturation (75% RAV threshold) if you presume that all stories are uncorrelated. Interestingly, you'd also get 11 nominees if you assume D, G, H, and I are maximally correlated (in which case, the only one of those that wins is D).

#50 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 12:41 PM:

@49: Thanks for catching that. I changed methodology mid-sentence and didn't spot it on review.

#51 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:42 PM:

@48: That is a pretty good explanation of strategy under RAV. There's just one thing I'd add. If, as @46 suggests, you like A, B, and C, and expect A to win without your vote, then the simple (imperfect, but generally good) strategic rule of thumb is to vote for A if and only if you prefer it to all other candidates. So there would be some people leaving off a popular work if they have lesser-known works they like better, but it's not generally a good strategy to leave off a popular work if it is actually your favorite. That latter strategy can work in some rare cases, but it's generally not worth the risk.

#52 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 12:45 PM:

Clay Shentrup @33:

1. We don't do .sigs here. Please leave that off.

2. We tend to look askance at commenters whose very first comment contains a link to another site, especially if it's connected to a business they run.

3. This thread continues the conversation started in the previous thread. "It should be fair" is one of the stated requirements of the voting systems we're discussing. It is not your place to abruptly show up and announce that we're having a different conversation, one whose terms you propose to define.

We are not talking about social choice theory. We are certainly not talking about whether a rational organism's goal is to maximize its expected utility.

The turf you're standing on belongs to science fiction fandom. We're talking about how our worldcons conduct the voting for our Hugo Awards. If you want to join us in respectfully discussing that subject, you're welcome to stay. If not, I'm sure you'll be much happier elsewhere.

#53 ::: J Thomas ::: (view all by) ::: April 13, 2015, 12:50 PM:

@33 Clay Shentrup

> It should be fair.

This is a basically meaningless desideratum frequently referenced by people not versed in social choice theory. Suppose we can select option X, which will give Bob and Alice utilities of 5. Or we can select option Y, which will give Bob a 6 and Alice a 7. Do you really want to choose X (thereby making Bob and Alice BOTH less happy) in the name of "fairness"?

Sometimes. It depends.

If Alice feels strongly that she does not want Bob to benefit *more*, Bob might agree to a more even result that's worse for everybody. This happens in real life. There's a debate going on in the bigger world about equality. I will parody it:

"In today's world we are all better off. I have an extra hundred billion dollars, and you have a better computer than you could otherwise have. Because of the Iron Laws of Economics, if anything happened that made me worse off, you would be worse off too. So let's all be happy with our lives, and if you just work harder and smarter you can be rich like me."

"It isn't fair and I'm not putting up with it."

People bring meaning to concepts. You might find no meaning in fairness, but others create meaning for it.

Is this off-topic?

#54 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 12:51 PM:

Jameson @44, Th Cntr fr lctn Scnc is not here. Jameson Quinn and Clay Shentrup are here.

#55 ::: albatross ::: (view all by) ::: April 13, 2015, 12:54 PM:

J Thomas:

I am curious how much non-slate voters agree in their nominations. My best guess is that people mostly tend to read clusters of related books--one person may read mostly MilSF, another may read mostly alternative history, still another may prefer urban fantasy. I'd expect that to lead to correlated nominations. Also, there are probably correlations between favorite authors (based partly on categories, but also on when the authors started publishing, political or social ideology, style of writing, hardness of their SF, series vs standalone works, etc.)

This matters because anything that decreases the weights of slates is likely to also decrease the weight of these correlated nominations. Suppose we have RAV with exponential weights, and we have a year where there are two really strong space opera choices. It would be easy for the less initially popular one to get cut out of the nominations by being in the shadow of the other one. That might be good or bad in a given year, but it's a potentially important change in how the nominations will work.

#56 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 12:55 PM:

I took a look at the 2014 short story nominee list — the puppy bête noire — and I think RAV might have "helped" there too. I mean, I actually liked "Dinosaur" and "Selkie" and "Water That Falls"; but these are three Escape Pod/ Podcastle stories that are all arguably more experimental from a literary point of view, less hard-science, and more "politically correct" in their themes than the "average" story in the genre would be (if such a thing really existed). So you don't have to posit any shadowy cabal to imagine that the voting for those three stories was correlated. Which means that, with RAV, "Selkie" (which is actually my favorite of the three, but whatever) probably wouldn't have made it. And then maybe there would have been less impetus for the puppies to go ape this year...

#57 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 01:04 PM:

@52: Clay is a co-founder of the Center for Election Science, and we respect him as such; but he is not a current board member, and has never been the executive director, so it's not exactly a "business he runs".

@54: Point taken. I thought my comment was just saying "welcome to join the conversation, but we don't need the talking points".

#58 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 01:06 PM:

Cheradenine @ 37:

Considering that you and Bruce have stated your support for a system that would knock out works with absolute majority support, I would be more comfortable if you just added a proviso that a work that appears on an absolute majority of ballots always makes it to the final ballot. If a single bloc manages to produce an absolute majority of the votes, then so be it, they win.

This is not far-fetched when we consider that a competing bloc to the Puppies might emerge and we end up with a two-party system. If one party dominates the votes to the point of having an absolute majority, they should reap the rewards.

#59 ::: J Thomas ::: (view all by) ::: April 13, 2015, 01:06 PM:

@55 albatross

I am curious how much non-slate voters agree in their nominations. My best guess is that people mostly tend to read clusters of related books--one person may read mostly MilSF, another may read mostly alternative history, still another may prefer urban fantasy. I'd expect that to lead to correlated nominations. Also, there are probably correlations between favorite authors (based partly on categories, but also on when the authors started publishing, political or social ideology, style of writing, hardness of their SF, series vs standalone works, etc.)

Yes. But the difference between people who vote for the same nominations because those are the ones they like, versus people who do it because they have an evil plan to dominate the nominations, is subtle.

You can't tell the difference from the ballots, you have to get stuff from their websites or listserves or emails where they are plotting. Or maybe you can just tell.

When the purpose is to reward diversity and independence, then when people vote in lockstep it doesn't matter whether their intentions are evil. We should punish them the same.

So if a bunch of people vote for the same 5 space operas or the same 5 feminist stories, they must be treated like evil slates. We will get more diversity if we give their opinions less weight when they agree.

Suppose we have RAV with exponential weights, and we have a year where there are two really strong space opera choices. It would be easy for the less initially popular one to get cut out of the nominations by being in the shadow of the other one. That might be good or bad in a given year, but it's a potentially important change in how the nominations will work.

Sure, but it only works that way if the same people vote for both. If the people who like one space opera hate the other one, then the weaker one will get its full chance to be nominated.

It's the usual strategic rule -- if you want nominee X to win, don't also vote for its closest competitors.

#60 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:07 PM:

@56: It actually looks to me like "Water hat falls" (the eventual winner!) had the fewest nominations of the 3, so if one of them was hurt, it would probably be that one. It's hard to say without full ballot info, though.

#61 ::: dh ::: (view all by) ::: April 13, 2015, 01:08 PM:

When the purpose is to reward diversity and independence, then when people vote in lockstep it doesn't matter whether their intentions are evil. We should punish them the same.

So if a bunch of people vote for the same 5 space operas or the same 5 feminist stories, they must be treated like evil slates. We will get more diversity if we give their opinions less weight when they agree.

The problem is that punishing slate voters gives enemies of a particular work or author or ideology a new weapon to fight with.

#62 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:11 PM:

Fred @58: We may have to agree to disagree here. I actually don't think it's appropriate to allow one group to completely domination the nomination pool. If 55% of people prefer SF and 45% prefer fantasy, is a list of 5 SF nominees most representative? I would tend to say no.

(OK, i would actually say that if we're in a situation where 5 works are all receiving an absolute majority of nominations, the Hugo nomination system is irretrievably broken anyway.)

#63 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 01:12 PM:

J Thomas @53, fairness and the perception of fairness must be on-topic. They're right there in Bruce Schneier's initial post.

I've helped administer TAFF, which is also a large idiosyncratic fannish institution that holds elections. Fairness and the perception of fairness are both essential in fannish systems. Of the two, losing the perception of fairness is the blunder that'll kill you faster.

#64 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:13 PM:

dh @51: Is a Hugo ballot consisting of five space operas ideal?

#65 ::: AD_FV ::: (view all by) ::: April 13, 2015, 01:28 PM:

In support of option 3d, the single transferable vote: Of the choices presented above, only a change in the voting system will have the desired impact. The primary flaw with the current Hugo nominating process is that it uses winner-take-all block voting, under which a cohesive plurality of the electorate can determine the winners, leaving the rest unrepresented.

The choice of particular voting system will always involve some compromise, but option 3d, the single transferable vote, provides an accurate and fair representation of the electorate while avoiding many of the pitfalls and compromises inherent in other systems.

Approval voting systems violate the later-no-harm principle, as a voter approving of more than one candidate may be harming the chances of a more preferred candidate, incentivizing voters to vote tactically rather than express their true preferences. More on concerns about approval voting and the comparative merits of STV can be found here.

Use of STV with rankings ensures a proportional outcome, eliminates the spoiler effect, and is immune to tactical voting, allowing voters to express their true preferences. It will produce a proportional and representative outcome regardless of the existence of slates or organized factions. It is currently used to nominate films for Best Picture at the Academy Awards, and is the natural counterpart to the instant runoff system used to select the Hugo award's ultimate winner.

The experiences of cities that have adopted ranked choice voting for single and multi-winner elections, like San Francisco, Oakland, Minneapolis, and Cambridge, MA, shows that voters are very likely to make use of the ability to rank candidates and understand and adapt to a ranked system. The same would certainly be true of Hugo award voters.

#66 ::: Seth Gordon ::: (view all by) ::: April 13, 2015, 01:32 PM:

Even in the absence of deliberate slates to defend against, I don’t think it would be an awful thing if a final Hugo ballot ended up with the most popular military SF novel of the year vs. the most popular feminist SF novel of the year vs. the most popular steampunk novel of the year and so on. OK, it would disappoint the author of the second-most-popular military SF novel of the year, but if I understand the voting algorithms correctly, the rocket would end up going to someone who not only appealed to one particular constituency, but was highly regarded in other constituencies as well.

#67 ::: J Thomas ::: (view all by) ::: April 13, 2015, 01:32 PM:

@63 Teresa Nielsen Hayden

Fairness and the perception of fairness are both essential in fannish systems. Of the two, losing the perception of fairness is the blunder that'll kill you faster.

I tend to agree. When you lose the perception of fairness, people who lack a personal relationship with the guys on top may start to vote with their feet etc.

If it's objectively unfair but people don't notice, then they suffer whatever consequences come from that. Since I don't know how to tell what's objectively unfair or what consequences come from it, I'm more likely to notice the perception of unfairness and write off the actuality of unfairness as bad luck.

#68 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 01:33 PM:

Jameson, use "politically correct" like that again and you will lose your vowels. So will anyone else who uses it. I am not kidding.

The casual political heterodoxy of the SF community has to be experienced to be fully appreciated. We will cheerfully hold intelligent, respectful political conversations where the participants include monarchists (early medieval model), extropians, neoplatonists, moderate liberals, Labour, LibDems, Parti Québécois, EU Greens, beleaguered US centrists, and six different strains of libertarian.

Anyone who thinks "pltcl crrctnss" is a significant literary criterion in our field is short on information, or has been fed bad information.

#69 ::: albatross ::: (view all by) ::: April 13, 2015, 01:38 PM:

Re nominating strategy:

ISTM that part of what we're wanting here is for it to be sensible behavior for someone to say "Of the new books I've read this year, I'd say A, B, and C were plausibly Hugo-worthy" and then nominate A, B, and C. Since most people probably haven't read everything worthy of consideration for an award, the nomination is an attempt to get lots of people to propose things they think might be worthy. But that does clash a bit with RAV, because now the same nominator should also think "Hmmm, I think A, B, and C are plausibly Hugo-worthy, but A is a heavily-marketed work by a popular author, so maybe I should just nominate B and C."

Re fairness:

In political elections, it's hard to opt out of the results if you think they're crooked or fixed. But most other things are voluntary. If you conclude that the federal government has lost legitimacy, you still have to pay your taxes and the FBI can still arrest you. If you conclude that an organization you belong to has lost legitimacy, you can just stop paying dues and attending meetings--enough of that, and the organization will fall apart.

#70 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 01:40 PM:

Cheradenine @ 62: It looks like your preferred system would solve the biggest issue of this year, that a tiny minority managed to get all the nominees in many categories. But it still wouldn't, by itself, stop that tiny minority from getting 1-3 nominees in every category, right? So as long as they can continue to get nominees, there's little incentive for that minority to stop blocing. As long as everyone else is not organized they can still get a win, even if it's not as big as this year's win.

I would then expect a large group of people to organize in response, and to form a significantly bigger bloc.

But guess what? Even if the puppies are a tiny minority bloc up against a much bigger bloc, they *still* get at least one nominee on each category in the ballot, because now under 3c the bigger bloc eats its children and the top minority bloc candidate can squeak in.

I think you should give more consideration to what happens after people absorb the lessons of this year. Assume that blocs are here to stay. Do we want to ensure that each bloc gets at least one nominee per category, or not?

#71 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 01:42 PM:

@68: I understand and take your warning. In my (somewhat irrelevant) defense, I was using the term in explicit scare quotes, to stand for what I believe a "puppy" would say. As I said, I personally thought that all three of the works in question were above average, and that "Selkie" was outstanding.

#72 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:44 PM:

@65: Apparently everyone will now drop by this site to make the case for their favorite system without reading the threads or considering the details of the situation in question.

STV is a good system (imo) with flaws like all voting systems. A number of your statements are hyperbolic or incorrect (STV neither eliminates the spoiler effect nor is immune to tactical voting, per the Gibbard-Satterthwaite Theorem.) It also cannot guarantee a proportionate outcome, particularly in a case where not all voters rank all nominees - which, as noted repeatedly in this discussion so far, is impossible for the Hugo nomination election.

#73 ::: albatross ::: (view all by) ::: April 13, 2015, 01:47 PM:

Re number of nominees:

It seems like there's a natural limit to how many nominations can be accepted, based on how much time people will have to read the nominated works. I wonder if it would make sense to allow some rule to expand the nominations but only within a narrow range--say, the default is five, but if the next highest nominee is close enough in some sense you can go up to six or even seven nominees.

#74 ::: Andrew M ::: (view all by) ::: April 13, 2015, 01:50 PM:

When the purpose is to reward diversity and independence, then when people vote in lockstep it doesn't matter whether their intentions are evil. We should punish them the same.

I think there is a serious issue here which it would help to be clear about. There seem to be two views at work in this debate. On one, the Hugos have worked pretty well up to now, and the new practice of slate voting has disrupted this; the aim is to restore the Hugos to something like their historic way of working. On the other, the aim is to improve on the Hugos as they have been; there is a feeling that they are not sufficiently diverse, and a modification of the system would make them more so.

Are the Hugos diverse? Well, I think there is more than one kind of diversity, and one kind may be the enemy of another. Clearly, Hugo nominees are not all the same kind of work - they can be incredibly different. But they don't reflect the full range of the field. There seem to be two factors which tend to make a work a Hugo nominee; one, which I mentioned in an earlier thread, is that they have, or at least might be imagined to have, cross-group appeal, rather than being in the core of a specific subgenre. The other, which someone else mentioned, is that they have a kind of uniqueness, rather than just being typical of their author. I think that these are good qualities for nominees to have; they help to pick out the most distinctive and significant work of the year; they mean that the final ballot does not consist of five works each of which is loved by 20% of the voters and hated by the other 80% [this is a rhetorical exaggeration], and that the winner, though not everyone's favourite, is not just the 'least hated' but has fairly wide support.

I think some people are assuming that if a lot of people vote for the same five works (not as part of an organised slate), this will be because they are all similar works - as dh says, five space operas or five feminist works. But I think it's quite likely that a fair number of people may vote for the same five works because they want to reward diversity and independence - because those works, diverse in nature, are the ones that stand out as significant. None of us knows what three works were knocked off the Novel ballot by the puppies, but I think we could name six or eight works and say with some confidence that the three missing works were among them; and a lot of ballots will have made their picks from among those works. I'm afraid that if the voting system positively rewards difference, we will end up with a duller set of nominees - the epic fantasy nominee, the urban fantasy nominee, the MilSF nominee and so on.

One other thing to bear in mind - I think this harmonises with some things that Brad Templeton has been saying - is the effect of the award as a recommendation. The voters are not the only beneficiaries of the process; we are sending a message to the wider world, about the most significant things in SFF. From the voters' point of view, it may be fair that clumped preferences should have less weight, so as to give some representation to more people. But if we are sending a message to the wider world, I think we should be telling them about the works which have the most support, not leaving things out because those who like them like a lot of the same other things.

#75 ::: Fred Bush ::: (view all by) ::: April 13, 2015, 01:51 PM:

Here's an example of what might happen once voting gets heavily bloc-ed up:

infernokrusher bloc: 65% of the vote
grimdark bloc: 15% of the vote
puppies bloc: 10% of the vote
non-bloc: 10% of the vote

If we use 3c/exponential weighting, I think infernokrusher gets 3 spots on each category, and grimdark and puppies each get 1. Is that what we want?

#76 ::: JonW ::: (view all by) ::: April 13, 2015, 01:53 PM:

3c3 looks appealing to me (as it does to a bunch of folks here), but I don’t pretend to say that from a position of technical sophistication.
A word on banning slates: Douglas @#35 explains the problem here, so I’ll just speak to the weaker suggestion of an honor pledge that one hasn’t “voted a slate,” per #17. That’s still a problem unless we can adequately define “voting a slate,” from the perspective of the individual nominator, so that nominators know what it is they’re pledging that they haven’t done. But I don’t think we want to tell Some Jane With A Blog that it’s against the rules for her to publish a list of the five novels she was most excited about this year, or to tell Some Joe Who Reads A Blog that it’s against the rules for him to be inspired to read those novels and really like them and decide to nominate them. The best we can really do with an honor pledge is to make people pledge only to nominate works they’ve actually read or viewed (which wouldn’t do much, because folks inclined not to comply wouldn’t be deterred). The beauty of 3c3 it that it would adequately address the ways that slate-driven nominations tend to break a FPTP nominations system, with minimal downsides, without telling people that certain sorts of enthusiasms and voting behavior are forbidden.

#77 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 01:59 PM:

Fred Bush @70: If there are competing blocs, I do think that each of them being represented on the ballot, and popular non-bloc works having a shot, is a better outcome than the larger bloc running the table (as will happen under the current system). In part this is because I see the ideal outcome of this election as providing Hugo voters a range of choices which is as representative as possible of the range of Hugo voters as a whole. So "avoid one group completely controlling the ballot contents by collusion" was a high priority in making my recommendations.

I think that STV would provide a slightly higher chance of a smaller bloc being shut out entirely, but due to the dispersion of Hugo nominations I'm not sure it would work. In a situation like the short story nominations this year, the bloc would win because no story would ever accumulate enough votes to exceed the bloc's votes or reach the quota. Balloting behavior might change in response to the system change, but I don't know if that would help here.

Voting system tweaks can keep blocs from taking over the ballot, but I don't think they can provide a strong disincentive to bloc voting. If we want to make blocs less powerful than single voters, a change like Option 4 would be necessary. (And as I noted above, I don't think enforcing that is viable.)

#78 ::: John A Arkansawyer ::: (view all by) ::: April 13, 2015, 02:06 PM:

Fred Bush @ 75: I can haz moar infernokrusher pleez?

JonW @ 76: The honor pledge against having voted a slate and the honor pledge that one has read the works which one is nominating have two good qualities in common:

1) They appeal to the better angels of people's nature.

2) They are imperfect but work well enough.

That last is so important. A perfect solution (should there be one) is gameable or authoritarian, in perception surely and in fact probably. Imperfection is a feature, not a bug.

#79 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 02:11 PM:

AD_FV@65: Sigh.

It is simply untrue that STV "is immune to tactical voting". The Gibbard-Satterthwaite theorem shows that no voting system is. (Dictatorships can be, but that's not voting, is it?)

In particular, all proportional systems, STV included, can in some cases reward a voter for leaving off a candidate that they prefer, but that can win without their help.

"Later no harm" is not, in my opinion and that of many voting theorists, a desirable characteristic in a voting system. It is incompatible with other, more-important criteria such as participation, consistency, monotonicity, Condorcet, and favorite-betrayal-proofness. There is some arguable benefit to a post-hoc, strategic later-no-harm-like criterion, such as a continuous median system would evince. But not to full out LNH.

...

I'm really sorry to have to do this, but I'm going to have to air some dirty laundry.

The "FV" in AD_FV refers to "FairVote", an election reform advocacy organization. (I also recognize the initials before the underscore.) FairVote promotes IRV and STV. FairVote is older than my organization, the Center for Election Science (CES). The CES was founded a few years ago to promote voting systems that are in line with "recent" (since the 80s) advances in voting theory. Prior to founding the CES, most of the people involved had attempted to convince FairVote to reduce its emphasis on IRV and embrace newer methods. Of course, I'm not going to claim that "my side" never overstated our case, or that we did everything right. But the outright refusal of FairVote to listen to us, and their continued use of what we view as outright falsehoods in promoting IRV (similar to the falsehoods about STV in AD_FV's comment; STV is a good system, and I'd support it here, but there's no need to lie about it), led us to be disenchanted with FairVote, and was part of the impetus for founding the CES.

I have been upfront about my own affiliation in this and the prior thread. And I honestly believe that there's not much "ulterior motive" in my suggestions here. RAV is not a system that the CES strongly promotes, unlike the case with STV and FairVote.

So, now you know. You can discount both my and AD_FV's statements accordingly.

I do NOT intend to dox anyone here, and I believe that I have not done so. If any moderators disagree, I encourage them to delete everything after the ellipsis. If any of this is over the line, then it would still be so after disemvowelment or rot13.

#80 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 02:12 PM:

Andrew M @74: That's a very sound point. I'm not quite convinced that uniqueness and cross-group appeal have typified the Hugo nominees over the past few years, but that's a subjective assessment. I think that even if a lot of people are selecting from the same list of 8-10 books, the effect size is likely to be pretty small. If we could agree on some parameters for what we think the distribution might look like, I'm happy to sim it. (Though again, real data would be the Holy Grail here.)

#81 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 02:18 PM:

For what it's worth, as far as I know, Cheradenine is unaffiliated with either CES or FairVote. More importantly, their comments seem knowledgeable yet entirely unbiased. My intention is that my own comments should be unbiased, too, of course, but that's not for me to judge.

#82 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 02:24 PM:

(As weak evidence for the assertion that I'm not biased by my CES membership: note that when my fellow-CES-member Clay came by, his proposal was different from mine.)

#83 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 02:36 PM:

J Thomas @67, my only disagreement is that there are no people on the top. The worldcon is run by fans, not pros, and every worldcom committee is different.

#84 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 02:45 PM:

Jameson @71: I understand; but you know, sooner or later someone's going to think you mean it. I'd rather keep that mess from happening than try to clean it up after the fact.

In any event, it really was intended as a general warning, and everyone should understand it as such. (Unless they're Bruce Schneier. If he wants to use it, he can.)

#85 ::: rcade ::: (view all by) ::: April 13, 2015, 02:50 PM:

Cheradinine @ 7: Option 4: Banning Slates. I can see two ways to do this: lay out in exacting detail what a slate is (which means that people will immediately game their way around the definition), or leave identifying slates up to the Hugo administrators, which is a responsibility I suspect most concoms do not want and are not prepared for.

I think there's a third way: Worldcon can adopt language to discourage slates without specifying any enforcement, such as something like what I proposed earlier: "3.7.4: Members should not vote for nominations by copying any slate of nominees suggested by others, but instead should make their own individual choices for what they believe are the best works."

Without enforcement, the purpose of such language would be to take away the argument that slates are within the rules. Any Worldcon member who launched one would be in direct contravention of the WSFS Constitution, which fans would no doubt remind that person.

To the people who favor "do nothing" as an option, I'd like them to consider at least finding some means of expressing the idea that Worldcon strongly opposes bloc voting.

#86 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 02:54 PM:

Jameson again, @79: Thank you for the explanation. I think we could all tell something was going on. A brief explanation is less distracting than mysterious scuffling noises ongoing behind the arras.

Also, what you did there wasn't doxing. You were giving us relevant identifications of some participants. We'd have had a lot more trouble sorting that out on our own. Not that we couldn't have done it, if we'd felt motivated; but it's better to not get distracted.

Now let's all go back to talking about the mechanics of voting.

#87 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 02:54 PM:

rcade@86:

"3.7.4: Members should not vote for nominations by copying any slate of nominees suggested by others, but instead should make their own individual choices for what they believe are the best works."

I think we should do this no matter what method we eventually decide to use. It does no harm, might do some good, and certainly expresses what the Hugos are supposed to be about.

Kilo

#88 ::: Joshua Kronengold ::: (view all by) ::: April 13, 2015, 02:54 PM:

As should be obvious, I support RAV (which was my first thought wrt slates, although I didn't know it had a name), or SDV (which I came up with while cogitating on an easier to explain and similar outcome equivalent of RAV). There are corner cases where they have different outcomes, but they are both fundamentally purportional representation systems that try to avoid a dedicated plurality controlling multiple slots in nomination. (I'd like to see how SDV does with competing slates -- my instinct is that the larger slate will tend to knock out the smaller one (if everyone votes party-line; if they don't then one smaller slate work will likely make it)--and then still not get everything it wants on the ballot.

Some considerations:

How do we ease the impact on administrators of nominees declining nomination.

How do we coherently display the results of the election in a way that's comprehensible to the average fan?

My guess -- since both loser-first and winner-first systems (which is the central difference between RAV and SDV; you could even have SDV act more like high-weight RAV systems by having ballots give, say, 1/2**(N-1) or some other calculated divisor to their candidates rather than 1/N) have unstable results -- that is, that elminating one candidate before election can result in substantial cascatding changes to the results, that any adoption of a similar system would also need to discuss how refusal would happen. The choices would be:

1. When a nominee declines, offer the slot (if any; it's possible that there are more than 5 nominees at that point) to the 6th place candidate.

2. When a nominee declines, re-run the calcualtion as if they were not present (eliminating them from all ballots), then offer the slot to the highest ranking candidate who is not already on the ballot.

3. Only ask people one at a time, so the results can be recalculated in full if people decline.

3 can obviously be dismissed as far too great a burden on Hugo administrators.

Of the remaining two, the first is by far easiest to explain. But I'd have to prefer the second -- because it would be most likely to maintain the goal of proportional representation, rather than potentially eliminating a bunch of people's second pick on the ground that we were taking their first pick. It might sometimes add a page to the newsletter (or just extend the web page, since at least last year we didn't bother with paper vote breakdowns IIRC) to summarize the runnoff elections, but that's worth not disenfranchizing people.

Regarding the second question, we've had a bit of attention paid to it, but I'd guess for RAV we'd want:

Round 1:
Candidate 1 selected (XXX votes)

Remaining top 15 candidates:
Candidate 2: XXXX votes at full value, YYYY votes with one successful candidate
Candidate 3: XXXX votes at full value, YYYY votes with one successful candidate
...
Candidate 15: XXXX votes at full value, YYYY votes with one successful candidate

Candidate 2 selected as winner.

And then similar breakdowns after the selection of candidates 2, 3, and 4 (none needed after 5, of course).

For SDV, I'd probably instead start after all below threshold candidates are eliminated, eg:

(after 400 candidates are eliminated for not having enough nominations to make the ballot):

Candidate 1: 50 votes in 200 ballots
Candidate 2: 35 votes in 89 ballots
etc.

I think SDV would require more elision or more space to handle, since it goes through many more steps prior to selecting winners (ie, eliminating candidates one at a time rather than 5 steps of approving them).

Question re RAV: The assumption seems to be that rank with RAV still matters, right--just only in tie situations? If not, you end up with slates being able to break it by exactly lining up their votes so the tie cannot be broken. This was the other reason I came up with SDV -- because it seemed to produce decent results without having to change the balloting system. Mind, ties are only a theoretical issue--we won't know until August who well SP/RP managed voting discipline (my guess, given that the louder parts say they didn't vote for the whole slate, is that they had at best 80%-90% discipline--enough to dominate the nominations but not enough that ties were that much of an issue). But it's good to have reasonable behavior on ties.

So difficulty in brevity explaining runoffs for SDV is a substantial point against it relative to RAV, whereas being able to cleanly avoid having rank matter at all (rather than just in ties) is a point in SDV's favor.

#89 ::: J Thomas ::: (view all by) ::: April 13, 2015, 02:58 PM:

@64 Cheradenine

Is a Hugo ballot consisting of five space operas ideal?

It depends.

Imagine that the nominations were done with approval voting, and the top 5 space operas got 55%, 54%, 53%, 52%, 51%.

It was just a space opera year and a lot of people really liked them that year.

Meanwhile the top candidates in other genres were

10% steampunk
9% cyberpunk
8% SF Regency Romance
7% knights and unicorns
6% Piers Anthony
5% military SF.

We would get a more diverse nomination to take one from each genre, but probably the space opera would win the election. If we had 5 space operas we'd find out which one of them won.

On the other hand it's a different situation when it's 55% of the nominators who like space opera, versus every single nominator voting for 2 or 3 space operas and other things too.

If everybody likes space operas then maybe that's what they want. But if it's only a small majority maybe we should give the minorities some of what they want too. But then on the third hand, if the minorities are all over the map and don't agree about anything, how do we decide which of them to throw some nominations to?

So my own preference is to let the voters see what's going on and make choices themselves. Maybe they will know better what they want than we know for them. On the other hand, future voters might have bad intentions, while we know that we don't have bad intentions now. If we make mistakes they will be honest mistakes because we can't predict what will happen, but the more power we give to future voters the more chance they will misuse it. On the third hand, we really don't know what they'll need and we are likely to make mistakes now.

#90 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 03:02 PM:

Joshua @89: What am I missing? It seems like SDV would potentially run into ties as well if we hit an elimination step with multiple tied candidates.

I was thinking on the way in about dealing with RAV ties. I'd be tempted to break them by which candidate appears on the highest number of ballots; if they're tied there too, then things get messier. (If the tie is for the last space, we can just take both candidates. The issue is if the tie is for second place, but picking one of the tied candidates would knock the other one down in the order.)

#91 ::: BillDoor ::: (view all by) ::: April 13, 2015, 03:23 PM:

No nominations.

When you register you get a (paper) ballot. Hand it in at the convention (appropriate mail-in arrangements, just like IRL elections). (Look to Oregon for a model)

Enter author & title (or equivalent identification)

Every vote is a write-in, including No Award.

Last day of of the con, see who's won.

#92 ::: Andrew M ::: (view all by) ::: April 13, 2015, 03:25 PM:

Cheradenine@80:

I'm not quite convinced that uniqueness and cross-group appeal have typified the Hugo nominees over the past few years, but that's a subjective assessment.

Well, I think it's a tendency; it's clearly not universal. There are a couple of writers of non-unique books who do well; and as for subgenres, borders shift. (I think you could say that GRRM, for instance, when he started out, was breaking the borders of his subgenre, epic fantasy, but now a new subgenre has formed around him.) I think it does explain some things, though. You do get occasional complaints - not only from puppies - that the Hugo nominees are all the same, and while that's clearly not true, this can help to explain why they don't cover everything.

#93 ::: Buddha Buck ::: (view all by) ::: April 13, 2015, 03:27 PM:

BillDoor @92:

I don't believe that particular method is one of the ones our hosts are entertaining in this thread. You can see the specific changes we are discussing listed at the top of the page.

I see you've never posted here before. Welcome. Do you write poetry?

#94 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 03:30 PM:

JThomas @89, if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, the people of fandom have spoken.

The Hugos reall are about what the Hugo voters like. This worthy principle keeps us out of fights to the death over the difference between SF and fantasy, or whether technothrillers are SF, or whether "YA" on the package makes a book ineligible.

And don't assume the space opera would win. I learned long ago to distrust statements like "Of course (Title) is going to win." Not only does (OtherTitle) win too often to be ignored, but when the voting breakouts are released after the ceremony, they may show that it had the lead all along; i.e., was a genuinely popular choice.

#95 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 03:33 PM:

JThomas @89, if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, fandom has spoken.

The Hugos really are about what the Hugo voters like. This worthy principle keeps us out of fights to the death over the difference between SF and fantasy, or whether technothrillers are SF, or whether "YA" on the package makes a book ineligible.

And don't assume the space opera wins. Not only do less obvious works win too often to be ignored, but when the voting breakouts are released after the ceremony, they may show that the winner had the lead all along; i.e., was a genuinely popular choice.

#96 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 03:40 PM:

Andrew M @92: Do the people who think the Hugo nominees are all the same actually read them? I am having real trouble with this concept. Where on the map of subgenres does Among Others belong?

#97 ::: J Thomas ::: (view all by) ::: April 13, 2015, 03:56 PM:

Here is my current concern with RAV.

In the nomination I vote for A, B, C, D, E. I think A is the best and I really care about it, but if A doesn't win I want B. C is pretty good. D is OK too. E, meh, I can't think of a better fifth choice.

But then it turns out E wins handily. My other four choices get recalculated as half votes. And then D wins. My other three choices get recalculated as quarter votes. And none of them win.

I didn't care that much about D or E in the first place, but the result of putting them on the ballot was that my vote for A got turned into a quarter of a vote.

Given my druthers, I'd rather be able to cast a full vote for A, half a vote for B, a quarter vote for C, etc and then if A loses I give B a full vote, C a half vote, etc.

I guess. Maybe I'd rather give A and B equal weights, it depends.

It's all so complicated.

I can see the reasoning. After I win something I don't deserve to have as much influence. Give somebody else a chance.

IRV lets you influence *just one*, but you get to pick which one. You can only vote your second choice if your first choice loses. You don't get penalized if your fifth choice wins before your first choice has lost. IRV has its problems, but maybe some of that is that it was a front-runner for so long that people have had a lot of time to figure out the problems. These newer approaches look good because we haven't had as long to knock them down.

#98 ::: J Thomas ::: (view all by) ::: April 13, 2015, 04:08 PM:

@95 Teresa Nielsen Hayden

if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, the people of fandom have spoken.

Yes, that's how I think it ought to be.

But as I understand it, by some of the new rules we're discussing, if a majority of fans each nominate the same five space operas, then we pass the one with the most votes and we reduce their votes for the second one, and reduce the votes more for the third one, and with enough reduction in votes some others that got fewer votes will get on the ballot.

Maybe space operas that got fewer votes but that got votes from *other people*.

Say that the best space opera got 52% of the votes, and the second-best got 51%, and the overlap was 3%. Then they both get nominated.

But if everybody who voted for the second-best also voted for the best one, then it doesn't get 51% of the votes. It gets 25.5% of the votes. Anything that gets 26% will beat it.

It isn't about how many votes it got. It's about who voted for it.

if five space operas, or five closely-written works of cutting-edge sociological SF, or five quasi-YA wizards-at-school romps get the most votes, the people of fandom have spoken.

With this proposed rule it isn't about whether it gets the most votes. It has to get the most voters who didn't also vote for something else. It's complicated.

#99 ::: J Thomas ::: (view all by) ::: April 13, 2015, 04:12 PM:

@96 Teresa Nielsen Hayden

Do the people who think the Hugo nominees are all the same actually read them?

Some years ago I read about a Chinese convenience store in San Francisco that got robbed by a white man. The clerk could not give the police a description. She said they all look alike.

It's like that.

#100 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 04:41 PM:

Something to think about...

Whatever system we choose, it needs to be easily explained to the Hypothetical Average Hugo Voter (tm). As an astrophysicist, I've got a pretty good background in statistics, so while the voting theory aspect is new to me, I think I'm understanding fairly well. I'm not sure we can say that about our HAHV. You only have to look at all the confusion surrounding the "no award" process this year to see the problem -- and the "no award" process is a much, much simpler process than most of, for example, the option 3 variants.

I'm not saying we should avoid complex voting systems. But whichever system we choose, I think we need to come up with a simple "mission statement" for that particular system, along with some examples that illustrate why that system fulfills the mission.

As a completely made-up template (and not one that is explicitly intended to match any particular system), a sample mission statement might read something like, "Ideally, Hugo voters should have one favorite work that they want to win, and failing that, some backup candidates that they wouldn't mind winning. As a result, your first choice is weighted the most, and if your first choice ends up being successfully nominated, then your other choices will be counted less -- in essence, you've had your turn to nominate, now let someone else propose something."

Is it possible to define a similar mission statement for all of the options under consideration? If not (though I personally think we can), I think think we should seriously consider not using such a system. If the HAHV can't intuitively see how the process works, that's going to tend to engender mistrust in the system, which undermines everything we're trying to do here.

Is anyone up to the challenge of writing a mission statement for each option now? It might help focus our discussions.

Kilo

#101 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 04:51 PM:

@86: Thanks for clarifying.

One more bit of dirty laundry; the last, I promise.

FairVote and CES agree on a number of things. We agree that plurality is a horrible system, with distortionary and pathological effects wherever it is used. We agree that proportional representation is a good idea in most multi-winner contexts. We agree that you also need some form of single-winner reform, since not all elections are or could be multi-winner. We agree that no single-winner system is ideal in all respects; there are tradeoffs involved, depending on what characteristics you value.

We even agree that arguing about which single-winner system to propose is counterproductive and makes us all look like the Judean People's Front ("splitters!"). Unfortunately, we have not yet found a way to agree on how to stop arguing.

I'm going to try to present a fair guess at what I think 4 hypothetical people would say. You can probably guess who they're supposed to stand for, except the last one. There is no hidden meaning in the names besides the most obvious pun.

IrishWhiskey_CES: The things that matter about voting systems are their strategic properties and their outcomes in realistic scenarios, as well as their simplicity, expressiveness, and auditability. There are a number of good systems; good single-winner systems include approval, Condorcet, majority judgment, score, and SODA, while good multi-winner systems include BTV, PAL, RAV, RRV, SAV, and STV. Unfortunately, IRV has enough problems with outcomes, simplicity, and auditability to make it worse than other single-winner systems. If we want to stop arguing, we should settle on the simplest proposal, and the one which has the fewest downsides. That means approval voting is the first step.

Dirt_CES: The main thing that matters about a voting system is its outcome from a utilitarian perspective. It is possible to measure this numerically using monte-carlo simulations (yielding numbers called "voter satisfaction efficiency" or "bayesian regret"), and if you do enough such simulations, questions of strategy become moot. The best single-winner system is score voting, and approval voting is an acceptable as a first step towards that. By analogy, RRV is probably the best multi-winner system. IRV is basically a step in the wrong direction.

SUM_FV: The main thing that matters about a voting system is whether it can be passed as a reform. This in turn depends mainly on three things: how much of a track record it has in political use, how little strategy it encourages from a coordinated/elite perspective, and how well you sell its advantages. For multi-winner systems, STV has the longest track record. IRV has the most track record of any single-winner reform. It also is a good stepping stone to STV, and its later-no-harm characteristic means that there are few incentives for candidates to encourage dishonest strategy. Approval voting, since it is not IRV, is a threat and must be fought against.

Nonaligned_Electorama_mailing_list_theorist: I have my favorite single-winner system X for reasons Y but I could live with approval as a first step.

#102 ::: felice ::: (view all by) ::: April 13, 2015, 04:52 PM:

J Thomas @97: "In the nomination I vote for A, B, C, D, E. I think A is the best and I really care about it, but if A doesn't win I want B. C is pretty good. D is OK too. E, meh, I can't think of a better fifth choice. But then it turns out E wins handily. My other four choices get recalculated as half votes. And then D wins. My other three choices get recalculated as quarter votes. And none of them win."

Which is why a system that takes rank into account is a good thing. People already rank their nominations, because there's no way not to do so; I really don't think telling them that the order matters is going to discourage participation. In the nomination instructions, just say something like "Does it matter what order I list them in? It can make a difference, but how many other people list the same works is a much bigger factor; if you're not sure, don't worry about it."

#103 ::: Joshua Kronengold ::: (view all by) ::: April 13, 2015, 05:10 PM:

#90: Cheradenine: Re ties, because SDV eliminates weaker candidates first, it has the luxury of considering the remaining field when deciding how to handle ties, and thus still end up being more or less fair to the voters.

The approach I used for my initial draft was that there's a threshold of the maximuim number of nominees (as well as the desired number). If removing all tied candidates would not put the number of nominees below the desired number, or if the current number of candidates is above the maximum number, then all tied candidates are eliminated as part of a single elimination step. If, on the other hand, this is not true (if the current number of candidates is within the minimum and removing all tied candidates would drop us below the desired number), then the runnoffs stop with a slightly higher than usual number of nominees.

This does act as a weak anti-slate measure (since a slate that manages to enforce strict voting discipline risks being below the threshold of nomination and having the entire slate eliminated rather than the entire slate accepted and added to the ballot), but I think manages to handle ties in most cases (obviously, ones where you end up with an empty or no-candidate balllot aside) reasonably well and fairly.

#104 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 05:12 PM:

J Thomas @97: Under RAV, if you feel "meh" about a candidate, you probably shouldn't list it. Actually that's true about the current system as well - E could end up keeping one of the ones you felt better about off the ballot by one vote. (If you look at the nomination counts, it's not unusual for 5th and 6th place to be this close!)

STV does give you more control over where your votes get placed, but possibly a lot less transfer. If A gets nominated but isn't way over the minimum votes, your ballot is basically done, whereas with RAV, it would count as 1/2 a vote for your remaining nomination choices. So there are scenarios where your "bang for your voting buck" ends up higher with either system. STV is probably more appealing for nominators with a strong favorite, RAV for ones without one. I don't know which group is larger.

It would be interesting to look at the ballots for a particular election and see what proportion named none of the nominees, 1 of the nominees, 2 of the nominees, 3 of the nominees, and so forth, to give us a sense of how often this will happen. "Interesting" might be an understatement. "Vital" might be better. :)

#98: I think your comment "It isn't about how many votes it got. It's about who voted for it." is actually a very cogent statement of how proportional voting systems work. (I might add "just" before the "about".) Note that this is true of STV as well, and, to an extent, of the IRV system that's used in the final round. If this idea makes you balk, then I think your best option is to switch to single nontransferable vote (basically allowing one vote per nominator - which will switch the focus back to campaigning for individual works). You could also keep the current system and try to effectively ban slates (tough in a system where they're so powerful), or alter the electorate.

#105 ::: felice ::: (view all by) ::: April 13, 2015, 05:19 PM:

Bruce Schneier @16:
"nathanbp @10:
"On option 5, what if the current nomination status was released just once during voting, at say one month before voting closes? Release a list of the top 15 in each category in alphabetical order without vote counts. I feel like this would help to concentrate votes without giving out too much information that could be used strategically or being subject to gaming."

I really dislike this. It's too much responding to the tactics, and not looking at the broad picture. The risk of unforseen consequences are great."

I think it addresses a bigger, longer term problem, not just the Puppy tactics. Most nominations are always for works that have no chance of getting on the final ballot; that means they're essentially wasted. Publishing the current frontrunners allows a lot more people to give their opinions on the works that do have a chance - as well as letting people who nominated unpopular works change their nominations, it would encourage participation from people who currently don't nominate at all.

This variant of Option 5 isn't particularly vulnerable to DDoS, sniping isn't a problem, it doesn't require much extra work for the admins (they'd have to check for duplicates under variant/typo names before releasing the top 15, but that leaves less to do before announcing the final ballot), and I don't see how the information it provides is of any use to the Puppies. Can anyone think of any examples of what "unforeseen" consequences there could be?

Do you object to a three stage voting system with a longlist between nominations and final ballot for any reason other than the extra time and work involved? A one-off publication of the top 15 half way through the nomination period serves essentially the same purpose, while being far simpler to implement.

#106 ::: JonW ::: (view all by) ::: April 13, 2015, 05:44 PM:

Felice@#105 -- Without regard to whether this is a good plan, I think strategy for the business meeting has to be "fix one thing at a time" -- it's exponentially harder to convince the business meeting to make *multiple* changes in the voting system all at once. So this proposal makes sense for the next business meeting only if it would be adequate all by itself to address the problem slates present (without also switching the nomination-tallying method to RAV). But I don't think it would be.

#107 ::: J Thomas ::: (view all by) ::: April 13, 2015, 06:01 PM:

@104 Cheradenine

It looks to me like with proportional voting, basicly it's "When you vote for more than one, you give full support to the one that's easiest to win. The more help a nominee needs, the less you can give it."

For this particular case I currently prefer straight approval voting. "When you vote for more than one, you don't choose between them. You are voting for all the ones you vote for, and basicly voting against all the ones you don't vote for."

The more you vote for, the less choosing you do. When you vote for just one, there are (N-1)!/(N-6)! different ways to lose. When you vote for two, there are (N-2)!/(N-7)! ways to completely lose. Fewer. (I may have the formula wrong, I don't feel like thinking it out right now. I think it's qualitatively right.)

There could be some choices you think would be fine Hugo candidates that you don't vote for because you don't want to dilute your vote. But if you find out that the ones you did vote for can't win, you have nothing to lose by voting for other good choices too.

You only choose between the ones you vote for and the ones you don't vote for. So if you only care about voting against a slate, one possibility is to vote for every nomination that has a chance, except those on the slate. That maximises your chance that things not on the slate get nominated, but it gives you no choice at all about which items not on the slate get nominated.

#108 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 06:12 PM:

Kilo @100: Good point. I'm pretty confident that given time and thought, I can write a cogent explanation of HOW a given method works, but that's distinct from WHY it works.

A first try:
SAV/PAV: "Pick the slate that will make the average voter as satisfied as possible with the results."

RAV: "Ballots count more if none of their nominees have been selected yet." (In this case, I'm kind of reversing the mechanical description. An equivalent way to model RAV with exponential weights would be "Whenever you select a nominee, double the votes on every ballot that didn't select that nominee," but this would be more work in practice.)

Joshua @103: Thanks, that was indeed what I was missing. It does mean that a slate with perfect discipline and no support outside the slate would either get all its candidates in or none of them - but I suspect we're as likely to have to deal with unicorns interfering with the election. I definitely will do some SDV simming and see what happens.

Felice @105: I fear that in a dispersed election, giving the top 15 at the halfway point will be misleading if anything. Looking at past ballots, the difference between the vote counts for 5th place and 16th place, with (probably) less than half the vote in, would quite likely be less than ten votes. But if most late voters choose one of the top 15, it will skew the election towards those that got a few more early votes. This would increase the power of campaigning and slates.

So I'm still pretty nervous about it. When I think of votes that run over a period of time and release the results at some point during that period of time, I think of those Internet polls that no one credits with any particular accuracy. It might well drive up interest in the Hugo Awards, but probably at the cost of a lot of added politicking, campaigning, and a skewed result.

#109 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 06:25 PM:

@107 JThomas: Just for clarity of language, you probably mean "RAV" for "proportional." (Proportional voting systems include STV, SAV, and PAV, and the dynamic you describe isn't true of the others.)

I understand your preference for the straightforwardness of approval, but under straight multi-candidate approval voting, the largest voting bloc is always bound to control all nominations (there's a reason it's also called bloc voting). The continuous voting system you suggest is basically an attempt to form the largest bloc dynamically over time - in other words, it's essentially a caucus. I don't think inviting everyone to an online caucus will end particularly well, but at this point I'm going to start repeating my arguments against it, so I'll stop.

#110 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 06:33 PM:

@104: STV is probably more appealing for nominators with a strong favorite, RAV for ones without one. I don't know which group is larger.

Here's how I'd put it:

STV is better for you if:

- it's easy for you to put your nominations in order of preference.
- you think that for two works with the same amount of overall appeal, the one whose appeal is concentrated within a subgenre appeal is usually better than the one with crossover appeal.

RAV is better for you if:

- you'd find it simpler just put "the good stuff this year" in a pile, without deciding which of them is better than another
- you think works with crossover appeal tend to be better than those with specialized appeal, assuming the total support is equal.

(Why do I say that about crossover appeal? Because STV can have a problem with premature elimination; something can be eliminated if it's nobody's first choice, even if it's everyone's second choice.)

#111 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 06:40 PM:

Cheradeine@109:

The continuous voting system you suggest is basically an attempt to form the largest bloc dynamically over time

I'm not wedded to continuous voting (though I do find it intriguing), but your comment struck me as odd. Isn't that, essentially, what we are trying to do in selecting a Hugo winner, no matter which system we choose? I've seen this proposed as an objection several times, but I'm still confused -- it seems to me it's a strength.

I agree there are unknowns that haven't been considered, but I don't think anyone has really considered what those unknowns might be. The attraction of the option 3-class systems is that you can mathematically model how an election will go, so we can get some determinate feel for how effective the system is. I'm wondering if we might be biased towards this type of system as a result. That may be a good thing, but I think we should, in good conscious, examine any biases that we might have. Just because something has a bias doesn't mean it's not biased in a direction that the Hugo voting community as a whole doesn't value and approve of, naturally.

Kilo

#112 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 06:53 PM:

Jameson@110:

Is there a system that can be summarized as "you'd find it simpler just put 'the good stuff this year' in a pile, without deciding which of them is better than another" without regard to cross-over or subgenre appeal? From my point of view, the whole purpose of nominations is just to "put the good stuff" out there and let the actual voting process decide which is the best. I can see arguments against either system if it favors subgenre or cross-genre appeal. That may be unavoidable, however, hence my question.

I find myself wondering how each system aligns to a given "philosophy" of what the Hugo award actually is. As someone (Teresa, I think) pointed out, to date no one has ever had to define what a Hugo was intended to be -- it was simply whatever was the result of the Hugo process. I'm reminded of the same debate in my own field over the definition of a planet. It was never a problem that we lacked a definition until we started discovering objects that didn't clearly fit in any definition. I'm wondering if the different voting systems (in addition to the "mission statement" I mentioned previously) also have a "value statement" that runs something like, "This is what we believe the Hugo is, and this is how this voting system realizes that."

Again, as a completely hypothetical template, a value statement might read, "The Hugo is awarded for the work which appealed the most to the entire fandom community, regardless of subgenre interest." (As an aside, given what we know about people's loyalty to subgenres, I'm not sure that was or ever could be the case, but that's my own opinion.) An equally valid value statement might be something like, "The Hugo recognizes the diversity of subgenres in the SF field, and awards its prize to the work that garners the most support, whether that majority comes from a dedicated group of subgenre fans or from overall broad appeal."

I'm just thinking that we may never be able to choose among the options presented until we all agree on what we're trying to accomplish.

Kilo

#113 ::: J Thomas ::: (view all by) ::: April 13, 2015, 07:00 PM:

@109 Cheradenine

I understand your preference for the straightforwardness of approval, but under straight multi-candidate approval voting, the largest voting bloc is always bound to control all nominations (there's a reason it's also called bloc voting).

The way I say that is "The nominations with the most votes tend to win". Somehow it sounds worse when you say it.

Thank you, that's a clear argument!

You could be right. My view is that when we specifically try to change the rules to exclude the Sad Puppies, and we judge how well the changes work by how well they would have excluded the Sad Puppies given historical data, we will have some difficulty explaining to journalists that we are not doing it to exclude the Sad Puppies. We can say that we aren't trying to break the Sad Puppy slate, we are trying to break all the slates that people might hypothetically try to form.

When we take away fractions of every Sad Puppy vote so their votes are worth less, because they deserve it, we may have some difficulty explaining to outsiders how this is fair.

If instead we *can* dynamically agree on nomination candidates that get more nominating votes than the SP slate, it's much easier to argue that it's fair. They win now because most of us waste most of our votes on things that cannot win. That's how they do it. If we can overcome this flaw in our own voting we can beat them without jiggering with the rules to take away fractions of their votes.

Since it *looks* unfair to jigger their votes, and maybe it *is* unfair, naturally I want an alternative that can work. The better that fans can come to agreement on some sort of consensus, the harder it is for outsiders (or for that matter insiders) to take over.

#114 ::: Cassy B. ::: (view all by) ::: April 13, 2015, 07:23 PM:

May I make a modest request? I've been seeing the term "bad guys" used in these threads, and I'd really rather we didn't use such loaded language. "Block voters" or "slate voters" is what we're trying to deal with, after all; I seriously doubt that most Sad Puppies are actually bad guys. Misled, perhaps. Misinformed, very likely. (A secret cabal? REALLY?) But not actually bad. (I reserve judgment on Rabid Puppies....)

We're all SF fen. Maybe we don't all read the same thing, but that's ok. Let's leave the demonization to the Rabid contingent.

Cassy

#115 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 07:51 PM:

Kilo @111: Sorry for poor communication. I was trying to explain why the power of blocs in an approval voting system is consistent with the ability of a negotiated set of candidates to defeat a smaller bloc, and then struck by the fact that what we were proposing was similar to a caucus. It wasn't intended as an argument against the proposal.

My issues with continuous voting basically boil down to: (1) implementation difficulties, (2) increased ability for bad actors to wreak havoc, and (3) that it basically embraces making the Hugo award a political process with campaigning, negotiation, arguments about who needs to compromise to push so-and-so out of the top 5, etc. Which is certainly a way that we could go, but it seems to be a prospect that many members of the community regard as undesirable and distasteful.

J Thomas @113: I probably should have called the slates in my example elections something else. I have strong personal feelings about the Puppies but my goal in looking at the voting system is less, "How do we stop the Puppies?" than, "The current voting system is vulnerable to a small but organized bloc controlling the entire ballot, how can we reduce this vulnerability?" Essentially, in my view, the bloc has revealed an exploit in the current voting system, and now we have reason to fix the exploit to stop them or any other group from taking advantage of it. Otherwise, I think the voting structure will lead us to warring blocs, because once enough voters are in blocs the price of not being in one is irrelevance.

Essentially, my goal coincides with the Sad Puppies' stated goal of aiming for a diverse ballot representing a cross-section of fan opinion. I happen to believe that ballot will end up looking a lot more like the pre-2014 ballots than the 2015 ballot, but voters would have the chance to prove me wrong.

#116 ::: Jameson Quinn ::: (view all by) ::: April 13, 2015, 08:01 PM:

@112:

I think that it's probably a rube's game to try to state all the mathematical differences between STV and RAV in philosophical terms. Any words I use are going to be loaded. Even if we had full historical data and could give examples of the differences, how would we agree on which answer was better? And even if we could, what actually matters is next year's election, not last year's. As a statistician, I can't even think about trying to decide on that basis without alarms in my brain shouting "OVERFITTING" and "POORLY-DEFINED ESTIMAND" and "MISSING DATA".

So I'm sorry I even brought up the "broad vs genre appeal" thing. And I'm definitely not going to say a word about the options for hybrid STV-RAV systems, or for tuning the STV quota, or for Bucklin Transferrable Vote, all of which could work but you absolutely didn't hear that from me.

To stay within the bounds of sanity, I think that a choice between STV and RAV should be made based on ballot format. STV is more expressive. Some people like that. On the other hand, that could mean more work. It could also mean more strategic opportunities ("masking" strategy... yuck). That's really the choice.

When it comes to SDV vs RAV: they're really similar. Probably too similar to be worth arguing about. But if the motivation for SDV was to avoid the possibility of ties in RAV, I can say that that's misguided; in theory, ties in SDV between two works with fewer than 5 votes each could snowball and end up affecting which totally-unrelated works win.

#117 ::: J Thomas ::: (view all by) ::: April 13, 2015, 08:57 PM:

@115 Cheradenine

"The current voting system is vulnerable to a small but organized bloc controlling the entire ballot, how can we reduce this vulnerability?" Essentially, in my view, the bloc has revealed an exploit in the current voting system, and now we have reason to fix the exploit to stop them or any other group from taking advantage of it.

I fully agree. The existing system is broken.

Essentially, my goal coincides with the Sad Puppies' stated goal of aiming for a diverse ballot representing a cross-section of fan opinion.

My view is that what's broken is that normal fans do not produce a lot of votes for winners at the nomination stage. That leaves an opening to be exploited by anybody who can get a lot of votes.

I have trouble seeing how we can explain that nominees that get the votes should be discarded in favor of other nominees that get less votes. I can see it when I think about abstract voting systems and abstract goals. But when it gets specific....

Here's a Hugo award nominee, and it has 250 votes. It will not be on the final ballot but will be replaced by another nominee that has 72 votes. I think about how I would explain that to a Fox News guy who wants a 15 second soundbite for his "Corruption in the Hugos" exposee, and my mouth gets dry and my forehead starts sweating.

Would you volunteer to be on call to field that interview if it ever happens?

#118 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 09:07 PM:

J Thomas @117: Sure! Though I hope I would get an assist from whoever's in charge of explaining the final result in the case that the candidate with the most first-place votes loses...

In terms of presentation and PR, though, this does seem like an advantage of a system like STV or SDV, where votes are always getting added to candidates instead of subtracted from them. (Or of reformulating RAV as "double the votes of any ballots who didn't get nominated," but that could produce eyebrow-raising numbers as well. "First we chose X with 100 votes, then Y with 160 votes, and now the third choice is Z with 180 votes...")

#119 ::: Cat ::: (view all by) ::: April 13, 2015, 09:24 PM:

Okay, based on what I've seen so far I'm liking RAV with exponential weighting.

It's not really possible for a slate to *hurt* a despised work, as dh brings up at one point, because the slate still contributes votes to that work, just fewer of them. Unless the slate choses a despised work that has such wide support that it wins before anything else on the slate, in which case all the slate's votes for wanted work just got cut in half, so the slate can go right ahead and throw me in the briar patch; next problem.

The more I think about it, the more I like the idea of more voters getting to have at least something they like on the ballot, which seems to be the point and product of RAV.

One thing does occur to me though. Has anyone modeled how RAV would behave in the presence of two slates? Because we already have two. So if we ended up in some hypothetical future year with the Redshirts Slate and the Puppetmistress Slate, it would be nice to know how the system would behave, or fail.

#120 ::: Emily H. ::: (view all by) ::: April 13, 2015, 09:25 PM:

J Thomas:

I just don't see it as a problem that "normal fans do not produce a lot of votes for winners at the nomination stage." It's just a natural consequence of having a big, diverse field, of which nobody can read more than a small chunk every year. Any way to increase coordination on the part of voters seems to encourage voting for what you haven't read but your friends say is good, or for what you only mildly liked but stands a decent chance of winning, and I think that's more broken than the status quo.

I can't even theoretically manage to believe that Fox News might take an interest in the Hugo Awards even as a "making fun of political correctness" talking poing, but if people can understand that North Dakota has two senators and California has two senators, then they can understand setting up a voting system in a way that gives a louder voice to some than to others.

#121 ::: Edmund Schweppe ::: (view all by) ::: April 13, 2015, 09:28 PM:

Cheradenine @115:

Essentially, in my view, the bloc has revealed an exploit in the current voting system, and now we have reason to fix the exploit to stop them or any other group from taking advantage of it.
This actually ties nicely into the question I've been trying to figure out how to ask for the last hour. I had an idea that could prevent blocs hijacking the nomination process. Details are at entry 96 of the "Clean Living" thread; the quick summary is for the Worldcon Committee to create a section 3.3.17 special category of Best Bloc Nominee (BBN) and use their powers under sections 3.2.7 and 3.2.8 to move bloc-nominated works into the BBN special category, where they won't interfere with the rest of the nominating process.

However, the Hugo admins for that Worldcon would need a good, transparent, mechanical way to figure out which ballots represented bloc nominations, in order to figure out which works to reassign to BBN. The bloc nominators will howl in protest, but the rest of fandom would probably accept it - if that good, transparent, mechanical way exists and the Hugo admins use it.

So, my question: is there a good, transparent, mechanical way to detect (a) that bloc voting is occurring and (b) which ballots are voting for which blocs? And, if so, what is it?

#122 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:36 PM:

3b, 3c, and 3d will all work. Pick whichever is easiest to compute and most comprehensible to the voters.

I've studied election systems for decades. These options (3b, 3c, 3d) are all variants of proportional representation.

Proportional representation is *always* the correct method of choosing a parliament, a committee, or a list of nominees, for very good reasons -- you're trying to get diversity on the committee or list of nominees which represents the diversity of the electorate.

Any proportional representation system prevents bloc voting from working.

If you're trying to get a single winner, or deciding something like the location of next year's convention, you will do better with straight approval voting.

Approval voting also prevents bloc voting from working, but only works with single-winner results.

Every election system is subject to tactical voting, but not all of them are subject to *bloc* voting -- these are systems which are not subject to bloc voting. (Tactical voting in proportional representation systems is much subtler, and quite risky, since it involves voting your second choice and not your first choice, and so risks letting your first choice lose out entirely.)

#123 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 09:42 PM:

JThomas @98: Every one of these proposed changes necessitated by the introduction of slates to the Hugo process is less democratic than the system that preceded it.

J Thomas @99:

@96 Teresa Nielsen Hayden

Do the people who think the Hugo nominees are all the same actually read them?

Some years ago I read about a Chinese convenience store in San Francisco that got robbed by a white man. The clerk could not give the police a description. She said they all look alike.

It's like that.

The hell is it like that.

If they care so little about science fiction that they can't tell one Hugo nominee from another, why are they even voting?

#124 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:45 PM:

FWIW, many serious election math theorists concluded years ago that reweighted approval voting (RAV, 3c) was the best theoretical method for selecting a committee (and thus, a slate of nominees). You can look into the papers on this if you like.

People in the field agree even more strongly that approval voting is the most democratic way to go for single-winner elections. The major problem with approval voting, which has been used successfully by many organizations, is that it keeps getting repealed by self-interested organizing committees in favor of First Past The Post... while the voters aren't paying attention.

#125 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:48 PM:

"One thing does occur to me though. Has anyone modeled how RAV would behave in the presence of two slates?"

Yes. There are papers on this. It's been studied in the context of modeling US legislative elections should we adopt RAV (assume a starting position of two political parties).

RAV gets the third parties some seats in the legislature.

#126 ::: J Thomas ::: (view all by) ::: April 13, 2015, 09:54 PM:

@118 Cheradenine

"Would you volunteer to be on call to field that interview if it ever happens?"

Sure!

OK, try a quick trial run. The cameras are pointed at you, and the microphones, and the bright lights. It's hot.

The interviewer starts his lead-in. "As we have all heard, for the majority of nominations for this prestigious award, the nominations with the most votes were thrown out and replaced by other nominations with a quarter -- or less than a quarter -- as many votes. For example, Jameson's Genociders, a story about patriotic soldiers winning a war on a desert world, was replaced by Mothra Faker, where a transsexual hobbit repeatedly tricks the sexists who try to get him pregnant. But the people responsible for this -- caught red-handed -- admit the facts but claim that what they did was the right thing to do and not corrupt at all. Here is Cheradenine to tell us their excuse."

What do you say in three or four short sentences?

#127 ::: Nathanael ::: (view all by) ::: April 13, 2015, 09:58 PM:

J Thojmas wrote: "Here is my current concern with RAV.

In the nomination I vote for A, B, C, D, E. I think A is the best and I really care about it, but if A doesn't win I want B. C is pretty good. D is OK too. E, meh, I can't think of a better fifth choice.

But then it turns out E wins handily. My other four choices get recalculated as half votes. And then D wins. My other three choices get recalculated as quarter votes. And none of them win.

I didn't care that much about D or E in the first place, but the result of putting them on the ballot was that my vote for A got turned into a quarter of a vote. "

J Thomas: in this situation you should have voted for A, B, and C. If you don't really care about D and E, don't vote for them.

In approval voting systems (where you vote up or down on each story), there is a tactical question of where to make the cutoff: what's "good enough" to be worth spending your vote on.

Deal with it. Every system has tactical considerations. The fact is that approval voting systems are more democratic, and less "game-able", than any other voting system ever proposed. This is confirmed both by mathematical theory and by practical tests.

I'm a bit of a partisan on this; I switched a club to running all its meetings on approval voting a decade or so back, and it saved SO MUCH TROUBLE. Every other voting system featured people complaining that the winners weren't "really" the most popular.

With single-winner approval voting, you can prove mathematically that the winning option is the one the largest number of people approved of.

With reweighted approval voting, using the correct weights, you can prove mathematically that the list of "winners" (nominees in this case) is the list with the largest number of voters approving of at least one winner.

If your goal for your organization is to have lots of members who feel that they have been represented, this makes approval voting variants obviously the best. (If that isn't your goal, well, democracy and voting may not be right for you.)

#128 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 09:59 PM:

Nathanael@124 (and others):
FWIW, many serious election math theorists concluded years ago that reweighted approval voting (RAV, 3c) was the best theoretical method for selecting a committee (and thus, a slate of nominees). You can look into the papers on this if you like.

This kind of gets to the heart of my concern expressed in #100: I think very few fans are willing to take "trust me, I'm an expert" as an acceptable reason to support a particular voting system. It may, in fact, be true that it does work, and works well, but so far I've only seen "appeals to authority" when what we need is a coherent, easy-to-understand explanation that (most importantly) explains why the system is, in fact, fair. As others have said, if you lose that appearance of fairness, then all is lost no matter you do.

The discussion of "how do you explain why the person with the most votes lost" gave me a cold shiver. Consider that there will be those who will delight in making whatever system that is proposed look bad, and certain people have shown that they aren't above misrepresenting what has been said to make their points. Now imagine that you've just made a statement similar to the quote above. I predict that saying, "Well, yes, but..." isn't going to cut it with a large portion of the HAHV's.

I'd almost go so far as to say that a system which seems fair may be the single most important of Bruce's criteria.

Please keep in mind that I'm not saying any of the option 3 systems are not fair, but I'm beginning to see that it's critical that the system be explained in such a way that the HAHV's can -manifestly- see that it is fair.

Kilo

#129 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 10:09 PM:

Cat @119: I tried a two-slate model as one of the examples I ran in post #3. Each slate got some nominations, as well as one or two popular non-slate candidates. (The counts depended on the popularity of the slate choices and how many slate voters there were.) This matches the behavior Nathanael cites in #125.

Nathanael @124: Do you have some citations? I'm certainly not up to date on the literature in any comprehensive way - I'm a teacher, not a researcher, and this is a small fraction of what I teach - but almost all the mathematical analysis I've read has given phrases like "most democratic" a fairly wide berth, because it's loaded and difficult to define.

#130 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:13 PM:

The usual explanation for proportional representation in legislatures goes like this:

40% of people vote Democratic. 40% vote Republican. 10% vote Green. 10% vote Libertarian. So the legislature should be 40% Democratic, 40% Republican, 10% Green, 10% Libertarian. We need a system which does this.

The same argument can be used for selecting a list of *nominees*. If 10% of the nominating population support colonialist military SF and 10% support gender-bending SF and 20% support mass-market explosion-oriented SF and 10% support wordplay-heavy literary SF, the list of nominees should represent these tastes in proportion to their presence in the electorate.

Explaining *why* a system like RAV achieves proportionality requires a little math. No getting around it. It's simple math, but there's nothing to be done for people who refuse to read a couple of pages of it.

Approval voting is much, much, much simpler to explain -- the most popular candidate always wins -- but you'd have to run a very different system in order to use it (you don't really need nominees with an approval voting system).

#131 ::: nathanbp ::: (view all by) ::: April 13, 2015, 10:16 PM:

RAV doesn't really seem that hard to explain. "Each time a work on your ballot is nominated, it counts for half as much for picking the rest of the nominees." Maybe I'm missing something, but I think that covers the gist of it accurately?

#132 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 10:17 PM:

Nathanael, the moderators are discussing your most recent comment. This may take a while.

#133 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:27 PM:

Upon looking again for the first time in a decade, I see that most of my academic citations for this stuff have dropped off the internet (sigh). Election math is an old field. I suppose I'm not surprised that people are reinventing the wheel.

There are some cites. Straight-up approval voting has its own advocacy group, which has some links:

http://www.electology.org/

This isn't the first such group, but the last one, which had a lot of academic links, gave up and dropped off the internet...

Range voting has its own advocacy group: it's mathematically equivalent to approval voting, but may or may not be an easier sell (the Olympics uses it):
http://www.rangevoting.org/

And some other links.

http://www.tursiops.cc/idhop/av/

Reweighted range voting (using the very specific weights needed to guarantee proportionality):
http://www.rangevoting.org/RRVr.html

I spent a lot of time studying this stuff two decades ago, first in the context of governmental elections, and then in the context of running committee meetings, where I got some practical experience. (My god, approval voting simplifies parliamentary procedure with regard to amendments massively -- throw out all the possible amended versions at once and have everyone vote up-or-down on each of them. Most popular version wins.)

The conclusions haven't changed in two decades. For most purposes which people might want to use a voting system for, you want either Approval Voting or Range Voting (single-winner) or Reweighted Approval (or Range) Voting (multi-winner). Multi-winner STV works reasonably well too. (Single-winner IRV doesn't, by the way.)

If you're actually *trying* to make bloc voting successful at suppressing candidates with wide popularity (and I'm sure some people are), then you might try a different voting system.

The real problem is that most people don't give a single thought to whether the voting system actually does what it's supposed to. This is a worse problem in the US than in the UK, where voting systems remain a hot topic of discussion.

Anyway, upon Googling, apparently the Oscars use reweighted range voting to determine the Best Visual Effects nominations. Maybe the message is getting through slowly.
http://www.goldderby.com/news/7980/oscars-boyhood-birdman-selma-imitation-game-entertainment-13579086-story.html

#134 ::: David Dyer-Bennet ::: (view all by) ::: April 13, 2015, 10:36 PM:

I don't think STV and IRV and the like work well for sparse long-tail things like...Hugo nominations. (I'm slightly surprised at how well some of the STV tests seem to have gone; but we don't have real data, and I wouldn't be surprised if real data is more diverse and thinner than the test data we've used.) Minneapolis is a ranked-choice voting city now, which is good, but it's a LOT different from Hugo nominating. Hugo voters of course are already intimately familiar with the procedure since it's used for the final ballot since time immemorial (i.e. before I got involved).

Possibly some statement against promulgating slate ballots might be useful. I'm utterly certain that any attempt to solve the problem by simply forbidding them will have horrific outcomes, though, probably in the very first year. Secrecy is one counter-measure, gaming the rules is usually possible, *and* it's a hammer that can be waved against legitimate attempts to stir up interest in Hugo nominating. It would be a huge disaster. I wonder if anybody vaguely competent would consent to be Hugo administrator under such rules? And then a less-competent admin would make things worse. Etc.

At some point a group stops being a "voting block" and becomes "a solid majority of the members" :-) . Reasonable democratic rules can't protect us from giving a Hugo award to a work that's the favorite of every single Worldcon member (of course, that degree of popularity is also not really a worry; then again even half easily suffices). And one of our better protections is to get more of our existing members involved in nominating. I don't see rules-changes relevant to that, but maybe some publicity dealing with the "I haven't read enough to nominate" idea?

#135 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:39 PM:

Teresa -- no worries from me about moderation, do as you like. I just found myself here and went "what, reinventing the wheel on voting systems again?" and sighed. Sometimes it feels like human knowledge does not advance over time.

The problem of having a list of nominees which reflect the diversity of the voters (rather than ending up with a "bloc slate") is known as the "representative committee" problem in the literature. Any proportional system will do. I happen to think that the range/approval variants are easier to vote, easier to count and provide less weird gaming-the-system opportunities, but Single Transferrable Vote with multiple winners has worked just fine in many countries, so I'm not going to complain about it.

#136 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 10:39 PM:

Nathanael @130: Fandom is not going to Balkanize the Hugos according to your schema of what's important in literature. You need to recognize that your map of subgenres and categories is not an objective map of the world. It's your personal map of your personal reading. The rest of us are unlikely to adopt it.

The Hugos are about what the voters love. When you vote in the Hugos, vote for what you love. Take the risk of loving it for its own sake, whether or not you think it has a chance of winning. Let other voters do the same.

#137 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 10:42 PM:

Me @136: Because believe it or not, that's actually how it has worked until now.

#138 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:50 PM:

Mr. Dyer-Bennett -- you might be surprised how sparse and long-tail most municipal elections actually are. :-P I've seen municipal elections with a large number of candidates (more than 12) and a widely scattered vote. From a distressingly small number of voters.

Any proportional voting system seems to work fairly well for selecting a "representative committee", regardless of how sparse and long-tail the voting is. The issue of "more people have heard of this candidate than that one" remains unavoidable, of course.

STV with multiple winners (at least three) is proportional; IRV with a single winner isn't, and has some problematic mathematical properties which tend to reduce it to first-past-the-post, but can actually make it worse in some cases.

The difference between STV and IRV in practice is stark. It can best be demonstrated by the Australian Parliament's lower house (single-winner IRV, behaves exactly like first-past-the-post, two-party system, winning party isn't always the most popular party) and its upper house (multi-winner STV, has proportional representatives from many parties, including both the Greens and "Palmer United" which is basically backers of coal mining).

I do not think of STV and IRV as the same voting system, because the results are so different.

#139 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 10:50 PM:

Nathanael@130:

Explaining *why* a system like RAV achieves proportionality requires a little math. No getting around it. It's simple math, but there's nothing to be done for people who refuse to read a couple of pages of it.

Well, and I say this with much respect for your obvious expertise, but I think being dismissive of a (potentially) large portion of fandom in this way is exactly what we must avoid at all costs.

I can certainly see working out the merits of various systems mathematically (as we are mostly doing here) and then coming up with a popular way to explain it after the fact. I'm also okay with (as I suggested above) coming up with a "mission statement" that we want to achieve and finding a system that best accommodates it. What we cannot (again, in my opinion) do is to say to fandom, "If you can't be bothered to figure out why this is a manifestly fair system on your own, then tough for you."

Ultimately, I think we can all agree that the Hugo -is- the fan's award, and I feel our primary responsibility is to make sure that the fan's continue to feel it is their award.

Kilo

#140 ::: Cat ::: (view all by) ::: April 13, 2015, 10:56 PM:

Cheradenine @129 Oops! Sorry I missed that, and thanks for pointing that out; I'll look at that again.

As for explaining RAV so that everyone thinks it's fair, I don't think that will be possible. For some people, no system is fair until they win all the marbles and own the field. Some people, for example, were very angry about the final Hugo voting process last year, because a person can have the most 1st place votes and still end up in last place if everyone else puts them low enough on the ballot.

You need to be able to explain RAV well enough that a person without a specific axe to grind thinks it's fair. I don't think Fox News will ever be in that category. But the people we need to please are the fans.

For them I think "the thing with the most votes becomes the first nominee. Once one of your picks becomes a nominee, your other votes count for half. We count the votes and half votes up. The thing with the next most votes becomes the second nominee. If two of your picks have become nominees your other votes count for a quarter, if only one, they count for half, if none, they still count full. We count the votes and half and quarter votes again. The thing with the next most votes becomes the third nominee... and so on. The end result of this is that as many people as possible have at least one thing they liked on the ballot."

One of the questions that Joshua brought up I haven't seen answered yet. Sometimes nominees decline the nomination. Is the ballot recalculated from there? If it is, could that lead to some second nominee that did make the ballot the first time failing to make the ballot the second time? Because that's the kind of thing that could produce hard feelings, I think.

#141 ::: Nathanael ::: (view all by) ::: April 13, 2015, 10:59 PM:

Teresa: *shrug* I'm not sure how you got that response from my comment.

I'm pretty sure you have specific tastes which tend towards particular genres or subgenres or styles. I've never met anyone who didn't. Some people have a more conscious understanding of their own tastes than others. Most people fall into groups regarding their tastes.

Award winners for well-run awards tend to appeal to people who have a wide variety of different tastes. (That's often what makes them so special.)

Creating a good list of *nominees*, however, is quite another matter. The goal of a pre-published nominees list is, if I remember correctly, to get people to read and consider stuff which they might not ordinarily have read. To do this, you *deliberately* want to get stories put on the list from people who tend towards different tastes.

Because you have to assume that before the nominees list comes out, a lot of people will simply not have looked at the stuff which wasn't to their 'typical' taste.

If this exposure wasn't the goal, you could skip the entire nominee procedures and just let people vote straight up for anything for the final vote.

So if you want the nominee list to work this way... you should design the system for nominee selection to *make* it work this way. You can call this paranoid, but it's really the "security mindset" that Bruce Schneier describes: assume that someone's going to try to break your system and preemptively design a system which doesn't break.

#142 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:05 PM:

J Thomas @126: OK, I'd give this more thought (and have more data at hand) if it were a realistic possibility, but, "The current system is designed to represent as wide a group of fans as possible on the final ballot. The final ballot includes choices from 52% of the ballots that were submitted. Using the system you're suggesting, only 24% of the ballots would have any representation."

And then if they ask if it isn't true that the system was just changed, say, "Yes, the concern was raised that the old system didn't guarantee a wide range of nominees."

This won't sway Fox News, but the best you can do in this scenario is try to give a clear and accurate answer. (I should note that this took me a while because it took a lot of gyrations to find a scenario where this could even happen to estimate what the percentages would be: the slate has to get 2 votes, Jameson's Genociders has to be listed on slate ballots and absolutely no others, and at most two non-slate books can get more than 5-6% of the total vote (and they can't share any votes with . It's basically a perfect storm, the equivalent of a book with 6.26% of the first-place votes winning the IRV round, which is also conceivably possible. But Fox could still generate outrage over a win by a work with 281 votes over one with 283, if they cared enough to.)

Kilo @128: I genuinely am trying not to argue from authority here. I don't consider myself an expert, for one thing. You see above my stab at arguing that it's "fair," but I'm honestly a little uncomfortable even using the word. There are a lot of ways to define fairness, and no voting system can meet all of them.

Nathanael @127: I'm uncertain that you can contribute much to this conversation by telling people to deal with it and making statements like, "With single-winner approval voting, you can prove mathematically that the winning option is the one the largest number of people approved of." Well, yes, because that's how approval voting is defined. I might as well say that with Condorcet voting, you can prove mathematically that a Condorcet candidate will always win. It doesn't add anything meaningful to the discussion if everyone pops in and rattles off the fairness criteria their chosen method satisfies.

#143 ::: Brad from Sunnyvale ::: (view all by) ::: April 13, 2015, 11:08 PM:

Bruce asked me last night to do some commentary on this posting, but I had too much work to do. And still do. But I have written up something fairly detailed on the philosophy around the Hugos and why I have come to a different opinion from most here.

Second Musings on the Hugos

For the TL;DR crowd, I outline what I think the purpose of a Hugo voting system is -- "to aggregate the opinion of the members, as accurately as possible, on what the best work of the year is, if it were the case that most of them actually read/evaluated most of the leading contenders." The nomination process helps find those contenders to give people a chance to look at them.

In looking at all the voting systems, I see too many flaws. Aside from complexity, they all add strategy. There are too many times when a member, armed with only a little information on likely popular choices, is best guided to deliberately leave certain works they actually loved off their ballot to improve the chances of others. I find this unacceptable.

The second thing I find unacceptable is that all these systems still put at least one, and as many as 3 slate candidates on a ballot from a group that is a minority (like 10%) of members. If we strive for the goal of an accurate measurement of true fan opinion, this is not right. Sure, you stop them from getting 5. But why is one OK if it needed collusion to get it on there?


In the end, I conclude something different. The only defence against attack by humans is not a set of rules, but a system of justice. Pit humans against humans. The humans on our side, the side of justice, will have some rules, some transparency, some accountability, but they will also have discretion so they can solve not just the current problem, but anything else the attackers throw at the system.

I know many will not like some of the examples I give of how to do this, but I want to focus on that principle. Only human judgement will foil clever attackers. Attempt to foil them by committee with voting system complexity will both fail, and also cause side effects.

#144 ::: David Dyer-Bennet ::: (view all by) ::: April 13, 2015, 11:08 PM:

My memories of nominating, and discussing nominating, definitely agree with people talking about diversity within ballots. My actual experience is that while each of us prefers some sub-genres of SF, we are able to recognize outstanding works in most of them. (Note, *most*; at least some of us despise some sub-genres enough to be unable to recognize outstanding works there.)

Thus for example, long ago, lots of *the same people* thought it was clear that both The Dispossessed and The Mote in God's Eye belonged on the ballot that year. They were also new works from major authors, so very broadly read, which always helps.

So I don't think that, absent slates, the probable distribution of real nominations is one pile with all space opera and one pile with all feminist fantasy, but a much more mixed arrangement.

#145 ::: Tammy Coxen ::: (view all by) ::: April 13, 2015, 11:08 PM:

I think the best way to handle declined nominations is just to offer it to the next person on the list. The way the process works is that the Hugo Administrators reach out to the nominees and ask if they'd like to accept or decline. If they are unable to reach someone, the assumption is acceptance. I don't know what happens if they receive a decline late, after the finalists have already been announced, but you definitely wouldn't want to recompute the entire finalist list after it's been made public. So it seems to me that just going to the subsequent nominee is the best choice.

#146 ::: P J Evans ::: (view all by) ::: April 13, 2015, 11:11 PM:

134
I'm hoping to have real not-exactly-live data next week, maybe the week after.

#147 ::: TomB ::: (view all by) ::: April 13, 2015, 11:13 PM:

I would like the Hugo nominations process to be better at representing diversity than the current FPTP process. Both the diversity in the candidates, and also the diversity of the community making the nominations. Of course it has to minimize the effects of slates, because slates are terrible for diversity, but it shouldn't just be an anti-slate measure, it should be pro-diversity.

This is another way of saying that Option 3 looks good. I don't know which of the variations would be best. They all have an aspect of "okay you got one of your choices, now let the other fans have a chance."

#148 ::: Brad from Sunnyvale ::: (view all by) ::: April 13, 2015, 11:18 PM:

Ok, odd, my link in #143 is missing even though I double checked it. Let's try again.

Second Musings on the Hugos

OK, looks like you really need the quotes! See #143 for TL;DR summary.

#149 ::: Cheradenine ::: (view all by) ::: April 13, 2015, 11:27 PM:

Brad @143: I respect your opinion here (though I think it's a bit tangential to a thread dedicated to "exclusively focused on voting systems and their relative merits"), but I do want to challenge the idea that more complicated systems necessarily add more strategy. Determining whether leaving some candidates off your ballot is strategically beneficial or not in RAV, for example, is a very hard problem even if you have exact knowledge of how everyone else voted; in general, you will get the best chance of an optimal result by voting the precise candidates you want to see on the ballot. (This is actually not true of the current system. Reference is here, but it's technical.)

P.J. Evans @146: I'm trying really hard not to drool on my keyboard. Mmmm... real data. (I suppose "live data" is not the way to go when the data is considerably older than most of my students.)

Cat @140: Declines are a bit tricky. My instinct would be to say "rerun the process, but only to the point that it selects a person who wasn't in the original nomination group, and offer it to that person." I'd have to look at this more and/or check the literature to see if it leads to any problematic behavior.

#150 ::: AnnieY ::: (view all by) ::: April 13, 2015, 11:30 PM:

This will be long but I had been thinking...

No matter what new system is chosen, it will be gameable. Not as easy as the current one but there is no system that cannot be gamed provided enough will. And the puppies can be a lot of things but I would not call them stupid and I would not be surprised if they hire someone to help them figure out whatever system the Hugos come up with.

So the most logical answer seems to be to stop the system from being gameable by making it impossible for anyone to predict what system will be used this year. I don't mean announce it after the nominations close (that obviously will not work) but tie it to a number that cannot be predicted. Number of valid ballots for novel. Or for short stories. Or for fan artist. If need be vote for which category to be used on each Worldcon for the next nomination cycle.

Then build a table that defines which algorithm is to be used based on the interval the number goes into. Make the intervals 10 wide, 50 of them (so 0-10, 10-20 and so on up to 490-500) and if there are more than 500 ballots divide by 500 and use the remainder.

And just to make sure that noone calls fault, build the table on every Worldcon, if need be with double blind draw in two separate rooms - one of them choosing the order of the intervals, the other one choosing the order of the algorithms. If we have 5 algorigthms, each takes 10 spots so this is the 50 elements that you can draw out of a hat/ball/whatever.

Once the table is built, just publish it on next Worldcon site or on the federation site and forget about it. Until it is time for the next nominations counting.

Yes - this eliminated ANY strategic voting but.. technically that is the idea - trying to vote strategically puts you in a path that leads to slates and decision for other people (most people won't go that far but...). This also will close all the rumors on a cabal that decides anything - noone can predict how many valid ballots there will be in a category in a year - unless if someone decides to say that the Hugo administrators are part of a cabal or something - and noone from any side had done that so far (and they would not). And even if numbers leak, noone can predict when the last 10 fans from somewhere will decide to vote - I added a nomination for a movie on my ballot ~3 minutes before closing time - before that this category was empty for me.

The main issue will be that the people that count will need to know all the systems but there is enough people in this thread that can help with that I suspect and that should not be such a problem.

Under these rules, a slate can end up with all works on the ballot as this year but can also go against an algorithm specifically built against slates. Somehow I do not see the puppies running the chance. And if they do, well - at least we had done all we could to try to stop the gaming.

#151 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 13, 2015, 11:43 PM:

Comments will be shutting down at midnight. We'll be back tomorrow morning.

#152 ::: Brad from Sunnyvale ::: (view all by) ::: April 13, 2015, 11:49 PM:

#149: I disagree. The reason Option 4 above is my current choice is because there are too many flaws in the proposed systems, which is what we are discussing. This thread and the prior have outlined them, and so I think Option 4 (and grander versions of it) are the only remaining approach that works towards the goals.

I think RAV is one of the easiest ones to be strategic on, but for almost all the proposals above, which "punish" your ballot for having named a popular winner. The strategy is simple -- if you think there's one or two obvious choices, don't vote for them. They will get on the ballot without you, and your ballot will weigh more strongly to promote your less obvious selections.

Yes, if lots of people do this, then we get a strange counter-result, which would be even worse, which is one reason why you don't want your ballot system to have a strategy. But more than that, the existence of strategy makes the system inherently more complex, because you must not just understand the system, you must understand the strategy.

So I view that any system which seriously increases the effectiveness of strategy fails the complexity test, as well as the most important test of all -- does it make members express their true desires? If people don't vote what they really feel, the ballot is useless.

Approval worked in the past because it has really minimal strategy. You can leave off a popular choice, but the only thing that buys you is the ability to name another item on your ballot. Risk hurting one of your top choices to give a boost to your #6 choice -- not a very good deal, and that's why there is not much strategy to the current system, and why we liked it.

It was our last, best hope for peace. It failed.

I'll tell you now, if we use RAV or similar, then I'm going to take any choices I have which showed their popularity by winning the Locus poll, or doing well in other awards that came out earlier, or any other thing I can find that's a good predictor, and I am not putting it on my ballot.

I don't think so many will do it as to take it off the ballot, so I won't cause any harm. Though when the final tally is published, it will not have done as well as it would have in an honest and independent poll.

That's why we like STV for the final ballot, with all its flaws. There is no strategy. It's a much more important feature than people are imagining.

#153 ::: Keith "Kilo" Watt ::: (view all by) ::: April 13, 2015, 11:57 PM:

A question for the experts:

I'm going to try to take my own advice and lay out an explanation first, then propose a voting system around it. This particular one almost has to be a variation of one of the option 3 systems, but I'm uncertain which one. Can you point me to it?

Here's a proposed system:

Everyone is told that at the end of the day they will be able to nominate no more than one nominee per category (and possibly none, if there is insufficient support for all of your preferred candidates). The goal is to give you the best chance of having one of your preferred candidates make the ballot, and preferably, one that you ranked highest on your list.


The process is that they list their top five nominees, in order of preference. Important: Even though the nominators have listed their nominations in order of preference, that is only to determine when they are "finished" having a say in voting. A vote for a work anywhere on the list counts the same. If, after tallying all of the nominations, their first choice wins, none of the rest of their nominations in that category are counted. Poof. Their nomination ballot no longer exists for that category.

For the next round, we eliminate the first nominee's name from all nomination ballots. We go through -all- of the remaining nomination ballots and select the nominee who gets the most nominations. If your first choice candidate was already chosen, then you have no say in this round -- you already got your candidate. If your first choice candidate did not win, however, then your nomination ballot is counted, exactly as if this were the first round (though you may only have four nominees on your ballot if one of your lower-ranked nominees came in first in the last round). Again, a vote for a name anywhere on your ballot is a vote for that work to be nominated. If your first choice has now been selected, again, you're done. If not, you get to participate in the next round of nominating. This continues for a total of exactly five rounds, selecting the five nominees for the Hugo award that will actually be voted on.

I -think- this is similar option 3d, but it's not exactly as Bruce describes it; I claim it's a little bit simpler to understand. It may not work as well mathematically (I'm not qualified to judge, but I'm hoping someone here is), but it seems to me that the "perceived fairness" rating is high (whatever that may mean). There is almost certainly some tweaking that would need to be done, hence my appeal to the experienced ones here.

A slate will almost certainly get one nominee on the ballot -- but only one. If they have a large enough representation, I don't see that as a problem -- there is something to be said for giving a majority a say, yes? I'm probably not seeing all the ways to game the system, and I suspect I'm a little more tolerant of "tactical voting" than some here are, so long as it leads to a genuine consensus, so I may be blind to it. I'm immediately curious what happens if there are two (or more) slates. Is this similar to the case we were discussing where basically if there are five slates, then they will each get to choose one of the final nominees? If so, can you explain why for me?

Additionally, what if a truly well-organized slate says, "We want 20% of our nominators to list this as their first choice, 20% to list it as their second choice, 20% to list it as their third choice, and so on." Does this break the process? The way to fix this might be to change the system so that if -any- of your five candidates makes the ballot, then your nomination ballot vanishes. I think this, too, is true to the original proposed philosophy.

Can one of the experts help me out? Again, my goal here is to try to explain what I mean by coming up with an explanation that everyone can intuitively understand and then designing a system around it. I have no problem using the "pure form" option 3 variations, so long as we can meet this criteria, but right now, I don't see that any of them really do.

Thanks,
Kilo

#154 ::: Cheradenine ::: (view all by) ::: April 14, 2015, 01:18 PM:

Kilo @153: At first I thought this was a slight variant of STV, but after some consideration I realized that, assuming informed voters, this system works out as equivalent to single non-transferable vote (everyone gets exactly one vote and we take the 5 works with the most votes) – i.e. the current system with each voter reduced to one nomination. The reason is that there’s no motivation to list anything in slots 2-5, because all that having a work there can do is get your ballot tossed out – it’s never going to help that work get elected. So optimal strategy is to only list one nomination.

In general I think we’re better off using a system that has been proposed and studied in the past, because there will be existing research on its implications, advantages and disadvantages, vulnerabilities, etc. (That said I do like SDV a lot more than I did at first – which brings me to...)

Joshua @88: I ran two sims using SDV, one using the 2013 nominations with bullet vote for #6 case, and one using the slate of 150 case. In both cases, the results agreed with RAV. I didn’t simulate more elections because SDV is considerably more work than RAV (since if there are N candidates above the cutoff, you need to go through N-5 iterations instead of only 5), though this could be eased by program support.
I’m warming up to this method. It yields a similar outcome to RAV with d’Hondt weightings, but I think its fairness is more intuitive. (“Everyone gets one vote split among the works on their ballot, and as we eliminate the works with the least votes, the vote gets redistributed to the choices left on the ballot. So everyone continues to have the same amount of say until all their nominees are eliminated from the ballot.”) The reported vote counts only increase, so dealing with the investigative reporter is a bit easier.
I would really like, though, to find some literature on this method, or find an actual voting systems expert to analyze it. Using an untried system can yield unexpected surprises.
Incidentally, it’s interesting to see how different the results look for slate and non-slate candidates. Non-slate candidates tend to slowly and steady gain votes as other candidates are eliminated, whereas slate candidates would stay at a fixed number of votes and then jump up when one of the slated candidates was eliminated. (This might be less apparent with real data, where there might be more correlations between votes for non-slate nominees.)

Brad @152: In general, all voting systems have flaws. The flaws in bloc approval are just flaws we’re more used to accepting, but that doesn’t make them less significant.

“The strategy is simple -- if you think there's one or two obvious choices, don't vote for them.” I’m glad you said this, because it prompted me to think through the details of why RAV is more strategy-resistant than straight AV (despite appearances), instead of relying on a theorem in a paper. Here’s the basic issue: being among the top few vote-getters in a proportional voting system is not an absolute guarantee that a work will be nominated.

Suppose that your top five choices are A, B, C, D, and E (in any order), with F as your sixth choice. You are certain that a lot of people are going to put A on their ballot.

Case One: You are sure that A is going to get far more nominations than any other candidate.
In this case, it’s safe to leave A off your ballot. Under AV, it’s probably better strategy to list BCDEF, on the theory that you’d like a final ballot with A, F, and three things you hate better than one with A and four things you hate. Under RAV, listing either BCDEF or BCDE could make sense, depending on how strongly you feel about F.

Case Two: You are sure that A will be among the top 2 to 4 vote-getters or so.
Under AV, it’s still good strategy to vote BDCEF. A doesn’t need your vote to make the ballot, and as with the previous scenario, you might prefer that A and F get nominated over a scenario where A is your only choice that gets nominated.

Under RAV, voting BCDEF or BCDE can help you, but it can also hurt you, because the order in which candidates are selected matters. If your withholding a vote for A means that G gets nominated before A, it’s possible that adjusting weights for G’s selection could hurt the standing of A and cause it to drop out of the nominee list. In fact, ballot configurations exist where you voting for ABCDE causes both A and B to be nominated, but voting for BCDE or BCDEF causes none of your choices to be nominated.

This scenario has low probability, but the scenario where you having 1 vote for B instead of 0.5 vote determines whether B gets nominated also has low probability. And in the absence of complete information about how everyone else is voting, you cannot determine which scenario is more likely. Therefore, there is not a clear advantage to voting against your true preferences.

One issue here, though, is that the strategy of withholding votes from more popular works can appear to be best in the absence of a more sophisticated analysis. We would need to get the word out that it’s not the case, but some people would probably still follow their instincts to leave off their perceived frontrunners.

(This leaves aside that AV also has a dominant strategy of “join the largest bloc,” as we’ve seen.)

On a side note, the statement that there is no strategy with STV/IRV (in your last sentence) is incorrect. It’s more strategy-free than first-past-the-post (or bloc approval), but there’s a significant spoiler effect still present (where it can be tactically better to list your second choice first if your second choice is likely to perform better against your last choice). More detail is here. It also has some vulnerability to a tactic called “push-over.”

All: just so you know, I’m in classes all day today. I wrote this in the early morning and am hoping to steal a minute to get it posted after the thread reopens, but if people have questions about any of this, I probably won’t be able to read and reply until tonight.

#155 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 01:25 PM:

nathanael@124, 125: I absolutely agree with your conclusions. But I think you’re overstating the consensus in the field a bit. And I’d like to see your references.

… I did this for AD_FV, so I’ll do it for nathanael to be fair. Nathanael seems to be knowledgeable about voting theory and seems to be basically on “my team”. Election theory is a small world — I recognized Clay and AD_FV immediately, and I think would have strongly suspected I knew who they were even if their names hadn’t been a giveaway. However, I don’t recognize nathanael by name or style. And again, I agree with him, but personally I’d tend to qualify my statements a bit more.

“How would you explain how the thing with the most votes lost”: that would never happen. The thing with the second-most votes might lose. In that case, I’d say something like what Cheradenine@142 suggests.

Brad@152:I think RAV is one of the easiest ones to be strategic on,

Actually, that’s pretty much the opposite of the truth. That is to say: there are various ways you could precisely define “easiest” and “be strategic” in that sentence, but for the definitions I find most natural, I believe you could prove that RAV is one of the hardest to be strategic on, especially in the kind of situations I think would be common in Hugo voting (where overwhelmingly dominant winners are rare). Certainly it’s harder to use strategy in RAV than in STV.

I’m not saying that the strategy you’re suggesting here would never work; something like that kind of strategy is possible in any PR system I know of, and I bet you could rigorously prove that it’s possible in any deterministic, Droop-proportional system with fixed numbers of seats and finite candidates.

But if your preferences are ABCDE in that order, and you think C is going to be nominated without your vote, under RAV you might decide to vote AB, or you might decide to vote ABCDE, but voting ABDE is a very risky strategy; I think it would be attempted rarely and successful basically never. If everybody uses the strategy where they vote AB in that situation, I think that leads to a pretty good chance of a good slate of nominees; perhaps even better by some diversity criteria than if everybody votes ABCDE. And in the long run, people would learn that their prior “certainty” that C will be nominated is often unwarranted, so plenty of people will give up on such finicky strategies and just vote ABCDE.

@153: Under that system, a slate could take 4 nominees in each category by just telling all their members to find random obscure works to put first.

And yes, I’m sure you can come up with a way to fix that pathology. But you’ll only create some other pathology. Patching problems is not a good way to design voting systems. And I trust a system whose problems I understand much more than one which seems too perfect. I think I understand the issues with RAV, and they’re minor (as I argued just above with Brad).

In a larger sense, I appreciate what you’re trying to do. You want a system that is easy to explain in terms of simple, easy-to-grasp principles. The trouble is, well-designed voting systems involve finding elegant solutions to various trade-offs between competing values. (In fact, you could say that that’s what a good voting system is; a way for groups of people to find a good trade-off among their different values). If you look at a voting system “from only one side” — in terms of only one basic principle — it’s always going to look a bit flawed.

I’m a grad student in statistics. When teaching statistics to undergrads, this kind of problem is common. There’s a lot of times when some result is based on more than one abstract principle, and it’s really hard to “get” it intuitively because without some practice, your naive intuition can’t see it from enough sides at once.

So what’s the kind of practice that gets you there? Concrete examples. In the case of voting theory, voting scenarios. I could certainly make a set of simple examples that shows how RAV works and why it’s a good thing that it works that way.

But there’s a danger with that. No voting system is perfect, and if you want to make some system look bad, it’s always possible to create a scenario that does so. So from the point of view of a non-expert, there’s always going to be competing experts with competing systems and scenarios and arguments. Each person has to decide for themselves how much of that to carefully decipher and analyze, and how much to tune out. For most people, they’ll start tuning out pretty quickly. So if they want to have an opinion, it’s going to be based on whom they trust and whom they don’t.

As an expert, how do I get people to trust me? I have some ways, but I’m not going to pretend I have it all figured out. I’m upfront about my biases. I try not to overstate my case, but I also try to avoid making overly-technical qualified statements that sound like fine print. I try to say things in a way that I think a non-expert could understand, but that would also make sense to an expert; I’m sure I often err on the side of the latter. I try to give people good heuristics for detecting fake experts; for instance, if somebody tells you their voting system is perfect, or they that this one precise scenario shows that some other voting system is bad, don’t trust them. Look for simple, realistic scenarios, along with some analysis of how they could vary and/or go wrong.

So… anyway, as yet another “nothing up my sleeves” gesture, I’m going to invite others to create simple scenarios to show off RAV. I imagine it would look at the first three winners in simple elections with

#156 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 01:47 PM:

Grr, another borked comment.

Here's another try:

nathanael@124, 125: I absolutely agree with your conclusions. But I think you’re overstating the consensus in the field a bit. And I’d like to see your references.

… I did this for AD_FV, so I’ll do it for nathanael to be fair. Nathanael seems to be knowledgeable about voting theory and seems to be basically on “my team”. Election theory is a small world — I recognized Clay and AD_FV immediately, and I think would have strongly suspected I knew who they were even if their names hadn’t been a giveaway. However, I don’t recognize nathanael by name or style. And again, I agree with him, but personally I’d tend to qualify my statements a bit more.

“How would you explain how the thing with the most votes lost”: that would never happen. The thing with the second-most votes might lose. In that case, I’d say something like what Cheradenine@142 suggests.

Brad@152:I think RAV is one of the easiest ones to be strategic on,

Actually, that’s pretty much the opposite of the truth. That is to say: there are various ways you could precisely define “easiest” and “be strategic” in that sentence, but for the definitions I find most natural, I believe you could prove that RAV is one of the hardest to be strategic on, especially in the kind of situations I think would be common in Hugo voting (where overwhelmingly dominant winners are rare). Certainly it’s harder to use strategy in RAV than in STV.

I’m not saying that the strategy you’re suggesting here would never work; something like that kind of strategy is possible in any PR system I know of, and I bet you could rigorously prove that it’s possible in any deterministic, Droop-proportional system with fixed numbers of seats and finite candidates.

But if your preferences are ABCDE in that order, and you think C is going to be nominated without your vote, under RAV you might decide to vote AB, or you might decide to vote ABCDE, but voting ABDE is a very risky strategy; I think it would be attempted rarely and successful basically never. If everybody uses the strategy where they vote AB in that situation, I think that leads to a pretty good chance of a good slate of nominees; perhaps even better by some diversity criteria than if everybody votes ABCDE. And in the long run, people would learn that their prior “certainty” that C will be nominated is often unwarranted, so plenty of people will give up on such finicky strategies and just vote ABCDE.

@153: Under that system, a slate could take 4 nominees in each category by just telling all their members to find random obscure works to put first.

And yes, I’m sure you can come up with a way to fix that pathology. But you’ll only create some other pathology. Patching problems is not a good way to design voting systems. And I trust a system whose problems I understand much more than one which seems too perfect. I think I understand the issues with RAV, and they’re minor (as I argued just above with Brad).

In a larger sense, I appreciate what you’re trying to do. You want a system that is easy to explain in terms of simple, easy-to-grasp principles. The trouble is, well-designed voting systems involve finding elegant solutions to various trade-offs between competing values. (In fact, you could say that that’s what a good voting system is; a way for groups of people to find a good trade-off among their different values). If you look at a voting system “from only one side” — in terms of only one basic principle — it’s always going to look a bit flawed.

I’m a grad student in statistics. When teaching statistics to undergrads, this kind of problem is common. There’s a lot of times when some result is based on more than one abstract principle, and it’s really hard to “get” it intuitively because without some practice, your naive intuition can’t see it from enough sides at once.

So what’s the kind of practice that gets you there? Concrete examples. In the case of voting theory, voting scenarios. I could certainly make a set of simple examples that shows how RAV works and why it’s a good thing that it works that way.

But there’s a danger with that. No voting system is perfect, and if you want to make some system look bad, it’s always possible to create a scenario that does so. So from the point of view of a non-expert, there’s always going to be competing experts with competing systems and scenarios and arguments. Each person has to decide for themselves how much of that to carefully decipher and analyze, and how much to tune out. For most people, they’ll start tuning out pretty quickly. So if they want to have an opinion, it’s going to be based on whom they trust and whom they don’t.

As an expert, how do I get people to trust me? I have some ways, but I’m not going to pretend I have it all figured out. I’m upfront about my biases. I try not to overstate my case, but I also try to avoid making overly-technical qualified statements that sound like fine print. I try to say things in a way that I think a non-expert could understand, but that would also make sense to an expert; I’m sure I often err on the side of the latter. I try to give people good heuristics for detecting fake experts; for instance, if somebody tells you their voting system is perfect, or they that this one precise scenario shows that some other voting system is bad, don’t trust them. Look for simple, realistic scenarios, along with some analysis of how they could vary and/or go wrong.

So… anyway, as yet another “nothing up my sleeves” gesture, I’m going to invite others to create simple scenarios to show off RAV. I imagine it would look at the first three winners in simple elections with <10 voters and candidates, starting with one where there are three works with separate, equal support, and progressing to cases where the supporters of two of the works explicitly ally against the third.

If nobody else makes scenarios, I'll do it later.

#157 ::: J Thomas ::: (view all by) ::: April 14, 2015, 02:14 PM:

@142 Cheradenine

"The current system is designed to represent as wide a group of fans as possible on the final ballot. The final ballot includes choices from 52% of the ballots that were submitted. Using the system you're suggesting, only 24% of the ballots would have any representation."

Thank you! I doubt I could make a better response even spending significant time on it.

The central point is that the nominations are not an election. People who think of them as an election will want the ones with the most votes to win. But they are intended to do something else.

I don't have it exactly straight what they're supposed to do, which leads me to propose doing without separate nominations. It used to be, we didn't have to define what it was for, we could just do it and get an adequate result. For a long time there weren't so many subgenres, they weren't big enough to credibly argue for more than one or two Hugo-level works in one, fans tended to read everything, etc. SF and fandom have gotten bigger and many fans have less catholic tastes, and this is the result.

We want diversity in Hugo nominations. We don't want the same people making all the nominations unless they suggest enough diversity. We don't want too many nominees from the same subgenre unless that's what the consensus is among the fans. Something like that.

We want a variety of nominations that represent SF. They should all be high quality, and should give a sense of what fandom is about. Yes!

Ideally, when two works that are too similar get nominated, one of the authors will decide that his own work is inferior and will decline so that we can get more diversity in the nominations. Of course we can't expect that, but that's the spirit we're looking for.

The nominations are not an election to pick the best. They are an attempt to choose a diverse group of great SF, so that each member of the group is worthy to represent SF and fandom.

It follows that the following election is not an election to choose the best SF novel etc of the year. The nominations are intended to provide a diverse group of high quality work, not to choose the five best. So the election among them does not imply that the winner is the best.

I started off not knowing what the nominations were for, and now I don't know what the election is for. I guess that's progress.

#158 ::: J Thomas ::: (view all by) ::: April 14, 2015, 02:29 PM:

@153 Keith "Kilo" Watt

Here's a proposed system:

Everyone is told that at the end of the day they will be able to nominate no more than one nominee per category (and possibly none, if there is insufficient support for all of your preferred candidates). The goal is to give you the best chance of having one of your preferred candidates make the ballot, and preferably, one that you ranked highest on your list.

I found your explanation a little bit unclear. I'll repeat it in my own words:

Each voter lists up to 5 choices in order of preference.

Then the votes are tallied by acceptance voting. Each of your 5 choices counts at this stage. A winner is found.

If the winner is your first choice, your ballot counts this time but is then discarded. If you listed it #2 through #5, your vote is not discarded. If none of your choices won, of course your vote is not discarded.

Do the same thing again, ignoring the known winner. A second winner is found. If this winner is your current top choice, your ballot is discarded after the vote. Otherwise not.

Repeat this until there are five winners.
So you get up to five votes. When your best remaining choice is selected, you're done.

The advantage I see for this approach is that you don't lose your vote if your fifth choice is selected early. Regardless what happens with the votes you care least about, your vote is not done until you have won the one you most care about, that can win.

A slate will almost certainly get one nominee on the ballot -- but only one.

If I understand you, no. If the slate voters arrange the same 5 names in random order, when the first one wins about a fifth of the ballots will be lost. Then if they win a second time, another fifth will be lost. If they lose the second time and win the third time, they lose the second fifth of their votes then and are down to 60%. They get as many nominations as their slowly-dwindling ballots happen to get.

If they were devious, they might try to have all of them vote in the same order, but have two voters who vote only for the last, two who vote for the last two, etc. Then their last choice wins and they lose two votes, their next-to-last choice wins and they lose two votes, etc. But it takes a lot of organization to make it go smooth.

I'm immediately curious what happens if there are two (or more) slates.

Say there are two slates, one with 20% of the votes and the other with 15%. Both with votes in random order. Assume that nothing else has more than 10%. Then the first slate wins one, and is cut down to 16%. It wins again and is cut down to 12%. The second slate wins one and is cut down to 12%. One of them wins and is cut down to 8% or 9%. The other wins one. If there was a sixth round then the first non-slate choice would win.

#159 ::: P J Evans ::: (view all by) ::: April 14, 2015, 03:11 PM:

Ideally, when two works that are too similar get nominated, one of the authors will decide that his own work is inferior and will decline so that we can get more diversity in the nominations.

Why do you think this should happen? (Unpack your assumptions about why you think people should be doing it your way: we can't actually read minds.)
Nominations are for works, unless otherwise specified. It's quite possible for someone to have read one for those two works but not the other, and they'd nominate the one they read.

#160 ::: J Thomas ::: (view all by) ::: April 14, 2015, 03:29 PM:

I got inspired by a misreadng of Keith "Kilo" Watt's proposal and made another.

I call it Reverse Single Acceptance Voting. It goes like this:

The purpose of the Hugo nominations is to present a variety of excellent works that are worthy to represent Worldcon, fandom, and SF. There is no guarantee that the best works will be nominated.

A nominator nominates as many titles as he wants, maybe up to 5, maybe more – I'm not interested in that limit, do it how you want. Acceptance voting, he needs to think each of them is worthy to represent Worldcon etc.

The voting program adds up all the votes for all the nominations. It then selects the nomination with the fifth most votes and adds it to the final nomination list. Then all ballots that include a vote for that title are discarded.

The program adds up all the votes for the remaining nominations. It selects the nomination with the fourth most votes and adds it to the final nomination list. All ballots that include a vote for that title are discarded.

Etc.

When explaining to voters, here's what it means. “You basicly get to vote once. Your vote will go to the one of your nominations that needs it the most to win.”

Side effects: The one that gets the most votes may not win. It starts out with the most votes, but then maybe it loses some when the fifth place winner is chosen, and maybe it loses more when the fourth place winner is chosen, and maybe it doesn't have the second-most when we pick the second place winner, or the most when we pick the first-place winner. This is OK. We have made diverse choices that represent us well. Each of them got pretty many votes that didn't overlap too much. When the most popular choice overlaps too much with other good ones, we're better off with them instead.

Slates: A big slate will get one win. A smaller slate may get one win. Slates might as well limit themselves to one candidate each because if they do two they can't predict how their vote will be split. If a slate nominates only one candidate, and has the most votes at the beginning, it will still have the most votes at the end when we select the first place. Meanwhile a smaller slate might be in fifth place when we look for the fifth place, fourth place when we look for the fourth place, etc. You just don't know.

Strategic voting: It does you no good to avoid voting for a popular work so your vote will count for a less-popular one. If your less-popular vote can win in fifth place or fourth place or whatever, it will. If it can't win, it won't. You might however consider whether you want the vote for a less-popular work to have the chance to interfere with your vote for the popular one. If it does win, your popular vote is lost. Only vote for more than one if you are willing to support whichever one needs the support more.

I am sure that this approach is better than the version which is otherwise just like it, but which chooses the most popular one first. For that, imagine the following extreme case:

Terry Pratchett's last posthumous book gets 80% of the vote. 20% of the ballots remain.

Something really good gets 40% of the 20% to get second place. 12% of the ballots remain.

Something else good gets 25% of the 12%, leaving 9%.

Something you probably wouldn't expect gets 44% of the 9%, leaving 5%.

That last 5% of the voters, the ones who don't like Terry Pratchett or any of the other stuff a lot of people thought was good, may be extremely diverse. If there are five votes for Brick Bronson the Bulletproof Commando and only four for Little Black Sambo at the Lesbian Orgy then the Hugos have dodged a bullet.

If we're going to dilute people's votes when they've already won something, then there's something to be said for STV which can minimize the votes lost, or maximize them, or anywhere in between. It is, however, complex and hard to explain. With my proposal your vote is completely gone after it wins something, but the number of votes lost at each step is minimized. We get five nominations based on uncorrelated votes, and each of them will tend to be a reasonably large number of votes.

It's simple to say what you're doing, and simple to do it. Not quite intuitive, unfortunately. If the numbers come out wrong you can get a bad result, but that's true of any system.

I expect it would be hard to game because your less-popular choices affect the chances of the popular ones instead of vice versa, and it's harder to guess just how unpopular your less-popular choices are.

#161 ::: Cheradenine ::: (view all by) ::: April 14, 2015, 03:30 PM:

Kilo @153: Ugh, I managed to miss the two times you said that all votes on the ballot counted and draw the conclusion that only first-place votes were counted. If I'm understanding it correctly this time (I swear I read it multiple times!), the problem is that there's no actual advantage to you of ranking your favorite work #1. It doesn't make that work any more likely to win than if you put it #2; it just creates a condition when your ballot stops counting. In a lot of cases, you'd be better off to rank the words you just scrawled on your napkin (i.e. something no one else will nominate) #1, and your favorites in spots #2-#5. (If everyone does this, it's equivalent to the current system with 4 nominations per person instead of 5.)

Sorry to misread you the first time! (Back to class now.)

#162 ::: Andrew M ::: (view all by) ::: April 14, 2015, 03:57 PM:

Teresa@96: You may well ask. I suppose Among Others belongs to the 'interesting and distinctive work' subgenre.

Cheradenine@80: Well, it may be that you wouldn't have much of a problem if votes were distributed over 8-10 works; but then in a thin year it might be fewer; Patrick noted near the start of this discussion that this was arguably a very strong year, and so votes might be more divided than they often are.

But I do think there is a general problem with carrying over assumptions from political voting. There, it's reasonable that voters will tend to clump around particular kinds of candidate, and so approval voting does a good job of counteracting the effects of clumping, and giving more voters representation. Some comments here seem to assume that the same is true in the Hugos, and the point is to resolve conflicts between fans of different kinds of work. But in fact voters themselves may well be seeking diversity, in which case a model derived from politics won't fit them. It's a rare voter who decides to vote for one conservative, one liberal, one socialist, one libertarian and one green, in the interests of diversity; it's not at all surprising that someone should vote for one work of epic-ish fantasy, one space opera, one piece of near-future SF, one this-worldly fantasy and one alternate history, or the like.

#163 ::: J Thomas ::: (view all by) ::: April 14, 2015, 04:05 PM:

"Ideally, when two works that are too similar get nominated, one of the authors will decide that his own work is inferior and will decline so that we can get more diversity in the nominations."

Why do you think this should happen? (Unpack your assumptions about why you think people should be doing it your way: we can't actually read minds.)

Nominations are for works, unless otherwise specified. It's quite possible for someone to have read one for those two works but not the other, and they'd nominate the one they read.

OK, I'll try to write in more detail.

Originally I thought of the nominations as being like an election. In an election, I want the ones with the most votes to win. When you throw away some votes but not others, when you say that some votes count as one vote each while other votes count as two votes or three votes, it sounds like corruption.

But the Hugo nominations are not an election with five winners. The final Hugo election is an election, and we want to everybody who votes to have read all the alternatives. We cannot possibly expect the nominators to have read all the alternatives. The various things that make the final election an election, are not present in the nominations. For example, it's perfectly OK in the nominations for the entry with the second-most votes and the third-most votes etc not to be chosen. In an election that would be vote-stealing. But the nominations are something else.

I looked at what people here said the nominations are. Every winner ought to be high-quality. And they should be diverse. They should get votes from diverse nominators -- we don't want the same small majority or small-but-focused bloc to nominate all the winners. And it would be rare that we would want them to all be space operas or all be sword-and-sorcery or all be hard as ten-point steel.

We aren't choosing them in isolation, the five individual best nominations we can find. We want a diverse group of excellent works. Things that people can use as a reading list. A canon, maybe.

I got the idea that the purpose of the nominations is to create a short list of excellent SF that represents the field. "This is what Worldcon is about." "This is what fandom is about." "This is what SF is about."

So we nominated Rite of Passage and Protector, but not the same year. Dune and Courtship Rite -- different years. A Civil Campaign and To Say Nothing of the Dog -- different years.

We want a diverse showcase of what SF has to offer, and if two excellent works are too similar in one year, probably one of them will lose out.

It is not a contest to pick the best five. If it was that, then the ones that got the most votes would be the ones that ought to win. If Planetbusters gets 300 votes you don't replace it with something that got 80 votes, just because you already accepted Starsmashers and Moonwhompers.

We are choosing a diverse collection of excellent work, and to do that we want work that is chosen by a diverse collection of nominators.

Are my assumptions clearer now? I could be wrong about what "we" want, but this is the explanation I see that seems to fit best so far.

#164 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 04:06 PM:

J Thomas: that would be crazy. Since it's basically impossible to predict what might end up in 5th place and suck up your vote, a strategic voter would be highly motivated to bullet vote. And then it reduces to plurality — the worst system.

If you want something like this, you use SDV, except transfer overvotes. So:

Vote for any number
while more than 5 candidates remain:
Divide each vote into n "shards" for each of the remaining candidates on that ballot
For i=1 to 4
redistribute any excess that the 1 through i'th top candidates have above the amount the (i+1)th candidate has
Eliminate the lowest candidate

It's SDV, but doesn't waste any "overvotes" on strong candidates. In other words, your ballot will tend to give more of its weight to the candidates that need it more to win.

Not hard to understand conceptually, but it would be utterly impossible to carry out without a computer. But it would be pretty easy to see if the computer were "cheating" somehow.

#165 ::: Derek Balsam ::: (view all by) ::: April 14, 2015, 04:06 PM:

Re: option 2

The minimum number of nominees is three, not five, though five is the usual number. See the WSFS constitution:

> 3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.

#166 ::: felice ::: (view all by) ::: April 14, 2015, 04:14 PM:

Cat @140: "One of the questions that Joshua brought up I haven't seen answered yet. Sometimes nominees decline the nomination. Is the ballot recalculated from there? If it is, could that lead to some second nominee that did make the ballot the first time failing to make the ballot the second time? Because that's the kind of thing that could produce hard feelings, I think."

If the calculation is rerun as Joshua suggests, it's only used to find the highest ranked nominee not in the current top 4; whether or not the current top 4 are still all in the recalculated top 5 is ignored. It's not a perfect solution - if the new winner would have pushed one or more of the others off the ballot, then it means the nominators who supported the works that would have been pushed are getting a disproportionately high number of their preferred works shortlisted - but it's better than throwing away the nominations of everyone who supported the declined work, and withdrawing a shortlisting because of recalculation really isn't an acceptable option.

If someone known to be likely to decline ends up on the ballot, ask them first before contacting any of the other nominees! Then if necessary the entire ballot can be recalculated, because none of the others on the original shortlist have been contacted yet.

#167 ::: J Thomas ::: (view all by) ::: April 14, 2015, 04:43 PM:

@164 Jameson Quinn

J Thomas: that would be crazy. Since it's basically impossible to predict what might end up in 5th place and suck up your vote, a strategic voter would be highly motivated to bullet vote. And then it reduces to plurality — the worst system.

Jameson Quinn, I don't see it that way, but that does not mean you must be wrong. On the other hand you probably haven't thought about it as much as I have. On the third hand, you have more experience with voting systems in general than I do. On the fourth hand, maybe we find out what it does better by testing.

Here's what I think: If there is only one work you care about winning, then of course you should only vote for that one. This is true in any voting system.

If you care equally about five different works, then it makes sense to vote for all five of them. You don't know which of them your vote will be cast for. But you know that it is the one which can win, that has the least support.

If it turns out that the one you think is a long shot actually has no hope, then you have lost nothing by voting for it. (Unless you are limited to five slots and there was another fifth choice you wanted.)

If it turns out that your vote helps to put your long-shot choice into fifth place, then your vote made a difference!

You made more difference bringing that into fifth place, than you would if your vote helped put something else into first place that would otherwise be in second place.

But you don't lose anything by voting for the popular one, if it is in fact popular, because it will have no effect unless your long shot loses. And it just might turn out that the one you thought was popular wasn't that popular, and your vote might be the one that lets it take fifth place.

So my strategy is "Vote for the ones you think ought to win." Your vote will benefit one of them (provided any of them make it). It will help the one that can win, that has the least number of votes.

#168 ::: Brad from Sunnyvale ::: (view all by) ::: April 14, 2015, 04:51 PM:

Hmm, is it a requirement the nominees be diverse? For that, you want a jury, I suspect.

When I say that strategy for some of these ballots is "easy" I mean it's not hard to figure out what your strategic ballot would be. You take the clear leader and leave it off your ballot (or perhaps 2.) That's also the easy strategy in AV.

The difference is, with AV, removing the leader from your ballot buys you very little. It buys you some support for your 6th place choice. That's such a minor win that most people will not be strategic -- which is good.

With some of the systems proposed here, including RAV, leaving the popular choice off your ballot buys you much more. It buys you improved support for all your other choices and in particular, it buys you improved support for what may be your top choices, not your 6th choice.

This is a really, really big difference. So big a difference that I think it's one of kind, not degree. Because if you are going strategic, you balance the cost (the risk that one of your choices which you think is a shoo-in doesn't make the ballot because you didn't support it) against the benefit.

With AV, the cost is low but the benefit is so low that I doubt many people even think of it. With RAV, you may argue the risk is slightly higher, but the benefit is so much more that many would consider it.

Now, I don't think for a moment that the majority of voters will go strategic. But I think a large enough fraction would that it compromises the accuracy of the nomination ballot.

All of this logic is complex, but the instructions to the voter who wants to go strategic are simple:

"Are there choices you are confident are shoo-ins? Leave them off your ballot, it will help your other top choices a lot, and even let you add some extras. Don't worry, only about (small percentage) of people do this, so you won't hurt the shoo-in."

Easy to do and understand but bad.

Some of the other methods are worse. In the approaches that give more weight to each candidate if you list fewer than the maximum, this is even stronger. "Are your top two choices a shoo-in and a more obscure work? Definitely nominate only the more obscure work, you will be doubling your support for it."

And as I've pointed out, all of the above approaches in option 1 2 and 3 still mean 1 or 2 slate candidates get on in each category. We've been shocked by a slate sweeping a category, but I think 2 out of 5 being slate choices is also quite bad, if not at the level of calamity.

#169 ::: albatross ::: (view all by) ::: April 14, 2015, 04:52 PM:

I think all these schemes (RAV, STV, AV) give you an ordered list, assuming you don't have any ties. So it seems like the natural way to handle declined nominations with a minimum of hard feelings and confusion is something like:

a. Run the vote-counting process to get the top 10 or more candidates.

b. Announce the top 5.

c. If someone declines, go to the next person on the list.

Is there some reason this isn't the right way to do things? The ballots were cast with the belief that this nominee was in the running, so it seems like the other ways to handle this situation mean that people cast ballots that now have to be treated in ways the voters never expected--like ignoring or removing some nominee from all the ballots and rerunning the election.

Of course a bigger problem is that when someone refuses a nomination, you've probably already called the other nominees. If the vote counting scheme could conceivably change who the other winners are based on one candidate being removed (so that taking Alice off the ballot knocks Bob from 2nd place to 7th place, and thus keeps him from getting a nomination), you've set up a situation where someone is going to feel, pretty understandably, like he got screwed out of a Hugo nomination. (And so will people who nominated him.)

#170 ::: felice ::: (view all by) ::: April 14, 2015, 05:09 PM:

albatross @169: "If someone declines, go to the next person on the list.
Is there some reason this isn't the right way to do things?"

Yes. It depends on the counting method (it's not a problem under the current system), but generally, it means discarding (at least part of) the nominations of everyone who supported the candidate who declined nomination - potentially a very large percentage of nominators. That means everyone who didn't like the declined nomination gets disproportionate influence. Rerunning the calculation allows their second choices to be counted instead.

#171 ::: albatross ::: (view all by) ::: April 14, 2015, 05:14 PM:

Assume that 2016 is the year of the Hugo Slates. Let's assume there are a half dozen big slates and several smaller ones, some using SP-like ideology, some trying to ensure that their favored category, which never seems to win a best novel, gets in this year, others trying to campaign for a particular writer they think has been unfairly overlooked.

My intuition here is that in this case, the slates will largely dominate the nomination almost no matter what we do, because bloc-voting gives you more of a voice, and who doesn't want more of a voice? It would be easy for the slates to capture a majority of the nominators--not any one slate, but one of the top half-dozen.

RAV and Ron Rivest's original scheme from the previous thread make it hard for a single slate to dominate. But suppose you have six slates, each with a substantial fraction of people, each voting for more-or-less disctinct things. The SPs get Manly Men In Space With Blasters on the ballot, the SJPs[1] get Ass Kicking Hermaphrodite Subverts the Patriarchy on the ballot, the SWVPs[2] get Sexy Sparkly Vampires Fall In Love with Misunderstood Tween on the ballot, the TCPs[3] get Atlanta Nights on the ballot, etc.

Each slate recommends only voting for their category of works--perhaps single nominations, perhaps simply "vote for our kind of work, especially this one, certainly not one of those evil selections by the wrong kinds of puppy."

We *still* don't get everyone voting their conscience. The slate voting still seems likely to dominate the individual votes, even with mechanisms to weaken any single slate. The SPs can't run the table anymore, but any slate that coordinates will be more likely to get their nominees onto the list of finalists.

[1] Social Justice Puppies
[2] Sexy Wereworf and Vampire Puppies
[3] Tor Conspiracy Puppies

#172 ::: Buddha Buck ::: (view all by) ::: April 14, 2015, 05:15 PM:

albatross @169:

I think treating the selection method as a way to get a fixed ordered list of all nominees and picking off the top until there are 5 accepted nominees is about the only way to go.

Worse than someone going "if the balloting had been rerun without Tom, who declined, I would have been nominated" is the unenviable phone call of "Um, yeah, you know how we said you were the #2 nominee? Well, the #5 guy dropped out, and re we re-ran the nomination software without him, and, um, you came in 6th..."

#173 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 05:21 PM:

@167: Sorry, as a person who's spent thousands of hours writing about, analyzing, testing, and programming voting systems, I'm pretty sure that your proposal doesn't work. I can think of several reasons why not. I don't think it's worth our time to go through the reasons one by one.

But I do like the basic idea. If you avoid the pathologies, it's actually quite similar to SDV in its philosophy. SDV is not my favorite proposal, but in light of your arguments for yours, it's actually growing on me. In my last message, I proposed a complex modification to SDV to make it more like what you want. I now have a simpler one:

Divvy up the votes, a la SDV.
Of the two works with the smallest amount of divvied votes, eliminate the one which is on fewer ballots.
Repeat until there are 6 candidates left
Take the top 5.


... This system is actually a tiny bit friendlier to slates than vanilla SDV; but on the other hand, there's less of an incentive to "leave off popular candidates" strategy. It's also, maybe, easier to explain philosophically than RAV. I still would choose RAV as my first choice, but the above is good. For a name, how about Fractional Approval with Popularity Eliminations (FAPE)?

#174 ::: nathanbp ::: (view all by) ::: April 14, 2015, 05:22 PM:

Brad from Sunnyvale @168: On diversity, you're right that selecting a diverse group of nominees (from different genres of SF&F) is not the goal of changing the voting system. Instead, it's to select a system that will take into account the votes of as diverse a group of voters as possible. I think in this discussion at times we've used the first as shorthand when we actually meant the second.

albatross @169: I see a couple choices for picking the next nominee when a nominee refuses a nomination.
1) Offer the nomination to the 6th (etc.) winner of the election.
2) Cross the nominee that refused off everyone's ballots and rerun the election.
2a) The new nominees are the top 5 returned (which may in extreme cases not match any of the old nominees)
2b) The new nominee is the top result from the new election that was not previously nominated

Under the current Hugo voting system, (1), (2a), and (2b) are all identical. Under most of the systems we've been discussing, they are not. For example, under (1), if the top candidate from a slate refused the nomination, that could (depending on the exact system in use), result in all of that slate's voters having no voice in who is nominated. No matter what you think about slates, I don't think that's really fair. (2a) seems the most fair to the voters to me, but could result in very awkward phone calls withdrawing someone's Hugo nomination. Under RAV, I believe (2a) and (2b) are identical, and if nominees are phoned in order would never result in a nomination being taken away.

#175 ::: J Thomas ::: (view all by) ::: April 14, 2015, 05:30 PM:

@W172 Buddha Buck

Worse than someone going "if the balloting had been rerun without Tom, who declined, I would have been nominated" is the unenviable phone call of "Um, yeah, you know how we said you were the #2 nominee? Well, the #5 guy dropped out, and re we re-ran the nomination software without him, and, um, you came in 6th..."

Don't call the people and say "You came in #2.". Call them and ask, "If you got onto the final ballot at the Hugo, would you accept?".

Far more work but better if you can handle it, call them earlier and say "You got a preliminary nomination for the Hugo, would you accept it if you got onto the final ballot?" If they say no then you can take them off the list and warn voters not to nominate them because they won't accept.

#176 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 05:52 PM:

me@173: actually, it's simpler to just say "repeat until there are 5 candidates left".

albatross@171: Say the competing slates had 30%, 20%, and 10%, leaving 40% non-slate voters. If there was a widely popular work that year that wasn't on any of the slates, it could still easily beat out the 10% slate. It is also possible that the non-slate voters would be frozen out.

But I think most fans would rather not vote a slate. So I suspect it would be more like 20%, 15%, and 8%. At that point, there's nothing really "slatey" about the 8% slate; they might as well be just similarly-minded voters, which has existed since forever.

#177 ::: felice ::: (view all by) ::: April 14, 2015, 05:56 PM:

Can we call publishing the unranked top 15 once halfway through the nomination period Option 5b?

Cheradenine @108: "Felice @105: I fear that in a dispersed election, giving the top 15 at the halfway point will be misleading if anything. Looking at past ballots, the difference between the vote counts for 5th place and 16th place, with (probably) less than half the vote in, would quite likely be less than ten votes. But if most late voters choose one of the top 15, it will skew the election towards those that got a few more early votes. This would increase the power of campaigning and slates."

I'd expect more people to nominate before the halfway point in order to get their first choices published in the top 15; it's a new deadline that would motivate people to nominate earlier than they do now. And if the difference is less than ten nominations, it's less than ten nominations; Hugo winners have been decided by less than 10 votes before. Most people who care about the Hugos don't want to support slates; and a Puppy-style slate that gets in the top 15 will just encourage people to nominate the other ten candidates. And that's before you take into account any other measures to reduce the impact of slates, such as the alternative counting methods most of this thread is discussing. Campaigning might help get a work in the top 15, but if it doesn't have genuine broad support, it's unlikely to make the final five, and increasing the total number of nominations contributing to the final ballot strongly counters the effects of slates and campaigns.

Cheradenine @115: "My issues with continuous voting basically boil down to: (1) implementation difficulties, (2) increased ability for bad actors to wreak havoc, and (3) that it basically embraces making the Hugo award a political process with campaigning, negotiation, arguments about who needs to compromise to push so-and-so out of the top 5, etc."

Most of that isn't an issue with Option 5b.

Emily H. @120: "Any way to increase coordination on the part of voters seems to encourage voting for what you haven't read but your friends say is good, or for what you only mildly liked but stands a decent chance of winning, and I think that's more broken than the status quo."

More broken than letting the Puppies sweep entire categories? You're right about the issues with increasing coordination, but it's also getting more people's opinions taken into account, which I think is a good thing even if they're not as informed as would be ideal. The problems you mention already exist to some extent in the final voting - not everyone reads/watches every nominee in every category they vote for. And knowing the midpoint top 15 does give people a month or so to investigate which candidates are deserving of their nominations - reading some that look of interest, checking the opinions of reviewers (not just friends).

#178 ::: J Thomas ::: (view all by) ::: April 14, 2015, 05:58 PM:

@171 albatross

Assume that 2016 is the year of the Hugo Slates. Let's assume there are a half dozen big slates and several smaller ones

As I understand it, the SPs this year were something like 15% of the vote. If we have six separate slates that are each 15% of the vote, that's 90%. They will dominate the elections because there will be nothing left.

#179 ::: felice ::: (view all by) ::: April 14, 2015, 06:12 PM:

J Thomas @126: "The interviewer starts his lead-in. "As we have all heard, for the majority of nominations for this prestigious award, the nominations with the most votes were thrown out and replaced by other nominations with a quarter -- or less than a quarter -- as many votes. For example, Jameson's Genociders, a story about patriotic soldiers winning a war on a desert world, was replaced by Mothra Faker, where a transsexual hobbit repeatedly tricks the sexists who try to get him pregnant. But the people responsible for this -- caught red-handed -- admit the facts but claim that what they did was the right thing to do and not corrupt at all. Here is Cheradenine to tell us their excuse."

How about "No, Jameson's Genociders didn't make the ballot, but all the same people also nominated Samson's Slaughterers, and that did get a place. Only 10% of the electorate nominated those works; once they got Samson's Slaughterers on the ballot, the other 90% got their turn, and Mothra Faker was the most widely supported amongst the 90%. You're not suggesting that 90% of voters should be ignored, are you?"

#180 ::: J Thomas ::: (view all by) ::: April 14, 2015, 06:16 PM:

@168 Brad from Sunnyvale

Hmm, is it a requirement the nominees be diverse? For that, you want a jury, I suspect.

I don't know what the requirements are. I've been trying to reverse-engineer the requirements from what people say they want and the voting systems they think encourage that.

They say they don't want slates. But voting systems can't tell what a slate is, they can only give less weight to ballots that are correlated. The more you and I agree about what to vote for, the less our votes count.

There are many fascinating ways to do that. If it's straight correlation that's the issue, then if you and I both vote A B C D E while Felice and Volts vote F G H I J both pairs get punished for agreeing. But if instead I vote A B C I J and Felice votes F G H D E then the four of us get punished less.

On the other hand, if A wins they could punish you and me for voting A, when the time comes to count up B C D and E.

I try to imagine this in a traditional context. "Brad, your Presidential candidate lost, so you get two votes in the Senatorial election. Meanwhile J, it appears your Senatorial candidate has lost so you get two votes for the congressman." People would scream bloody hell.

#181 ::: Brad from Sunnyvale ::: (view all by) ::: April 14, 2015, 07:08 PM:

I think the chaotic results when a candidate withdraws -- which could result in retracting a prior nomination -- are another sign of how complex these systems are, and thus how they fail the criteria I would set for them. (This is not fixed by saying you could just take the 6th place winner from the first run. The chaos is still present.)

It is possible to detect slates. With algorithms, but even better with algorithms and human oversight and jurisprudence. This is part of what leads me to option 4. Collusion should be banned. Any action (which is distinct from any pure speech like plain advocacy, which compromises the independence and honesty of the ballots is forbidden, and the committee should be able to correct it if they possibly can.

Yes, even though that could lead to mistakes and abuses. They would be fewer than the problems we are heading for.

#182 ::: J Thomas ::: (view all by) ::: April 14, 2015, 07:10 PM:

@173 Jameson Quinn

Sorry, as a person who's spent thousands of hours writing about, analyzing, testing, and programming voting systems, I'm pretty sure that your proposal doesn't work.

I notice that I do not respect your attempt to pull rank. Perhaps others might. When you give us useful ideas, and explanations that make sense to us, that's a valuable contribution.

But your field has changed a lot in the last X years and there's every reason to think it will change just as much in the next X years. You are trying to balance contradictory goals, and fashions change about which goals are worth balancing. You are trying to assist us when our statement of the problem we want to solve is basicly incoherent. So there are various reasons why your unsupported opinion may not fit my needs.

But I do like the basic idea. If you avoid the pathologies, it's actually quite similar to SDV in its philosophy. SDV is not my favorite proposal, but in light of your arguments for yours, it's actually growing on me.

Good! I'm glad our discussion is not useless to you.

Divvy up the votes, a la SDV.
Of the two works with the smallest amount of divvied votes, eliminate the one which is on fewer ballots.
Repeat until there are 6 candidates left
Take the top 5.

Interesting! If we did that without SDV it would be plain acceptance voting. In your hybrid system you first choose the two lowest when ballots that include some winners are devalued, and then you discard the one of the two lowest that has undevalued votes from the fewest people.

Earlier I presented a voting plan that would tend to stop slates. I didn't think to name it, and that looks important. I think I should call it Independent Voting.

The idea is, your ballot contains five nominees that will be handled with acceptance voting. But before the votes are counted, we adjust them according to how much they correlate with other ballots.

If your ballot has exactly the same five nominees as someone else's ballot, they both get a weight of zero.

If your ballot has four nominees that are the same and one different compared to someone else's ballot, they both get a weight of 0.2.

For three the same it's 0.4, for two the same 0.6, and for one the same 0.8. If your ballot is absolutely unique and nobody else voted for any of the same things, it gets a weight of 1.0.

We add up the votes with these weights and take the top five nominees.

Obviously, if you make up nominees that do not exist hoping to increase your weighting, your ballot should be disqualified.

With this system nobody in his right mind would attempt slate voting. That problem is solved. We reward independent voting, not going with crowds and cabals and slates.

I am not being sarcastic. Not entirely.

#183 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 08:42 PM:

@182: Notice that I "pulled rank" only after having cited a specific problem that you discounted.

Your voting system could fail to elect a candidate that was voted for by all but 1 voter. Obviously, that's an extreme, but less-extreme cases of an "obvious winner" who didn't get nominated would sour people on the system. Furthermore, all voters who prefer one specific work — anybody with a "favorite author" — will have a dominant strategy of voting for only that one. Anybody with wide-ranging tastes will contribute their judgment to helping only the least-popular of their choices. The way to get your work nominated would be to have fans who don't really read much. It's a bad system. Yes, there's the germ of a good idea there.

Experience with voting systems doesn't give you an infallible nose for a good system. SDV, for instance, is growing on me. But once in a while, you can smell a stink. I've learned that from having invented my share of stinker voting systems. It's no shame.

#184 ::: Keith "Kilo" Watt ::: (view all by) ::: April 14, 2015, 09:36 PM:

Jameson@183:

I'm sorry, but I don't see your counter-examples as being evidence that JT's proposal is bad at all. Maybe I just need more details.

Your first objection is that "a system could fail to elect a candidate that was voted for by all but one voter." Isn't that possible only if all the other ballots are identical? In other words, in the case that all the other ballots are slate votes? Isn't that precisely the effect we're looking for? Even though I'm not a specialist, I am a scientist -- my statistics background is enough for me to feel fairly certain that such a situation is impossible due solely to random chance.

Secondly, you say, "all voters who prefer one specific work — anybody with a "favorite author" — will have a dominant strategy of voting for only that one." Well... yes. Again, isn't that exactly the effect we want to achieve? If a fan likes a work, he/she should nominate it. If the fan likes -only- that work, that's within his/her right as well. It's still not a slate, and it's not preventing anyone else from getting their nominations on the ballot.

Finally, you conclude that it's a bad system because "you can smell a stink". Well, as others (and I) have pointed out, "Because I said so" isn't really going to fly with Hugo voters, so I really think we should cease using this type of argument for or against a system.

I'm beginning to suspect that an optimal voting system for a political election (which is what our experts have mostly studied, as I understand it) may not, in fact be the optimal system for the Hugos. This leads to an issue for this thread, however, that the moderators may need to rule on.

Teresa/Bruce: I think JT's proposal may have merit in that it is a minimal change to the existing Hugo system that -may- have no other effects (plus or minus) than to remove the influence of slates. It also may have other consequences that affect the nominations, but not in ways that we as Hugo voters care about. However, I'm not sure it fits as one of Bruce's options. Is it still kosher to discuss in this topic, or would you prefer that we stay focused on the options Bruce has presented? I can definitely see the need not to let the discussion get too far from the original intent, else we won't get anywhere, but I do think there is merit in the idea that is worth exploring. So before going further, I thought it best to check in with you -- I don't want to be guilty of transgression! :)

Thanks,
Kilo

#185 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 10:18 PM:

Consider the following election:

(num voters:works supported; ? stands for various works supported by fewer than 5 voters each)

4:?????
1:F????
5:FA???
15:EA???
20:DA???
25:CA???
30:BA???

The system elects E,D,C,B,F. A is not elected, despite being supported by an overwhelming 95% of the voters. None of the voters are voting more than 2 in common with any other, and of those 2, 1 is always A, which is just that good.

Yes, this is not realistic. But scale A down to a reasonable number — say, 50% — and increase F's support to 12% or so, and it becomes so.

As to strategy: you don't want all people who prefer P>Q>R>S>T to be voting for P only.

As to information: this system throws away ballots after they have one candidate elected. That's more radical a punishment than needed for proportionality. Aside from encouraging strategy, this also decreases both statistical validity and consensus validity.

As I said, the way to get your work nominated would be to have fans who don't read much, or whose taste is wildly different from the norm. If they haven't read as many of the other good works, or if they don't like what others like, they won't have their ballots discounted for supporting something else.

So that's 4 objections. This is not a good system. I understand that "pulling rank" made me look like an arrogant ass, and I'm sorry. I just don't know how else I can convey the difference between "interesting, but it may have a problem with X" and "no, that system is a non-starter". It's not that I don't like the idea of "use your vote where it's needed the most"; I found it worth rescuing, with two separate proposals that would accomplish the same thing. But starting a sequential procedure from the middle, that is not going to work.

#186 ::: J Thomas ::: (view all by) ::: April 14, 2015, 10:20 PM:

@183 Jameson Quinn

Notice that I "pulled rank" only after having cited a specific problem that you discounted.

You suggested a possible problem that I don't see is much of a problem. If you like five works enough to nominate them for the Hugo, it may turn out that some of them cannot win. And it may turn out that some of them are shoo-ins. If the one given your vote on is the one that barely makes it, rather than one that has a very good chance without you or one that cannot win, why would you complain about that?

Your voting system could fail to elect a candidate that was voted for by all but 1 voter. Obviously, that's an extreme, but less-extreme cases of an "obvious winner" who didn't get nominated would sour people on the system.

It depends on what people want. What I understand the purpose is supposed to be, is that we get a list of five excellent works that are supported by very different people, not all by the same clique. We want diversity.

If that is the goal, we can't do better than five works that each have 20% of the votes with no overlap. That is maximum diversity. If each of them had a unique 19% of the vote, but 1% of voters voted for all five of them, that would be less diversity. If 20% of the voters voted for all five of them, that would be a slate and a travesty.

So, say we have five works that each have a unique 20% of the vote, the best case, and then there is a sixth work that has 100% of the vote. Every ballot contains one of the magic five, and also this one. For diversity, this one is no better than the fifth one of the others.

Imagine this less-extreme case: We have selected four winners, each with about an uncorrelated 20% of the vote. At the last step, the 1st place, there are two candidates that both get all 20% of the remaining votes. One of them doesn't have anything else, and the other also has another 20% because it is on a slate with a previous winner. Does that make it better or worse?

Furthermore, all voters who prefer one specific work — anybody with a "favorite author" — will have a dominant strategy of voting for only that one.

That's true under almost any voting system, unless you forbid them to cast a lone vote. If they only care about this one work winning, how would it help them to vote for anything else? If they thought the rules favored a devious strategy of voting for something else too to help their one choice win, that would not make those rules look better to me.

Anybody with wide-ranging tastes will contribute their judgment to helping only the least-popular of their choices.

They don't necessarily know which one is the least popular. Their best strategy is to vote for the five they like best, if they like five enough to want them to win the Hugo. The one that is really the least popular -- but that can win with their support -- gets their vote. All their other choices either have no chance, or have a better chance than that one. I get the impression you don't understand how this voting system works. (It's possible I misunderstand it myself.)

Or maybe you think this person should have more than one vote? Maybe so. With some voting systems, you could be the deciding vote for all five winning nominations. If you made five different choices, all the outcomes would be different. With this system you can determine at most one. I'm not sure which is better.

The way to get your work nominated would be to have fans who don't really read much.

With any voting system, the surest way to get votes is to have a lot of readers who like your work and don't read anything else.

It's a bad system.

It may be, but you have not yet presented your case. So far you have one valid observation which is a feature and not a bug.

You are within your rights to consider it a bug, since the criteria we are working toward has so far been left totally unclear except by me.

And while I think I've been somewhat clear there hasn't been much feedback to say whether I have it right. The main feedback I've noticed is that I thought there was a sentiment that we shouldn't have too many Hugo nominees from the same sub-genre, and some people have said they think that's fine. (I like it better that way too, since there's no obvious way for the voting system to encourage people to nominate from multiple sub-genres.)

You have a second observation that each voter really only gets to vote for one nominee. Many other voting systems have that too, and I don't know whether it's a feature or a bug.

With this system, like many others, a slate can win one nomination if they have enough votes. Then they lose their votes and can't win any more nominations. It reduces the power of slates as a side effect of creating diversity among the winners, which is what various people have said they wanted.

You might possibly find a better system to achieve this goal. Or you might find a goal you think is better, that another system satisfies.

Given the goal that I think we're trying to meet, it will not improve diversity to contaminate some of the vote counts with fractional votes left over from other winners. But if you find some additional goal that justifies the reduced diversity, then you can do better at the combined goals.

#187 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 10:26 PM:

Oh, by the way, I agree completely that "an optimal voting system for a political election (which is what our experts have mostly studied, as I understand it) may not, in fact be the optimal system for the Hugos." I certainly wouldn't suggest my modified-SDV idea for a political election, for instance. Or the best system here may be something else unheard-of. But if a system is bad enough, it is bad for both politics and Hugos.

This is the last I'll comment on J Thomas's proposals. At least on that one point, I've obviously blown whatever goodwill I'd accumulated in this thread, so further argument from me isn't going to accomplish anything. I'd still be happy to answer questions but as I said when I first arrived, I'm not a Worldcon member so this is really y'all's decision not mine, and I fully respect that.

#188 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 10:40 PM:

Hmm... My "I won't comment again" obviously crossed with J Thomas's response. I am not going to go point-by-point, but his comment does raise another scenario:

25: A???
24: B???
23: C???
22: D???
21: E???
20: EF???

Your system will elect F,D,C,B,A. E will be punished because it was "on a slate" with F (that is, liked by an overlapping set of people), even though it had more non-slate than "slate" votes, and absolutely nobody voted F over E.

Unlike the previous scenario I gave, I find this one entirely plausible, more or less as-is.

#189 ::: heresiarch ::: (view all by) ::: April 14, 2015, 10:44 PM:

albatross @ 55: "I am curious how much non-slate voters agree in their nominations. My best guess is that people mostly tend to read clusters of related books--one person may read mostly MilSF, another may read mostly alternative history, still another may prefer urban fantasy."

I agree, though my suspicion is that people define their clusters by criteria far more idiosyncratic than recognizable sub-genres.

"This matters because anything that decreases the weights of slates is likely to also decrease the weight of these correlated nominations."

I tend to see it as a good thing. Partially because, as others have mentioned, proportionality in a nominees list is useful in the same way as proportionality in a committee or parliament is useful: if you are trying to bring multiple groups to a common decision, they all need to be (and feel) represented. I also think it is useful on a nominator-by-nominator scale: insofar as it encourages people to nominate widely across their reading rather than within their favored genre, that feels like a good thing. What do I think deserves a Hugo that isn't the sort of thing a lot of people will think of/is outside my normal reading cluster feel like good questions.

Cassy B. @ 114: "May I make a modest request? I've been seeing the term "bad guys" used in these threads, and I'd really rather we didn't use such loaded language."

I whole-heartedly second this. I think a fairly fundamental assumption of the current project is that Sad Puppies are only one iteration of the larger problem, which is that the Hugo nomination is prone to capture by dedicated and organized minorities. Such a minority could echo my politics down to the smallest detail, I could love each and every one of them like my own child, and if they didn't have more than 20% of the vote they shouldn't get more than one nominee out of five. Talking about this in terms of thwarting the puppies makes this project seem a) petty and vindictive and b) distracts from coming up with robust, long-term solutions.

#190 ::: J Thomas ::: (view all by) ::: April 14, 2015, 10:45 PM:

@185 Jameson Quinn

Sorry, I was writing and didn't see your response.

Consider the following election:

Good! I wrote about basicly the same thing, I think your example is written far more clearly.

The goal I understand so far is diversity. We want nominees to win that are supported by different nominators, not by the same nominator. (It's OK if some of the same nominators support more than one winning candidate, but it's important that they each have diverse support.)

What is the additional goal that the Hugo nomination system should achieve? I can easily imagine it should fit some other goal too. Can we clearly specify what that goal should be?

As to strategy: you don't want all people who prefer P>Q>R>S>T to be voting for P only.

If they think T is good enough to be a Hugo finalist, why not vote for T also? If enough of the people who vote for P also vote for T that it endangers P's chances, maybe T is more popular than it looks.

As to information: this system throws away ballots after they have one candidate elected. That's more radical a punishment than needed for proportionality. Aside from encouraging strategy, this also decreases both statistical validity and consensus validity.

If our main goal is to get nominees with ballots from diverse sources, we do not increase consensus validity by choosing winners because they correlate with other winners. If this is the goal, then any statistical validity or consensus validity we get by depending on the same ballots for multiple winners, is bogus.

I understand that "pulling rank" made me look like an arrogant ass, and I'm sorry.

It's a natural thing to do, faced with a system that looks awful and a proposer who refuses to understand. I don't blame you for trying it. I just don't want it to work. ;-)

It's not that I don't like the idea of "use your vote where it's needed the most"; I found it worth rescuing, with two separate proposals that would accomplish the same thing.

Thank you! Every tool in the toolbox is potentially useful.

But starting a sequential procedure from the middle, that is not going to work.

I don't know yet whether it can work. It's much simpler than other methods I've seen to do similar things. With luck, it will create five independent groups of ballots that are about the same size. The alternative of using this method starting at the beginning is *bad*. First you throw away the biggest group of votes, then you throw away the second biggest. By the time you get to the fifth group it's likely to be tiny.

#191 ::: nathanbp ::: (view all by) ::: April 14, 2015, 10:46 PM:

J Thomas, I don't think your Independence Voting system does what you think it does. At best I believe it performs about as well as any of 3a-d, and at worst it probably has other holes. I don't think it's reasonable to expect everyone to fill out all 5 slots with valid nominees or to disqualify them if they fail to do so, so any slate can continue on, just with reduced numbers of nominees. However, their power is not reduced as much as you might have thought. First, every ballot that matters shares at least one work with another ballot, so everyone's votes go down to being valued at 0.8. Most ballots likely share 2 works with another ballot (the 2nd-7th placed nominees for best novel totaled at 54.3% last year, so roughly speaking they were likely on at least half of the ballots that also had the 1st place nominee), bringing the average nominating power down further. A slate performing relatively as well as it did this year with 2 or 3 works listed per category would probably still have gotten all of their nominees in.

In my mind it also fails the fairness test, although this is somewhat subjective. Why should someone's vote for a popular work count less just because they nominated the same less popular works as someone else? In effect it's reducing the power of the supporters of works 7-15 to decide who is nominated. In the strategic voting sense, it disincentivizes putting more than 1 or 2 works on the ballot because if you happen to match up with someone else then your vote won't count for as much. At least with RAV or STV when your vote's value is reduced in later rounds it's because something you liked actually got nominated.

As to your other proposed system, I think the way I would describe it is arbitrary. Why pick the 5th, 4th, 3rd, 2nd, 1st candidates at each step instead of the 1st each time (same as Bruce's #1 in the old thread)? Or the 3rd each time? I think that any system which has a demonstrated example of not picking the most popular candidate is not a good choice and one that would also be difficult to describe as fair or likely to lead to outcomes that would be viewed as fair.

#192 ::: J Thomas ::: (view all by) ::: April 14, 2015, 11:05 PM:

Jameson Quinn

This is the last I'll comment on J Thomas's proposals. At least on that one point, I've obviously blown whatever goodwill I'd accumulated in this thread, so further argument from me isn't going to accomplish anything.

I welcome your recent substantive comments, and you are welcome to continue if you want to. If it's just too painful dealing with my failure to understand, I'll have to accept that.

25: A???
24: B???
23: C???
22: D???
21: E???
20: EF???

Count the votes.

41: E
25: A
24: B
23: C
22: D
20: F

The 5th place winner is D.
The 4th place winner is C.
The 3rd place winner is B.
The 2nd place winner is A.
The 1st place winner is E.

A similar version:

5: A???
24: B???
23: C???
22: D???
21: E???
20: F???
20: FA??

40: F
25: A
24: B
23: C
22: D
21: E

D takes 5th place
C takes 4th place
B takes 3rd place
A takes 2nd place

21: E
20: F

E takes 1st place.

F had 40 votes and did not place.

But we did get five nominees that had diverse votes.

#193 ::: Jameson Quinn ::: (view all by) ::: April 14, 2015, 11:22 PM:

@192: Sorry, you're right. Here's the scenario I meant:

25: A???
24: B???
23: C???
21: D???
20: E???
19: EF???
2: F???

#194 ::: Keith "Kilo" Watt ::: (view all by) ::: April 14, 2015, 11:23 PM:

(I haven't heard from the mods on my previous question as to whether it's okay to discuss an option that is not one of the original five, so please feel free to delete this if it's a problem.)

@185:

I would like to consider just the "slate weighting" that JT proposed. As a first cut, this is a minimalist change to the Hugo nominating rules. The original nominating rules stand with the addition that:

3.8.8 It is the intent of the Hugo nomination process that Worldcon members will only nominate works that they personally have read and enjoyed, and will not allow their nominations to be co-opted by any other individual or group.

3.8.9 Any nominating ballot which has five nominations, all five of which are duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.0.
Any nominating ballot which has four nominations, all four of which are duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.2.
Any nominating ballot which has three nominations, all three of which are duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.4.
Any nominating ballot which has two nominations, both of which are duplicated on any other ballot shall have each nomination on that ballot weighted by a factor of 0.6.
Any nominating ballot which has one nomination which alone is duplicated on any other ballot, shall have each nomination on that ballot weighted by a factor of 0.8.
The purpose of this rule is to discourage nominating that violates the spirit of 3.8.8


Okay, let's apply the system to your hypothetical ballots:

Consider the following election:

(num voters:works supported; ? stands for various works supported by fewer than 5 voters each)

4:?????
1:F????
5:FA???
15:EA???
20:DA???
25:CA???
30:BA???

Group 1's ballots are weighted at 1.0.
All other groups' ballots are weighted at 0.8.

Remember, the goal is to discourage -slates-, not a given work appearing on multiple ballots. There is only one duplicate for each of the other groups. Group 2 duplicates work F with Group 3. Groups 3, 4, 5, 6, and 7 all duplicate work A. Why doesn't Group 3 get penalized twice, then? That's not what we're proposing. We are only weighting copies of slates, not works that happen to be held in common by many voters -- the whole -point- is to find works that are held in common, yet not through artificial means.

So, tallying the votes:

Work A: 95 nominations * 0.8 = 76 nominations
Work B: 30 nominations * 0.8 = 24 nominations
Work C: 25 nominations * 0.8 = 20 nominations
Work D: 20 nominations * 0.8 = 16 nominations
Work E: 15 nominations * 0.8 = 12 nominations
Work F: 5 nominations * 0.8 + 1 nomination * 0.8 = 4.8 nominations

Final ballot is therefore ABCDE. I'm not sure I see how this is a problem.


As to strategy: you don't want all people who prefer P>Q>R>S>T to be voting for P only.

Why not?


As to information: this system throws away ballots after they have one candidate elected. That's more radical a punishment than needed for proportionality. Aside from encouraging strategy, this also decreases both statistical validity and consensus validity.

I think you may be misunderstanding the system. Even so, if it's simpler to understand, then a "more radical punishment than needed" is fine. In my opinion, it's a worthwhile trade-off.


As I said, the way to get your work nominated would be to have fans who don't read much, or whose taste is wildly different from the norm. If they haven't read as many of the other good works, or if they don't like what others like, they won't have their ballots discounted for supporting something else.

Sorry, but I guess I'm not understanding this at all. I'm probably just not parsing it correctly, however.


So that's 4 objections. This is not a good system.

It is not a system you -value-. Your objections are not hard and fast objective facts, they are value statements. We, as Hugo voters, may agree with all of your value statements. Or we may agree with none of them. That's a debate worth having, and one I'd still like to see the community undertake. I think that question is arguably more important than that of specific voting systems.


I just don't know how else I can convey the difference between "interesting, but it may have a problem with X" and "no, that system is a non-starter".

I think the problem is not that you're an "arrogant ass" (feel free to disemvowel as required!), but the fact that -you- have decided it's a non-starter. I don't believe anyone here has the power or right to declare that, save only Teresa and Bruce, as it's their playground, as I understand it.

Kilo

#195 ::: J Thomas ::: (view all by) ::: April 14, 2015, 11:23 PM:

@!191 nathanbp

J Thomas, I don't think your Independence Voting system does what you think it does.

Wow, I didn't think anyone would actually take it seriously. It fits some criteria for what we say we want, but....

If you vote for just one, that's worthless unless other people also vote for it. So your weight is .8 on one vote.

If you vote for two, also worthless unless other people also vote for them, your weight is .6 on two votes for a combined voting power of 1.2

If you vote for three with a weight of .4, you also have a power of 3*.4=1.2

If you vote for four with a weight of .2, you have a power of 4*.2=0.8

If you vote for five and anybody else votes for the same five your power is zero.

It destroys naive slates. Of course real slates would vote for three like anybody else who had three candidates they liked.

It was a joke but the idea of applying different weights to ballots depending on how similar other ballots are, and/or how many are similar, has a whole lot of possibilities provided we can get away from the idea that it is supposed to be a fair election.

Why should someone's vote for a popular work count less just because they nominated the same less popular works as someone else?

Almost everything we do with voting systems to reduce the power of slates, is doing this.

That isn't true for allowing 4 nominations per ballot and six winners. That means even a winning slate leaves a couple of slots for others.

But in general, we are looking for ways to make slate ballots count less because they nominated the same works as other slate ballots.

Why should someone's vote for a popular work count less just because they nominated the same less popular works as someone else?

No, I take it back. Often they make the *second* and later votes count less because they nominated the same popular work. They let the first one slip through and then clamp down on later votes, which in the order of things tend to be the less popular ones.

#196 ::: heresiarch ::: (view all by) ::: April 14, 2015, 11:25 PM:

Single Distributed Voting and Reweighted Approval Voting approach the same problem, ensuring a widely representative ballot, from two directions: SDV says if one of your choices has been eliminated then we shall increase the weight of your remaining choices; and RAV says if one of your choices has been selected then we shall decrease the weight of your remaining choices. They both have the same strategic problem: voters who do not want to "waste" their full-power vote on a less-favored selection have a reason to reduce their total number of nominations (bullet voting).

Both SDV and RAV have a maximum vote power: for SDV it is conveniently one, and for RAV it is either 1.79 (St.-Laguë; 1 + 1/3 + etc.) or 1.94 (D'Hondt; 1 + 1/2 + etc.). Hugo-Classic has a maximum vote power of 5. It's possible to exert less effect on the election than this, but no more. Under RAV, what voters are doing when they strategically vote for a single candidate is sacrificing the .79-.94 extra vote in favor of being absolutely certain where the remaining one will go. (SDV entails no sacrifice: this implies that SDV favors bullet voting more than RAV.)

However, the rate at which votes are altered as candidates are selected/eliminated is open to adjustment. What if SDV, rather than redistributing the entire vote according to the number of candidates remaining, was redistributed at a rate less than 100% and according to the number of candidates eliminated? So, for example, if one of your five nominees was eliminated, the other four would go from 1/5 each to 1/4.5, rather than 1/4? And if you only listed four to begin with, they would still only start with 1/4.5? What if RAV votes were reweighted at double St.-Laguë--2/3, 2/5, etc. as winners are removed, rather than 1/3 and 1/5? Under this kind of proposal, bullet voting under RAV would mean sacrificing 1.57/2.57ths of your maximum voting power, all on the hopes of one boost. A voter would have to be very committed to that single work to give up being able to help out four other works one and a half times as much.

#197 ::: nathanbp ::: (view all by) ::: April 14, 2015, 11:36 PM:

J Thomas@195:
Wow, I didn't think anyone would actually take it seriously.

I'll confess I got confused about which of your proposals Jameson was responding to and then typed all that up before realizing I was talking about the wrong one.

No, I take it back. Often they make the *second* and later votes count less because they nominated the same popular work. They let the first one slip through and then clamp down on later votes, which in the order of things tend to be the less popular ones.

I think this is important. Decreasing the value of the remainder of someone's ballot (to half, or even to zero) seems fair to me because they got something.

#198 ::: J Thomas ::: (view all by) ::: April 14, 2015, 11:47 PM:

25: A???
24: B???
23: C???
21: D???
20: E???
19: EF???
2: F???

39: E
25: A
24: B
23: C
21: D
21: F

A tie! I hadn't thought of that. As a first guess, I ought to say that in case of ties we should pick the one that has less overlap with other tentative winners, and also pick the one that has more disqualified ballots due to previously confirmed winners. I mostly don't want disqualified ballots to matter, but when it's to break a tie, why not?

But that gives the wrong answer for the example. Let's say that F wins fifth place.

25: A
24: B
23: C
21: D
20: E

The next winners are D, C, B, A.

E went from 1st place to fifth and never won. It had the most votes.

If we had chosen A B C D E instead of A B C D F it would have been better. E got more votes than F, and none of the shared ones were shared with other winners. Only with F.

So here is an example where RSAV creates a good selection of uncorrelated nominees, but not the best selection. I don't know whether it's possible to fix it in a way that that solves this and all other problems, or not.

#199 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 01:41 AM:

Regarding ties:

Is that a bad thing? We're just trying to get a list of nominations, not select a winner. It ultimately doesn't even matter how items on the final ballot are ranked. Is there problem with just saying:

3.8.10 In the event of a tie for the fifth nominated slot, all works that are part of the tie will added to the ballot.

So if we get a tie for fifth, we have six nominees for that category. I don't see that as a big deal...

Kilo

#200 ::: Brad from Sunnyvale ::: (view all by) ::: April 15, 2015, 01:47 AM:

This thread should be giving us some idea of how well the voters and business meetings would understand these systems. (Though to be fair, I think with effort, simpler explanations can be made.)

Once again, these systems aren't coming close to matching the non-strategic qualities of the simple system we used that was spoiled by the slates. Not only is there effectively zero reason to be strategic on the approval ballot, people are even openly non-strategic.

For example, I have often, in a Hugo nominating category, only listed two, three or even just one work. And I know I am not alone. I am "wasting" my "power" and entirely not caring. It's just that I don't follow that category super closely, but I know the people I am nominating were good enough by my standard that year.

Many of the above systems will give this "only care modestly" ballot greater weight, as though all ballots were written by people who have 20 things they want to nominate and have to carefully pick the best 5. Now the hidden message will be "nominate fewer, your choices will do better."

Yuck.

#201 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 02:03 AM:

A hasty P.S. to my #194:

Groups 3, 4, 5, 6, and 7 all duplicate work A.

... whoops, I've got the weighting wrong. Each of group 3, for example, is obviously a two-work "slate", so should be weighted at 0.6, not 0.8. I was only comparing them to other slates, forgetting that they themselves are a slate.

Let me try the tally again:

4:?????
1:F????
5:FA???
15:EA???
20:DA???
25:CA???
30:BA???

Group 1's ballots are all weighted at 1.0 (no duplicates on their nomination ballots).
Group 2's ballot is weighted at 0.8 because it duplicates one work from Group 3.
Group 3's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 4, 5, 6, and 7 (so 0.8 weighting), and a different work with Group 2. (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights.
Group 4's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 5, 6, and 7 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights.
Group 5's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 4, 6, and 7 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights
Group 6's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 4, 5, and 7 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights
Group 7's ballots duplicate two works among themselves (so 0.6 weighting), but they also duplicate one work with Groups 3, 4, 5, and 6 (so 0.8 weighting). Final weighting is 0.6, simply the lower of the two weights

Work A: 95 nominations * 0.6 = 57 nominations
Work B: 30 nominations * 0.6 = 18 nominations
Work C: 25 nominations * 0.6 = 15 nominations
Work D: 20 nominations * 0.6 = 12 nominations
Work E: 15 nominations * 0.6 = 9 nominations
Work F: 5 nominations * 0.6 + 1 nomination * 0.8 = 3.8 nominations

Winners are ABCDE as before, but it's important that I get the example weighting right.


If something like this could work without having to select "rounds of winners", it might be closest to what the system we have now while still discouraging slates.

But maybe not -- I'll try to come up with counter-examples where a slate can still control the nomination list under this change. I'd be interested in someone else doing the same, since I likely can't see the forest for the trees.

Kilo

#202 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 02:05 AM:

Brad@200:

Now the hidden message will be "nominate fewer, your choices will do better."

Yuck.

Why yuck? It seems to me that if someone nominates fewer choices, then they feel more strongly about those choices they do make. Isn't that the behavior we want to see?

Kilo

#203 ::: heresiarch ::: (view all by) ::: April 15, 2015, 02:38 AM:

Brad from Sunnyvale @ 200: "Once again, these systems aren't coming close to matching the non-strategic qualities of the simple system we used that was spoiled by the slates."

You write as though the previous system lacked strategy, when the very fact that it was spoiled by the formation of a slate shows that that is not true. The current nomination system always had a dominant strategy--it just went blissfully unexploited.

"For example, I have often, in a Hugo nominating category, only listed two, three or even just one work. And I know I am not alone. I am "wasting" my "power" and entirely not caring. "

The goal of discussing "power" and "strategy" is precisely to protect this kind of voting. Now that gaming the nomination system has happened, its bare existence is forcing Hugo voters to agonize over a series of unpleasant strategic decisions (form a slate, form anti-slates, boycott slates, give slated works a fair read) rather than doing what they ideally ought to be doing, which is, as abi eloquently describes, thinking deeply on what works have moved them enough to be worth nominating.

It is by thinking seriously about how various voting schemes can be gamed that the impact of those strategies can be minimized, leaving the best choice, both morally and also strategically, to simply vote for what you think deserves your vote. That's the point.

#204 ::: felice ::: (view all by) ::: April 15, 2015, 02:41 AM:

Keith "Kilo" Watt @202:"Why yuck? It seems to me that if someone nominates fewer choices, then they feel more strongly about those choices they do make. Isn't that the behavior we want to see?"

It's the exact opposite of what we need to counteract slates. The fewer nominations in total, the fewer needed to get on the ballot, and the fewer slate nominators needed to swamp it. If total nominations were low enough, a slate could win against any counting method by dividing their supporters into multiple sub-slates (potentially, five slates of one work in each category).

#205 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 03:01 AM:

felice@204:

I think I see what you're saying: Fewer nominations total means a slate needs to gather fewer nomination votes to run the nominations, correct? I can see that, though to a certain extent that's true even if with lots of nominations, if they are widely scattered among different works.

But is your objection truly the case for all systems? It seems to me that the slate-weighted system solves that problem. If I nominate a single work that I really love, the odds are my vote will count with a 0.8 weight. But a slate is likely to get a 0.0 weight regardless of what I do personally.

I do see that one could argue that the system encourages me to choose only one work to support. I don't think that's a bad thing, particularly since you still have the option of nominating other works if you want, though at the increased risk that your weighting may go down. But because the odds of randomly matching someone else's nomination ballot go way down the more works you nominate, the actual chance of your weighting going down also decreases. If, on the other hand, you have intentionally matched your nomination ballot to someone else for ask five slots, then the odds of you getting a weight of 0.0 are 100%. So, even in the case you suggest, the system seems to function okay, doesn't it?

Kilo

#206 ::: felice ::: (view all by) ::: April 15, 2015, 04:45 AM:

Keith "Kilo" Watt @205: "It seems to me that the slate-weighted system solves that problem."

I'm not quite clear on how the slate-weighted system works (what's the weighting on a nominating ballot with 5 works, of which 4 are duplicates and one unique?), but in any case there are ways for a slate to avoid a zero weighting (ie instruct supporters to nominate a random subset of the slate instead of the whole thing). The power of the slate is reduced, but the power of non-slate nominations is also reduced, both because there are fewer of them (more people only nominating one work), and non-collusion overlap in ballots (the distribution of nominations isn't random, there's some correlation based on tastes). I'm not convinced it would help against slates at all, and I think the side-effects are problematic.

#207 ::: J Thomas ::: (view all by) ::: April 15, 2015, 05:25 AM:

@201 Keith "Kilo" Watt

I intended that Independent model at most to focus discussion. It stops 5-item slates cold. A 4-item slate would suffer an 80% penalty -- 300 votes would amount to 60 votes. A 3-item slate would have a 60% penalty -- 300 votes would amount to 120 votes. A 2-item slate would have only a 40% penalty, for those two items 300 votes would amount to 240 votes which is not so bad. And if a bunch of people agreed to push just one item? They would have no more penalty than anybody else does who votes for a single item that might win.

Meanwhile, regular voters would suffer random effects. If you accidentally vote for the same five works as somebody else, both your votes are lost. Etc.

This is actually kind of representative of the voting systems we're looking at. We're looking for ways to punish people for voting alike, so that slates will have less effect.

So for example, if a ballot has power 5, each slate ballot can affect five nominations. If we reduce that to power 1 then a slate ballot can affect only one nomination. We've taken away 4/5 of its power. That doesn't affect regular votes nearly as much because it's far more likely that a regular vote would mostly be wasted anyway. If you vote for 5 nominations and 3 of them are things that cannot win, you only lose half your power. With RAV you lose hardly anything, while the slate still loses around 3/5.

These are palliatives. Our central problem is that most of our nominations are wasted. Slates win because currently they each have five (5) votes that they do not waste. We're looking for rules to make them waste most of their votes too.

If you look at the 2013 Hugo statistics,
http://www.thehugoawards.org/content/pdf/2013HugoStatistics.pdf starting page 19
for novel there were 1113 ballots, and there were 717 votes for nominees that won. At least 396 ballots had no winners at all.

Out of more than 5500 opportunities to vote, 717 of them were for winners. Another 785 were for the rest of the top 15. That's about a quarter of them, total. Three quarter of the votes were for nominees that -- in hindsight -- could not have won.

Slates do not have this problem unless they are small pathetic slates. They never vote for works that nobody else does. Because they have each other.

There's a limit to how much we can fix this by adjusting the voting procedure to take away votes from slates.

#208 ::: J Thomas ::: (view all by) ::: April 15, 2015, 07:13 AM:

I think that Felice's proposal goes some way toward fixing this.

Announce the top 15 nominations halfway through, and let people who have not voted for them vote for them if they want to, during the second half.

This is almost the same as a run-off vote. One difference is that your earlier ballot still counts. You don't lose your vote if you fail to vote again. To the extent that the vote is online, it is not much extra work for the administrators.

If you want a sense of everybody's honest feelings, after the election they can post the half-time voting results that show how many votes every nomination got. Or just the top 15 with their votes, if that's what you care about.

But the more people who choose to vote for nominees that can win, the less influence slates can have.

So, the first half of the nominations is about expressing your honest feelings. You have an incentive to vote in the first half to improve the chance that your nominees get into the top fifteen. As far as I'm concerned you can express your honest feelings in the second half too. Nominate new things that can't win. Vote for nominees that aren't in the top 15, it's like a write-in vote but it isn't completely impossible to make a difference.

The second half can be about winning. With acceptance voting, you might as well vote for all the nominees you like. If you think it over, and you find yourself thinking "I'd hate it if B won and A didn't", then probably you shouldn't vote for B. If you'd be satisfied with either of them winning, then vote for both.

I suggest we let nominators vote for as many nominees as they want, at least in the second half. So if you are particularly angry at some slates, you can vote for everything but those slates. You do nothing to decide who wins, except to vote against them. Or you might want to vote for only one nominee. Or none. Whatever you choose.

In 2013 the top novel got about 18% of the ballots (maybe 4% of the votes) and the 2nd got around 12%. A slate with about 140 votes could have gotten 4 nominations. The highest novella got 103 votes, the same slate would get everything.

Now, a quick sanity check. In the very best case, imagine that 1113 nominators each chose 5 out of the top 15. That's 5565 votes. About half in the top 5, around 2800. A slate would only have to be about 4 times as big to get the same result. In the best case. About 560 nominators on top of the 1100 that are already there.

Best Novella had 587 nominators instead of 1113. Whatever the rules, a slate half as big would hit it.

Small awards are inherently slateable. A slate that's big enough to affect a big award will roll right over a small award. Period. We can look for a way to get a lot more honest nominators, or get them to vote more for winners, or get rid of the award, or find some clever way to nerf slates, or all four. But whatever we do has to work *really well* or we have to write off the small awards.

Bottom line: Helping more nominators nominate winners is another palliative. In the best case it could increase the number of votes for serious competitors against slates, by about 4 times. If we also use a voting system that reduces voting power to 1 (which reduces slates by about 80%) that will reduce votes for winners by about 80%. The two approaches do not work together.

#209 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 07:27 AM:

@207: "There's a limit to how much we can fix this by adjusting the voting procedure to take away votes from slates."

Yes. Voting theorists study limits like this.

I can't find it right now, but there's a graph of proportional systems, with "average satisfaction"—how much the average voter likes the average candidate—on one axis, and "representativeness"—a measure of how evenly-distributed satisfaction is across voters—on the other. You can see a clear pareto front on that graph; that is, some systems do well on one dimension but poorly on the other, some do OK on both, and then there are the poorly-designed systems that do badly on both. But there is no magic "perfect" system that does perfectly on both.

You could make that kind of graph for any two competing goals that you could define and measure in a monte carlo simulation. For instance, the power of "slate" strategy versus the power of "leave off popular candidates" strategy. On this latter graph, the current system would have a nice low power for the latter, but a crazy high power for the former. The STV-like systems which don't transfer overvotes would err in the other direction. RAV, SDV, and FAPE would be on the pareto front, representing reasonable compromises, with RAV giving more weight to avoiding slate strategy, FAPE giving more weight to avoiding "leave off popular" strategy, and SDV somewhere in the middle.

(Remember, FAPE is the system where you divvy up votes, then successively eliminate whichever of the bottom two has support from fewer voters.)

I'd recommend choosing a system with the following characteristics:

1. Resolvable without NP-complete calculations. That rules out SAV and PAV.
2. Uses an approval-like ballot, as with the current system. That rules out standard STV.
5. Proportional. That rules out the current system.
4. On or near the two pareto fronts I discussed above. I suspect, but cannot currently cite or prove, that that rules out STV-without-ranking, which I suspect is dominated by SDV.

That leaves RAV, SDV, and FAPE. So: if you want to encourage people to vote for more candidates, without giving too much power to slates, choose FAPE; if you want to punish slates as much as possible, without causing collateral damage among groups with common tastes who don't organize slates, choose RAV-exponential or, even further in that direction, RAV-Saint-Lagüe; and if you want a compromise between those "reasonable extremes", choose SDV.

Note that if the goal is to "beat" slates, it's better to do it "naturally", by encouraging non-slate voters to vote broadly and thus find common ground, than to do it "artificially", by having the voting system deliberately deweight slate votes. This argues that, in terms of the tradeoff above, you'd rather err on the FAPE side of things.

On that basis, I'd say FAPE is the best choice. But my personal self-interest is in having you choose a previously-known system which would have wider application in other use cases, that is, RAV. Really, I'd be happy with any of the three.

It is likely that a system somebody here just invented is not going to be on the pareto front, unless they at least took care to make it proportional. You don't have to be a super-expert to invent a good voting system, but you do have to have some awareness of the pitfalls and be careful about things like proportionality.

#210 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:20 AM:

The other argument in favor of SDV and FAPE over RAV is that, based on this thread and reactions like GRRM's, "dividing up the votes" seems to be easier to explain and to seem fairer to people than "reweighting the votes".

For instance: above, people are talking about RAV in terms of the sum of your ballot's voting weight over the rounds. But actually, that's not really a meaningful concept. Consider:

10:A
10:B
10:C
Elect 3. Winners: ABC

All votes have equal weight. Now:

20:AB
10:C
Elect 3. Winners: ABC

The first two groups have added a second preference, but the result is unchanged. Would you say that the first two groups have 50% more power? I wouldn't. Mathematically, the things that matter are how many of the nominees you supported, and how much power your vote has to choose the next nominee, and the ratio between those numbers. Adding a ballot's power across rounds doesn't really make sense.

Consider the extreme case. You have RAV with D'Hondt weightings, and you're electing 100 nominees. 99% of voters choose slate A1-A100, and 1% choose B. The winners are A1-A99 and B. The sum of 1/n from 1 to 99 is 5.177. Who would say that the A voters have 5.177 times the voting power of the B voters here? That's just silly; you could say they have 99 times the power, or the same proportional power. but 5.177 corresponds to nothing.

But I understand that it feels right to add up the voting weights. And that's because RAV is pretty unintuitive. I think my "cookie grab" metaphor helps understand it, but it's still tough.

So, SDV and FAPE avoid that problem. It's easy to understand dividing up a vote; it's treating the vote as a fixed resource, and our intuition knows how to handle that.

I still think RAV is a great system. But basically I'm beginning to talk myself around to thinking that FAPE is better.

#211 ::: J Thomas ::: (view all by) ::: April 15, 2015, 08:30 AM:

@209 Jameson Quinn

I can't find it right now, but there's a graph of proportional systems, with "average satisfaction"—how much the average voter likes the average candidate—on one axis, and "representativeness"—a measure of how evenly-distributed satisfaction is across voters—on the other. You can see a clear pareto front on that graph; that is, some systems do well on one dimension but poorly on the other, some do OK on both, and then there are the poorly-designed systems that do badly on both. But there is no magic "perfect" system that does perfectly on both.

I expect that many of us would prefer a graph with "average satisfaction of non-slate nominators" on one axis, and "representativeness for non-slate nominators" on the other. The preference would be that the slate nominators quit in utter disgust that their tactic failed.

But we don't have a certain way to identify slate voters from the voting data. "A large number of identical ballots with nominees nobody else votes for" is a slate, but they can disguise that pretty easily.

We can define "probably a slate" as "a significant number of ballots that are too correlated" and look for ways to punish probable slates. I'm not sure whether that requires NP-complete calculations or not.

(NP-complete calculations are OK for elections that don't get too big. The problem is only that they don't scale well.)

#212 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:32 AM:

As for the idea of publicizing a "long list" of the 15 frontrunners a month or so before nominations close, and allowing voters to redo their nomination ballot on that basis if they wish: it's not a bad idea. It would help with the problem where too many voters supported zero winners because the votes are too dispersed. But it wouldn't solve the problem of slates, and could even potentially make it worse. Personally, I'd suggest making one change at a time; so right now, this wouldn't be the most important change.

#213 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:42 AM:

@211: Any system built to ignore slate voters, as in the graph you suggest, would have collateral damage in groups of voters who just innocently had similar tastes. In general, insofar as there is "good taste" and "bad taste", and voters with "good taste" are roughly identifying some objective quality measure in works while voters with "bad taste" are just randomly voting based on what the weather was like when they read the book, any such system would tend to decrease the power of the "good taste" voters. This is true even if there are several conflicting versions of "good taste"; the truly random "bad taste" voters would still have more than their share of power.

So, I strongly feel that a good voting system gives slate voters their proportional share. It should be no more than that; but also no less than that.

#214 ::: Joe in Australia ::: (view all by) ::: April 15, 2015, 08:42 AM:

We're looking for a gate-keeping system, not a voting one: the actual voting system is pretty robust against slates.

The existing nomination system is broken because it lets attackers exclude other works. All we need from the nomination system is a list that includes works that are worth voting on. Having extra works on the ballot is only a failure if it creates too much work for the scrutineers; a solution to this problem that creates more work anyway is no solution at all!

For those reasons I support Option 2 (change the number of winners of the nomination election). I see no reason to change the number of nominations a member can make or to change the ranking of nominees.

It's better to err on the side of inclusivity to ensure that we have enough "genuine" (non-slate) nominees. For this reason I think we should have overlapping mechanisms for determining the length of the list.

My preferred mechanisms are:
1) Ask the members - it's an additional vote, but a very quick one. Take the median response, because it is hard or impossible to manipulate;
2) Leave it up to the ConCom. If attackers control the ConCom we have worse problems;
3) Set it by a formula - I don't care much for this because any fixed formula will be a target for attackers.

I'd be very happy with the first two options (no fewer than five nominees, plus as many more as the members elect in a followup vote, plus as many more as the ConCom elect). But any of these should work well and should not be too burdensome..

#215 ::: Cassy B. ::: (view all by) ::: April 15, 2015, 09:25 AM:

I know this is a naive question.... but some of those systems above are talking about (if I understand correctly) comparing *every* ballot to *every other ballot* in order to determine weighting.

Is this feasible in terms of computational power? Are we talking about running a script on a laptop, or about hiring the time of a supercomputer? I honestly have no idea.

That said, as a non-theorist who has been struggling to keep up with the proposals, I have one other question; in the case of the 80/60/40/20/0 weighting, does one of the identical ballots get full weighting, or does (in the impossible situation of every ballot being perfectly identical) every iteration get thrown out? That is, in an election with 101 ballots, and 100 have five identical nominees, and 1 has one (different) nominee... do we end up with one item on the ballot, because every single instance of the 100 are thrown out? or does one ballot survive of the 100, and we have six items on the ballot due to the tie? (Yes, I know, impossible hypothetical, but I'm trying to understand how this works.)

On the gripping hand, if I'm understanding the 80/60/40/20/0 system at all correctly, I rather like it. It's simple and easy to explain. And since it compares only within categories, the fact that EVERYONE liked The Avengers won't matter for Best Short Story...

#216 ::: P J Evans ::: (view all by) ::: April 15, 2015, 09:47 AM:

214
A motion has been proposed to change the number of nominations that can be made from five to four. (I know there are other changes also being proposed, but my brain isn't retrieving.)

#217 ::: J Thomas ::: (view all by) ::: April 15, 2015, 09:54 AM:

@213 Jameson Quinn

We don't have to worry about increasing the share of voters who vote at random. They have essentially no power because there are so many random things to vote on that they cannot be in the top 15 except by rare accident.

What I call a "probable slate" would show up basicly by cluster analysis. A bunch of ballots that are very similar to each other, and not very similar to anything else. We could punish it more the bigger it was, which would reduce the problem of accidental clusters.

Punishing cliques that all like the same things, and rewarding diversity.

I am getting a better idea of the criteria -- probably we want to reward diversity and also to a lesser extent reward things that get a lot of votes whether they are diverse votes or not. More later, I have a minor crisis to attend to.

#218 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 09:58 AM:

@215: You are correct; in principle, SAV and PAV involve comparing every possible set of 5 possible winners to every other. In practice, there are ways to home in on the "reasonably good" sets. So the exact computation should be feasible with high probability, and even if it becomes prohibitive to find the optimal winner set, a nearly-optimal one would definitely be easy to find. Still, you might reasonably count that as a downside to those two systems. But I haven't seen anyone in this thread supporting those systems so I think they're pretty much dead.

The 80/60/40/20/0 idea does not have good theoretical properties. At all. It can "punish" anybody who just happens to agree with somebody else. Weighting votes at 0 is a very bad idea; for instance, it means that if you have some way of knowing how somebody else voted, you can cancel out their vote entirely by just casting an identical one. It's a real mess.

If it were up to me, we'd be focusing any voting system discussion ("option 3") on RAV, SDV, and FAPE. I'd support whichever of these is seen to have the best chance of passing in the business meeting. If additional information, proofs, or simulations would help that passage, I'd be willing to help make those.

Ideas on options 1, 2, 4, and 5 are also welcome of course.

#219 ::: J Thomas ::: (view all by) ::: April 15, 2015, 09:58 AM:

@212 Jameson Quinn

As for the idea of publicizing a "long list" of the 15 frontrunners a month or so before nominations close, and allowing voters to redo their nomination ballot on that basis if they wish: it's not a bad idea. It would help with the problem where too many voters supported zero winners because the votes are too dispersed. But it wouldn't solve the problem of slates, and could even potentially make it worse. Personally, I'd suggest making one change at a time; so right now, this wouldn't be the most important change.

I doubt the membership will have a lot of tolerance for many repeated changes. We need to change to one system that works, and we will make a second change if the first system demonstrably fails. We will think about how to change it more after an extended discussion about whose fault it was and how we will never listen to them again about anything.

Not that this is how it ought to be done, but it's par for the course, right?

#220 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 10:08 AM:

@217: If you think of it in terms of "cluster analysis", there are various clusters, each of which has a "strength" (number of voters) and a "cohesiveness" (aka precision or inverse variance). Statistical tools can try to detect clusters. But such tools are fundamentally subject to both false positives and false negatives; the latter especially so, because any actual organized slate will presumably try to avoid being detected. Because of that, it is my firm belief as both a statistician and a voting theorist that looking to punish slates by more than a proportional system would (for instance, by more than highly anti-slate standard proportional mechanisms such as FAPE, RAV with Saint-Lagüe reweighting, or STV with a Hare quota) will do more harm than good. Explicit late voters will dodge it, so you'll only end up punishing innocents.

#221 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 10:13 AM:

In particular: cluster analysis in statistics is not an easy thing, but it gets easier the more items there are in a cluster. In this case, that would mean that unless you used some really advanced math, you'd probably end up punishing "large groups of voters with naturally similar tastes" more than "nasssty sssslate voterses".

#222 ::: Mark Wonsil ::: (view all by) ::: April 15, 2015, 11:08 AM:

I posted this on Bruce's site as well but adding it here for those who didn't go there. The process of determining rules for elections is covered quite well in a book called, "Choosing in Groups: Analytical Politics Revisited" by Michael and Kevin Munger. You can find it on Amazon if you wish or listen to a podcast about the book here: http://www.econtalk.org/archives/2015/02/michael_munger_1.html One may have the "perfect" voting system but a powerful few can make elections look very "democratic". It's an interesting read and this discussion is fascinating in light of the book.

#223 ::: Brad from Sunnyvale ::: (view all by) ::: April 15, 2015, 11:55 AM:

Yuck because this is not supposed to be a war. It is not supposed to be an election, with power and influence and clever strategies. That's why we're in crisis, because people treated it that way.

It's supposed to really be a survey, to learn from fans what they thought the great works were of the year that should be contenders for the Hugo.

For that, we want their true opinions, their independent opinions. I'm OK if they rank them (Hugo voters are used to that) because that's a true opinion. But when it becomes better for you to not give your true opinion because you can do better for "your" candidates by lying, I say "yuck,"

You are not in a competition with other Hugo nominators to see who can get more of their choices onto the ballot. You're trying to help the awards figure out good nominees.

#224 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 11:57 AM:

Regarding "pulling rank", I have an analogy.

Say the Holy Paper Airplane has just crashed and burned, so the faithful organize a contest to make a replacement for it. One or two experts and one or two hundred amateurs enter the contest. Chances are still good that the winner will be one of the amateur planes; expertise is worth something, but building good paper airplanes isn't that hard, and some of the amateurs will be naturally talented.

But say one of the amateurs says "the Holy Plane crashed because it had a tendency to turn left. We should avoid that problem by making a normal paper airplane and then cutting off the right wing." The expert will reply, "That won't work," and she will be right.

Designing voting systems takes a light touch and a balance of conflicting principles. If you make something designed only to avoid the last disaster, you'll only get a different disaster. Not all good systems look like each other, but there are subsets of bad systems that are bad for the same reason, and it's possible to learn to recognize those.

#225 ::: nathanbp ::: (view all by) ::: April 15, 2015, 12:29 PM:

Brad from Sunnyvale @223: I'm pretty sure it's not possible to have a voting system without the possibility for clever strategies at edge cases. Even in the current system, big fans of a work on the edge of nomination are potentially costing that work the nomination if they list other works in the same group on their ballot. For example, last year Parasite was nominated for Best Novel by only 2 votes over The Shining Girls. It's not likely, but possible that 2 people may have listed both works but would have preferred The Shining Girls over Parasite.

All the systems without ranking assume that the people voting will be equally happy with whichever of the works they vote for being nominated, which is obviously not true.

#226 ::: kimiko ::: (view all by) ::: April 15, 2015, 12:30 PM:

214,
All we need from the nomination system is a list that includes works that are worth voting on. Having extra works on the ballot is only a failure if it creates too much work for the scrutineers; a solution to this problem that creates more work anyway is no solution at all!

Okay, what's the largest number of works we could have in a given category, that a reasonably able person could read and evaluate? More for short stories than novels, yes?

#227 ::: P J Evans ::: (view all by) ::: April 15, 2015, 12:37 PM:

225
Traditionally, we haven't had rankings on nominations because we're creating a list of the stuff that people like and think is good enough for a rocket. If we have to rank them at that stage, why bother with a final ballot?

Short: make a case for ranking at the nomination stage.
Because so far, I haven't heard one that doesn't come out sounding like 'we don't want anyone to get a lot of nominations, and we want as many things nominated as possible, because diversity is more important than what people actually like'.

#228 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 12:46 PM:

Brad@223:

It's supposed to really be a survey, to learn from fans what they thought the great works were of the year that should be contenders for the Hugo.

I agree with you -- strongly, in fact. The purest and simplest form of nomination, and one that directly achieves this goal, is to simply have everyone nominate one work. Everything makes ballot, we all vote, and we're done. I still think this is a good idea, the simplest idea, and immune to almost all of the problems we've been discussing. The objection seems to be that no one can possibly read the huge numbers of works that would be suggested, and I think that's probably true. I still claim it's open to debate as to whether that might be the lesser evil. What a great reading list that would be!

But we all seem to be holding conflicting aims at the same time:

1) We want a wide and diverse field of nominations so that every fan's opinions are equally valid and considered. Every fan is voting for works he or she personally enjoyed.

2) We want to constrict the potential nominations to a small subset so that people can sample works they didn't necessarily read themselves previously. Their personal choices may need to be sacrificed to the choices of larger groups of people that share similar choices.

3) Larger groups of people who share similar choices should not be able to force the personal choices of others to not be considered.

Do you see what I'm getting at?

Kilo

#229 ::: Brad from Sunnyvale ::: (view all by) ::: April 15, 2015, 12:48 PM:

#225 nathanbp -- while I would hope we didn't have so many "nominate everything by my favourite writer" sorts, I will presume we do.

Your case just asks for ranking. There is no strategy available to the SMcG fans there that anybody would risk applying. Nominating just one of your 2 favourites is foolish unless you are sure about the other, and that is not the case here. Not nominating other favourites can work, but if they are other favourites, why do you want to hurt them?

It's not that there are no strategies at all to Approval. It's just that they all fairly ineffective, with low gains for the risk they provide, and hard to be sure will work. That is about as good as you can get in a strategy free system.

I am sure there were 2 fans like you describe, but how would they have done their ballots differently without foreknowledge? Or even with the foreknowledge that "it's probably gong to be close and only one will make it," which is actually an unrealistic amount of foreknowledge with a secret ballot.

However, it does not take a lot of strategy to affect the close races. Even if only 1% of nominators do strategy, it is going to affect races on a fairly regular basis, and the more strategy there is to do, the more this will happen. The battle for 5th and 6th -- possibly a career changing event for some writers -- is often quite close.

#230 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 12:52 PM:

A P.S. to my 228:

As a variation of the simple nomination system (everyone nominates one work, everything makes the ballot), one interesting thing would be to report all nominations on the ballot, but also include how many people nominated that work, strictly as informational purposes. A person who does not have time to read the entire list (and who would?) could decide for themselves how many people need to recommend a work before he or she will consider reading it. And even then, if something with fewer nominations still catches his eye, there nothing stopping him from reading it too. The advantage of this simple system is that it puts the power to decide what to read and vote on in the hands of the individual Hugo voter. No one can stop them from reading and/or voting on anything they like.

Kilo

#231 ::: Steve Halter ::: (view all by) ::: April 15, 2015, 01:01 PM:

Keith "Kilo" Watt@230:I kind of like that. Here's everything* that was nominated with some information (or not?).

*culled for invalid nominations.

#232 ::: kimiko ::: (view all by) ::: April 15, 2015, 01:05 PM:

Keith "Kilo" Watt@230, Steve Halter, 231,
That sounds like a pretty attractive proposal to me too. I have some doubts about it getting adopted, but I really like it.

#233 ::: nathanbp ::: (view all by) ::: April 15, 2015, 01:10 PM:

P J Evans @227: I don't feel strongly that the nomination ballot should be ranked (my preference is for RAV, 3c-3). Having said that, as I see it the case for ranking the nomination ballot is if people have strong preferences within their list on which work they'd like to see nominated, instead of being equally happy with any of them being nominated.

Brad from Sunnyvale @230: The point I was perhaps failing to make was in response to your comment @223 about the proposed systems:
But when it becomes better for you to not give your true opinion because you can do better for "your" candidates by lying, I say "yuck,"
In all systems, with perfect foresight of all the other voters, there are going to be cases where it's better not to give your true opinion. In a good system, including I believe all of 3a-d above (and the current system), without perfect foresight, voting your true opinions should be the best you can do. So just because there are hypothetical examples that make you say "yuck" does not mean the system would produce those results in practice.

#234 ::: Joshua Kronengold ::: (view all by) ::: April 15, 2015, 01:39 PM:

I am fond of the no system nomination system myself (i.e. take N nominations from each person; report the results minus invalid nominees and let people vote for what they want), but there are several reasons it's entirely impractical:

1. The quality of the Hugos is directly predicated not just on high nomination quality, but also on people generally watching/reading everything in a category before voting. That's not possible for this system, and the difficulty of convergence would be a real hit. Also, the Hugo Packet (which really, really helps with this) would more or less be demolished by this scheme.

2. It's already a big job to eliminate invalid nominees only culling from the top. Doing the same but listing everyone would be a collosal job, and we simply don't have the manpower to do it reliably.

3. It's important to give nominees the chance to decline nominations. This system wouldn't allow for this, which is a big no.

4. Except in years like this one, being a Hugo nominee is a big deal; this would pretty much put paid to this idea.

So no. And also no.

Jameson: SDV seems viable as an alternative tiebreaker for SDV that is kinder to works that tie without a big slate -- while still being quite harsh (presuming that you use the original tiebreaker for otherwise unbreakable ties) for constructed ties due to coordination. Since FAPE is really a slight modifier to vanilla SDV for extraordinary situations, I suggest referring to the combination as SDV/FAPE.

I don't think overvotes as you described are a good idea with SDV -- adding that complexity throws away the biggest advanvage that SDV (or SDV/FAPE) has over other proportional schemes -- the ease of understanding by the average voter/WSFS member. I could see a similar scheme that assigned votes after elimination not evenly among your votes, but instead to the weakest thing you selected -- so after elimination your vote would be divided 1/5 / 1/5 / 1/5 2/5. But I don't think that gives enough advantage for the simplicity cost.

I -could- see an overvote scheme in RAV. There, you could compare the winner to the nearest contender. On a tie, you use straight weighing, but otherwise you adjust that weighing based on the proportional number of ovtervotes. I'm not sure what the right spillover scheme would be -- the simplest one would just be to multiply the weight by votes/nearest competitor, maxing out at 1. So if you vote for something that's twice as popular as anything else, you don't lose any votes at all (giving you no real incentive to avoid voting for the popular thing you really like; the more votes there are for it relative to anything else, the less your other votes will be affected.

This would argue going for a harsher rather than a kinder weight (powers of 2 is good here), since the overvote system will act as a counterweight.

#235 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 02:09 PM:

@234: the "FA" in FAPE is basically synonymous with the "SD" in SDV. So if you want a hybrid name, it would be SDV/PE. So mote it be.

The overvotes thing was my first attempt at getting the "don't waste your vote unless it's needed" idea inspired by J Thomas. SDV/PE was a later, simpler, better attempt to get the same effect. So forget about the overvote thing.

An overvote scheme in RAV would be basically STV with approval ballots, fractional assignment, and a flexible quota. For instance, you could run RAV once to find the 6th-place total as a rough quota, then run unranked-STV using that quota. Or you could keep doing it over and over again using the 6th place total as a new quota each time until it converged (which it would). I think this system would have good properties — comparable to SDV/FAPE — but it's really hard to explain.

#236 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 02:13 PM:

This last suggestion I made in @235 is a bit like a second-price auction, only it's a "proportional sixth-price auction". There may be a way to figure out the result using systems of linear equations rather than running until convergence. And while nice linear convergence would indeed be the norm, there could be edge cases where it stuttered between two candidates. If that happened, you could take whichever of them led to the lowest quota at the end.

#237 ::: Tammy Coxen ::: (view all by) ::: April 15, 2015, 02:45 PM:

Any proposal that "everything makes the ballot" is not paying attention to the number of works/people that are nominated. In 2014 the lowest numbers of unique entries in a category were 100 (semi-prozine) and 124 (editor-long form) and the highest numbers were 728 (short story) and 587 (novel). With the rest of the categories being scattered in between.

Which is also a question for folks speaking to computational requirements - has the huge number of entries been considered?

#238 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 02:52 PM:

Joshua @234: Ooh, I like the idea of RAV with overvote, though I don't know if it makes it too complicated for the masses. It reduces the apparent incentive to omit favorites from the ballot. (As I've mentioned, that isn't as good a strategy as it appears, but my conversation with Brad convinced me I'd need to offer each Hugo voter a 4-hour class on game and probability theory to convince them of that, and that's probably just a bit impractical.)

I'm interesting in simming SDV, RAV, and their tweaked versions some more, but I'm inclined to wait until we have the 1984 Hugo nominations data. Admittedly it may not be representative of the distribution of votes 30 years later, but it's probably closer than my random ballots are.

#239 ::: P J Evans ::: (view all by) ::: April 15, 2015, 02:53 PM:

237
I doubt it.

#240 ::: Tammy Coxen ::: (view all by) ::: April 15, 2015, 02:55 PM:

Sorry, I said "in 2014" above - that data is for the current Hugos, which recognize works from 2014, sorry I was unclear.

#241 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 03:01 PM:

Tammy @237: Thanks for that info! I knew they'd be large, but not exactly how large. That's one of the reasons I wrote off PAV, which with 728 candidates would run until the heat death of the universe. (I'm kidding. Mostly.)

RAV scales up very nicely because you only need to iterate 5 times, and the number of computations you need varies linearly with the number of candidates and voters.

SDV has more computational complexity, but with a fixed number of voters it should scale up as the square of the number of candidates, which isn't unreasonable. (Some implementations of SDV might begin by discarding all ballot entries appearing less than N times. In addition to reducing computations, that might reduce the pain of Hugo administrators, who otherwise might spend a lot of time figuring out if two similar entries appearing on one ballot each represented two votes for one thing or one vote each for two things.)

#242 ::: Tammy Coxen ::: (view all by) ::: April 15, 2015, 03:03 PM:

I was really excited about this thread and all the options. But as the conversation has evolved, I'm afraid that all this effort is for naught. The inability of any of the proportional voting systems to elegantly handle a declined nomination is going to make it dead in the water in the business meeting, I think. I think you'd be hard pressed to find a Hugo Administrator who'd support something that required re-running all the data whenever someone declined. What if a decline was received an hour before the scheduled announcement? What if, like now, works were deemed ineligible and removed after the ballot went public?

If past and future Hugo administrators at the business meeting say it's not workable, it will not pass, no matter how good of a system it is.

#243 ::: P J Evans ::: (view all by) ::: April 15, 2015, 03:04 PM:

241
Generally, the voters are smart enough to only list something on the nomination ballot once. Generally....

(I want to get hold of the data because the Hugo site only has the list of nominees, not the numbers.)

#244 ::: Buddha Buck ::: (view all by) ::: April 15, 2015, 03:04 PM:

Tammy @ 237:

The most computationally complex proposal that I saw was (at first analysis) O(n^5), meaning that the time requirements scaled at approximately the fifth power to the number of nominees. That was the class of proposals which attempted to find the set of 5 nominees which maximized "happiness". With 587, n^5 is in the trillions, so it's probably infeasible to use a naive approach, but there may be not-so-naive approaches which would work better.

REV, which seems to be one of the favored proposals, appears to be doable in O(n^2), or O(n^3) at the worst. For n=587, that's relatively trivial.

None of the methods appear to have extensive memory requirements.

#245 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 03:11 PM:

Joshua@234:

At least two of these are indications of what you value about the Hugo process, and I think they are good ones. Some people may also value them, some people may not. What I want to be clear about is that they are not, in and of themselves, objectively good or bad. It's painfully apparent that none of these systems are ideal. Trade-offs are going to have to be made. We have to decide what trade-offs we want to make, though, of course. Some may agree with you and value (for example) the Hugo voter packet over a simpler system. Some may not. It may even be that -most- people value the packet more, but I don't think that's been established (certainly I don't, but I have no way of establishing whether I'm in a majority or not).

To address your #1, I strongly disagree that the quality of the Hugos has anything to do with people watching/reading everything in the category. There's room for disagreement, of course. As mentioned above, the Hugo packet is an excellent freebie that I would be sad to see go, but if that were the cost to pay for a better system, I personally would pay it. It may be that not many people would.

Regarding #2 and #3, this is manpower issue, and is certainly not insurmountable. In fact, I'd put my money where my mouth is and volunteer. Alone. Yes, it's a big job, but I've handled bigger ones. So, I don't think we can automatically declare that this is a "no" based solely on these criteria.

And finally, regarding #4, this is again a value statement, and one I happen to agree with. Saying something is "Hugo nominated" used to mean something, and I agree it would not under this system. That would be a cost, and maybe more than most fans would want to pay. For me, even recognizing that it is a cost, it's one I'd gladly pay, but again, I may be in a minority.

I guess what I'm trying to point out is that every single system we are discussing, without exception, involves a cost. The real question is what costs are the fans willing to pay? If we want a system with no cost, then we should give up, and simply make no changes at all. There are those on this board and others who are proposing exactly that, and I can see why. We're just going to go in circles until we do.

===

Tammy@#237: This is a really excellent point. And if the number of nominations is too high, is it reasonable to set a threshold cut off as we do now? For example, if a work doesn't get at least, say 5% of the nominations then it is not put on the ballot? Everything else would be kept as originally proposed.

I recognize and accept the argument that "patching" a new system could be seen to be worse than using a known and well-studied system. I get that. But it is looking to me like none of the well-researched systems are turning out to be very popular. They were new once as well, so for me, I'm not intimidated by trying to find an alternative.

Just my thoughts,
Kilo

#246 ::: felice ::: (view all by) ::: April 15, 2015, 03:53 PM:

Jameson Quinn @ 209: "2. Uses an approval-like ballot, as with the current system. That rules out standard STV."

Nominations use a write-in ballot - that's inherently ranked, so STV (ranked with redistribution of overvote) would not need any change to the ballot at all. And I don't believe knowing that ranking mattered would discourage nominations. With STV, you can rank an obscure work over a popular one, to maximise the obscure work's chances without taking nominations away from the popular work if/when the obscure one is eliminated; that encourages nominating a full ballot rather than just your first choice, which is a good thing.


Jameson Quinn @212: "As for the idea of publicizing a "long list" of the 15 frontrunners a month or so before nominations close, and allowing voters to redo their nomination ballot on that basis if they wish: it's not a bad idea. It would help with the problem where too many voters supported zero winners because the votes are too dispersed. But it wouldn't solve the problem of slates, and could even potentially make it worse."

How could it make it worse, and why wouldn't it solve the problem? It could drastically reduce dispersal of nominations, pushing up the number of nominations received by non-slate works, and thereby increasing the number of supporters a slate needs to overwhelm legitimate voters. No reasonable system is going to help if the slate supporters become a majority, and I think this stands the best chance of keeping slate works off the final ballot entirely rather than merely reducing them to one or two places (and through the obviously fair method of other works getting more nominations - nothing for Fox News to complain about there :)


P J Evans @216: "A motion has been proposed to change the number of nominations that can be made from five to four."

Which is trivial for the Slate to overcome with a simple randomiser to assign Puppies with 4 of the 5 slate works each. The slate power is slightly reduced, but there are less non-slate nominations too, which will at least partially cancel out.


P J Evans @227: "Traditionally, we haven't had rankings on nominations because we're creating a list of the stuff that people like and think is good enough for a rocket. If we have to rank them at that stage, why bother with a final ballot?"

Because more people vote on the final ballot, and the nominators might change their minds after reading shortlisted works they were previously unfamiliar with. Ranking removes the incentive to leave good-but-not-your-first-choice works off your nomination ballot, and we want as many genuine nominations as possible to counter slates.

#247 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 03:57 PM:

Andrew M@162: "It's a rare voter who decides to vote for one conservative, one liberal, one socialist, one libertarian and one green, in the interests of diversity"
Actually, I do vote somewhat like that in primaries. I'm one of Those Annoying Libertarians, and California has a "top two open primary" system where you can vote for anybody in the primary and only the top two get into the general election, which usually means a Democrat and a Republican, but sometimes two D or two R. Since I don't want Republicans getting elected, and the Democrats will be getting my vote in the fall unless somebody gets lucky, I vote for minor parties and independents in the primary; last election that included a Green and an Occupy activist in the various districts I'm in, as well as a Libertarian and the potential second-spot Democrats. But it's a hopelessly gameable system that's designed to keep third parties from annoying Democrats and prevent the traditional open-primary problems of one big party voting for unelectable candidates from the other big party.

#248 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 03:59 PM:

Tammy @242: Assuming the computer implementation is reasonable (and most of these would need to be implemented via computer - RAV is almost reasonable, but anything on the complexity level of STV isn't), the new results could be generated in a minute. It would also be an easy programming problem to, when the nominee list is generated, generate the "6th choice" for each possible withdrawal. (In the absence of slates, these will often all be the same.) So re-running would only be necessary if multiple candidates declined in one category.

Alternatively, you could figure out the 6th, 7th, etc. place candidates when you run the original election. They could yield a somewhat skewed slate in terms of being proportional, but probably no worse than a non-proportional method would yield.

Kilo @245: If the ballot has a large number of nominees and most people haven't read most of them, voting blocs become much more powerful in the finals. Since many ballots will "run out" of votes, non-bloc works may never accumulate enough votes to defeat the bloc works.

You can certainly attempt to develop a new system from scratch, but keep in mind that this is a field of study that's been active since at least the 13th century (Ramon Llull's work) and heavily researched over the last 50 years in a wide variety of settings. The Academy Awards commissioned studies when they adopted STV. The paper I cited on SAV, PAV, and RAV notes that there's great interest in them in AI research, as a way of forming a consensus based on the different results from different modules or sensors.

If I want to bake my first loaf of bread, I can use a recipe, or I can throw random ingredients into a pan, see what happens, and then try again. If I iterate the second process enough, I might get bread eventually, but I'll produce a whole lot of inedible junk en route. If I met a baker and asked them to explain what was wrong with each set of ingredients, it would be understandable if eventually they said, "Why don't you try one of these reliable bread recipes?"

I also have to admit that I'm not quite getting the conclusion "none of the well-researched systems are turning out to be very popular" from what I see in this thread, which is "the same three people are loudly and repeatedly dissatisfied with the ones that have been presented." (And we by no means have exhausted "the well-researched systems." We've been discussing "the ones that people with knowledge of this area of study and those who participated in the last thread thought were most appropriate to this situation.")

I have tried to avoid appeals to authority, I'm happy to explain things, but when I take an hour to write up a summary of what I know from the existing research and the response is basically, "Well, what do those experts know?" there doesn't seem to be much point in continuing that conversation.

#249 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 03:59 PM:

Tammy@242: Interesting point. The simplest answer to that is to have the rule for replacing declined nominations be to use Joshua@88's #1: "When a nominee declines, offer the slot... to the 6th place candidate".

How crazy is it to do that? That is to say, how likely is that to give a different nominee than you would have gotten by re-running the process with the declined candidate removed from all ballots? Not particularly likely for any of the systems we're considering. But if you want to compare the systems, it's relatively more likely with RAV, in the middle with SDV, and relatively less likely with SDV-PE. That is, SDV-PE has the least "problem" with ties. That's because in SDV-PE, being on a lot of ballots can tend to protect a candidate from elimination, and that doesn't depend on which other candidates are in the election at all.

So, that's another argument in favor of SDV-PE, which is continuing to grow on me.

#250 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 04:10 PM:

nathanbp@174 - Huh? Your choices for what to do if somebody withdraws (1: Replace them with nominee#6,etc. vs. 2/2a/2b: Remove them and rerun the nominating-election) would produce radically different results.
If the slate voters uniformly picked their top five candidates, and dominated, then #6 is the first non-slate candidate, etc., as happened in Best Novel this year. But if you rerun the election, providing information about the top N choices and a week or two for "All of fandom [to be] plunged into war", then probably the slate loses.

If you allow this to happen more than once, and don't restrict new nominations, the Puppies can probably keep all of fandom in war for a month or more; even if you only allow one round of reelection, the Committee to RE-Elect the Puppies can still cause few weeks of chaos, cackle their Bwahahah, and invite the Reavers in to help.

#251 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 04:16 PM:

Minor followup to me@250 - For Best Pro Artist, since one of the Puppies was deemed ineligible, nominee#6 was chosen, and was also a puppy (rabid but not sad), so there's also a case for what happens when slates don't vote quite uniformly, or there's more than one slate, or whatever.

#252 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 04:34 PM:

felice @246: I think Jameson means "ballot" in a voting-theory sense of "what information are we requesting from voters," not "what does the ballot physically look like?"

There's some research suggesting that approval ballots lead to higher participation levels than ranked ballots, but (as far as I know - this is more on the poli sci side of voting theory, and I'm more on the math side) it isn't conclusive. We'd probably have to try it and see how participation rates are effected (though I suspect they'll be up for the next few years regardless of rule changes or their lack!).

I do note that STV is not quite as strategy-free as you suggest. You can safely rank an obscure work over a more popular one if you are *sure* the obscure work won't compete with the popular one for a nomination, but if there's any possibility they might compete, ranking the obscure work higher can cause both to fail to be nominated.

A danger in releasing votes and then allowing everyone to change the vote is that bad actors (and by this, I don't mean just "slate voters" but "people intending to gum up the process") get more opportunities to game the system. If they enter one set of votes en masse before the data release and then change them all, they can potentially affect voter behavior. (Or they can claim they changed them all and then not do so, etc.)

Bill Stewart @251: In this context, "rerun the election" means "take the existing ballots, remove the candidate who declined, and generate a set of winners again," not physically having another election.

#253 ::: nathanbp ::: (view all by) ::: April 15, 2015, 04:35 PM:

Bill Stewart @250: Sorry, by re-run the election I mean take the existing set of ballots, cross off the nominee who refused, and then recalculate the results under whatever voting system is being used. I did not mean to suggest that everyone be given a chance to vote again, that would indeed not work at all.

Jameson Quinn @249: Could you point me to which post(s) you described SDV-PE in? I seem to have missed it and don't see exactly which system you're referring to.

#254 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 05:05 PM:

@253: When I first proposed SDV-PE (single divisible vote with popularity eliminations), I called it FAPE. The rules are:

(Optionally: First, clean things up by eliminating all candidates with fewer than N votes, where N is, say, two thirds the number of votes for that the weakest nominee last year had. This shouldn't change the result, but makes things simpler, and involves less drudge-work cleaning up irrelevant data.)
Then:
While there are more than 5 candidates:
Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
Find the two candidates with the smallest pile of fractional votes.
Eliminate whichever of those two is on the fewest ballots.

Here are the benefits of this system, as I see them:
-This is easy to program, and in my opinion it's easier to intuitively explain or grasp than RAV.
- Of the proportional systems that have been proposed here, it does the least to encourage the "don't nominate popular works" strategy (both in a strict strategic sense, and, I'd guess, in an intuitive sense of "how likely is it that voters will try that strategy in real life"). This means that it would encourage the most "broad" voting of any of the proposals here, which would help fight slates by encouraging more nomination votes overall, and thus a higher threshold for slates to be winners.
-It is the most stable under declined nominations; while no proportional system can be perfectly stable in that case, with SDV-PE, eliminating a candidate before the election is (as) unlikely (as possible) to change the relative ranks of other candidates.

In other words, I think it's a good system for the Hugos. (I do not think it would be a particularly good system for a political election, by the way.)

#255 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 05:12 PM:

If you think of each ballot as a pizza of a unique flavor, then the elimination step of SDV-PE is: of the two candidates with the least pizza, eliminate the one with the fewest flavors.

#256 ::: felice ::: (view all by) ::: April 15, 2015, 05:17 PM:

Cheradenine @ 252: "felice @246: I think Jameson means "ballot" in a voting-theory sense of "what information are we requesting from voters," not "what does the ballot physically look like?""

The distinction is pretty ambiguous. We currently ask voters to write down up to five works in order (because there's no way to avoid ordering them), but tell them we won't take the order into account. Is using the order information instead of discarding it really a change in the information being requested?

"There's some research suggesting that approval ballots lead to higher participation levels than ranked ballots, but (as far as I know - this is more on the poli sci side of voting theory, and I'm more on the math side) it isn't conclusive."

That could well be the case in selecting from a shortlist; ticking your preferred option is certainly less work than putting numbers next to several options. Does it apply to write-in ballots, though? Where the information requested is the same in either case, and how many other people nominate the same candidates is much more important than the order?

"I do note that STV is not quite as strategy-free as you suggest. You can safely rank an obscure work over a more popular one if you are *sure* the obscure work won't compete with the popular one for a nomination, but if there's any possibility they might compete, ranking the obscure work higher can cause both to fail to be nominated."

Good point. Does #173 help with this? "Of the two works with the [fewest first place nominations], eliminate the one which is on fewer ballots" rather than just eliminating the one with the fewest first places nominations each round.

"A danger in releasing votes and then allowing everyone to change the vote is that bad actors (and by this, I don't mean just "slate voters" but "people intending to gum up the process") get more opportunities to game the system. If they enter one set of votes en masse before the data release and then change them all, they can potentially affect voter behavior. (Or they can claim they changed them all and then not do so, etc.)"

Bad actors are always going to be a minority, or there's no way to save the awards. If they get a slate in the top 15, it can be ignored and people can choose from the remaining 10. What else can bad actors do? Nominate works they think genuine voters will like, to improve those works' chances over other works we like? That hardly seems like much of a win for them. Claims of changed nominations can safely be ignored, whether they're true or not; we don't care what bad actors nominate, we just want to get more genuine nominations. If bad actors have sufficient power to take over the entire top 15, then they've done us a service by warning us, and a counter-slate can be prepared that ignores the official top 15 (counter-slates aren't good, but they're certainly the lesser evil in this case).

#257 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 05:28 PM:

Cheradenine@248:

Oh, whoops, profound apologies. I didn't mean to imply "Well, what do those experts know?" at all. I'm truly sorry I gave that impression. My observation was that we don't seem to be any closer to coming to a consensus about any of the option 3 systems (at least not that I see -- we seem to keep proposing and discarding or modifying several different systems). That is entirely different from saying none of them are popular, so I definitely shouldn't have phrased it that way. Your work in particular has taught me more about voting systems than I've learned in a lifetime so far, so I very much appreciate that. Please don't stop!

So let me ask plainly, is there a consensus forming? Can someone lay out the pluses and minuses (there have to be both in any system, of course -- not a bad thing)? It might be good to have a summary of where we are at this point, do you think?

If the ballot has a large number of nominees and most people haven't read most of them, voting blocs become much more powerful in the finals.

Do they? Okay, that doesn't seem intuitively obvious to me, but I think I can see it. Let's see if I can reason it out , and let me know where I've got a misconception. If there are a large number of nominations, then there might also be a large number of works that serve to dilute the concentration of votes in the finals, correct? And so, those votes are distributed so widely that a voting bloc (as opposed to a nomination bloc) can get more than the 50% of the total number of votes, right? I think I see that, but if that's not right, then my next statement isn't going to valid: Wouldn't that mean that they legitimately have more than half of the total electorate anyway? And so actually deserve to win? Under the voting rules, if they don't get quite 50%, then the last place candidate is eliminated and those votes are added to the second-place choice. Does this still allow them to run the election? If so, isn't that possible under the current system as well?


If I met a baker and asked them to explain what was wrong with each set of ingredients, it would be understandable if eventually they said, "Why don't you try one of these reliable bread recipes?"

A very adept analogy, and I do agree with what you're saying. I suppose my feeling was that no one was willing to even consider another system because it was not well-studied. I've never thought that was a path to innovation, but as you point out, the field is 800 years old, so maybe there is literally nothing new under the sun to really even consider. I think that's the point you're trying to make, right? In that case, if I'm wrong that no consensus is forming, then maybe we could move on to a way to "sell" the consensus candidate system.

I can definitely see why you wouldn't want to waste your time with an amateur system when there are perfectly good and well-researched systems out there. If any of those systems are viable (which, of course, means we have to define what viable means) for the Hugos, then I'll strongly support them. I'm not seeing that the experts here agree on that yet, but maybe I'm missing that part of the conversation. Could it be that we (and mostly I mean you folks, as the experienced ones) just haven't settled into a choice yet, and I'm not being patient enough? :)

I honestly don't mean to offend anyone or disregard their work. Yours in particular has been very helpful, and I've never seen you resort to saying "trust me!" I do appreciate that.

Again, apologies,
Kilo

#258 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 05:42 PM:

Jameson#254:
A quick comprehension check (mine, not yours ;-) )...


While there are more than 5 candidates:
1. Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
2. Find the two candidates with the smallest pile of fractional votes.
3. Eliminate whichever of those two is on the fewest ballots.

So the single divisible vote part is step 1, correct? And the popularity elimination part is steps 2-3?

I'm trying to figure out the interaction between the two parts as I go back through and try to follow your scenarios for each part.

Thanks,
Kilo

#259 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 06:08 PM:

Kilo @259: My apologies in turn; I think my frustration with a couple different phenomena and people focused inappropriately on your last comment, and you got a rant that only partly related to you. I'm sorry for that, and I appreciate your apology and your willingness to engage.

With a large number of nominees in an STV election, the danger is that ballots will be exhausted (that is, all choices on them will be eliminated) before the end of the election. So if the bloc only has 15% of the vote, and the order of elimination takes out any candidate that appears on more than 15% of the ballots, the bloc can end up with the win despite having a minority of votes. They were a minority of the electorate, but now the electorate has shrunk so much that they have a majority of the remainder.

(One way to deal with this would be to declare no winner if the winner doesn't have a majority, but I suspect that might mean that the short story award, for example, is never given again.)

It's very helpful to hear your view of the discussion, because my sense was that what we've done is narrow down the map of voting systems to a fairly small corner of territory (essentially, proportional systems that reduce slate power by either reducing the weight of ballots that have already gotten nominees, or increasing the weight of ballots that haven't gotten nominees yet or had candidates eliminated) and are now tinkering around with the details. But it makes perfect sense that if you don't know the landscape, there's no easy way to distinguish between "tinkering with the details" and starting over from scratch.

At this point, I think the most helpful step will be to get some actual ballot data so we can run some better-informed simulations and talk about how the results vary with each method, and which set of results is most desirable (as well as other factors like difficulty of implementing and explaining each system). I'm certainly happy to answer questions and talk over systems in the interim, but doing a full analysis of the pros and cons of a new system is pretty time-consuming. It may be more productive to discuss a set of goals and how we could modify an existing system to better achieve them than to start with a new set of rules and figure out their implications.

#260 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 06:09 PM:

That should, of course, be Kilo @257. Does the numbering sometimes change, or am I just being especially bad tracking numbers over the last few days? (Rather embarrassing in my line of work.)

#261 ::: P J Evans ::: (view all by) ::: April 15, 2015, 06:11 PM:

260
If the gnomes take out a duplicate post, it messes with the numbers. But they'll usually say something.

#262 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 06:18 PM:

Keith@258: Yes, that's correct.

#263 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 06:27 PM:

@257, @259: I agree with Cheradenine. Also, note that there are now several "custom made" voting systems under discussion. These may have slightly improved characteristics, but really I think that the (non-drive-by) experts in this thread generally agree that even if we had to just take one of the better "off-the-shelf" voting methods (say, RAV with Saint-Lägue weights), it would work pretty well in general, and in particular adequately resolve the slate crisis. So yeah, we're tinkering at this point.

#264 ::: Bill Stewart ::: (view all by) ::: April 15, 2015, 06:29 PM:

Cheradenine@252, nathanbp@253 - Thanks for the clarification. That makes a lot more sense (and yeah, with the current Hugo nominating system, that should produce the same results, while under at least some of the various voting systems that have been proposed, you might need to do a whole recalculation and get a different result.

#265 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 06:39 PM:

Cheradenine@259

It may be more productive to discuss a set of goals and how we could modify an existing system to better achieve them than to start with a new set of rules and figure out their implications.

I wholeheartedly agree with this. It's sort of the point I've been trying to make in some of the other threads in the discussion. Until we all agree on what we value in a system, it would be rather difficult to choose any system, particularly since my read on the discussion is that many of our goals are contradictory.

It seems to me no system is perfect and all will have costs/trade-offs. Is this a fair statement of voting theory? (My engineering background biases me to see all systems that way, of course, so I may have blinders there.)

Kilo

#266 ::: P J Evans ::: (view all by) ::: April 15, 2015, 06:46 PM:

265
Stuff I've read in the past on voting systems (Scientific American had at least one article on them) says that every system has at least one flaw.

#267 ::: nathanbp ::: (view all by) ::: April 15, 2015, 07:32 PM:

Kilo @265: There is actually a proof that no voting system can be perfect (under a defined set of conditions). See Wikipedia's article on Arrow's impossibility theorem.

#268 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 07:42 PM:

PJ and Nathan, thanks, I appreciate it.

A quick request:

I've been reading up on voting systems and the math behind it in hopes that I can communicate more easily in the discussion. Are there a few particular canonical references that I should be looking at? I'm an astrophysicist, college professor, and former NASA engineer, so consider me "intelligent but inexperienced," to use the famous quote. :) My statistics background is pretty good (I also have a grad degree in education research), but not at the specialist level. Any suggestions at an appropriate level would be appreciated!

Thanks in advance!
Kilo

#269 ::: J Thomas ::: (view all by) ::: April 15, 2015, 07:52 PM:

@248 Cheradenine

I have tried to avoid appeals to authority, I'm happy to explain things, but when I take an hour to write up a summary of what I know from the existing research and the response is basically, "Well, what do those experts know?" there doesn't seem to be much point in continuing that conversation.

I welcome your explanations. I notice that the Hugo goals are poorly defined but sound like they might be pretty unusual. Existing methods might or might not be a good fit to them. Of course, new methods might or might not be a good fit also. At least with well-studied methods you have a better idea what you're getting, whether or not it's what you want.

I want to look closely at Hugo goals and see how much of a consensus we can get about those.

#270 ::: J Thomas ::: (view all by) ::: April 15, 2015, 08:01 PM:

@254 Jameson Quinn

When I first proposed SDV-PE (single divisible vote with popularity eliminations), I called it FAPE. The rules are:

(Optionally: First, clean things up by eliminating all candidates with fewer than N votes, where N is, say, two thirds the number of votes for that the weakest nominee last year had.
This shouldn't change the result, but makes things simpler, and involves less drudge-work cleaning up irrelevant data.)
Then:
While there are more than 5 candidates:
Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
Find the two candidates with the smallest pile of fractional votes.
Eliminate whichever of those two is on the fewest ballots.

That's beautiful! It deserves careful simulation to look for flaws, but it *looks* great.


#271 ::: conrad6 ::: (view all by) ::: April 15, 2015, 08:07 PM:

Nn-sltns t vry rl prblm!

Frst prblm - thr hs bn scrt slt snc th 1990s (SMF/SJW/Shvr cnsprcy(1950s fr y jvnls)).

Scnd prblm - y hv pkd th slpng br t mny tms, nd thr s nw nn-scrt slt. Dl wth t - thr vstrs ctlly by bks (s lng s thy dn't hv th dprctd Hg/Nbl tx stmp).

Thrd prblm - why d y dny nclsvnss t fns? Wld y ccpt $40.00 pll tx t vt n lcl lctn? ntnl lctn?

Frth prblm - Why th ht fr n pn slt? nlk SMF ths ws LWYS t n th pn n wb st.

S th vtng lgrthm nd dt r nt th prblm. nlss sbrnd lk mny lctrnc vtng mchns.

Th prblm s nt th bllt r th mthd - t's tht y ddn't gt th rslt tht y wntd.

#272 ::: P J Evans ::: (view all by) ::: April 15, 2015, 08:18 PM:

271
Y'all own it, entirely, because y'all invented the first problem.

We are not responsible for your actions or your lack of money. (Many of us are spending our grocery money to have a say in the mess y'all created.)

#273 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 08:56 PM:

@268: I can't think of anything that's perfect for this situation. "Gaming the Vote" is a good read, but it doesn't spend too much time on the technical side, and focuses on single-winner systems. "Choosing in Groups: Analytical Politics Revisited" (as discussed above) isn't bad, but it's also a bit skewed to single-winner issues and spatial models (which are pretty wrong for a Hugo context). There's a UN publication on proportional representation systems, but it's for politics, and only really talks about systems used by existing political democracies. Hmm... looking through my library of papers, I'd say "Divisor-Based Biproportional Apportionment in Electoral Systems: A Real-Life Benchmark Study" is not a bad one as a technical intro to PR.

#274 ::: Kimiko ::: (view all by) ::: April 15, 2015, 09:09 PM:

269,
On Hugo goals:
picture a 2x2 grid:
Good & Known ........................ Not-so-good & known
Good & Unknown .....................Not-so-good & unknown

For the nomination round, we want to select items from the first column, and we are particularly interested in the Good & Unknown, since the Good & Known can take care of themselves. We don't want to over select from Good & Known, since we want the final vote to be about ranking, not the nomination.

Our current problem is how to prevent noise from drowning out the signal.

#275 ::: conrad6 ::: (view all by) ::: April 15, 2015, 09:17 PM:

@ P J ?? r y sppsd t b frm Wst Vrgn? lbm/ Wst L? Pls lrn t tlk.

y'll??? WTF tht mn? Y sm t b prtndr lk ----- 'll stp xcrtng y s wn't gt bnnd, bt y r rlly stpd.

#276 ::: P J Evans ::: (view all by) ::: April 15, 2015, 09:23 PM:

277
Try California and four years in Texas.
Also try reading before you make any more remarks about others, because y'all sound like a troll to me.

#277 ::: Jameson Quinn ::: (view all by) ::: April 15, 2015, 09:25 PM:

"Y'all" is a great word. Everybody understands it, and allows English to join most languages in distinguishing singular and plural in the second person.

@270: I'm glad you like it. Your proposal was the inspiration.

#278 ::: Lori Coulson ::: (view all by) ::: April 15, 2015, 09:33 PM:

conrad6 -- you are nekulturny, comrade.

And if there is anyone lacking in intelligence in that exchange, you'll see them in your mirror.

#279 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 09:33 PM:

I was born in the great thriving metropolis of Bells, Tennessee, population 635 and a couple of dogs. It was years before I knew there even was a second person pronoun other than y'all. :)

Kilo

#280 ::: P J Evans ::: (view all by) ::: April 15, 2015, 09:43 PM:

281
It's my understanding that the plural, in Texas anyway, is 'all y'all'. (I don't generally like their political views, but most of the people I met were fine, and I miss them.)

My grandfather was born in, or near, Plummer's Landing, KY, population almost enough to count.

#281 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 09:57 PM:

Back on topic: Jameson, I also love the SDV/PE proposal -- the simplicity of it is marvelous. As I mentioned, I've been working through the examples that have been given, and it's growing on me even more. That's in no small part to your effective and clear statement of how the system works, so hat's off to you for that.

I'm also interested in the simulations that Cheradenine is running; I had actually started coding something myself when I realized that had already been taken care of. It'd be educational to see how it performs in various other situations, particularly since that would give scenarios that could be put out to further explain the system (as well as look for problems, of course).

Also, thanks for the recommendation; I'll see if I can find that reference.

Regards,
Kilo

#282 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 09:59 PM:

felice @256:
"Is using the order information instead of discarding it really a change in the information being requested?"

This is not a super scientific answer, but based on the number of "Top 5"/"Top 10" Tumblr memes I see where people say "In no particular order, because I couldn't possibly rank them," I think it's not that numbering items is by itself more work, but that assigning meaningful rankings to a set of 5 items involves a higher cognitive load.

Whether this would have a significant effect on Hugo nomination rates is harder to say. I got the impression in the last thread that people thought it would be an issue, I guess?

Good point. Does #173 help with this? "Of the two works with the [fewest first place nominations], eliminate the one which is on fewer ballots" rather than just eliminating the one with the fewest first places nominations each round.

I think that if something similar is applied to STV, it does reduce though not eliminate the spoiler effect. I've worked more with IRV than STV, though, so my intuition for the one-winner case may not match the behavior with multiple candidates.

Bad actors are always going to be a minority, or there's no way to save the awards. If they get a slate in the top 15, it can be ignored and people can choose from the remaining 10. What else can bad actors do?

Having thought more about it, a single slate is probably best off not even to release the slate before the revelation of nomination counts. The more slate works on the ballot, the more likely other people's votes will clump on the non-slate might actually benefit most from the list of 15 being a bunch of varied, highly appealing works to split anti-slate votes.

If there are 3+ slates and they vote early, then there's a danger that all 15 works will be from slates.

With this suggestion, the early nominations sort of serve as a nonbinding poll of what people are planning to nominate, which could serve as a crowdsourced recommended reading list. I don't know how many people would realistically have time to read a bunch of works during the 2nd half of the nomination period, but they could, or could just let their ballots stand.

I do think this approach avoids many of the flaws of either open continuous voting or releasing nominations at the halfway point but not allowing changes. Ultimately how well it would work comes down to predicting how voters would respond, which is a guess. I'm not confident that by itself it would be a sufficient anti-slate tactic, but if it were the only reform we could get through, I'd at least be interested to see what happened.

#283 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 10:04 PM:

@274: Hmm..that article seems to be behind a paywall. I'll have to see if we get Management Science on campus.

#284 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 10:31 PM:

Kilo @268: I don't know of a great overview, alas; most of my reading has been piecemeal. I really like the overview of the "classic" mathematical approach to voting in Taylor and Pacelli's Mathematics and Politics, but it's a math book, so it spends very little time on questions of implementation, and it doesn't cover multiple-winner situations like the nomination round. It would give you an idea of how mathematicians have tended to evaluate voting systems, though. (It also has good coverage of other problems in the field like apportionment, fair division, and yes-no voting systems. The general rule of thumb is that there's almost always a list of desirable qualities and a proof that no system can have them all.)

@271: Ah, I long for those bygone days of yesteryear, where people who posted without reading the thread were here to beat the drum for their favorite voting system. Now we've gone from people with "voting system" Google alerts to people with "Sad Puppies" Google alerts. (For clarity, Jameson, I don't mean you, I mean the folks who showed up once to tell us why their chosen system was Superior In All Ways and then disappeared again.)

Kilo @283: If you have any programming knowledge at all, you can probably code something much better than what I'm doing, which for SDV is basically having Excel calculate the weights, then me manually entering which candidate gets eliminated, then having Excel recalculate the weights. I'm happy to share my ersatz data if you'd rather not generate your own ersatz data.

@265, 269, 276 RE Goals: I'm operating on the assumption - which is admittedly partly reverse engineered from the current final voting system - that the overall goal of the award is something like "recognize the SFF work in the category most acclaimed by WorldCon members." ("Most acclaimed" instead of "most popular" because a book that 60% put 1st will win over a book that 100% like, and I avoided "best" because, well, measurement difficulties.)

In that case, it seems like a reasonable goal for the nomination procedure would be, "Produce a set of works which includes the books likely to be most acclaimed by WorldCon members, but is not so large that WorldCon members cannot reasonably compare them." (Note that the set of books might also include books that are NOT likely to be most acclaimed by WorldCon members. Screening those out entirely would be really hard.)

My assumption (which could be wrong) is that we can approach closest to this goal with a procedure that best balances the following 3 ideas:
1) Select works which are popular (on more nomination forms).
2) Select a group of works that together represent as large a set of nominators as possible (to reduce the chance that the entire set of nominees is acclaimed by a minority of the voting group).
3) Don't select too large a group of works.

Of course, there's the word "best" again... I think we've seen proposals that would prioritize #1 and #2 by disregarding #3, and ones that would prioritize #2 and #3 by disregarding #1. (The current system prioritizes #1 and #3 by disregarding #2, which is how we got here.) SDV and RAV variants are basically attempts to balance #1 and #2.

Does this make sense? If so, what do people think the relative importance of these ideas is? Or are they barking up the wrong tree in achieving the goal of the Hugo nomination system? Or do I have THAT goal, or the one of the Hugos entirely, wrong?

Whew.

@254, 270: OK, you've twisted my arm, I'll go simulate SDV-PE now. :)

#285 ::: P J Evans ::: (view all by) ::: April 15, 2015, 10:39 PM:

286
Repeating your crap argument does not make it stronger.
You start from a false premise (that there's a conspiracy), and you won't get any answer that will be good. (Ever hear of GIGO? This is GIGO.)

#286 ::: felice ::: (view all by) ::: April 15, 2015, 10:57 PM:

Cheradenine @284: "This is not a super scientific answer, but based on the number of "Top 5"/"Top 10" Tumblr memes I see where people say "In no particular order, because I couldn't possibly rank them," I think it's not that numbering items is by itself more work, but that assigning meaningful rankings to a set of 5 items involves a higher cognitive load."

True, but "if you don't have strong preferences, just write them down in the order you think of them" is I think a pretty easy and reasonable way to avoid the cognitive load in this scenario. I'd expect people would feel less pressure to rank them accurately with a write-in ballot than assigning numbers to an existing list, because in the former case the ordering is secondary to the main goal of specifying which works you're nominating, while in the latter the ordering is the sole focus. Alternatively, put a "ranked in order of preference?" checkbox on the form, and assign random orders when the checkbox isn't ticked.

"Having thought more about it, a single slate is probably best off not even to release the slate before the revelation of nomination counts. The more slate works on the ballot, the more likely other people's votes will clump on the non-slate might actually benefit most from the list of 15 being a bunch of varied, highly appealing works to split anti-slate votes."

Yep; though hopefully they wouldn't benefit enough, since even if nominations are pretty evenly divided between 15 works, the total number of nominations for the top 5 should still be much higher than under the current system.

"If there are 3+ slates and they vote early, then there's a danger that all 15 works will be from slates."

That would require three times as many bad actors as we have now, which seems unlikely. And of course there's the question of how the top 15 are counted - if our preferred proportional option (whichever that ends up being) is used, then that makes it much harder for slates to dominate this stage.

"With this suggestion, the early nominations sort of serve as a nonbinding poll of what people are planning to nominate, which could serve as a crowdsourced recommended reading list. I don't know how many people would realistically have time to read a bunch of works during the 2nd half of the nomination period, but they could, or could just let their ballots stand.

In some cases, people will have read works in the top 15 but not originally nominated them; if they agree that those works are good enough to be on the final ballot, and their original choices aren't in the running, it would be reasonable for them to change their nominations accordingly. Other people will not have nominated at all previously, but see something they like on the top 15, and nominate it. Others might read one or two of the most appealing works from the top 15 and nominate them if they like them. The nomination stage is about "this work is good enough to be in the running for an award", not "this work was better than anything else published last year", because nobody can possibly know if the latter is the case. Reading the entire top 15 isn't necessary.

"I do think this approach avoids many of the flaws of either open continuous voting or releasing nominations at the halfway point but not allowing changes. Ultimately how well it would work comes down to predicting how voters would respond, which is a guess. I'm not confident that by itself it would be a sufficient anti-slate tactic, but if it were the only reform we could get through, I'd at least be interested to see what happened."

One key point - as far as I can see, it's not against the current rules, so it could in theory be done next year. Would it be possible for the Business Meeting to vote to make Option 5b an explicit requirement for future awards, and recommend that MidAmeriCon II voluntarily carry it out before the change is ratified? As well as hopefully protecting next year's awards from Puppies, this would give the system a trial run before ratification, allowing for a more informed vote at next year's Business Meeting.

#287 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 10:59 PM:

On the two data sets I ran with SDV earlier, I got the same nominees with SDV-PE. However, in the situation where all the people who voted for #6 voted only for #6, it was a lot closer. In the situation with 150 slate ballots, the eliminated slate candidates stuck around just a bit longer.

Overall, I think SDV-PE reduces the incentive to vote for only one candidate that's present in SDV. This helped the slate a little bit since they were all voting for 5 candidates in this model, but the effect wasn't pronounced. I'd tentatively say that makes SDV-PE a bit more strategy-proof than SDV. I'd want to run a lot more sims, but first I need to make my spreadsheet more sophisticated so I'm not repeatedly searching a column of numbers for the two smallest, then looking to another column to find which corresponding number is smaller...

#288 ::: David Dyer-Bennet ::: (view all by) ::: April 15, 2015, 10:59 PM:

That's not an argument; it's simply contradiction.

#289 ::: P J Evans ::: (view all by) ::: April 15, 2015, 11:01 PM:

289
It should be possible to run both methods in parallel, to get good comparisons, assuming software is available for it.

#290 ::: Keith "Kilo" Watt ::: (view all by) ::: April 15, 2015, 11:03 PM:

Cheradenine@287:

If you have any programming knowledge at all, you can probably code something much better than what I'm doing, which for SDV is basically having Excel calculate the weights, then me manually entering which candidate gets eliminated, then having Excel recalculate the weights. I'm happy to share my ersatz data if you'd rather not generate your own ersatz data.

That would be outstanding, particularly the dataset. Having the Excel sheet would be helpful too, if only so that I can check my implementation. It's exam week for my students, so I'll be grading all weekend, but this would make an interesting break as I have time.

I've been reading "Voting Systems" by Paul E. Johnson, and found it to be an interesting introduction, but without any referents, I've got no idea if it's a good source or not. I did recognize immediately that there were papers put out by some organizations that had an obvious axe to grind.


Regarding goals:

I like your first cut. I might add a #2a to be something like, "No minority should be able to absolutely prevent any other set of nominators from having their acclaimed works considered for nomination." Which is essentially what you're saying in #2, I realize, but there is a slightly different focus. I would contend that a nominating ballot that was not diverse would be okay if no one tried to propose anything different. Not likely to happen, I realize, but the possibility should exist. I'm not for forcing diversity if there is none, if that ever could possibly become the case. For me personally, my problem is not so much that the slates got their works on the ballot, it's that their method prevented other works from even having the opportunity to be on the ballot. Does that make sense?

Kilo

#291 ::: nathanbp ::: (view all by) ::: April 15, 2015, 11:03 PM:

My preference is still for RAV over SDV or SDV-PE because I think RAV does a better job of reducing the power of slates.

For example, consider this case:
A - 50 Votes
B - 39 Votes
C - 35 Votes
D - 30 Votes
E - 25 Votes
QRSTU - 160 Votes (the slate)

In SDV the winners are A with 50 votes, then QRST with 40 votes each. In RAV, the winners are Q with 160 votes, R (80), A (50), S (40), B (39). Effectively, SDV does the work of figuring out the optimum size for the slate for free instead of making the organizers of the slate guess ahead of time.

I don't think my example is that different from potential situations in the less popular categories and more spread out categories.

#292 ::: Martin Schafer ::: (view all by) ::: April 15, 2015, 11:32 PM:

In trying to come up with a mission statement for the nominating process, I think about a universe of reactions that the group of people who care about the Hugos might have.

"Those are the best works in the genre"

"Those are all worthy works"

"Some of the things I like best are on the list"

"Some of the things that I like are on the list"

"Nothing on the list embarrasses me"

Given the imperfect world we live in I think having as many people as possible react with 3 or at least 4 is the most achievable goal that will keep the Hugo's going on much as they have.

My personal preference is for the RAV 3c with exponential as I think it privileges those reactions.

Something of a side note: I get really uncomfortable when people use sub genre as a stand in for faction in the nomination selection process. I think the taste clusters are much more complicated than that. Nearly everyone reads more than one sub genre and will have only a few if any that they can't appreciate an exceptional work in. The things that make a story beyond the pale for me is a strong reaction of "people don't work like that" or "the world doesn't work like that."

#293 ::: Cheradenine ::: (view all by) ::: April 15, 2015, 11:37 PM:

Kilo @293: Let me know where I should send them. I might take a day to make my Excel coding somewhat less embarrassing.

I skimmed Johnson's piece and it looks like a decent introduction to the original main thrust of voting theory. It covers more or less the same systems as are covered in a liberal arts math class, but covered in more depth (particularly the different Condorcet methods) and with a more rigorous mathematical approach. He doesn't touch on approval methods, strategic voting, or fairness criteria that aren't the four covered by Arrow's, and he doesn't have a lot of detail on STV. Occasionally his terminology seems a little nonstandard (I've never heard it called Arrow's Possibility Theorem instead of his Impossibility Theorem), but I don't think in a way that would cause you major trouble. I do think the Taylor & Pacelli book gives a clearer presentation, but it's also not available on the Internet.

This is all based on quickly flipping through it, so it's possible I missed a giant mistake, but it looked OK.

#294 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 15, 2015, 11:45 PM:

I hate to put the brakes on such a good conversation, but I've got some acts of corporeal charity to take care of before I crash for the night. I'll be shutting down comments in 5-10 minutes. See you all tomorrow --

#295 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 09:20 AM:

@294: Yes, you've hit on the key difference between RAV and SDV. However, I think the issue is more subtle than what you've said.

In voting theory, when you're comparing systems XYZ and PDQ, and superficially XYZ seems more hoopy, it's often the case that those froods who are voting will strategically overreact, making PDQ come out more hoopy in the end. For example, look at Condorcet and IRV. If voters from a one-dimensional continuum vote honestly, IRV is subject to center squeeze, prematurely eliminating a centrist and electing a more extreme candidate; while Condorcet would elect the centrist. But with strategic voting, IRV voters may learn to strategically vote for the "lesser evil" centrist to prevent this from happening, leading to a system that is actually more biased towards centrist parties than Condorcet.

I think that could happen in this case, too. Let's take your example, slightly modified for extra realism:

45 Votes: AXX
9 Votes: ABX
36 Votes BXX
35 Votes: C (no X because C is a bit of an upstart)
30 Votes: DXX
25 Votes: EX
106 Votes: XXX
7 Votes: QXX
5 Votes: RXX
3 Votes: SXX
QRSTU - 160 Votes (the slate)

The changes I made make no difference to either of the outcomes you worked out. I just added a bit of overlap, and added 105-120 ineffective voters.

My point is that SDV and SDV-PE are designed not to incentivize much "don't vote for popular candidates" strategy. So, of the 320 non-slate voters in this scenario, if just 10% made one more approval and 5% made 2 more approvals, that would be 64 extra approvals. If 1/8 of those are for B, then B gets 8 more approvals — easily enough to win in both SDV and SDV-PE.

So I think that in practice, the outcome of RAV, SDV, and SDV-PE would probably be the same in scenarios like this. But that's not to say that they're identical. With SDV and SDV-PE, the outcome would be more variable dependent on how voters acted. If they reacted to the presence of a slate by approving more works, they could easily keep the slate to just 3 slots, as with RAV, and possibly even keep it to just 2. (In my estimation, that would take about 110-160 extra approvals; not a trivial amount, but certainly achievable).

So the questions for comparing the systems are:

1. Do we want a more off-the-shelf method (RAV), or something custom-build (SDV or SDV-PE)?
2. Which is more important: the risk that SDV or SDV-PE might not be able to guard against a slate "rounding up" its slots, or the opportunity that these systems could help people could come together to ensure a slate "rounded down"?
3. Which is easier to explain? (Probably argues for SDV, SDV-PE, and RAV in that order)
4. Is it better for a system to be more flexible based on what the voters do (SDV-PE), or should it try to be more foolproof and "protect them from themselves" (RAV)?

#296 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 09:47 AM:

Morning all!

Jameson@298:

Good questions! In my opinion:

1. I don't think (obviously) custom-build is a problem, so long as we have tested it to the greatest extent we can (simulations/proofs, etc.).

2. I'm not 100% sure I'm parsing "rounding up" vs "rounding down", but I think this is related to the goal 2a I proposed to Cheradenine last night: It's important that a system not allow a slate to prevent works from being considered. It is not important at all (again, in my opinion) that a slate's suggestions be kept off the ballot. Does that relate to this one?

3. Probably the single most important criteria, in my opinion. If it doesn't get buy-in from fandom and (perhaps more importantly) the business meeting, then it's all for nothing anyway.

4. I'm never in favor of protecting someone from themselves. If it's the "fans' award", then whatever the fans want is by definition the proper course.

Kilo

#297 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 09:52 AM:

Cheradenine@296:

I'll see if the work you recommend is in the library on campus; thanks for the recommendation.

It occurs to me that posting an email address here is probably a singularly bad idea. I'm not sure how to get that to you. Teresa/Abi: I presume you have both mine and Cheradenine's email addresses; can you possibly send my address to her directly?

Many thanks,
Kilo

#298 ::: Carrie S. ::: (view all by) ::: April 16, 2015, 10:31 AM:

conrad6 @#271: Not that you'll read it (or care if you did), but consider: the whole point of this discussion is to nix the power of slates. So your hypothetical "SMOF Slate", insofar as it would be represented here, is going to a great deal of time and effort to kill its own effect.

Now I suppose that makes sense in the universe in which They plan to simply start ignoring the voting altogether and just giving the Hugo to whoever They like best (surely the simplest way to get what They want, since They have such a lock on the process!), but it seems like a lot of energy to waste on mаскировка.

#299 ::: J Thomas ::: (view all by) ::: April 16, 2015, 10:50 AM:

Here is a first draft of a somewhat-formal description of what the Hugo nomination might be for. Anyone who wants to crib from it toward a document to be sent to the business meeting or whatever is welcome to.


The traditional Hugo nominations are broken. To fix them we must think carefully about what we need them to do.

Here is a possible general statement:

The Hugo nominations are a survey, to learn from fans what they think are the great works of the year that should be contenders for the Hugo award.

They should produce a final ballot which will be used to vote for a single winner. This ballot is a short list of excellent SF or SF-related items or people, chosen by a large number of Worldcon members. The nominations are a survey, not an election. They do not need to be (and cannot be) “fair” to a particular work.


Detailed goals:

1. All Hugo voters should have an equal opportunity to suggest great works.

2. Hugo voters should each be able to suggest multiple great works, up to some maximum.

3. Other things equal, works suggested by more fans should be chosen over works suggested by fewer fans.

4. Other things equal, two works suggested by different fans should be chosen over two works suggested by the same fans. Other things equal, between two candidate final ballots, the one that includes suggestions from more fans should be chosen.

5. To the extent possible, the nominating system should not encourage “strategic” suggestions. The preference is that a fan should think “What are the most excellent works I have experienced” and not “How can I game the system to make sure the ones I most want will win”.

6. If a nominee declines at the last minute, it should not cause undue disruption of the process.


Discussion of detailed goals:

2. We get various complications when single fans suggest multiple works, which would go away if they could suggest only one. But we learn more about the great works of the year when each fan can suggest more of them.

3. We cannot hope to be “fair” to works, and discussion of what would be fair to fans is complex and nuanced. But it makes sense that a work which is suggested by more fans is preferable to one which is suggested by fewer fans, other things equal.

4. Similarly, other things equal it is preferable to have two works that together are suggested by more fans, than two works that together get the same number of suggestions, but the suggestions come from fewer fans. Similarly for groups of three works, and so on.

3 & 4. Goals 3 and 4 can conflict with each other. We might sometimes prefer two works that get fewer total suggestions, because they have more total fans suggesting them. Somehow we must choose how much weight to give these different goals when they conflict. It might occasionally happen that the work that is suggested by the largest number of fans is not chosen. This is because all the chosen works put together got suggested by more fans, compared to the total fans for any combination of works that includes the one with the most. Etc.

5. The goals of the nomination system are best met if each fan nominates the most excellent works he knows. But fans may have other goals. A fan may want one particular work to be chosen, even though he thinks others are also excellent. If he wants one to win, why would he suggest any others? Or he may feel that a particular excellent work is so popular that it will be suggested by sufficient people that he can suggest something else in its place. Or many other possibilities. We must accept that fans may have goals that oppose our goal of getting an accurate survey. But we should attempt to design a voting system which encourages people to suggest the works they think are great. It is always possible to game the system, but one criteria for a good system is that it should not make it obvious in detail how to game the system.


This document is provided without copyright. Anyone can use or modify it however they want, provided they do not restrict the right of other people to use or modify the original.

#300 ::: J Thomas ::: (view all by) ::: April 16, 2015, 11:04 AM:

@301 Carrie S.

It would be possible to hold an attitude similar to Conrad6 and be logically consistent.

First, given the chaotic nominating system, small groups could maneuver to get their choices onto the final ballot, crowding out everything else. The groups could tend to win and still be small enough to be deniable.

By creating an undeniable threat, he helps to get us to fix the problem he sees. If there was no problem before, still it is a potential problem that is worth fixing. Presumably the small secret groups would be against any fix but they can't very well say so. We, trying to fix it, become his allies or his dupes.

It can fit together. The previous conspiracies are still deniable but not disproven. Ideas similar to his can fit together.

#301 ::: J Thomas ::: (view all by) ::: April 16, 2015, 11:06 AM:

Brad from Sunnyvale had a complaint:

For example, I have often, in a Hugo nominating category, only listed two, three or even just one work. And I know I am not alone. I am "wasting" my "power" and entirely not caring. It's just that I don't follow that category super closely, but I know the people I am nominating were good enough by my standard that year.

Many of the above systems will give this "only care modestly" ballot greater weight, as though all ballots were written by people who have 20 things they want to nominate and have to carefully pick the best 5. Now the hidden message will be "nominate fewer, your choices will do better."

Applying that to this:

While there are more than 5 candidates:
1. Split each vote into m pieces, worth 1/m each, where m is the number of non-eliminated works it supports; and give those pieces to the works.
2. Find the two candidates with the smallest pile of fractional votes.
3. Eliminate whichever of those two is on the fewest ballots.

If a single vote counts as m=1, then it counts as m votes in the early stages and other votes only catch up after all the other votes on their ballots have been eliminated. That might be a powerful advantage.

If instead, a single vote counts 1/m and never increases because there are no other votes on the ballot to be eliminated, that might be a powerful disadvantage. But someone who wants to game that can list his single choice and then include four garbage works that he thinks no one else would consider nominating. They will be eliminated quickly, giving his real vote weight 1.

Is this an issue? If so, is there any way to fix it?

I tend to think it isn't. The only way for a candidate to lose is to be eliminated for having one of the fewest adjusted votes and the fewest total voters. If it has a chance at being in the top five, most of its voters' votes will have been eliminated before its own turn comes. Giving it a full vote early probably won't matter -- if it's in danger of being eliminated early, it probably doesn't have a chance later.

#302 ::: Lydy Nickerson ::: (view all by) ::: April 16, 2015, 11:24 AM:

On the topic of being easy to explain and fair: The truth is that the muse of math and I don't get on. Simple arithmetic and I are on fighting terms. 4s and 7s are confusing, and the numeral 8 and I haven't been on speaking terms in years. I genuinely can't evaluate the mathematics of the various voting systems. What I can evaluate are the people who recommend a change. I do this partly by reputation, partly by reading the posts looking for obvious self-serving bullshit, and partly by noting the response of people I know and trust to people I don't know and trust. If Bruce Schneier and Kevin Standlee assure me that the new nomination system is fair, and will yield a slate which is diverse but reflects the informed interest of Worlcon members, I'm likely to be in favor of it, although I am incapable of understanding it.

I am also looking for a system where I can follow the instructions. As per above, I don't and can't really evaluate the exact reasons why one does a certain thing. But I do need to know if it is important that I rank below No Award, or some other arcane thing. The instructions need to be reasonably understandable. I'm capable of following instructions that are somewhat complex, but I do need sufficient information to be able to figure out how to deal with my own complicated preferences in order to get a result close to what I want to achieve.

My best guess is that I'm not alone. It is important that the math make sense to anyone who can follow it. But reputation is also going to be important, here. This is probably true for the SP as well. They are likely to refuse to accept any explanation from anyone they don't trust. Do they have any good statisticians on their side? Of course, if all they really want to do is break the Hugos, then no amount of analysis will matter.

#303 ::: J Thomas ::: (view all by) ::: April 16, 2015, 12:23 PM:

@305 Lydy Nickerson

I am also looking for a system where I can follow the instructions. .... The instructions need to be reasonably understandable.

OK! If you aren't interested in how to game the system or how to stop somebody else from gaming the system, then it's simple. Here's the most important part.

Nominate the five you think are best. If you don't think there are five that are good enough, then nominate four or as many as you choose. Your vote will help the ones you nominate.

That's it. That's all.

Now here are some complications you can think about if you want to.

We have two different ways we can arrange counting. With one of them, you just list all the choices you think are good enough. The other way, you list them in order. The one you most want to win first, then the second, then the third, etc.

If it's the first way, then here's something you might choose to think about. Say that you think Fires of Europa is the best, and you think Feldspar's Luck is good enough. Imagine that you nominate both of them, and then it turns out that Feldspar wins and Fires does not. If that happens and you're ready to get all upset about it, maybe it's better that you not nominate Feldspar. Because it might happen. You can put as much calculation as you like into this. If you're pretty sure that Feldspar isn't that popular, not like it can win when Fires doesn't, then you might think you can afford to nominate it anyway. Or if you wouldn't be *that* upset. If you could just yell a little "How could they *do* that, what were they thinking?" and get over it, you might figure it's still worth doing.

If your attitude is that you'll just nominate the ones you think are good enough and let whatever happens happen, then you don't need to think about this at all.

If they say to list them in order, then do that. If you can't decide between two which you want more, then it doesn't matter which comes first.

Finally, No Award. This is for voting. You can vote to not have any award made.

The special issue since the votes count from best to least, is that things you don't list you don't vote for, but things you do list below No Award may get voted on if No Award loses. Basicly, if No Award is not that popular but they are, then voting them below No Award means your votes for them might count. You're saying that if No Award won't win, you'd like them to win.

If that's too confusing then just don't list anything below No Award. Then you're saying that if the things you want don't win, and No Award doesn't win, then you don't really care which of the others wins. That isn't a terrible stand to take.

I can't think of anything else you need.

If you want to play games to win more than your share, or stop other people from playing those games, then it's endlessly complicated. But if you just want to do the right thing yourself then the complications dissolve away.

#304 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 12:53 PM:

OK, read all the comments so far, now commenting.

Bruce @6: I want nominating to be as easy as possible -- "It's okay that you haven't read everything, just send us names of works you think are worthy"

Gah! I want nominating to be easy as well, but I think suggesting that people nominate things they haven't read is more pro-slate than anti-slate. (And, for the record, I'm a Sad Puppy saying that. I can't speak for anyone else involved with SP3, but I didn't nominate anything I hadn't read.)

I've seen a lot of discussion of various option #3s, but very little discussion of frobbing option #2. I think playing with option #2 offers one of the best chances to reduce and/or eliminate the effect of slates by simply eliminating the 5 candidate limit for the final vote, and putting everyone who gets 5% of the vote on the final ballot. If the Gonosypherpalitic Juvenile Canid Slate votes in absolute lockstep, all 300 of them, then yes, their 5 candidates will be on the final ballot. But since they only comprise 14% of the total population (historically speaking, 2100-ish nominators), there could be as many as 17 other candidates.

It makes the pool fairly large, but doesn't leave it quite as wide open as the "no nomination system at all" or "everything that gets nominated at all is on the final ballot" proposals back around the #230's.

I suppose it's still somewhat subject to things which came close getting swept off if there are lots and lots of competing slates, but not nearly the possibility of the effect of this year's nomination round.

#305 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 12:55 PM:

@302: That looks good. A few notes:

2: I'd suggest removing "up to some maximum". Yes, I understand that Worldcon members may be reluctant to remove the maximum, and that's not the end of the world. But really, most if not all of the systems under discussion work better without a maximum. In any case, a detail like that certainly shouldn't be in a general statement of principles like this one.

Add:

7: The voting system should make as much sense as possible to as many voters as possible. Complicated mathematics should be kept to the minimum necessary, and if possible that necessity should have a good intuitive explanation.

and then in "Discussion":

2. Well said.

5. "It is always possible to game the system" is a little too pessimistic. Rather: "No system is impossible to game." That is, there may be some limited circumstances when it is indeed impossible to game a given system.

.....


So, let's see how the systems under consideration measure up on those goals. Here's my subjective rating each of RAV, SDV, and SDV-PE from 0(worst) to 10(best) on each of those criteria.

1 (equal opportunity) All three get a 10.
2 (multiple works) RAV 9, SDV 10, SDV-PE 10
3 (more over fewer) RAV 9, SDV 9.5, SDV-PE 10
4 (diversity) RAV 10, SDV 9.5, SDV-PE 9
5 (strategy resistance) RAV 8, SDV 9, SDV-PE 10
6 (declined nominations) Depends on the rules, but assuming you build a list of the top 10 and then when somebody declines, just pull from the list: RAV 8, SDV 10, SDV-PE 9 or 10 (???)
7. (understandable and seems fair) RAV 3, SDV 6, SDV-PE 5


#306 ::: P J Evans ::: (view all by) ::: April 16, 2015, 12:58 PM:

307
Generally, people aren't going to be nominating anything they haven't read or seen. How else can you decide if it's worth a rocket?

If you think only one item is that good, that's all you need to put down. (It isn't really a contest for how many you read or saw.)

#307 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 01:04 PM:

Actually, I think I just realised that I might be confused about how the 5% rule works.

Presuming everyone who sends in a nomination ballot picks five nominations for Best novel, and there are 2000 total ballots, is that "5% of 2000" or "5% of 10,000"?

If it's the former, my above proposal might end up with 100 nominees on the final ballot. Which is, um, a lot.

If it's the latter, if the Canids are a block of 200 out of 2000, and no-one else votes for any of the works on their slate, I don't think any of their slate picks make the final ballot, because each of their picks only have 2% of the final vote.

#308 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 01:09 PM:

P J @309: Sure. I only put 4 things on my ballot this year for Best Novel for that very reason. I just got a frisson of twitch when I read that comment by Bruce, so... *shrug*

#309 ::: P J Evans ::: (view all by) ::: April 16, 2015, 01:11 PM:

310
It's by category. So 5% of 2000.
This rule matters a lot more in the categories with fewer nominees - not usually in novel and dramatic presentation.
At the Hugo Award site, some years have the numbers posted, so you can see the cutoff points.

#310 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 01:12 PM:

@299: "I'm not 100% sure I'm parsing "rounding up" vs "rounding down"....

In order to explain, let's use some numbers. Say that in a certain category, 15% of votes are ineffective long tail "undervotes" — that is, only voting for things that almost nobody else mentioned; and 10% are ineffective "overvotes" — that is, only voting for things that could have won anyway even without their votes, or things that have no chance, but nothing in the middle. So that leaves 75% effective votes; 1/5 of that, or 15%, should be about right to get one nomination slot.

Now imagine that a slate has 33% of the voters. So it "deserves" about 2.2 slots. Of course, it can't get 2.2, so there's a choice between 2 or 3. I think most people here would agree we'd rather "round down" and give it 2 slots, than "round up" and give it 3.

RAV would tend to "round down" in this case until it had about 40%, so that's a good system. SDV and SDV-PE would tend to "round up". However! SDV and SDV-PE might tend to encourage "undervotes" less than RAV would, so it's I think it's probable that if you account for that, they would get the same answer as RAV, and also get it "for better reasons" (in a way where it's easier to demonstrate that it satisfies more of the voters).

And those numbers were my guesses of what might happen in a category like "Best Novel". In something more diffuse, like "Short Story", under- and overvotes could make up almost 50% of the votes. In that case, RAV would probably give the slate 3 slots, while something like SDV-PE could conceivably give it 2 ("rounding down), 3, or 4 ("rounding up") slots.

So is the chance of keeping the slate to 2 worth the risk of it getting 4? That's the basic question here. And it hinges on how people would respond to the strategic incentives particular to a given system — something we can't possibly know for sure ahead of time, even if we had all the old election data.

I don't think this is the most important difference between RAV and the two SDV options. But it is a difference.

#311 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 01:15 PM:

Note that numbers are probably going to jump next year anyway, so the idea that slates will be motivated enough to grab 4 slots even in a proportional system may be far-fetched.

#312 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 01:36 PM:

@313: Thanks, that makes sense. My personal preference would be for SDV-PE, then, specifically because there is more uncertainty in how the nomination would go. It seems to me that uncertainty of this type (in general) makes the system somewhat less easy to game, since -- as you point out -- neither we, nor a potential slate, can possibly know what voter behavior is going to be ahead of time.

Kilo

#313 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 01:49 PM:

#312: No, I mean, a ballot has five slots on it for Best Novel. If all 2000 ballots have all five slots filled, that's 10,000 "votes" cast for nominations for Best Novel, even if they were only cast by 2000 people.

#314 ::: P J Evans ::: (view all by) ::: April 16, 2015, 01:54 PM:

That's why I mentioned that for some years, at the Hugo site, you can see the actual numbers with the cutoff points.

#315 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 02:07 PM:

#317: Ah. Yes, good call. I'll go do some math.

#316 ::: Mary Aileen ::: (view all by) ::: April 16, 2015, 02:29 PM:

perlhaqr (307): Bruce @6: I want nominating to be as easy as possible -- "It's okay that you haven't read everything, just send us names of works you think are worthy"

Gah! I want nominating to be easy as well, but I think suggesting that people nominate things they haven't read

I'm pretty sure Bruce was saying "nominate worthy things from the subset that you *have* read", not "it's okay to nominate things you haven't read".

#317 ::: J Thomas ::: (view all by) ::: April 16, 2015, 02:43 PM:

@308 Jameson Quinn

Thank you for the comments. I have incorporated them in my copy, I'll wait in case there are more rather than repost it.

While I was looking at ways to say that, I found criteria to judge the result of a nomination system. Not a way to find out how to pick the winners, but a way to rate how well it's been done.

How to score a nominating result:

1. Count each ballot that includes at least one of the five winners.

2. Count each ballot that includes more than one winner.

3. Add #1 plus a factor n times #2, divided by the number of ballots.

4. Take the number of votes for the fifth nomination divided by the number of votes for the fourth nomination. Multiply this fraction times the fraction from #3.

As a first guess I would set n = 0.1.

The first part is a sort of "dissatisfaction" score. The more ballots that have no winners, the more dissatisfaction.

The second is an "antidiversity" score. It's better to have one ballot get more winning votes than to not have more winning votes, but it's much better to have a ballot that gets its first winning vote.

The third is an "antievenness" score. It's better not to have any of the winners with too little support.

It should be clear where this is heading. If 20% of the voters are a slate that wins everything, that gives a score of .22

If 5 groups of voters, each 20% of the total, get one winner each with no overlap, that's a score of 1.0.

If 5 groups of voters get one winner each and two of them overlap, so that the vote comes out 20%, 20%, 20%, 30%, 30%, the score is 1.02. Times 2/3.

If 5 groups of voters get one winner each and two of them overlap a different way, so that the vote comeds out 20%, 20%, 20%, 20%, 20% and 10% of ballots have no winner, the score is 0.92. Times half.

This doesn't match up perfectly with "average satisfaction" versus "representativeness". But that's OK. A system that caters too much to slates can't do well because it doesn't spread the votes out well enough, it doesn't promote diversity.

But diversity isn't the only goal, we also care about getting more votes even when they aren't diverse.

Fraction of ballots with at least one win will not correlate strongly with fraction of ballots with more than one win.

I like this way to decide how good a result is, but I am not certain what value the n parameter should have.

Ideally knowing the desired outcome would lead to a voting procedure that would produce that result, but I don't see it this time.

#318 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 02:51 PM:

J Thomas @302: This draft looks good to me. A few thoughts:

I think I would reword the statement "This is a survey, not an election," to clarify what distinction it's intended to make, since different people will have different associations with each word. To me, technically, the process of gathering and analyzing the data is a survey; once we're using a method to determine a set of choices based on that data, we're having an election (or, if we want to be super formal, "applying a social choice function"). I wonder if something like "This is intended to be a representative process, not a political one" captures what you meant? But you're a better judge of that than I.

I might trim down #4 by deleting the second sentence (which I think repeats the thrust of #3, unless I'm missing a distinction?) and reword #5 to something like, "The voting system should minimize the incentive for a voter to think strategically, instead of casting an honest vote for the works they consider excellent."

#319 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 03:14 PM:

J Thomas #320: "4. Take the number of votes for the fifth nomination divided by the number of votes for the fourth nomination. Multiply this fraction times the fraction from #3."

I think this rule introduces some odd behavior into the metric. Say we have the following distribution of ballots (assuming bullet voting for simplicity):

A 20%, B 20%, C 20%, D 17%, E 12%, F 11%

The score for ABCDE is .89 + 0 (no one has more than one winner) multiplied by (12/17) = 0.628.

The score for ABCEF is .83 + 0 multiplied by (11/12) = 0.7608.

So according to the metric, ABCEF is a better set of candidates than ABCDE, which doesn't seem right.

If we disregard #4, this is quite similar to the metric used in PAV. In PAV, a voter has a satisfaction score determined by how many of their nominees made the ballot: 1 for 1 nominee, 1 + 1/2 for 2 nominees, 1 + 1/2 + 1/3 for 3 nominees, ... This proposal is equivalent to those scores being 1, 1 + n, 1 + n, 1 + n, ... The fact that getting more than 2 of a ballot's nominees doesn't increase satisfaction at all could lead to some weird behavior in edge cases (say, if 60% of the ballots all agree on 3 candidates); 1, 1 + n, 1 + n + m, ... where m is less than n.

The most straightforward way to design a system around a metric like this is the PAV approach: compute this score for all possible winning sets, and take the one with the highest score. As with PAV, though, this runs into computational difficulties.

We could also use it as you suggest to analyze different choices of voting systems, though we'd need to tune the metric first.

#320 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 03:45 PM:

If you're looking for a measure of evenness, one way is to use squares of fractions. For instance: divide each vote among the winners, a la SDV; then take the sum of the squares of the amount of votes for each winner.

However, as Cheradenine (and Clay, earlier) pointed out, this kind of measure will sometimes favor a slate of winners that seems objectively worse than another from point of view of simply maximizing happiness. For instance:

10: AB
10: AC

This measure prefers BC to AB or AC, even though all the voters like A.

#321 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 03:48 PM:

Aak... I got the math very wrong in my last comment. Please ignore.

#322 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 03:58 PM:

I think if you just turn everything I just said completely backwards, it is correct. Take the negative square of the number of people not voting for each candidate.

#323 ::: J Thomas ::: (view all by) ::: April 16, 2015, 04:24 PM:

@322 Cheradenine
"4. Take the number of votes for the fifth nomination divided by the number of votes for the fourth nomination. Multiply this fraction times the fraction from #3."

I think this rule introduces some odd behavior into the metric.

Oops! I guess so! I don't know how that slipped by, I meant the fifth divided by the first. The fractional difference of the biggest and the smallest.

Maybe it would be better to use the one with the least votes divided by the total number of ballots. The point is we want the smallest to be big, not that we give it a pass if the largest is small too.

If we have four nice big groups and then the fifth is a tiny scattering of votes for a Gor novel, the four extra-good groups don't make up for the bad one.

#324 ::: albatross ::: (view all by) ::: April 16, 2015, 04:41 PM:

Jameson Quinn:

I was thinking more about SDV-PE, and tried to hack something up in Python to make sure I understand it. I noticed a couple places where I wasn't sure what to do:

First: Am I right in thinking that as we eliminate candidates from the list of possible candidates, we also remove those candidates from all the ballots? That is, if I have a ballot

[Alice, Bob, Carol]

then in this round, Alice, Bob, and Carol each get 1/3 vote from my ballot. At the end of this round, I eliminate Carol. It seems like I should now change that ballot for the next round to

[Alice, Bob]

for the next round. Is that right?

Second: When we're deciding which candidates are up for possible elimination, we can have ties for lowest or second-lowest fractional vote total.

Suppose at the end of a round, I have these ballots:

3: ADE
9: E
1: AB
1: C
1: BC
1: BE
8: A

We thus get these fractional vote totals:

A 9.5 votes
B 1.5 votes
C 1.5 votes
D 1.0 votes
E 10.5 votes

Your description is that we want to choose the lowest two fractional vote totals in this round, and decide which one to eliminate by seeing which is on fewer ballots. But in this case, we'd have to make an arbitrary choice between Bob and Carol to keep it down to two potential candidates to eliminate.

The only sensible way I can see to handle this case is to put Bob, Carol, and Dave all three up for possible elimination.

Third: When considering candidates for elimination, we could have ties for lowest number of ballots. (Again, I think this will happen a lot early on, when there are lots of candidates with few votes.)

That is, suppose we followed my idea above and now are considering Bob, Carol, and Dave for elimination in this round. And suppose when we look at it we see

Bob: 3 ballots
Carol: 2 ballots
Dave: 3 ballots

Now, we have a tie between Bob and Dave. In this case, they have different fractional vote totals, but that's not always going to be true--they could be tied on everything.

The simplest way to deal with that is to just eliminate all the tied candidates. But that does leave the possibility that one round of the election could eliminate all the remaining candidates. That's just what will happen if we have really dispersed votes (like everyone votes for 3 candidates, and they're all different candidates).

It's always going to be possible to tie in this elimination step, and have more than one candidate that is equally worthy of elimination.

#325 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 04:59 PM:

Fifth divided by first actually leads to some even funkier behavior:

A 60%
B 10%
C 9%
D 8%
E 7%
F 6%

ABCDE gets a rating of .93 X (7/60) = 0.1085.
BCDEF gets a rating of .40 X (6/10) = 0.24.

Using fith place votes/total ballots would fix this pathology, but I think it may weight too heavily toward taking the five top vote-getters, even if some or all of them are on a slate. Consider:

20% vote ABCDE (the slate)
40% vote F
15% vote G

ABCDF would have a score of 0.64 X (.20) = .128.
ABCFG would have a maximum possible score (assuming no overlap between F and G voters) of 0.79 X (.15) = 0.1185.

So a 15% increase in the number of fans represented is overpowered by the fact that the non-slate work only has 3/4 as many votes as the slate works.

I don't think it's unreasonable to have a factor that reflects "we'd like the individual works to have larger numbers of votes," but I don't think it can be anywhere near this strong.

#326 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 05:10 PM:

albatross @327: I had a few ties in the vote count come up while simming SDV-PE (partly because I have a bunch of unnamed candidates with 50 votes) and broke them by going back to the candidate with the least fractional votes. It's certainly possible to have a tie in both of these; in that case, I would lean toward a protocol of

1. If eliminating all the tied candidates leaves 5 or more candidates, eliminate all the tied candidates.
2. If eliminating all the tied candidates leaves 4 or fewer candidates, expand the nomination pool to include all the tied candidates (which is the current practice).

This procedure isn't perfect; #1 in particular could lead to a different outcome than eliminating one tied candidate at random. The alternative would be to develop more tiebreakers.

In the other case, I think your solution of "compare all 3 candidates and eliminate the lowest" is definitely the best. So, if two or more candidates tie for the lowest fractional votes, consider them all and eliminate the one on the fewest ballots. If two or more candidates tie for second lowest, consider the lowest and all the tied candidates, and eliminate the one on the fewest ballots. (And if a tie exists in the ballot count, follow the procedure above.)

#327 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 05:17 PM:

@327

Eliminations (Alice etc.): Yes, you're right.

Ties for lowest two: consider all three for elimination, but eliminate just one.

Ties for elimination: (Your example is not tied for last, so it's easy, but I know what you mean.) Various possibilities. Flipping coins, using some other attribute of the ballot to decide. One possibility would be to eliminate both. This would make slates which didn't get votes from non-slate voters extra-risky, and actually probably wouldn't affect any other candidates (since if they don't have significant overlap, both will probably be eliminated soon anyway).

#328 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 05:20 PM:

Jameson @330: I think a nondeterministic method is preferable to flipping coins here. I don't want the Hugo administrators to have to get their coin flips notarized. :)

#329 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 05:21 PM:

A deterministic method. Apparently it's State Things Backward Day.

#330 ::: CHip ::: (view all by) ::: April 16, 2015, 05:43 PM:

J Thomas @ 269: I don't think the Hugos ever had a goal as such; the object was to perfect the process of picking a winner. (As noted previously, some people offer education as an excuse for the Retro Hugos; I disagree that this is even an effect, let alone a justification -- but the RH are a separate and self-limiting case.) Your attempt at a mission statement is interesting, but I wonder where it will fit in; the WSFS Business Meeting usually deals with much more focused/quantitative matters. Possibly you should put this up in a wider forum?

piling on to conrad6 @ 271: If you want to convince us of anything, you might try getting your most checkable fact correct. Shaver was a 1940s phenomenon.

PJ @ 312: This rule matters a lot more in the categories with fewer nominees. I think you've got that backwards; the 5% rule has historically taken effect in the shorter fiction categories, where there are many more nominees. Long DP will never have many nominees (because a long-form DP costs a lot to bring out) and is very unlikely to get trimmed by the 5% rule.

ddb @ 291: <grins>

#331 ::: P J Evans ::: (view all by) ::: April 16, 2015, 05:49 PM:

333
Getting it backwards day, indeed: that was what I meant. Me braining not good today.

#332 ::: perlhaqr ::: (view all by) ::: April 16, 2015, 06:00 PM:

OK. So, looking at the math at thehugoawards.com for past nomination rounds, it looks like the percentage is calculated as #votes / #ballots.

Which means that my "take everything with 5% or more" leads to a potential situation where there are 100 items on the final ballot, which is probably untenable.

OK, so... what if that got changed? #votes for a work / #votes cast total? Of course, then you might not get anything that gets 5% of the vote, so you might want to change the 5% rule as well, to something more like "everything between the highest percentage of the vote * $X and the work which got the highest percentage of the vote", where $X is .5 or .75 or something determined by looking at actual vote totals, and figuring out what fraction there should be based on how many nominees one ends up with from various years, and with a floor value so that you never have fewer than $Y works on the final ballot.

It also has the side-effect of causing slates to either self-throttle or drown themselves. A slate could vote for one work in lockstep, and do fairly well for it, because their votes only increase the vote total by 200 as well. But a slate of five nominees, voted in lockstep, only increases the vote count for each novel by 200, but increases the total vote count by 1000, effectively reducing the voting power of the slate votes.

#333 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 06:08 PM:

CHip@333: I agree the goals of the Hugo have not been stated, historically. That's because there was never a need to. But now a situation has arisen where it's become clear that we actually need to define what we're talking about.

I'm an astronomer; the same situation famously came up with Pluto. We realized we didn't actually have a definition of a planet -- but we never really needed one before. Once Eris was discovered, we realized we needed to be precise in what we were talking about.

The definition that eventually passed is ridiculous, but I digress...

Kilo

#334 ::: Brad from Sunnyvale ::: (view all by) ::: April 16, 2015, 06:10 PM:

Several people have said to me "every system has strategy" -- this misses the point which is that some systems have much more strategic voting incentive in them than others. And I think the systems proposed above all have too much.

Even single nomination has more strategy than 5 nomination approval that we do now. And single nomination can still be dominated by slates.

I still don't see anything that comes even close to the plain approval 5 ballot that we do now when it comes to the influence of strategy. I know this because people very rarely talk about strategy for it, because there almost inn't one.

Only option 4 fights slates without increasing the complexity and strategy in the nomination process.

#335 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 06:18 PM:

Brad @337: Please stop making authoritative-sounding statements about strategy and gameability. You're basically talking about game theory and probability, which are well-studied fields, and human instincts for probabilities in complex systems are terrible.

#336 ::: J Thomas ::: (view all by) ::: April 16, 2015, 06:45 PM:

@323 Jameson Quinn

If you're looking for a measure of evenness, one way is to use squares of fractions. For instance: divide each vote among the winners, a la SDV; then take the sum of the squares of the amount of votes for each winner.

However, as Cheradenine (and Clay, earlier) pointed out, this kind of measure will sometimes favor a slate of winners that seems objectively worse than another from point of view of simply maximizing happiness.

That is inevitable.

If you start out with criterion A, and then you let some other criterion B influence your results, you will no longer maximize A. It will look worse to people who think you should just maximize A.

Of course, it's sometimes possible to fall between two stools. To get a compromise that's worse than either extreme.

#337 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 06:46 PM:

Brad@337: I agree that option 4 is simpler, and honestly to be preferred (since it alone -directly- addresses the problem). Personally, I'd love to see it happen. I just don't see any way to make it work, apart from giving a human committee the power to make the call, as you suggested on your blog. My problem with that is that this effectively proves what the puppies have been saying: "There's a committee which gets to approve which works can be nominated." You and I know this committee wouldn't exclude something because of content or who wrote it, but it would certainly look that way -- or at the very least, could be taken out of context to look that way.

Then of course there's the problem of how to unambiguously define and detect slates. I've got the same problem detecting cheating when I give an exam. The more similar two wrong answers are, the less likely it is to have occurred by chance, but can I -prove- it was cheating? It's surprisingly hard. I'm not saying it can't be done, but after reading your blog, I thought very hard about it and couldn't figure out a way beyond "trust the committee's judgement." I just don't think I could accept that, and I'm fairly certain the puppies and their ilk couldn't either.

If there were enough nomination slots, then the odds of having (say) ten identical nomination ballots just by chance is very low -- so it's probably a slate. Another example: What if a large group pushes three works out of a possible five, is that a slate? What if they divide into two groups and each pushes three different works? I'm not sure how the committee would be able to throw that out. I dunno, I'd be interested if there were way, though.

Kilo

#338 ::: Keith "Kilo" Watt ::: (view all by) ::: April 16, 2015, 06:48 PM:

Hmm, number swapping mania... I guess Brad's message is now #334..

K

#339 ::: J Thomas ::: (view all by) ::: April 16, 2015, 06:52 PM:

@332 perlhaqr

But a slate of five nominees, voted in lockstep, only increases the vote count for each novel by 200, but increases the total vote count by 1000, effectively reducing the voting power of the slate votes.

I may not be following what you're saying here.

It looks to me like it increases the total vote count by 1000 for everybody else too, but increases their individual vote counts not at all.

#340 ::: J Thomas ::: (view all by) ::: April 16, 2015, 07:03 PM:

@325 ::: Cheradenine

I don't think it's unreasonable to have a factor that reflects "we'd like the individual works to have larger numbers of votes," but I don't think it can be anywhere near this strong.

Yes. Three different rules leave me wondering just how to bolt them together.

I think these are the parameters I want. The winners should have more votes. They should have votes from more fans. And the least of them should have votes from more fans too.

#341 ::: William Hay ::: (view all by) ::: April 16, 2015, 07:16 PM:

The works you nominate will greatly depend on which books you happen to have read. The books you have read will in turn depend on accidents of your personal history and acquaintances. Why not admit this and use chance to make the final decision as to nominees?

For each work nominated place one token into a tombola for each person who nominated it. Mix well. Draw tokens one at a time until you have five unique nominees for the final ballot.

Should be roughly proportionate over time, won't overweight slate candidates, will encourage diversity and discourage a tyranny of the majority by occasionally throwing up something that wouldn't have a chance in more deterministic systems.

#342 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 07:18 PM:

J Thomas @340: I think adding a small weight for "ballots with 3 works nominated," "4 works nominated," etc. may do enough to weight the metric toward avoiding works that appear on a small number of ballots. If we have separate elements in the rating representing "more votes" and "least with more votes," what it basically ends up doing is increasing the weight on "more votes." I think we'd be better served to increase that weight directly (I actually think .1 is probably too low for n).

#343 ::: Cheradenine ::: (view all by) ::: April 16, 2015, 07:22 PM:

William Hay @341: Random ballot came up for discussion in the first thread, and I'm not sure how much is to be gained from reopening that discussion.

#344 ::: nathanbp ::: (view all by) ::: April 16, 2015, 07:27 PM:

Jameson Quinn@305
7. (understandable and seems fair) RAV 3, SDV 6, SDV-PE 5

I'm curious what you (or others) think is difficult to understand about RAV? Or are you knocking it down for seeming less fair than SDV? To me, RAV seems among the easiest to explain and understand of all the proposed systems.

#345 ::: Jameson Quinn ::: (view all by) ::: April 16, 2015, 07:39 PM:

@344: Oh. I was really trying to guess what other people would think. Basing it on my own understanding is a bad idea, because I think about voting systems all day.

So, what do other people think? Which of the three is easiest to understand?

#346 ::: Brad from Sunnyvale ::: (view all by) ::: April 16, 2015, 08:18 PM:

#335 -- what is the statement I have made from authority, and what analysis shows you it is false?

Having a human based system of justice does not require exactness, though it is a virtue. The world has a lot of experience with such systems, some better than others. If humans are to exercise judgement, we know things we want -- transparency, accountability and a few others.

I am not entirely comfortable with the idea. I am led to it by finding too many flaws in the systems being proposed when viewed through the lens of what I believe is the goal of the Hugo system.

If you dislike the term "survey" I offer the term "measurement." Fans have opinions. There is such a thing as "The works which are the most supported, according to metric X, by the fans" It is something that can be measured -- if fans will express their true (and independent) opinions to us. Any factor, such as collusion, or voting strategy compromises how truly fans tell those views.

#347 ::: J Thomas ::: (view all by) ::: April 17, 2015, 10:15 AM:

I found a better way to say what I want for nominations, which may turn out to be similar to what a lot of fans would want.

What we want is for the nominations to provide a good set of results. We want to have five very good nominations. We want diversity, meaning a lot of fans have input. When one faction mostly gets its way, that's bad even when the total votes for their choices is more than the total votes for a more even set of nominations.

If at all possible, we want to wind up with five great choices. If the nominations result in one great choice and four mediocre ones, that is not so good. Four great choices and one mediocre one is not that good. We want five great choices. It's more important that the fifth choice be great than that the first choice be the best.

Here's a picture of the result I want:

You have six piles of ballots. Five of the piles are ballots for the five winners. The sixth pile is the pile of ballots that didn't win anything.

We want that sixth pile to be small. And we want the smallest of the five winning piles to be large.

If a slate is big enough to get a pile, then it gets one pile.

Five piles of winners and we want the smallest of those piles to be pretty big.

Imagine that there's a different way to arrange the piles where one of the piles is giant and the others are small, and the nominee in the giant pile is not in any of the winning five piles the first way. There is a big winner that we did not choose the first time. Would that be a better way to pile them up? Not necessarily.

Imagine this situation: One book got marketed very well and every fan in the world has read it. 20% of them liked it. A second book got read by only 10% of fans and they all liked it. Then in the nominations say the first gets 200 votes and the second gets 100. If we nominate the first one, everybody has already read it and four fifths of them didn't like it. If we nominate the second one, a bunch of people will read it and like it. I say in that particular situation the second book is the better one to nominate. Because in the nominations we can't expect everybody to have read everything. It's more important to have five nominations where each of them has a lot of fans that like it, than to have the one that the most people like.

I don't know exactly how much emphasis to put on making the pile of losers small, or how much to put on making the fifth-largest pile of winners large. That's two different goals that might conflict. And after we decide how to weight those, I don't know how to divide things up to get the best result.

But that's the result I want.

Oh, one more thing. If there are two different ways to pile up the ballots that get the same score, then I'd want to count all the second winning votes on winning ballots, and break the tie that way.

#348 ::: J Thomas ::: (view all by) ::: April 17, 2015, 10:18 AM:

@347

Why is this a good way to do it?

When I make suggestions to people, I don't say “Here's the suggestion I most want you to do, and here's the suggestion I second-most want you to do.” I just make suggestions. My suggestions are not a ranked vote. They are not even an approval vote. The nominations are not votes for best SF whatever of the year. They are suggestions for what to put on the short list.

The point of the nominations is to offer five great things that are all worthy. We want each of them to be suggested by a lot of fans, because when there are too few fans suggesting it then it might not be that great.

We want to accept suggestions by varied fans, we don't want all the choices to come from the same ones. Fans are likely to read everything on the short list, and we want that list to represent the diversity of fandom, for fans to sample.

#349 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 10:30 AM:

@346: I believe Cheradenine is using "authoritative-sounding" to mean "overconfident", not "based on argument from authority". So we're talking about @334 (which used to be @337). In particular:

Several people have said to me "every system has strategy" -- this misses the point which is that some systems have much more strategic voting incentive in them than others.

Correct, though I don't see anyone denying the point you say is missed.

And I think the systems proposed above all have too much.

This is your opinion, and you have a right to it.

Even single nomination has more strategy than 5 nomination approval that we do now. And single nomination can still be dominated by slates.

These are "authoritative-sounding" statements, and they are unsupported and in conflict with both the common voting-theoretic and common-sense definitions of the terms you're using. There may be some definitions under which they are correct, but you should at a very least be more careful.

I still don't see anything that comes even close to the plain approval 5 ballot that we do now when it comes to the influence of strategy. I know this because people very rarely talk about strategy for it, because there almost inn't one.

Are you kidding? Slates are strategy. That's the only reason we're having this discussion in the first place. Do you think we would be here if each puppy had independently nominated their 5 favorite works in each category?

Only option 4 fights slates without increasing the complexity and strategy in the nomination process.

Option 4 is not without complexity. And as regards strategy, you are either wrong, or at least have failed to explain under what definitions you could be considered right.

Strategy is anything which considers how other people are likely to vote. Say my preferences are A-Z in that order. If I always vote for A-E, or always vote for half of the candidates (A-M), or always vote for candidates I consider above average, then that is not strategic. But if I vote ABCD because E is too popular, or worse, EAIHD because I want to make sure TONSR lose, then that is strategic. Strategy can be honest or dishonest; it can favor or disfavor candidates perceived to be popular; and it can lead to a high-strategy equilibrium or a low-strategy equilibrium. The current system strongly favors dishonest strategy which helps popular candidates and leads to a high-strategy equilibrium (a two-party system). The proposed changes weakly (in some cases, very weakly) favor honest strategy which helps relatively-unknown candidates and leads to a low-strategy equilibrium. I really don't see how that could be considered a step down in terms of strategy, though I'm willing to hear any arguments you have.

#350 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 10:39 AM:

@347: This very much aligns with the SDV-PE process. SDV-PE is basically a bunch of steps where in each step you:

Divide everything into piles
Take the smallest two candidate piles
Eliminate the one of them which leads to the "didn't get a nominee" pile growing the least.¹

This is not guaranteed to be optimal by your definition, but it is pretty much exactly a greedy algorithm for approaching that, and greedy algorithms tend to do pretty well even when they're not optimal.


¹This is only approximately what the elimination does. You could define the elimination so it more strictly led to this outcome, but I don't think that would actually improve this system, for at least 3 reasons. Basically, it would be harder to understand; would lead to winners with less-broad support; and would encourage "avoid popular candidate" strategy more. I could state all of those with more mathematical precision if needed.

#351 ::: J Thomas ::: (view all by) ::: April 17, 2015, 10:54 AM:

@346 Brad from Sunnyvale

Having a human based system of justice does not require exactness, though it is a virtue.

Depending on the judgement of a large committee might not work well. For example, it might tend not to do anything until a lot of people get worked up, and then tend to do something drastic and counter-productive. But it might do some good and it might be that it will happen regardless.

Independent of that, we still need a way to measure the Hugo nominations. That's what this topic is about.

If you dislike the term "survey" I offer the term "measurement."

I like that. When we talk about voting it seems to imply obligations to the winners which I don't think we are obliged to honor, certainly not in the nominations.

Fans have opinions. There is such a thing as "The works which are the most supported, according to metric X, by the fans" It is something that can be measured -- if fans will express their true (and independent) opinions to us. Any factor, such as collusion, or voting strategy compromises how truly fans tell those views.

Your goal is that the fans should provide their true opinions about which five X are the best.

Sometimes something that looks to you like strategy might be an honest opinion. And regardless, other people have their opinions about what they want the Hugos to do.

For example, if you believe that there was only one X this year worthy of a Hugo, then the obvious way to express that opinion is to nominate only that one and then if it's on the final ballot to vote for that one. That's honest.

But if you believe that there were thirty X that were good enough for the Hugo but you want your one to win for some reason (like you want the author to survive as a professional writer and write more stuff you like) then you might also only nominate the one.

No way to tell which way a fan thinks unless you ask him and believe his answer.

Either way, he votes for just one candidate. We want to include as many ballots as we reasonably can, we want a nomination system that gives at least one win to a lot of fans. Either we give him that one, or his ballot is a loser. He has given us this implicit ultimatum. Any voting system has to handle it somehow.

If a bunch of people agree about which X they think are best, why shouldn't they vote that way? Most fans don't agree that much. But if some of them do, should we try to stop them from agreeing?

If they are shills who don't care about SF but just vote to help out their friend or patron, that's bad. But if they deserve to vote, then they get to vote their way. The most a voting system can do is to find some fair way to give them the same limited influence others have.

#352 ::: P J Evans ::: (view all by) ::: April 17, 2015, 11:28 AM:

351
Either way, he votes for just one candidate. We want to include as many ballots as we reasonably can, we want a nomination system that gives at least one win to a lot of fans. Either we give him that one, or his ballot is a loser. He has given us this implicit ultimatum. Any voting system has to handle it somehow.

I don't think that's going to fly very well at all.

#353 ::: TomB ::: (view all by) ::: April 17, 2015, 11:39 AM:

J Thomas @347: "We want five great choices. It's more important that the fifth choice be great than that the first choice be the best."

I agree that the relative ranking of the nominees doesn't matter. Due to the humongous size of the field, it is not possible to ensure that the five very best works are nominated, much less that they are nominated in order. One just hopes that the nominees are representative of the best of the year, and that the very best managed to get in as one of them.

What I want is for the actual best of the year to somehow get onto the ballot as one of the choices. (And a pony.)

"One book got marketed very well and every fan in the world has read it. 20% of them liked it. A second book got read by only 10% of fans and they all liked it. Then in the nominations say the first gets 200 votes and the second gets 100. "

Imagine that the second book was ready by 20% of fans and they all loved it. Then in the nominations it gets 200 votes, the same as the first book. There is no way to tell, just by counting votes, the difference between the widely marketed clunker and the neglected gem.

#354 ::: albatross ::: (view all by) ::: April 17, 2015, 11:43 AM:

Brad:

You've proposed giving the people running the award process the power to intervene in some way when they see evidence of slate voting. I wonder wheher any of the people who've run these processes would like that responsibility. It sounds like a nightmare to me.

My guess is that, say, disqualifying a bunch of nominations because they came from a slate, or disqualifying a nominated work because it was mostly nominated by a slate, would cause a lot of conflict and discord--probably many times what we're having now. It would certainly bolster the narrative of "see how our kind of works are robbed of their deserved awards?" It wouldn't be too shocking to see lawsuits arising from that, and at the least, I'd expect years of ongoing bad blood. This is especially true if next year's slate is explicitly not a slate (but rather another Greek of the same name)--you can imagine language that would make a reasonable-sounding case that you weren't doing a slate. (And you can't really forbid people making recommendations or suggesting a list of works you think are Hugo-worthy.)

There may be administrator actions that wouldn't kick off an endless conflict--letting the administratior add a work or two to the list on his own authority might work. But it looks like a bucket of snakes to me.

#355 ::: J Thomas ::: (view all by) ::: April 17, 2015, 12:12 PM:

@350 Jameson Quinn

@347: This very much aligns with the SDV-PE process.

Yes, kind of. What I want here is a simple obvious way to describe the goal.

Without that, somebody is going to say "Wait a minute, this candidate here had more votes than #5 and didn't get onto the ballot. This is wrong, bogus, corrupt."

I want to make an intuitive explanation why the five with the most votes don't have to be the winners. Otherwise, we have to either let a slate have most of the winners or else explain why the slate's votes are disqualified.

SDV-PE does not choose what I want but it chooses something vaguely similar, and it will often give a pretty good result. I would like to find something that does better, of course.

Also I will try to find a simple explanation for what SDV-PE does, to make a clear pitch for it.

#356 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 12:23 PM:

J Thomas @355: I think I'm having a little trouble perceiving what you're looking for that isn't covered by the goal of "balance selecting the works with the most nominations with selecting the works which together represent the most nominators' preferences." Does that capture it, or am I missing a nuance?

(Incidentally, in an SDV system, I'd be inclined to refer to the fractional votes at each stage as "votes" (or "points," if you really want to avoid election vocabulary) and the number of ballots on which a candidates is listed as "nominations" or "ballot appearances.")

#357 ::: J Thomas ::: (view all by) ::: April 17, 2015, 12:53 PM:

@353 TomB

"One book got marketed very well and every fan in the world has read it. 20% of them liked it. A second book got read by only 10% of fans and they all liked it. Then in the nominations say the first gets 200 votes and the second gets 100."

Imagine that the second book was ready by 20% of fans and they all loved it. Then in the nominations it gets 200 votes, the same as the first book. There is no way to tell, just by counting votes, the difference between the widely marketed clunker and the neglected gem.

Yes, exactly!

I can think of a way that would tend to palliate that, if the side effects weren't bad. If people could make negative nominations, then the things that a lot of people read and didn't like could show up. Things that people hadn't read would tend not to get negative nominations.

However, consider something that everybody has read and 80% didn't like. It might get 20% positive votes and 20% negative votes. (Because people tend not to think to make negative votes.)

Meanwhile something that only 10% of people have read that 20% of the readers liked and 80% disliked, might get 2% positive nominations and 0 negative nominations because the people who read it and didn't like it wouldn't imagine that it would get nominated at all. 2% won't get it on the final ballot, but there could be variations in there that go bad.

To know what to mark down you'd need to know what was nominated. It's hard to make just one change, one leads to another.

Also, people might play politics. Mark down things they think are good because they want their own favorites to win. If even 10% mark down everything they think is popular, it will be hard to find winners.

I don't know whether it's worth trying to fix this.

But if we don't, then we have to accept that the nominees with the most votes may not at all be the best.

So I think it isn't bad to ignore which have the most votes. Pick five that have the most varied votes, that together do the best job of getting something from each ballot, where each one got pretty many votes. That might not include the ones with the most votes. (If the ones with the most votes come from a slate, it won't include more than one of them.) But it will be a good survey of the ballots.

#358 ::: J Thomas ::: (view all by) ::: April 17, 2015, 01:16 PM:

@356 Cheradenine

J Thomas @355: I think I'm having a little trouble perceiving what you're looking for that isn't covered by the goal of "balance selecting the works with the most nominations with selecting the works which together represent the most nominators' preferences." Does that capture it, or am I missing a nuance?

I want to balance selecting the works which together represent the most nominator's preferences, with dividing the selected works' ballots into discrete groups while maximizing the size of the smallest group.

So other things equal, it's better for the stack of ballots that don't include a single winner to be 24% of the total than 25%.

And other things equal, if we can divide the winning ballots into 17% for A, 16% for B, 15% for C, 14% for D, 13% for E, that's better than if the best we can do is 20% for A, 16% for B, 16% for B, 15% for C, 12% for D, 12% for E.

Given ballots that include both A and B, and both A and E etc we have some leeway in how to divide them up.

The point here is that the winner with the least votes can carry its own weight. It is on a significant number of ballots. It isn't just something we stuck in at the end to recover the largest number of losing ballots after we maximized the first four.

#359 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 01:30 PM:

Cheradenine@356:

(Incidentally, in an SDV system, I'd be inclined to refer to the fractional votes at each stage as "votes" (or "points," if you really want to avoid election vocabulary) and the number of ballots on which a candidates is listed as "nominations" or "ballot appearances.")

I like calling them "points" a great deal, as I think we are pre-conditioned to thinking of "one person, one vote". It makes a person wonder, "How can I have a fractional vote?" We're writers (and readers), so I think we all have an intuitive understanding that words matter and often have a context beyond their actual meaning, so must be chosen carefully. One of the great things about physics is that I get to define a quantity and it means exactly what I want it to, regardless of what my reader wants it to mean. :) As a writer, I don't get that luxury!


JT@357 (and others):

I like the goal of (ideally) seeing nominators get -one- of their choices,but not necessarily their first choice, on the ballot as a good thing. In fact, I'd prefer that choices not even be ranked at all, even though we know each voter isn't going to like all of the works equally. It just seems to me that ranked choices still has too strong a connotation of a "winner", which isn't supposed to be the point at this stage. If there was no expectation that your nominations were ranked, I think most of fandom would accept it as just "part of the rules". There will always be the, "Well, this one isn't bad and probably deserves to be nominated, but I really want this other one to win!" But I'm all for letting the individual decide what constitutes "nomination worthy". Because, ultimately, I think that's what the nomination process is supposed to be -- not an election, really (though while there may not be a winner, there are obviously losers), but instead a threshold (set by the individual reader) as "above this, it's worth being considered, below this it's not". Of course, from a logistical standpoint, we have to decrease the number of nominees on the final ballot, so we choose a system that will narrow the field while still staying true to that principle. I've been going through all of the SDV-PE examples, as well as looking through some of the literature on SDV itself, and I'm starting to become convinced that SDV-PE may just meet those goals. I really think it's worth testing as a candidate system, and maybe even drafting some language to present it.

Kilo

#360 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 01:34 PM:

@356: In @347, there are three principles, which I interpret as:

1. Lots of average votes for each nominee.
2. Lots of ballots "satisfied"; few voters get 0 nominees.
3. No nominees with few votes.

Your question, if I understand it, is: why isn't 3 redundant with 1 and 2? So here's a scenario:

20: AB
5: AI
19: CD
5: CI
18: EF
5: FI
15: GH
7: GI
5: JH
1: J

Possible winner sets:
ACEGJ: 100 total votes from 100 total voters; worst winner J with 6 votes
BDFHI: 99 total votes from 99 total voters; worst winner F with 18 votes.
ACEGI: 116 total votes from 94 total voters; worst winner I with 20 votes.

I think that you could make an argument that BDFHI might be the best answer in this scenario. But according to principles 1 and 2 alone, it is dominated.

#361 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 01:39 PM:

J Thomas @358: I worry that that metric, as described, winds up actually disadvantaging the works with the most nominations, unless it's only the vote totals for E that are relevant. And, in that case, any criterion that focuses on the votes for a single nominee (the fifth in this case) is likely to lead to some distorted behavior where tweaks to the status of the fifth nominee have outsized effects on any measurement.

#362 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 01:45 PM:

In order to get SDV-PE to get the "right" answer in the scenario I just gave, you'd have to make the JH voters be a bigger pile and switch them to voting BDFHJ. It would also help if you switched some GI voters to IJ.

There's no way RAV can get the "right" answer in any scenario like this; there's no way to recover from the inevitable misstep on the very first nomination.

#363 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 02:01 PM:

Jameson @360: I see what you mean; a strict reliance on 1 and 2 would rank ACEGJ as better than BDFHI, and it probably shouldn't be. Nice example.

Isn't ACFHI probably superior to any of the listed options, though? (114 total votes from 99 total voters, worst winner H with 20 votes.)

#364 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 02:18 PM:

@363: You're right about ACFHI being better. You could probably tweak the example until it wasn't. But anyway, the main point of the example was comparing ACFGJ against BDEHI. (I mixed up E and F earlier.)

#365 ::: Brad from Sunnyvale ::: (view all by) ::: April 17, 2015, 02:28 PM:

A few notes on various:

I am using "strategy" to refer to individual strategy, as opposed to collusion. Slates are collusion, your votes are not independent.

There are many factors which might lead a nominator to not just offer their true opinion. I hope to find the system which minimizes the incentive to do that, and that is what I mean by choosing the system with the least strategy.

I do think Hugo admins would definitely not want to yield the power to intervene. I hope they would not want to yield it! But I think they would yield it this year if they had it; in general I hope this would be rare.

At the same time, a system of human justice offers a deterrent to attack. There is no point to attempting a slate if you know that if it is detected, it will be nullified as best as possible, and perhaps (though for liability reasons this may not happen) you will be identified as having tried to cheat, and your result will be negative.

As to why I say single-nominee has more strategy, this is because with multiple entries (both in nomination and final ballot) we reduce the extent to which you must "sacrifice" some choices to promote others.

#366 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 03:14 PM:

Brad@365:

I am using "strategy" to refer to individual strategy, as opposed to collusion. Slates are collusion, your votes are not independent.

OK, that helps me understand where you're coming from. But...

There are many factors which might lead a nominator to not just offer their true opinion. I hope to find the system which minimizes the incentive to do that, and that is what I mean by choosing the system with the least strategy.

...do you see how this is a different definition of strategy than the one you just gave? Slate voters are "not just offering their true opinion".

Furthermore, do you see how slates can happen through only individual strategy, without any collusion? Imagine the candidates were letters; my preferences are alphabetical, but I think that other votes will be approximately frequency-based (ETAOINSHRDLU...). Do you see how it behooves me to vote EAIHD rather than ABCDE? And how if there's a group of voters like me, we can end up all voting EAIHD without having to collude?

Finally, do you see how this kind of strategy is more problematic than the kind that can happen (much less, and much more riskily) under RAV or SDV-PE? If it's strategic to vote for popular candidates you kinda like ("lesser evils"), that becomes a self-reinforcing equilibrium. But if it's strategic not to vote for them, then strategy becomes less effective the more people use it, so it should tend to burn itself out naturally.

I do think Hugo admins would definitely not want to yield the power to intervene. I hope they would not want to yield it! But I think they would yield it this year if they had it; in general I hope this would be rare.

At the same time, a system of human justice offers a deterrent to attack. There is no point to attempting a slate if you know that if it is detected, it will be nullified as best as possible, and perhaps (though for liability reasons this may not happen) you will be identified as having tried to cheat, and your result will be negative.

It's better if enforcement is not a "nuclear option". When it is, the enforcers might be tempted to let minor violations slide; and if, as in this case, violations are a self-reinforcing cycle, that's a recipe for disaster.

As to why I say single-nominee has more strategy, this is because with multiple entries (both in nomination and final ballot) we reduce the extent to which you must "sacrifice" some choices to promote others.

I'm sorry, I don't understand what you're saying here.

#367 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 03:18 PM:

Brad @365: Any time you're adjusting your vote based on what other people are doing, your votes aren't independent, whether it's conforming to a slate or avoiding voting for the leading contender.

Block plurality (the current system) has a known serious vulnerability to slates (see here - yes, it's a Wikipedia link, the primary literature is too technical), but is also subject to being "gamed" even in a low-information environment. For example, it offers the strategic choice of nominating one candidate ("bullet voting") vs. nominating more than one. This isn't a decision offered with single-nominee. Saying the system where you have to choose one candidate to vote for involves more strategy than a system where you can make a tactical choice as to whether to choose one candidate or not is pretty much nonsensical.

(That Wikipedia entry is a little painful. "A coalition has substantial incentive to nominate a full slate of candidates, as otherwise supporting voters may cast some of their remaining votes for opposing candidates." Yeah, I think we just got to see how that works.)

#368 ::: perlhaqr ::: (view all by) ::: April 17, 2015, 04:02 PM:

J Thomas @339: "It looks to me like it increases the total vote count by 1000 for everybody else too, but increases their individual vote counts not at all."

Well, yes, that's true from a purely numeric sense, but not an accurate reflection of the effect.

Imagine you have a voting block of 200 Poor-Digestion Foxes, and they all nominate the same five books.

"Book One", by A Writer
"Book Two", by B Lister
"Book Three", by C Monster
"Book Four", by D Best
"Book Five", by E Pluribus

So, as it stands now, they vote that way, the vote count for each novel goes up by 200, and the divisor goes up by 200.

Under my theory, the vote count for each novel goes up by 200, but the divisor goes up by 1000.

Now, you've got a different pool of 1800 Trufen, of whom, 200 of them all happen to nominate "Book Six", by F Miner, and purely by happenstance, none of their other suggestions overlap in any way. (Ok, unrealistic, there just aren't that many books published in a given year.)

...

Actually, y'know, I think you're right that this doesn't accomplish what I thought it did. Bugger. :-/ Sorry. Nevermind.

#369 ::: albatross ::: (view all by) ::: April 17, 2015, 04:02 PM:

Short-term effect of SP/RP actions: Messing up the Hugos for a few years.

Long-term effect: SFF fans become major force in driving reform of election systems, having become fascinated en masse with election theory.

#370 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 04:38 PM:

@369: That's my devious plan. I'm even part of an organization devoted to the latter outcome. But I'm not supposed to post the name of that organization here, even legalistically endorsing common themes of lookouts on ... oh, forget it.

Our triumph is inevitable! Rot13 is futile!

#371 ::: J Thomas ::: (view all by) ::: April 17, 2015, 04:53 PM:

@361 Cheradenine

J Thomas @358: I worry that that metric, as described, winds up actually disadvantaging the works with the most nominations, unless it's only the vote totals for E that are relevant. And, in that case, any criterion that focuses on the votes for a single nominee (the fifth in this case) is likely to lead to some distorted behavior where tweaks to the status of the fifth nominee have outsized effects on any measurement.

I can't be sure that won't happen. I had a vague idea about a scheme to actually achieve the goal, and I hope it is easier to raise the minimum than to try to reduce the variance.

If I rate outcomes by how even all five are, that might involve a lot of computation.

I can imagine the works with the most nominations might often be disadvantaged, because sometimes they will tend to have the most overlap with other nominations, and so that might make them the most disposable. In a group of five works, the one that has the fewest unique ballots -- ballots that have none of the other four votes on them -- is the obvious choice to replace with something else. When the work with the most votes also has the most unique votes it's no problem.

I'd want some simulation to get a feel for what to expect, and it will take me some time to ramp up.

#372 ::: felice ::: (view all by) ::: April 17, 2015, 05:28 PM:

J Thomas @351: "Either way, he votes for just one candidate. We want to include as many ballots as we reasonably can, we want a nomination system that gives at least one win to a lot of fans. Either we give him that one, or his ballot is a loser. He has given us this implicit ultimatum. Any voting system has to handle it somehow."

We really want to encourage people to nominate as many works as possible. With fewer works nominated, there's likely to be less overlap between ballots, fewer nominations for the best works, and more power for slates. I'd recommend that in "making the pile of losers small" we count the number of nominations in the pile, not the number of ballots. So, for example, a pile of 10 losing ballots each nominating 5 works is worse than a pile of 45 ballots each nominating 1 work. That could theoretically advantage relatively small slates by ensuring they get one work on the ballot when they'd otherwise get none, if lots of genuine voters nominate single works anyway, but I think it's more likely to increase the number of non-slate nominations. And slates aren't the only form of manipulation; this is helpful against attacks like the Scientologist bulk nominations for Hubbard in '87.

#373 ::: J Thomas ::: (view all by) ::: April 17, 2015, 05:49 PM:

@372 Felice

We really want to encourage people to nominate as many works as possible.

I see no workable way to make people nominate more works when they don't want to. A higher limit on the number of nominations they can make would let them nominate more if they wanted to. And if a slate mostly just gets one of it nominations no matter how many it makes, that particular downside of allowing more is blocked.

With fewer works nominated, there's likely to be less overlap between ballots, fewer nominations for the best works, and more power for slates.

Yes, agreed.

I'd recommend that in "making the pile of losers small" we count the number of nominations in the pile, not the number of ballots. So, for example, a pile of 10 losing ballots each nominating 5 works is worse than a pile of 45 ballots each nominating 1 work.

Interesting! That's easy to game, though. Just put down the name you want, and then put down four bad names -- awful writing that can't possibly win. Then your ballot counts just as much as one with 5 sincere nominations.

That could theoretically advantage relatively small slates by ensuring they get one work on the ballot when they'd otherwise get none, if lots of genuine voters nominate single works anyway, but I think it's more likely to increase the number of non-slate nominations. And slates aren't the only form of manipulation; this is helpful against attacks like the Scientologist bulk nominations for Hubbard in '87.

I think it takes simulation or something to get a feel for what we're talking about. Say there are eight slates and each of them has 6% of the nominations. They can't all win. But if we throw them all out then that throws out almost half the ballots. If it reaches that point there may not be any adequate answers.

#374 ::: felice ::: (view all by) ::: April 17, 2015, 06:04 PM:

J Thomas @373: "Interesting! That's easy to game, though. Just put down the name you want, and then put down four bad names -- awful writing that can't possibly win. Then your ballot counts just as much as one with 5 sincere nominations."

True, but at least it makes you think about it; and you have to be sure that other people aren't writing down any of the same four bad names (otherwise there'd be a high chance of ending up with Terry Goodkind on the ballot!), so they have to be obscure bad names; and we can throw out ballots that nominate non-existent works. It would be easier and safer for most people to just nominate other works they do like, and since extra nominations that can't win do no harm, even a small proportion of voters nominating extra works would be a win.

"I think it takes simulation or something to get a feel for what we're talking about. Say there are eight slates and each of them has 6% of the nominations. They can't all win. But if we throw them all out then that throws out almost half the ballots. If it reaches that point there may not be any adequate answers."

If slates make up 48% of nominations, we're screwed anyway.

#375 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 07:43 PM:

@372: Good point. I think that all three proportional systems we're talking about now are better than the "average" proportional system in terms of not motivating a bullet voting strategy; and SDV-PE is the best of the lot. Frankly, I can't imagine a better proportional system in that regard. So I think they'd all do pretty well. It is also worth lifting the limitation of 5 votes per ballot, and even explicitly suggesting to voters that more than 5 votes is good.

#376 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 08:30 PM:

Jameson@375:
It is also worth lifting the limitation of 5 votes per ballot, and even explicitly suggesting to voters that more than 5 votes is good.

I can see why more than 5 nominations better meets the goals we're developing, but I'm not 100% certain I see all the effects of that with SDV-PE. Say I have 10 nominations on my list. Each nomination now gets fewer "points" (to use Cheradenine's proposed terminology) for each work listed (which seems reasonable and fair). Eventually, as nominations are eliminated, I think I can see that you're going to end up in the same place (at the slimmed down list of nominations), but getting to that point might be quite different, correct? Is SDV-PE going to be independent of what happens in the middle rounds of elimination? It's not clear to me that it will -- though it's also not clear to me that it would matter if it wasn't. Can you or Cheradenine give me some guidance as to how that would play out (i.e., the same, different and it matters, different and it doesn't matter, etc.)?

Thanks,
K

#377 ::: Jameson Quinn ::: (view all by) ::: April 17, 2015, 09:01 PM:

@376: Almost certainly the same, except that with more votes overall, the top works would tend to have more votes too, so that slates working with a fixed number of voters would be less effective.

The other difference is that some works which are more "everyone likes it but few really love it" would do slightly better relative to others. I suspect this is a minor effect.

In theory, it is possible for the eliminations to work out differently aside from the above changes, but in practice, I very much doubt anything else would change. Most things would be in about the same place relative to everything else (in both fraction of total points and in number of voters), and so end up eliminated at much the same time.

#378 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 09:26 PM:

@377

Okay, that's the impression I was getting too, but I couldn't convince myself it was right. That's a good thing.

So we could conceivably say in the new system, "Nominate as many works as you like, the more the merrier!" I can see that actually as an attraction to voters, both because it seems more inclusive and because it removes some of the pressure of how many works you should be putting forward. If they can put up as few or as many as they wish, you've got a system that further leaves the choice to the voters -- something I strongly approve of for a "fans' award". One could argue that might even be a strength over the current system.

Of course, since every nomination is a write-in, I can see the extreme cases getting to be a tedious headache for those who have to verify that the write-in nominations are valid. But maybe if that step wasn't applied until further along into the rounds (when the field has been narrowed), it wouldn't be such an issue.

Kilo

#379 ::: felice ::: (view all by) ::: April 17, 2015, 09:58 PM:

felice @286: "as far as I can see, it's not against the current rules, so it could in theory be done next year. Would it be possible for the Business Meeting to vote to make Option 5b an explicit requirement for future awards, and recommend that MidAmeriCon II voluntarily carry it out before the change is ratified? As well as hopefully protecting next year's awards from Puppies, this would give the system a trial run before ratification, allowing for a more informed vote at next year's Business Meeting."

Does anyone have any thoughts on this?

#380 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 10:16 PM:

Felice@379:

As I understand it, a given Worldcon can do any kind of sample election they want, but the -actual- Hugo has to be awarded based on the current constitution, which can't be changed for two years at least. That said, if a system were approved at the business meeting, I think it would be a logical step to test it out at the next Worldcon, even if the results were non-binding and unofficial. That would apply to any system, not just option 5. That said, I strongly suspect no one would undertake the hassle of a test unless a specific system had been formally approved.

Kilo

#381 ::: Cheradenine ::: (view all by) ::: April 17, 2015, 10:45 PM:

Felice @379, Kilo @380: I think the reference is to releasing the names of the top 15 nominees at the halfway point of the nomination period. I suspect the answer is political - would the WorldCon meeting support it? Would it reduce the chance of other reforms passing the meeting? I'll defer those to people with more WorldCon experience than my zero. Trying something - even if I'm not sure it will help - to fight slates in the "gap year" when we can't reform the rules system certainly has some appeal.

#382 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 11:02 PM:

Oh I see, I did misunderstand. So basically the idea is to see if a counter slate is needed or desired, yes?

Hmm, not sure how I feel about counter slates even as a temporary meyasure. It just seems hypocritical. I think for me, anything that moves towards political parties is a bad thing. I realize option 5 doesn't require that, but I suspect it would inevitably lead that way based on what we've discussed here, so I guess I would oppose it.

I'm dreaming of course, but I would like even the puppies to buy into whatever new system was proposed. If it can be shown to be fair to everyone, it would lay their arguments to rest, but ultimately, their actual argument is that "our stuff isn't on the ballot" and nothing else will satisfy no matter what is done. VD had already gone on record as saying if any no award takes any category (i.e., if he doesn't win), he'll make sure that category never sees another Hugo. So I'm not sure there's any reasoning with that. I would hope the sad puppies are more sincere in their beliefs, but who knows.

If I had the time and energy to deal with the inevitable abuse, I'd really like to post on Brad T's site and ask. But I suppose I can guess how that would go.

Kilo

#383 ::: felice ::: (view all by) ::: April 17, 2015, 11:15 PM:

Cheradenine @381: "Felice @379, Kilo @380: I think the reference is to releasing the names of the top 15 nominees at the halfway point of the nomination period."

Yep; the top 15s from the actual nominations, not a separate sample election. And allowing people to update their nominations at any point during the nomination period, in the same way they've been able to update their final votes during the voting period in the past.

http://www.thehugoawards.org/the-voting-system/ says "The full details of the nominations are not released until after the final ballot. This is to prevent the nomination numbers influencing voters in the final ballot. " - Option 5b doesn't involve releasing actual numbers, just a pseudo-longlist, so it shouldn't influence voting on the final ballot (though it will influence what ends up on the final ballot - that being the whole point).

("Pseudo-longlist" because works don't have to appear on that list to get on the final ballot; there could be a surge in nominations for something else due to "What? I can't believe X didn't make the top 15! I'd better nominate it myself")

Keith "Kilo" Watt @382: "Oh I see, I did misunderstand. So basically the idea is to see if a counter slate is needed or desired, yes?"

Not exactly; the idea is to get more nominations for the works that have a chance of winning, so a greater number of people have a say in the makeup of the final ballot. This is particularly important for categories like short story where nominations are spread out so much that we can get as few as three works crossing the 5% threshold to make the ballot. It helps against slates (irrespective of whether the slates put in their nominations before or after the publication of top 15s), but that's not the only advantage.

#384 ::: Keith "Kilo" Watt ::: (view all by) ::: April 17, 2015, 11:30 PM:

Okay I got you now. That's basically what I (and JT, I think) were supporting a while back, I believe. I've become convinced that it doesn't mesh with what most people feel the Hugos should be (which, in my mind, is a key factor), but as a temporary measure? I might be able to see that. I had pretty much written off the next two Hugos anyway, so maybe it's not so bad as a short term thing.

The only problem I see is that, the way I read the constitution, while there may not be anything against posting the top 15, I think that the ability to -change- your nomination would have to be a change to the constitution (which doesn't help).

Kilo

#385 ::: felice ::: (view all by) ::: April 18, 2015, 12:06 AM:

Keith "Kilo" Watt @384: "Okay I got you now. That's basically what I (and JT, I think) were supporting a while back, I believe. I've become convinced that it doesn't mesh with what most people feel the Hugos should be (which, in my mind, is a key factor), but as a temporary measure? I might be able to see that. I had pretty much written off the next two Hugos anyway, so maybe it's not so bad as a short term thing."

It's only next year that's the problem (all going well at the Business Meetings, anyway); changes can be voted for at Sasquan this year, ratified next year, and in place for the 2017 Hugos. Do most people really feel the Hugos should let lots of nominations be wasted on works that have no chance of making the final ballot? 15 is a fairly arbitrary number - is there some other way of determining how many works to list at the halfway point that would work better for you? Eg the larger of everything nominated by at least 5% of voters in the category and the top 15 works nominated by at least 5 people? (That would be a theoretical upper limit of about 100, but probably much lower in practice since a large percentage of first-half nominations would be for works below the threshold or for works supported by significantly more than 5% of voters).

"The only problem I see is that, the way I read the constitution, while there may not be anything against posting the top 15, I think that the ability to -change- your nomination would have to be a change to the constitution"

No, they just don't count as submitted till nominations close; the website stores your current choices, and whatever's in the database at the deadline is what's submitted. There's nothing in the constitution about changing votes on the final ballot either, and that's not a problem, so it shouldn't be an issue for nominations either.

BTW, who here is actually able to attend the next two Business Meetings? Getting to the WorldCons is well out of my budget, unfortunately.

#386 ::: Brad from Sunnyvale ::: (view all by) ::: April 18, 2015, 03:58 AM:

Yes, just to note: For some reason my brain typed "yield" a few times when I meant "wield." It's been a stressful week. (Those who are comic book fans will know why.)

To further clarify terms, when I say independent, I don't mean independent of information out in the world, I mean independent of collusion with other nominators. A "slate" is not a slate unless there is some sort of agreement between two or more people to both vote it to increase its chances. (And yes, the person proposing a slate is not strictly colluding, it is the person who follows through who is colluding, the former person is colluding only if they know of the other. But there is still collusion by the latter in any event.)

This is the thing we want to get rid of, and trying to get rid of accidental slates is one of the problems we face, as well as strategic voting.

My general feeling is:

a) Only a modest percentage of fans will be strategic in their choices.
b) Those who are will do it based on fairly strong signals. However, those come often enough, there are many years when most fans will correctly identify a shoo-in for that year's nomination. (In certain categories you can often predict 3 or 4 of the 5.)

In some cases it will be "that semiprozine is always nominated." In other cases it will be "that book won 3 awards and the Locus poll." Either way, that's a real thing that happens, where you can get a high degree of certainty. Knowing the likely order of nominations is not.

With those givens, examine some simple situations:

1) The shoo-in is your favourite, and a less popular work is your 2nd choice.
2) The less popular work is your 1st choice, the shoo-in is your 2nd.

Consider each of the above systems. In basic approval the course is pretty simple. Name them both.

But in several of the above systems, the correct strategy, if you are trying to "win," is to name only the less popular work. Not always, but too often.

There are strategies in all the systems. What concerns me is how easy they are to understand (they will be done more, and blogs will advocate for them,) how much risk they present, and what reward they offer. Sadly, a number of the proposed approaches do poorly here -- fairly simple plans offer good reward for little risk.


#387 ::: J Thomas ::: (view all by) ::: April 18, 2015, 05:04 AM:

Hmm, not sure how I feel about counter slates even as a temporary meyasure.

I don't see it as a slate.

I'll use the same numbers that I've almost memorized:

In 2013, for novels, there were 1113 nominating ballots. There were a total of 717 votes for winners, and 785 votes for the 10 losers with the most votes. That's 1502 votes that had some kind of chance to win. If I read it right, there were 3837 votes cast. 2335 of the votes cast could not possibly have won. 60%. (There were an average of around 3.5 votes per ballot.) Brad of Sunnyvale is right that in the current system there is no advantage for nominating only once (except that there are four fewer votes for competing works). All of the votes not cast were wasted.

Obviously, people who vote a slate have no votes wasted, they have 5 votes for their candidates.

If the people who have already voted for the ones they thought were the best were to then vote for the ones that could win that they wanted, I don't see how that could be a bad thing.

Suppose the top 15 were announced, and that consisted of 5 from a slate that nobody but the slate wants, and 10 others. Voting for 5 out of 10 is not a slate. Voting for as many as you want out of 10 is not a slate.

If the slate waited until the last minute to make their nominations, then the top 15 would be 15 others and the vote would be diluted more. Doing this would imply that the slate did not expect others to join them. A slate which is among the top 15 could get votes from others. A slate that tries not to do that, is admitting they don't think others would want to vote with them.

#388 ::: J Thomas ::: (view all by) ::: April 18, 2015, 06:15 AM:

@386 Brad from Sunnyvale

1) The shoo-in is your favourite, and a less popular work is your 2nd choice.
2) The less popular work is your 1st choice, the shoo-in is your 2nd.

Consider each of the above systems. In basic approval the course is pretty simple. Name them both.

In basic approval, if you would hate for your second choice to win while your first choice loses, maybe you should vote only for the first choice. A vote for the second choice increases the chance of the outcome you hate.

But in several of the above systems, the correct strategy, if you are trying to "win," is to name only the less popular work. Not always, but too often.

If you like six things but you are only allowed to nominate five, then it might make sense to leave off the choice which is most likely to be nominated anyway. However, only five can win. If you are nominating your last choice -- the one you wouldn't nominate if you were expressing your true feelings -- because you think something you like better doesn't need your vote, what is going on here? Only five can win. You think your sixth choice won't win without your vote, so you are giving it your vote so it has a chance to displace something you rank higher?

Maybe you think your sixth-best-choice can't win, but you want it to be listed as something nominated for? Something in the top 15, maybe? Well, OK.

I'll try again. You think the first choice is so popular it will be nominated regardless. If you vote for it, you will reduce the chance your second choice will also be nominated. So you only vote the second. How does that work?

Oh! Proportionate voting. As long as you have two choices that haven't been eliminated, your vote counts less, for example half. The more popular choice will not be eliminated while the less-popular choice is still there. So you never get more than half-a-vote for the less popular choice.

Yes, there are voting systems like that. Let's not use them.

I'll look at SDV-PE which is the main one I'm interested in at the moment.

Barring ties, if a particular nominee is among the top five for votes, it will win any challenge to be removed. And barring ties, if it is among the top four with the system of proportional voting, it will never be considered for removal.

So apart from special cases coming from ties, the top four will be the ones with the best proportional votes, and the fifth will be the one with the best absolute vote among the remainder.

So if you are sure that one of your choices will have the most votes even without your vote, you can let that one win its place in the fifth slot. By voting for only the other one, you improve its chance to get into one of the four other slots.

That only works when you are sure it has the most votes, though. And mostly when there is only one other you care about. If you have four other nominations then they each start out with 1/5 vote, and leaving off the best would only bring that up to 1/4. But the last one, the one with the best chance, would still only get half a vote.

I could be wrong, but I'm afraid this is one of the ones where that strategy works. Provided you are sure that one of your choices will be the one that gets the most votes, and won't come in second. (As a picky point, any of the four that win on proportional vote could have a higher absolute vote and not get in the way.)

#389 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 08:20 AM:

@386: Let's take the example I used before. Your preferences are ABCDE...etc, your expected outcome without your vote is approximately ETAOINSHRDLU...etc. You are inclined to vote strategically.

Under the current system, your logic would be like this:
A? Of course, it's my favorite, and has a chance to win.
B, C? No, don't want to waste one of my 5 precious votes on something with no chance.
D? OK. It probably won't win but I'll give it a shot.
E? Well, maybe. I mean, it isn't bad, but it can probably win without my vote. Depends on how strategic I'm being, but let's say I don't vote for it.
F, G? See B,C.
H, I? Yes, I hope this wins, and beats the hated O, N, and S.
J, K? See B,C.
L? Yes.
Final ballot: ADHIL (extreme strategy) or ADEHI (reasonable strategy)

Under SDV-PE, here's your logic:
A? I expect this can win without me, but I'm not certain. So I'd probably vote for it. But if I'm highly strategic and feeling lucky, I guess I could risk leaving it off.
B-D: Why not? B and C will be eliminated early but that's no skin off my back for voting for them; D has a bit of a chance.
E: Probably not.
F-I: Why not?
J-L: Pretty much a toss-up.
So, final ballot BCDFGHI (extreme strategy) or ABCDFGHI (moderate strategy)

By any reasonable measure, the current-system ballot is more dishonest than the SDV-PE one. In particular, it includes various kinds of strategy (leaving off candidates for being too weak and for being too strong), while the SDV-PE only leaves off those that are too strong, and even that, probably only if the voter doesn't like them very strongly.

---

@387: Everything you say here is true. Just to play devil's advocate for a moment, though: announcing a "long list" early would mean that early voters would have extra influence. Is that fair?

---

@388: You've explained it pretty well. But you misspeak at one point: "Barring ties, if a particular nominee is among the top five for votes, it will win any challenge to be removed." No. If it is the top for votes among those which are not among the top four for proportional support, it will win any challenge. A slate could easily have all of the top 5 for votes, and still end up with only 1 or 2 slots. (It's clear from the later stuff that you actually understand this, but I just wanted to correct it so others wouldn't read this and get a wrong understanding.)

#390 ::: J Thomas ::: (view all by) ::: April 18, 2015, 09:04 AM:

@389 Jameson Quinn

Your preferences are ABCDE...etc, your expected outcome without your vote is approximately ETAOINSHRDLU.

It's plausible you might know your preferences are ABCD....etc. But the expected outcome without your vote is likely ET? or ETA? or possibly ETAO? or in utterly extreme and unlikely cases ETAOI?

If you can with any certainty predict the fifth spot in the Hugos, you probably have skills that can make a whole lot of money.

With uncertainty your strategy would probably be to leave off E and maybe leave off A. BCDFG would be plausible. Maybe BCDFI if you think I has a chance but it's kind of iffy.

On the other hand, a lot of people prefer to win over having the maximal chance to make a difference. To be reasonably sure you would have one winner, you might do ABCDF or ABCDI. But you might not be sure A will win either. You could lose every single time! The safest is to do ABCDE. Then you're reasonably sure E will win, you're helping out A which might not, and BCD deserve their chance. You could do ABCEI on the assumption that I has a chance while you can't predict D's chance. But if you vote for I which you don't like that much, and later you find out that D almost made it and I didn't, you'll feel bad.

It's just hard to tell. If you wander around blogs etc people will try to persuade you that their favorites are ahead so you'll vote for them. What do you really know? It's a fog.

If you vote for your real favorites you at least know that you did right by them.

#391 ::: J Thomas ::: (view all by) ::: April 18, 2015, 09:14 AM:

@387: Everything you say here is true. Just to play devil's advocate for a moment, though: announcing a "long list" early would mean that early voters would have extra influence. Is that fair?

It depends. If you vote early you're flying blind. You *might* vote for things that do well and so your vote influences people. You might vote for things that do badly and your vote has no influence at all.

Meanwhile the people who vote late get to influence which of the top items win, if they choose to vote that way.

I would like for each voter to get at least two ballots. Vote early, and then vote for different things late. That could be equivalent to letting each voter do acceptance-voting for up to 10 things. Five early and five late. So everybody who bothers to vote early gets influence, and everybody who bothers to vote late gets the other kind of influence.

But it's easy to stretch the rules enough to announce early leaders halfway through. Allowing people to split their ballot and vote on two different occasions might be a bigger stretch. They might be more likely to think they have to vote on a rule change and not do it for 2 years.

#392 ::: J Thomas ::: (view all by) ::: April 18, 2015, 09:20 AM:

"Barring ties, if a particular nominee is among the top five for votes, it will win any challenge to be removed."

I apologize, that was bad proofreading.

The first four slots will be filled by the top four according to SDV. The fifth slot will be filled by the one with the most absolute votes among all the others.

#393 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 09:49 AM:

"If you can with any certainty predict the fifth spot in the Hugos, you probably have skills that can make a whole lot of money."

I think it's actually plausible that you could divide the pool into about 4 tiers, such as the scrabble-point-based division:
E
TAOINSRLU
GD
(everything else)

You'd have no idea if T had a better chance than L, but you could be pretty sure that three-point-and-up scrabble letters like B and C had no chance at all. (And you'd be wrong in some regards — for instance, H has a better chance than you think it does — but there's no way to know where you're wrong.)

Based on that division, the current-system strategic vote is ADGIL or ADILN, and the SDV-PE strategic vote is ABCDFGIJKL. It's clear that the former are more dishonestly strategic.

#394 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 09:55 AM:

Oops, I left out H. The SDV-PE strategic vote is ABCDFGHIJKL

#395 ::: Keith "Kilo" Watt ::: (view all by) ::: April 18, 2015, 10:57 AM:

JT@390:

It's just hard to tell. If you wander around blogs etc people will try to persuade you that their favorites are ahead so you'll vote for them. What do you really know? It's a fog.

I think you've just hit on one of the prime qualifications a system should have to be considered "strategy-resistant". I think what matters is not if there exists a strategy, given perfect knowledge of the order of potential winners. What matters is that the uncertainties in any strategy are so large, that you may as well vote your preference.

Kilo

#396 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 10:59 AM:

So there's been a lot of discussion of option 3, a bit of 4 and 5, and general skepticism to option 1. But that still leaves option 2. It's actually pretty easy to incorporate option 2 into SDV-PE. Just say:

1. No option will be eliminated if it has over X% of voters supporting it and no other non-eliminated candidate, where X is the lowest percent of voters supporting any nominating candidate in that category in the previous year (and thus, X is never less than 5%).

2. For the purposes of elimination, any ties are both eliminated at once, unless that would take the number of candidates below 5, in which case neither is eliminated and the process stops. In this case, any totals separated by under 3 ballots are considered to be ties.

Both of these rules could expand the number of nominations. But neither of them would be likely to be a factor in "best novel", where the burden of reading an extra work would be highest. In general, they might expand the "Short Story" nomination list to 6 or 7 — still quite manageable. And that would be an additional bit of insurance against slates, since the extra nominations would likely be non-slate works.

#397 ::: Keith "Kilo" Watt ::: (view all by) ::: April 18, 2015, 11:38 AM:

Jameson@396:

I see what you're saying, but it seems to me that it adds complexity for not a lot of benefit, doesn't it? The argument could be made that the more changes to the current system that we propose (e.g., now there are more than five nominees on the final ballot), the harder it will be to pass the business meeting. The idea of "nominate 4 and have 6 winners" is apparently gaining traction elsewhere in the blogosphere, apparently, mostly because of it's simplicity, I think. I don't really think that solves the problem, since the ballot would still be dominated by a couple of slates.

I do like the opposite idea, though, of letting voters nominate as many works as they like, with the understanding that doing so means each work gets fewer of their "points" (1/N). That seems an intuitive trade-off that most anyone can understand and agree with.

#398 ::: Kevin Standlee ::: (view all by) ::: April 18, 2015, 01:23 PM:

As Chairman of this year's WSFS Business Meeting, I'm trying to avoid proposing new business or taking a stand on any specific proposal, but I did recently propose something that is not a constitutional change in that it's legal under the current system, but unprecedented and not in keeping with existing tradition. It doesn't necessarily change the finalists, but it does shed some light on the process:

  1. Instead of announcing the five finalists in each category, announce fifteen "semi-finalists" in random order with no indication of who has the most nominations. (For the actual number of semi-finalists, pick a number greater than five; I use fifteen here because there is an existing rule about the top fifteen nominees, that's all.) The Administrator doesn't contact in advance the semi-finalists and doesn't necessarily do exhaustive eligibility checking. Announce that the finalists will be drawn from this list of semi-finalists and that the final ballot will be announced on a specified date.
  2. The Administrator starts contacting the semi-finalists, saying, "If you are selected as a finalist, will you accept your position on the shortlist? If you do not decline by date X, you are assumed to have accepted. This decision cannot be undone after that date." BTW, inasmuch as some nominees turn out to be difficult to contact, having the semi-final list in public helps this step, even as it increases the number of nominees the Administrator has to contact, because you can ask in public for help finding a nominee.
  3. During this same period and in parallel with it, the Admin does eligibility checking. This also is helped by the "crowdsourced" nature of the semi-finalists, because you can bet that people will turn up prior publication and the like during the process.
  4. At the specified date, the Admin announces the five finalists in each category, based on the top 5 semi-finalists who (a) did not decline and (b) appear to be eligible. Note that this means that finalist simply cannot break a news embargo because they don't know whether or not they made the shortlist until it is announced.
  5. The final ballot then continues as usual.

As I said, I don't think this process violates the existing WSFS Constitution, but it is clearly not the traditional process. When I proposed this to a group of people with considerable experience (the SMOFS e-mail list, the secret of which is that it's not secret), the reactions included:

  • The extra workload on the administrator (255 potential finalists instead of 85 including the Campbell) far outweighs the strain of contacting the 85 current finalists and determining eligibility during a very short period between the end of nominations and the finalist announcement. (This from a recent Administrator who has had to do the detail work himself, which I admit that I never had to do during my turn in the barrel.)
  • Such a change, while legal, is so unprecedented that it should not be done without explicit constitutional sanction. This is the "no elephants in mouseholes" principle, in that WSFS rules aren't made in a vacuum but in the context of the traditional way of administering the election.

While I cannot easily dismiss either of the above two arguments, I also think that my proposed procedure would deal with a couple of the known logistical flaws in our existing process that are not part of the issue over how five-highest-pluralities can be dominated by an organized minority bloc.

#399 ::: J Thomas ::: (view all by) ::: April 18, 2015, 01:43 PM:

@398 Kevin Standlee

I like your proposal, for what that's worth.

The extra workload on the administrator (255 potential finalists instead of 85 including the Campbell) far outweighs....

Given the nonsecret and crowdsourcing side, it might not in fact be that much extra workload. It might turn out easier. So if you could get permission for the administrator to do it or not, his choice, then the business meeting would not have to decide about this question of fact. Of course, if the administrator in question thinks he wouldn't want to do it, it probably isn't worth considering further for this year at all. If he does want to do it, that's pretty strong in its favor.

Such a change, while legal, is so unprecedented....

I can't argue with that.

#400 ::: Brad from Sunnyvale ::: (view all by) ::: April 18, 2015, 02:15 PM:

By the way, I do think STV is the least strategic of the proposals. STV (and its cousins) are known to be the least strategic systems, and that's why they are used in the final ballot. I have not had time to study the multi-winner version -- if a work has 40% of the 1st place votes, how is it decided to pick which half of those votes which get eliminated? Randomly, I presume? Or perhaps with one of the happiness algorithms?


Anyway, I am afraid I must not be communicating my view on how strategy gets applied very well. Several keep citing strategies for approval. They exist, but are very weak. People just don't care that much about their 6th place choice and how much they help it. That's by definition -- it's their 6th place choice. Because they don't care that much, it reduces their tolerance for putting any risk on higher placed choices. The most I can ever see is somebody not nominating a 4th or 5th place shoo-in to get their 6th on their ballot.

This is not true of some of the proposals, where you can affect how much you support your 1st place choice. Again, by definition, that is the one you care about. Because you know your intuition about shoo-ins is uncertain, you will only use it to help your #1 or #2 choices in most cases.

So it doesn't matter if you lay out a scenario which shows some mild strategies for Approval, or STV or any other. The key is, how often will we get the combination of factors which pushes a voter to the strategy. I believe it is necessary that:

a) They have strong confidence in their prediction of the shoo-in. (This happens reasonably often in novels, though not every year, and very often in certain categories.)

b) The work they wish to help with strategy is one of their top choices.

There are extreme situations where this would modify your plain approval ballot -- that's not what I think we need to discuss. What is important is, "how likely is it that the reward exceeds the risk, and does so clearly in a member's mind?" That's how much strategy will be applied.

And again, one point I have made which I am surprised I don't see any comment on. I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so. Yes, there is a backup system -- Vox Day lost to No Award in 2014, but that hardly discouraged him in 2015. However, it did keep Ken Liu off the ballot. (Though he is on it now, as a translator, in 2015.) (It may not be fair, but I feel less pain for Seanan McGuire who may also have been nudged off.)


Another note -- while not nearly as important as winning, there are people who pay attention to the total nominee counts published after the awards. Writers want to know how they did. Fans look for up and coming writers. These systems tweak these numbers as well. Only honest and independent fan opinion gives valid numbers there. This year people will look with special interest, in fact, for the "real" nominees.

Kevin#398. I suspect promoters of slates will take more care in future about which works are eligible, and you won't see a repeat of 2015.

#401 ::: nathanbp ::: (view all by) ::: April 18, 2015, 02:48 PM:

Brad@400:
Excess votes in STV are typically divided according to the proportion of voters' second place choices. For example, if the quota is 300, 500 people vote for work A, and their second place choices are evenly split between B, C, D, and E, each of B, C, D, and E gets 50 ((500-300)*(500/125)) votes for the next round. This does mean that in a widely divided field you can get very small fractional votes being added.

And again, one point I have made which I am surprised I don't see any comment on. I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so.
I'm pretty sure there's no voting system that prevents slates from getting at least 1 nominating slot. Our discussions have mostly been focused on preventing a slate from getting all 5. I don't believe there's any way that would work and be perceived as fair to eliminate slates entirely.

Anyway, I am afraid I must not be communicating my view on how strategy gets applied very well. Several keep citing strategies for approval. They exist, but are very weak. People just don't care that much about their 6th place choice and how much they help it. That's by definition -- it's their 6th place choice. Because they don't care that much, it reduces their tolerance for putting any risk on higher placed choices. The most I can ever see is somebody not nominating a 4th or 5th place shoo-in to get their 6th on their ballot.
Not nominating your 4th or 5th place shoo-in to increase the chances that one of your other top 6 choices is nominated seems very similar to the amount of strategy there is in RAV or SDV. I guess for approval this only shows up if you have 6 top choices to choose between, whereas for RAV and SDV it shows up even if you have less than 6 top choices.

#402 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 03:59 PM:

@400:

"STV (and its cousins) are known to be the least strategic systems"

By whom? On what planet? This is really, really not true. There's a whole literature on how many seats have been strategically swayed by vote-management strategy in STV elections. And if by "(and its cousins)" you mean IRV, then I really don't even.

In particular, in the Hugo, a "shielding" strategy in STV would be common, and could lead to very pathological results. The idea is that (in my alphabet example) you'd vote QA rather than A so that if A has a quota to start out with it doesn't use up any of your vote. This strategy could even lead to a popular candidate getting eliminated prematurely.

Then later you say:

"a) They have strong confidence in their prediction of the shoo-in. (This happens reasonably often in novels, though not every year, and very often in certain categories.)

b) The work they wish to help with strategy is one of their top choices."

You're right, these are indeed the conditions for "strategy" to be worthwhile in RAV or SDV-PE. But if, as you say, people don't care much beyond their first few preferences, in that case, the alphabet voter will vote ABCD. This is "strategic" in the sense that they have chosen their cutoff based on their expectations of what others will do; but it is still a perfectly honest ballot. If everybody voted like this, it's possible (though unlikely) that E would fail to be nominated; but in that case, it would be in part precisely because E had already won enough acclaim, sales, and/or prizes and didn't really need a Hugo. Why, then, is it a bad thing if, through perfectly honest voting, the Hugo voters decide to focus on other worthy works?

#403 ::: J Thomas ::: (view all by) ::: April 18, 2015, 04:42 PM:

I found this mildly amusing.

I was putting together a potential nominating system that depended on arranging ballots in patterns. To do that I was using some set theory, I'd take piles of votes and do intersections and unions etc. I noticed that a few votes somehow disappeared, and I couldn't tell where they were going.

It turned out that with that programming language, when you convert a list to a set it throws away all duplicates. A set has no two the same. So by accident -- just because of the rules of math, it replaced every slate (and accidental slate) with a single vote.

I think I see a way to work around that, but for the moment I'll keep it. It's simpler to play with. And I like it.

#404 ::: felice ::: (view all by) ::: April 18, 2015, 05:22 PM:

Kevin Standlee @398: "Such a change, while legal, is so unprecedented that it should not be done without explicit constitutional sanction."

Does that mean it shouldn't be implemented till after it's ratified and formally added to the constitution, is is the approval of one Business Meeting sufficient?

"my proposed procedure would deal with a couple of the known logistical flaws in our existing process that are not part of the issue over how five-highest-pluralities can be dominated by an organized minority bloc."

Possibly not the best time to bring it up, then? It seems likely that the minority bloc issue will generate quite sufficient business to fill up this year's meeting all by itself...

#405 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 05:32 PM:

I didn't mean to imply that STV was the highest-strategy system in general. But I do believe that it would lead to more dishonest strategy than any of the other voting system options under discussion here (except the status quo). You could make an argument the other way. But "Everyone knows STV is low-strategy" is not an auspicious beginning for such an argument.

#406 ::: felice ::: (view all by) ::: April 18, 2015, 05:45 PM:

Jameson Quinn @389: "@387: Everything you say here is true. Just to play devil's advocate for a moment, though: announcing a "long list" early would mean that early voters would have extra influence. Is that fair?"

If everyone knows early voters will have extra influence, and everyone can choose to be an early voter if they want that influence, then yes, perfectly fair.


J Thomas @391: "I would like for each voter to get at least two ballots. Vote early, and then vote for different things late. That could be equivalent to letting each voter do acceptance-voting for up to 10 things. Five early and five late. So everybody who bothers to vote early gets influence, and everybody who bothers to vote late gets the other kind of influence."

That's an unnecessary complication. Just do it the same way final ballot votes are already done; you can edit the content of your form any time you like, and only what's in the database at closing time counts. (Paper form nominations don't have that option, but is there anyone who still has to use paper forms?) No rule change required, and it really should be implemented irrespective of whether or not Option 5b is approved. Let people input their current favourites as soon as nominations open, and edit them if they read anything new during the nomination period that they think's better than their original choices.


Jameson Quinn @393: "Based on that division, the current-system strategic vote is ADGIL or ADILN, and the SDV-PE strategic vote is ABCDFGIJKL. It's clear that the former are more dishonestly strategic."

How much of that is counting system, and how much removing the limit on number of nominees? Would the SDV-PE strategic vote be ABCDF if it was restricted to five choices? (And I have reservations about allowing more than five works per nominator)

#407 ::: J Thomas ::: (view all by) ::: April 18, 2015, 06:24 PM:

@406 felice

J Thomas @391: "I would like for each voter to get at least two ballots. Vote early, and then vote for different things late. That could be equivalent to letting each voter do acceptance-voting for up to 10 things. Five early and five late. So everybody who bothers to vote early gets influence, and everybody who bothers to vote late gets the other kind of influence."

That's an unnecessary complication. Just do it the same way final ballot votes are already done; you can edit the content of your form any time you like, and only what's in the database at closing time counts.

I did not realize that's how it was already.

There were some drawbacks to that which I think are minimized if the extended list is revealed only once. With a similar proposal it might be possible to game the system, to vote for something else to trick people and then change later.

But if the leading nominees are revealed only once, a cabal would need tremendous communication among themselves to even track what they were doing. Complicated plans would fail. And they couldn't trick people very much. Things are either on the list or off the list. Vote for something that otherwise wouldn't be on the list, and then take away the votes later. Big deal. To do that they risk their own stuff not getting on.

It looks at worst harmless.

People who want an accurate look at what people want, would do better to look at the voting before the reveal. Afterward it's contaminated by what other people want. We can save the old opinions, so that's potentially available.

OK, I don't see any harm in letting people unvote things they've already voted, instead of just add new votes. All the things people complained about it before were connected with other parts of the proposal that are gone.

It looks good to me.

#408 ::: TomB ::: (view all by) ::: April 18, 2015, 06:39 PM:

Here are my thoughts so far on the proposed voting systems:

Option 3a: Satisfaction Approval Voting (SAV)

The way "satisfaction" is defined under SAV doesn't match how I feel when I am nominating for the Hugos. If one of my nominations wins, I am thrilled. Under SAV if I made 4 nominations in that category, I am supposed to feel only 25% satisfied. But my definition of satisfaction is not my winning percentage, it is the quality and diversity of the winners. In some categories such as Short Story, I have been thrilled with the final ballot even though none of them were what I nominated.

SAV rewards voters for making fewer nominations per category, because they receive a higher satisfaction score per nominee. I don't think that is a good thing. In the Hugos, voters who make more nominations in a category tend to be more knowledgable about that category (with the glaring exception of the slate voters).

SAV weights additional winners on a ballot just as strongly the first, so its anti-slate effect is weak.

In summary, SAV is not a good fit for the Hugo nominations. It maximizes an arbitrary definition of satisfaction that not what I (and I think other Worldcon members) actually want. It has a negative effect on knowledgeable voters, and it is not very effective against slate voting.

#409 ::: Brad from Sunnyvale ::: (view all by) ::: April 18, 2015, 06:39 PM:

#402 -- Sorry, the planet I was referring to where STV is low in strategy is this thread, and the world of Hugos in general. And I was referring to the broad class of preferential ballots. I don't contend they are the best in a broad sense, but one of their attributes when doing comparison is that preferential ballots reduce the value of strategy in a number of problems.

But I have not done much study of STV for multi-winner, but I am sure like other IRVs it has highly chaotic results from time to time. Chaotic results can engender strategy, though they tend to be "hard to predict" situations which in turn discourage strategy, if you don't know whether your strategy will help or hurt.

I think the classic situation I would describe as likely to cause too much strategic voting is the relatively common case where your first choice is not a shoo-in, and you feel it "needs your support" while other works you like somewhat less need it less.

I believe the goal is that you state your true opinion, which is to list the works you think best qualify for the award, and not to treat it as a contest where things need your support.

I can think of a few recent situations where this was true for me. Works like Super Sad True Love Story, The Quantum Thief, and in DP-Short, a few episodes of Black Mirror (including last year's White Christmas.) I even toyed in these cases with being strategic on Approval voting, but did not. In 2012, it actually would have worked on Approval -- leaving off Leviathan Wakes from my ballot would have given TQT a nomination, but I listed them both.

In RAV and other systems that reduce the effect if other choices when one wins, I would have considered leaving Embassytown offf the ballot to give more to these works of lesser known writers. That would not have made a difference. But a system should not encourage me to do it.

But let's bring this to a close. While I do think the proposed solutions are not adequate, their increase of strategy is just one of their flaws. They also downgrade honest and independent correlated opinion, and they don't keep all the slates off the ballot.

There are ways to keep all the slates off the ballot, such as option 4, which would not work all the time but generally should work. The slates are coordinated but not ultra-coordinated, which is to say I don't they they would easily collude a complex set of ballots to stay under certain types of algorithmic and human radar. Rather, I think the slate supporters simply see the virtue of voting the slate, knowing that they assure a set of choices they generally like a slot, vs. wasting their time with their own particular choices. In their internal survey, the puppies did not have much agreement.

If a "slate" only wishes to try for one nominee, it can succeed against most approaches here, except systems which inherently make everybody pool resources, colluding or not. This would include the open ballot proposals, they can stop even a single collusion candidate. A version of the open ballot, namely a "real runoff" as opposed to the instant runoff, would also destroy the slates.

In "real runoff" where you see the results of the first round and vote again, everybody changes their vote to not waste support on the works that don't have a chance, and support concentrates on your best alternative. IRV also does this, but doesn't give you a chance to step back and see the slate and decide on a more radical reallocation of your votes. In real-runoff, you would get political actions, such as withdrawing support for a work you liked because you see it's in a slate, and giving more support to a work that you otherwise only liked too mildly to rank, but which you would rather see there instead of "those bastards."

Of course, any system which reveals intermediate results is going to be more political.

#410 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 06:40 PM:

@406: "(And I have reservations about allowing more than five works per nominator)"

Why?

#411 ::: P J Evans ::: (view all by) ::: April 18, 2015, 06:54 PM:

410
Why do you think more is better in nominations?

#412 ::: TomB ::: (view all by) ::: April 18, 2015, 07:18 PM:

Option 3b: Proportional Approval Voting (PAV)

PAV fixes most of my concerns with SAV. Its definition of satisfaction more closely matches how I (and I think most people) feel: "Hey, one of my nominees made it!". It does not discriminate against knowledgeable nominators who fill a category. It is more effective against slates because it gives a lower weight to each additional winner on the same ballot.

The downside of PAV is computing it. If I understand the math correctly, given n total nominees for m, roughly n^m combinations need to be considered. (Actually: n * (n -1 ) * (n - 2) * ... * (n - m + 1)).

The upside is the certainty that all possible combinations have been evaluated. Other techniques such as Sequential PAV (aka RAV) are easier because they avoid evaluating all possible combinations. In theory, PAV is better. I don't know if it is enough better in practice to justify doing all that work. My guess is that it isn't worth it.

#413 ::: Jameson Quinn ::: (view all by) ::: April 18, 2015, 07:19 PM:

@411: Because only with more non-slate votes, enough so that good non-slate candidates can get around 20%, can we ensure that slates don't get more than their fair share. Slates can always maximize their overlap, the only way to have that much honest overlap is if the average voter votes more than the current approx 3.5, and that means increasing the ceiling, because there will always be some who vote for few.

#414 ::: felice ::: (view all by) ::: April 18, 2015, 07:35 PM:

Jameson Quinn @410 & 413: "@411: Because only with more non-slate votes, enough so that good non-slate candidates can get around 20%, can we ensure that slates don't get more than their fair share. Slates can always maximize their overlap, the only way to have that much honest overlap is if the average voter votes more than the current approx 3.5, and that means increasing the ceiling, because there will always be some who vote for few."

A slate can nominate 20 works in each category as easily as 5; what would that do to the results? If the current average number of nominations is only 3.5, it seems likely that only a small proportion of genuine voters would nominate more than 5 if they could. This gives more power to a minority, rather than getting more people's opinions taken into account. I think Option 5b is a much more effective way of getting more nominations for good non-slate candidates.

#415 ::: TomB ::: (view all by) ::: April 18, 2015, 07:35 PM:

Option 3c: Reweighted Approval Voting (RAV)
Option 3d: Single Transferable Vote (STV)

This will be brief because I have dress up and go out now.

RAV is basically a better PAV. It looks good, especially with exponential weights because it would be very effective against slate voting and give more voters a chance to get at least one of their choices on the final ballot.

STV might be even better because the vote reallocation means voters are not punished for listing obviously great shoo-in candidates.

#416 ::: Cheradenine ::: (view all by) ::: April 18, 2015, 08:11 PM:

Felice @414: Slates don't benefit in the same way because their individual nominees don't pick up as many more votes from non-slate voters. They might have 20 nominees with 150 nominations apiece, but non-slate candidates would be more likely to pass 150. And under either RAV or SDV, the slate candidates are going to get winnowed down in the same fashion whether there are 5 or 20 of them.

A ballot with more than 5 nominations is not so much more powerful as more flexible. You're expanding the group of candidates you can help win, at the expense of selectivity among that group.

That said, my suspicion is that most people would list 5 or fewer, with a small portion of people listing 6-7 and a very small group listing any more than that. So while the voting system has better theoretical properties with no limit on the number of candidates one can list, I don't think that in practice it will make a huge difference. I'd favor increasing the maximum number of candidates listable, but it's not a hill I'd be inclined to die on.

#417 ::: Cheradenine ::: (view all by) ::: April 18, 2015, 08:13 PM:

It occurs to me that in combination with 5b, expanding the number of candidates would let someone add more popular candidates to their ballot, without sacrificing their original choices. I'm not sure that would have a big effect on results, but it might be psychologically preferable.

#418 ::: Cheradenine ::: (view all by) ::: April 18, 2015, 08:16 PM:

TomB @415: It kind of depends on how dramatic the number of votes for the shoo-in candidate is. If it's less than twice the quota, you'll get more vote transfer with RAV than with STV; if it's higher, the reverse.

#419 ::: TomB ::: (view all by) ::: April 19, 2015, 04:02 AM:

Back from the symphony. It was awesome, but I did manage to think some more about Hugo nomination voting procedures.

Option 3c: Reweighted Approval Voting (RAV)

The downside is it penalizes voting for obviously great shoo-in candidates. This is a dramatic change from the current FPTP system which rewards consensus candidates.

Option 3d: Single Transferable Vote (STV)

The "simplified version here", referenced in the main post, looks to me like RAV with a weight factor of 0 (or 1/∞). That makes it super effective against slates, but it maximizes the problem RAV has with penalizing votes for consensus winners.

The standard version of STV has the nice effect that if everyone votes for the same thing, it wins by a huge margin and there are a lot of leftover votes that are redistributed to the remaining candidates. Therefore there is a reward for backing a consensus winner. But if a candidate just barely wins, there are no leftover votes and there is no reward for backing it. This biases the system towards consensus and against slate candidates.

The downside of STV seems to be that it is only weakly anti-slate, unless the number of "votes" you get is less than the number of nominations you are allowed to make. In which case, try explaining that to the voters.

#420 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 05:11 AM:

@409:

First off: I still disagree with you, but I do want to thank you for backing off from saying that STV is "known to be" less strategic. Now that you're not saying that, what you're saying is reasonable, if in my opinion not quite right.

So, we're basically talking about the situation where you love A and kinda like the shoo-in E. You're right that in this situation, SDV-PE and RAV do encourage voting a bit more strategically than in STV. But the actual outcome is almost certainly the same, and if not, arguably better.

To see why, let's look at how that situation could play out in SDV-PE and STV.

SDV-PE: you're right that a strategic voter might be inclined to leave E off their ballot, voting something like ABCD or even ABCDFGHI. What are the possible outcomes?

1. A and E win: No harm no foul.
2. E wins, A doesn't: Unsuccessful strategy, not a concern.
3. E loses, A wins: If E was such a shoo-in, this seems unlikely. If it does happen, it's because E's support was lower than people thought. And if that's the case, is this outcome such a bad thing? If E's support seemed higher than it actually was, that's probably because it has high sales or has won other awards. In that case, giving A a chance in the sun is worthwhile.
4. E and A lose: again, if this is what the honest ballots lead to, why is this bad?

The only ways strategy can be harmful here, in my opinion, have to do with I:
4b. E and A lose, but I wins, because the voter voted ABCDFGHI: Sure, this is arguably a problem with strategy. It's also extremely unlikely.
1b. E and A win, but I, who could have also won, loses, because the voter voted just ABCD. In this case, whichever other candidate replaced I probably deserved it more. And again, it's very unlikely.

How would STV have come out? Generally speaking, you're right, the voter has no strategic reason to vote ABCD rather than ABCDE. So let's say they vote the latter. What happens if:
1. SDV-PE would have elected A and E: STV probably gives the same result. But in fact, there is some chance that STV eliminates either A or E due to weak first-choice support, even though they clearly have strong support when considering the ballot as a whole. The voter's ballot does nothing to help prevent E from being eliminated here.
SDV
1b. SDV-PE could have elected I with help from this ballot, but didn't. In STV, this ballot is never going to make it to I, even if the voter thought to vote ABCDEFGHI, so STV seems to be doing worse than SDV-PE if anything for this situation.
2. SDV-PE elected E but not A: STV gives same result.
3. SDV-PE elected A but not E: STV probably gives same result. Yes, this ballot included E, where it wouldn't have with SDV; but the vote was used up on A. In theory it is possible that the transfers from A put E over the top; in practice, it seems very unlikely. And again, arguably electing E in this scenario isn't even better than not doing so.
4. SDV-PE elected neither: STV does the same.
4b. SDV-PE elected I but not A or E: STV would probably do the same, though possibly not if the quota was set using a formula that didn't account for the long tail.

So yes, you're right: the voter could plausibly vote "more strategically" under SDV-PE. But if you actually look at the outcome of the election, it is very likely the same as under STV. When the outcomes differ, the SDV-PE outcome is either obviously or arguably a better one. And the SDV-PE "strategic" ballot is probably the semi-honest ABCD; personally, I think that semi-honest "strategy" is barely strategy at all, and certainly not worth worrying about.

The only way this could actually be a problem is if voters under SDV-PE started to recklessly over-apply this strategy, to the point of bullet voting. I find that highly unlikely.

Note that I've focused on SDV-PE here, but the same arguments all apply to RAV. The only difference is that RAV encourages this strategy a bit more strongly than SDV-PE does; but the outcomes of the strategy are still probably the same as the unstrategic STV result, and if not, arguably better. I might be convinced that misguidedly-strategic bullet voting could become significantly more of a problem under RAV than under SDV-PE, but frankly I still doubt it.

#421 ::: TomB ::: (view all by) ::: April 19, 2015, 05:16 AM:

Here is another option. It is an attempt to create a RAV/STV hybrid that has the best characteristics of each. I am not at all an expert at this stuff, so please excuse me if it is reinventing the wheel.

Option 3c-?: Popularity Reweighted Approval Voting (PRAV) — This system first nominates the candidate with the most votes. Then, all ballots featuring that candidate are “reweighted” so that votes on them are worth proportionally less. The weight is nc/n where nc is the number of votes for candidate c and n is the total number of ballots. (Or it could be the total number of non-blank ballots, or the total number of votes cast.)

For example, let's say the first candidate with the most votes is "The Dispossessed" with 1500 votes out of 2000 total. Therefore it is nominated. All the ballots that listed "The Dispossessed" are updated and a weight adjustment of 1500/2000 = .75 is applied to the remaining candidates listed on those ballots. The next candidate with the most votes is "Atlanta Nights" which was pushed by a slate of 400 voters. It gets nominated. All the ballots that listed "Atlanta Nights" are updated and a weight adjustment of 400/2000 = .2 is applied to the remaining candidates listed on those ballots. At this point there are four possible groups of ballots:

- 120 did not list D or AN: Weight is a pristine 1.0
- 1480 listed D: Weight = .75
- 380 listed AN: Weight = .2
- 20 listed both LHoD and AN: Weight = .15

Because the weight adjustment is based on the popularity of the work, slates that push unpopular works get adjusted downwards faster than RAV with exponential weighting, so they are unlikely to succeed at nominating more than one. Because the penalty is less for nominating popular consensus candidates, people can simply vote for what they prefer and don't have to vote strategically.

Computationally I think this is even simpler than regular RAV, because the same weight is applied to all the ballots that list the winning nominee. With regular RAV, the weight has to be determined for each individual ballot.

#422 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 05:24 AM:

@409: Aside from the technical arguments I just made above, I understand that you'd rather focus on options 4 and 5. I personally think 4 (outlawing slates) wouldn't solve the problem; the fallout from disqualifying ballots would be too high especially when both sides can make reasonable arguments that the other side is getting away with it, so the slates would be ever more of a problem. So while I don't oppose mild forms of option 4, I don't think a solution should hinge on it. As for 5 (publishing preliminary results such as a "long list"), it may help, but I still think that it's unwise to rely on this alone without also fixing the voting system (option 3).

Cheers. I apologize if the "what planet" comment was too harsh. When you're not claiming that your side is "known" to be right, your arguments are definitely a help to this conversation; I may disagree with you, but you are clearly thoughtful, reasonable, and sincere.

#423 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 05:32 AM:

@TomB: Interesting, but I think it has various problems.

Basically, you're trying to ensure that the "reweighting" factor depends on the popularity of the work. That is how unranked-STV already works. (Note that people here are using "STV" mostly to refer to ranked-STV, but unranked-STV has been referred to various times in this thread.) Unranked-STV, unlike your proposed system, is guaranteed to preserve proportionality. I suspect your system would deweight too harshly in practice, so that it's intended result of not discouraging votes for a popular candidate would not work, or even come out worse than RAV. Note also that SDV-PE was designed with much the same goal, though with different mechanisms.

#424 ::: J Thomas ::: (view all by) ::: April 19, 2015, 06:12 AM:

@420 Jameson Quinn

3. E loses, A wins: If E was such a shoo-in, this seems unlikely. If it does happen, it's because E's support was lower than people thought. And if that's the case, is this outcome such a bad thing? If E's support seemed higher than it actually was, that's probably because it has high sales or has won other awards. In that case, giving A a chance in the sun is worthwhile.

Is this one a concern?

People who want E think it will win without them, so they don't vote for it. If too many of them use this strategy then E loses even though the strategic voters wanted it to win.

They all thought they could depend on somebody else to do it. They were wrong.

But this is a stupid strategy. If what you want most is ABCDE and you instead vote ABCDF because you think E will win without your vote? You are voting for your sixth choice, hoping it will displace one of your five best choices. That doesn't make sense.

How about this one:

You vote ABCD because you think that if you vote E it makes it more likely that one of ABCD will lose. You want D to win more than you want E to win. The chance that your vote will help E is not worth the chance that it will hurt D (and C and B and A).

Imagine that the vote counting happened entirely in public and you were watching. You voted ABCDE, honestly. You watch while B loses, your second best choice. You gave 1/16 of a vote to help B. Then you watch C lose. You gave C 1/8 of a vote. Then you watch D lose. You gave D 1/4 of a vote. Then A loses. You gave A half a vote. And finally E wins, with your full vote.

Meanwhile your friend is standing beside you. He gave 1/8 of a vote for B, 1/4 of a vote for C, 1/2 of a vote for D, and a full vote for A. Nothing he voted for won, but he might easily feel better about his votes than you do.

You both might feel like you'd rather have the choice to give 1 vote for A, 1/2 vote for B, 1/4 vote for C, 1/8 vote for D, and 1/16 vote for E.

Or maybe 7/8 vote for a, 5/8 vote for B, 3/16 vote for C, 3/16 vote for D, and 1/16 vote for E.

But you don't get to choose. Some sort of mathematical process assigns your vote, and it gives the biggest part to the things that are already the most popular.

I would much prefer that we could honestly say "Your vote goes to the one on your list that *can* win, that needs it the most."

Or get away from the voting concept. "Your ballot will help to provide a well-rounded collection of works where the worst of them was suggested by many fans and the winning suggestions as a whole came from many fans."

If you just plain can't predict how your ballot will fit into the whole thing, then you might as well suggest a full slate of the ones you think are best.

#425 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 09:46 AM:

@424: I understand your point, but you get the details a bit wrong. Here's a corrected version, using SDV. Imagine that the text letter frequency times 4 (because the average voter approves 4 letters) are the raw totals, and the proportional results are as the dictionary letter frequency (second table), except that A has an unexpectedly weak showing — the 6th highest proportional total, below R and N. Thus, clearly there's a slate for THE.

Imagine that the vote counting happened entirely in public and you were watching. You voted ABCDE, honestly. You watch while B, your second best choice, comes up against H, and is eliminated by a crushing margin in raw ballots. You gave 1/5 of a vote to help B, but it's clear that even if you'd given more, H would have beaten it easily. Then you watch C lose to D, even though C had more proportional votes. Even if you'd not voted for D, D would easily have beaten C in raw totals. You could have given C 1/3 of a proportional vote over D, but the fact that that could have helped put C even further ahead of D wouldn't have mattered; in the contest that mattered, the elimination, C would have had your full support. After D squeaks past C and L on raw votes, you watch it lose to O. You gave D 1/3 of a vote in the proportional contest, and full support in the actual elimination. Then, A loses to T. You gave A half a vote, which helped put it ahead of T in proportions, but T had a higher raw vote, which you couldn't have changed even if you'd left E off. The final winners are EISR (proportionally) and T (total ballots). E was never in danger, and didn't really need your vote.

Meanwhile, your friend left E off. Thus, their vote counted 1/4 for B, (1/3 on both sides for C and D), 1/2 for D, and fully for A, in their respective proportional counts. But in none of those cases would changing the proportional count made one whit of difference to how the letter was eliminated, because the key in all cases was the margin. So, you're not upset by your non-strategic vote.

However, your strategic friend says, "Look! The dastardly THERS slate got four of their candidates on the ballot, even though they had only just over 30% of the vote, so they only deserved 1! If the RISEN voters had been as strategic as ABCDE voters like me, maybe we could have made it so the last elimination was between T and E, and so only one of them would have been on the final ballot! (That is, as long as there weren't too many ERIS voters sowing chaos.) I don't regret my strategy one bit!"

(Your friend is, of course, wrong about THERS being a slate. The real slate is just THE, which only got 2 of its candidates on the ballot. But that's still out of proportion to THE's numbers, so the rest of what she says is correct.)

Personally, I think that most Hugo voters would be more like you than like your friend. And even if they were like your friend, the result is only that they (for instance) leave E off and vote ABCD. And in that case, unstrategic STV would have gotten the same result anyway.

....

I would much prefer that we could honestly say "Your vote goes to the one on your list that *can* win, that needs it the most."

But with SDV-PE, we can say that. In the pairwise elimination comparisons, which is what actually counts, your vote does go to each of the works on your list when they need it the most (unless they are up against another work you supported).

#426 ::: J Thomas ::: (view all by) ::: April 19, 2015, 10:28 AM:

@400 Brad from Sunnyvale

And again, one point I have made which I am surprised I don't see any comment on. I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so.

There are few possible voting systems that can prevent this. Particularly if the colluders realize how the voting system works and respond to it.

The first obvious approach would be to throw out every pair of ballots that have all five votes the same. This will not affect regular voters at all, it isn't one in ten thousand that's identical to another. But the slaters would of course retreat to using four.

It would be hard to tell the difference between a slate that all voted for a single work, versus a bunch of people who independently voted for that work.

It would be hard to tell the difference between a slate that dishonestly voted for a single work because their group identity made them do it, versus a bunch of people who believed that single work was the one that deserved the Hugo.

And the argument could be made that some slates deserve a single win. If 10% of the voters want a particular work, nominated, there might not be five other works that get 10%. For 2013, the novel got 10.6%, the novella got 12.4%, but various awards did worse, plus the same slaters can hit smaller awards harder.

If we're going to go after 10% of the voters because of who they are, that's definitely politics.

If we find a voting system that gives us results we like better than what we have, it would make sense to change. It won't entirely solve the problem of slates.

What do you think the goals should be for the nominating survey? Whatever specific goals we choose will suggest an ideal voting system (which might be too cumbersome to use in reality) and we can rate other voting systems by how well they approach the goals.

#427 ::: Jameson Quinn ::: (view all by) ::: April 19, 2015, 01:05 PM:

Brad@400:

I think the core of your point is: "I think even one nominating slot attained through collusion is an abuse of the system, and 2 is very much so." If I understand you correctly, you're not objecting to a faction with 20% of the vote getting one slot, because in a situation where the non-slate voters are highly diverse, there's no voting system that could p prevent that. Nor are you objecting to the puppies becoming voters in the first place; while stricter membership qualifications may be desirable, this is not the thread for that discussion. What you object to, and rightly so, is that in an environment where even the second- and third-place non-slate candidates get around 10-15% at best, a second-rate slate candidate can elbow its way onto the ballot with well under 20% support, just by coordinating votes; and with 30%, a slate can hope for 2 or even 3 slots.

I agree, this is bad. Very bad.

There are several things that could be done to fight this. For instance, option 5 — announcing a preliminary "long list" — could help. But focusing only on voting systems, what's the best we can do?

If that's the question, I can see how you've convinced yourself that the answer is STV. You're worried about an honest ABCDE voter voting ABCD, because that leads to fewer non-slate votes, which lowers the hurdle for the slate voters. And you've figured that that kind of strategy is (generally) viable in systems like SDV-PE, but not in STV. All of that is correct.

But there's one thing you're missing. If there are enough voters who honestly put E fifth, so that their leaving off E would drop E out of the winning set and allow a slate candidate in, then STV won't elect E, even if voters honestly rank her fifth! The whole reason STV doesn't encourage a "leave popular candidates off of the bottom of your ballot" strategy is that it doesn't even see the bottom of your ballot until the higher choices have all been eliminated. So a few extra honest votes for E down at the bottom of people's ballots will not protect E from getting eliminated early on.

So that means that if SDV-PE only causes slightly more "truncation" strategy than STV, then the former will actually be better at fighting slates by finding the honest overlap between non-slate voters. In fact, even if truncation is used whenever it is strategically appropriate, SDV-PE will be as good on average as STV in this regard. It's only if truncation is significantly over-used, that SDV-PE becomes worse than STV.

Is there any reason to think that would be the case? I think, absolutely not! Consider:

- Adding further candidates to a ballot is more of a bother with STV, as you have to make sure to put everything in the right order, rather than just throwing it all out there. So STV could in principle actually lead to less honest overlap, as well as being less able to take advantage of the overlap that exists.

- Because truncation doesn't help with the pairwise comparisons, only with the points, SDV-PE only encourages truncation if you think that you can somehow get a rival eliminated before it comes up against one of your preferred candidates. Pretty much the only way that can happen is if the rival is a slate candidate, and your preferred candidate gets enough extra points to jump past another candidate on the same slate and into the top 4. It's pretty tough to imagine that an extra fraction of a point from your ballot is going to be enough to do that. So probably, truncation will be pretty rare in SDV-PE.

In other words: I think that if the goal is to take advantage of non-slate overlap to help defeat slates, SDV-PE is the best system we've discussed, RAV is pretty close behind, and I'd be very surprised if there were anything significantly better than these.

#428 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 03:30 PM:

My first problem with strategy is it makes the member not say the truth on their ballot, and leaves us the hope that the strategy is not likely to neither backfire (giving us a ballot without a choice that's actually popular) nor work (giving us a ballot where a work with rabid fans displaces a work supported by more, but less strategic fans.)

I think the only way to do that is the minimize the advantage of strategy. If you can make a case that it has little chance of affecting the result, you get a feedback loop, because people are less likely to do it. We can all paint stories of how one strategy attempt might work, another might backfire etc.

The best thing to do would be to get real sets of ballots and run tests of different systems on them, and also different systems in the presence of certain strategies operated by some fraction of the voters. Doing this a lot would give some hard evidence.

As to Option 4 -- I have not outlined here just how you would design a rule against slates, and the rules under which that would be judged. Bruce has requested not to do so at length here. It is not something hugo admins would relish, but then again, they don't relish what's happening now. I think there may be other systems of justice we can just import -- it probably isn't practical to invent a whole fannish one.

I do think the strongest effect of rule 4 might well be deterrence. If there is such a rule, it's hard to do a slate. It's probably not going to work, and in addition, you have to ask all your supporters to waste their ballots (and their membership fees, if that's the only reason they joined the con) on something that won't work. At "best" you can use it to claim victimhood, which they already do.

The systems above are perhaps best described as anti-sweep, not so much generally anti-collusion. They will prevent sweeps. It is a good thing to prevent sweeps, of course, but is it enough?

Anti-sweep does offer a deterrence against a classic slate. The slate might just say, "Here are the 2 slate candidates, be sure to nominate them, and if it doesn't hurt them, add your own."

Now I do agree that getting only a single candidate on is also a harder sell to the conspirators, but more rewarding than just getting grumbling points.

Of course, one of the first things I thought about was to see if you could get self-selection out of the process entirely. If nomination were done by randomly selecting 1,000-2,000 fans from a pool of 30,000 or so you could probably make it very difficult for a conspiracy to be that many of the sample. Though if they got to be 5% or more of the sample space (and thus the sample) they start having the ability to have an effect.

Could we get a sample of 30,000? If the sample were "Anybody who has ever attended any major convention, ever, for whom we still have the valid e-mail?" (Yeah, that has issues, like people who attended under different names getting 2 samples, and the fact that people will think it's unfair that only the first John Smith to respond gets to nominate, since it's way too expensive to try to tell them apart.)

I wonder what the sample size is for "People who have in the past, received a hugo voting PIN at any time?" Another alternative -- a smaller space -- is people who have ever voted in a past Hugos.

No supporting memberships clearly, that's highly self-selected. Sorry to those who never attend but regularly support. You are great fans but you are the definition of self-selected.

#429 ::: felice ::: (view all by) ::: April 19, 2015, 04:16 PM:

Brad from Sunnyvale @428: "If nomination were done by randomly selecting 1,000-2,000 fans"

What proportion of fans actually read a lot of what's published as soon as it comes out? I for one usually wait till things are out in mass market paperback, and a lot of my reading is catching up with older work I haven't read yet. And as for shorter fiction, the voter packet is virtually all I read some years. I think I'm a good voter, but I'd be a lousy nominator (in the absence of 5b). Self-selection of nominators isn't a bad thing.

#430 ::: JonW ::: (view all by) ::: April 19, 2015, 04:31 PM:

#428 - If Bruce has asked us not to discuss the design of a rule against slates, I've missed that. Bruce has asked us not to discuss options other than the five set out above, but a rule banning slates is one of those options. In my view, no version of a rule banning slates would be workable. It would either unduly trammel the ability of fans to communicate with each other (as in the rule "no fan may use the Web to bring SF works to the attention of other fans during Hugo nominating season"), or would be easy to evade (as in the rule "nobody may explicitly urge others to nominate particular works"), or would give too much discretion to administrators. If you have a version of a rule against slates that you think actually would work, you should suggest it.

#431 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 05:15 PM:

Ah. It depends what you mean by a rule. Unlike the mathematical formulations above, which are precise, rules are written in English and are expected to have interpretations, and humans make judgments. The rules have a spirit, as well as a letter, and the adjudicators can be permitted to enforce that spirit. And also told to do it very rarely.

You can also try to define the rules strictly. That permits loopholes. People find bugs in the rules and exploit them. This particular "bug" wasn't secret, everybody knew you could collude and take over the Hugos, it just wasn't felt anybody would be so slimy as to do so. That approach failed.

Here are a couple of quick possibles ways to structure the fannish inquisition.

a) A simple court of say 5, elected in some proportional system by fans. If puppies are 20% of fans they probably get a member. The court might be required to be unanimous, 4-1 or just 3-2.

b) A larger delegate group, perhaps of 50 or more, and a supermajority needed.

It might be controversial, but it could be the right structure is to give the group very broad powers, with everything made public at the end. Their duty: "If something has gone really wrong, fix it." Possibly as vague as that.

Possibly not as vague as that too. You could name specific wrongs -- collusion, voting by non-natural persons (already not allowed), buying memberships for others with an expectation of how they will vote. The more specific you are, the more possible it is a loophole can be found, but that might be OK -- it's not like we don't know what most of the loopholes are.

Now I know there will be debate about one issue. What's the status of holding a fancy semi-closed party at the worldcon and giving gifts and food to some members and not others to promote publications? There definitely are writers and publication houses that spend a lot of effort and money to build themselves up and become well known and liked in the community, and it does give them an advantage in voting, I would say. Not that much of one, and I doubt these parties are done with that motive -- but marketing does work, we are more likely to read and nominate books that have been well promoted to us, though one hopes most of us are not swayed in the final ballot.

I think we can come to a reasonable conclusion about whether that's OK, and it might be a different conclusion from whether people agreeing to vote the same to game the system is OK.

My primary role -- speech vs. action -- is blurry here, since giving away free drinks is an action, and saying on your blog, "here's my slate, I ask my supporters to vote for it" is speech.

(What's not speech is the supporter agreeing to that and voting the slate.)

But even so, I think it can be worked out. I would hope the fannish inquisition would be called upon almost never, and only would awaken if things are clearly very wrong. Perhaps it could not even meet unless 66% of fans asked it to meet.

In other words, one should not expect the fannish inquisition.


Unless you are planning something nasty, as it should deter you.

#432 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 05:27 PM:

Ok, writing off the cuff, let me outline a more specific system.

a) At any time, from before nominations to after the awards, any fan may propose a problem they see happening on the con web site.

b) Fans may, with their hugo pins, indicate they wish to see an inquisition into the matter proposed.

c) If any matter gets the support of X% of fans within a certain time frame, an inquisition is called on the matter. (How the members of the fannish inquisition are chosen is another issue. They might be pre-elected, or only elected then.) Two proposals may overtly declared to overlap -- only the one with greatest support is acted on.

d) The inquisition, of 5 members, may by 4-1 vote or better do just about anything related to the problem for which they were summoned. After discharging their duties, members are not eligible for another inquisition for some amount of time (a couple of years at least.)

e) Fans may invoke another inquisition (with different members) to appeal. But this should be extremely rare.

Now, getting 50% of fans would be a pretty major event. I don't believe 50% of the members of a worldcon have ever nominated or voted or attended a single event or any other fanac. 50% might even be too high, though it conveys a democratic fairness than any minority number would not.

I do suspect, that in this year, there would have been a call for an inquisition even before nominations closed but it would not have broken 50%. However, I think it might have after nominations were disclosed. I think it would have met, and corrected the ballot to remove the slate votes in a fair way, and the Hugos would have gone on. I think the puppies would have called for an appeal, and not gotten 50% for it.

Modest bit of work to code this, but not that large.

#433 ::: felice ::: (view all by) ::: April 19, 2015, 05:49 PM:

Brad from Sunnyvale @431: "Perhaps it could not even meet unless 66% of fans asked it to meet."

As a simpler alternative that doesn't require election of delegates, how about empowering the Hugo subcommittee to take actions not otherwise specified in the constitution, if 66% of voters endorse their proposed action? (With a "quorum" of 50% of the WorldCon membership?) The Hugo subcommittee could potentially be aware of a problem before everyone else, and in case like this, propose a special anti-slate measure before announcing the shortlist, describing the nature of the problem without specifying which works are involved. Other people could of course draw their attention to a problem if there's an issue that they aren't in the best place to spot first.

#434 ::: Keith "Kilo" Watt ::: (view all by) ::: April 19, 2015, 05:59 PM:

Brad@431:
Here are a couple of quick possibles ways to structure the fannish inquisition.

Heh, okay, that one got a chuckle out of me... Nice one. And the best thing about this type of rule is that No One Will Expect it!!!

Kilo

#435 ::: Joe in Australia ::: (view all by) ::: April 19, 2015, 10:25 PM:

I think "punishing" slates is too risky: the consequences of getting this wrong would be awful and delegitimise the vote. Also, how do you deal with non-malicious things that *resemble* slates, like nominations of local authors when the Worldcon is held outside North America? The problem of slates is that they exclude non-slate works; if we can't block slates the solution must be to give voters a more useful and genuine choice by including additional works.

I propose keeping the present nomination system, but having a final ballot that can be adjusted in length when necessary. The problem of reading all these nominated works will only be an issue when slates have distorted the results: the ballot will otherwise be at or around its present length. I suppose if that's a problem for anyone they can just ignore the works that were artificially thrust forward.

I favour the present nomination system because it's less burdensome for electoral officers: a ballot that's twice as long would take twice as long to count, but this can be done gradually as votes are received. In contrast, you can't start calculating some re-weighted voting systems (e.g., STV) until all votes are received. If people are impatient (if!) this means a bunch of people all working at the same time, which is harder to organise. Re-weighted voting systems are also trickier and more error-prone than simply enumerating the number of nominations for each work separately.

I think the easiest way to determine the length of the final ballot is to show people the ranked nominations and have an additional vote to determine the number of works they want on the final ballot. If you take the median response then the answer is effectively impossible to manipulate.

I understand that some people think that revealing the nominations will bias the voting process. I don't know whether we should make a big deal of this problem, but I would happily settle for alternatives like leaving it to the ConCom's discretion or using a formula to determine the length of the ballot. My only caveat is that any formula can be attacked: a smart attacker would design their slate(s) to minimise the number of alternative works that get included.

#436 ::: felice ::: (view all by) ::: April 19, 2015, 11:23 PM:

Another advantage to 5b - spotting ineligible works early, and giving potential nominees the opportunity to decline early, reducing the chance of the administrators having to deal with last-minute or after-announcement withdrawals. If a work in the top 15s is withdrawn for whatever reason, recalculate the results (by the same method that the final ballot would be recalculated with in the event of a late withdrawal) and publish the revised top 15s. Recalculation should only use the original half-way point data, not anything more recent, to prevent gaming the system.

#437 ::: Brad from Sunnyvale ::: (view all by) ::: April 19, 2015, 11:48 PM:

There is some merit to what you suggest -- allowing the Hugo committee to, if and only if they rule there has been attempted manipulation of the nominations, to expand the number of nominations so that exactly 5 nominations ruled as unlikely to be due to manipulation are on the ballot. They would not disclose this until after the ceremony. But if there are more than 6 it will be obvious what they did, but you would not learn from them which works they ruled against.

This works because the online community will certainly have a good idea of what the slates are, because it's very hard to run a slate in secret. As such, many fans may decide that they can not bother to read the works they regard as from slates but still be acting fairly, and so the burden on the fan to read the whole ballot is not expanded by this expansion.

The only negative is that the slate voters do get to call themselves Hugo Nominees, and if you get multiple slates it could get a bit crazy. Sadly, even though it would not do much good for them, I could see the puppies (and others) continue to run slates in order to provide publicity (good or bad) for their works and cause, and to be able to call themselves Hugo finalists.

However, it has a lot to recommended it because the committee's actions can not truly harm anybody, not with a preferential ballot. Having extra nominees is only harmful because it puts a burden on the honest fan who wants to read them all and be fair in judgment. It is "harmful" in that you might lose your Hugo to a work that, once fans saw it on the ballot, they realized was better than yours. That's not harm at all.

As for things that "resemble" slates, I don't see that as too likely. A slate is a large conspiracy. You can't keep it secret. It will almost always involve public declaration of the slate, and if it's done in private, all it takes is one person who got it to make it public or forward it to the Hugo admins.

Most of my proposals for a fannish inquisition have had it do all the work, most of the proposals here have involved complex rule changes. The right answer may well be a combination -- minor rule changes that give the Hugo admins the tools to fix the problem with minimal harm.

At least until the attackers come up with something that was not anticipated by those rules -- thus my idea of 50% of fans being able to invoke a Fannish Inquisition that can fix almost anything.

#438 ::: J Thomas ::: (view all by) ::: April 20, 2015, 07:05 AM:

A slate is a large conspiracy. You can't keep it secret. It will almost always involve public declaration of the slate, and if it's done in private, all it takes is one person who got it to make it public or forward it to the Hugo admins.

Can that be gamed?

Somebody sends the Hugo committee a document they claim somebody sent them, with two names they recommend voting for. They say this list was sent to a lot of people. Both names show up on a lot of early ballots, two popular works that appeal to a certain sort of person, the sort of social undesirable that might want to do a slate. It looks like one or both of them might place on the final ballot.

Maybe two people send them the same document.

So they don't accuse anybody of anything, they instead add one or two slots to the short list.

Adding slots doesn't help a slate very much, if it only had one or two on the short list in the first place. Say you started with two slate nominations and three others. Instead you have two slate nominations and five others. The slate still gets all its votes for either the one item the slate has agreed to vote for, or for two. Everybody else (who presumably dislike the slate nominations) have five others to split their votes among instead of three.

It isn't a big help to a slate, but maybe enough that members of an actual slate might want to reveal it to get that result.

The good side is the slate doesn't get crowd out other works from the short list. But you might have other ways to prevent that.

Would anybody try to game it the other way? Create the illusion of a slate to get more things on the ballot? That doesn't seem plausible to me at first thought. It's a big effort that could possibly backfire, and the result is you get 2 extra chances to be nominated, but there's no reason to think the one you want would come in 6th or 7th. The votes are split among more choices which could help or hurt a particular works' chance of winning. I don't see how it does much good to fool the Hugo Committee into believing in fake slates.

#439 ::: Brad from Sunnyvale ::: (view all by) ::: April 20, 2015, 10:57 AM:

If the slate is not obvious, I doubt the committee would act. And they would be on the lookout for fakes.

Now, if there are more than 5 slate candidates, the committee need only put 10 (the top 5 slates and the top 5 non slates) but I hope that would not happen.

#440 ::: JonW ::: (view all by) ::: April 20, 2015, 11:00 AM:

Brad @# 431 & 432,

As I understand you, you’re answering the “what would be forbidden” under a rule against slates and “what would Worldcon be authorized to do in response” questions with “anything the Inquisition thinks is Really Bad” and “anything the Inquisition wants.” This strikes me as a terrible idea.
First, if we’re giving Worldcon authority to act arbitrarily against certain of its members just because folks think (on a non-rule-constrained, I-know-it-when-I-see-it basis) that the members have done Bad Stuff, then those guys will complain that they’ve been treated arbitrarily, and they’ll be right. If we want people to feel that they’re being treated fairly, we need to tell them the rules in advance rather than telling them that there are no rules, but they’re at risk whenever the Inquisition decides after the fact that they’re done wrong.
Second, it’s only human nature that the Inquisition will be inclined to cut slack to well-respected members of the community, and disinclined to do so for jerks like the Puppies. So we’ll be moving from a system that at least has the virtue of rule-bound fairness (something the vote-tallying proposals now on the table would retain) to one that would actually provide basis for the Puppies’ complaints about the in-group excluding and arbitrarily impeding outsiders.
Finally, if it’s unclear what sort of stuff will get Inquisitory attention, a lot of people will be inclined to steer clear of those lines. But that’s not an unambiguously good thing, because what they’ll be inclined to steer clear of is promoting books, which is an ill-defined subset of talking about books – and talking about books is a feature, not a bug. So if people have reason to worry about being Inquisitioned for recommending books for Hugo consideration, we want to be really thoughtful about what we're chilling them from doing.

#441 ::: JonW ::: (view all by) ::: April 20, 2015, 11:02 AM:

Brad @# 431 & 432,

As I understand you, you’re answering the “what would be forbidden” under a rule against slates and “what would Worldcon be authorized to do in response” questions with “anything the Inquisition thinks is Really Bad” and “anything the Inquisition wants.” This strikes me as a terrible idea.
First, if we’re giving Worldcon authority to act arbitrarily against certain of its members just because folks think (on a non-rule-constrained, I-know-it-when-I-see-it basis) that the members have done Bad Stuff, then those guys will complain that they’ve been treated arbitrarily, and they’ll be right. If we want people to feel that they’re being treated fairly, we need to tell them the rules in advance rather than telling them that there are no rules but they’re at risk whenever the Inquisition decides after the fact that they’re done wrong.
Second, it’s only human nature that the Inquisition will be inclined to cut slack to well-respected members of the community, and disinclined to do so for jerks like the Puppies. So we’ll be moving from a system that at least has the virtue of rule-bound fairness (something the vote-tallying proposals now on the table would retain) to one that would actually provide basis for the Puppies’ complaints about the in-group excluding and arbitrarily impeding outsiders.
Finally, if it’s unclear what sort of stuff will get Inquisitory attention, a lot of people will be inclined to steer clear of those lines. But that’s not an unambiguously good thing, because what they’ll be inclined to steer clear of is promoting books, which is an ill-defined subset of talking about books – and talking about books is a feature, not a bug. So if people have reason to worry about being Inquisitioned for recommending books for Hugo consideration, we want to be really thoughtful about what we're chilling them from doing.

#442 ::: JonW ::: (view all by) ::: April 20, 2015, 11:03 AM:

Damn. Internal Server Error strikes again. Sorry.

#443 ::: J Thomas ::: (view all by) ::: April 20, 2015, 12:04 PM:

@441 JonW

Everything you say here is reasonablem but let me briefly take the devil's advocate position.

When we are at war we must assume that any rules we impose on ourselves will be used by the enemy in their attempts to hurt us. We are the good guys and they are the bad guys, and we must not have any rules of war against bad guys.

Since we are the good guys and they are the bad guys, anything we do to them is justified while nothing they do to us is justified.

They will complain that we are treating them arbitrarily, but they will do that regardless so what rewards do we get to do otherwise?

Anyway, if you are in a position of authority and you treat your friends no better than your enemies, soon you will have no friends.

Taking the bigger picture, there are two sides here, Sad Puppies and Social Justice Warriors. Neither wants to tolerate the other. They want a war. If you try to come between them and make their war less likely, both of them will consider you an enemy. But if you take one side and push for war, the others on your side will like you a lot and the other side will develop a grudging respect for you.

The more the hate builds up the better your position. Some day in the distant future your side may win and then you will get fading appreciation for old victories, but while it rages, the more it rages the more important you will be.

The SP side has done something outrageous to get their enemies arrayed against them. Now it's our turn to do something outrageous to help their side get stronger, so it can turn into a bigger and more glorious conflict.

Why would you get in the way of all that? Is this the hill you want to die on?

#444 ::: JonW ::: (view all by) ::: April 20, 2015, 12:14 PM:

J Thomas @#443,
I don’t think you actually believe any of that, and I think you and I can agree that none of the sentiments expressed in your post are a path down which we want to go.

#445 ::: albatross ::: (view all by) ::: April 20, 2015, 12:15 PM:

Let me be the first to announce my definitely-not-a-slate list of SFF works which I heartily recommend reading and nominating if you find them worthwhile. With exactly the same words on my part, this will or won't be a slate depending on:

a. What various nominators do with my suggestions
b. What you started out expecting of me w.r.t. slate voting
c. How much you dislike my choices or politics or whatever else.

I think the natural points to get to w.r.t. an explicit rule against slates are either:

a. Nobody has a slate, because everyone carefully puts the right not-a-slate wording in their webpages. The SPs and RPs and various other Ps now have hearty reading recommendations rather than a slate. Nothing changes.

b. Deciding who has a slate turns on your judgment of who's basically up to no good and who's not. The resulting judgment calls are probably almost impossible to do without pissing off a big chunk of fandom, and they will inevitably be based partly on the prior opinions of the people making the judgments. People on the other end of those judgments will basically never be convinced they weren't just attempts to fix the rules to prevent the wrong kind of works being nominated.

#446 ::: J Thomas ::: (view all by) ::: April 20, 2015, 12:25 PM:

@444 JonW

Agreed, I was taking a devil's advocate position.

However, I think that what I said is definitely a big influence in a lot of people's back-brains.

There isn't a lot of wiggle-room between trying to live in peace and trying to start a war. That middle ground is real unstabl.

Holding to that ground, I say that I don't want a minority slate to get all or most of the Hugo nominations just because it's better-organized. On the other hand if it's 10%-20% of the ballots, it probably does deserve one win. If we decide it can't have any because it's a slate, that is going to seem way unfair to 10% to 20% of the voters and all their friends.

A combination of a better voting system and a mid-time announcement of the ones that have a chance, is probably enough to get that result on average. It is a move against the SPs, but it is not an intemperate move against them -- they got more than they deserve and we scale it back to what they deserve.

I don't think we need more at this point.

#447 ::: Keith "Kilo" Watt ::: (view all by) ::: April 20, 2015, 11:32 PM:

Hi all -

I've just finished coding a simulator for the SDV-PE nomination system. The code will read in a text file of works (called, creatively enough, "works.txt"), one per line. My sample just has the alphabet:
A
B
C
D
E
F
G
H

...etc., but you can use actual titles if you are using historical Hugo data. It's not necessary that all of the works actually get any nominations (the code will eliminate them in the first round, since they won't get any points).

The code will then read if a comma-delimited text file of nomination ballots (again, stretching my creativity here, called "ballots.txt"). Right now I'm assuming there are five nominations per ballot, but the code can (in theory) handle any number of nominations up to whatever maximum we decide. Any blanks are just listed as "none" (any invalid nominations -- i.e., nominations not appearing on the works list -- will automatically be changed to "none", so in practice it shouldn't matter what the blanks are called).

So, for example, I ran the following:

A,B,C,D,E
A,B,C,D,E
A,B,C,D,E
A,B,C,D,E
A,B,C,D,E
A,F,G,H,I
B,G,H,I,J
C,H,I,J,K
D,I,J,K,none
E,J,K,none,none
K,none,none,none,none

This is obviously an extreme case with half the ballots coming from a slate. Interestingly, the slate works ended up getting the least number of points, but since they all had six nominations, they were all tied for elimination in what would end up as the last round. Since removing them all would put the ballot at less than five works, the process was stopped and the remaining works were put on the final ballot (we had decided, you'll recall, to eliminate all ties, unless that would reduce the final ballot to less than five works). The winners were:

A,B,C,D,E,I,K

If any one of the slate votes had not gotten an "external" nomination, they would have been eliminated.

So! I need your help. I need to test the code. Are there any situations that you would like me to run? Just send me the ballots in the format above (I'll assume you're using A->Z for the works unless you say otherwise). I'll run the code and post the results (including the round-by-round results, if desired), and maybe our experts can see if the results look reasonable. In particular, Cheradenine, if we could run it against the cases you've already tested, that would give us some confidence that the code works. If it looks good, we can in theory thoroughly investigate the properties of SDV-PE with some hard numbers. Here's a chance to test any situation that's been bothering you!

There are undoubtedly bugs (there always are!), but hopefully with some test data we can squash them quickly.

Thanks in advance for the help!

Kilo

#448 ::: P J Evans ::: (view all by) ::: April 21, 2015, 12:02 AM:

447
I did something similar, but I had the file tell it how many nominees it would be reading in that category (so that input file had category name, nominee count, list of nominees - I believe in letting the computer do as much of the work as possible).

#449 ::: Keith "Kilo" Watt ::: (view all by) ::: April 21, 2015, 12:20 AM:

PJ@448:

I agree, let the computer do the work! The code reads in the files line-by-line, resizing the dynamic array the data is stored in as needed. The total number of entries is just the length of the array. I don't see any reason why the code as written couldn't handle any number of nominations, I just haven't tested it. I love it if you could post your input data so I can see if I get the same thing you do.

I just added a formatted "results.txt" file that contains the round-by-round results, as well.

Kilo

#450 ::: J Thomas ::: (view all by) ::: April 21, 2015, 01:20 AM:

Here's a preliminary attempt at a routine to generate ballots, in python 2.75.

It assumes that the ballots fit a modified zero-truncated poisson distribution.

Here is my reasoning: Assume that votes are not correlated, that just because a fan likes one thing has no influence over whether he also likes something else.

Then according to how popular things are, there is some probability he will pick work #1, and if he doesn't do that, there is some probability he will pick work #2. If he doesn't pick either of the first two there is some probability he will pick work #3, and so on. But at some point it gets too ridiculous and if he hasn't picked one of the top N he'll leave a line blank rather than pick the N+1st one.

If those probabilities were all the same, it would be a zero-truncated poisson. But I figure as we go from more to less popular, the probabilities should get lower. The less popular something is, the less likely it will be what he picks if he doesn't pick anything *more* popular.

So I modeled that, and found parameters that seemed to vaguely fit the 2013 novel nominations. It isn't done right because I thought about independent votes and then actually worked on ballots with up to 5 votes. But it sort of vaguely fits the shape of the curve.

It isn't bad for 20 minutes effort.

If you use this to generate random ballots that reflect the votes that aren't slates, you'll get something that's kind of like the real ones and you can have as many of them as you like while you look for unexpected results and aberrations.

You can't use this to estimate probabilities of things happening with real Hugo nominations, but if something happens pretty often with this, it's likely to be fairly common in reality too.

http://notes.io/YJ8

#451 ::: Jameson Quinn ::: (view all by) ::: April 21, 2015, 07:53 AM:

@450 What you're looking for is an Indian Buffet Process. I have an idea for a heirarchical buffet that is to a buffet as a buffet is to a Chinese Restaurant, but it doesn't fit in this margin.

#452 ::: David Harmon ::: (view all by) ::: April 21, 2015, 08:11 AM:

Popping in late and responding to the last few dozen messages:

The basic problem with trying to set up the "fannish inquisition", is legitimizing it. ISTM that merely getting 50% of "fans" (that is, Worldcon registrants) -- even 30%, perhaps 20% -- to respond promptly to a call for action, is itself implausible. There are times when we're tempted to think of "The Internet" as a single big room, where we can yell and get everyone's attention. But it's nothing of the sort -- not only does a broadcast depend on an assortment of relays and pass-alongs, but not everyone is "on the net" all that regularly, and they won't necessarily be watching the Hugo hotline for developments. E-mail itself is an increasingly secondary medium (thanks, spammers), and even roping in other channels... well, a lot of folks still live basically all of their lives out in "meatspace", only occasionally poking their noses into the 'nets.

On a separate line, any committee is necessarily composed of humans, and thus any attempt to judge the motivation of voters will be subject to bias. Accordingly, the committee and the system itself really need to be limited to reponding to behavior, as opposed to motivation -- and preferably, limited to behavior within the system. That's going to lead to a lot of "limited restraints", where bad actors can still get something, but only so much.

As a simple example, (and this may well have been covered before, apologies if so) the previously-suggested rule that one person can only win nomination in 1 or 2 slots (with corporate entities also blocking their officers and vice versa) would by itself have squelched a fair chunk of the problems in this year's ballot. Not all of them, but part, and that's something. The problem of one author's multiple works splitting the vote, might be handled by allowing "Best Short Story", and perhaps other "Best <works>", to pool same-type works by a common author, automatically shifting from a "single-work" to a "personal/plural" award.

#453 ::: J Thomas ::: (view all by) ::: April 21, 2015, 09:50 AM:

@451 Jameson Quinn

What you're looking for is an Indian Buffet Process. I have an idea for a heirarchical buffet that is to a buffet as a buffet is to a Chinese Restaurant, but it doesn't fit in this margin.

Model first, categorize models later.

What I have is quick enough at our sample sizes that I don't need to use theory to find something more efficient. It can stand in for a better model until I need a better model.

The flaw I found in my model is that if the average nominator has a 1/6 chance to choose the most popular work on his first vote, and he doesn't, then the chance he will choose the next-most-popular vote is less, let's call it 1/8. But if he does choose the most popular on his first chance, then the chance he picks the 2nd-most-popular work on his second vote goes up to 1/6, and if he doesn't choose that then the chance for the third-most-popular goes up to 1/8, etc.

I had no theoretical basis for choosing how fast the 1/6 1/8 etc rate goes down, and I had already adjusted it to look pretty good by eyeball before I noticed the flaw which I then did not fix.

It occurs to me that it might be faster to fix it than to explain why I didn't, but then again it might not. If something went wrong and I spent 2 days solving some fascinating problem that doesn't help me achieve my goals, I'd be 2 days behind.

I figure, get it good enough to use and use it for the next step. If I then find a bad assumption that means I can't possibly do what I wanted to, or if the results come out completely boring, I haven't wasted as much time getting set up. If it's worth getting right (and particularly if it gets funding), then go back and make sure about all the details.

#454 ::: Jameson Quinn ::: (view all by) ::: April 21, 2015, 12:30 PM:

Don't reinvent the wheel, at least not before you've looked at prior wheels: https://cocosci.berkeley.edu/tom/papers/indianbuffet.pdf

#455 ::: J Thomas ::: (view all by) ::: April 21, 2015, 12:49 PM:

When you're doing simple preliminary work, use whatever works. The real world is likely not to exactly fit prior work, and it might be worth shoe-horning it into place -- later.

#456 ::: Steve Halter ::: (view all by) ::: April 22, 2015, 04:11 PM:

Jameson Quinn@454:Interesting paper--thanks. I'll have to mull that a bit.

#457 ::: Keith "Kilo" Watt ::: (view all by) ::: April 22, 2015, 10:38 PM:

Okay gang, we seem to have settled down to a nomination system, since discussion seems to have died off. I'd like to propose a draft language to be presented at the business meeting. It includes FAQ's and commentary, so it's a bit long -- I'll post it in the next message. One thing: I suggest we change the name of the system to "Single Divisible Vote with Least Popular Eliminated" or SDV-LPE. I do understand "Popularity Elimination" is intended to mean the same thing, but on first reading it sounds like the -most- popular work is being eliminated. I think the clearer we make things, the easier it will go.

Please keep in mind this is very rough draft with items pulled from this thread and the original thread, then edited into a cohesive whole. Hopefully it will serve as a start for discussion.

Thanks,
Kilo

#458 ::: Keith "Kilo" Watt ::: (view all by) ::: April 22, 2015, 10:45 PM:

A Proposed Revision to the Hugo Award Nomination System

The Worldcon Constitution shall be modified as follows:

3.3.1 The guiding principles of the Hugo Award are as follows:

3.3.1.1 The Hugo Award should reflect the opinion of Worldcon members that the winners are the most-enjoyed works of the previous year.

3.3.1.2 The Hugo nominations are a survey, to learn from fans what they think are the great works of the year that should be contenders for the Hugo award.

3.3.1.2 All Hugo voters should have an equal opportunity to suggest great works. No minority should be able to absolutely prevent any other set of nominators from having their acclaimed works considered for nomination.


3.7.4 It is the intent of the Hugo Award nomination process that Worldcon members will only nominate works that they personally have read and enjoyed, and will not allow their nominations to be co-opted by any other individual or group.


3.8.1 Worldcon members may nominate any number of works for each category, up to the limit specified in 3.7.1. Each Worldcon member gets one nomination “vote”, which is divided equally among all nominations on his or her ballot. For example, if only one work is nominated in the category, that nomination gets a full “vote”. If two works are nominated in a category, each nomination gets 1/2 of a vote. If three works are nominated in a category, each nomination gets 1/3 of a vote, etc.

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the two works getting the least total number of votes will be compared. The work which appears on the fewest number of nominations ballots shall be eliminated.

3.8.1.2 The eliminated work is removed from all ballots. For the next round, votes are assigned to each work as in 3.8.1. Note that if a ballot contained, for example, five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now get 1/4 of a vote for the current round.

3.8.1.3 The least popular works are eliminated until only five works remain in the category.

3.8.1.4 In the event that more than two or more works are tied for the lowest number of votes received in 3.8.1.1, all tied works will be compared to see which appears on the fewest number of ballots.

3.8.1.5 In the event that two or more works are tied for appearing on the fewest number of ballots, all the tied works will be eliminated, unless eliminating all the tied works would bring the total number of works on the final ballot to be less than five. In this case, all the tied works will be retained, and the final ballot will be extended to include more than five works, as required.


Commentary:
1. The system is very simple and intuitive. It has two major components:
a. The least popular work is eliminated each round.
b. Nominators may decide how strongly they feel a given work or works should be on the final ballot. If there is one work they feel very strongly about, they can give that work their entire nomination vote. If there are a number of works they feel are Hugo-quality, they can nominate as many as they like, while keeping in mind that because they don’t feel as strongly about any one of them, they are spreading their one nomination vote among the works they have nominated.

2. Nominator instructions are equally simple:
a. Nominate whichever works you feel are Hugo worthy, up to the maximum permitted.
b. You have one nomination “vote” for each category, however, you may choose to divide your nomination vote among more works by simply listing more works on your nomination ballot. Your nomination vote will be split equally among all the works you recommend.
c. It is not necessary to nominate the entire maximum number of works.
d. If you feel very strongly that a particular work should be nominated, you can choose to put the entire weight of your entire nomination vote behind that work. If you have several works you feel are worthy, you can choose to recommend each one slightly less strongly.

3. The guiding principles in recommending this system are:
a. Hugo voters should each be able to suggest multiple great works, up to the maximum indicated in 3.7.1.
b. The works selected for the final ballot should be works which are popular among Worldcon members.
c. The nomination system should select a group of works that together represent as large a set of nominators as possible.
d. The nomination system should not select too large a group of works.
e. Other things equal, works suggested by more fans should be chosen over works suggested by fewer fans.
f. Other things equal, two works suggested by different fans should be chosen over two works suggested by the same fans. Other things equal, between two candidate final ballots, the one that includes suggestions from more fans should be chosen.
g. To the extent possible, the nominating system should not encourage “strategic” suggestions. The preference is that a fan should think “What are the most excellent works I have experienced” and not “How can I game the system to make sure the ones I most want will win”.
h. If a nominee declines at the last minute, it should not cause undue disruption of the process.

4. A wide variety of nomination systems were considered, drawing from the experience of experts in the field of election theory as well as from experienced members of fandom. The system chosen here, formally called “Single Divisible Vote with Least Popular Eliminated” or SDV-LPE, was found to be the system which best fits all of the desired properties listed above. The system is both mathematically and intuitively rigorous and robust.

FAQ’s
1. How does this system eliminate slate or bloc voting?

It doesn't, exactly, nor should a work be automatically eliminated just because it appears on a slate. On the other hand, any slate which nominates a full set of five works will find that each of its nominations only count 1/5 as much. With a large enough support behind the slate (five times as much), the slate may still sweep a category; however, if that many voters support the slate, they arguably deserve to win, and no fair and unbiased system of nomination will prevent that. The answer is, simply, to increase the general pool of voters. Regardless, with SDV-LPE, slates will never receive a disproportionate share of the final ballot, as occurred in the 2015 Hugos.

2. What if there are multiple slates (slate wars, “parties”, etc.)?

As with a single slate, the more works that anyone nominates, the less their votes count for each work. The end result is that even multiple slates are unable to sweep the nominations.

3. What happens if a genuinely popular work is nominated by a group of unrelated people?

If it is genuinely popular, the system will still select that work for the final ballot.

4. What happens if a genuinely popular work also appears on a slate?

Even if it is on a slate, if the work garners support from individuals – particularly if they list it as their only nomination, or with just a few nominations – then the system will select that work for the final ballot.

5. Isn't it true that any voting system can be gamed (or strategized, etc.)?

Yes, there is a theorem which proves that all voting systems must have inherent flaws. The objective is to choose a system whose flaws are not in an area of concern to the electorate.

6. What are SDV-LPE’s flaws?

A significant one is that ties may occur somewhat more easily than with the current system. The result of that is that it is possible that a category could have as many as 7 or 8 works on the final ballot. This still isn't common, but it is more likely than it is now. This is considered to be a worthwhile trade-off for SDV-LPE’s benefits.

7. What are SDV-LPE’s benefits?

Simply put, it reduces the power of bloc voting without eliminating the chance that works appearing on slates will make it to the final ballot. Conversely, it makes it very difficult for slates to prevent non-slate works from appearing on the ballot.

8. Couldn't slates just recommend a single work for a candidate, and it will automatically appear on the final ballot?

Yes, that is certainly a viable possibility – it’s also completely fair. It does not force any other works off of the ballot, and the final Hugo winner is determined by the same voting process we have always had. Just appearing on the ballot isn't a guarantee of winning a ballot. However, if a large section of fandom strongly believes that a work deserves a Hugo nomination, then it should, in fact, be represented on the ballot.

9. What happens with a large field with no stand-out favorites when a slate votes?

Even in this case, we were unable to find a simulation in which no non-slate works appeared on the final ballot. Slate works did receive a larger proportion of nomination slots than they did otherwise, however, again, this could be considered a fair and valid result. If there was no general favorite, then voters really had no collective preference.

10. Isn't this system too complicated for the average voter to understand?

No, it’s actually quite simple and straightforward, both in terms of voter instructions and in how the system operates. Essentially, the total number of votes for each work are totaled (and this will usually be the sum of factional votes). These votes are used to determine our two candidates for elimination, since the voters felt the least strongly about them. We then look at the number times the two works appeared on any ballot. The work that appeared the least number of times must be the least popular of the two, so is eliminated. This process continues until the five finalists remain.

11. I think we should just increase the number of nomination slots on the final ballot to (for example 6), and decrease the number of slots a voter can vote for to a smaller number (for example, 4). Wouldn't that be simpler and easier?

Unfortunately, this simply means that the largest slate will receive four of the nominations and the next largest will receive the remaining two. It doesn't solve the problem of forcing works off the ballot that had a chance to win the final election.

12. I think we should set up a committee to handle these situations as they occur. The committee would be empowered to add nomination slots or throw out slate-influenced ballots as required.

This could work. The problem is that now you have a small group of people who serve as literal gatekeepers to the Hugo Awards. In spite of the word on the Internet, this has never been the case in the past. Establishing it now means that those groups who believed it existed in the past will now be correct. Ultimately, human judgement is fallible. The fairness of a committee’s decisions will forever be subject to opinion. The end result is that the prestige of the Hugo Awards will forever be tarnished.

13. I think we should use [insert other mathematical voting system].

We considered essentially every type of voting system currently in the literature, guided by two experts in the field. Some of these systems do in fact have positive properties that speak for them. None of them were as simple or as intuitive as SDV-LPE, yet SDV-LPE meets all of the stated goals for a Hugo nomination system.

14. Won’t SDV-LPE be complicated to code and implement?

Actually, no. One of our non-experts coded a full simulator for the system in a matter of days. A full web-based app would not be much more difficult to handle.

15. Wasn't this system just designed by Social Justice Warriors to block the Good Stuff?

It is true that much of the discussion for this system occurred on Teresa and Patrick Nielsen Hayden’s “Making Light” discussion board, and it is also true that groups such as the Sad Puppies and the Rabid Puppies consider TNH and PNH to be The Enemy, and therefore completely biased and not to be trusted. TNH and PNH had absolutely no input in the discussions, however. Those of us who worked on the system were very clear that our goal was not to keep the Sad/Rabid Puppies off of the Hugo ballot and that any system which specifically targets any type of work is inherently wrong and unfair. One of the members of the group is a retired US Naval officer, a combat veteran, a certified Navy marksman, a Christian, and considers Robert Heinlein to be the greatest science fiction author who has ever lived. In short, he is exactly the Puppies’ demographic. But any slate, of any sort, be it a Sad Puppy or a Happy Kitten of Social Justice, breaks the Hugo Award because a small percentage of voters can effectively prevent any other work from appearing on the final ballot. This is a major flaw in the Hugo nomination system, and it is a flaw that must be fixed if the integrity of the award is to be maintained. Politics should play no role whatsoever in whether a work is Hugo-worthy or not.

#459 ::: Keith "Kilo" Watt ::: (view all by) ::: April 22, 2015, 10:49 PM:

Okay, it seems to have formatted reasonably well. Please let me know what you think, particularly errors and misconceptions I have (if I have them after all the research I've been doing, chances are other non-experts will have them, too). We may need more simulations of different cases, as well.

Looking forward to hearing your thoughts!

Kilo

#460 ::: David Wallace ::: (view all by) ::: April 22, 2015, 11:35 PM:

Keith@458: A couple of thoughts. First, I thought the conclusion above was to call the fractional "votes" points rather than votes, so a ballot for ABCDE would award 0.2 points to each work.

Second, I have a question about the SDV-LPE system: suppose there are 26 works being nominated, A-Z, and a slate of 200 voters is going for VWXYZ. All the other voters are going for various combinations of works from A-U, but no single work is selected by more than 190 voters (and in fact, most of the works are pretty close to being mentioned on 190 ballots). As I understand the way the system works, that means in any paired elimination from 3.8.1.1, any work from V-Z will beat any work from A-U. So it looks to me like the slate will sweep (in spite of not getting lots of points in the later rounds) unless two slate works face off against each other in one or more of the paired eliminations. Can someone simulate this a few times to see how many slate works survive?

#461 ::: David Wallace ::: (view all by) ::: April 22, 2015, 11:44 PM:

Followup to my @460: Actually, under my exact scenario, the slate has a pretty good chance of being eliminated entirely due to the tie provision of 3.8.1.5. So instead, assume that the slate gives a different number of votes between 195-205 to each of its 5 works, and simulate that case.

#462 ::: felice ::: (view all by) ::: April 23, 2015, 01:14 AM:

Keith "Kilo" Watt @458: "3.3.1 The guiding principles of the Hugo Award are as follows:"

I'd suggest moving that section to the commentary, rather than including it in the rules. It's not a substantive change, and I suspect is more likely to cause debate than aid the passage of the motion. Ditto 3.7.4.

Section 3.7.1 will have to be changed; it currently says "up to five (5) equally weighted nominations in every category".

David Wallace @460: "in any paired elimination from 3.8.1.1, any work from V-Z will beat any work from A-U. So it looks to me like the slate will sweep (in spite of not getting lots of points in the later rounds)"

Only the fifth place work gets compared for elimination. As works from A-U get eliminated, the votes for the remaining works go up (since their supporters' votes are less divided), potentially as high as 190, while the vote for each work on the slate stays around 40 (200 divided 5 ways), so the first four slots should go to the most popular works from A-U. For the final slot, the most popular slate work will win the elimination comparison and get on the final ballot.

#463 ::: David Dyer-Bennet ::: (view all by) ::: April 23, 2015, 01:43 AM:

FAQ #15: we can't reasonably claim "TNH and PNH had absolutely no input in the discussions, however." Teresa, for example, has commented on this very post.

More broadly, I'm not comfortable backing away from my friends quite so hard. Shouldn't the argument be more that this system is better, and if anything *helps* the puppies stated goal (diversity), but is less gameable?

#464 ::: Soon Lee ::: (view all by) ::: April 23, 2015, 02:49 AM:

Removing "TNH and PNH had absolutely no input in the discussions, however." doesn't change the point of FAQ#15.

Would it be useful to include URLs to point to this & the previous discussion? Explicit pointers to those interested to see how the sausage was made?

#465 ::: J Thomas ::: (view all by) ::: April 23, 2015, 06:37 AM:

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the two works getting the least total number of votes will be compared.

As I understand it, the rule is confusing and this simple statement is not adequate -- it makes the later explanations harder to understand.

Here are the situations to deal with:

1. The lowest three point scores are 21, 21, 24.
We compare the nominations with 21 and 21.

2. The lowest three point scores are 21, 22, 24.
We compare the nominations with 21 and 22.

3. 21, 21, 21.

We compare all three.

4. 21, 24, 24.

We compare all three.

It isn't the two works with the least number of points, because sometimes there are more.

It isn't the works with the two lowest numbers of points, because when there are two or more works tied at the lowest number then the 2nd-lowest is spared.

When there is a tie at the lowest number of points, that's all that's compared.

When there is not a tie at the lowest number but there is a tie at the second-lowest number, then we take more.

If you try to say it too simply then it hides the reality.

I don't particularly like the following:

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, at least two of the works with the lowest total number of points are compared. The rule is, take the two with the lowest number of points, and add more if and only if they are tied with the second-lowest nominee.

There ought to be a better way to say that.

#466 ::: J Thomas ::: (view all by) ::: April 23, 2015, 07:38 AM:

I am fundamentally dissatisfied with SDV-LPE but I don't have a better alternative to propose.

SDV-LPE does not work as effectively against slates as I'd like. In my simulations with reasonable estimates of the spread of other voting, with 1000 votes spread over hundreds of other nominees and 200 votes for the slate, I average about 1.7 slate wins out of five. I would prefer an average closer to 5/6, their fraction of voters.

I don't have much spare time and no uninterrupted time, so my explorations have been slow. I have the idea that something which actively tries to spread the winning votes among ballots would do better. But I haven't finished even working out the true result to measure voting routines against.

I want to grade a winning combination of nominations by first if you try to divide the ballots into five piles, each containing votes for a different nominee, how big is the smallest pile? and second, how small is the pile of votes that don't have a single winning nomination?

The first part looks pretty easy. When you have votes in a more-or-less power function that kind of fits some Hugo data, it's usually easy to bring the lowest pile up close to the average (or its maximum) and have no others lower. We're basicly solving five linear simultaneous equations in ten variables, with some restrictions on those variables. When two of the piles are from a slate then it's hard to reduce the number of ballots left out. So something that achieved that result would probably work well.

There are only 3003 combinations to choose 5 out of 15. It should be possible to just do all of them and compare, pretty quickly. But I haven't done it.

#467 ::: David Harmon ::: (view all by) ::: April 23, 2015, 08:17 AM:

Typo watch: 3.8.1.4 In the event that more than two or more works

#468 ::: Mercy ::: (view all by) ::: April 23, 2015, 08:33 AM:

I have a question that I didn't see an answer to in either the proposal language or the discussion (although searching on an ipad is less than optimal, so my apologies if I missed it).

What happens under this proposed change if someone nominates something in the wrong category? Is it simply discarded? Is it treated as it is in the current rules, moved if there's a free spot in the appropriate category?

#469 ::: Mercy ::: (view all by) ::: April 23, 2015, 08:40 AM:

Er. I realized after I previewed and hit post that I was a little unclear. I meant, if someone's nominating ballot has a work in the wrong category, right now it gets moved if they left a blank spot in the appropriate category. There's no mention of this case (which I'm guessing happens not that rarely with the shorter fiction categories) in this proposal.

#470 ::: nathanbp ::: (view all by) ::: April 23, 2015, 08:47 AM:

One thing that's not listed is there is the proposed system under SDV-LPE for picking the replacement nominee when one of the nominees refuses.

#471 ::: Keith "Kilo" Watt ::: (view all by) ::: April 23, 2015, 09:48 AM:

JT@465:

One quick clarification: The way I've coded it, in the situations you list, we compare all three in cases 1, 3, and 4. Case 2 has two with the lowest point totals, so is the "standard result". That's not to say it can't be coded some other way, but as you point out, I don't see any inherent reason to muddy the explanation with anything more complicated.

Maybe the proper way to say it is that it's the works that have the two lowest point totals.

All: Please keep the comments coming -- as we come to a group consensus, we may end up with a proposal we can use!

Kilo

#472 ::: dotless ı ::: (view all by) ::: April 23, 2015, 10:11 AM:

Keith "Kilo" Watt@458:

For FAQ#13 ("I think we should use X"): It's probably worth pointing out here that there's difference in requirements between a good system for choosing, say, a representative committee from multiple political factions, and the requirements for choosing Hugo finalists, especially given the diffuse distribution of favorite books among most of the membership. It seems to have taken a while for the discussion here to make that difference explicit, and I doubt that the difference will immediately come to mind if someone in the business meeting has a favorite voting scheme.

For FAQ#10 ("Is it too hard to understand?"): Should this section point out that the final Hugo vote already uses a system of eliminating the lowest-ranked contenders and rescoring ballots until there's a final result? That may make it seem less strange. The different rules for elimination and rescoring are because of the differences in requirements between choosing a single winner and choosing a good set of finalists.

#473 ::: Stewart ::: (view all by) ::: April 23, 2015, 10:19 AM:

As a matter of wording 3.3.1.1 and 3.3.1.2 are mildly inconsistent - enjoyable and great (best) as applied to books are imperfectly correlated concepts.

#474 ::: Steve Halter ::: (view all by) ::: April 23, 2015, 10:20 AM:

Here is an attempt to reword/reorder 3.8.1.1 through 3.8.1.5 to smooth them out and make tie handling more immediately apparent:

3.8.1.1 The final Award ballots shall list in each category the five eligible nominees as determined by successive rounds of elimination. In each round, the two works with the least total number of votes will be compared. Of those works, the work that appears on the fewest number of ballots will be selected for elimination from all ballots with ties handled as described in 3.8.1.2 and 3.8.1.3.

3.8.1.2 In the event that more than two or more works are tied for the lowest total number of votes received in 3.8.1.1, the work that appears on the fewest number of ballots will be selected for elimination.

3.8.1.3 In the event that two or more works are tied for appearing on the fewest number of ballots, all the tied works will be eliminated. If eliminating all of the tied works would bring the total number of works on the final ballot to be less than five, all the tied works will be retained, and the final ballot will be extended to include more than five works, as required.

3.8.1.4 For the next round, votes are assigned to each work as in 3.8.1. Note that if a ballot contained, for example, five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now get 1/4 of a vote for the current round.

3.8.1.5 Works are eliminated until only five works remain in the category.

#475 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 10:20 AM:

That's great work! A few comments:

1. Maximum votes per ballot: I understand the desire not to change things about the current system if the changes aren't essential. But one of the key advantages of SDV-LPE over other proportional voting systems is that it usually doesn't discourage voters from voting inclusively for more than 5 candidates. That's important because it can help achieve more overlap in non-slate ballots, which helps give inclusive, high-quality results. So if it were me, I'd definitely lift the limit on votes per ballot.

2. In the same spirit, I think commentaries 1b and 2d encourage "narrow" voting too much. I'd say :

b. Nominators may decide how strongly they feel a given work or works should be on the final ballot. If there is one work they feel very strongly about, they can give that work their entire nomination vote. If there are a number of works they feel are Hugo-quality, they can nominate as many as they like. There is an inherent tradeoff here, which no voting system can entirely avoid. Voting for many works, especially if they are popular, risks spreading your one nomination vote among the works they have nominated. However, voting for few works, or only unpopular works, risks having all of the works you voted for eliminated. Also, note that no matter how many other works you voted for, your vote still has full power to protect all your choices from being eliminated by a less-popular work.

2. Nominator instructions are equally simple:
a. Nominate whichever works you feel are Hugo worthy, up to the maximum permitted.
b. You have one nomination “vote” for each category, however, you may choose to divide your nomination vote among more works by simply listing more works on your nomination ballot. Your nomination vote will be split equally among all the works you recommend, but will still count fully to prevent all of them from being eliminated by a less-popular work.
c. It is not necessary to vote for any partcular number of works, only whichever ones you honestly believe may deserve a Hugo. Idividually, it is best to vote for enough of your favorites so that one or two of them have a good chance of becoming nominees. Collectively, the system can give the best slate of nominees if the average voter votes for around 6 or 7 candidates, so that the ballots carry plenty of information about which works are broadly popular.
d. If you feel very strongly that a particular work should be nominated, you can choose to put the entire weight of your entire nomination vote behind that work. If you have several works you feel are worthy, you can choose to recommend all of them. Voting for more cannot decrease the chances that at least one of your choices will be nominated, but it can in some rare cases decrease the chances for particular work you favor, as splitting your vote might cause that work to fall out of the set of 4 candidates which are never considered for elimination.

#476 ::: J Thomas ::: (view all by) ::: April 23, 2015, 10:24 AM:

@471 Keith "Kilo" Watt

One quick clarification: The way I've coded it, in the situations you list, we compare all three in cases 1, 3, and 4.

Interesting! I saw no hint of that in your explanation.

The two works with the lowest points. If more than two are tied for lowest, take them all.

20, 20, 23 the two with 20 are tied. The one with 23 is not tied with them.

20, 20, 23, 23 take all four.

If this is the way you want to do it, then it gets extra-easy to say. Are you sure this is what you want? The way I assumed you meant is just as easy to program. I don't know which gives better results, if there is a difference.

Here's a possible way to write what you want.

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the works getting the two (different) lowest levels of points will be compared. The work which appears on the fewest ballots shall be eliminated.


FAQ 16: How does that nomination thing go again?

First, count up the points. Say that one ballot has (A B C). Then you give 1/3 point to each of A B and C. Another ballot has (A D E F). You give 1/4 point to A D E and F. A now has 7/12 point while B has 1/3 point. Count all the ballots this way.

When you have finished adding points, you look for the two lowest levels of points among the nominees. Say that D G J and K have 21.3, 22.5, 21.3, and 22.5 respectively, and everything else has more than 22.5. Then choose D G J and K to test for elimination.

Find the total number of ballots that list D G J and K. Suppose that the numbers are 78, 89, 82, and 78, respectively. Then D and K have the lowest number of ballots, and those two are eliminated.

Repeat this process. If at some step there are fewer than five nominees remaining, restore the nominee(s) removed last time.

#477 ::: dotless ı ::: (view all by) ::: April 23, 2015, 10:39 AM:

Jameson Quinn@475: if it were me, I'd definitely lift the limit on votes per ballot.

Two reasons not to do that:

Political: Anything that makes the balloting look less like the current system might incline people to avoid the proposal as being too innovative/experimental.

Administrative: Kevin Standlee has already expressed concern about the current administrative load of determining which of the permutations of author and title refer to the same work, giving an actual example of someone kept off the final ballot due to a failure in the current system. More nomination slots means more load, which means a greater chance of failure.

#478 ::: cyllan ::: (view all by) ::: April 23, 2015, 10:42 AM:

Speaking as someone who has not been following this particular thread in any detail, I have to say that I found the language, the FAQs and the accompanying explanations quite clear. It's probably not a perfect system -- and I think some of the latter tweaks have been useful -- but it comes across cleanly. The proffered reason and 'how it works' makes sense. Give me a bit to ponder it just to make sure I understand it, and I can easily see that this is a solution that I'd be willing to support.

#479 ::: Keith "Kilo" Watt ::: (view all by) ::: April 23, 2015, 11:23 AM:

All: I was thinking about it on the way into work, and I think that the way JT suggests handling ties might actually be better. Basically, the only difference is that if there is a tie in the lowest point total, then we don't consider the second lowest point works for elimination in that round. The advantage I see to doing it this way is that you may have fewer situations where you also get ties in the number of nominations, so you will also have fewer situations where the final ballot gets extended to more than 5 works. I don't think these cases will come up often, but as JT says, it's really no harder to code, so it makes sense to do it this way. I can also think of some (essentially statistically impossible) situations where you might get some other issues.

Can the experts weigh in on this? Are there problems with just considering the lowest point total works if they are tied?

I'm keeping a list of the other suggestions, so as the group agrees on which should be incorporated, I will add that to the draft.

Thanks again,
Kilo

#480 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 11:35 AM:

dotless@477: I understand what you're saying about the political concerns, and can't really speak to that. I'm just saying that it would be better from a theoretical point of view without a limit.

As to the administrative concerns: again, the people with experience running these elections are of course the true experts on that, but I'd suggest that there are ways to make sure this isn't a problem. Basically, it's worth including an initial "blanket elimination round". Here's how I'd structure that to make it simple and robust:

1. Make a quick rough tally of the number of ballots that vote for each candidate. Don't worry too much about disambiguating the tough cases for now. Also, for this initial rough tally, do NOT count any votes on any ballot after the 7th vote per category. (This last condition is merely to prevent a slate from gaming this process by voting for 16 candidates. Since it prevents that strategy, it is very unlikely to have any impact in practice.)

2. Sort the candidates in order of number of votes.

3. Find the biggest gap in votes between two successive candidates in positions 15-21, breaking tied gap sizes by choosing the one closer to the 15th candidate. For instance, if the tallies 15-21 were 75, 73, 73, 69, 66, 62, 61 the gap in question would be 73 to 69; that is, 17th and 18th place.

4. So, the "number to beat" would be fixed at 73. Go back and disambiguate more carefully, but do not worry about cases where disambiguation could not bring a candidate above 73 or affect a total already above that level.

5. Once you've finished step 4 properly, eliminate any candidates with fewer than the "number to beat".

6. Proceed with SDV-LPE.

Theoretically, it is not impossible that the end result of the process would hinge on whether you set the initial mass-elimination "cutoff" at 15 or 20 candidates. But in practice, it is astronomically unlikely that that would happen; I would literally bet my house that it would never happen.

#481 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 11:36 AM:

dotless@477: I understand what you're saying about the political concerns, and can't really speak to that. I'm just saying that it would be better from a theoretical point of view without a limit.

As to the administrative concerns: again, the people with experience running these elections are of course the true experts on that, but I'd suggest that there are ways to make sure this isn't a problem. Basically, it's worth including an initial "blanket elimination round". Here's how I'd structure that to make it simple and robust:

1. Make a quick rough tally of the number of ballots that vote for each candidate. Don't worry too much about disambiguating the tough cases for now. Also, for this initial rough tally, do NOT count any votes on any ballot after the 7th vote per category. (This last condition is merely to prevent a slate from gaming this process by voting for 16 candidates. Since it prevents that strategy, it is very unlikely to have any impact in practice.)

2. Sort the candidates in order of number of votes.

3. Find the biggest gap in votes between two successive candidates in positions 15-21, breaking tied gap sizes by choosing the one closer to the 15th candidate. For instance, if the tallies 15-21 were 75, 73, 73, 69, 66, 62, 61 the gap in question would be 73 to 69; that is, 17th and 18th place.

4. So, the "number to beat" would be fixed at 73. Go back and disambiguate more carefully, but do not worry about cases where disambiguation could not bring a candidate above 73 or affect a total already above that level.

5. Once you've finished step 4 properly, eliminate any candidates with fewer than the "number to beat".

6. Proceed with SDV-LPE.

Theoretically, it is not impossible that the end result of the process would hinge on whether you set the initial mass-elimination "cutoff" at 15 or 20 candidates. But in practice, it is astronomically unlikely that that would happen; I would literally bet my house that it would never happen.

#482 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 12:30 PM:

@466:

SDV-LPE does not work as effectively against slates as I'd like. In my simulations with reasonable estimates of the spread of other voting, with 1000 votes spread over hundreds of other nominees and 200 votes for the slate, I average about 1.7 slate wins out of five. I would prefer an average closer to 5/6, their fraction of voters.

Yes, those are about the same numbers I would have guessed.

In defense of how SDV-LPE is working here: consider that, among the 1000 non-slate voters, there are probably at least 400 who, purely by chance, are voting for only no-hope "long tail" candidates which no voting system would elect. So from that point of view, it's a matter of 200 slate voters vs. 500-600 non-slate voters who support any viable candidate. In that case, 1.7 slate candidates is not too far from the "correct" proportional answer.

In my opinion, the best way to change that is to encourage the 400-500 "nonviable" voters to vote for more candidates overall, so that they have a better chance of including something viable. I think that if the average votes per ballot were higher - 5 or 6 or 7 instead of 3 or 4 - chances are that in this kind of scenario there would be only 2-3 hundred "nonviable" voters, so that SDV-LPE (like any other reasonable proportional system) could reasonably keep the slate to no more than 1 slot. And I think that SDV-LPE, of the proportional systems I've considered, has the best chance of encouraging that kind of "broad" voting (certainly, more so than STV, in my opinion).

I realize this isn't a perfect solution to the issue you're raising, but I think it's the best one that's available.

#483 ::: J Thomas ::: (view all by) ::: April 23, 2015, 12:49 PM:

@480 Jameson Quinn

Find the biggest gap in votes between two successive candidates in positions 15-21, breaking tied gap sizes by choosing the one closer to the 15th candidate. For instance, if the tallies 15-21 were 75, 73, 73, 69, 66, 62, 61 the gap in question would be 73 to 69; that is, 17th and 18th place.

The 5% rule which is currently in the WSFS constitution, will tend to cut it off at somewhere in the range 3 to 18.

Would you suggest lobbying to get the 5% rule removed? As you point out, nominations beyond the top 15 are unlikely to make a difference, and it could be argued that nominations that get less than 5% should not make a difference.

#484 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 01:03 PM:

@483: Personally, I think that the 5% threshold is unnecessary, because it's redundant with the No Award option in the final round. But speaking as a voting theorist, there's nothing wrong with a little redundancy. If you decide to keep the 5% rule, the initial absolute cutoff should be calculated in the way I suggested, or at 5% of the votes in that category, whichever is more.

#485 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 01:34 PM:

@479: Either way is fine from the point of view of theory. It's just a matter of what's easier to explain, pass, and program.

#486 ::: J Thomas ::: (view all by) ::: April 23, 2015, 01:35 PM:

In defense of how SDV-LPE is working here: consider that, among the 1000 non-slate voters, there are probably at least 400 who, purely by chance, are In defense of how SDV-LPE is working here: consider that, among the 1000 non-slate voters, there are probably at least 400 who, purely by chance, are voting for only no-hope "long tail" candidates which no voting system would elect. So from that point of view, it's a matter of 200 slate voters vs. 500-600 non-slate voters who support any viable candidate. In that case, 1.7 slate candidates is not too far from the "correct" proportional answer.voting for only no-hope "long tail" candidates which no voting system would elect. So from that point of view, it's a matter of 200 slate voters vs. 500-600 non-slate voters who support any viable candidate. In that case, 1.7 slate candidates is not too far from the "correct" proportional answer.

I think it's defensible, it just isn't the result I want. If we want a representative result that reflects the voting population, then it's justifiable to give a slate that's in the 10% to 25% range one spot to represent them, and rather than give them a second slot give it to something else that represents a reasonably large group of voters provided there is a good enough alternative.

I'm reasonably sure my approach would do that, but it looks like it might be hard to sell. If we had the 2015 situation over again it would under-represent the Sad Puppies if we think of the nominations as a one-man/five-votes thing. It would appear to fit their claim of being persecuted. Anybody else who thinks it should directly go by the number of votes might also dislike it.

After all, if the important thing is to nominate the best so the final vote can be for the best, then we wouldn't want to admit that the best one might be thrown off the ballot because it didn't fit together well enough with the other good choices. If we want the best, then we have to include the one that got 17% of the vote and not replace it with something else that got only 12% of the vote. ;)

SDV-LPE has that criticism too, but it sounds more like a fair vote and also it will keep slates from winning more than 2 nominations except in categories that don't get many votes. Anyway, if the best one gets onto the ballot along with slate nominations, then the best one will presumably win.

#487 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 02:09 PM:

@458:

13. I think we should use [insert other mathematical voting system].

We considered essentially every type of voting system currently in the literature, guided by two experts in the field. Some of these systems do in fact have positive properties that speak for them. None of them were as simple or as intuitive as SDV-LPE, yet SDV-LPE meets all of the stated goals for a Hugo nomination system.

Two responses to this.

1. We haven't come close to considering "every type of voting system currently in the literature". You should qualify that somehow, such as "every _applicable_ type".

2. I presume that the "two experts" are Cheradenine and me. If so, I'm glad to have been of assistance. My goals in participating here were primarily, to help a community I care about (though I'm not a member) to use systems that will help you make good decisions; and secondarily, to help show the expertise of my organization, Electology.xyz¹ (The Center for Election Science).

As a 501(c)3 organization, we've already been officially recognized as close consultants for the Webby awards, and have played unofficial consultant roles with other awards/hall of fame processes. We view our role as non-partisan, merely helping organizations to be more responsive to their members or voters, and helping to avoid results that make people unhappy. I hope that I've lived up to that goal here; while I've at times expressed my own personal opinion on some matters, I've been clear when I was doing so.

So frankly, I'd be grateful if this FAQ referred to me as something like "Jameson Quinn, from the voting reform organization Electology.xyz¹." I also understand if that is not in the offing. Obviously, even if it were a possibility, it would be only fair if Cheradenine also agreed to be referred to specifically by whatever name or handle they prefer.

¹ Of course, the xyz is actually the first three letters of "organization", but I want to be clear that I'm not trying to break the rules by posting a url, it's just what we call ourselves. (The board has made a decision to make the short name more prominent in the future; though many of our materials still focus more on the older, longer name.)

#488 ::: Cheradenine ::: (view all by) ::: April 23, 2015, 03:02 PM:

Sorry to be so quiet lately, I've been sick as a dog and all the energy I could muster went to work.

If I'm on of the two experts referred to - honestly, I'm hesitant to describe myself as an expert of any sort on this stuff. I'm just a person with a degree in a completely different area of math who happens to have developed an interest in voting theory. I feel like I've learned far more than I've taught during this discussion and exploration.

On the subject of ballot files: I do have the ones I used to sim and can send them to you, Kilo, if that would be helpful. I think I have probably found your work email, but don't want to use it without your OK.

#489 ::: David Wallace ::: (view all by) ::: April 23, 2015, 03:37 PM:

felice@462: No, I get that. The question is how many of the "other" works (A-U) make it above the 40 point threshold before getting eliminated. If the answer is 5 or more in my @460 scenario, I think the slate gets eliminated completely due to rule 3.8.1.5 on ties. If the answer is 4 or fewer, then I think the whole slate gets on the ballot, together with those other works.

In my modified scenario @461, I think the slate work with the highest number of votes always gets on, and I'm not sure what happens to the rest. Basically, I think the effect of 3.8.1.5 in its current form is "if there is a single work with a plurality of ballot mentions, that work is always on the final ballot; if two or more works are tied with the largest number of ballot mentions, usually either all of them will get on or none of them will (with a possible exception where all but one of them are eliminated in a single round due to being tied with each other.)"

It's the "all of them" possibility that concerns me a bit, although I suspect the probability of it happening in most reasonable scenarios is pretty low. We're probably not going to get a perfectly disciplined slate where nobody else votes for any of the slate works, which is basically what you would need for an n-way plurality tie.

#490 ::: felice ::: (view all by) ::: April 23, 2015, 03:57 PM:

Jameson Quinn @475: "I think commentaries 1b and 2d encourage "narrow" voting too much. I'd say :

b. Nominators may decide how strongly they feel a given work or works should be on the final ballot. If there is one work they feel very strongly about, they can give that work their entire nomination vote. If there are a number of works they feel are Hugo-quality, they can nominate as many as they like. There is an inherent tradeoff here, which no voting system can entirely avoid. Voting for many works, especially if they are popular, risks spreading your one nomination vote among the works they have nominated. However, voting for few works, or only unpopular works, risks having all of the works you voted for eliminated. Also, note that no matter how many other works you voted for, your vote still has full power to protect all your choices from being eliminated by a less-popular work."

I think we also need to highlight that as works on your ballot get eliminated, the other works you nominated get that share of your vote back, so when there's only one left, it gets your entire nomination vote, just as if you'd only nominated one work in the first place.


Jameson Quinn @482: "In my opinion, the best way to change that is to encourage the 400-500 "nonviable" voters to vote for more candidates overall"

Option 5b should be very effective in encouraging more votes for viable candidates, and reducing the impact of the 5% threshold. It should be a separate motion, though, not lumped in with SDV-LPE.


David Wallace @489: "It's the "all of them" possibility that concerns me a bit, although I suspect the probability of it happening in most reasonable scenarios is pretty low. We're probably not going to get a perfectly disciplined slate where nobody else votes for any of the slate works, which is basically what you would need for an n-way plurality tie."

I'd suggest that we amend the rules so that if there are more than three works tied for 5th place, then we eliminate all of them and only put four works on the ballot, giving us a ballot size ranging from 4 to 7.

#491 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 03:58 PM:

@489, @461: Generally, there are a couple of possibilities for this scenario (5 slate works have 195-205 votes, highest non-slate work has 190 votes). If the 4th-place non-slate vote has more than 102 divisible votes (when there are 4 non-slate and 2 slate candidates remaining), then the last elimination round will be between the top two slate candidates; one will be eliminated, leaving 4 non-slate and 1 slate winners. Otherwise, the 4th-place non-slate candidate will face a slate candidate in the last elimination round, and so the result will be 3 non-slate and 2 slate candidates. Generally speaking, if the second-place non-slate candidate has over around 150, it's about an even bet that the 4th-place will be over 100; with the first place at 190, I'd actually bet that the 4th place would be under 100 in over half the cases, but not too much more than half.

#492 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 04:00 PM:

@490: I agree with you on all three counts. Good points/ideas.

#493 ::: Keith "Kilo" Watt ::: (view all by) ::: April 23, 2015, 04:21 PM:

David, I will recode the sim to handle ties as JT suggests and run some other situations. I'm thinking it may satisfy your concerns in essentially all likely cases.

Cheradenine, feel free to send that data to my work address if you have it. I'd appreciate it! I'll let you know if out comes through.

Jameson/Cheradenine: I hate to say it, but there may be risk in using personal information with the proposal. I'd like to think not, since as someone else pointed out, this really does meet the Puppies' stated goals. Still... Ultimately, it's up to you guys.

Kilo

#494 ::: Jameson Quinn ::: (view all by) ::: April 23, 2015, 04:40 PM:

@493: Speaking for myself: I think I understand what you're talking about. Obviously, it's only the most rabid of the rabid that we're thinking of here. Without trying to minimize the issue, I believe their bark is worse than their bite, especially when it comes to somebody like me, who, to put it delicately, tends to land close to the top of the kind of totem pole those people care about. So in other words: bring it on.

(As for my organization: there's no such thing as bad publicity.)

I obviously don't know about Cheradenine, but they've comported themselves admirably in every respect, and while it might not be impossible to dig up a real identity on them, in their case I don't see why anyone would maliciously try. But obviously that's up to them.

#495 ::: J Thomas ::: (view all by) ::: April 23, 2015, 08:37 PM:

Here is a way to say it for the other approach, that I believe is correct though not fully intuitive.

3.8.1.1 The final Hugo Award ballot shall be determined by a number of “rounds” of elimination. In each round, the works with the lowest total number of points are compared, and the one(s) present on the fewest number of ballots is eliminated.

The rules are: Compare at least two with the lowest number of points. If two works have the same number of points, they must either both be compared or both be spared this round. Compare the smallest number of works that fit these rules.

FAQ 16: How does that nomination thing go again?

First, count up the points. Say that one ballot has (A B C). Then you give 1/3 point to each of A B and C. Another ballot has (A D E F). You give 1/4 point to A D E and F. A now has 7/12 point while B has 1/3 point. Count all the ballots this way.

When you have finished adding points, you look for the nominees with the least points. Say that D G J and K have 21.33, 22.5, 22.5, and 22.5 respectively, and everything else has more than 22.5. Then choose D G J and K to test for elimination. We accept D because it is the lowest. We include G because we must have two. We accept J and K because their points are the same as G and we must accept G.

Find the total number of ballots that list D G J and K. Suppose that the numbers are 78, 89, 82, and 78, respectively. Then D and K have the lowest number of ballots, and those two are eliminated.

(Note that on the next round, the ballot (A D E F) will give 1/3 point to each of A E and F instead of 1/4 point, because D is no longer counted. Each time we eliminate a nominee, the points increase for other nominees on the same ballots. So we must recompute the points each time.)

Repeat this process until there are five works remaining.

If there are two or three works tied for fifth place, include them so that there are six or seven final nominees. If there are more than three tied for fifth place, discard them all, leaving four final nominees.

#496 ::: David Wallace ::: (view all by) ::: April 23, 2015, 08:57 PM:

@490, @495: I'm not sure that fifth place should be considered special. You could also go directly to a five-way tie for 2nd, 3rd, or 4th (something like my scenario @460 with 1, 2, or 3 non-slate works meeting the survival threshold above the slate). It's even mathematically possible to get something like a 10-way tie for 2nd, which would give the organizers a nasty choice between an 11 item final ballot or declaring the top choice the winner without a final vote. But it would take truly impressive levels of coordination or coincidence to pull that last off on behalf of one or two slates, so I doubt it's a possibility we need to worry about too much.

#497 ::: felice ::: (view all by) ::: April 23, 2015, 10:06 PM:

David Wallace @496: "@490, @495: I'm not sure that fifth place should be considered special. You could also go directly to a five-way tie for 2nd, 3rd, or 4th"

How about this?

"3.8.1.5 In the event that two or more works are tied for appearing on the fewest number of ballots, all the tied works will be eliminated, unless eliminating all the tied works would bring the total number of works on the final ballot to be less than five. In this case, the final ballot may be extended to include up to seven works as required. If that is not sufficient to include all tied works, but eliminating all tied works would leave four works on the ballot, then all tied works will be eliminated. If eliminating all tied works would leave fewer than four works on the ballot, tied works will be randomly selected for elimination until the ballot size is reduced to seven."

Random selection isn't ideal, but if multiple works are tied on both number of ballots and sum of fractional votes, it seems the only fair option. We can't allow an unlimited number of tied works on the final ballot; it's theoretically possible to have a hundreds-of-ways tie for a spot on the ballot (albeit statistically improbable).

#498 ::: J Thomas ::: (view all by) ::: April 24, 2015, 05:26 AM:

I'm not sure that fifth place should be considered special.

Ties are rare. If there's a two-way tie above the fifth place in the final vote, you can still keep it to five by throwing out the fifth place. If there's a three-way tie above the fourth place you can throw out the fourth and fifth. If there's a tie in fifth place you're stuck. You can throw them both out or leave them both in or flip a coin.

Fifth place is special for this kind of nominating, because it's usually fifth place that's at stake.

But then, ties are rare and we don't really need to make a special rule for extremely rare cases. I got carried away.

If we want to make unresolved cases rarer, when we want to resolve ties we could use a third rule -- count the ballots that include one of the tied votes, and for each of them count the number of other winners on it. Add up the other winners, and break the tie by eliminating the one(s) that have the most overlap with other winners.

#499 ::: David Wallace ::: (view all by) ::: April 24, 2015, 04:44 PM:

J Thomas@498: Ties may be rare historically, but they become a lot more likely in the presence of slate voting, which is why we are having this discussion in the first place. If you have idealized slate voting (every slate voter votes for all the works on the slate, no one else votes for any of them), then a five-way tie in some position becomes a certainty.

That ideal model may still be rare in practice, because slates won't normally vote with perfect discipline, and some other voters may vote for one or two slate works either because they liked them or just to mess with the slate, but it's certainly a lot more likely to have a multi-way tie in the presence of a slate than if everyone is just voting their individual preferences. I think it's reasonable to ask how a voting system designed to combat slate voting can cope with ideal slate voting, and tweak the voting system if that model reveals a problem.

#500 ::: Joshua Kronengold ::: (view all by) ::: April 24, 2015, 06:20 PM:

I wouldn't include the 5% threshold in a proposal. If the meeting doesn't like its removal, it can get struck via amendment, but over the past few years it's done far more harm than good, as Best Short Story got spread thinner and thinner.

It ends up being about 5th place, because if there's a 5 way tie for 2nd, we encounter it when we run the elections for 5th. So that's ok. In general, the Hugos are better with a "long tail" candidate or three on the ballot than with only 3 candidates.

While I don't disagree that increasing the number of nominees per ballot is potentially desirable, I am very resistant to doing so with this measure. There are potentially complications with doing so in terms of the load on administrators, and on the design of nomination forms, and I think it would distract from the base point, so I've left it out of the following draft.

The usual form for this kind of thing is something like:

(note: ML only allows italics and strong -- but WSFS constitutional ammendments should use strong (bold) for the title, underlines for new text, and strikeout for removed text. I've used [strikeout] pseudotags for strikeout and <em> instead of underlines.

Short Title: Change Nominations to Single Divisible Vote, Least Popular Elimination

Moved, to amend section 3.8 (Tallying of Nominations) as follows:

Section 3.8: Tallying of Nominations.

[strikeout]3.8.1: Except as provided below, the final Award ballots shall list in each category the five eligible nominees receiving the most nominations. If there is a tie including fifth place, all the tied eligible nominees shall be listed.[/strikeout]

3.8.2: The Worldcon Committee shall determine the eligibility of nominees and assignment to the proper category of works nominated in more than one category.

3.8.3: Any nominations for “No Award” shall be disregarded.

3.8.4: If a nominee appears on a nomination ballot more than once in any one category, only one nomination shall be counted in that category.

[strikeout]3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.[/strikeout]

3.8.6: The Committee shall move a nomination from another category to the work’s default category only if the member has made fewer than five (5) nominations in the default category.

3.8.7: If a work receives a nomination in its default category, and if the Committee relocates the work under its authority under subsection 3.2.9 or 3.2.10, the Committee shall count the nomination even if the member already has made five (5) nominations in the more-appropriate category.


3.8.8 The final Award ballots shall list in each category the five eligible nominees as determined by successive rounds of elimination, where each nominator gets a single vote, divided equally among their nominations.


3.8.8.1 In each round, the two works with the least total number of (fractional) votes will be compared. Of those works, the works (at least 2) that appear on the fewest number of ballots will be selected for elimination.


3.8.8.2 In the event that more than two or more works are tied for the lowest total number of votes, the work that appears on the fewest number of ballots will be selected for elimination. If that elimination would reduce the number of ballots to fewer than 4, then instead none of the works should be eliminated and all remaining works appear on the final ballot. (In the unlikely event that more than 8 works would make the final ballot, the administrators should use their judgement).


3.8.8.3 For the next round, votes are reassigned to each work as in 3.8.8. (Example: If a ballot contained five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now gets a 1/4 of a vote)

Submitted by: (insert members here, ideally including at least one attending member who will be at all the relevant WSFS meetings. A proposing member gets the privlege of speaking first to the pro side of the proposal)

Commentary: As has been shown by numerous recent Hugo ballots, but most explicitly by the 2015 ballot, the current Hugo nomination system can be too easily dominated by the largest single faction members nominating in that category. This proposal would replace the nomination tallying system with a system designed to produce proportional representation among nominators, so that every sizable group generally gets to elect a nominee, but none get to set the terms of the entire ballot.

#501 ::: Avram ::: (view all by) ::: April 24, 2015, 06:43 PM:

We actually do support the strike tag, along with a bunch of others that aren’t listed. You can use b instead of strong if you want, as well as cite or i instead of em. And tt and code are supported (but for not kbd for some reason), and sup, sub, and small. And a few block-level tags that can wind up looking really ugly in combination with the automatic paragraph formatting.

#502 ::: David Harmon ::: (view all by) ::: April 24, 2015, 06:46 PM:

Joshua Kronengold #500: Re: tags, it's <strike>, not "strikeout". <s> doesn't seem to work, though.

#503 ::: J Thomas ::: (view all by) ::: April 24, 2015, 09:00 PM:

I was all along confused by 3.8.4, 3.8.6, and 3.8.7, but ignored it because it looked like it wasn't what we were doing. What are they doing here?

Are they saying that if they notice two blocks of votes that turn out to be for the same nominee, that they discard one of them instead of merging them? That when they move a work to a more appropriate category they discard no more than five votes for it in the "default" category?

If we're going to write about it, should we find out what it means and write it clearer? Or is it clear to other people?

About 3.8.8.2, the ways we can get in trouble are:

1. a 3 or more way tie when there are 6 left gives us too few left if they are all discarded. So we should include them all and have six nominees.

2. A 4 or more way tie when there are 7 left gives us too few left if they are all discarded. We should include them all hand have seven nominees.

3. A 5 or more way tie when there are 8 left gives us too few if they are all discarded. We should include them all and have eight nominees.

4. A 6 or more way tie when there are 9 left gives us too few if they are all discarded, and too many if they are all kept.

But one slate can't give us a 6-way tie even if they coordinate perfectly. For a 6-way tie we pretty much need for a slate to give us the same number of votes for each member which accidentally matches up with the number of votes for one non-slate choice.

My preference is to go ahead and discard them even if there are only three nominees left. This sort of tie looks very unlikely without a slate and pretty unlikely with a slate. Why go to extra trouble to keep 3 or 4 or 5 slate members on the ballot?

So if the number of final entries can be 3 to 8, then it's trouble when there's:

1. a 4-way tie when there are 6 left.
2. a 5-way tie when there are 7 left.
3. a 6-way tie when there are 8 left.

I hope these will be rare.

#504 ::: felice ::: (view all by) ::: April 24, 2015, 10:56 PM:

J Thomas @503: "I was all along confused by 3.8.4, 3.8.6, and 3.8.7, but ignored it because it looked like it wasn't what we were doing. What are they doing here?

Are they saying that if they notice two blocks of votes that turn out to be for the same nominee, that they discard one of them instead of merging them? That when they move a work to a more appropriate category they discard no more than five votes for it in the "default" category?"

3.8.4 says if a work is listed multiple times on the same ballot it's only counted once for that ballot, eg if someone nominates AAABC, it gets counted as ABC instead. Though for SDV-LPE, this rule isn't actually necessary, and we could let people weight their ballots by giving their preferred works a bigger share of their vote.

3.8.6 means that if eg someone nominates Avengers: Age of Ultron for best short story, it will be counted as a nomination in the best dramatic presentation (long form) instead, unless they've nominated five other movies in bdp(lf), in which case the nomination in the wrong category is discarded. The second half of this is also not needed for SDV-LPE with no limits, though it should stay if we keep the five works per category limit for nominations.

3.8.7 appears to be broken in the current rules - I think it should be referring to subsections 3.2.7 and 3.2.8. But it means if someone nominates eg Avengers for bdp(lf), but the administrators move it to bdp (short form) instead, the bdp(lf) nomination counts even if they've also nominated five TV episodes. Not necessary if we drop the five work limit. (And technically Avengers probably isn't eligible to be moved to bdp(sf), I'm not sure on the runtime)

#505 ::: Kevin Standlee ::: (view all by) ::: April 24, 2015, 11:33 PM:

Joshua Kronengold @500:

I wouldn't include the 5% threshold in a proposal. If the meeting doesn't like its removal, it can get struck via amendment,...
There has already been such a proposal submitted ("The 5% Solution") and it will be on the agenda. I suggest that any other proposals avoid touching that section so that it can be considered in isolation.

(This time, speaking officially from Business Meeting; my life is going to be difficult enough as it is sorting out overlapping and conflicting proposals.)

#506 ::: Chris ::: (view all by) ::: April 24, 2015, 11:51 PM:

"3.8.8.1 In each round, the two works with the least total number of (fractional) votes will be compared. Of those works, the works (at least 2) that appear on the fewest number of ballots will be selected for elimination."

I think this needs some editing. It looks like two or more of two works will be eliminated, and that can't be right.

#507 ::: P J Evans ::: (view all by) ::: April 25, 2015, 12:14 AM:

504
Under current rules, I believe that A would only be counted once under those circumstances. (It would be looked at very closely, because it looks like someone is trying to pack the ballot box. Which is what we don't want.)

#508 ::: J Thomas ::: (view all by) ::: April 25, 2015, 01:30 AM:

3.8.4 says if a work is listed multiple times on the same ballot it's only counted once for that ballot, eg if someone nominates AAABC, it gets counted as ABC instead. Though for SDV-LPE, this rule isn't actually necessary, and we could let people weight their ballots by giving their preferred works a bigger share of their vote.

Thank you! Now it makes sense.

3.8.7 appears to be broken in the current rules - I think it should be referring to subsections 3.2.7 and 3.2.8. But it means if someone nominates eg Avengers for bdp(lf), but the administrators move it to bdp (short form) instead, the bdp(lf) nomination counts even if they've also nominated five TV episodes.

I think 3.8.7 is written backward. If you nominate a work in the wrong category, they shouldn't move it to the right category if you already have 5 nominations in the right category.

But they should move it even if it's the sixth nomination in the wrong category. Don't throw it away for being the sixth nomination in the wrong category.

I think the wording of those was clumsy, but most of the reason I misunderstood it was my own misreading. I probably see what they mean just fine once you pointed it out.

#509 ::: felice ::: (view all by) ::: April 25, 2015, 01:57 AM:

J Thomas @508: "I think 3.8.7 is written backward. If you nominate a work in the wrong category, they shouldn't move it to the right category if you already have 5 nominations in the right category.

But they should move it even if it's the sixth nomination in the wrong category. Don't throw it away for being the sixth nomination in the wrong category."

Under SDV-LPE, yes, there's no need to throw away a sixth nomination. But under the current rules (and the text for 3.8.6 and 3.8.7 in #500 is the current version, not a proposed change), you're only allowed five nominations per category, and you shouldn't be allowed to get around that by deliberately or accidentally nominating extra works in the wrong category. But you shouldn't be penalised if you nominated correctly, but the administrators changed the work's category.

#510 ::: J Thomas ::: (view all by) ::: April 25, 2015, 07:28 AM:

@509 Felice

This is an academic question and it probably doesn't matter what we think. But I can see it either way.

Yes, it makes sense not to penalize somebody who acted in good faith. Also, it makes sense not to penalise them if they have room in the category you're moving them too, even if they don't have room in the category you're moving them from.

One reason I'm inclined to think it's backward is that it specifies *less than* five in the category it comes fron when it's legal at five or less there. But less than five in the category it's going to would leave it at five or less in the new category.

On the other hand, why would they specify about ballots that have six nominations with one in the wrong category? Why would that ever happen? Maybe for the same reason we are looking into five-way ties?

Sorry to clutter the discussion with this. We can ignore it, it doesn't affect what we're looking at. But thank you for explaining it.

#511 ::: J Thomas ::: (view all by) ::: April 25, 2015, 08:27 AM:

I'm ready to propose an alternative to SDV-LPE.

I know it works now.

It would be tedious to do by hand, but it's quick by computer.

Here is what it does that I want:

The Hugo nominations are a survey of what voters think deserves a Hugo. This method chooses according to two criteria.

One is that we want the largest number of voters to be represented. We don't want one minority to determine too many of the winners.

But imagine that we did that by first choosing the work with the largest number of votes, and remove those ballots from further consideration. Choose the work with the largest number of remaining votes.

By the time we get to the fifth choice, we have a much smaller number of ballots, and they are likely to be more random. The fifth winner could easily be something that has only a few votes.

So the second criterion is that each one of the five in the combination should have pretty many votes too.

We want five that cover the field about as well as five can, and each of them has a following.

The method

Take every combination of five nominees that has a plausible chance to win. (With the 5% rule there are not a large number of likely plausible winners, almost certainly 18 or less. So there are fewer than 20,000 combinations.)

For each combination, count how many ballots are saved, that is, they include a vote among the five.

For each combination, arrange the saved ballots into five groups labeled by the five nominees. Each ballot in a nominee's group includes a vote for that nominee. Do this so that the smallest group is as large as it can be. (I have a method to do this which works. It is not tuned to be efficient but still is fast enough not to need to be more efficient.)

If M is the number of ballots saved, and N is the size of the smallest group, the combination's score is N + aM where a is a tunable parameter and will probably be less than 0.5.

The winning combination has the largest score. In case of a tie, choose the tied combination that has the fewest lost votes.

If there is still a tie, choose the tied combination that has the largest second-smallest group.

Discussion

This may be NP-hard for number of winners. I haven't really checked. That is, if there were 30 winners and we had to check every combination of thirty, it may take too long. But there are only five winners and it's fine for that.


We can tune it to decide the relative importance of the two criteria. With some tunings, large slates will get one nominee but no more unless the nonslate competition is utterly pathetic.

If we put all the emphasis on M, the number of ballots saved, a slate will never get more than one nomination but some of the other winners may have only a few votes.

#512 ::: Jameson Quinn ::: (view all by) ::: April 25, 2015, 11:49 AM:

@584: That system clearly accomplishes its goals with honest votes. It's also the kind of thinking I've learned to distrust. In particular, I've learned that when you optimize too hard in one direction — in this case, trying to make sure that a slate has a hard time getting more than one nomination — there can be a kind of "strategic rebound" in an opposing direction — in this case, a strategic incentive for bullet voting. It may be that, with actual Hugo voters, bullet voting won't be a problem with your system. But to me, it is clearly strategically advantageous, far more so than under SDV-LPE.

To be honest, I think I'm a little bit biased against your system. It reminds me too much of too many other attempts to devise a voting system that optimise the result for honest votes, and which all too often end up creating a strategic nightmare. Looking rationally, I think your system has strategic problems, but certainly not at nightmare levels. But it still rubs me wrong on an aesthetic level, and there's part of me that's still saying "there has to be a catch, or even, more of a catch than you've noticed yet".

When I try to ignore my aesthetic biases, it's not a bad system. I still think that I'd rather choose SDV-LPE, because the better strategic incentives are worth the slightly weaker resistance to slates. And I also think that the discussion is far enough along that bringing up a new option isn't really what's needed.

#513 ::: felice ::: (view all by) ::: April 25, 2015, 04:49 PM:

J Thomas @510: "One reason I'm inclined to think it's backward is that it specifies *less than* five in the category it comes fron when it's legal at five or less there. But less than five in the category it's going to would leave it at five or less in the new category."

Ah, I think I see what you mean. 3.8.7 is talking about works being moved, not nominations being moved (see 3.2.7 and 3.2.8 in http://www.wsfs.org/bm/const-2014.html), and "the more-appropriate category" means the destination category, not the default category; if the new category wasn't more appropriate than the default, the admins wouldn't have moved the work. It's saying that if the admins move a work, it keeps all its nominations regardless.


J Thomas @511: "For each combination, arrange the saved ballots into five groups labeled by the five nominees. Each ballot in a nominee's group includes a vote for that nominee"

But not necessarily all the ballots with a vote for that nominee, even in the smallest pile. Interesting, but the tunable parameter is a bit of a red flag, and it seems harder to understand the process than SDV-LPE; does it produce significantly better results?

#514 ::: J Thomas ::: (view all by) ::: April 25, 2015, 05:10 PM:

@512 Jameson Quinn

You could be right. I haven't studied it much yet, beyond confirming that my coding is correct. I wanted to announce it as soon as I could since I'm so late.

In almost any system, if what you care about is getting one nomination to win, additional votes for anything else will only hurt because anything which helps another nominee is good for the competition and not for your single nominee.

That's true for SDV-LPE. Your half-vote for your preferred nominee does less good than a full vote until the second nominee loses. There's the chance that you preferred nominee will lose first. The times to vote for two are when you don't care which one wins, or when you hope they can both win with a half-vote from you for each of them.

But if what you care about is that your preferred nominee wins, it's silly to vote for a second one also, hoping that the second one will lose in time to give your full vote for the one you want.

I will present some reasoning about this, although to my way of thinking simulation trumps reasoning, if done right.

If you vote for one, you contribute to every combination of 5 nominees that includes that nominee. Any combination that doesn't include that nominee cannot save your vote. Your vote adds to your nominee's pile and to no other, which could be good or bad.

If you vote for two, you contribute to every combination of 5 nominees that includes those two nominees. It becomes easier to include your vote. It contributes somewhat more to combinations that include both of your nominees because your ballot can contribute to whichever of the two piles of votes that's smaller.

So if everybody voted for one, all the voting system could do would be to pick the 5 nominees that have the largest number of votes. There is nothing else to do in that case. But if enough people vote for two, then there's increasing flexibility.

Say that for one case we pile up votes for A B C D E as follows:

A 194
B 172
C 168
D 132
E 112

But there are 12 AE votes in A. We could move those votes to E.

A 182
B 172
C 168
D 132
E 124

And there are 10 BE votes in B. Move those to E too.

A 182
B 162
C 168
D 132
E 134

There are 24 CD votes in C so we move 18.

A 182
B 160
C 150
D 150
E 134

There are 10 DE votes in D so we move 7.

A 182
B 160
C 150
D 143
E 143

When you vote for two you like, your vote can increase the diversity score for some of the combinations that include those two.

So if you like both of them, it's good. If you only like A then I'm not sure you do better to also vote for E.

If there are 15 different nominees that have a chance, and you only vote A, you are improving the 1001 combinations that include A and not the 2002 combinations that don't contain A. You improve them more when A is the fifth choice.

If you vote A and E then you improve the 1716 combinations that include one or the other of them, and you do more for the 284 combinations that include both. So you are also improving the 717 combinations that include E but not A.

I'm not clear what strategies would work, but at first look I'd guess that if you only want one, you do better to also vote for others that look promising but not too promising. It looks very complicated to figure out, and also it may heavily depend on guessing the odds, which tends to be hard to do particularly when the result depends on the number of cross-votes for the fifth candidate.

I think it has a lot of potential, sad that I didn't get it working until now. It takes a lot of careful testing to find problems. We haven't done enough of that for SDV-LPE, much less an alternative.

"Mother Nature always sides with the hidden flaw."

#515 ::: Keith "Kilo" Watt ::: (view all by) ::: April 25, 2015, 05:35 PM:

JT@514:

We haven't done enough of that for SDV-LPE, much less an alternative.

I'm still trying to think my way through your proposal, but FYI, Cheradenine and I have been running a multitude of sims, and so far everything is looking very reasonable with SDV-LPE. We haven't yet found a case that breaks it when using variations of the 2013 Hugo data (in fact the baseline test gave the actual result of the 2013 nomination list). That's not to say those cases aren't out there, though, so I'm hoping more people will propose cases that might be troublesome so we can test them.

BTW, since suggestions have died down a bit, I'll post a summary tomorrow so that maybe we can nail down some actual proposal language. I'm not opposed to doing compare and contrast tests with the system you're describing though. My only initial complaint so far is that it's not intuitively obvious how it works philosophically yet, but as we've said before, there's value in describing the system first, then developing an explanation for it.

Kilo

#516 ::: J Thomas ::: (view all by) ::: April 25, 2015, 05:43 PM:

@513 Felice

J Thomas @511: "For each combination, arrange the saved ballots into five groups labeled by the five nominees. Each ballot in a nominee's group includes a vote for that nominee"

But not necessarily all the ballots with a vote for that nominee, even in the smallest pile.

Here's how that can happen. Say the two smallest piles are tied with 60 D votes and 60 E votes. And there is one DE vote.

If the DE vote is put in D then the smallest is E and E doesn't have all its votes. If the DE vote is put into E then the smallest is D and D doesn't have all its votes.

It takes a 5th place tie to get that. If there are more D votes, you can put the DE vote into E and if necessary get another D vote from some other pile.

Interesting, but the tunable parameter is a bit of a red flag, and it seems harder to understand the process than SDV-LPE; does it produce significantly better results?

It was intended to get better results. It hasn't been tested enough yet.

It was intended to get different results. Whenever you have more than one goal, you must choose among them. If you have a simple system that hides that choice, it has hidden the choice in the simplicity, it has made that choice without you noticing. Remember the STV systems that had different weights for the transferred parts of the vote? They made that choice explicitly, to get the result they wanted. SDV-LPE makes it seem reasonable that when you split a vote it's one vote that you're splitting so each fractional vote should be 1/n. But it's a choice. If we wanted to encourage people to vote for multiple candidates, we could make the fractional votes bigger so that they win something by voting for extras.

Two goals. If we count only the total number of votes, then the second nominee on a slate gets nothing. But the fifth winner might be from another small slate.

If we count only the spread then a slate of 200 votes will be run as one nomination with 200 votes that are all unique, and two nominations with 100 votes each, and 3 nominations with 67, 66, 66, and so on.

But I'm talking about it when I should be actually running simulations and finding out whatever I find out.

#517 ::: Cheradenine ::: (view all by) ::: April 26, 2015, 12:32 AM:

The scoring system in #511 has the potential to lead to some very weird rankings of choices of nominees.

Consider the following set of 100 ballots:

1x A
1x B
1x C
1x D
1x E
19x AF
19x BF
19x CF
19x DF
19x EF

F appears on 95% of the ballots; A, B, C, D, and E each appear on 20% of them. BUT:

ABCDE represents all ballots and can be divided in 5 equal piles of 20, giving a score of 100 + 20a.

ABCDF (likewise ABCEF, etc.) represents 99 ballots and can be divided into 4 piles of 20 and 1 of 19, giving a score of 99 + 19a, which is a lower score than ABCDE's regardless of the value of a.

I don't think a system that potentially rejects a work appearing on 95% of the ballots in favor of works which each appear on 20% of the ballots is viable.

#518 ::: Brad from Sunnyvale ::: (view all by) ::: April 26, 2015, 12:37 AM:

Long delay, working in Shanghai, but...

There are arguments for and against a lot of power for a fannish inquisition or special committee rule.

A high threshold for triggering of the power (like support of 50% of all members, something I think would be very hard to get) means it's very hard to abuse the power or even use it unless the problem is very, very clear to fans. One could learn and revise the threshold over time. The goal, as I said, is to not expect (to need) the fannish inquisition.

With a high threshold, you can give it more leeway. More leeway means more ability to fix whatever problem the attackers can dream up. Less leeway means that attackers can find loopholes -- as is an issue with purely algorithmic rules.

The amount of power can be part of the process calling for the inquisition. Which is to say, the proposal for an FI would state what it is charged to do, and 50% of fans (or whatever) would need to say yes to that. It would be easier to get 50% support for a very specific action like "Remove works with too much cluster voting from the ballot this year" or "increase nominations this year" and harder for one that authorizes doing whatever it takes.

The chief weapon is fear and surprise. Which is to say the attacker knows that whatever they do, they might get it undone. This stops the attack from happening in the first place, or makes it more mild. (The more mild the attack is, the less likely fans would approve a fannish inquisition.)

It is an interesting question. This year's ballot, with some categories swept by the slates, would probably get an inquisition. Last year's, with just a copule of slate candidates, might well not.

In effect this is a use of some direct democracy as a fallback. Permanent rule changes still need ratification and will be slow, as planned. Short term fixes can be fast, but need overwhelming support.

No change to voting, very minimal strategy, no explaining of voting system -- though there is explaining of the system by which the inquisition is activated. After which, their chief weapon can be a ruthless efficiency, and an almost fanatical devotion to fandom.

#519 ::: Joshua Kronengold ::: (view all by) ::: April 26, 2015, 01:32 AM:

Avram, thanks for the info -- I tested with <s> (which is what I usually use) but not <strike>.

I don't like JT's newer proposal. It's gameable in non-obvious ways (probably) and feels too clever to me. And SDV-LPE seems to do the job nicely for expected results and, hugley importantly, is very comprehensible both in how it operates and why it works, which is a key component to acceptance. Also, my programmer's heart rebels at NP-hard solutions even if their N is theoretically (I'm not convinced) low enough that NP-Hard isn't non-computational. Also, the lack of ease of understanding makes it much harder to vet the process.

#513: Exactly. Obviously, if we relaxed restrictions on the number of works we could drop this restriction. The idea of course is that if the will of the nominators is such that a work gets moved (this is the usual reason that the Admins move a work), then people who voted it in the "right" category don't lose the their votes--but you still don't get to nominate 6 things by dropping a nomination in the wrong category when the rules say you can only nominate 5 of them.

Re duplicates: I think avoiding the gamability of being able to list a work but give another work most of your votes, while cool on a gamer level, is a very good idea.

#506: yeah, I put the "or more" in the wrong part of the clause. Best fix that.

Kevin@505: I'll accept your preferences on this -- other Chairs have requested that measures modifying the same section be collapsed (and I thought that one was a mistake at the time even though the measures had similar but not identical purposes), but in this case the nitpicking and flyspecking committee can clean up the mess if there is one (it won't be a huge deal, if so).

Also, they really called the measure "The 5% solution?" Calling back to "The Seven Percent Solution?" I'm afraid I only know one song from that show, but it seems apropos.

TTTO: "I Never Do Anything Twice/The Madam's Song"

When I was just a young fan,
I don't recall the date,
We made a rule for our premier award,
Even if it made the cut, a nomination met its fate,
If one in twenty didn't think it scored

At first it proved a good rule,
Avoided the long tails,
But later, when the field ballooned in size,
If everbody fails,
To read all the same tales,
Where the ballot's concerned there's too much for the prize,

Then, yes, the genre was small,
Now, though, you can't read it all,
Then, tastes were more concentrated,
The best stories rated,
And found themselves slated

We must this rule amend,
At this point, it's hard to defend,
Think, you'll see it makes sense,
To not restrict works by percent.

....

Anyway, here's another draft:

Short Title: Change Nominations to Single Divisible Vote, Least Popular Elimination

Moved, to amend section 3.8 (Tallying of Nominations) as follows:

Section 3.8: Tallying of Nominations.

3.8.1: Except as provided below, the final Award ballots shall list in each category the five eligible nominees receiving the most nominations. If there is a tie including fifth place, all the tied eligible nominees shall be listed.

3.8.2: The Worldcon Committee shall determine the eligibility of nominees and assignment to the proper category of works nominated in more than one category.

3.8.3: Any nominations for “No Award” shall be disregarded.

3.8.4: If a nominee appears on a nomination ballot more than once in any one category, only one nomination shall be counted in that category.

3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.

3.8.6: The Committee shall move a nomination from another category to the work’s default category only if the member has made fewer than five (5) nominations in the default category.

3.8.7: If a work receives a nomination in its default category, and if the Committee relocates the work under its authority under subsection 3.2.9 or 3.2.10, the Committee shall count the nomination even if the member already has made five (5) nominations in the more-appropriate category.


3.8.8 The final Award ballots shall list in each category the five eligible nominees as determined by successive rounds of elimination, where each nominator gets a single vote, divided equally among their nominations.


3.8.8.1 In each round, the two works (or more, in the case of a tie) with the least total number of (fractional) votes will be compared. Of those works, all but one will be selected for elimination, removing the works that appear on the fewest number of ballots.

3.8.8.2 If (due to a tie) elimination would reduce the number of ballots to fewer than 4, then instead none of the works should be eliminated and all remaining works appear on the final ballot. (In the unlikely event that more than 8 works would make the final ballot, the administrators should use their judgement).

3.8.8.3 For the next round, votes are reassigned to each work as in 3.8.8. (Example: If a ballot contained five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now gets a 1/4 of a vote)

Submitted by: (insert members here, ideally including at least one attending member who will be at all the relevant WSFS meetings. A proposing member gets the privlege of speaking first to the pro side of the proposal)

Commentary: As has been shown by numerous recent Hugo ballots, but most explicitly by the 2015 ballot, the current Hugo nomination system can be too easily dominated by the largest single faction members nominating in that category. This proposal would replace the nomination tallying system with a system designed to produce proportional representation among nominators, so that every sizable group generally gets to elect a nominee, but none get to set the terms of the entire ballot.

#520 ::: J Thomas ::: (view all by) ::: April 26, 2015, 02:15 AM:

@517 Cheradinine

Yes, you can create cases that look like not what you'd expect. But this example actually does fit the goals for the surveying system.

It's better to represent 100% of the votes than 95% of them. That is one of the goals. It is not a stated goal to include the one with the most votes, though that may happen as a side effect of achieving the other goals. The one with the most votes is a strong candidate for helping to provide the most votes total -- particularly when there are many votes for just F -- and a strong candidate for helping to provide a balanced combination -- particularly when there are many votes for F combined with other members of that combination.

Consider the following example:

7 votes for A
7 votes for B
7 votes for C
7 votes for D
7 votes for E
5 votes for FG
4 votes for GH
4 votes for HI
4 votes for IF

With SDV-LPE that's

Absolute votes:
A-E 7 votes F-G 9 votes H-I 8 votes

1st round:
A 7
B 7
C 7
D 7
E 7
F 4.5
G 4.5
H 4
I 4

H and I tie and are removed.

2nd round:
A 7
B 7
C 7
D 7
E 7
F 6.5
G 6.5

F and G tie and are removed.

The winners are ABCDE, because all of the candidates with more votes were eliminated.

But this has a good side. Consider

7 votes for A
7 votes for B
7 votes for C
7 votes for D
7 votes for E
34 votes for FGHIJ

The first round, we get
A 7
B 7
C 7
D 7
E 7
F 6.8
G 6.8
H 6.8
I 6.8
J 6.8

There is a 5-way tie for the slate, and they all lose. Every nominee that got 34 votes has lost and every nominee that got 7 votes has won.

However, 35 of the people who cast have a winner and only 34 of them have no winners.

By some ways of looking at it, this is the right outcome. It just doesn't seem intuitive when you think in terms of voting systems where the one with the most votes is supposed to win.

#521 ::: J Thomas ::: (view all by) ::: April 26, 2015, 02:43 AM:

@519 Joshua Kronengold

3.8.8.1 In each round, the two works (or more, in the case of a tie) with the least total number of (fractional) votes will be compared. Of those works, all but one will be selected for elimination, removing the works that appear on the fewest number of ballots.

This needs changes for technical reasons.

In each round, the two works (or more, in the case of a tie) with the least total number of (fractional) votes will be compared.

Suppose the lowest fractional votes came out

3.75
3.5
3.5

Then there is a tie for the lowest, but no need to include the third. That one isn't so bad, and you could include the third one if you wanted to.

Of those works, all but one will be selected for elimination, removing the works that appear on the fewest number of ballots.

Say that you have two to compare, and they each have 19 votes. You will want to remove both unless as you say in the next section it's the last round and that removes too many. Not so bad, the next section sort of implies there can be ties where you'd remove both.

Say you have three to compare, and the votes are 19, 19, 17. Obviously you will remove the one with the low vote. Will you also remove the two that tied? If you leave them in and go another round, you may get a different result after that one is gone. They may not be tied any more, they may not even have the lowest fractional votes.

Similarly when the absolute votes are 36, 18, 9. Do you remove the lowest or all but the highest?

So I think there are things that are worth spelling out here, which the current wording does not spell out.

I'm not sure which way it ought to go. I tend to think that when the bottom two are tied for fractional votes there is no need to add more, and when the bottom two tie for absolute votes they should both usually be eliminated. IMHO.

I think that when there are more than one that don't have the lowest absolute vote, just remove the ones that do have the lowest absolute vote. Again IMHO.

#522 ::: J Thomas ::: (view all by) ::: April 26, 2015, 04:16 AM:

@519 Joshua Kronengold

I don't like JT's newer proposal. It's gameable in non-obvious ways (probably) and feels too clever to me.

I believe that to game it, you need to know things about the other votes that would be very hard to know. But it's new, and I don't know everything.

And SDV-LPE seems to do the job nicely for expected results and, hugley importantly, is very comprehensible both in how it operates and why it works, which is a key component to acceptance.

Would you say that this new system has hardly any chance of being accepted? I have the impression that you know a lot about the Business Meeting, and I don't. If it's too hard a sell, then I should either give up, or put some of my limited spare time into simple descriptions what it's about hoping to get it more palatable. (As well as some into checking that it performs as promised.) How much time is left? If there isn't enough time, or if people are just too impatient, I should drop it.

Also, my programmer's heart rebels at NP-hard solutions even if their N is theoretically (I'm not convinced) low enough that NP-Hard isn't non-computational.

That prejudice has occasionally been very good to me. "Don't do that, at N=35 the universe will end before you can get an answer." "We can sell it at N=20 right now. If somebody offers us a lot of money to do it for N=35 we can tell them to look elsewhere." ;-)

Also, the lack of ease of understanding makes it much harder to vet the process.

See if this makes sense:

Pick a possible solution, with five winners.

We want to include as many ballots as possible. So we count the number of ballots that have at least one of these five. (This is easy.)

We also want each of the five nominees to make a significant contribution to the whole. We don't want a combination where four of them mostly cover the field and the fifth one adds in five offball ballots which is enough to tip the combination into first place. So find a way to split up the ballots so that each nominee gets its own pile, and make the smallest pile be pretty big. Obviously the smallest pile can't be any bigger than the total number of votes for that nominee. (That lets us rule out some nominees early on, and rule out more as we get better solutions.)

There might be some combinatorics method that gives a super-easy solution for this, but a super-easy solution is not needed.

Start out with five piles, with the ballots that include more than one of the five distributed any which way among their choices. Take the total number of ballots in the piles and divide it by 5 to get the average. Find the smallest pile and the largest pile. If there are ballots in the largest pile which can go into the smallest pile, put them there but no more than will increase the smallest pile to average or reduce the biggest pile to average. Do that with all the other piles.

Find the new smallest pile and do it again. If you can't bring the smallest pile up to average, then pick a large pile and look for ways to send ballots to another small pile and send just as many from the small pile to the smallest. When there's no path left to send ballots from any large pile to the smallest using any number of intermediaries, then you're done. It doesn't take as many steps as you'd think. The size of the smallest pile gives the second number.

The intention is to get diversity, inclusiveness, and strength. We want to get input from as many different fans as we can, while we pick five works that are each of them strong.

A strong slate can expect to get one nomination. To compete for a second nomination, they have two nominees each with half the votes, and the second contributes nothing to total ballots included. A nonslate nominee may do better by contributing ballots that none of the other four have, even if it has less than half the number of votes that the slate does.

#523 ::: felice ::: (view all by) ::: April 26, 2015, 04:22 AM:

J Thomas @520: "H and I tie and are removed. F and G tie and are removed."

That looks like a good reason for not ever eliminating two works in one round. I'd suggest in the event of a tie on both fractional votes and number of ballots, that doesn't involve fifth place, randomly select one of the tied works to eliminate. Random selection isn't great, but it's better than denying both works the opportunity to increase their fractional vote totals. In this example, I think ABCFH, ABCGI, CDEFI, etc (34 or 38 satisfied) would all be better results than ABCDE (35 satisfied), or AFGHI (24 satisfied). Random elimination of ties produces a better balance between including the most popular works and satisfying as many people as possible.

#524 ::: Joshua Kronengold ::: (view all by) ::: April 26, 2015, 05:06 AM:

Re J Thomas@521: I've shown up to a bunch of BMs, and submitted at least one successful amendment and a measure that made it through passage (eviscerated, but with one important piece left unchanged), but like any political process it's hard to predict unless you've already polled the room and therefore have a decent baseline.

So while my instinct is that your "most respresentation" proposal would see a speedy death, I'm not about to claim any oracular authority. I do really hate that it specifically fails at proportional representation -- as Cheradine said, a proposal that if 95% of the con liked 7 specific works and the remaining 5% divided into roughly 4 groups that liked four other works, would ignore the remaiing 6 works and instead pick other tiny market segments is very much the opposite of what we want. You want a rough image of the field, not an attempt to leave as few people out. Proportionally is more important than diversity, and your proposal fails significantly on that score.

The deadline for new proposals is August 6. We have some time yet, although having stuff out and available for review will give more of a chance for technical fixes and for people to get used to the ideas.

Also, I'm not convinced that there will be numbers for any revision to nomination despite the proven brokenness. Splitting our efforts among different proposals will not help, particularly if there's one that enough people like enough.

Re ties and elimination, I don't like randomness. It's hard to predict and impossible to vet in these conditions. Frankly, if we're not just going to throw out all candidates in the case of a tie (and I do see some good reasons to do so), I'd rather in this one case make nomination order matter and break ties by how high in people's ballots they were placed -- and failing that, by order of nominator number, just to have an entirely reproducable process.

The right fix IMO to my wording re ties is to say "may" in the case of a tie--and rely on common sense that only if you have a three or more way tie for lowest -- or if you have a tie for second lowest -- do you include more than one item in the contest. This is a constitution, not a set of game rules, and some things can be left to sense and judgement.

#525 ::: J Thomas ::: (view all by) ::: April 26, 2015, 05:43 AM:

@523 felice

If we keep moving the goalposts we're never going to score.

SDV-LPE was not designed to satisfy as many people as possible. It was designed to put some emphasis on fractional votes (so that voters can vote for multiple nominees without getting too much power), and also to put some emphasis on popularity, on how many votes a nominee gets.

If you want to satisfy as many people as possible, then put that into the criteria the voting system should satisfy.

It's easy to make up pathological cases that demonstrate a voting system failing to meet criteria it was not designed to meet.

The original problem was that the existing nominating system was wide open to abuse by slates. SDV-LPE is not designed to interfere with slates, but it does interfere with slates by reducing the effect of ballots with multiple candidates. Since slates have to have multiple candidates, their effect is reduced.

It isn't designed to make the nominees with the most votes win, so sometimes they'll win and sometimes not. It isn't designed to maximize the number of satisfied voters, so sometimes it will do that and sometimes not.

If we decide what we want, we can probably get a voting system which will do that. But if the agreed goals are opposed or perpendicular, we may have to accept some trade-offs.

Jameson Quinn in #209 above had an explanation about two important criteria.

http://nielsenhayden.com/makinglight/archives/016206.html#4089097

#526 ::: J Thomas ::: (view all by) ::: April 26, 2015, 06:07 AM:

@524 Joshua Kronengold

I do really hate that it specifically fails at proportional representation -- as Cheradine said, a proposal that if 95% of the con liked 7 specific works and the remaining 5% divided into roughly 4 groups that liked four other works, would ignore the remaiing 6 works and instead pick other tiny market segments is very much the opposite of what we want. You want a rough image of the field, not an attempt to leave as few people out. Proportionally is more important than diversity, and your proposal fails significantly on that score.

I don't expect it to do that, though if you can find an example where it would, we can look at how likely that is.

If we take one popular work and fill in the gaps with four tiny ones, the smallest of them will not have many votes and so we cannot make a pile with a lot of votes for it. That means a low score.

If we instead have five popular works we can split up their votes so the smallest of them has a large pile of votes. You have postulated that a big fraction of the ballots will include at least one of them, too. So they ought to do fine unless they're a slate.

Also, I'm not convinced that there will be numbers for any revision to nomination despite the proven brokenness. Splitting our efforts among different proposals will not help, particularly if there's one that enough people like enough.

I am not convinced that we have reached a consensus about what it is a nominating system ought to do. Not to the point of choosing one. I haven't seen a traditional voting system as good as SDV-LPE for our needs. If it's that or something traditional, I'll support that. I'm not convinced that we should settle on one, yet.

We don't have it clear what we want or how we'd know when we found it.

The right fix IMO to my wording re ties is to say "may" in the case of a tie--and rely on common sense that only if you have a three or more way tie for lowest -- or if you have a tie for second lowest -- do you include more than one item in the contest.

Here you're talking about when to have 3 or more items to compare with SDV-LPE? Yes, that looks good. You can specify it or you can assume that whoever programs it will be sensible.

I'd rather in this one case make nomination order matter and break ties by how high in people's ballots they were placed -- and failing that, by order of nominator number, just to have an entirely reproducable process.

I don't like it to be random either. If you're willing to leave the other thing to committee judgement, you could do that with this one too.

#527 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:13 AM:

J Thomas: I think your system is interesting, and, as I've said, it's a surprisingly good system for one that is neither proportional nor as tradeoff-conscious as I'd like. But I think you should drop it. Several people have responded skeptically, and you've responded with further rationalizations.

-I stated, and still believe, that it does significantly more to encourage bullet voting than SDVLPE. You responded that SDVLPE can also encourage bullet voting. While true, that's not really a counter argument. Note that bullet voting makes the long tail problem tougher and the task for slates and other strategic voters easier.

-Others have said it's harder to understand.

-Others have said it seems less likely to pass the BM.

-It is not Droop proportional unless a is infinity.

-It is probably NP hard (though feasible in practice )

In defending your idea, you've made several statements about SDV-LPE that I'd disagree with.

I don't think that your system's upsides (mostly: clearer design goals) are enough to overcome all those objections.

#528 ::: J Thomas ::: (view all by) ::: April 26, 2015, 10:38 AM:

But I think you should drop it. Several people have responded skeptically, and you've responded with further rationalizations.

I might drop it, but I'm not ready to yet. It's clear that I have not explained it adequately, since people respond with claims about it that are not true.

I stated, and still believe, that it does significantly more to encourage bullet voting than SDVLPE.

How would you like that to be tested?

You responded that SDVLPE can also encourage bullet voting. While true, that's not really a counter argument.

Agreed. If encouraging bullet voting is a big drawback, how would we measure which one encourages bullet voting more?

Others have said it's harder to understand.

Yes, and I'm just beginning to learn how to describe it. I have the idea that my central description could be very clear. You make six piles of ballots, one for each of five nominees and one that has none of them. We want the smallest of the five to be large (showing that all of them have strong followings) and the pile that is not represented should be small.

Two goals, and they are right there in the picture.

Others have said it seems less likely to pass the BM.

That's a concern, but I've only just started trying to make it clear. If I can get it very clear, maybe it will look good. I have a suspicion that they wouldn't like it because it's new and different. But then, the same applies to SDVLPE. In the short run either one looks more likely than not to be rejected. I think that the runoff idea would do a lot to resolve the slate problem, and either of these would contribute more. But the runoff by itself would work better than SDVLPE by itself plus there is not much to explain. They have to do something, and if they do the minimum it will probably be that.

More than push this one alternative, I want to get it straight what criteria a nominating system should meet. If we get the criteria clear we can probably get a voting system that will meet them.

It is not Droop proportional unless a is infinity.

Should it be? Why should it be? Droop sets a quota for the number of votes a nominee should have to pass. This one maximises the number of votes the lowest candidate has, subject to also satisficing number of total votes represented.

It is probably NP hard (though feasible in practice

This is a red herring. I should not have mentioned it. I don't even know that it is NP hard, I only imagined it might be. If it is NP hard maybe somebody else should take the time to show that it is.

In defending your idea, you've made several statements about SDV-LPE that I'd disagree with.

These were not defending my idea, they were attacks on SDVLPE. My intention was to show that this sort of handwaving attack did not show much. It would probably be good though to show where I have gone wrong about SDVLPE. If my claims are wrong I will apologize. If they are right then maybe they can be palliated, or they may turn out to be right but not that important.

I don't think that your system's upsides (mostly: clearer design goals) are enough to overcome all those objections.

You may be right. If everybody's too impatient to consider it then I'll just slink away.

I think the design goals are central, and I want to get those clear, and so far mostly no one has cooperated on that. If nobody's interested in doing that then I'll slink away on that one too.


#529 ::: J Thomas ::: (view all by) ::: April 26, 2015, 11:12 AM:

What Goals Should A Nominating System Have?

Candidate goals:

Slates should tend to lose.

The nominees with the most votes should win.

The system should not encourage bullet voting or any other voting strategy.

It should be Droop-proportional whether it is a proportional voting system or not. (Is that too snarky?)

Every winner should be pretty popular. (This is a variation on "biggest vote is best vote".)

It should be politics-free and should not encourage bloc voting. People should always vote for what they personally think is best regardless of whether they think it can win. (Did I mention that it should discourage slates?)

If we had a way to estimate voter satisfaction (like how satisfied they are to have one win, or two wins, up to five wins), then we would want to maximize total voter satisfaction.

If we had a way to estimate voter satisfaction, then we would want to minimize the variance in voter satisfaction. It's better for everybody to have one win than for a small group to have 5 wins, even if the total satisfaction comes out bigger when the small group has a giant victory over everybody else. (This is another way to say we want to discourage slates. Maybe we should remove the slate goal and just use this one?)

We want a way to deal with ties that is not arbitrary. We never want to have a tie where one is accepted and the other is not, and we can't explain why it happened that way.

Have I left out a goal?

#530 ::: Cassy B. ::: (view all by) ::: April 26, 2015, 11:38 AM:

J Thomas @529, we want Joe and Jane Average Fan to understand the process at least enough to trust that it's fair, even if they don't get the nuts-and-bolts of it.

#531 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 11:57 AM:
More than push this one alternative, I want to get it straight what criteria a nominating system should meet. If we get the criteria clear we can probably get a voting system that will meet them.

And that... is why you fail. In this world, there is no "do or do not", there is only "try".

That is to say: if you explicitly maximize non-strategic criteria in a voting system, you generally make it easy for strategic voters to take advantage of that maximization, and often for them to outright subvert it.

SDV-LPE is designed with essentially the same goals as your system. I know, because I designed it. The difference is, I was also keeping an eye on minimizing strategic exploitability, and on computability, and on ease of explanation (not only in terms of answering the question "how does this work", but also the question "why did X lose?").

If you want to help us make sure we're not overlooking the best system, you need to look at plausible scenarios with an open mind, not tell more stories about why your system is best. I'm satisfied in my own mind that it's not really better. And yes, it's not entirely fair of me that I'm not willing to put in the work of explaining all the scenarios I've considered in my head, but as long as it's you against everyone else, and as long as you seem to be in defensive mode, I'm not sure that doing that explaining would have enough chance of changing many minds. I'm sorry, I know that must seem infuriating. If you can convince somebody else that your system is probably the best option, I'd put more work into explaining what I think about it.

#532 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 12:25 PM:

David@461: I ran the sim you requested, and the results were A,B,C,D,V where V was the most popular slate choice. A-D were the most popular non-slate choices. So, in this case, I think SDV-LPE did pretty well.

All: I'm going through the list and organizing the modifications that have been suggested. Once we've reached concensus, I'll type it it in a similar format to what Joshua suggests.

Thanks,
Kilo

#533 ::: J Thomas ::: (view all by) ::: April 26, 2015, 12:55 PM:

@531 Jameson Quinn

That is to say: if you explicitly maximize non-strategic criteria in a voting system, you generally make it easy for strategic voters to take advantage of that maximization, and often for them to outright subvert it.

If you can clearly say what it means for strategic voters to subvert the system, then we can design a system with thought to minimizing that.

Until then, we'll still just be floundering.

For example, people talk like it's a bad thing when voters make only one choice. But for a voter who only cares about his one choice winning, that's the right thing for him to do. So we want a voting system that won't give him inappropriate power when he does that.

But we probably don't want a system that discourages him from doing it, because we don't want him to load up on choices he doesn't really want so he can get special benefits by avoiding bullet voting.

Would it be fair to say that in general we don't want to give particular voters undue power?

Slate voters get undue power in the existing system because by working together they can get most of the wins even when they're only 20% or less of the voters.

Bullet voters get undue power in some systems because their single votes count too much.

Specify what is undue power, and we're ready to roll.

If you want to help us make sure we're not overlooking the best system, you need to look at plausible scenarios with an open mind, not tell more stories about why your system is best.

Hey, my system *is definitely the best at the two criteria it was designed to balance. That's built in, the same way that acceptance voting is the best when you define the goal as acceptance voting.

Now people are adding an ill-defined collection of other criteria. That's fine, it probably isn't the best at criteria it wasn't designed to handle. Let's define the criteria we care about, and then look at how to achieve them.

And yes, it's not entirely fair of me that I'm not willing to put in the work of explaining all the scenarios I've considered in my head

No, it's not fair at all. Not that I want to prove that this particular system will be best at all the random criteria people come up with. But we need to set up the specifications we want to build a system for, not just come in with random new requirements after we have one we don't like.

Does this remind you of the bad old days of software development? Management says they want something new. It should use genetic algorithms -- no, make that simulated annealing -- to help investors pick stocks. Oh, and quaternions. It should have a database of the stocks they're considering, and a spreadsheet they can use to calculate with, and an internet browser they can use to look up stuff, and a word processor they can make notes with, and something like PowerPoint so they can present their results to other people. It should have enough encryption that they'll feel like their data is safe. Never mind the details, just do it.

So you come up with a demo and ask them if that's what they wanted, and they say "No, the simulated annealing interface is too hard for investors to understand, and the spreadsheet looks too much like Excel but the wordprocessor doesn't look enough like Word. And the borders ought to be green and yellow." You point out that the user can set the borders to whatever colors he wants. "They need to be green and yellow."

So you make another demo, and another, and eventually you have a demo that they think they like. "OK, that's it. We'll ship that." And you know it's time to dust off your CV...

There's no reason to expect an ad hoc group to be organized, but of course until we agree about the specs we'll keep flopping around.

You might persuade people without testing that your pet system is good, but if they accept it and then they use it at WorldCon and don't like it, your name will be on it.

#534 ::: J Thomas ::: (view all by) ::: April 26, 2015, 01:01 PM:

What Goals Should A Nominating System Have?

Candidate goals:

It should be easy to explain, so that average fans will trust that it's fair even if they don't understand the details.

Slates should tend to lose.

The nominees with the most votes should win.

The system should not encourage bullet voting or any other voting strategy.

There is a minimum number of votes that every winner must get.

Every winner should be pretty popular. (This is a variation on "biggest vote is best vote" and also on "Every winner must be so big".)

It should be politics-free and should not encourage bloc voting. People should always vote for what they personally think is best regardless of whether they think it can win. (Did I mention that it should discourage slates?)

If we had a way to estimate voter satisfaction (like how satisfied they are to have one win, or two wins, up to five wins), then we would want to maximize total voter satisfaction.

If we had a way to estimate voter satisfaction, then we would want to minimize the variance in voter satisfaction. It's better for everybody to have one win than for a small group to have 5 wins, even if the total satisfaction comes out bigger when the small group has a giant victory over everybody else. (This is another way to say we want to discourage slates. Maybe we should remove the slate goal and just use this one?)

We want a way to deal with ties that is not arbitrary. We never want to have a tie where one is accepted and the other is not, and we can't explain why it happened that way.

Have I left out a goal?

#535 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 01:38 PM:
Does this remind you of the bad old days of software development?

No. As far as I can tell, nobody's asking you to add bells and whistles, because people seem to be satisfied with SDV-LPE.

I understand you're not satisfied, because it's not clear to you why we would want the thing that SDV-LPE is explicitly maximizing, and you think that should be clear. But if you want others to be dissatisfied, show them why SDV-LPE doesn't meet worthwhile goals in some realistic situation. In this case, unlike the case of your software-design memories, I think the burden of proof really should be on you.

As for your list of goals: it's pretty complete. However, I think there's one you missed. You mention not encouraging bullet voting "or any other strategy". I also think the system should make broad voting (listing relatively many candidates) easy and strategically decent, insofar as that's possible without encouraging slates or "dummy" votes for unserious candidates.

My experience tells me that your urge to explicitly maximize is pretty much at it's limit with the system you've designed; and that by trying to explicitly pursue yet more goals, you're going to find the pitfalls multiplying. But I could be wrong; as I said, your proposal is already better than I would have expected likely to come from your design attitude.

I honestly wish you luck. I do know that if you do succeed at making a better system than SDV-LPE and convincing people you've done so, it's not going to be because I've carefully tried to shoot it down and you've responded in detail, but because you yourself have carefully tried to shoot it down.

#536 ::: P J Evans ::: (view all by) ::: April 26, 2015, 01:41 PM:

534
That sounds reasonable to me.

#537 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 02:17 PM:

Okay, as promised, here are the discussion points that have been made regarding SDV-LPE so far. Please forgive the inevitable typos, as this is consolidated from my raw notes.


1. Felice@462: Move the guiding principles of the Hugo Award (3.3.1) to the commentary rather than the proposal itself.
2. Felice@462: Move the “no slates allowed rule” (3.7.4) to the commentary rather than the proposal itself.
3. Felice@462: Modify the current 3.7.1 to fix the phrase “up to five (5) equally weighted nominations in every category”
4. David Dyer-Bennet@463: We can’t claim “TNH and PNH had absolutely no input in the discussions, however.” Nor should we distance ourselves from TNH/PNH. [Kilo’s comment: I contend that their input into SDV-LPE itself has been minimal to non-existent, but I realize that’s not the same thing as no input in the discussions.]
5. Soon Lee@464: Include URLs to the discussion in the commentary section so that people can see how the system was developed
6. J Thomas@465 (see also Steve Halter@474):If there is a tie for the lowest-scoring works, then don’t consider the second-lowest work(s) for elimination. [Kilo’s comment: This is already implemented in the current SDV-LPE code, but a better description needs to be written.]
7. David Harmon@467: Typo in 3.8.1.4 In the event that more than two or more works
8. Kilo: Typo -- FAQ#10 should say “fractional” not “factional”
9. Mercy@468: What happens if something is nominated in the wrong category?
10. nathanbp@470: What do we do if a nominee withdraws?
11. dotless i@472: In FAQ#13 point out the difference between a good system for choosing representatives from various political factions and choosing a final Hugo nomination list, given the diffuse distribution of favorite works.
12. dotless i@472: In FAQ#10 point out that the Hugo voting system already uses a system of eliminating lowest-ranked contenders and rescoring ballots until there is a winner.
13. Stewart@473: 3.3.1.1 and 3.3.1.2 are inconsistent: enjoyable and great are imperfectly correlated in literature
14. Jameson Quinn@475 (see also dotless i@477): Expand number of works that can be nominated beyond five
15. Jameson Quinn@475: Soften the language of commentaries 1b and 2d so that voters aren’t encouraged to only nominate one work [Kilo’s comment: I don’t yet see that as a bad thing in the sims I’ve run, but there are likely several scenarios I haven’t tried.]
16. Jameson Quinn@480: Have an initial “blanket elimination round” to eliminate works that have no chance of winning
17. Jameson Quinn@485: Remove the 5% rule
18. Jameson Quinn@487 In FAQ#13, change “every type of voting system” to “every applicable type of voting system”
19. Jameson Quinn@487: Identify Jameson’s organization in the FAQ
20. Felice@490: Highlight that as works on your ballot get eliminated, the remaining works on your ballot count more; if there is only one of your choices remaining, it gets your full vote.
21. felice@490: If there are more than three works tied for fifth place, eliminate all of them, giving a range of 4 to 7 on the ballot.
22. J Thomas@495: New FAQ#16 explaining nomination process
23. felice@497 (see also@523): Random elimination when ties occur
24. J Thomas@498: Eliminate number of nomination ties by eliminating work that overlaps the most with other winners [Kilo’s comment: I’m not entirely sure I understand this, so may have summarized it incorrectly.]
25. Joshua Kronengold@500 (see also @519): Reformatted proposal with strikeouts [Kilo’s comment: I haven’t completed gone through each section to see if anything new has been added.]
26. Joshua Kronengold@500: At least one of the proposal writers should present it at the business meeting. [Kilo’s comment: I hadn’t planned on going to Worldcon this year, but I can, particularly since I can run any sims on the fly that might be requested. It would be great if one of our experts could go as well…]
27. J Thomas@503: Discard ties, even if that leave only three works on the ballot
28. felice@504 (see also P J Evans@507): Under current rules, if someone nominates AAABC, it is counted as ABC; how should SDV-LPE count this? [Kilo’s comment: My thoughts are to change as little of the existing system as possible, to ease passage at the business meeting.]
29. felice@504: Under current rules, works in the wrong category are moved to the correct category unless that would cause it to extend beyond the 5-work limit; how should SDV-LPE count this? [Kilo’s comment: Again, better to change only what is absolutely needed, IMO.]
30. felice@504 (see also J Thomas@508): 3.8.7 in the current rules is broken. [Kilo’s comment: If so, that should probably be brought up as a second proposal.]
31. Kevin Standlee@505: Avoid touching the 5% rule to prevent issue with other proposals.
32. Joshua Kronengold@524: Deadline for proposals is August 6.
33. Joshua Kronengold@524: Opposed to randomness in any form
34. Joshua Kronengold@524: Change wording to “may” remove ties and leave it to [someone] to judge when it is required [Kilo’s comment: I’m still pretty much opposed to any human intervention, just because the thought that it exists is what led to this in the first place.]
35. J Thomas@529 (see also@534): Goals for a nomination system
36. Cassy B@530: Nomination system should be easily understandable at a general level by average fans
37. Jameson Quinn@535: Broad voting should be easily strategic without encouraging slates or votes for unserious candidates.

Please note, I have only included commentary on SDV-LPE. JT’s new system is worth looking at as well, but I haven’t considered that here.

Can we start narrowing down the list of changes that we think are important? I do think that the less we change the current system (and, as some have mentioned, point out how it is similar to the current voting system), the easier it will be to garner support in the business meeting.

Secondly, who here is planning or willing to go to Worldcon to present the proposal?

Thanks,
Kilo

#538 ::: J Thomas ::: (view all by) ::: April 26, 2015, 02:28 PM:

revised.

Candidate goals:

1. It should be easy to understand, so that average fans will trust that it's fair even if they don't understand the details.

2.
-------------
The nominees with the most votes should win.

There is a minimum number of votes that every winner must get.

Every winner should be pretty popular.

If we had a way to estimate voter satisfaction (like how satisfied they are to have one win, or two wins, up to five wins), then we would want to maximize total voter satisfaction. We don't have a reliable way to measure voter satisfaction, and this theoretical statement basicly says that the nominees with the most votes should win (and if the ballots are rank-ordered then the higher ranks should count more).

These four are different ways to address similar issues. Basically, we don't want nominees to win even in fifth place unless they at least get "enough" votes.
------------

3. If we had a way to estimate voter satisfaction, then we would want to minimize the variance in voter satisfaction. It's better for many nominators to have one win each than for a small group to have 5 wins, even if the total satisfaction is bigger when the small group has a giant victory over everybody else.

This basicly says that votes should count more when they're spread out among ballots. Better to have more ballots with winners than fewer ballots with winners.

4. The system should not encourage voting strategy. Specific voting strategies we have thought of include bullet voting, slate voting, and adding unserious nominations to improve the chance that your real nomination will win (shotgun voting?). But while we don't want to reward adding extra nominations to ballots, we also want to do nothing to punish additional nominations. We want nominators who think in terms of strategy to feel they have little to lose by adding additional nominees that they think deserve the Hugo.

5. We want a way to deal with ties that is not arbitrary. We never want to have a tie where one is accepted and the other is not, unless we can show a good reason for it.

Have I left out a goal?

#539 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 02:59 PM:

@537: Wow, good job.

An attempt at categorization:

Minor uncontroversial fixes:

3. Felice@462: Modify the current 3.7.1 to fix the phrase “up to five (5) equally weighted nominations in every category”
7. David Harmon@467: Typo in 3.8.1.4 In the event that more than two or more works
8. Kilo: Typo -- FAQ#10 should say “fractional” not “factional”
25. Joshua Kronengold@500 (see also @519): Reformatted proposal with strikeouts [Kilo’s comment: I haven’t completed gone through each section to see if anything new has been added.]
18. Jameson Quinn@487 In FAQ#13, change “every type of voting system” to “every applicable type of voting system”

Handling special cases:


6. J Thomas@465 (see also Steve Halter@474):If there is a tie for the lowest-scoring works, then don’t consider the second-lowest work(s) for elimination. [Kilo’s comment: This is already implemented in the current SDV-LPE code, but a better description needs to be written.]
9. Mercy@468: What happens if something is nominated in the wrong category?
10. nathanbp@470: What do we do if a nominee withdraws? [JQ's comment: this is important. I'd suggest the simplest answer is just to add in the last to be eliminated, LOFI style, as long as the list is under 5 works and any absolute threshold is passed.]
21. felice@490: If there are more than three works tied for fifth place, eliminate all of them, giving a range of 4 to 7 on the ballot.
23. felice@497 (see also@523): Random elimination when ties occur
24. J Thomas@498: Eliminate number of nomination ties by eliminating work that overlaps the most with other winners [Kilo’s comment: I’m not entirely sure I understand this, so may have summarized it incorrectly.]
27. J Thomas@503: Discard ties, even if that leave only three works on the ballot
28. felice@504 (see also P J Evans@507): Under current rules, if someone nominates AAABC, it is counted as ABC; how should SDV-LPE count this? [Kilo’s comment: My thoughts are to change as little of the existing system as possible, to ease passage at the business meeting.]
34. Joshua Kronengold@524: Change wording to “may” remove ties and leave it to [someone] to judge when it is required [Kilo’s comment: I’m still pretty much opposed to any human intervention, just because the thought that it exists is what led to this in the first place.]

Simplifying administrative burden:

16. Jameson Quinn@480: Have an initial “blanket elimination round” to eliminate works that have no chance of winning


FAQ and explanatory improvements:


4. David Dyer-Bennet@463: We can’t claim “TNH and PNH had absolutely no input in the discussions, however.” Nor should we distance ourselves from TNH/PNH. [Kilo’s comment: I contend that their input into SDV-LPE itself has been minimal to non-existent, but I realize that’s not the same thing as no input in the discussions.]
5. Soon Lee@464: Include URLs to the discussion in the commentary section so that people can see how the system was developed
11. dotless i@472: In FAQ#13 point out the difference between a good system for choosing representatives from various political factions and choosing a final Hugo nomination list, given the diffuse distribution of favorite works.
12. dotless i@472: In FAQ#10 point out that the Hugo voting system already uses a system of eliminating lowest-ranked contenders and rescoring ballots until there is a winner.
15. Jameson Quinn@475: Soften the language of commentaries 1b and 2d so that voters aren’t encouraged to only nominate one work [Kilo’s comment: I don’t yet see that as a bad thing in the sims I’ve run, but there are likely several scenarios I haven’t tried.]
19. Jameson Quinn@487: Identify Jameson’s organization in the FAQ
22. J Thomas@495: New FAQ#16 explaining nomination process
20. Felice@490: Highlight that as works on your ballot get eliminated, the remaining works on your ballot count more; if there is only one of your choices remaining, it gets your full vote.


Philosophy and goals

1. Felice@462: Move the guiding principles of the Hugo Award (3.3.1) to the commentary rather than the proposal itself.
2. Felice@462: Move the “no slates allowed rule” (3.7.4) to the commentary rather than the proposal itself.
13. Stewart@473: 3.3.1.1 and 3.3.1.2 are inconsistent: enjoyable and great are imperfectly correlated in literature
33. Joshua Kronengold@524: Opposed to randomness in any form
35. J Thomas@529 (see also@534): Goals for a nomination system
36. Cassy B@530: Nomination system should be easily understandable at a general level by average fans
37. Jameson Quinn@535: Broad voting should be easily strategic without encouraging slates or votes for unserious candidates.

Points of order

26. Joshua Kronengold@500: At least one of the proposal writers should present it at the business meeting. [Kilo’s comment: I hadn’t planned on going to Worldcon this year, but I can, particularly since I can run any sims on the fly that might be requested. It would be great if one of our experts could go as well…]
32. Joshua Kronengold@524: Deadline for proposals is August 6.

Related changes to existing rules:

14. Jameson Quinn@475 (see also dotless i@477): Expand number of works that can be nominated beyond five
17. Jameson Quinn@485: Remove the 5% rule
29. felice@504: Under current rules, works in the wrong category are moved to the correct category unless that would cause it to extend beyond the 5-work limit; how should SDV-LPE count this? [Kilo’s comment: Again, better to change only what is absolutely needed, IMO.]
30. felice@504 (see also J Thomas@508): 3.8.7 in the current rules is broken. [Kilo’s comment: If so, that should probably be brought up as a second proposal.]
31. Kevin Standlee@505: Avoid touching the 5% rule to prevent issue with other proposals.


...

Looking at this, there's one thing I've been thinking about, that I see I probably haven't said out loud in relation to the specific proposal. I think it's worthwhile including a rule that expands the number of nominations in cases of near-ties. For instance: "3.8.1.6 When there are only 6 candidates remaining, works will be considered 'tied for appearing on the fewest number of ballots' if the number of ballots they appear on differs by 2 or less."

#540 ::: J Thomas ::: (view all by) ::: April 26, 2015, 03:07 PM:

@537 Keith "Kilo" Watt

24. J Thomas@498: Eliminate number of nomination ties by eliminating work that overlaps the most with other winners [Kilo’s comment: I’m not entirely sure I understand this, so may have summarized it incorrectly.]

Here's the idea. We're having various problems with ties. Sometimes we would do better if we can avoid ties, particularly when the result of a tie is that we have too few or else too many final nominees.

One possible way to break a tie is to look back at the fractional votes. They might have had the same fractional vote or maybe not. If the fractional votes are different we could discard the one with the lower fractional vote and then go on to the next round.

I was thinking that the votes could be the same and the fractional votes the same and something else related be different, but now I think that isn't possible. If A and B both have fractional vote 2.5 but A has A A AC while B has B B BCDE BCDE they will have different total votes. It's rare that you could have A AC AC AC and B B BCDE BCDE and have both the votes and the fractional votes come out the same. It just wouldn't help enough to distinguish between those.

But OK, we can benefit from other ways to choose between ties. Here's one way -- for each ballot, count the total number of votes on that ballot including votes for nominees that have already lost.

So if A had A A A A A while B had BCDEF BDFHI BCGJM BEKLN BFILM then I think we could consider B the winner. I'd like to subtly reward extra votes. But of course, this subtly rewards extra votes for dummy candidates that have no chance. So it isn't that good.

If the top seven are currently ABCDEFGH, and A and B are tied for elimination, maybe we could reward the one whose ballots also have more of CDEFGH. It's a way to reward nominating additional serious candidates. One side effect of that is that we'd eliminate the one that has less effect on the other nominees for the next round. I don't know whether that's good or bad.

It might seem like a somewhat arbitrary rule for breaking ties. But an explicit rule whose side-effects are known might sometimes be better than the alternatives of making it random, leaving it up to individual judgement, or treating both the same.

#541 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:13 PM:

Jameson@539: Thanks for the organization; my notes were simply in chronological order.

All: To my thinking, here are the major questions that need to be answered, roughly in order of importance:

1. How do we handle ties, in both points received and number of nominations received?

2. How do we handle refused nominations?

3. How do we handle works placed in the wrong category?

4. Should we expand the maximum number of nominations?

5. Should we change or eliminate the 5% rule?

6. Should we include a definition of what the Hugo nomination system (including an explicit "no slates" rule) in this proposal, or should that be proposed separately?


Most of the rest come under language and FAQ clarifications, so we could deal with them separately in successive drafts after these more fundamental questions have been answered.

Kilo

#542 ::: P J Evans ::: (view all by) ::: April 26, 2015, 03:21 PM:

540
people should not be putting the same item more than once in a category - that's a Bad Thing. It's going to be looked at ballot-box stuffing. (It's not likely to happen, but don't encourage it by letting people use it to get their favorite more than one try at being selected!) Just say 'duplicate nominees on a ballot won't be counted'.

#543 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:22 PM:

And now, my opinions:

1. As long as it's -simple- and doesn't lead to odd cases, I'm open to almost anything (other than randomness -- I don't think that will be well-received by the Hugo voters). I think some minor variation of JT's tie system for SDV-LPE (if there is a tie in the lowest points, don't consider the second-lowest points works for elimination) is the best way to go. I would support a simple system with some rare odd results over a complicated system that -might- be bulletproof. Whatever the group decides, I'm pretty sure I can code it.

2. The code reports the results of each round of elimination. Just put the work(s) that were eliminated in the previous round back in. If there's interest in seeing a sample output, just let me know and I'll post it.

For the rest, my thoughts are to leave the current rules unchanged -for this proposal-. None of those issues are essential to SDV-LPE -- it will work either way. I suggest that if we want those things changed, they be done as a separate proposal. My guiding principle is that the fewer changes to the current rules are made, the more likely it is to pass the business meeting as well as be accepted by fandom.

Kilo

#544 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:25 PM:

JT@540:

So if A had A A A A A while B had BCDEF BDFHI BCGJM BEKLN BFILM then I think we could consider B the winner. I'd like to subtly reward extra votes.

Okay, I think I see what you're saying, but I actually feel just the opposite way. To my mind, if someone felt so strongly about a work that it was the only one they nominated, then that should be weighted more heavily (in this case) than someone who ranked it equally with 4 other works.

K

#545 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 03:32 PM:

PJ@542:
(It's not likely to happen, but don't encourage it by letting people use it to get their favorite more than one try at being selected!) Just say 'duplicate nominees on a ballot won't be counted'.

Just a point of clarification, listing the same work multiple times won't give them more than one try. If a work is eliminated, then it will be wiped from all instances in which it appears on a ballot. What it does do is let them give more of their vote to one work than another. For example, with ABC, each work gets 1/3 of a point. With AAABC, work A gets 3/5, work B gets 1/5 and work C gets 1/5. While that might be considered a feature instead of a bug, I still don't really like it. I really don't want to encourage any sort of games with the nomination process, even ones that don't necessarily hurt anything.

Kilo

#546 ::: P J Evans ::: (view all by) ::: April 26, 2015, 03:38 PM:

545
That's why I don't like it - and in the past, duplicates like that (when they've occurred) haven't been counted because it's gaming the system in ways that are wrong. It doesn't fly with me.

#547 ::: J Thomas ::: (view all by) ::: April 26, 2015, 04:04 PM:

@544 Keith "Kilo" Watt

"So if A had A A A A A while B had BCDEF BDFHI BCGJM BEKLN BFILM then I think we could consider B the winner. I'd like to subtly reward extra votes."

Okay, I think I see what you're saying, but I actually feel just the opposite way. To my mind, if someone felt so strongly about a work that it was the only one they nominated, then that should be weighted more heavily (in this case) than someone who ranked it equally with 4 other works.

If we want to use this as a criterion for breaking ties, it can work either way. We just need an explanation that sounds good.

I would prefer to count the votes for surviving nominees instead of all votes, because those are more serious contenders than the ones that are already eliminated.

Another possibility is to go back and look at the points. A lot of the time the points will not be the same.

When it's the bottom two that are not tied on points, but are tied on votes, that's easy. When it's three or more that are not all tied on points, likewise. It's only when you get two or more tied for last place on points and then tied on votes too, where eliminating them both (all) results in too few winners but leaving them both (all) results in too many winners, that it's an issue. So that obvious, sensible rule would take care of most of the troublesome cases.

I kind of like eliminating all the tied nominees unless it causes a problem, because that would be more likely to hurt slates than anything else. But that's only a preference.

Also, when it's two items from a slate that are tied, probably every rule we might use to pick one from the other will fail. They'll be as similar as clones in a vat, we could count the letters in their names or put them in alphabetical order or something.

#548 ::: J Thomas ::: (view all by) ::: April 26, 2015, 04:21 PM:

5. Should we change or eliminate the 5% rule?

That could and probably would be a separate proposal. The voting system can live with the 5% rule or do without it.

In theory, I think the 5% rule probably should be changed. Currently small categories have trouble getting 5 candidates that qualify, maybe because they don't get a lot of votes but they do have a lot of potential nominees. But I have some vague theoretical reasons to think that for the Hugo nominations, larger samples may have bigger variance than smaller samples. It might turn out in the future as the number of voters goes up to 3000, 5,000 etc, it might get harder to have 15 nominees or even 5 that each get 5% even for big categories.

I'm not ready to stand behind that prediction, but *maybe* the 5% rule simply won't keep working as well as it has so far. We probably don't need it.

An intermediate runoff among the top 15 would probably give you nominees with more than 5%.

The 5% rule expresses a worthy goal -- we want winners that are not tiny minorities. But the rule itself may not be an adequate way to get the result the rule is supposed to enforce. If we do have other ways that do reliably get the result we want, then we don't need the rule to require them to get that result.

#549 ::: Keith "Kilo" Watt ::: (view all by) ::: April 26, 2015, 04:33 PM:

JT@547

I would prefer to count the votes for surviving nominees instead of all votes, because those are more serious contenders than the ones that are already eliminated.

This is actually how it works now. Basically each round starts completely from scratch, except that at least one work has been removed from both the ballots and the list of potential works that have been nominated. It has to be this way, since we have to recalculate the points for each work every round (because a ballot that originally gave a work 1/5 of a point now may be giving it 1/4 of a point, etc.). Once a work is eliminated, it has no further effect on the process.

Kilo

#550 ::: felice ::: (view all by) ::: April 26, 2015, 04:50 PM:

J Thomas @525: "It's easy to make up pathological cases that demonstrate a voting system failing to meet criteria it was not designed to meet."

I don't think this is anything to do with criteria; rather, certain rare ties can break the intended functioning of SDV-LPE.


Keith "Kilo" Watt @537: "My thoughts are to change as little of the existing system as possible, to ease passage at the business meeting."

Sensible. So we're sticking with the limit of five nominations per category?


P J Evans @546: "545 That's why I don't like it - and in the past, duplicates like that (when they've occurred) haven't been counted because it's gaming the system in ways that are wrong. It doesn't fly with me."

Under the current counting system, yes, it would be gaming the system if it wasn't prohibited. But under SDV-LPE, there's nothing wrong with it at all, and it's not reasonable to criticise it for what effects it used to have. But as Kilo says, change as little as possible, so we should stick with the current rule on duplicates within a ballot.

#551 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 04:52 PM:

J Thomas? Drop it.

The next few times you post, I'd like to see you concentrate on discussing other commenters' opinions.

Ignore this at your peril.

#552 ::: P J Evans ::: (view all by) ::: April 26, 2015, 04:57 PM:

550
The one year I had my fingers actually in that pie, it wasn't a problem. Most people aren't likely to try it. (And really, it isn't a winning move.)

#553 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 05:36 PM:

Can we have a round of applause here for Joshua Kronengold @519?

Because even the best threads are better if there's poetry.

#554 ::: J Thomas ::: (view all by) ::: April 26, 2015, 06:09 PM:

@551 Teresa Nielsen Hayden

The next few times you post, I'd like to see you concentrate on discussing other commenters' opinions.

Drop which? One alternate voting proposal? The attempt to get a consensus about criteria that voting proposals should meet? Responses to Keith Watt's open questions about how to set up a particular proposal?

Please be more specific. I want to honor your opinion about this, and I don't see what you mean.

#555 ::: felice ::: (view all by) ::: April 26, 2015, 06:14 PM:

Joshua Kronengold @519: "When I was just a young fan"

*applauds*

(Would have done so when originally posted, but I was unfamiliar with the tune so didn't fully appreciate the filk)

#556 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 06:39 PM:

J Thomas @554: Huh. I didn't realize that the concept might be unfamiliar to you. Take a time out from the thread while I think about this.

#557 ::: felice ::: (view all by) ::: April 26, 2015, 07:03 PM:

Jameson Quinn @539: "When there are only 6 candidates remaining, works will be considered 'tied for appearing on the fewest number of ballots' if the number of ballots they appear on differs by 2 or less."

Ties are bad; we don't want to introduce more of them. Sucks to be 6th by two points, but what if 7th, 8th, and 9th are only one point behind 6th? If we let 6th in, they've got even more cause for complaint. But the near-ties issue isn't specific to SDV-LPE; if you think it's a good idea, you can propose it separately.


J Thomas @547: "when it's two items from a slate that are tied, probably every rule we might use to pick one from the other will fail. They'll be as similar as clones in a vat, we could count the letters in their names or put them in alphabetical order or something."

How about all else being equal, eliminating the last one to appear in the database? So whoever nominated first wins, and if they nominated all the tied works, the one(s) they listed first win. Technically breaks the "equally weighted" bit in 3.7.1, and theoretically encourages nominating as early as possible, but I don't think in practice anyone's going to care enough to affect behaviour.

#558 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:03 PM:

@556: In JT's defense: he really was carrying on at least three threads, and was only being stubborn in one and self-centered in at most two. I'm not going to tell you you're being too harsh; it's your blog. But I think you could at least be more specific.

#559 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:12 PM:

And I've no intention of putting myself above anyone here. 33% stubborn and 67% self-centered would probably be better than my usual averages.

#560 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 26, 2015, 08:18 PM:

Jameson: That could be a useful distinction. In what part of the discussion would you say he's being most helpful and responsive?

#561 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:31 PM:

For instance, @547 was very much a responsive part of the general conversation, with no hint of going off on any hobby horses.

#562 ::: Jameson Quinn ::: (view all by) ::: April 26, 2015, 08:34 PM:

@557: You're probably right. I think that it would be possible to make a good rule to minimize "almost made it" cases, but it's not related to the main problem here, and so it should probably be a separate proposal.

#563 ::: Kevin Standlee ::: (view all by) ::: April 26, 2015, 11:21 PM:

J Thomas @548:

Should we change or eliminate the 5% rule?
It's already on the agenda under the title "The Five Percent Solution" (that was my suggestion when the maker was looking for a short title). I'd really prefer that proposal be debated separately from other proposals. Remember that people need to understand what they're voting on. If they're confused, they'll probably vote it down on principle. (I know I would.) Loading too many subjects into a single proposal confuses people.

#564 ::: Joshua Kronengold ::: (view all by) ::: April 27, 2015, 03:02 AM:

Felice: This youtube is a good rendition of the base Sondheim tune. It's a saucy song, but not dirty: https://www.youtube.com/watch?v=4mf33LcIe1A

Theresa: Thanks! *bows*

I think JT has been a reasonably good citizen recently except regarding his new pet proposal.

I like a first tiebreak of score.

I -prefer- lowest sum of placement for 2nd tiebreak, because despite now making placement matter, it still doesn't matter much, and it's an actually meaningful tiebreak.

For third tiebreak, I favor something entirely unambiguous, since as we've determined, SDV produces much fairer results if ties are broken than if all tied works are eliminated. We've exhausted meaningful tiebreaks at that point, so I'd favor "the item appearing first on a ballot going chronologically" (so the item appearing first on the earliest ballot where any tied item appears).

I do think we should keep the rule that when we get a tie that includes 5th place, we should keep all tied items. This means we're using our slightly terrible (except for #1) tiebreak only in cases where we -have- to break a tie. But otherwise, we do have to break the tie if we want to keep the election proportional.

I originally proposed eliminating all tied votes because it hurts slates more than other things -- but it would have unintended consequences, and frankly we don't -need- to hurt slates that much, and randomly, while also punishing non-slate candidates for being randomly involved in a tie. So I repent of that notion.

#565 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 06:53 AM:

I do think the "eliminate all members of a ballots tie" works well. It naturally leads to allowing 6 or more nominees. And I think it would literally never hurt a non-slate candidate. If you save one of the two using a tiebreaker, the only way that candidate avoids elimination in the very next round is if it had a significantly elevated overlap with the work it tied with. Thus, both the tie and the ability to be helped by a tiebreaker are evidence of slates at work; each weak evidence on its own, but together reasonably strong.

#566 ::: J Thomas ::: (view all by) ::: April 27, 2015, 07:20 AM:

@558 Jameson Quinn

Thank you. I feel like I owe you one.

@565

I do think the "eliminate all members of a ballots tie" works well. It naturally leads to allowing 6 or more nominees.

This is a feature or a bug, according to opinion.

If you save one of the two using a tiebreaker, the only way that candidate avoids elimination in the very next round is if it had a significantly elevated overlap with the work it tied with.

This is a picky point, but not necessarily. You could have had two contestants both with low points due to many votes on the ballots, and the next one up has a lot of bullet votes and a low absolute vote count so it loses next time. If you disagree I could write up a scenario for it, unless I'm wrong. But I don't see that there's a lot riding on it.

Thus, both the tie and the ability to be helped by a tiebreaker are evidence of slates at work; each weak evidence on its own, but together reasonably strong.

I tend to agree with you on this. We can find examples where it isn't so, but if there's a slate it will usually be the slate that creates this situation.

Sometimes we need to break ties, particularly if they result in too few or too many final nominees. We have candidate methods to do that now, and I like Joshua Kronengold's method assuming I understand it.

So the other question is which ties are we better off breaking, and which ones should we eliminate all the tied ones?

I agree with you that it's usually better to eliminate both, but we could get by without that. What data would inform us about which way is better in the usual cases where it isn't clear whether we need to break the tie?

#567 ::: J Thomas ::: (view all by) ::: April 27, 2015, 07:34 AM:

@564 Joshua Kronengold

I like a first tiebreak of score.

I like that one. We've already calculated it and we already think it's important. Sometimes it will be the same, particularly for slates.

>o/I -prefer- lowest sum of placement for 2nd tiebreak, because despite now making placement matter, it still doesn't matter much, and it's an actually meaningful tiebreak.

Are you saying, number the order of votes on each ballot from 5 to 1, and give 5 points for each vote where it's in first place down to 1 point when it's in last place, and the higher sum wins?

That looks good to me! So if two nominees on the same slate are competing, the one that comes first will win. That's probably what the people who did the slate wanted, and it will almost always break a tie. They could possibly have an even number of ballots with the nominees exactly balanced, but it would not be easy.

For third tiebreak, I favor something entirely unambiguous, since as we've determined, SDV produces much fairer results if ties are broken than if all tied works are eliminated. We've exhausted meaningful tiebreaks at that point, so I'd favor "the item appearing first on a ballot going chronologically" (so the item appearing first on the earliest ballot where any tied item appears).

That one will always break a tie, so with these three methods any tie can be broken. Good.

I do think we should keep the rule that when we get a tie that includes 5th place, we should keep all tied items. This means we're using our slightly terrible (except for #1) tiebreak only in cases where we -have- to break a tie.

Here it sounds like you're saying to use it only to break ties we have to break. I'm assuming those are when the number of final nominees is otherwise less than 5 or bigger than 6.

I originally proposed eliminating all tied votes because it hurts slates more than other things -- but it would have unintended consequences, and frankly we don't -need- to hurt slates that much, and randomly, while also punishing non-slate candidates for being randomly involved in a tie. So I repent of that notion.

This looks more ambiguous, it could imply that you want to break all ties unless they result in six final nominees. If that's your meaning, I want to suggest we try to get data about its likely effects before we decide. We can have unintended consequences either way.

#568 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 27, 2015, 10:47 AM:

Okay, this is good.

#569 ::: Joshua Kronengold ::: (view all by) ::: April 27, 2015, 12:15 PM:

My concern with my "kill them all" tiebreak is that while it punishes slates, it also punishes highly correlated nominations (like, say, fandoms for particular TV shows) that happen to tie.

I'd hate to simply ignore a very popular show just because its fans happened to manage a true tie between two episodes.

Unlike a slate, a fandom (which is the type of constituency that has heretofore dominated categories) is voting entirely honestly based on things that it's pretty much guarunteed everone involved has watched. But they will (not unreasonably) discuss what the best episodes are, and thus can act slate-like in their category without breaking any of the customs of the Hugos (or intendeding to make things difficult).

#570 ::: Joshua Kronengold ::: (view all by) ::: April 27, 2015, 12:21 PM:

Oh, sorry for the double:

J Thomas@567: I think you have it backwards. It's necessary to break a tie if we're not making the final selection, since otherwise we have to eliminate all tied candidates, and as above that can have undesired consequences of disenfranchizing some of the electorate. So in that case I favor a bad but consistent rule (with some good rules mixed in early) over no tie break.

On the other hand, if all remaining tied candidates are tied for a set of places including 5th (and we don't have so many candidates that the ballot would be too large -- I think 7 or 8 is probably an upper limit here, not 6), we don't have to break a tie--we can instead stop eliminating. So in that case I favor stopping on a tie rather than breaking it.

There's no good advantage to the voters on ending up with fewer than 5 nominees unless there really -aren't- 5 nominees with any popular support. So I don't see any reason to go below 5 except in very odd situations.

#571 ::: Jon Lennox ::: (view all by) ::: April 27, 2015, 12:31 PM:

@519 Joshua Kronengold, and others making proposals:

A proposal also needs to update section 3.11.4 of the WSFS constitution, to describe how the nomination results should be published.

The practical (as opposed to legalese) part of this requirement is that we'll need to figure out how the steps and outcome of the SDV-LPE process (or whatever) should be presented in the statistics packet, in a way that's comprehensible to readers. (Separately, we should figure out how much of this presentation should be legislated in the constitution.)

#572 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:03 PM:

@569: "But they will (not unreasonably) discuss what the best episodes are, and thus can act slate-like in their category without breaking any of the customs of the Hugos (or intendeding to make things difficult)."

I understand that this behavior is not unreasonable, but it's also something that a fandom can easily avoid if they realize it risks causing simulelimination. And I think there's nothing wrong, and arguably something right, with discouraging it like that. For instance, even if one whole fandom voted a set of 2 episodes, they could still probably prevent a tie by making sure that each of them voted for something outside their common interest, so that the SDV votes for the two would be unlikely to be sequential.

Really, ties are very unlikely to exist in the absence of explicit strategic coordination, and even less likely to matter.

#573 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:06 PM:

@571: With the bulk elimination round at the start, a round-by round account of vote totals (along with a single list of ballot totals) would be pretty easy to publish. Without the bulk elimination round, it would be pretty tedious.

#574 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:11 PM:

@570: I agree, there's no reason that the voting system should go below 5 nominees, except because of the threshold. Even if there were a 5-way tie for 5th place, you could simply publish that fact. Many final round voters probably wouldn't need to read all of the 5 tied works to figure out where to put them; that is, the community could probably reach some informal consensus on 1 or 2 of the tied works to consider as "the real nominee(s)", and ignore the rest.

#575 ::: Jameson Quinn ::: (view all by) ::: April 27, 2015, 01:12 PM:

(not that evaluating 9 nominees is impossible if it occasionally happens in categories besides Best Novel)

#576 ::: J Thomas ::: (view all by) ::: April 27, 2015, 01:16 PM:

@573 Jameson Quinn

@571: With the bulk elimination round at the start, a round-by round account of vote totals (along with a single list of ballot totals) would be pretty easy to publish. Without the bulk elimination round, it would be pretty tedious.

I recognize there are potential problems with this, but it might be possible to put up the software and the anonymised ballots into a public software repository. Let anybody who's interested run the code and see what it did.

It can be as tedious as anybody wants to look at.

That might put too much knowledge into the hands of the public. But it wouldn't have to be a bad thing.

#577 ::: J Thomas ::: (view all by) ::: April 27, 2015, 03:53 PM:

@573 Jameson Quinn

With the bulk elimination round at the start, a round-by round account of vote totals (along with a single list of ballot totals) would be pretty easy to publish. Without the bulk elimination round, it would be pretty tedious.

If you never eliminate more than one nominee per round, then:

Take the ones with the top five votes. Calculate their point scores. Look at the lowest of them.

Any nominee whose total votes is less than that point score cannot possibly win. Sooner or later it must compete with one of those five. (Or something that has beaten one of those five, which can't happen.)

If its point score was high enough it might possibly avoid competing with them until some of them have done each other in, and that would give it a chance to win. But their point scores are already higher than its can ever be. So it must compete and lose.

That provides some mass elimination, if the rules wind up working that way.

Is my logic right?

#578 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 05:35 PM:

Here's a sample output from the SDV-LPE code. This was run using the baseline data from the 2013 Hugo ballots. The ballots were randomly generated with frequencies that match the reported frequencies from the 2013 Best Novel category. This output could easily be made public to anyone who was interested.

- Works 1-15 are the top works in order of number of nominations received. (#15 received 55 votes)

- Works 16-19 are unknown works receiving 54,53,52 nominations each.

- Works 20-24 are unknown works receiving 50 nominations each.

- Works 25 is slate candidate #1 (2 nominations); work 26 is slate candidate #2 (1 nomination) -- note we increased these to actual slates in other scenarios.

- The remaining works are for various minor works that received few or no nominations.

Here are the results:

Results of Round: 1
Work(s) Eliminated:
28 Points: 0 Nominations: 0
29 Points: 0 Nominations: 0
30 Points: 0 Nominations: 0
31 Points: 0 Nominations: 0
32 Points: 0 Nominations: 0

Results of Round: 2
Work(s) Eliminated:
27 Points: 0.25 Nominations: 1

Results of Round: 3
Work(s) Eliminated:
26 Points: 0.583333333333333 Nominations: 2

Results of Round: 4
Work(s) Eliminated:
23 Points: 21.1666666666667 Nominations: 50

Results of Round: 5
Work(s) Eliminated:
20 Points: 22.4166666666667 Nominations: 50

Results of Round: 6
Work(s) Eliminated:
24 Points: 23.5833333333333 Nominations: 50

Results of Round: 7
Work(s) Eliminated:
21 Points: 24.9833333333333 Nominations: 50

Results of Round: 8
Work(s) Eliminated:
19 Points: 22.25 Nominations: 52

Results of Round: 9
Work(s) Eliminated:
22 Points: 26.2833333333333 Nominations: 50

Results of Round: 10
Work(s) Eliminated:
25 Points: 27.0666666666667 Nominations: 50

Results of Round: 11
Work(s) Eliminated:
16 Points: 27.2833333333333 Nominations: 55

Results of Round: 12
Work(s) Eliminated:
17 Points: 30.2 Nominations: 54

Results of Round: 13
Work(s) Eliminated:
18 Points: 30.9166666666667 Nominations: 53

Results of Round: 14
Work(s) Eliminated:
15 Points: 31.8666666666667 Nominations: 56

Results of Round: 15
Work(s) Eliminated:
14 Points: 34.4166666666667 Nominations: 58

Results of Round: 16
Work(s) Eliminated:
13 Points: 32.95 Nominations: 61

Results of Round: 17
Work(s) Eliminated:
12 Points: 40.6666666666667 Nominations: 62

Results of Round: 18
Work(s) Eliminated:
10 Points: 39.25 Nominations: 69

Results of Round: 19
Work(s) Eliminated:
11 Points: 44.5 Nominations: 68

Results of Round: 20
Work(s) Eliminated:
9 Points: 45.5 Nominations: 74

Results of Round: 21
Work(s) Eliminated:
8 Points: 62.9166666666667 Nominations: 90

Results of Round: 22
Work(s) Eliminated:
7 Points: 68 Nominations: 91

Results of Round: 23
Work(s) Eliminated:
6 Points: 69.6666666666667 Nominations: 101

Final Hugo Ballot:
1 Points: 152.083333333333 Nominations: 193
2 Points: 106.75 Nominations: 138
3 Points: 102.666666666667 Nominations: 135
4 Points: 100.583333333333 Nominations: 133
5 Points: 94.9166666666667 Nominations: 118

#579 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 05:39 PM:

JT@577:
Any nominee whose total votes is less than that point score cannot possibly win.

I'm not sure that's true, if I understand what you're saying. It's entirely possible -- likely, even -- for the points a work receives to go up in the next the round under SDV-LPE. As a work is eliminated from a ballot, the remaining works get more of that ballot's vote. Maybe I'm just misunderstanding, though.

Kilo

#580 ::: felice ::: (view all by) ::: April 27, 2015, 06:05 PM:

Jameson Quinn @572: "I understand that this behavior is not unreasonable, but it's also something that a fandom can easily avoid if they realize it risks causing simulelimination. And I think there's nothing wrong, and arguably something right, with discouraging it like that. For instance, even if one whole fandom voted a set of 2 episodes, they could still probably prevent a tie by making sure that each of them voted for something outside their common interest, so that the SDV votes for the two would be unlikely to be sequential. "

We really, really don't want nominators to have to worry about that sort of thing! And I don't think your suggestion is very practical - if you're requiring them to nominate things other than their top two preferences, the extras have to be both diverse (it obviously doesn't help if they all nominate the same extras), and popular enough to not all get eliminated before the potential tie is reached, but not so popular that supporting them is likely to help them push the two really popular works off the ballot.

Ties aren't that likely with slates, because bloc voters don't appear to all stick exactly to the slate (and some works may have supporters who aren't following the slate). Ties are unlikely but perfectly possible through coincidental genuine nominations of popular works (eg there's likely to be broad consensus on the best two Doctor Who episodes of the year without any deliberate coordination, which increases the probability of an accidental tie). There's no meaningful advantage to not breaking ties, and potential real harm, on occasion.

#581 ::: P J Evans ::: (view all by) ::: April 27, 2015, 06:32 PM:

580
My imperfect memory says that of the nearly 1500 final ballots in 1984, more than 1200 had 'Return of the Jedi' ranked. I can't remember how many nominations it got (if I ever knew).

Still waiting on Frisbie: he's baking tapes for one last read, which is 10 hours of baking (under controlled conditions) per batch of tapes. And some of the stuff in on 8-inch floppies, and he can't yet get at that computer.

#582 ::: dotless ı ::: (view all by) ::: April 27, 2015, 08:38 PM:

Keith "Kilo" Watt@579: I read the suggestion as comparing the number of nominations for each work against the fifth-best initial point totals. The final point total for any work can't exceed its number of nominations, so that should be a safe initial filter.

I can't tell how useful this would be without more insight into actual ballot distributions. In the scenario you just posted, for example, the fifth-lowest initial point total could have been as little as 23.6 (118/5), which doesn't eliminate very many works.

It's been a long day, so I may be misanalyzing this.

#583 ::: J Thomas ::: (view all by) ::: April 27, 2015, 09:06 PM:

@579 Keith "Kilo" Watt

"Any nominee whose total votes is less than that point score cannot possibly win."

I'm not sure that's true, if I understand what you're saying. It's entirely possible -- likely, even -- for the points a work receives to go up in the next the round under SDV-LPE. As a work is eliminated from a ballot, the remaining works get more of that ballot's vote.

Maybe you will find a flaw in the logic.

We have five nominees whose minimum score is, say, 20.

A 31
B 29
C 26
D 22
E 20

We have a sixth nominee whose total votes add up to less than the minimum of the five scores.

votes for F = 19.

F can never have a score higher than 19, because when its votes are completely unsplit it gets one vote per ballot.

There are two ways for F to win. One is to have the most votes. The other is to have a score so high that it does not get challenged.

F can never have the most votes, because there are five that already have scores higher than F has votes. So all five must have at least as many votes as they have scores, more than F.

Maybe some of them will be eliminated? Maybe, but their scores are higher than F's score can ever be, so F will be challenged and eliminated before any of them. Even if four of them have such high scores that they are never challenged, the fifth has more votes than F and will eliminate F. If something else has a high score and the result is that D and E face a challenge and E is eliminated, that will happen only after F is eliminated because F's score is lower than E's and lower than D's.

F cannot avoid challenge, and cannot win challenge. So F cannot win.

I think this generalizes. At any time, if you look at the five highest scores, any nominee whose total number of votes is less than the fifth highest score cannot win.

Is this wrong?

#584 ::: felice ::: (view all by) ::: April 27, 2015, 09:15 PM:

dotless ı @582: "I can't tell how useful this would be without more insight into actual ballot distributions. In the scenario you just posted, for example, the fifth-lowest initial point total could have been as little as 23.6 (118/5), which doesn't eliminate very many works."

It eliminates every work nominated by fewer than 24 people, which I would expect to take out most of the candidates in a typical set of Hugo nominations; there'd probably be plenty of works nominated by just one person, for example.

#585 ::: Chris ::: (view all by) ::: April 27, 2015, 10:01 PM:

J Thomas @583:
That looks like a sound argument to me, provided that "total votes" means the total number of ballots the nominee is on. You'll probably want to clarify that if this becomes part of the formal proposal

#586 ::: dotless ı ::: (view all by) ::: April 27, 2015, 10:04 PM:

felice@584: I'm just being cautious because I don't have a good intuition about the actual distribution of the votes. Scanning through all the 2013 nomination stats I see only one category (Best Editor, Short Form) where any of the top 15 listed would definitely be eliminated by that rule.

#587 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 10:09 PM:

JT@583:
F can never have a score higher than 19, because when its votes are completely unsplit it gets one vote per ballot.

Unless I'm totally messed up, this isn't true. Unless you mean total nominations and not total points?

It's virtually guaranteed that F will have more than 19 points in a later round. If it appeared on a ballot with, say, G then that ballot only gave it half a point. Once G is eliminated, that ballot will give it a full point and F's total goes to 19.5, doesn't it?

Kilo

#588 ::: felice ::: (view all by) ::: April 27, 2015, 10:17 PM:

dotless ı @586: "Scanning through all the 2013 nomination stats I see only one category (Best Editor, Short Form) where any of the top 15 listed would definitely be eliminated by that rule."

We don't particularly want to eliminate anything in the top 15 in this initial step; there are probably dozens if not hundreds of works nominated in some categories, and it's this long tail we want to snip off. Is there any historical information available about nominations beyond the top 15?


Keith "Kilo" Watt @587: "Unless you mean total nominations and not total points?"

That's my understanding, yes. Works with fewer total nominations than the initial points of the five highest scoring works in the first round get eliminated immediately.

#589 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 10:44 PM:

felice@588:
Works with fewer total nominations than the initial points of the five highest scoring works in the first round get eliminated immediately.

Ah, okay, that's different. Yes, that'd be correct then.

But I guess I'm still wondering if it gains us anything. The code runs fairly quickly (less than 2 seconds with the ballots we've been using as test data), and I haven't even attempted to optimize it. If the concern is the reporting of results, anyone can download a text file and read it. It seems to me that even though what you're saying is true, all it does is to complicate a simple system to no real purpose -- and is just one more thing that must be explained at the business meeting.

Kilo

#590 ::: Keith "Kilo" Watt ::: (view all by) ::: April 27, 2015, 10:50 PM:

By the way, if anyone wants to know the current point total for -each- work in each round of SDV-LPE, just let me know. That's a fairly trivial modification to the code. The text file would be rather long, but it would let you better see what's happening at each step.

Kilo

#591 ::: J Thomas ::: (view all by) ::: April 28, 2015, 06:43 AM:

@589 Keith "Kilo" Watt

"Works with fewer total nominations than the initial points of the five highest scoring works in the first round get eliminated immediately."

Ah, okay, that's different. Yes, that'd be correct then.

But I guess I'm still wondering if it gains us anything.

I now think it has no immediate practical use. If the 5% rule falls then there will be lots of things with 1 vote that tie, and a bit fewer with 2 votes that tie, and so on, but those won't cause any problem.

It's helping me get a sense of what wins. Obviously, the five with the most votes have a big advantage, as they should. But at any given time, the two of those that have the most crossover voting with other survivors may have low point scores and may challenge each other. In that case one of them is eliminated, leaving room for something else to win.

The new winner will tend to be the one with the sixth-highest vote, unless that one has too many ballots in common with other winners. (Ballots it has in common with the 7th highest vote-winner will affect both their scores equally.) Because if the 6th vote-winner and also one of the first 4 both have lower scores than the 7th, they will compete and the 6th will lose, leaving room for the 7th.

If two of the first 4 vote-winners both have lower scores than the 7th vote-winner, they will compete and one will lose, leaving room for the 7th to win independent of the 6th winning.

But no, it isn't so deterministic. If your nominee is the 6th or 9th then at any time -- maybe when your score is half your vote, one of the top 5 may have its score be a quarter of its vote and the two are chosen to compete and yours is eliminated. That could possibly happen *any time* after the scores being tested have risen to 1/5 of the votes of anything with more votes than yours.

This has implications for strategic voting, but they're complicated. Strategic voting requires game theory. You have to guess not only how much everybody else likes the different works, but also what game strategies they will use to get the results they want. The honest outcome suffers just as much when people use bad strategies that fail for them, as when they use effective strategies.

Honest voters and strategists both agree that most voters ought to use no strategy but just vote for what they like.

It's complicated and I don't have it thought out yet.

#592 ::: Joshua Kronengold ::: (view all by) ::: April 28, 2015, 12:07 PM:

I think the only real strategy for SDV-LPE is to decide what "grade" to cut off your votes. Which is really a true strategy for any approval system. Obviously, if you include A- choices you're putting less "oomph" behind your A+ choices, so you need to decide what's more important to you.

There's also the false ungroup -- but that's such a risky strategy unless you have way too much knowledge of the pool that it's not usually worth doing. If your 2nd place item is in close enough contention that it's at risk of being knocked out by your first place item, then your first place item is probably also at at least some risk.

The most important thing is that reasonable, obvious strategy results in good results and strategic play doesn't result in a substantial advantage -- and in both cases I think we're pretty good here.

Re direct elimination (ie, remove all items whose total # of ballots are less than the number of votes for the fifth place nominee), this is a mathematical collorary, not a rule, but I do see a use for it, not algorithmically (as said, running a computer algorithm makes the first few hundred rounds go fast very quickly), but in terms of display.

Display is something of an issue for SDV-LPE, since there are so many rouns of elimination. But if we do direct elimination every time the scores of the top 5 items changes (since the scores of the top 5 can only ever go up until the last round of elimination), we can eliminate more candidates in a step--which substantially simplifies display. At any time a candidate has fewer votes than the fifth place candidate has score, they are eliminated--since at that point they cannot possibly make the final ballot. This is probably worth mentioning as a display simplification in commentary.

#593 ::: dotless ı ::: (view all by) ::: April 28, 2015, 01:01 PM:

Joshua Kronengold@592: I was assuming this was to optimize the display. As Jon Lennox@571 noted, any proposal is going to need to make at least a minimal change to 3.11.4 (on the reporting of nominations). That's why I was trying—mostly unsuccessfully—to compare the "can't win" cutoff to the current "top 15" rule. A minimal change reporting the last 15 works surviving elimination would be closest in spirit to the current rule; but I wouldn't be surprised if there's some sentiment in favor of also making sure to report, say, the top 15 in terms of nomination counts, so that people can see what's getting eliminated by the new voting scheme.

I have no strong intuition, though, about what would make the proposal most acceptable to the business meeting, so this may be needlessly complicating things.

I may have missed the answer to this earlier in this or the previous thread, but are there generally accepted reasons why the full anonymized nomination ballots are not made generally available? Is it out of a desire or requirement to present the results in print form? Or that there's never been a reason for it? Or something else? In particular, is there any requirement that's likely to be broken if the natural way of presenting the detailed results is exceptionally verbose?

#594 ::: P J Evans ::: (view all by) ::: April 28, 2015, 01:09 PM:

593
I'd say no reason for it: it really hasn't come up. (Generally, people haven't been interested in that end of the process, nor has there been much interest in the actual final ballots as ballots. It's the results that are most wanted.)

#595 ::: nathanbp ::: (view all by) ::: April 28, 2015, 01:39 PM:

593
One thing to consider about releasing the full nomination ballots is that given the number of categories and slots they will likely be unique (except for slates). It seems to me that we shouldn't release a list of who nominated what, and releasing the set of (likely) unique ballots is a step closer to that than the current process. I think that releasing some summary that covers the top 15 candidates should be enough.

#596 ::: J Thomas ::: (view all by) ::: April 28, 2015, 02:41 PM:

The following is a description of what strategic voting is, and how it works. There are no policy ideas here.

Strategic Voting

When you vote not for what you want, but for what you think can win, you are doing strategic voting.

People do this a whole lot when there are more than two choices, in primaries etc. But we want to minimize it here.

You mostly can't tell whether somebody is doing strategic voting by watching them vote. Maybe what they vote for is what they really want. It depends on what's going on inside their heads, and you don't know that unless they tell you.

Mostly, to vote strategically you have to guess how other people will vote. Some strategies will backfire if you guess wrong. From our point of view, strategies which succeed and strategies which fail are both bad. Ideally we would have a voting system in which strategies do not work and also people believe they don't work and don't try to use them.

Possible strategies include:

1. Vote only for the one you want most, to maximize the chance it wins. (Bullet voting)

2. Don't vote for one you want which you think is popular enough to win without you. (Deserter)

3. Vote for your favorite and several other unpopular works, on the assumption that people who like the other works will also vote for yours. (Slate)

4. Vote for something that you think can win instead of your favorite which you think cannot. (Compromise)

5. When you can rank nominees, rank a popular one lower than you believe it deserves hoping you can defeat it. (Burying)

6. For any system with runoffs, when you think your favorite will get to the runoff, vote for the worst alternative so it will be easier to beat later. (Pushover)

7. Etc.

There's a difference between strategies that stop working if too many people use them, versus strategies that just keep working better. If, under a particular voting system, bullet voting works, and it keeps working better when more people use it, then you don't need to estimate how many of your comrades will use it. Use it yourself and hope they all do. But the Deserter strategy fails if too many people use it. The popular nominee that many people want to win fails because they all depended on somebody else to vote for it. Self-limiting strategies are probably less noxious, if you must allow one or the other.

Say you like A and B. When there are 5 winners, and A wins, then there are only 4 places left that B can have. So it looks like A affects B's chances just by existing. And that's usually true. (I have an example where they are in fact independent, where voting AB has the same effect on A's chances as just voting A would. I'll post it if somebody wants it. The bottom line is, even obvious things aren't always true in voting theory.) Anyway, it's usually true that A affects B's chances just by existing, and in that case it's impossible to prevent strategic voting entirely.


Bottom line: Strategic voting depends on how the voter thinks other voters will vote. It's impossible to do really great strategic voting without secret access to the votes, but people will base strategies on their guesses. The best voting system to reduce strategic voting would arrange things so that the most you could say without knowing illegal details was “Voting for A will increase A's chance to win. Voting for B will increase B's chance to win. Voting for AB will increase both A's chance and B's chance almost as much as voting for just one would increase that one's chance.”

#597 ::: J Thomas ::: (view all by) ::: April 28, 2015, 08:10 PM:

Strategic voting with SDV-LPE

1. First the good news. If you like three nominees equally, your best strategy is to vote for all of them. If you like five nominees equally, vote for all five. This is the best approach.

2. If you only like one, vote for that one.

3. If you like several, and you think the one you like best is the most popular one this year, then great! Vote for all of them. If you're right, the one you like best will win, and you can also support anything else you like. You might possibly do better by not voting for your favorite but only for the others. But remember that you could be wrong, and your vote for your favorite might be needed.

4. If you like several, and you think the one you like best will be among the top five in votes, then it's probably safe to vote for all of them. Unless maybe one that you don't like as much is probably higher among the top five. Then you might do better not to vote for that one. You can do more good for your favorite if you don't vote for the others either, but they aren't as dangerous.

5. If you like several, and you think the one you like best is not among the top five but may be among the top ten or so, vote for that one and no other.

6. If the one you most care about is not very likely to be a winner, get as many of your friends as possible to vote for only it and nothing else. Start a movement! Alternatively, follow the Compromise strategy and drop it for something that can actually win.

Here's my reasoning.

3. The one with the most votes will win. If you're right that your favorite has the most votes, then you're sitting pretty, you can do whatever you want. You can do more good for your other choices if you don't vote for it, but then again, what if you're wrong? It's your favorite, and if you're not certain it will win then vote for it.

4. If your favorite is really among the top 5, there is only one way it can lose. It can get a lot of divided votes that reduce its point score, and one of the ones above it also gets a lot of divided votes, and at some point those two have the lowest scores and compete, and yours loses. This is the only way to lose.

Imagine that you think your favorite has the fifth-highest votes, and you cast a ballot divided between it and the four higher. That does nothing at all to help your favorite win! If you vote for weaker works, there's a good chance they will be eliminated before your favorite ever has one of the bottom scores, though there is a risk. But every vote for something that can beat your favorite is a vote against your favorite.

5. If you think that your favorite is in place N for votes, it can't win unless N-5 of the ones above it are eliminated. That can't possibly happen unless its score is higher than the beginning scores of that many of them. Otherwise it will definitely be eliminated first. If you cast a divided ballot you are not helping it much at all. However, it's OK to vote for other works that are so much weaker they will be eliminated quickly. If you want to.

6. If you don't have a plausible chance that N-5 of the ones above your favorite will be eliminated, your only hope is to treat it as a cult classic. Get enough dedicated followers to bullet-vote for it that it has a chance after all. What to do for an encore after it gets on the final ballot is something else, but if you can get enough people to bullet-vote you have a chance to get there. If you aren't that fanatical, then punt.

All of this looks reasonable to me, but I haven't done simulations to confirm that these strategies work. There could be something I haven't thought of that would make them fail. I'll start on simulations in my copious free time.

#598 ::: Keith "Kilo" Watt ::: (view all by) ::: April 28, 2015, 09:48 PM:

JT@597:

I'd be happy to run any sims you might be interested in, just let me know the specific parameters (or better yet a data file) that you want me to test.

K

#599 ::: J Thomas ::: (view all by) ::: April 28, 2015, 11:26 PM:

@598

OK, I'll email you some files when they're ready. Actually choosing what to test and building the datasets is of course the tedious part. ;)

#600 ::: J Thomas ::: (view all by) ::: April 28, 2015, 11:27 PM:

@598

OK, I'll email you some files when they're ready. Actually choosing what to test and building the datasets is of course the tedious part. ;)

#601 ::: felice ::: (view all by) ::: April 29, 2015, 03:25 AM:

Here's a draft formal proposal for Option 5b; any thoughts?


Short Title: Don't Get Wasted

Moved, to amend the WSFS Constitution to require publication of an indicative longlist part way through the nomination period of each year's Hugo Awards, in order to reduce wasted nominations and increase the number of nominations for the works on the final ballot, by adding words as follows:

3.7.4: Half way through the nomination period, the Committee shall take a copy of nominations received to date. These early nominations shall be tallied as per Section 3.8, and an unranked longlist of the most nominated works in each category shall be published as soon as is reasonably practical.

3.7.5: The longlist shall consist of 15 works in each category, provided that each listed work has been nominated by at least five people. Should fewer than ten works in a category qualify, the five nominations requirement shall be waived for the top ten works. In the event of a tie, all tied works shall be included or excluded, provided that this does not reduce the number of listed works below ten, and the number of listed works should be as close as possible to 15.5. If an unreasonably large number of works are tied, the Worldcon Committee may use any tie-breaking methods they deem appropriate.

3.7.6: Should any work on the longlist be found to be ineligible, or the nominee declines nomination, the published longlist shall be updated to replace that work with the next most nominated work, based on the original copy of the nomination data.

3.7.7: Nominators may update their nominations following the publication of the longlist.

Commentary:
Far too many nominations are currently wasted. The field of science fiction and fantasy is vast, and many nominations are made for works that don't have any realistic chance of making the final ballot. Conversely, works that do make the final ballot can get there with the support of a relatively small proportion of nominators (*cough*puppies*cough*).

Publishing an indicative longlist before the closing of nominations allows people to see which works are genuinely in the running, and alter their nominations accordingly. This means a greater number of people will have input into the makeup of the final ballot. This will make it harder for bloc voters to dominate the final ballot, but it is also a good thing in its own right, even in the absence of a slate. For example, works E and G may both have enough devoted supporters to get in the top 15, and under the current system, E will make the final ballot, while G won't. But if many of the people who nominated P through Z realise that their favourites don't have a chance, they might choose to nominate G instead, because they believe it to be far superior to E. This means that each work ending up on the final ballot will tend to have broad approval as well as a significant fanbase that believes it to be one of the very best works of the year.

As well as concentrating the nominations of the existing pool of nominators, the longlist would encourage participation by people who don't usually nominate. "Which of these works do you think is most worthy of a Hugo?" is a much easier question to answer than "What were the best works published last year?". Increasing participation is a good thing, whether there's bloc voting to resist or not.

The longlist is only indicative; it is possible for a work to end up on the final ballot despite not appearing in the longlist, for example by the fans of a popular work being motivated to nominate it because it isn't on the longlist.

The effect of the provision for ties in 3.7.5 is that extending to 16 is prefered over shrinking to 14, which is preferred over extending to 17, and so forth. An unreasonably large number of tied works could occur if say only four works in a category were nominated by more than one person, with a hundred other works tied for fifth place, in which case the committee could for example accept the first 11 single-vote nominations received for positions 5 through 15, or select them randomly. Such an extreme tie is pretty unlikely, though.

The original copy of the nomination data is specified in 3.7.6 because we want to know what would have been on the original longlist if the removed work had never been included, without being affected by changes to nomination patterns following the initial publication.

Editing nominations as per 3.7.7 is simply an extension of the process already followed for the voting stage: "You can submit a partial ballot and update/edit/complete the ballot at a later date by returning to this page and re-authenticating. Your previous selections will be reloaded into the form, which you can then alter, delete or leave as is."

This motion is entirely compatible with changing the method of tallying nominations to a proportional system such as SDV-LPE; indeed, such a change would result in a more representative longlist and reduce the number of works from a bloc voting slate that would appear on the longlist.

NB There is nothing in the current constitution prohibiting the publication of an indicative longlist. This motion would merely make it a requirement, and if it's passed at Sasquan, MidAmericon II could voluntarily publish a longlist before the change is ratified. While perhaps somewhat irregular, this would allow a trial run of the process before the ratification vote, and it's a measure that could help protect next year's Hugo Awards from bloc voting.


FAQs
1. Won't publishing a longlist distort the subsequent nominations?
Yes - that's the whole point. It encourages nominators to indicate what they think of works that do have a chance of winning, instead of telling us that they prefer works that can't win. The latter may give a more accurate picture of voter preferences, but does nothing to help select works for the final ballot, which is the real purpose of the nomination process. If we switched instead to a true three stage system with formal longlist, shortlist, and winner rounds, nobody would see a problem with choosing the shortlist from the works on the longlist, and this indicative longlist process is essentially equivalent.

2. So why not switch to a proper three stage system instead?
Becuase it would take longer to run and be more work for everyone, both nominators/voters and administrators. The indicative longlist has pretty much all the benefits and none of the drawbacks.

3. Doesn't this give an unfair advantage to people who nominate early?
Since everyone knows that those who nominate early will influence the makeup of the longlist, and everyone can be one of the people who nominate early if they want, no, there's nothing unfair about it.

4. Isn't this unfair to works that just miss out on the longlist?
It will probably reduce those works' chances of winning, sure, but not as much as failing to make the final ballot reduces a work's chances, and that's generally accepted as reasonable. It's bad luck for the works that miss out on making the cut at any stage, but no, it's not unfair.

5. What if a slate gets all their works on the longlist?
Then that gives genuine voters more incentive to pick some of the non-slate works on the longlist to nominate. And in the unlikely event that the voting bloc is big enough and organised enough to run three separate slates to sweep the entire longlist (or if there are three or more separate powerful voting blocs), then they've warned us how much of a problem they are, and we've got a few weeks to take other action such as coming up with a democratically chosen unofficial counter-slate.

6. So what if the slate waits till after the longlist is published to nominate?
Then we'll still get more people nominating works from the longlist, raising the bar a voting bloc needs to reach to get any of its slate on the final ballot. We don't need to know about the existance of a slate for this measure to help combat it.

7. People can't possibly read all the works on the longlist in such a short time!
That's not a question. And it doesn't matter; nominating a work means you think it's good enough to deserve a Hugo, not that it's better than anything else published last year, because nobody can possibly know if the latter is really the case. Some people can read some works on the longlist that they wouldn't otherwise have considered for nomination, especially short fiction, and that's beneficial for the awards. Reading everything on the longlist isn't necessary.

8. Would there be a voter packet for longlisted works?
Highly unlikely.

9. Isn't this extra hassle for the administrators?
Not much; and it may even make things easier for them. The data needs to be cleaned before tallying the nominations for the longlist, but that reduces the workload when preparing to tally the nominations for the final ballot. The tallying itself should be automated, so running it an extra time should be trivial. And publishing a longlist reduces the chances of having to make changes to the final ballot after it's already been announced, as ineligible works are likely to be spotted earlier, and anyone who doesn't want an award can decline nomination earlier.

#602 ::: J Thomas ::: (view all by) ::: April 29, 2015, 07:02 AM:

I would suggest

3.7.5: The longlist shall consist of 15 works in each category, provided that each listed work has been nominated by at least five people. Should fewer than ten works in a category qualify, the five nominations requirement shall be waived for the top ten works. In the event of a tie for last place, all tied works shall be included or excluded, provided that this does not reduce the number of listed works below ten, and the number of listed works should be as close as possible to 15.5. If an unreasonably large number of works are tied, the Worldcon Committee may use any tie-breaking methods they deem appropriate.

If the fifth and sixth works are tied, the fifteenth work is still fifteenth. Unless you want other ties to increase the length of the list, we could keep it at 15 most of the time.

If you get 12 tied for 5th place, that's still a tie for last place. But if you get 5 tied for 5th place, it isn't.

#603 ::: J Thomas ::: (view all by) ::: April 29, 2015, 07:14 AM:

I tentatively suggest:

FAQs
1. Won't publishing a longlist distort the subsequent nominations?
Yes - that's the whole point. It encourages nominators to indicate what they think of works that do have a chance of winning, instead of telling us that they prefer works that can't win. The latter may give a more accurate picture of voter preferences, but does nothing to help select works for the final ballot, which is a vital purpose of the nomination process. If we switched instead to a true three stage system with formal longlist, shortlist, and winner rounds, nobody would see a problem with choosing the shortlist from the works on the longlist, and this indicative longlist process is essentially equivalent. When the longlist is later published with votes for each item, it will provide the same picture of voter preferences that the current published longlist does.

#604 ::: Joshua Kronengold ::: (view all by) ::: April 29, 2015, 11:19 AM:

I'll note that the unintended conseuqnece of the longlist proposal is that it makes it somewhat likely for something to pull a "Digger" run (i.e., relatively unknown work squeaks into the nominations in last place, then blows everyone away once they get a chance to read it). There's at least some chance of already-popular works picking up extra nominations due to their popularity rather than their quality after the longlist is published, which would be an unfortunate distorting effect.

That said, it's obviously a win/win for next year to have -some- kind of longlist unless the puppies don't (ha, ha) organize. And the benefits probably generally outweigh the costs.

#605 ::: Lydy Nickerson ::: (view all by) ::: April 29, 2015, 11:19 AM:

Just off the top of my head, as someone who doesn't (and can't) do the math, and who is hesitant to nominate at the best of times, I think that publishing a long-list makes the process seem more complicated, not less, and will raise the bar for nominating, not lower it. One of the things I like about the SDV-LPE is that while the math appears to be complicated, my actual interface with it is pretty simple: nominate the things I think are Hugo worthy, don't nominate the things I think aren't. Reduce the list to the things I'm actually passionate about. It also allows me to nominate without considering other people's potential nominations. If I have to look at a list half-way through in order to adequately strategize, there become a lot more choice-points, and I feel more confused and constrained.

One of the styles of voter/nominator that you should be trying to design for is me: reasonably involved in the field, reasonably passionate about the Hugos, but without a huge amount of spoons to spend on strategic voting or sussing out the exact math behind the system.

Also, I think you may minimize the amount of effort necessary to publish the long-list. I could be completely wrong, since I've never been involved in running the Hugos, per se, but as a con-runner, I can tell you that any additional deadline of any type whatsoever is a significant additional burden. Keeping track of the time-line, keeping track of all the various deadlines, is real administrative work.

Also, you don't address (unless I missed it) how the long-list will be published. I believe you are assuming that it will be put up on the web. But the argument over whether or not we're comfortable disenfranchising people who don't use computers is not over, yet. Hence, the paper ballots. This means that unless you want to take on that argument in tandem with this amendment, you are asking the Hugo administrators to add an additional cost of publishing and mailing the long list. And that, again, is a real, substantial administrative burden.

#606 ::: Lydy Nickerson ::: (view all by) ::: April 29, 2015, 11:24 AM:

Joshua @604: I think the assumption here is that people will use the longlist as a reading list. I suspect that this is optimistic. People like me use the nominations as a reading list, and at that, it's rather long. The primary use of the longlist will be strategic voting, if I understand the design correctly. Which I freely admit I may not. Also, is there any chance that this will increase the ability of puppies to refine their strategies?

#607 ::: Nancy Lebovitz ::: (view all by) ::: April 29, 2015, 11:51 AM:

My hope is that the long list will help people remember what they're liked that's eligible. I agree that it doesn't make sense as a reading list.

#608 ::: Joshua Kronengold ::: (view all by) ::: April 29, 2015, 12:02 PM:

s/likely/-less- likely/

It will act as a force multipler for works that are already generally popular -- as their presence on the long list will encourage people who liked them to nominate them.

This is good in the presence of a minority slate, of course, since boosting numbers of all popular works makes it hard for the slate to push less popular work based on concentrated numbers. But it's not clear to me that the overall result will be positive. As Lydy mentions, it's more demanding for nominators (who if they want maximal input, need to nominate early, then check the longlist and possibly adjust their nominations, and maybe post to draw attention to works that they think are worthy but didn't make the longlist...). And it's more demanding for the admins, who need to do at least a basic check of the longlists to try to collapse duplicates and remove obvious errors, coordinate making sure it goes up on time, etc.

I'd overall rather that longlisty things happened outside the Hugos per se (if people want to make an open polling site with some thin voter-confirmation to make it less open for stuffing for next year, frex, I'd be up for helping), so they could adjust based on need.

#609 ::: Joshua Kronengold ::: (view all by) ::: April 29, 2015, 12:05 PM:

@606: The benefit to the general electorate would be far higher than that to a slate -- the slate is already converging, but a longlist would allow/encourage the general public to converge as well. I suspect a longlist would in a sense get used as a reading list for those very invested in a category -- but as someone who reads a 30+ book long longlist every year (Mythopoeic Childrens, and I often get somewhere into the adult list as well), it takes -quite- a lot of time, more than we'd have after it went up. Maybe for short story and Best Fanzine (which admittedly are some of the least convergent categories).

#610 ::: Kevin Standlee ::: (view all by) ::: April 29, 2015, 12:16 PM:

Do you want to address the observed fact that most nominating ballots have historically been cast during the final two weeks of the voting period?

#611 ::: P J Evans ::: (view all by) ::: April 29, 2015, 01:24 PM:

610
Kevin, I think they haven't had to deal with that particular feature before. (Some of this makes me think they're short on real-world experience.) I described it on one of these posts as a 'bathtub-shaped curve' - that was our experience with the final ballots in 1984, but it applies to nominations also: initial hit from the early-birds, a long slow trickle, then a big big load of incoming ballots at the end. Plus, with paper nominations, figuring out exactly what they're nominating so it can be put in the computer.

#612 ::: Cassy B. ::: (view all by) ::: April 29, 2015, 01:41 PM:

For whatever it's worth, I, as a run-of-the-mill Hugo nominator and voter, am very uncomfortable with the idea of a long-list.

To me, it seems to be just another slate.

#613 ::: Buddha Buck ::: (view all by) ::: April 29, 2015, 01:43 PM:

PJ Evans @611:

How bad do the nominations get in terms of "figuring out exactly what they're nominating"? I suspect "'Redshirts, by John Skalsey" would be annoying, but acceptable, but what about "The last Discworld novel, about trains" (AKA Raising Steam, by Terry Pratchett)?

#614 ::: Joshua Kronengold ::: (view all by) ::: April 29, 2015, 01:44 PM:

610, 611: It's an interesting wrinkle. I suspect that having a "longlist" feature might push some people to nominate earlier, but maybe not. Paper ballots aren't as much of a problem any more apparently (supposedly, this year there were three).

This is another reason that I think having loose polling that wasn't explicitly tied to the Hugo administration would work better, as it would be a tool that could achieve convergence (when it was useful), but not locked in stone.

#615 ::: P J Evans ::: (view all by) ::: April 29, 2015, 01:51 PM:

613
Misspellings, definitely. With fanzines, where it's possible that no one on the administrative side knows them, it can be more interesting. (ISTR we dealt with both 'Space & Thyme' and 'Space & Time' as variant titles. I can answer in more detail if I can get the stuff from Frisbie before everything has been decided.)

#616 ::: Steve Halter ::: (view all by) ::: April 29, 2015, 03:07 PM:

I'm also not really comfortable with the long list idea as it seems to somewhat go against the idea of nominating what you like rather than what you think is going to get nominated. There is a certain amount of gamesmanship that seem implicit within the notion.

#617 ::: Jameson Quinn ::: (view all by) ::: April 29, 2015, 03:29 PM:

In ideal theory, the point of the longlist would be to enable honest strategy. To take the alphabet example again, if I'm a reverse-alphabetical voter, my naive vote might be ZYXWV. When I saw that the longlist is ADEHILMNOSTU, I might want to change to voting ZYXWVU or ZYXWVUTS. That is, if you don't already have any longest members on your ballot, you might consider adding 1-3 of them, but you generally wouldn't take anything off your ballot, or change a ballot like ABCDE which already overlapped with the longlist.

So in theory, publishing the longlist should be a relatively "light" influence on how people vote. A "nudge". Would it be like that in practice? I don't know.

#618 ::: Keith "Kilo" Watt ::: (view all by) ::: April 29, 2015, 04:01 PM:

Jameson@617:
That is, if you don't already have any longest members on your ballot, you might consider adding 1-3 of them, but you generally wouldn't take anything off your ballot

Of course, I don't know any better than anyone else, but I actually think more people would say, "Oh, my choices can't win. I'd better nominate something I like from the top 15."

The real question: Is this a bad thing?

Kilo

#619 ::: Cassy B. ::: (view all by) ::: April 29, 2015, 04:10 PM:

Keith "Kilo" Watt @618, yes, that's how I'd react. "This thing I really loved isn't mentioned, so it can't possibly win, so I shouldn't nominate it".

It seems to me that a longlist actually suppresses nominations. I don't personally WANT to be instructed how to nominate strategically. It seems to me to go against the spirit of the Hugos.

Just my two cents.

#620 ::: Fade Manley ::: (view all by) ::: April 29, 2015, 04:41 PM:

As someone who finds the minutia of voting systems hard to follow, but would very much like to nominate honestly, I feel like a preliminary longlist is a bad thing that will skew nominations. I don't want to nominate things based on how likely they are to get on the final ballot; I want to nominate them based on how much I like them! And having an official step that reads like "Well, you should stop talking about things you actually like, and push for what you're closest to liking on this list" is...uncomfortable.

I mean, I accept that in the voting round, because the numbers have already been run for what everyone nominated. But now suddenly I need to strategize, when I'm still trying to suggest things to people! And I don't know if what's on the displayed list is actually popular, anyway, or just what's most popular among the groups that are most motivated to get their nominations in really early to get things on the 15-long list to make sure everyone else gives up on anything they liked that isn't on that list already, and are people going to tell me that I'm "wasting" my nomination if I don't nominate one of those things? Even if I didn't like it?

That was a somewhat incoherent sentence, but it's a pretty accurate representation of what I feel like when I look at a theoretical change of that sort.

In general, I don't like voting systems in an award like this that make strategizing more valuable than honestly stating my opinion. And the 15-long list looks very much like that's what it would do: push me to think about what could win, and who might be nominating earliest, and if I should nominate immediately rather than think about my choices just for a chance to affect the first round. It becomes increasingly distant from a statement about my own preferences on literature and a lot closer to feeling like...well. Politics. Or rules-lawyering.

#621 ::: Soon Lee ::: (view all by) ::: April 29, 2015, 06:11 PM:

Fade Manley #620:

In general, I don't like voting systems in an award like this that make strategizing more valuable than honestly stating my opinion.

So much this.

#622 ::: felice ::: (view all by) ::: April 29, 2015, 07:18 PM:

Joshua Kronengold @604: "I'll note that the unintended conseuqnece of the longlist proposal is that it makes it somewhat [less] likely for something to pull a "Digger" run"

Whether it makes it more or less likely is unclear. Yes, you're right that it's a force multipler for popular works; but it also provides a lower threshold for bringing a Digger-type work to people's notice. People won't read all the works on the longlist, but I'd expect most people to at least make themselves aware of what the longlisted works are, eg at a minimum reading back cover blurbs or equivalent, or a few sample pages; and if the longlist introduces them to something new that looks interesting, they've got a chance to read that. There's certainly time for most people to read a few short stories, watch a few dramatic presentations, and check out a few art samples.


Lydy Nickerson @605: "If I have to look at a list half-way through in order to adequately strategize, there become a lot more choice-points, and I feel more confused and constrained."

You don't have to. If you don't want to worry about the longlist, keep nominating the same way you do now, and your nominations still get counted. That's definitely something to add to the FAQ.

"Also, you don't address (unless I missed it) how the long-list will be published. I believe you are assuming that it will be put up on the web. But the argument over whether or not we're comfortable disenfranchising people who don't use computers is not over, yet. Hence, the paper ballots."

Text edited to specify online publication - thanks. Reacting to the longlist is optional, so it doesn't strike me as a huge problem that three people won't see it; their nominations will still get counted just like everyone else's. There's a significant difference between making information available online and requiring people to submit nominations etc online; presumably most WSFS members who don't use computers at least talk to people who do.


Joshua Kronengold @608: "And it's more demanding for the admins, who need to do at least a basic check of the longlists to try to collapse duplicates and remove obvious errors, coordinate making sure it goes up on time, etc."

Any errors/duplicates found at the longlist stage are errors/duplicates that don't need to be found at the final tallying stage; that's mostly just shifting the work, not adding to it. The actual tallying and publication is extra work, but not that much work.


Kevin Standlee @610: "Do you want to address the observed fact that most nominating ballots have historically been cast during the final two weeks of the voting period?"

I would expect the longlist deadline would encourage most of them to nominate earlier. Rate of submissions increasing as a deadline approaches is a universal principle, irrespective of when the deadline is. There are plenty of graphs showing it, eg http://www.nature.com/nphys/journal/v3/n11/fig_tab/nphys761_F1.html


Joshua Kronengold @614: "This is another reason that I think having loose polling that wasn't explicitly tied to the Hugo administration would work better"

Though that has the disadvantages of requiring more work (people have to nominate for the poll as well as for the actual Hugos, and of course the administration of the poll is entirely new work), and the questions of eligibility and spam (an unofficial poll wouldn't have access to membership data so wouldn't be measuring the same population as the nominations, and if payment isn't required to participate, you're likely to get a lot more gamergate types "contributing").


Cassy B. @619: "Keith "Kilo" Watt @618, yes, that's how I'd react. "This thing I really loved isn't mentioned, so it can't possibly win, so I shouldn't nominate it". It seems to me that a longlist actually suppresses nominations."

Only if you wait to see what's on the longlist before submitting any nominations yourself. If you nominate Z and it doesn't show up on the longlist, you can still keep it on your ballot, and maybe change your nomination to ZADGH to support some works that have a chance as well as sticking with Z. But honestly, if it does suppress nominations for works that can't possibly win, does that really do any harm? Is there any reason to think you could be wrong about it not being possible for your favourite to win? Under the current system, you won't know it's not getting many nominations, but not knowing won't do anything to help its chances!

Fade Manley @620: "I mean, I accept that in the voting round, because the numbers have already been run for what everyone nominated. But now suddenly I need to strategize, when I'm still trying to suggest things to people! And I don't know if what's on the displayed list is actually popular, anyway, or just what's most popular among the groups that are most motivated to get their nominations in really early to get things on the 15-long list to make sure everyone else gives up on anything they liked that isn't on that list already, and are people going to tell me that I'm "wasting" my nomination if I don't nominate one of those things? Even if I didn't like it?"

No, don't nominate anything you don't like! If you think none of the works on the longlist are Hugo-worthy, stick to promoting things you do like. Tell people the current frontrunners suck, and they should nominate X instead. It might work. Is there any reason to think particular groups would be more motivated than others to nominate before the longlist deadline?

"In general, I don't like voting systems in an award like this that make strategizing more valuable than honestly stating my opinion."

State your honest opinion upfront, then strategize in the second half if you want to. How is this strategizing different from ranking the five works on the final ballot irrespective of how you think they compare to other works that didn't make the final ballot?

#623 ::: David Harmon ::: (view all by) ::: April 29, 2015, 07:47 PM:

Another vote against releasing an intermediate longlist here. While it's easy to think "more information is always good", that's not necessarily true:

The nominations are a social process that already involves flows of information around the fannosphere. Releasing internal information from an intermediate stage of the nominations is not the same as fans making recommendations to each other. Rather, it's looping "privileged data" (certified by the Hugo administrators!) from the middle of the process back into the input (that being the nominators).

The idea smells like trouble to me, basically opening the door to all sorts of strategies and feedback loops.

#624 ::: P J Evans ::: (view all by) ::: April 29, 2015, 07:54 PM:

623
Looping data from the middle (or the end) of the process back into the input is literal feedback - it's how you make a flipflop, in electronics, by routing the output of a NAND gate into one of its two inputs!

#625 ::: David Harmon ::: (view all by) ::: April 29, 2015, 08:15 PM:

P J Evans #624: Just so... but exactly what circuit you get depends on the particular connections you make, and in this case, (with people in the loop) I think the result is likely to be instability.

#626 ::: Fade Manley ::: (view all by) ::: April 29, 2015, 08:29 PM:

felice @622: Is there any reason to think particular groups would be more motivated than others to nominate before the longlist deadline?

Given that several people, including me, seem to be under the impression that having that 15-item list is a way of focusing votes on those particular early front-runners... Yes, I absolutely think that the sort of people who want to do slate-voting would try very hard to take control of that early list for exactly that purpose. And it feels like you're suggesting that if I don't like that, I ought to...rally the vote around things I like?

But I don't want to rally votes. I want people vote for the things they like without any consideration to whether or not other people like it. "If you think none of the works on the longlist are Hugo-worthy, stick to promoting things you do like" is the exact kind of politicking I'm trying to avoid. I don't want people campaigning all over the place for their favorites! I want people saying what they already like. This is the stage where people talk about the things they've already read, not ask everyone else what other people like and think is appropriate.

How is this strategizing different from ranking the five works on the final ballot irrespective of how you think they compare to other works that didn't make the final ballot?

The 5-item list is set. I'm not "strategizing" when I rank them; I'm expressing how good I think they are relative to each other.

The 15-item list is not set. It's a potential short-list. Deciding if I want to "waste" my vote on something not on that list yet, or add to something I might like less but which looks to have a better chance of making it to the final round to keep off things I dislike even more already on that list, turns into strategizing. Trying to convince other people to nominate things, on that list or not, to make sure they make it onto the final five, based on the information I have about whether they've made it to the semi-final 15 or not, is both political and strategizing. Exactly what I want to avoid!

The information I want to give the Hugos is "What things did you think were good?" and "How good do you think these specific things are, relative to each other?" I do not want to have to consider what other people think about these things at any point along the way. (The 5-item list tells me what people thought, but at a point where it can have no input on what I stated that I liked.) I can see no reason for this midway list except to tell me that I ought to be thinking about what other people like, and other people ought to be thinking about what I like, and that we're supposed to be politicking about it. Ugh.

I mean, at that point you might as well have one of those 'awards' that's based on mass popular vote and shows percentages going to different items through the whole voting process, with people shouting on social media to, quick, go vote for THIS book, it's losing against THAT one. That is a type of award. But it's not the Hugos.

#627 ::: J Thomas ::: (view all by) ::: April 29, 2015, 09:13 PM:

@620 Fade Manley

I feel like a preliminary longlist is a bad thing that will skew nominations. I don't want to nominate things based on how likely they are to get on the final ballot; I want to nominate them based on how much I like them!

If that's what you want to do, then you should do that.

And having an official step that reads like "Well, you should stop talking about things you actually like, and push for what you're closest to liking on this list" is...uncomfortable.

If it makes you uncomfortable, don't read the list. It isn't there to tell you what to do. It's there to provide information, that might be helpful for some choices.

I mean, I accept that in the voting round, because the numbers have already been run for what everyone nominated.

That shortlist has a great big effect on the final voting. Are you comfortable with it because it's familiar? We *could* let everybody just do write-in votes for whatever they want, without a seeing a shortlist, and then they won't have as much strategy. One of the problems with that is that slates like the SPs are likely to win the Hugo award.

And I don't know if what's on the displayed list is actually popular, anyway, or just what's most popular among the groups that are most motivated to get their nominations in really early

If we have a published longlist, I would like it if you get your nominations in early enough to be on it. If you get your nominations in early enough to be on the shortlist, you can probably do it early enough to be on the longlist too. It isn't that different.

... to get things on the 15-long list to make sure everyone else gives up on anything they liked that isn't on that list already,

If something you want is not on the longlist, you can start a campaign to get people to nominate it. If enough people agree, it will wind up on the shortlist anyway. If you want to do that. But if you just want to honestly vote for what you want and ignore what wins, then that's fine too.

and are people going to tell me that I'm "wasting" my nomination if I don't nominate one of those things?

Of course they will. They want their favorites to be winners, and some of them will tell you whatever they think will get you to vote for those favorites. If they offer you ice cream or single-malt in exchange for your promise to vote their way, are you OK with taking their bribes and lying about it? If so your stomach might be pleased.

Even if I didn't like it?

Yes, they will probably say that. But you don't have to vote for anything you don't want to.

But now suddenly I need to strategize, when I'm still trying to suggest things to people!

You don't have to do that. Some people will. The SPs claim that some people have been doing that all along enough to get their favorites to win while you, behaving openly and honestly have lost unless you accidentally wanted the same thing the manipulators did. Whether or not they're right about that, they are openly trying to manipulate the vote. Others vote for things that can win in the second part of the nominating, have a chance to overcome the slates, voting for things they honestly like. Without that, it looks unlikely.

In general, I don't like voting systems in an award like this that make strategizing more valuable than honestly stating my opinion.

If you honestly state your opinion you will have an effect on the election.

If the final winner is important to you, we haven't found an adequate way to prevent strategizing and other people will use that to win. If you like one or more of the nominees that have a chance, you can affect their chances. If you don't like any of them enough to vote for them, then don't.

It's possible to be true to your honest beliefs and still have an effect on the Hugos. Or for that matter you can vote for what you honestly think is best with no thought for whether any of it wins or not. If you nominate before the proposed longlist is released, then you will have to do that because there will be no reliable information to influence you.

#628 ::: J Thomas ::: (view all by) ::: April 29, 2015, 09:31 PM:

@626 Fade Manley

The information I want to give the Hugos is "What things did you think were good?" and "How good do you think these specific things are, relative to each other?"

You absolutely can still do that.

I do not want to have to consider what other people think about these things at any point along the way.

You don't have to give any consideration to what anybody else thinks. Just, if nobody gives any consideration to that, the SPs (or somebody of that ilk) will probably win. People who want to stop them can set up their own slate and fight fire with fire. Nothing to do with you, you're just telling the Hugo's what you think is good, which is fine.

This longlist thing gives people a chance to work against slates without making slates themselves. It looks to me like a minimal thing that can work against the ugliest politics.

But if you don't care who wins and you aren't interested in doing anything about that, then none of this need affect you at all. It's when you care who wins that you start getting political.

If you don't care who wins then the whole thing turns into Somebody Else's Problem.

#629 ::: P J Evans ::: (view all by) ::: April 29, 2015, 09:36 PM:

If the 'longlist' looks like a slate, and is expected to act like a slate, in that people decide what to put in based on what they're seeing on it, then it might as well be a slate, and that's what we absolutely DON'T want.

#630 ::: Fade Manley ::: (view all by) ::: April 29, 2015, 09:51 PM:

J. Thomas @628: You don't have to give any consideration to what anybody else thinks. Just, if nobody gives any consideration to that, the SPs (or somebody of that ilk) will probably win.

So your answer to "I feel uncomfortable with policitizing the awards" is "That means the terrorists win"?

It is starting to feel in here like I am being told that if I don't do some special fancy technically-not-a-slate-we-swear version of slate-voting, other people doing slate-voting will control things. Well. Fine, then. If the only way to find slate-voting is with more slate-voting, they can have the whole thing. I would rather not have the awards than use their tactics against them. Even if it's a fancy version of their tactics with a wink and a nod to pretend it's not really a slate.

#631 ::: Steve Halter ::: (view all by) ::: April 29, 2015, 10:07 PM:

Fade Manley@626:Yes to all of that.

P J Evans@629:And yes, exactly to that.

I think that the "Long List/Don't Waste Your Vote" measure would not do terribly well in the business meeting. It seems to run against the general feeling we want out of the process that Abi expressed so well in her post introduction a couple of weeks ago.
One major change to the nomination process is probably the most that could be expected in one go. SDV-LPE seems more in line with the general goals and so seems like the better choice to try to get passed.

#632 ::: J Thomas ::: (view all by) ::: April 29, 2015, 11:48 PM:

@629 P J Evans

If the 'longlist' looks like a slate, and is expected to act like a slate, in that people decide what to put in based on what they're seeing on it, then it might as well be a slate, and that's what we absolutely DON'T want.

I have to respect the depth of your feelings.

I'm kind of flabbergasted by this. I don't understand it. Did people feel this way, with anything like this intensity, before the Puppy attacks?

I get that this proposal is unacceptable to you. I don't understand what you want well enough to begin to make a proposal that could be acceptable. So I want to try to get a feel for that. I have some beginning sense of what it is you don't want.

If you'd be willing to clarify it some, maybe we could find something workable. Again, I don't want to say you're wrong, or dismiss your concerns. I just plain don't understand and I want to.

#633 ::: Keith "Kilo" Watt ::: (view all by) ::: April 29, 2015, 11:51 PM:

JT@628:

I agree with you that no one has to look at the long list -- they can simply vote however they would have anyway. The problem with that is that if you are aware that -other- people are looking at the long list, then you are also aware that there is a good chance that your vote will count less, since you may not be voting for the "emerging slate". That puts the our voter at a perceived disadvantage. There may be something to be said for having the community publicly come to a consensus without even actually having an election, but because of the perceived disadvantage, it looks to me like posting a long list mid-way through fails the "appearance of fairness" test outlined at the beginning of this thread.

Just my thoughts,
Kilo

#634 ::: P J Evans ::: (view all by) ::: April 30, 2015, 12:04 AM:

632
The way its' been done is that the nominations are counted and the top 5 (or 6, if there's a tie for 5th), after any withdrawals and disqualifications, are what goes on the final ballot. The nomination process is easily broken by ballot-box stuffing, as we've seen this year. But it's also easy to explain, easy to understand, and doesn't require a lot of effort on the part of the people submitting nominations. (The people administering it have a bit more work, but it shouldn't be difficult.) Nor should it suggest what people should be nominating - that's what we're objecting to, is that it gives the appearance of suggesting how people should be filling out their nominations. Which is what we specifically don't want.

#635 ::: J Thomas ::: (view all by) ::: April 30, 2015, 12:22 AM:

@631 Steve Halter

It seems to run against the general feeling we want out of the process that Abi expressed so well in her post introduction a couple of weeks ago.

That would be this?
http://nielsenhayden.com/makinglight/archives/016191.html#016191

I like that.

...the passage also echoes what, precisely, is the difference between the rather chaotic means of choosing the Hugo that has evolved over time and the Sad Puppies’ slate-based, goal-oriented one.

The problem is precisely that the chaotic nomination process is susceptible to defeat by a small army marching in lockstep.

I thought the armies could be beaten by fans who gradually came to a consensus, but it sounds like that is not acceptable either.

I had been looking at a very different idea. If instead of an election, the nominations became a sort of survey. We would have a mechanical process to choose five nominations that are each reasonably strongly supported in themselves, and that together represent as many fans as possible.

The one that got the most votes wouldn't necessarily be included, if some other combination of nominees gave at least one "winner" to more nominators. It would be very hard to predict which nominees would be on the final ballot, except that your nominations would improve the chances of the works you nominated.

I thought that something like that might fit the Hugos' needs. But it looked like it would not be accepted -- it was too different, too chaotic, too different from an election where the most votes win.

But as I think about what you've said, and what Abi said, I begin to wonder whether this might be worth more study. The more it's like an election where the most votes win, the more political it's likely to be.

Maybe a surveying system which represents the people who make nominations, which tries to select a diverse collection of works that have significant support from diverse fans might perhaps be acceptable after all?

#636 ::: felice ::: (view all by) ::: April 30, 2015, 03:51 AM:

David Harmon @623: "The idea smells like trouble to me, basically opening the door to all sorts of strategies and feedback loops."

What sort of strategies? Getting as many people as possible to nominate a work before the deadline is hardly unique to this proposal. And the feedback is once only, not a feedback loop in the sense of past feedback influencing future feedback like the Doctor Who howlaround.


Fade Manley @626: "Given that several people, including me, seem to be under the impression that having that 15-item list is a way of focusing votes on those particular early front-runners... Yes, I absolutely think that the sort of people who want to do slate-voting would try very hard to take control of that early list for exactly that purpose."

Yes, it's intended to focus votes on the front-runners, and yes, slate voters would likely try to get their choices on the longlist. But the question is, who wouldn't nominate before the longlist deadline, knowing how important it is? If everyone nominates before the longlist deadline, there's no advantage to any particular group.

Voting for this year is open now; I've already made a few selections, and will gradually update my ballot over the next three months as I evaluate the finalists. Is there any reason not to nominate the same way? Ie enter your initial favourites as soon as nominations open, and update whenever you read anything new that you think is more deserving?

"And it feels like you're suggesting that if I don't like that, I ought to...rally the vote around things I like? But I don't want to rally votes. I want people vote for the things they like without any consideration to whether or not other people like it. "

In #620 you said "when I'm still trying to suggest things to people!" - how is that different from rallying votes for the things you suggest?

"The 5-item list is set. I'm not "strategizing" when I rank them; I'm expressing how good I think they are relative to each other. The 15-item list is not set. It's a potential short-list."

If you don't want to seriously campaign for something outside the longlist, it's close enough to being set that you may as well treat it as such. The chances of anything outside the top 15 making it to the top 5 is pretty remote. Would you feel differently if the longlist was set, and anything outside it was explicitly taken out of the running?

"I mean, at that point you might as well have one of those 'awards' that's based on mass popular vote and shows percentages going to different items through the whole voting process, with people shouting on social media to, quick, go vote for THIS book, it's losing against THAT one."

I don't want the Hugos to be that sort of award either, but I really don't think Option 5b would have an effect anything like it (the original Option 5 probably would have done, but I'm not supporting that proposal).


P J Evans @629: "If the 'longlist' looks like a slate, and is expected to act like a slate, in that people decide what to put in based on what they're seeing on it, then it might as well be a slate, and that's what we absolutely DON'T want."

It's no more of a slate than the five works on the final ballot are a slate. In fact, it's less of a slate; it essentially is the final five, plus the option to consider some of the runners up to decide if any of them would be more suitable finalists. If we said we were going to make a longlist from nominees received before 10 Feb, but then kept the longlist secret instead of publishing it, I'd bet money that the top five works on the secret longlist would be the same ones that ended up on the final ballot.

#637 ::: David Harmon ::: (view all by) ::: April 30, 2015, 08:31 AM:

felice #636: What sort of strategies?

Fade Manley already discussed the "focusing" issue...

The chances of anything outside the top 15 making it to the top 5 is pretty remote. Would you feel differently if the longlist was set, and anything outside it was explicitly taken out of the running?

And that's the flipside, because in the current regime, that's not necessarily true -- a lately-publicised entrant can easily come up from behind in the late voting (see above re: "bathtub curve").

So this encourages at least two opposed strategies: Stick to the longlist and push your favorites from there, or push a dark horse to come in from behind. And notice that I'm not just talking about "your own vote", because discussions among the voters are part of the process.

That sort of fight between two opposed strategies is exactly what I'm talking about when I worry about "instability" -- the outcome of the vote starts depending less on which work is, or was, more popular, and more on which voters talk to whom, and when they do so. Some instability is native to the process, but normally it gets sorted out quietly. An interim longlist would add more instability in a very prominent manner.

That is, it adds another layer of feedback (yes, even if it "only happens once") to the already-chaotic process of the voting, disrupting the normal settling process of "public opinion".

Again, I really don't think this is a good idea.

#638 ::: Andrew M ::: (view all by) ::: April 30, 2015, 10:08 AM:

Felice@636:

Voting for this year is open now; I've already made a few selections, and will gradually update my ballot over the next three months as I evaluate the finalists. Is there any reason not to nominate the same way? Ie enter your initial favourites as soon as nominations open, and update whenever you read anything new that you think is more deserving?

Yes. There is a very good reason why many people do not enter their initial favourites as soon as nominations open; when nominations open, they have not read the things that they might nominate.

This is a point that has been made before, but I feel needs to be emphasised; many people do not regularly read short fiction, and many people do not typically read novels in the year in which they first appear. When such people become nominators for awards, they have to run around at nomination season looking for significant stuff to read as potential nominees. I think it's clear that a lot of people do this; you see them discussing it on the internet. (When the schedule for the Nebulas was changed there were complaints that there wasn't time to read everything relevant between the close of the eligibility period and the end of nominations. Some people responded that you weren't meant to do that, you were meant to have been following the field throughout the year. But clearly a lot of people don't have time - or money - to keep up with the field in this way. And these people were writers, unlike the Hugo voters who are just fans.)

I realise, of course, that you don't have to read a lot in order to nominate; but you have to read something; and ideally more than one thing in each category in which you nominate, since you can't guarantee that the first thing you will strike you as award-worthy. (If you could guarantee that, you wouldn't have to read it.)

I see a lot of people not getting this, working from the assumption that one has read quite a lot of new stuff during the year, and needs to be reminded of stuff one has read that is eligible; but for many people it isn't like that.

This explains a number of things. It helps to explain why most nominations come in at the end of the period. It helps to explain why so few people nominate, of those who are eligible. It also goes a long way to explain why there is a a degree of convergence in nominations, so that historic results can look a bit slate-like to outsiders; we don't know exactly what the five Novel finalists would have been without Puppy intervention, but we can probably narrow them down to eight or ten. It's not that people, after reading works and forming impressions of them, are coordinating votes in order to maximise success; rather, people are coordinating their reading. When looking for works they might nominate, they look around for things that are being recommended and are getting a lot of discussion, and there will be a few such things that stand out each year.

And I am worried by the idea of a longlist, because the people who can nominate early will include those who have been reading new stuff throughout they year, but they will also include those who don't need to read stuff in order to nominate it - the committed slate voters.

#639 ::: Andrew M ::: (view all by) ::: April 30, 2015, 10:12 AM:

I am separating out this question so it does not get lost in the verbiage of my previous comment, since I'm still not clear about the answer. Do those of you who are proposing new methods of voting expect them to produce results similar to those of the Hugo process before the Puppy intervention?

#640 ::: Jameson Quinn ::: (view all by) ::: April 30, 2015, 10:31 AM:

@639: Yes.

In a statistical sense: as long as the correlations between works are low (technically speaking: the first moments dominate the second moments), the results will tend to be the same. This can be formally stated and demonstrated.

#641 ::: J Thomas ::: (view all by) ::: April 30, 2015, 11:16 AM:

Felice writes up the proposal and suddenly we have six people opposed who have not been participating much recently. One new lukewarm acceptance. Some of the opposition is viscerally strong.

Even if it turns out that it's a small but intense opposition, that's likely to turn into intense opposition at the business meeting. It might pass with a lot of public bad feeling, or it might fail.

I didn't expect this. I thought that this part would pass easily and it was the voting system that would run into trouble. We need to find out what changes would be acceptable. There might be a way to make improvements that would not call up such negative emotion.

#642 ::: Lydy Nickerson ::: (view all by) ::: April 30, 2015, 11:19 AM:

People care a lot about the Hugos. This is a true fact. People have complicated lives. This is also a true fact. A lot of people, people who care, get around to nominating and/or voting last minute because lives are complicated and this sort of thing gets pushed off to the last moment. Procrastination happens, this true and verified experimentally. See Kevin Standlee above.

If most of the electorate ate, drank, and breathed the Hugos, a step including a longlist might make sense. But most of us don't. And by creating the longlist, you are creating the feeling that there are "more qualified" voters, people who have the time and energy to engage in the longlist. The responses to those of us who don't like it has very much come across that way. I do understand that this is not what you mean to say, but I can tell you that even listening with sympathetic ears, what I keep on hearing is that I'm not really good enough to participate in this process. I know you think that this is a system where engagement is optional. That's not what it looks like from over here.

JT asked about the historical reaction to campaigning. It would be difficult to describe how incredibly antagonistic our community is to campaigning. Even the informational posts, stating what one has published which is eligible, are controversial. I personally like them, but they are viewed as campaigning, rather than simple informational items, and they upset a lot of people. The move to the longlist, which seems to actively encourage campaigning, is an incredibly bad idea in this context.

And no one has engaged with the fact that adding another deadline to the convention timeline is a significant piece of administrative work. You seem to think that it's trivial. I assure you that it absolutely is not. There needs to be a huge benefit to justify the administrative overhead.

#643 ::: Fade Manley ::: (view all by) ::: April 30, 2015, 11:42 AM:

felice @636: In #620 you said "when I'm still trying to suggest things to people!" - how is that different from rallying votes for the things you suggest?

Oh! I see the lack of clarity there. Let me try to explain this better.

When I make a nomination, I'm suggesting things I think are good to the general group of people who vote on the Hugos. "This," I'm saying, "is what I read and liked." If enough other people agree, it'll show up on the short-list; if not, people who are interested will see it on the long-list later. I might mention to friends which things I'm nominating, but that's mostly the general fannish chatter about things we liked. It's also a bit of a cross-check about "Remember X?" and "Oh, I'd forgotten about Y!"

Now, if EVERY item that was being nominated was being added to a long public list as it came in...this would be just fine by me. People who cared to look would be reminded of things they might've forgotten, and possibly see that other people were recommending something interesting-sounding that they hadn't heard of before. No problem. The more things to look at when sort of casting around for what to nominate, the better.

But as soon as you RANK things, and only display a subset that have gotten enough votes... that's when it gets political. That's when it turns into "I loved X and Y equally, but Y is on the 15 item list and X isn't, so I'd better tell other people who haven't nominated yet to push X, not Y, because X needs more help" and other sorts of...well. Campaigning.

The first version has no inherent politicizing or campaigning or any sort of pressure; it's dumping everything into a giant bucket, unaffected by and large by thoughts of how other people are voting.

The second version, with the long list, exists exclusively for the purposes of seeing how other people are voting, and with a bit of implied finger-wagging that you'd better start taking that kind of strategizing into account if you reeeeally care about what gets on the list. It skews the purity of the "I like X!"

And I would rather give up the Hugos and let the bastards win than start this whole tut-tutting at people who just wanted to throw into the big box the names of the stuff they liked. Because that "Just throw in the stuff you liked, don't worry about what anyone else is doing!" is the whole point of the nominations process.

J Thomas @641: Felice writes up the proposal and suddenly we have six people opposed who have not been participating much recently. One new lukewarm acceptance. Some of the opposition is viscerally strong.

...yup. This is definitely the case. There are some of the voting systems proposed that I end up side-eyeing a little, since I don't particularly like some of their stated goals or potential failure states, but a lot of it comes across as number-fiddling in the background by people who understand voting systems better than I do, and, well. On my end, presumably I am still doing the same thing: suggest things I like. Vote on the short list based on what I like of that.

But the proposal for the midway 15-item list gets a strong visceral impolite response from me. (I edit the profanity out when I try to explain this response, because this is Making Light.) It may not be intended to look like a slate and a prioritization of the things suggested by whoever can nominate the fastest. But that's what it looks like. And when I said so, it really felt like I was told it was, in fact, designed to encourage campaigning, which is the exact opposite of what I want out of any "fix" of the Hugo voting process.

The minority slate problem can, in a sense, be defeated by getting enough more people nominating in more categories with their own enthusiasm. (Look at the novels category, which got at least one legitimate candidate even with the entire slate to work against.) Adding a step that's all about politics doesn't defeat the slate; it just makes slate tactics more necessary.

#644 ::: dotless ı ::: (view all by) ::: April 30, 2015, 11:43 AM:

Andrew M@639: Unless there's a very strong and persuasive argument that the results are better in the absence of slates, then I think we need to get as close as possible to getting the same results. Otherwise the only people likely to support a change are those fascinated with voting mechanisms (and even on this thread that's measurably less than 100% of the population).

#645 ::: J Thomas ::: (view all by) ::: April 30, 2015, 11:44 AM:

@639 Andrew M

Do those of you who are proposing new methods of voting expect them to produce results similar to those of the Hugo process before the Puppy intervention?

Speaking for myself, I like the idea of new methods that get new results that fans like. The new system would have to be something that fans approve of, and it would be hard to get enough fans to quickly approve of anything that looks very different. But I like the idea and I want to see where it leads.

Here's what I would expect of SDV-LPE, the only voting system which is currently under consideration:

1. Next year I would expect the SP-etc slates to get two to three nominations in most categories, and three to four nominations in some. This is because I expect them to increase their numbers faster than regular fan voters increase their numbers. I expect that year after next SP numbers will be small regardless of next year's outcome -- they've been there, done that, and they'll be looking for somebody else to bother. My predictions are highly dependent on things I can't be at all sure about. If they hear that we're ready for them they may just declare victory and go elsewhere next year. I doubt they know themselves what they'll do.

2. Apart from slate wins, I expect a lot of the same winners. It's easy to test -- count the votes both ways. The top winner is guaranteed to be the same. The next four have a good chance. The only way they lose comes if at least two of them on average have ballots with more votes than the ballots for some less-popular work, and the luck of the draw results in them competing with each other and one is eliminated early. That is likely to happen with traditional slates because all their votes are on ballots with a lot of votes. So they will tend to be eliminated, resulting in only 2-4 slate works surviving even though they get more votes than anything else. But that could happen to some of the nonslate nominations with the most votes too.

So apart from the slates, the results should be similar but one or more of the top works may be replaced by less-popular ones. One that is not in the top ten might possibly win, particularly if its voters tend to vote for nothing else.

#646 ::: P J Evans ::: (view all by) ::: April 30, 2015, 11:58 AM:

Fade, that's pretty much how I was thinking of the nominations: you get a card for each category that you can write up to 5 names on (including none at all), and then you throw it in the big bin for that category, and at the end of the period, all the cards in the bin are taken and everything on the cards is counted. If a lot of people like something, it's going to show up on a lot of cards. For some things, like fanzines or short stories, maybe not a lot of people will write any one name, but you'll end up with a long list of names.

#647 ::: J Thomas ::: (view all by) ::: April 30, 2015, 12:03 PM:

@643 Fade Manley

Now, if EVERY item that was being nominated was being added to a long public list as it came in...this would be just fine by me. People who cared to look would be reminded of things they might've forgotten, and possibly see that other people were recommending something interesting-sounding that they hadn't heard of before. No problem. The more things to look at when sort of casting around for what to nominate, the better.

I like that.

I'd recommend that fans at least have the opportunity to see the list by author's name. There tend to be fewer ways to spell the name unrecognizably than the title.

If you have a list of all the current nominations, you are more likely to nominate one of them than start fresh with your own spelling. So it reduces the burden on the administrators.

It doesn't even take a lot of programming, and it's good for fans. It does nothing to solve the problem I want to solve, but it's worth doing for itself.

#648 ::: Steve Halter ::: (view all by) ::: April 30, 2015, 12:16 PM:

Fade Manley@643:That captures my thinking also.

I had actually just been thinking in response to J Thomas@635 asking what would be acceptable in terms of early results and that a list of ALL of the nominated works would be fine with me. Such a list wouldn't need to have the number of votes associated with it, just a simple list of all works nominated to date. It seems like such a list could be produced fairly easily from the electronic ballots and just a bit of administrative overhead to account for misspellings and such. It wouldn't have to have the final accuracy of the actual vote tallying.

#649 ::: Fade Manley ::: (view all by) ::: April 30, 2015, 12:50 PM:

Steve Halter @648: And thinking about the effects of having that full list (without numbers, for sure) building up as it goes, it strikes me that people could respond in a few ways:

1) "I don't see X on the list yet. I should get it in so that other people are reminded to nominate that!"

2) "Oh, there's X on the list! I should definitely nominate that, I liked that story, and at least one other person agrees."

3) "Well, I didn't read a lot of novellas last year, so I don't feel I can nominate any yet... but I could start looking at some of these that have been nominated, and see if any strike me as so good I should nominate them too."

4) "That list is a lot of effort to look at. I'll just nominate what I usually do."

It feels to me like 1 and 2 could cancel each other out relatively well, as distribution goes, while 3 would make for a slight skew towards things nominated early, but much less than the ranked-by-number-of-noms semi-short list. (And 4 is the "treat this like nothing has changed" version, which should produce the same results as always.) So to me this feels fair, while also having the benefit of giving more reading options and reminders to people who are interested in having them at this stage.

But if someone who knows more about how these things works can see a way in which this would significantly bias the results, I would like to hear about that! And of course this is all aside from the additional burden of keeping nominations updated at some reasonable speed, which is a burden on the people handling those things.

#

I am reminded of how Fourth Street Fantasy puts people who have registered to come to the convention on a long list available on the site, at least among those who wish to have their name so publicized. The first time I attended, seeing people I knew as pros and people I knew as fans listed all in that big democratic lump, as opposed to marked out as Guests vs. Attendees, was part of what encouraged me to go; the list felt like an implicit welcome to my point of view. I could be listed the exact same way, and in a sense have as much input in conversations there as anyone else.

So my idea for the really long list is following on how I reacted to something tangentially related in the past. All the things people like, without the numbers, makes it feel like even if my nomination doesn't make it to the final 5? I've still put it out there where interested parties might see it.

And I wonder in passing if that would help in any way with the subset of the Puppies who thought they were honestly trying to make their voices heard to a crowd that was ignoring them. There, on the Hugo website, a place where they can put up for consideration the things they really do love, and know other people will see those suggestions, without any need to do slates to force it onto the final round.

#650 ::: Cheradenine ::: (view all by) ::: April 30, 2015, 01:07 PM:

@647: I agree that one advantage of posting a running list is that it would allow some crowdsourcing of the "are these two items really the same thing?" checks. That probably wouldn't make up for the added labor of needing to update the list on some sort of regular basis, but it might help reduce the workload at the end of the process.

@645: I would actually be pretty surprised if SDV-LPE chose a nominee outside the top ten in the absence of slates; votes for the top ten nominees would have to be very heavily correlated for them to keep trailing #11 in points as members of the top ten get eliminated. (Or the vote counts for #6 through #11 would have to be really close together, which is probably more likely now that I think of it.)

#651 ::: Buddha Buck ::: (view all by) ::: April 30, 2015, 01:53 PM:

I think that if a "long-list" is published early, it should explicitly NOT be ordered by number of nominations. I think it would be giving out too much information to say that work A got more nominations than work B, or that if nominations were to close today, these top 5 works would be the finalist.

If, instead the long-list was explicitly sorted via some neutral criterion (alphabetical by title, alphabetical by author, chronological by date of publication, by increasing/decreasing word-count, etc), no one could use that info to nominate strategically.

#652 ::: J Thomas ::: (view all by) ::: April 30, 2015, 02:07 PM:

@648 Steve Halter

... a list of ALL of the nominated works would be fine with me. Such a list wouldn't need to have the number of votes associated with it, just a simple list of all works nominated to date.

The way I imagine it, any time you log in to nominate or change your nomination, you'd get a list of all the current nominations. You click on the ones you want, or write-in your own. That should reduce the number of misspellings etc a good deal.

If the list *did* include the number of votes so far, that would have some beneficial results but I think you would consider the side effects too bad to consider it. OK.

I have to admit I've never voted for the Hugos. Someone here has said that at any time you can or should be able to change your online vote. And they said that at any time when you are modifying your vote, you can or should be able to add new votes beyond the limit of 5. How much of that is true now? How much of it ought to be true?

I thought about my ideal dream where 5 works get more than 50% approval without politics. If everybody gets 5 nominations, that means half the nominations have to be for those 5 works. That seems implausible. Maybe if we reduced the bar some, and nominators are asked to include up to 25 of the best works, that they think are good enough to deserve a Hugo? (It seems like they'd have to accept a lower standard for that. I might come up with 25 good things but probably not 25 that are hugo-worthy as I think of it.)

As it is now, the 5th winner tends to be on around 10% of ballots, if each ballot had 5 times as many nominees as it does now, maybe that would come out to 50%.

#653 ::: J Thomas ::: (view all by) ::: April 30, 2015, 02:28 PM:

@650 Cheradenine

I would actually be pretty surprised if SDV-LPE chose a nominee outside the top ten in the absence of slates; votes for the top ten nominees would have to be very heavily correlated for them to keep trailing #11 in points as members of the top ten get eliminated. (Or the vote counts for #6 through #11 would have to be really close together, which is probably more likely now that I think of it.)

The scenario I'm thinking of, #11 gets a whole lot of bullet votes, and at least 6 of the others get a lot of votes with say 5 nominees.

So they start out with point scores at 1/5 of their number of votes, and each time they reach the bottom there's a chance two of them reach the bottom together and one is eliminated.

Every time something is eliminated there's a chance their scores will go up and they won't be at the bottom any more, but that isn't certain. If six of them get eliminated before their scores go up enough to make them immune, then #11 is in.

I would expect that to be fairly rare. More likely #11 doesn't have a higher proportion of bullet-votes than the others. It's possible. Nobody would know how to plan based on it, because nobody would know the number of votes on other people's ballots.

As a strategy, if you *think* that the one you care about most is #11, then bullet-vote for that is the obvious choice. Especially don't vote for anything you think is more popular, but if you vote for something you think is less popular that should do less damage. If both win with your half-vote, then fine. If your second choice loses then from that point on it adds no risk for your first choice.

And if (because other people did not bullet-vote when they voted for your first choice) your first choice loses early, then your second-choice still has the best chance you can give it.

#654 ::: Nancy Lebovitz ::: (view all by) ::: April 30, 2015, 02:52 PM:

The complete long list seems reasonable to me. I'd kind of like to see it in order nominated, with options to also show it ordered by length or by author's name.

One advantage of the long list is that checking for eligibility can be partially crowd-sourced.

#655 ::: Jon Lennox ::: (view all by) ::: April 30, 2015, 03:42 PM:

The idea of a complete long list feels fairly similar, in form, to the Pegasus Awards' brainstorming poll.

(The Pegasus awards have two phases: "brainstorming", in which people submit suggestions for eligible works; and nominations, which work a lot like the current Hugo nominations.) The administrators try to do a pass to verify eligibility of works on this list, though things slip through.

In the nomination phase, nominators are under no obligation to nominate works that were mentioned during brainstorming, but in practice the final ballot tends to be works that were on the brainstorming. Thus, people who actually remember to submit for brainstorming (and it's usually a pretty small number of people) have a lot of influence over what ends up on the final ballot.

One notable difference is that brainstorming is explicitly intended to be a "list as many works as you want" process, so it's vulnerable to people dumping every one of their eligible works, though this is usually pretty obvious when it happens.

Also, the list is a one-time publication, not one that's continuously updating during the nomination process. (Probably to keep the web maintenance easy.)

#656 ::: felice ::: (view all by) ::: April 30, 2015, 06:22 PM:

Andrew M @638: "When such people become nominators for awards, they have to run around at nomination season looking for significant stuff to read as potential nominees. I think it's clear that a lot of people do this; you see them discussing it on the internet. (When the schedule for the Nebulas was changed there were complaints that there wasn't time to read everything relevant between the close of the eligibility period and the end of nominations. Some people responded that you weren't meant to do that, you were meant to have been following the field throughout the year. But clearly a lot of people don't have time - or money - to keep up with the field in this way."

If they don't have the time or money to keep up during the year, how on Earth do they find the time and money to catch up on everything in just a couple of months? How many of these people haven't read anything from the last year, and how many are looking for worthwhile stuff they missed to supplement what they've already read? How much of the significant stuff recommended isn't going to get in the top 15 anyway due to nominations by the people who are recommending it to the people looking?

"I see a lot of people not getting this, working from the assumption that one has read quite a lot of new stuff during the year, and needs to be reminded of stuff one has read that is eligible; but for many people it isn't like that."

I assumed that most people who didn't read a lot of new stuff during the year didn't participate in the nomination process...

"the people who can nominate early will include those who have been reading new stuff throughout they year, but they will also include those who don't need to read stuff in order to nominate it - the committed slate voters."

A slate can only get 1/3 of the longlist at worst. Less if the nomination tallying switches to SDV-LPE.


Lydy Nickerson @642: "A lot of people, people who care, get around to nominating and/or voting last minute because lives are complicated and this sort of thing gets pushed off to the last moment. Procrastination happens, this true and verified experimentally."

Yes, absolutely true. And if we publish a deadline for the longlist, people will get around to nominating for that deadline at the last minute. It's human nature, and there's no reason to think people would respond to a longlist deadline any differently to the final nomination and voting deadlines.

"by creating the longlist, you are creating the feeling that there are "more qualified" voters, people who have the time and energy to engage in the longlist."

I do think there are more qualified voters, and I'm not one of them. Oh, do you mean engaging in creating the longlist by nominating early, or in evaluating the works on the longlist after it's published? If you've nominated already and none of the works you liked ended up in the top 15, then none of them would have ended up in the top 5 under the current system. You're not losing anything you have now by failing to consider the top 15 under the longlist system.

"And no one has engaged with the fact that adding another deadline to the convention timeline is a significant piece of administrative work. You seem to think that it's trivial. I assure you that it absolutely is not. There needs to be a huge benefit to justify the administrative overhead. "

It could be done entirely in code with no human involvement required at all. I could write that code if necessary, though other people could probably do a better job. (Though some human involvement would be preferable, of course; merging misspelled variants etc is the sort of thing people are much better at than computers.) And isn't countering the sort of bloc voting that's wrecked this year's Hugos a huge benefit?


Fade Manley @643: "Now, if EVERY item that was being nominated was being added to a long public list as it came in...this would be just fine by me."

That would be less effective at combating slates, but still possibly useful. How would you feel about making it, say, every item nominated by at least five people?

"That's when it turns into "I loved X and Y equally, but Y is on the 15 item list and X isn't, so I'd better tell other people who haven't nominated yet to push X, not Y, because X needs more help" and other sorts of...well. Campaigning. "

In that situation, I'd expect most people to think "well, Y is on the list and I loved that, so that's the one I'll recommend to people for the next few weeks". I'd mostly only expect serious campaigning when a really obvious candidate doesn't make the top 15 for some reason; eg if Ancillary Justice hadn't been nominated enough to get into the top 15 last year, there would have been a lot of people saying "hey, this is ridiculous, AJ has to be on the ballot - get nominating, everyone!" But in most cases, any such obvious candidate will be in the top 15, and campaigning for anything that isn't in the top 15 will be a waste of time. Anyone trying to promote "Plague of the Cybermen" for best novel would just have been ignored.

"The minority slate problem can, in a sense, be defeated by getting enough more people nominating in more categories with their own enthusiasm."

How would we achieve that, though? Even without the slate we've got the problem of nominations being so diverse that not enough works are reaching the 5% threshold in some categories, and only the fraction of extra nominations that support the five most popular works help combat slates.


Buddha Buck @651: "I think that if a "long-list" is published early, it should explicitly NOT be ordered by number of nominations."

Yep, that's the plan. Alphabetical, or on a web page, the order could be randomised every time the page is loaded.

#657 ::: David Harmon ::: (view all by) ::: April 30, 2015, 07:33 PM:

Fade Manley #643: a lot of it comes across as number-fiddling in the background by people who understand voting systems better than I do, and, well. On my end, presumably I am still doing the same thing: suggest things I like. Vote on the short list based on what I like of that.

But the proposal for the midway 15-item list gets a strong visceral ... response from me.

Yes, this. For the voting systems proper, I don't have the expertise or energy to really join the analysis. When I see the proposal for a "top 15" list, I immediately know how I'd react to that, and various ways I'd expect other folks to react, and I don't like it.

On the other hand, listing all the nominations for each category without ranking or counts, doesn't ping my trouble bump at all. (Especially if it's automated and ongoing.) Extra points for letting it be sorted by various factors: Title, author name, length, timestamp of first nomination. As a bonus feature, it allows crowdsourcing the problems of spotting non-obvious duplicates, and noms in a probably-wrong category.

#658 ::: J Thomas ::: (view all by) ::: April 30, 2015, 07:54 PM:

@656 felice

"There needs to be a huge benefit to justify the administrative overhead."

It could be done entirely in code with no human involvement required at all.

People come up with objections and you answer them, and they are not convinced. I think what's happening here is that they have a deep emotional objection, and they come up with rational objections to go along with that. But answering the picky rational points will not get you anything even if you are right. Because those are not the point. Those are what they come up with when we don't get the point.

As I understand it, the point is that they think any politics around the Hugos is bad. Any time people find out which works are ahead and let it influence their voting -- is bad. And this is not negotiable.

So our choice must be to either hope that this is a small faction that can't do much and ignore them, hoping we can get our proposal passed in spite of them. Or we can look for ways to help the situation that they will not oppose and preferably that they will support.

Forbidding any communication among fans that involves knowing which works are ahead, cripples this whole approach. We depend on fans coalescing around winners that they like, that they see can win. They say that's as bad as slates.

We could try to show why it is not as bad as slates, but it looks like they think it's too bad to allow, even if we convince them that actual slates are worse.

I tried to get a sense of what sort of changes are allowable, and so far what I've heard is that it's OK to change the voting system as long as it's fair and explainable. And they want to get more people to nominate.

Anyway, the bottom line is it won't do any good to argue the proposal point by point. They are opposed for important reasons, and we need a better sense of the important constraint.

#659 ::: Michael I ::: (view all by) ::: April 30, 2015, 08:35 PM:

My thought on publishing the "top 15" list is that you're effectively adding a third stage of voting.

Currently you have a nomination phase and a final selection phase. The SDV-LPE proposal is basically aimed at making the nomination phase work in a way that feels reasonable and prevents a small organized group from sweeping the nominations.

Publishing a "top 15" list effectively breaks the nomination phase into two phases: one that works similar to the current nomination phase but selects the top 15 instead of the top 5, and a second that winnows the top 15 down to a top 5. In that sense, it's a more fundamental structural change than the proposal to shift to SDV-LPE voting for the nomination phase is.

(And it seems to me that the actual effects are less predictable than the effects of a shift to SDV-LPE voting.)

#660 ::: Patrick Nielsen Hayden ::: (view all by) ::: April 30, 2015, 09:33 PM:

Lydy Nickerson's #642 is right on the money: "And no one has engaged with the fact that adding another deadline to the convention timeline is a significant piece of administrative work. You seem to think that it's trivial. I assure you that it absolutely is not. There needs to be a huge benefit to justify the administrative overhead."

Felice's assertion in #656 that "It could be done entirely in code with no human involvement required at all" makes me pinch the bridge of my nose. No it certainly could not. This is one of the primal nerd fantasies, the idea that we can fix our political and administrative problems with sufficiently clever code, "with no human involvement required at all". Yeah, that always works.

I think Felice has said some very cogent things, but in this particular she is demonstrating an incomplete understanding of the actual workflow effects, in a rules-based organization, of the stuff she's proposing. I admit that I am hampered here by having a certain amount of pertinent experience.

J. Thomas, in #658, characterizes people (like Lydy) who disagree with Felice like so: "People come up with objections and you answer them, and they are not convinced. I think what's happening here is that they have a deep emotional objection, and they come up with rational objections to go along with that." This is of course glib horseshit, but more to the point, it's factually wrong.

Many thoughtful and interesting things have been said in this thread, but relatively few of them have come from J. Thomas, and yet his posts are beginning to constitute a greater and greater percentage of what appears here.


#661 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 30, 2015, 09:42 PM:

J Thomas @658: I hope you're not falling into the "You are arguing from emotion, I am arguing from reason" trope. That one's like turning yourself into a giant snake: it never helps.

===

I agree with everyone who's saying that the long list idea imposes vast potential administrative overhead. Felice, I'm sorry, but Hugo rules are built to accommodate the inevitable edge cases, odd circumstances, and other regularly occurring anomalies that turn up in the awards. A change that imposes multiple requirements that have to function in relation to the other Hugo processes is going to generate a lot of work.

Fandom has an unnatural affinity for edge cases.

Also, I estimate the odds as just about zero that fandom will approve a rule change that excludes some fans from the nomination and election processes. We have a real aversion to things like that.

#662 ::: Lydy Nickerson ::: (view all by) ::: April 30, 2015, 09:48 PM:

J Thomas @ 658: You appear to be assuming that emotion and intellect are utterly divisible, or perhaps even antithetical. Which is a very broken view of how human brains work. There's actually a lot of neuroscience that shows this to be an unsound model of intelligent thought.

In terms of administrative burden: as far as I know, and I'm sure Kevin will correct me if I'm wrong, there isn't a mandated time period for nominations and voting. The convention opens up nominations when they've got their software and hardware and people-points ready, and close them at the point that they need to print ballots, which will be dependent upon a whole bunch of factors which will change from year to year. Mandating a longlist be published at the midpoint means, among many other complexities, establishing when that midpoint is. Which takes away a certain amount of flexibility for the concom. Flexibility which they really, really need.

It is also making me crazy that you are assuming that everybody is, of course, online and interacting with the nomination and voting process online, and that those three ballots that were paper can be ignored because it's such a small minority. These people, let's pretend for a brief moment that there really are only three of them, are IMPORTANT. You cannot write them off, ignore them, treat them as if they are not part of who and what we are. Despite the bizarre claims currently floating about in the data sphere, we care very deeply about disenfranchisement, and we are completely opposed.

#663 ::: J Thomas ::: (view all by) ::: April 30, 2015, 10:17 PM:

@660 Patrick Nielsen Hayden

J. Thomas, in #658, characterizes people (like Lydy) who disagree with Felice like so: "People come up with objections and you answer them, and they are not convinced. I think what's happening here is that they have a deep emotional objection, and they come up with rational objections to go along with that." This is of course glib horseshit, but more to the point, it's factually wrong.

Whoa. I am convinced that the deep emotional objection is primary, in that if all the practical objections are overcome but that is not, they still will not agree.

Do you think this is wrong?

If that's right, then there is no point trying to deal with the yes-buts before we handle the gotcha. If they think it is fundamentally the wrong thing to do, then it doesn't matter how many of the details we work out to show that it is feasible.

Imagine I was arguing for a nuclear first strike on Russia. I could present evidence that we had a good chance to destroy their entire second-strike capability. That there would be no nuclear winter. That radiation effects would be minimal. That our allies would not be harmed. But none of my arguments why the war would be successful would have much impact on you if you just plain don't want to have a nuclear war, successful or not.


This is mostly irrelevant when the big objection stands, but:

"And no one has engaged with the fact that adding another deadline to the convention timeline is a significant piece of administrative work."

Felice's assertion in #656 that "It could be done entirely in code with no human involvement required at all" makes me pinch the bridge of my nose. No it certainly could not.

The proposal calls for the code which handles the online-generated nominations to produce a list of the 15 nominations with the most votes, and publish them on the website. This should happen at a particular date.

If all the paper ballots which have been received must be entered before that date, that is a human thing. In one recent year there were 3 of those. If some paper ballots were received earlier but not entered by then, that would be a minor flaw in the procedure.

If some nominations were mismatched -- two people nominated the same thing with somewhat-different names -- and that did not get straightened out before that date, that would be another flaw in the system. Usually it would be works that would not get onto the longlist. There are typically hundreds of works nominated, and only 15 on the longlist. Usually when it was a work that would get onto the longlist if both nominations were combined, it would still get onto the longlist with only one. But there is a possibility for an unfair result. The idea to have the website make the whole nomination list public throughout the nomination period might reduce the chance of that happening, since many eyes could check it and report possible problems. That might not be less work, though, since there might be a collection of garbage reports to sort through.

If authors on the longlist declined at that point, it would take some human intervention. Probably better that they decline then than that they decline after they get onto the shortlist.

Felice has put a lot of effort into this, and she's thought out a lot of details. If we're willing to have paper ballots and mismatched nominations be resolved whenever, rather than put a deadline before the longlist publication, it could all be automated. Authors who decline could be an issue, but five of them have to be dealt with anyway, and this gives extra time to do it, plus there's the crowdsourcing side of it -- if they hear they're on the longlist some of them will step forward whether they want to decline or not. It might wind up less work with a less urgent deadline.

But how easy it is to do is irrelevant if we can't agree that it's worth doing.

#664 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 30, 2015, 10:37 PM:

J Thomas, this might be a good point at which to quit for the evening.

#665 ::: J Thomas ::: (view all by) ::: April 30, 2015, 10:38 PM:

@661 Teresa Nielsen Hayden

J Thomas @658: I hope you're not falling into the "You are arguing from emotion, I am arguing from reason" trope.

No, maybe it would be clearer to talk about means and ends.

It does no good to argue that the project is feasible -- the means -- when we haven't agreed that it's worth doing -- the ends.

I don't say that people are being emotional so they can be ignored while the rational adults solve the problems. Quite the reverse. People who care deeply about their concerns are not going to be argued out of those concerns with 2-line or 20-line or 200-line arguments that present logical arguments starting from the wrong assumptions.

So I want to get a clear sense of what the concern is. I'll have it right when I can say it in my own words and people agree that's it.

What I have now, is that Hugo nominators should not do strategic voting but should vote for what they really think deserves the nomination.

One form of strategic voting is the Compromise Strategy. Instead of voting for what you really want, instead you vote for something you don't want nearly as much but you think you can win. (Democrats and libertarians might gag a little when they think about that.) We can prevent that strategy if we never tell anybody which nominees can win.

Is that it?

#666 ::: Teresa Nielsen Hayden ::: (view all by) ::: April 30, 2015, 10:41 PM:

I do most earnestly desire the welfare of your soul, and your happiness here on earth.

Tomorrow is another day.

#667 ::: P J Evans ::: (view all by) ::: April 30, 2015, 10:45 PM:

665
We've never, AFAIK, told anyone which nominees can win. We haven't done slates, before the juvenile canines. We think that's the wrong way to go.
A list of nominations, alphabetical by author or title, if it can be done without taking a lot of time, might be a good idea - but, having been on the inside, I think you overestimate how much time people have. The administrators are doing this in their copious spare time, not as a full-time paid job. (And, having been dealing with databases for decades, they're a lot more work to maintain that most people think, even with good software.)

#668 ::: felice ::: (view all by) ::: April 30, 2015, 10:49 PM:

Patrick Nielsen Hayden @660: "Felice's assertion in #656 that "It could be done entirely in code with no human involvement required at all" makes me pinch the bridge of my nose. No it certainly could not."

if (now < DEADLINE) {
  print "longlist not yet available";
} else if (fileexists('longlist')) {
  print loadfile('longlist');
} else {
  $longlist = '';
  for each ($cat in CATEGORIES) {
    $longlist += '\n\nCategory: $cat\n\n';
    $orderedlist = select top 15 name, count(*) as noms from nominationstable where category = $cat group by name order by noms desc;
    $longlist += select name from $orderedlist order by random;
  }
  savefile('longlist', $longlist);
  print $longlist;
}

This doesn't deal with variant spellings etc, which as I pointed out in #656 are much easier for humans to spot; there are ways code can reduce them, though (eg getting the website to ask "did you mean X?" if nominators type something similar to but not exactly the same as work that's already in the database).

#669 ::: Keith "Kilo" Watt ::: (view all by) ::: April 30, 2015, 11:26 PM:

The idea of presenting a cumulative list all the nominated works in real time is interesting, but ultimately, it can be a separate proposal (and it was proposed as such), as it can work with or without SDV-LPE. If the community doesn't mind, I'd really like to finish up the SDV-LPE proposal before we get too embroiled in a second debate. Would it be acceptable to table the debate on this for just a bit?

In that spirit, it seems to me the biggest remaining issue to settle is how to handle ties in SDV-LPE. Allow me to summarize what has been proposed so far:

Ties in SDV-LPE

1.Current system:
If there is tie in the lowest point total for two or more works, then do not consider the second-lowest point total works for elimination this round (i.e., eliminate the lowest point total work that also got the fewest nominations).

If there is a tie in the second-lowest point total for two or more works, consider all of them, along with the lowest point total work, for elimination.

If there is a tie in which work (among those being considered for elimination) received the fewest nominations, eliminate all the tied candidates unless this would reduce the final ballot to fewer than five works. In this case, keep all the tied candidates and extend the final ballot to more than five works.

2.felice@523:
If there is a tie in which work (among those being considered for elimination) received the fewest nominations, randomly select which work to eliminate. {Kilo’s comment: I can’t see fandom accepting any non-deterministic method of eliminating a work from the potential ballot.]

3.JT@540 (version 1):
If there is a tie in which work (among those being considered for elimination) received the fewest nominations, then count the number of ballots in which the work received a fractional vote (i.e., appeared on a ballot with other works). The work which appeared on the most ballots along with other works is retained. [Kilo’s comment: Keep in mind that this is essentially the opposite of the SDV portion of the system, as in general the one with more fractional votes will have a lower total score than one that does not – I find this somewhat confusing.]

4.JT@540 (version 2):
If there is a tie in which work (among those being considered for elimination) received the fewest nominations, then count the number of times each candidate work appears on a ballot with the current top five works. The work that appears the fewest times with the current leaders is eliminated. [Kilo’s comment: This strikes me as somewhat arbitrary, since the current leaders can change from round to round.]

5.JT@547:
If there is a tie in which work (among those being considered for elimination) received the fewest nominations, then look at the point totals for those works being considered again. The one with the lower point total is eliminated. [Kilo’s comment: This doesn’t deal with the case in which they are tied for both nominations and points, which I think might not be rare – this would require adding yet another tie-breaker.]

6.felice@557:
If there is a tie in which work (among those being considered for elimination) received the fewest nominations, then eliminate the one that was nominated last chronologically. [Kilo’s comment: What do you do if two people nominate the same work at different times? Also, this rewards early nominating which may not be fair, given the historical patterns in nominations, as well as the issue that people may not have read all the works they want to consider yet.]

7.Joshua Kronengold@564:
If there is a tie in which work (among those being considered for elimination) received the fewest nominations, then as a first tie-breaker, eliminate the one with the lowest total points received. If there is a tie for lowest total points, then “lowest sum of placement” determines the work eliminated. If there is a tie for “lowest sum of placement”, then the item nominated last chronologically is eliminated. Keep all ties when there is a tie for fifth place. [Kilo’s comment: Sorry, but I don’t understand what is meant by “lowest sum of placement”, so I’m not sure I can evaluate this.]


Does this cover all the tie proposals so far?

Kilo


#670 ::: Keith "Kilo" Watt ::: (view all by) ::: April 30, 2015, 11:32 PM:

And some additional commentary:

felice@550: certain rare ties can break the intended functioning of SDV-LPE
Actually, I'm not sure what cases you're referring to. I've not yet run into a case that breaks it, though I admit I've probably missed it in the discussion. Could you lay that out for me?

Joshua K@569:My concern with my "kill them all" tiebreak is that while it punishes slates, it also punishes highly correlated nominations (like, say, fandoms for particular TV shows) that happen to tie.

It would only eliminate both if they were not near the final five. In other words, if they are likely to win, the fact that they are tied makes it more likely to -keep- both rather than eliminate both under the current system.

Honestly, the only problem I can possibly imagine with the current system is the case when you have a 6-way tie (or whatever) for fifth place. In this case, the rule says that we keep them all and have 11 works on the final ballot. In my view, if 11 works were tied for number of nominations, then it's not unreasonable to keep them all on the ballot. I also think the odds of this happening are vanishingly small, but again, not a problem if it does happen -- it's just an interesting year.

It seems to me that adding tie breakers dramatically complicates the system to not much gain (not zero gain, mind you). Eliminating all tied works in the early rounds really doesn't hurt anyone, in any case that I've been able to sim. The only real possible problem I see is that you might in very rare cases get a long final ballot in a category. Personally, given that it should be rare, I really don't see a problem if it happens every once and a while. Remember, we're just coming up with a list of candidates for the Hugo, not the winner itself.

What are your thoughts?

Kilo

#671 ::: felice ::: (view all by) ::: April 30, 2015, 11:52 PM:

Keith "Kilo" Watt @669: "If the community doesn't mind, I'd really like to finish up the SDV-LPE proposal before we get too embroiled in a second debate. Would it be acceptable to table the debate on this for just a bit?"

Or start a new thread?

"If there is a tie in which work (among those being considered for elimination) received the fewest nominations, then eliminate the one that was nominated last chronologically. [Kilo’s comment: What do you do if two people nominate the same work at different times? Also, this rewards early nominating which may not be fair, given the historical patterns in nominations, as well as the issue that people may not have read all the works they want to consider yet.]

Only consider the first time a work was nominated. In a two way tie, it would be more intuitive to say "keep whichever was nominated first", but that gets awkward for three-or-more ties. Any true tie-breaking method is inherently unfair, because if there was a fair way to pick one over the other, there wouldn't be a tie (methods that involve looking at more data, eg point total as well as number of ballots, I consider more like a photo finish that a tie-break, and you can theoretically get ties all the way down, eg a slate with perfect discipline and zero overlap with non-bloc voters). This should be the last resort tie-break, eg after 5.JT@547 and whatever else.


Keith "Kilo" Watt @670: "felice@550: certain rare ties can break the intended functioning of SDV-LPE
Actually, I'm not sure what cases you're referring to. I've not yet run into a case that breaks it, though I admit I've probably missed it in the discussion. Could you lay that out for me?"

See #520

#672 ::: Lydy Nickerson ::: (view all by) ::: April 30, 2015, 11:55 PM:

Felice @ 668: Do you understand that you are completely missing my point, here? I don't think than anyone, not even me, thought that the coding was a hard problem.

Putting on a convention requires a huge number of tasks, many of which are interrelated, some of which have complex dependencies, and the administrative task of keeping all these things on the rails is enormous. One of the truisms I learned at one point was that the publications schedule drives the schedule. This wasn't, to me, immediately obvious. However, there are a host of decisions that you don't really need to make right now -- except that you need to publicize them. So knowing when you are making public statements about what vastly constrains which decisions have to be made when. And that's just one example.

I have never been involved in the Hugo awards process, but I can see from the outside some of the obvious problems and time constraints. You are attempting to add an additional requirement, which has significant time-sensitivity, requires publication, maintenance, explanation, transparency, interacts with both the timeline of the nomination and voting process, and the larger convention, including but not limited to publications and publicity (not always the same thing). You are proposing to do this as an actual change to the actual rules, that is to say, you want to mandate it. Which means setting up a whole series of requirements, deadlines, and deliverables, which need to not only be clearly defined, but which can translate from year to year. And I have yet to understand why this will significantly improve the actual nomination process. There needs to be a huge payoff, here.

The SDV-LPE does seem to have that kind of payoff. Moreover, while it requires some additional futzing on the back-end, it doesn't affect the overall timeline that I can see. It changes the way the votes are processed, but it doesn't add a deadline, change requirements, interact with the schedule, etc. And it does have a clear benefit in that it gives a more diverse set of nominees which more clearly represents the interest of the membership. It also does not, as I understand it, encourage slates or campaigning, both of which are significant problems.

#673 ::: Lydy Nickerson ::: (view all by) ::: April 30, 2015, 11:59 PM:

Keith @ 669: Sorry. Didn't see your post before I posted. I think it would be a _great_ idea to finish the SDV-LPE discussion before haring off after the longlist discussion. While I have a lot less to contribute, I think it is actually hugely constructive. Thank you.

#674 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 12:11 AM:

felice@671: Actually I did look at 520. There's no real problem with that, as far as I can see...

It's true that candidates with more total nominations were eliminated, but that's only because their nominators didn't feel as strongly about them as A-E nominators did. That's how it's supposed to work, so I don't see that as an issue.

K

#675 ::: felice ::: (view all by) ::: May 01, 2015, 12:41 AM:

Lydy Nickerson @672: "And I have yet to understand why this will significantly improve the actual nomination process. There needs to be a huge payoff, here."

SDV-LPE is a good thing; it reduces the strength of slates, so they'll only get one or two works on the ballot in each category; or maybe three or four in some categories (in 2013, the second highest placed short story only got 38 nominations; a slate would only need 153 people giving a quarter of their vote to each story to get four places on the final ballot, and "Warbound" got 184 nominations last year). Ie, SDV-LPE helps, but it's not enough. 662 people submitted nominations for short story in 2013; if they'd been told what the top 15 were, and half of them gave one extra nomination to one of the top 15, second and third places would have been at least 60 nominations, and 153 bloc voters could only get two works on an SDV-LPE final ballot. If half of the 662 gave all five of their nominations to works in the top 15 they hadn't originally nominated, 153 bloc voters probably wouldn't get anything on the final ballot. That is the huge payoff of Option 5b.

#676 ::: felice ::: (view all by) ::: May 01, 2015, 12:53 AM:

Keith "Kilo" Watt @674: "It's true that candidates with more total nominations were eliminated, but that's only because their nominators didn't feel as strongly about them as A-E nominators did. That's how it's supposed to work, so I don't see that as an issue."

Not necessarily true. What if everyone nominated five works, but the people who nominated A-E also nominated a wide range of less popular works that got eliminated in earlier rounds? A-E may even have been the least preferred option of all those voters, but survived because they had more broad support. F-I have even more broad support, and could all have been passionately supported by the people who nominated them, but because their supporters happened to be backing more than one popular work, and a fluke tie lead to them being simultaneously eliminated, they don't get any representation on the ballot. If one person fewer had voted for I, the final ballot would have been FHABC (ignoring the five way tie on A-E for simplicity's sake).

#677 ::: Cheradenine ::: (view all by) ::: May 01, 2015, 01:06 AM:

Some brief thoughts on tiebreaking:

I think we want to minimize the number of situations where we eliminate multiple candidates at once, since it can potentially lead to strange behavior (even if the probability is not that high).

I looking up how STV systems handle ties deterministically; based on that, here's a suggestion I'd make that builds on Joshua's from #669:

1) Determine the group of candidates to consider for elimination as in #669, option 1.

2) If two or more candidates in the group are tied for least nominations:
a) If one candidate has the fewest points, eliminate them (as in #669, option 7).
b) If multiple candidates are tied for the fewest points, look at the previous step's results and eliminate the one who had the fewest points in that step. If there is still a tie, look back to the step before that, and so on. (This is referred to as "Backwards Tie-Breaking" in the STV literature.)
c) If multiple candidates are still tied (meaning that they have the same number of nominations and have had the same number of points in each round so far), eliminate all of them. (STV methods would usually resort to a random draw at this point; something like the chronological-order option would also be workable.)

#678 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 01:43 AM:

felice@675:

Actually, I ran both the #520 situation and the situation you suggest (one fewer vote for I -- I changed one HI ballot to HJ) and got identical results: ABCDE. The only difference is that in the original version, H and I get eliminated together, but in the modified version H gets eliminated first, then I gets eliminated the next round. This matches what you would intuitively expect to happen. I got exactly the sequence of rounds that JT predicted in 520 with his ballots, by the way.

Kilo

#679 ::: felice ::: (view all by) ::: May 01, 2015, 02:19 AM:

Keith "Kilo" Watt @678: "Actually, I ran both the #520 situation and the situation you suggest (one fewer vote for I -- I changed one HI ballot to HJ) and got identical results: ABCDE... in the modified version H gets eliminated first"

You can't change it to HJ; J is already eliminated by this point, so H gets a full point. How can reducing the vote for I result in H being eliminated first? Am I doing anything wrong here?

7 votes for A-E
5 votes for FG
4 votes for GH
3 votes for HI
1 vote for H
4 votes for IF

1st round:
A-E 7
F 4.5
G 4.5
H 4.5
I 3.5

I is removed, to the benefit of H and F

2nd round:
A-E 7
F 6.5
H 6
G 4.5

G is removed, to the benefit of H and F

3rd round:
F 9
H 8
A-E 7

#680 ::: J Thomas ::: (view all by) ::: May 01, 2015, 08:08 AM:

7 votes for A-E
5 votes for FG
4 votes for GH
3 votes for HI
1 vote for H
4 votes for IF

A-E 7
F 9
G 9
H 8
I 7

1st round:
A-E 7
F 4.5
G 4.5
H 4.5
I 3.5

I is removed, to the benefit of H and F

2nd round:
A-E 7
F 6.5
H 6
G 4.5

G is removed, to the benefit of H and F

H is removed. I think it helps to write out the votes for each ballot, because it's easy to make one picky little error.

3rd round:
F 9
G 6.5
A-E 7

#681 ::: J Thomas ::: (view all by) ::: May 01, 2015, 08:11 AM:

If you eliminate both tied works, you can eliminate more high-vote works at once. This potentially helps low-vote works to win.

For example:

Ballots:
A 9
BC 10
DE 9
AD 1
AE 1
F 7
G 6
H 6
I 6

Votes:
A 11
B-E 10
F 7
G 6
H 6
I 6

1st round scores:

A 11
B 5
C 5
D 5
E 5
F 7
G 6
H 6
I 6

If we eliminate ties, we now eliminate BCDE.

AFGHI wins.

This is more chaotic. There are more chances for works that got fewer votes to slip into the top because works that got more votes but temporarily low scores, eliminate each other.

Keith is pointing out that it doesn't take ties to do that, but when we only remove one work at a time, it has to happen more often.

For the work in 15th place to win, ten items with more votes have to be eliminated by getting compared to each other, and during that time none of them get compared to the 15th one or it loses. This is possible but unlikely. Eliminating more than one at a time with ties makes it a little more likely.

I don't know which way is better because I'm unclear what goal we are reaching for.

What we have is a voting system that punishes nominees that get a lot of split ballots by exposing them to selection. If they happen to match up with one that got more votes, that is also being punished for split ballots, they can be eliminated.

This will tend to punish slate nominees more than anything else because their ballots are split as much as possible. If we eliminate both when tied, then two ties are enough to reduce them to one winner. If we eliminate one (which will happen anyway if they get a few random extra votes) then the right conditions have to line up four times to reduce them to one winner. Each time one is eliminated, the survivors' scores jump up and we eliminate something else for awhile until they are the lowest scores again.

This is not a random process, it is determined precisely by the votes, but it is kind of chaotic. It's vaguely possible that one changed vote for the 20th place work could have effects that ripple up to change four of the winners.

This might be a feature and not a bug. If you think you know which work has the second-largest number of votes, you don't know that it will win. Something you can't possibly predict could eliminate it. So the Deserter strategy is undependable. If you want something to win, you do better to vote for it unless you're sure it's in first place.

#682 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 09:34 AM:

I'll read your replies in a sec, but you're right, I gets eliminated then H, of course. Latent dyslexia. :-)

Kilo

#683 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 10:12 AM:

JT@680:

You've made a mistake in your second round:

2nd round:
A-E 7
F 6.5
H 6
G 4.5

G is removed, to the benefit of H and F

H and G are compared for elimination, but since H has 8 nominations and G has 9, then H is removed, not G.

In the next round, F and G are tied for both lowest points and number of nominations, so are both eliminated, leaving ABCDE.

All this is assuming there's not a bug in the code somewhere, of course, but so far, it's worked against everything I've checked by hand.

Kilo

#684 ::: Andrew M ::: (view all by) ::: May 01, 2015, 10:19 AM:

Felice@656:

If they don't have the time or money to keep up during the year, how on Earth do they find the time and money to catch up on everything in just a couple of months?

They find the time by making time: saying 'my leisure activity for the next couple of months will be reading Hugo-(or Nebula-) eligible works'. They may be able to find the money because some of the books will be cheaper by the new year, having appeared in paperback. But in any case, they're not catching up on everything. They are catching up on a few outstanding things, things that are getting recommendations. You can't do that in real time: you don't know what's outstanding at the moment it appears.

I assumed that most people who didn't read a lot of new stuff during the year didn't participate in the nomination process...

Well, I'd agree that they would be the people best qualified, but if you want to confine it to them, don't be surprised that nominators are few in number.

A slate can only get 1/3 of the longlist at worst. Less if the nomination tallying switches to SDV-LPE.

One slate can. What if there are three slates?

#685 ::: Andrew M ::: (view all by) ::: May 01, 2015, 10:27 AM:

By the way, one problem with the longlist is that (I take it) there's no guarantee all the works on it will be eligible. Non-eligible entries do sometimes make the top fifteen: Anne Leckie was in the top fifteen for the Campbell last year, although she has short fiction publications going back years. The administrator only makes rulings on entries which make the cut-off for the shortlist, and since this proposal is meant not to increase the administrative burden I take it this will not change.

#686 ::: Jameson Quinn ::: (view all by) ::: May 01, 2015, 10:37 AM:

Here's an idea for accomplishing the same "good" as a longlist, that may be acceptable to the people who don't like the longlist.

On the web ballot, each person would get a personalized list of 5 works others have nominated. Those would be chosen by choosing a random ballot, and then a random work from that ballot. If you went back to edit your nomination ballot, it would tend to show you the same 5 "suggestions". You would be free to do whatever you wanted with those 5 suggestions; they'd just be to "jog your memory".

The first 10 voters wouldn't get any "suggestions". When the 11th voter got their ballot, it would have 5 "suggestions". Something like:

Others have nominated: "That railroad book by the discworld guy", "Flex by Ferrett Steinmetz", "Redpants by John Skalsey", "A Wind of Winter by GRRM", and "Galaxy Conquest 5: Space Marines Don't Eat Quiche by Semper Uzi". If you feel any of these deserve a Hugo, you are free to include them among your nominations.

Say those were from ballots 2, 3, 4, 6, and 10 respectively. Then when the 12th ballot came in, it would have a 1/12 chance of replacing one of those, so that the next time voter 11 logged in, they might see:

Others have nominated: "That railroad book by the discworld guy", "Flex by Ferrett Steinmetz", "The Repossessed by Ursula K Leguin", "A Wind of Winter by GRRM", and "Galaxy Conquest 5: Space Marines Don't Eat Quiche by Semper Uzi". If you feel any of these deserve a Hugo, you are free to include them among your nominations.

Notice that using this "replacement" scheme, a voter can't just log in over and over again to see all the nominations so far; but at any moment that they log in, any of the ballots so far has an equal chance of being among the ones they see. By setting a random seed using the voter's name or other unique ID, you could re-generate the "suggestions" each time they logged in, and they'd come out the same each time. (If somebody needs tips on how to program this, I can give more details, but the point here is that it's possible without burdensome requirements of storing the suggestions for each voter).

The upshot is that voters would be subtly nudged to overlap, but everyone would still be free to use their own judgment. Since the "suggestions" are randomly selected from other ballots, everybody knows that it's not enough information to really enable detailed strategies; it's more just a mechanism to jog people's memories. The administrative burden is essentially zero once it's programmed, and doing the programming is not that much harder than just enabling the web ballot in the first place. (That is, it might be as much as twice the work, but it is not 10 times the work.)

This does mean that early nominations have a smidgen more power than late ones, because they have more chances to "nudge" other voters. But it's not a big difference, and in the end, nothing will rise to the top unless it's cream.

(This scheme would work even better if the 5-nomination-per-ballot limit were lifted, but I understand that it's worthwhile to make the fewest changes possible, so I'm not going to reopen that debate.)

#687 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 10:40 AM:

JT@681:
This is more chaotic. There are more chances for works that got fewer votes to slip into the top because works that got more votes but temporarily low scores, eliminate each other.

I disagree it's chaotic. This is the LPE part of SDV-LPE in action. It sounds like you really want just pure SDV, i.e., the works with the most points "should" win. That's not what SDV-LPE is about. It's about eliminating the least popular work.

Jameson, please correct me if I have a misconception here. Popularity has two aspects: How strongly the voters felt about a work they nominated (fewer works nominated means they value those works they do nominate more -- so they get more points) and how many voters considered the the work worthy at all (and so get more votes for nomination). This means that a work that gets fewer points might get on the ballot because it was nominated by more unique people. On the other hand, if a somewhat fewer number of people feel very strongly about a work (so only nominate that one), then it will have an advantage in points, so may not be likely to be considered for elimination in the first place.

What this means is that slates have much less power if they nominate all five positions -- because they are saying, "We think these works are Hugo worthy, but don't really value any one over another." That weakens how much say they have in the results because each work only gets a fifth of a point. Slates could, on the other hand, push a single work, saying, "Only vote for this work!" and they are very likely to get that work on the final ballot. And that's perfectly okay. That's no different from any other popular work, and more importantly, they haven't taken up all the nomination slots, so other works will be still be considered.

(Gotta get the kids to school, so apologies for the lack of edits...)

Kilo

#688 ::: Jameson Quinn ::: (view all by) ::: May 01, 2015, 10:41 AM:

@685: note that my suggestion in @686 explicitly doesn't address this issue. Perhaps text to that effect should be included in the "suggestions":

Others have nominated: "That railroad book by the discworld guy", "Flex by Ferrett Steinmetz", "Redpants by John Skalsey", "A Wind of Winter by GRRM", and "Galaxy Conquest 5: Space Marines Don't Eat Quiche by Semper Uzi". If you feel any of these deserve a Hugo, you are free to include them among your nominations. Note that these works appeared on other ballots but have not been checked for eligibility; please do not vote for them if you believe they are ineligible.
#689 ::: Andrew M ::: (view all by) ::: May 01, 2015, 10:43 AM:

Jameson Quinn@640: Thanks. I take it, though, that what can be formally stated and demonstrated is that if the behaviour of voters does not change the results will not change: but from the way you put it I also take it that you are not trying to change voter behaviour.

The longlist proposal clearly does aim to change voter behaviour, and I therefore see it as problematic. I also feel that there have been a few places in the discussion where someone has objected 'But wouldn't that give people an incentive to do X' (where X is not something they do now), and the response has been 'But wouldn't that be a good thing?'.

I tried to make my question as non-committal as possible, but I do actually think that the Hugos have produced largely good results hitherto, and it would not be a good idea to change them, or to change incentives (other than the incentive to create slates).

#690 ::: Jameson Quinn ::: (view all by) ::: May 01, 2015, 10:44 AM:

@687: That's a good explanation.

#691 ::: Jameson Quinn ::: (view all by) ::: May 01, 2015, 10:53 AM:

@689: Right. If the system had been SDV-LPE, and voters had behaved the same as in past years, the results would probably have been the same (I'd guess around 95-98% the same nomination lists, and near 100% the same ultimate winners). But this year, of course, the slate would not have been as successful, because the whole point of slate voting is to induce artificial correlations between works.

Note that my "suggestion" suggestion in @686 is intended to slightly increase the overlap between voters, to help the cream rise to the top more clearly; but that it would not tend to increase the correlation. That is, if X was a good work were nominated early and thus appeared as a "suggestion" for many later voters, it would be more likely to get more votes later on, but this suggestion would not mean that people who voted for Y were more likely to vote for X than people who didn't vote for Y. So while it would "nudge" a slight change in behavior, in a way that would tend to reduce the power of slates (by increasing the totals of the best non-slate works), I believe it would not have changed the results in previous years, whether or not SDV-LPE had been used.

#692 ::: Bradley W. Schenck ::: (view all by) ::: May 01, 2015, 10:57 AM:

I've been following this discussion with a lot of interest and zero expertise, so nothing I've had to say about it was worth anyone's time. But I'm really glad to see that the emphasis has shifted back from the longlist for two reasons:

1. It's my understanding that nothing in the rules prevents the longlist from being displayed, so a rules change isn't necessary. All you need to do is to convince a future con committee to try it and see what happens. Wouldn't this be better? Otherwise you'd need to pass a rule change for something that hasn't been tested, may be unpopular, and may require changes before it's ready.

2. I think that a longlist would be a very contentious issue for the voting bloc(s). It could easily be interpreted as a way to pre-screen nominations, with the result that some nominees could be targeted as things to knock off the list. You can't prevent people from drawing an unsupported conclusion (which should be obvious by now) but I think this is an enormous elephant to put in the room. And for my part I share an emotional objection to the idea. It pretty much invites nominators to strategize. That simply feels like the wrong message to send.

I understand that I just put emphasis back on the thing I was happy to see de-emphasized. Don't mind me.

I also think that there must be a very clear description of how a new nominating method works in terms so simple that a casual reader will see its fairness. An infographic that shows how nominees fall out of (or back into) the short list would be very helpful once you have a final proposal. But what I, as a casual reader, would find most convincing is to see that when the new method is applied to historic nominating data it yields substantially similar results.

I've looked at the nominating data for a couple of past years. What I think is missing is the complete, anonymized ballots. That information - if it exists - seems like the best data to use to a.) prove that there have never before been organized voting blocs; and b.) demonstrate how a new nominating method compares to the old one. I have a feeling that those anonymized ballots may not exist, which seems like a problem for either a.) or b.). Isn't that exactly what's needed to test a new method?

#693 ::: Cassy B. ::: (view all by) ::: May 01, 2015, 11:15 AM:

Jameson Quinn @688, your suggestion, although I'm sure well-meant, seems to me to be just yet another way (like the 15-entry longlist) to sneak alternate slates into the Hugos. I really, really don't want ANYONE suggesting to me what I should nominate. Strategic voting offends me on a very basic level. I want to nominate what *I* love, not what someone else tells me I should love.

And won't a suggested list also fuel Puppy Paranoia? (And that of other interested parties, for that matter.) "You told me I should nominate a book by (John Scalzi/Brad Torgersen!) The Horror! It's a Conspiracy!"

Yes, they are only suggestions. But that won't stop the howling....

It seems to me that it doesn't matter if the suggestions are randomized. We not only want to avoid impropriety, but perhaps even more important we need to avoid the APPEARANCE of impropriety.

I really, really don't like this idea. In case I haven't expressed myself clearly -- PLEASE do not do this.

#694 ::: J Thomas ::: (view all by) ::: May 01, 2015, 11:21 AM:

#683 Keith "Kilo" Watt

You've made a mistake in your second round:

I did it right to that point, and I agree with your answer.

My mistake is in the third round where I put the wrong score for F. I guess the lesson here is don't do this by hand.

#695 ::: Nancy Lebovitz ::: (view all by) ::: May 01, 2015, 11:25 AM:

#685 ::: Andrew M

If the complete long list is published, there can be some crowd-sourced checking for eligibility. The crowd can check for basic publication date and mention earlier versions, though there will need to be a judgement call about whether early versions of a story are similar enough to render it ineligible.

The long list would also work as a memory prompt and a light-weight recommendation system.

#696 ::: Jameson Quinn ::: (view all by) ::: May 01, 2015, 11:27 AM:

@693: Hmmm... OK. That's really not the intention, nor do I believe it would be the result, but I understand that this does not resolve your objections, so I'll drop the idea.

#697 ::: Jameson Quinn ::: (view all by) ::: May 01, 2015, 11:38 AM:

@693 makes me sympathetic to J Thomas. I had an idea which I thought was kinda cool (I was even ready with all kinds of proofs of the cryptographic strategic-resistance of the sampling scheme!) but it got shot down for a reason I don't entirely understand. I realize there's probably no point in pursuing it, but there is definitely a part of me that wants to keep pushing... "Why don't you like this? What about this other similar idea which I just invented to try to address what I roughly guess you to be saying? What if I wrote an essay about why you're thinking about this wrongly?"...

But yeah, I gotta just let go.

#698 ::: Cassy B. ::: (view all by) ::: May 01, 2015, 11:44 AM:

Jameson Quinn @697, I'm truly sorry; I know you meant it well, and I really don't want to rain on your parade. And I'm only one Hugo voter; it's entirely possible others may react differently. Unfortunately, for me, as a garden-variety voter with no actual expertise in voting systems, this pushes my "HELL NO" button. Seriously, I know how rotten it feels when someone reacts negatively to a shiny new idea -- I'm truly sorry.

I welcome others' reactions to your proposal; perhaps I'm just a paranoid outlyer....

#699 ::: J Thomas ::: (view all by) ::: May 01, 2015, 12:40 PM:

@687 Keith "Kilo" Watt

I disagree it's chaotic. This is the LPE part of SDV-LPE in action. It sounds like you really want just pure SDV, i.e., the works with the most points "should" win.

That isn't what I want, that's what we already have.

Let me try to express it vividly. We want a vivid story to tell people, yes?

SDV-LPE is like a medieval tournament. We do pairwise jousts, and the loser is eliminated, and that happens over and over until there are only five knights standing.

The winner of each joust is the one with the most votes.

At first thought it looks like the five strongest knights will win. But the tourney schedule matters too. When you vote, if you split your vote among different knights, it has its full power for each of them when they're fighting. But also, your vote helps delay their jousts.

It's kind of like the weakest knight fights the second-weakest knight, and then the third-weakest and so on until he loses. But not quite! When you split your vote, it's only a fraction of the vote that keeps your knight off the lists. If everybody who voted for him split their vote five ways, he's likely to fight knights that are 1/5 his strength, and he'll cream them.

But every time he beats a knight that was on some of his ballots, he will share those ballots with fewer knights and his chance to miss the next tourney goes up. Once he clobbers enough of the knights on his ballots then he can take a rest.

So think of the five strongest knights. The strongest one, Sir Absolute, can clobber anybody. His voters shared the vote with lots of people, so he's down there among the also-rans, beating anybody he comes up against.

And the second-strongest knight, Sir Brawny, can clobber anybody but the strongest. He's down there too, clobbering people, but if he ever comes up against Sir Absolute he's had it.

The third knight, Sir Claymore, is clobbering everybody but Absolute and Brawny. And so on.

IF -- through the luck of the lists -- Absolute, Brawny, Claymore, Destrier, and Exploder fight each other, then somebody else can slip in. Otherwise, no chance.

A knight still has to be pretty good to win, even if his strategy is to win by avoiding combat. If he isn't strong enough then he'll have to fight and lose. But if Sir Sneaky's voters vote only for him, he has a chance to avoid combat with knights who are nearly 5 times as strong. Maybe they'll knock each other out before they get to him. Just maybe.

And if he's more than half as strong as Sir Exploder, he has a chance that Sir Exploder will still be mostly fighting weaker knights when Sir Aweful eliminates him. Sir Sneaky might squeak by, if the luck of the lists never forces him to fight anybody stronger than he is.

And he doesn't *have* to have voters who vote only for him. He could luck out and have voters who load up his ballets with such weaklings that they all get eliminated before he's in danger. He gets all the advantages of voters who really really want him, without actually having such voters.

When you get right down to it, it's the luck of the lists. We can expect that usually the strongest will still win, the way they used to. But every now and then Sir Sneaky has his day.

Also, what if Sir Awful, Sir Brute, Sir Crazy, Sir Dreadful, and Sir Excrement all have more votes than anybody else, but they share all the same ballots? They'll be down there fighting the weaklings, getting plenty of opportunities to possibly fight each other. Probably two or three -- or even four-- of them will drop.

Nobody can be sure how the tournament will go. Nobody knows who each knight will be matched up against, or (unless he's weak) whether he will have to fight at all. It isn't really chaos, but it looks like it to anybody who watches.

#700 ::: Fade Manley ::: (view all by) ::: May 01, 2015, 12:53 PM:

I'm a bit sheepish to mostly be showing up here to go "I don't like that!", but I'm going to agree that I have an intense negative reaction to the automated suggestions of other things to nominate.

It comes across as the Hugos committee trying to push things on me. "You should be looking at these items! We've decided they're appropriate for you." Amazon does that because it's trying to sell me things, but the Hugos aren't a giant corporation attempting to convince me to spend money. The Hugos are supposed to be soliciting information from me.

And the Amazon recommendations are wildly gameable. There are forums dedicated to discussing how to best game them to get specific books pushed towards specific customers. I really don't think that's the connotations you want here.

#701 ::: Kevin Standlee ::: (view all by) ::: May 01, 2015, 12:53 PM:

Lydy Nickerson @662:

In terms of administrative burden: as far as I know, and I'm sure Kevin will correct me if I'm wrong, there isn't a mandated time period for nominations and voting. The convention opens up nominations when they've got their software and hardware and people-points ready, and close them at the point that they need to print ballots, which will be dependent upon a whole bunch of factors which will change from year to year.
That is correct. There is no hard-coded set of dates other than the January 31 deadline (you have to be a member of one of the three eligible Worldcons by that date in order to nominate). The practical deadlines, of course, involve being able to determine the results in time to engrave the trophy plaques and working backwards from there.

If you were prepared to put off engraving until after the convention -- risky due to post-Worldcon ennui -- you can squeeze more time out of your schedule and present the winners with blank trophies.

Naturally, you want to leave enough time for the members of the current Worldcon to read all of the finalists.

Because you have to work all of your deadlines backwards from the Worldcon itself, it's generally considered impractical to hold a Worldcon earlier in the year than the beginning of July (the US Independence Day/Canada Day weekend).

felice @668:

I still think you are significantly minimizing how much human intervention is required to parse freeform text input, and even then, it's not perfect. Remember that Paul Cornell was cheated out of a spot on the shortlist because the 2009 Administrator didn't realize that three different variations on the same name were actually the same work.

Keith "Kilo" Watt @670:

As I recall, we've had as many as eight finalists in a category once.

Andrew M @685:

By the way, one problem with the longlist is that (I take it) there's no guarantee all the works on it will be eligible.
That's why I thought the three-stage "semi-finalist" round, even though it added administrative overhead and required sending out an intermediate ballot with Yes/No on each semi-finalist, worked better. The semi-finalists wouldn't necessarily be eligibility-checked nor would they have been asked whether they accepted a finalist spot yet. While the semi-final voting went on, the Admin would be contacting semi-finalists and checking eligibility, and the public would also be checking eligibility and calling out items that weren't eligible. But the current Admin rejected the idea as too much work.

Bradley W. Schenck @692:

It's my understanding that nothing in the rules prevents the longlist from being displayed, so a rules change isn't necessary.
Technically, yes. Practically, it's so out of the ordinary process that at the very least I think you would need a WSFS Business Meeting to pass a resolution asking the following year's Worldcon to do so and finding affirmatively that it's not prohibited. And the argument would be complex and difficult. I don't know how I'd vote if it came down to me using my casting vote.

What I think is missing is the complete, anonymized ballots.... I have a feeling that those anonymized ballots may not exist....
I think we should all proceed from the assumption that such information does not exist. While WSFS doesn't have the formal practice of ordering the Hugo ballots destroyed the way we do site selection, I think from a functional point of view we should assume that once the results of the election are official (that being when the winners are announced), the underlying data is gone and cannot be touched again.

This is another area where the Business Meeting would have to pass a resolution asking for anonymized data and finding that it doesn't violate existing rules. And I don't think that the Administrator is required to hand it over even if we ask nicely.

#702 ::: Kevin Standlee ::: (view all by) ::: May 01, 2015, 12:56 PM:

Lydy Nickerson @662:

In terms of administrative burden: as far as I know, and I'm sure Kevin will correct me if I'm wrong, there isn't a mandated time period for nominations and voting. The convention opens up nominations when they've got their software and hardware and people-points ready, and close them at the point that they need to print ballots, which will be dependent upon a whole bunch of factors which will change from year to year.
That is correct. There is no hard-coded set of dates other than the January 31 deadline (you have to be a member of one of the three eligible Worldcons by that date in order to nominate). The practical deadlines, of course, involve being able to determine the results in time to engrave the trophy plaques and working backwards from there.

If you were prepared to put off engraving until after the convention -- risky due to post-Worldcon ennui -- you can squeeze more time out of your schedule and present the winners with blank trophies.

Naturally, you want to leave enough time for the members of the current Worldcon to read all of the finalists.

Because you have to work all of your deadlines backwards from the Worldcon itself, it's generally considered impractical to hold a Worldcon earlier in the year than the beginning of July (the US Independence Day/Canada Day weekend).

felice @668:

I still think you are significantly minimizing how much human intervention is required to parse freeform text input, and even then, it's not perfect. Remember that Paul Cornell was cheated out of a spot on the shortlist because the 2009 Administrator didn't realize that three different variations on the same name were actually the same work.

Keith "Kilo" Watt @670:

As I recall, we've had as many as eight finalists in a category once.

Andrew M @685:

By the way, one problem with the longlist is that (I take it) there's no guarantee all the works on it will be eligible.
That's why I thought the three-stage "semi-finalist" round, even though it added administrative overhead and required sending out an intermediate ballot with Yes/No on each semi-finalist, worked better. The semi-finalists wouldn't necessarily be eligibility-checked nor would they have been asked whether they accepted a finalist spot yet. While the semi-final voting went on, the Admin would be contacting semi-finalists and checking eligibility, and the public would also be checking eligibility and calling out items that weren't eligible. But the current Admin rejected the idea as too much work.

Bradley W. Schenck @692:

It's my understanding that nothing in the rules prevents the longlist from being displayed, so a rules change isn't necessary.
Technically, yes. Practically, it's so out of the ordinary process that at the very least I think you would need a WSFS Business Meeting to pass a resolution asking the following year's Worldcon to do so and finding affirmatively that it's not prohibited. And the argument would be complex and difficult. I don't know how I'd vote if it came down to me using my casting vote.

What I think is missing is the complete, anonymized ballots.... I have a feeling that those anonymized ballots may not exist....
I think we should all proceed from the assumption that such information does not exist. While WSFS doesn't have the formal practice of ordering the Hugo ballots destroyed the way we do site selection, I think from a functional point of view we should assume that once the results of the election are official (that being when the winners are announced), the underlying data is gone and cannot be touched again.

This is another area where the Business Meeting would have to pass a resolution asking for anonymized data and finding that it doesn't violate existing rules. And I don't think that the Administrator is required to hand it over even if we ask nicely.

#703 ::: Kevin Standlee ::: (view all by) ::: May 01, 2015, 12:58 PM:

Argh! Someone delete 702 and this post, please. I got "Internal Server Error" the first time I tried to post it, and I guess it double-posted when I tried again.

#704 ::: J Thomas ::: (view all by) ::: May 01, 2015, 01:14 PM:

@697 Jameson Quinn

@693 makes me sympathetic to J Thomas. I had an idea which I thought was kinda cool (I was even ready with all kinds of proofs of the cryptographic strategic-resistance of the sampling scheme!) but it got shot down for a reason I don't entirely understand. I realize there's probably no point in pursuing it, but there is definitely a part of me that wants to keep pushing

We've had several approaches toward helping the membership coordinate, to find works that they tend to agree on, works that can compete well against slates.

Each new idea gets shot down very firmly.

I am pretty sure by this point that anything which helps voters tell which works are more likely to win, will be rejected. Even your extremely clever idea of giving individual voters a small sample of works, proportional to the votes so far. It would statistically give a push in the direction you and I think would help the problem, and I predict the others will respond like Cassy did.

What looks to you and me like a solution to the problem we are trying to solve, to them looks like more of the problem.

I'm pretty sure they will reject anything that results in a Compromise Strategy -- where you vote for something you otherwise would not have, because you think it can win. If we come up with a way to do that which doesn't look like it, they will reject it as soon as they see what's going on.

I am convinced this is not just tribal. "This is our tribal ritual and we do it our way." It is not just an overreaction to SPs. "They are wrong so we declare the opposite of them is right." There's an actual philosophy here, a way of thinking that fits together. I only get part of it.

#705 ::: Bradley W. Schenck ::: (view all by) ::: May 01, 2015, 01:26 PM:

#702 ::: Kevin Standlee This is another area where the Business Meeting would have to pass a resolution asking for anonymized data and finding that it doesn't violate existing rules. And I don't think that the Administrator is required to hand it over even if we ask nicely.

Is that a fairly simple rules change that could be considered? If the nominating ballots were anonymized and preserved, there could be a formal challenge for an audit of the data. Then if anyone suspected a Big Hidden Conspiracy they'd have a valid way to prove their suspicions instead of doing what's now been done.

#706 ::: Tom Whitmore ::: (view all by) ::: May 01, 2015, 01:44 PM:

Bradley W. Schenck @705: if the conspiracy's that big, they can easily fake the data. Or so the conspiracy theorists could claim.

#707 ::: Kevin Standlee ::: (view all by) ::: May 01, 2015, 01:52 PM:

Bradley W. Schenck @705:

Is [requiring the Administrator to release anonymized voting data] a fairly simple rules change that could be considered?
Maybe. I haven't considered how to write it up. Write to me (and the rest of the WSFS Business Meeting staff) at wsfs-business@sasquan.org if you are serious about wanting to propose it (assuming you have at least two Sasquan attending/supporting members to sponsor it) and we'll work on drafting it.

#708 ::: Duncan J Macdonald ::: (view all by) ::: May 01, 2015, 02:04 PM:

JT et alia @704
I'm pretty sure they will reject anything that results in a Compromise Strategy -- where you vote for something you otherwise would not have, because you think it can win. If we come up with a way to do that which doesn't look like it, they will reject it as soon as they see what's going on.

If I may toss my tuppence into the fray, the nomination balloting under discussion is one that deeply seated in what each individual believes is worthy of consideration. Any attempt to provide a(an) alternate or additional suggestion(s) that one might nominate in addition to one's own choice(s) goes against that individuality.

It really does boil down to that individuality -- I'd rather nominate nothing than something someone else (slate, SMOF, SP/RP, Cabal(TINC), etc) suggests/orders me to nominate -- unless I've read said work(s) and agree.

YMMV.

#709 ::: Bradley W. Schenck ::: (view all by) ::: May 01, 2015, 02:47 PM:

#707 ::: Kevin Standlee
Write to me (and the rest of the WSFS Business Meeting staff) at wsfs-business@sasquan.org if you are serious about wanting to propose it (assuming you have at least two Sasquan attending/supporting members to sponsor it)

Unfortunately I won't be there this year, though I have hopes for 2016. I did realize that it gets complicated quickly: you need an independent auditor, and all that data has be be accessible (somehow) even though the con's run by different committees each year. But I think this is similar to what large sweepstakes and lotteries do. It might be appropriate here, too.
I'm probably not even the guy to write up a usable proposal (I'm an artist, not an accountant!) but I'll think about it a bit more, and we'll see.

@#706 ::: Tom Whitmore

You can't change an extremist's mind. All you can do is show that the process is fundamentally fair, so that those who are not zealots may decide that you're right, and it is.

#710 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 03:02 PM:

JT@699:

As I understand it, that is -not- how SDV-LPE works. We don't do pairwise jousts, as that implies that each work is compared against all other works individually. They are not. Also, the tourney schedule does -not- matter, at least in any way that I can see it. Getting defeated early doesn't matter. It's not like a NCAA bracket where you can get knocked out early just because you happen to get put against a stronger team early.

To look at just one of your statements, if a ballot has a strong candidate and then four weaklings, then the result is essentially the same as if the ballot contained only the strong candidate. This is because the entire election is run -from scratch- each round with the only difference being that some works have vanished as though they were never nominated.

Unless I'm not seeing all the permutations (I fully recognize that could be that case -- I'm -not- an expert, but I'm pretty familiar with it since I coded it), this situation can't possibly matter in the slightest. In the vast, vast majority of cases, if the strong candidate is truly strong and the weak candidates are truly weak, then the weak candidates are eliminated long before the the strong one even gets close to being considered for elimination. In that case, that voter's ballot has only the strong candidate getting a full point -- exactly as if he had not nominated the weaklings at all. (As a side note here, I see this as a bonus: People shouldn't be afraid to nominate -anything- for the Hugos -- if it has no chance of winning, then it will be as if they didn't nominate them at all; the strength of their remaining candidates is unchanged from what they would have been.)

Suppose, however, that the "strong" candidate isn't really strong, but is razor-close to being one of the final five. The only way the situation you describe could be a factor is if somehow he was up for elimination before the four weaklings, because then he's only getting a fifth of a vote instead of a full vote. But if the weaklings haven't been eliminated yet, then he's -not- much stronger -- he's not really different in strength from the other works on the ballot, so your example is invalid to being with.

Another example, you say it's "the luck of lists" that A,B,C,D, and E don't fight each other. That's completely wrong. There's no luck involved. These five are, if I understand you right, the candidates that are likely to have the most support. The big difference is that you seem want the candidates to "fight to win", but that's SDV, not SDV-LPE. SDV-LPE simply eliminates unpopular candidates until there are only five left. It doesn't care who among those are the most popular. It's a subtle distinction, maybe, but it's important, and it's why your analogy doesn't really work.

Points don't determine who gets eliminated -- they only determine who is -eligible- for elimination. I've noticed that when you ran round results, you never counted the number of nominations -- but this is key to the LPE part of the system.


Let me summarize SDV-LPE again (for the tl;dr folks, start here ;-) ):

1. Look at all the points, and select (ideally) the two lowest point totals. No other work is competing or even looked at.

2. Of these works selected as candidates for elimination, now ignore points entirely. Count up nominations received. Eliminate the work with the least number of nominations.


A good deal of your example is kind of either wrong or misleading, I think, and that is leading to some hypotheticals that don't entirely make sense. That's good, because it shows we need to do a better job of explaining SDV-LPE, though.

I recognize that SDV-LPE is not the system you want. You propose what I think is an interesting, innovating, and most importantly completely new system. It may have strengths. As I said about the initial "real-time results" idea, I think it could be fun to watch. But it is a fundamental change to the Hugo process, and in general, the Worldcon electorate wants to keep the process as absolutely close to the original Hugos as possible. All of your proposals have moved away from that, and in fairness to you, keeping as close to current Hugo system as possible was not one of the criteria listed in the original post. But I think it's clear that it -is- a criteria and that may be why you're running into the resistance you are.

Kilo

#711 ::: Steve Halter ::: (view all by) ::: May 01, 2015, 03:20 PM:

J Thomas@641:Keep in mind that there are quite a few people watching as proposals evolve and only posting as ideas become concrete. Also, remember that there is an even (usually vaster) audience who are hardly ever posting but are thinking about things.

J Thomas@704:The key that we are trying to communicate is that in general, the membership does not desire to coordinate in a strategic fashion. Strategic coordination is the antithesis of what I think we want to accomplish. What I think is desired is a way of deterring strategic voting that is not itself strategic voting (including slates). The SDV-LPE voting system fulfills this requirement while the ranked long list does not.

#712 ::: Tom Whitmore ::: (view all by) ::: May 01, 2015, 03:31 PM:

Bradley W. Schenck @709: Let us remember that nobody was calling for this kind of accounting until extremists started talking about cliques and perceived irregularities. We're approaching the territory of Sturgeon's story "Mr. Costello, Hero" here.

Now, there are some interesting analyses that could be done if someone wanted to save that sort of information: but it has nothing to do with the actual awarding of appropriate Hugos, and I'm not all that fond of analysis for analysis's sake (and I worked in data analysis for several years -- this is not just Luddite fear of Big Data).

#713 ::: P J Evans ::: (view all by) ::: May 01, 2015, 03:50 PM:

712
Keeping anonymized nominations for a year: that I can see. The final ballots, not so much. (Even with pack-rat genes, I really can't see a point to keeping them. It mattered more before the internet, I suspect, when it might have come in handy for the next Worldcon in your neighborhood.)

#714 ::: J Thomas ::: (view all by) ::: May 01, 2015, 03:51 PM:

@710 Keith "Kilo" Watt

As I understand it, that is -not- how SDV-LPE works. We don't do pairwise jousts, as that implies that each work is compared against all other works individually. They are not.

Each work is not compared against *all* other works individually.

*Most* works are compared against *some* other works individually, and each time the loser has lost. There can be at most four works that never get compared against any other, unless we have double-elimination for ties.

Also, the tourney schedule does -not- matter, at least in any way that I can see it. Getting defeated early doesn't matter. It's not like a NCAA bracket where you can get knocked out early just because you happen to get put against a stronger team early.

As I understand it, yes, you can get defeated early when you would be much less likely to be defeated later. I do not call this a bug.

To look at just one of your statements, if a ballot has a strong candidate and then four weaklings, then the result is essentially the same as if the ballot contained only the strong candidate. This is because the entire election is run -from scratch- each round with the only difference being that some works have vanished as though they were never nominated.

Yes, it can easily work that way, and I think I described an example of it working that way. But it doesn't *have* to work that way.

If a particular ballot has a strong candidate and four weaklings, and each weakling has *mostly* ballots that have only that single nominee while the strong candidate has mostly ballots that have five spots filled, the strong candidate can have a lower score than the weaklings. And it can be matched against another strong candidate which also has a lower score than the weaklings, and it can lose before them. That will raise their scores. I don't say this is a bug.

In the vast, vast majority of cases, if the strong candidate is truly strong and the weak candidates are truly weak, then the weak candidates are eliminated long before the the strong one even gets close to being considered for elimination. In that case, that voter's ballot has only the strong candidate getting a full point -- exactly as if he had not nominated the weaklings at all.

I expect that to happen more often than the reverse. But you can't depend on it. While the strong candidate is at 1/5 score, or at 1/4 score, or 1/3, or 1/2, it can still have the lowest score and can get matched against a stronger candidate and lose.

Suppose, however, that the "strong" candidate isn't really strong, but is razor-close to being one of the final five. The only way the situation you describe could be a factor is if somehow he was up for elimination before the four weaklings, because then he's only getting a fifth of a vote instead of a full vote. But if the weaklings haven't been eliminated yet, then he's -not- much stronger -- he's not really different in strength from the other works on the ballot, so your example is invalid to being with.

If he is the 6th strongest, he desperately needs to have at least the 4th highest score. If two of the others fight each other, he wins. Otherwise he inevitably loses. So yes, he does better if his weaker companions have already been eliminated -- which will be the case unless they have higher scores than he does. And he needs two of the stronger works to have lower scores, which can only be true if they still have a number of multiple ballots.

Another example, you say it's "the luck of lists" that A,B,C,D, and E don't fight each other. That's completely wrong. There's no luck involved. These five are, if I understand you right, the candidates that are likely to have the most support.

Some of them can have the lowest scores even though they are the strongest. If two of them have the low scores, one will be eliminated. That's how F has a chance. F is less popular by number of votes, but F can be more popular by score.

I recognize that SDV-LPE is not the system you want. You propose what I think is an interesting, innovating, and most importantly completely new system.

I'm clear that my alternative is not viable. It is too different. I have put it aside.

I'm working on SDV-LPE both to get clarity on what it does, hoping to explain it clearly, and to attempt to show that strategic voting has strictly limited effect. We might still find some better approach, but we probably won't unless SDV-LPE turns out to be inadequate. I am not trying to make it inadequate, if it's good enough I'll go with it.

I believe my description as a single-elimination tournament is basicly correct. I didn't describe completely how the scores are counted, and how the scores determine who is next to fight. And I didn't describe ties at all. I think it's a good second draft. About the things you and I appear to disagree on, I think we agree about the most common cases. You talk like things I say are possible are not possible. Do you want examples? I'd prefer to wait until tomorrow since it takes some time and you might see some of it my way before then.

#715 ::: J Thomas ::: (view all by) ::: May 01, 2015, 04:21 PM:

It really does boil down to that individuality -- I'd rather nominate nothing than something someone else (slate, SMOF, SP/RP, Cabal(TINC), etc) suggests/orders me to nominate -- unless I've read said work(s) and agree.

YMMV.

I fully agree. I myself would not nominate something I hadn't read and liked.

I would like it if Hugo nominators actually agreed to the point that the top 5 each got more than 50% of the vote. If that happened, slates would have no chance.

I don't see that happening in an obvious way because when nominators vote for up to 5, the way it is now the winners get 10% to 20% of the vote. For some categories they can't get more than 3 nominees that have 5%.

If everybody nominated 5 times as many as they do now, they might likely get that level of consensus. Somewhere among the 15 to 25 you might nominate, would be a few that a whole lot of others nominated. That's too much work, and it might be too much dilution of the concept. Is the work you'd rate #25 really good enough?

If you looked at the things that everybody blindly nominated, and found some that were popular that you liked enough to nominate too, we might get a consensus that way. Maybe not. I thought it was very different to see which ones to rally behind if it was worth rallying behind something, than to join a slate.

But they could grade into each other. I looked at it idealistically, like fans would do the right thing. It could be misused, and maybe there would be an inevitable tendency to misuse it increasingly more. I don't know. I wasn't thinking that way.

So freedom means you aren't free to see early results. That isn't a snark at you, sometimes things just work out ironically.

No information released to assist strategic voting, because you want strategic voting not to happen. You want the Hugos to be free from bandwagon effects.

OK. I'm getting a somewhat better feel for it.
Thank you.

#716 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 04:23 PM:

JT@714:

Yeah, I guess we disagree on whether a single elimination tourney is a good analogy. I don't think that it is at all. I'd be very interested in seeing any odd cases you might can propose -- in fact, I think that's one of the most useful things we can do. Give me cases that might be odd, and I'll run the sim and see what happens. If it truly is odd (recall that so far none of the "odd" cases that were proposed have turned out to be odd when they were actually run), then we can discuss how likely an artificial case like that is to actually occur and whether we think we need to deal with that case or not.

To all:
I'd actually appreciate sample scenarios from anyone -- the more cases we can run, the better we can get an intuitive feel for the system. So far, I've yet to find a case that breaks the system, but we really do need to look at the low-probability stuff.

Cheradenine and I have run the 2013 Hugo data that we have (not actual ballots, but the correct nomination counts with random distributions that match them) and got exactly the 2013 final ballot. We've also tried variations (slate with 150 ballots, slate with 300 ballots, two slates, etc.) and none of them have broken the system either. I'd be happy to post those results if there is interest.

Regards,
Kilo

#717 ::: Cheradenine ::: (view all by) ::: May 01, 2015, 04:46 PM:

To be fair, it is pretty likely that the actual nominations aren't randomly distributed - but any guesses at the level of correlation present are pretty much sheer speculation. I'm still hoping we can run a sim with the 1984 ballots; in the meantime, I'd love to see any conjectures people come up with for what ballots might plausibly look like. The more data we throw at the system, the better.

#718 ::: Kevin Standlee ::: (view all by) ::: May 01, 2015, 05:07 PM:

Bradley W. Schenck @709:

Unfortunately I won't be there this year,...
You don't have to be there; you have to be a member, either supporting or attending. Any two members, supporting or attending, can submit proposals. You can't debate or vote upon them if you're not present, but you can submit them and they'll be on the agenda.

#719 ::: J Thomas ::: (view all by) ::: May 01, 2015, 05:10 PM:

Give me cases that might be odd, and I'll run the sim and see what happens. If it truly is odd (recall that so far none of the "odd" cases that were proposed have turned out to be odd when they were actually run)

My "odd cases" did what I said they did, and we agreed that was what the system was designed to do.

It will be similar with the others.

Ballots:
A 50 B 50 C 50 DG 10 DH 10 DI 10 DJ 12 EG 10 EH 10 EI 10 EJ 10 G 12 H 12 I 12 J 11

Votes:
A 50 B 50 C 50 D 42 E 40 G 32 H 32 I 32 J 33

1st Score:
A 50 B 50 C 50 D 21 E 20 G 22 H 22 I 22 J 22

Anomaly: G H I J are weaker than E, but E is eliminated on the first round because E shares votes with the weaker works. Eventually J wins.

Ballots:
A 50 B 50 C 50 DG 10 DH 10 DI 10 DJ 12 EG 10 EH 10 EI 10 EJ 10 G 12 H 12 I 11 J 11

Votes:
A 50 B 50 C 50 D 42 E 40 G 32 H 32 I 31 J 33

1st Score:

A 50 B 50 C 50 D 21 E 20 G 22 H 22 I 21 J 22

Anomaly: One vote for I is different. The result is on the first ballot I loses, then so do G H and J. E wins.

E can lose on the first ballot or win, depending on one vote for a weaker competitor. This is not a bug, this is how the system is designed to work.

We can make much more dramatic cases with 4 or 5 votes in the ballots, but they take a lot more typing. I don't say anything is wrong with this happening. I say that my tournament description of the process is basicly correct.

#720 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 05:39 PM:

JT@719:

Looking at your first scenario, you're characterization is incorrect. You say:

Anomaly: G H I J are weaker than E, but E is eliminated on the first round because E shares votes with the weaker works. Eventually J wins.

No, E is not stronger. Again, you are only looking at number of nominations but ignoring the second half of the system. E has very nearly the same number of nominations as GHIJ. But a significant number of voters felt so strongly about GHIJ that they nominated them exclusively. No voter felt that way about E. From this perspective then, E is less popular than GHIJ. If E got significantly more nominations (in any form), then it could possibly have been popular enough to win over J (the final results are ABCDJ, btw), but this didn't happen. It was a near tie, yet no one -really- liked E above all other works.

This is not an anomaly, yet you characterize it as such. You have a different definition of "weaker", and it shows, but your definition fundamentally wants to define "popular" as "received the most number of nominations". That definition is fine for SDV and a number of other systems, but it is explicitly not the definition of "popular" in SDV-LPE. The definition you are subtly putting forward as a "strong" or "popular" work is the definition that allows a slate to sweep the ballot: Their works got the most nominations, so their works are the "strongest". Until we agree on the meaning of popular, I'm not sure how far we're going to get.

Even so, running these scenarios is worthwhile. I'll run the others here soon.

Kilo

#721 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 05:41 PM:

And I cancelled the post when I noticed "you're" instead of "your". Honest, I did.

:-)

Kilo

#722 ::: felice ::: (view all by) ::: May 01, 2015, 05:56 PM:

felice @679: "Am I doing anything wrong here?"

Duh. I forgot the LPE part. Ok, H and I are irrelevant, it's the tie between F & G that needs to be broken. One less vote for either, and the other shoots to the top of the ballot, giving us FABCD or GABCD as winners, a result that represents the nominations of two more voters than ABCDE does.

How about this example? After less popular works are eliminated, L and M are the only works left with significant overlap. They're not a slate; more people voted for each one alone than voted for both; but at 69 points and 89 ballots, they're tied for elimination well before the fifth place contest. With simultaneous elimination, the winners are ABCDE; with any kind of tie-break, it becomes LABCD or MABCD.

A 80
B 79
C 78
D 77
E 76
F 75
G 74
H 73
I 72
J 71
K 70
LM 40
L 49
M 49

And if you add a few more ties:

NO 40
N 48
O 48
PQ 40
P 47
Q 47
RS 40
R 46
S 46
TU 40
T 45
U 45

Tie-breaking this would give a final ballot of LNPRT, MOQSU, or some mix thereof. Am I doing something incredibly stupid this time?

#723 ::: J Thomas ::: (view all by) ::: May 01, 2015, 06:24 PM:

720 ::: Keith "Kilo" Watt

your definition fundamentally wants to define "popular" as "received the most number of nominations". That definition is fine for SDV and a number of other systems, but it is explicitly not the definition of "popular" in SDV-LPE.

OK, how do you state that definition?

But a significant number of voters felt so strongly about GHIJ that they nominated them exclusively. No voter felt that way about E. From this perspective then, E is less popular than GHIJ.

We have been talking some about strategic voting and how we don't want it. One of the things we don't want is "bullet voting" where you vote just for the one you like most and not any of the others, so it will have the best chance.

Now you are saying that bullet-voting is the sign that you cared about this work, and if you don't bullet-vote for it then of course it's only proper that it should fail.

In the vast, vast majority of cases, if the strong candidate is truly strong and the weak candidates are truly weak, then the weak candidates are eliminated long before the the strong one even gets close to being considered for elimination. In that case, that voter's ballot has only the strong candidate getting a full point -- exactly as if he had not nominated the weaklings at all.

Of course people don't really know how strong their candidates are. They have some sense of how popular they are, particularly if they're in say the top 5. But guessing how many people will bullet-vote for them?

If you are sure that some extra candidates are so weak that they will definitely be eliminated before your main choice ever gets to the lowest score, then it's safe to nominate them.

But if they might be somewhat stronger than you think, then they aren't safe after all. If they get enough votes (particularly bullet votes) to survive until after the first time your choice does a joust, he might lose right then. And they will probably be weak enough to also lose soon after.

#724 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 06:40 PM:

I've got to take my daughter to gymnastics, so I'll have to revisit this tonight, but I did want to say that I don't have a problem with bullet voting, nor have I seen a good explanation of why it is a problem. I did run a sim with the 2013 Hugo data with bullet voting and it did put the bullet-voted work on the ballot. I'm not sure why that's a bad thing.

Regarding popularity definitions, I posted my definition in #687:

Popularity has two aspects: How strongly the voters felt about a work they nominated (fewer works nominated means they value those works they do nominate more -- so they get more points) and how many voters considered the the work worthy at all (and so get more votes for nomination).

Kilo

#725 ::: felice ::: (view all by) ::: May 01, 2015, 07:31 PM:

Keith "Kilo" Watt @724 "I don't have a problem with bullet voting, nor have I seen a good explanation of why it is a problem"

More bullet votes means fewer total nominations, meaning (probably) fewer nominations for the most popular works, meaning a smaller target for slates to beat. And SDV-LPE only makes a slate sweep of the ballot harder, not impossible.

"Popularity has two aspects: How strongly the voters felt about a work they nominated (fewer works nominated means they value those works they do nominate more -- so they get more points"

SDV-LPE doesn't exactly conform to that; how many points a work has will change as other works are eliminated, and part way through the process a work getting more points from a particular voter doesn't necessarily mean they valued it more. What we're trying to do is balance pure popularity (measured in number of nominations) against representation (including works nominated by as many members of voting population as possible). We don't have any reliable measure of how strongly voters feel; someone could reluctantly bullet the only work they thought was semi-decent from what they've read , while someone else is wildly enthusiastic about everything on their ballot and agonising over not being able to list a sixth brilliant work (and not necessarily because they're easier to please - they might just have read a lot more and been more fortunate in their choices).

#726 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 11:04 PM:

felice@725:
More bullet votes means fewer total nominations, meaning (probably) fewer nominations for the most popular works, meaning a smaller target for slates to beat.

I may be misunderstanding, but I'm not sure the second follows from the first. If more bullet votes means fewer nominations (I assume you mean fewer different works nominated), then by definition won't the works that are nominated tend to be the more popular ones? And if so, then each of those ballots counted for a full point, which means they are very likely to beat any slate of five works (which only get 1/5 of a point per nomination).

I also can see that a slate could say "Hey, just vote for this one work so that we can maximize it's chances of getting on the ballot!" But is a slate of one work really a slate? I'm not interested in -- and in fact, I am rigidly opposed to -- keeping Puppy works off of the ballot. I just don't want them to keep -other- works off.

And SDV-LPE only makes a slate sweep of the ballot harder, not impossible.

This is very true. But it doesn't just make it harder, it makes it astronomically harder. The slate needs five times as many people voting. At that point, they have such a huge chunk of the electorate, that -not- giving them their choices wouldn't be fair.

SDV-LPE doesn't exactly conform to that; how many points a work has will change as other works are eliminated, and part way through the process a work getting more points from a particular voter doesn't necessarily mean they valued it more.

I agree with this. But the popularity is based on the original ballot, not on what happens once it's in the system. So the voter -has- expressed an opinion based on how many works are listed (and whether a given work is listed at all). It's very true that we can't know why he or she listed what was listed, but no system can determine that, so it's hard to argue one way or the other about that.

Kilo

#727 ::: felice ::: (view all by) ::: May 01, 2015, 11:20 PM:

Should we start a new thread for discussing longlist proposals, while leaving this one open for refining SDV-LPE? It seems pretty safe to assume that the Hugo rules discussion will continue beyond the comment limit of this one.

#728 ::: Keith "Kilo" Watt ::: (view all by) ::: May 01, 2015, 11:39 PM:

felice@722:

I ran the first situation you posted and did in fact get ABCDE using "eliminate all ties". I don't have any way to simulate exactly another tie-breaking system (and I'm not really motivated to code anything until the community decides what they want), so I simulated a tie break by just reducing the number of single-nomination ballots that M got. As you predicted, the winners were ABCDL.

But I'm not sure this is as significant as you think. In both your example and JT's 719 example, all of the works that are affected are -very- close together. I'm not sure it's accurate to say that a work "shoots up" from a low place to a winning position when the difference, as you mentioned, is just a few votes either way. I submit that -any- time you have a close race, any small change can alter the outcome among the competitors that were close. I don't really see this as an odd or unexpected result.

In all the cases when I've given a more realistic distribution to the ballots, this type of situation has never come up. I'm not sure it -can- come up, but I can't do a proof of that, obviously.

That said, as I mentioned previously, I'm happy with whatever kind of tie-handling system the community wants to do, so long as it's simple. I don't see a lot of advantages over just eliminating the tied works, but that doesn't mean it can't be done. I'm not opposed to it, if that's what everyone wants to do, but they shouldn't misinterpret cases in which -all- the works are very close to being tied.

Kilo

#729 ::: felice ::: (view all by) ::: May 01, 2015, 11:56 PM:

Keith "Kilo" Watt @726: "If more bullet votes means fewer nominations (I assume you mean fewer different works nominated), then by definition won't the works that are nominated tend to be the more popular ones? And if so, then each of those ballots counted for a full point, which means they are very likely to beat any slate of five works (which only get 1/5 of a point per nomination)."

No, I mean fewer nominations in total. A thousand people bullet voting one work each is 1,000 nominations, while a thousand people nominating a full ballot of five is 5,000 nominations. And you can't assume bullet voters all prefer the most popular works; worst case scenario, they all champion a unique first choice, allowing a bloc of six to sweep the ballot. In practice, yes, nominations will be concentrated amongst the more popular works, but it's likely everything will get a lot fewer nominations. Eg without bullet voting, the most popular work might be first choice of 100 people, and second through fifth choice of another 400. Bullet voting cuts its nominations by 80%. That isn't good.

"I'm not interested in -- and in fact, I am rigidly opposed to -- keeping Puppy works off of the ballot. I just don't want them to keep -other- works off."

The only fair way to keep Puppy nominations off entirely is to have more nominations for other works.

"This is very true. But it doesn't just make it harder, it makes it astronomically harder. The slate needs five times as many people voting. At that point, they have such a huge chunk of the electorate, that -not- giving them their choices wouldn't be fair."

As I pointed out in #675, in 2013 it would only have taken a bloc of 153 slate voters to take four of the short story slots, even with SDV-LPE. (Assuming tiebreaking eliminates one slate member; otherwise you've got a five-way tie for 4th place, since 1/5 of the slate vote is sufficient to beat "No Place Like Home" with 30 nominations. Or a bloc of 157 could stagger their nominations to avoid ties.) This isn't just not astronomically hard, it's downright probable. And a slate getting four of the spots on the final ballot is virtually as bad as five; a victory for the lone non-slate finalist would be pretty hollow.

#730 ::: Keith "Kilo" Watt ::: (view all by) ::: May 02, 2015, 12:23 AM:

felice@729:
A thousand people bullet voting one work each is 1,000 nominations, while a thousand people nominating a full ballot of five is 5,000 nominations.

This still doesn't quite follow. It's not clear what you're using for your basis of comparison. 5000 nominations is not 5000 unique nominations, obviously (and I don't -think- that's what you're saying). Are you saying the overlap in those 5000 nominations is better able to counter the inherent overlap (lockstep, actually) of the slate? Well, maybe, but that's irrelevant under SDV-LPE. Remember, we are no longer counting just raw number of nominations. A slate by definition has less nominating power than a non-slate under SDV-LPE.

Let's consider a 200 ballot slate. Under SDV-LPE the slate has 40 points for each work. With 1000 bullet voters, what are the odds that they will have an overlap of 40 nominations among them? I think the odds are pretty good. If so, then they have beaten at least one work for the slate. For each overlap of 40 nominations, they beat another slate work. It seems to me that bullet voting actually makes it -easier- to defeat a slate, not harder. And I really think that's as it should be. If the voters feel that one work really deserves the nomination, it will get it.

So, from my point of view, then, bullet voting is an okay thing, not bad. That's why I like the nominator instructions for SDV-LPE: If you honestly like five nominations about equally (there's no need to like them -precisely- equally because the extreme cases aren't going to happen), then nominate five. But if you feel very strongly about a smaller number of works, then you can express that feeling by only nominating those works -- they will likely do better.

Is this strategy? Well, sort of. But isn't it the kind of strategy we want? Vote for as many things as you -really- like?

Again, it is entirely possible I'm missing something here, because it seems my conclusions are the exact opposite of what Jameson was saying, so I hope one of the experts will jump in and explain it to me.

Kilo

#731 ::: Greg Q ::: (view all by) ::: May 02, 2015, 12:31 AM:

"::: Bruce Schneier ::: (view all by) ::: April 07, 2015, 04:22 PM:

On identifying bloc voting. "

This sounds like a great idea. Let's do it for the last 10 years, both for the nominations, and the awards. Let's get all the ballots released, without any identifying information. It would be a fascinating study, no?

The problem with SAV / PAV / RAV is that they utterly ignore the issues of preference. I nominated 5 novels, unranked. But I most certainly did have a ranking for them. If my #5 novel gets a nomination, and that causes my #1 novel NOT to get nominated, then the voting system sucks. If you're going to do something like that, i propose the following (flawed) system:

1: 5 Nominees, everyone gets 5 ordered votes.

2: Work A with most #1 votes got X votes. Work B with second most #1 votes got Y votes. Let
Z = (X - Y) / X
Work A is removed from everyone's ballot, and moved into the "Nominees" pile. One every ballot that had an "A", the lesser ranked nominations get moved up.

3: Everyone who had A in the #1 position gets their "vote strength" set to Z. E.G. if A had twice as many votes as B, each of the A voters' top nominations is now worth 1/2 a vote

4: Repeat Step 2, finding A', B', X', Y' and Z' given the new ballots. Set Z = Max (Z, Z'). Update the ballots with A' removed.

5: Everyone who had A or A' as their top choice has their "vote strength" set to Z. For those who had both A and A' as their top choices, can do one of four things:
A: Treat same as single winners, set vote strength to Z
B: Set vote strength to Z^N where N = Number of nominees in the "winner" pile
C: Set vote strength to Z/N
D: Set vote strength to Max (Z/N, 1 - ((1 - Z) * N)

6: Loop back to #4, repeating until you have 5 nominees.


Spreads the love around, honors voters preferences, hurts block voting, rewards broad-based popularity, treats all winning nominees, and their voters, equally.

#732 ::: P J Evans ::: (view all by) ::: May 02, 2015, 12:44 AM:

731
If you have a preference for your nominations, put in the the ones you think are best. (And remember that YMMV.)

#733 ::: felice ::: (view all by) ::: May 02, 2015, 01:14 AM:

Keith "Kilo" Watt @730: "Are you saying the overlap in those 5000 nominations is better able to counter the inherent overlap (lockstep, actually) of the slate?

Yes, that's what I'm saying. In my example in #729, a work could start at 100 points with either bullet voting or full ballot, but in the latter case, it might have 500 nominations for purposes of elimination comparison (ie it beats a 200 person slate work easily), and its points go up as less popular works get eliminated; but with bullet voting, if it ever faces a slate work, it gets eliminated. Eg if it's the 4th placed work up against the last two remaining slate works, a three way tie of 100 points each, under bullet voting it would get eliminated leaving the slate works as 4th and 5th finalists; while without bullet voting, the 5th non-slate work would probably have enough nominations to beat even a sole remaining slate work.

"Let's consider a 200 ballot slate. Under SDV-LPE the slate has 40 points for each work. With 1000 bullet voters, what are the odds that they will have an overlap of 40 nominations among them? I think the odds are pretty good. If so, then they have beaten at least one work for the slate. For each overlap of 40 nominations, they beat another slate work."

Eliminate one slate work, and the rest go up to 50 points. Two, and the remaining three have 66.67 points. And short story 2013 only had one work with enough nominations to beat even a 40 point slate, and that's without bullet voting. "Immersion" was nominated by 107 people, most of whom may well have also nominated four other works. If a third of them would have bullet voted for it (an optimistic guess - a random distribution would give it only 20% of bullet votes, and it might have been everyone's second choice and ended up with no bullet votes), it would only have gotten 36 nominations, and the slate would have taken all five spots.

#734 ::: felice ::: (view all by) ::: May 02, 2015, 02:21 AM:

Keith "Kilo" Watt @728: "I simulated a tie break by just reducing the number of single-nomination ballots that M got. As you predicted, the winners were ABCDL. But I'm not sure this is as significant as you think. In both your example and JT's 719 example, all of the works that are affected are -very- close together. I'm not sure it's accurate to say that a work "shoots up" from a low place to a winning position when the difference, as you mentioned, is just a few votes either way."

Ok, how about this?

A 80
B 79...
...J 71
K 70
LM 139
NO 138
PQ 137
RS 136
TU 135

The ten works eliminated first are all nearly twice as popular as the final five with simultaneous elimination. And while getting five such ties is statistically incredibly unlikely, it's not impossible, and one tie isn't at all implausible. Eg it's not inconceivable that the two standout Doctor Who episodes in a year could tie with heavy overlap and both get eliminated; or even best novel, with the two most popular works heavily overlapping and five other strong works supported by disparate groups. Such ties may be rare, but they can happen, and the results of simultaneous elimination can potentially be really bad.


Cheradenine @677: "2) If two or more candidates in the group are tied for least nominations:
a) If one candidate has the fewest points, eliminate them (as in #669, option 7).
b) If multiple candidates are tied for the fewest points, look at the previous step's results and eliminate the one who had the fewest points in that step. If there is still a tie, look back to the step before that, and so on. (This is referred to as "Backwards Tie-Breaking" in the STV literature.)"

That sounds good. Step c can't be eliminate them all, though.

#735 ::: Greg Q ::: (view all by) ::: May 02, 2015, 03:05 AM:

So, applying my method to the test data from Post 1, and assuming the rankings are in order:
#1: ABC
#2: ABC
#3: ABC
#4: ABC
#5: ABF
#6: AC
#7: EA
#8: BD
#9: DFG
#10: DG
#11: DIJ
#12: EH
#13: EI
#14: EJ
#15: FGH
#16: H

Round 1:
A: 6, E: 4. A wins, Z = 1/3

Round 2:
E: 4, B: 8/3. E wins, Z' = 1/3

Round 3:
D: 3, B: 8/3. D wins

So, same as many other results, but people aren't put in the position of having to strategically leave off books they like, because they don't want to risk having a their 5th choice book causing their 1st choice book not getting nominated.

Personally, I'll strongly oppose any *AV voting proposal that doesn't include ranking.

#736 ::: Greg Q ::: (view all by) ::: May 02, 2015, 03:28 AM:

Now, let's apply my method with one change to the data: flip the slate ordering:

#1: CBA
#2: CBA
#3: CBA
#4: CBA
#5: FBA
#6: CA
#7: EA
#8: BD
#9: DFG
#10: DG
#11: DIJ
#12: EH
#13: EI
#14: EJ
#15: FGH
#16: H

Round 1:
C: 5, E:4. C Wins, Z = .2

Round 2:
E: 4, D: 3. E Wins, Z' = .25 -> Z = .25

Round 3:
D: 3, B: 2 D Wins

Winners are C, D, E

Because preferences matter

732: You're really saying that you value all five of your novel choices the same?

Do we WANT to force people to engage in tactical voting? Because that's what a *AV system without ranking does. With ranking, I can submit my choices, in my preferred order, and not worry that voting for a book I like will cause the book I love the most not to get nominated.

#737 ::: felice ::: (view all by) ::: May 02, 2015, 03:36 AM:

Greg Q @736: " With ranking, I can submit my choices, in my preferred order, and not worry that voting for a book I like will cause the book I love the most not to get nominated."

Some people think making people rank their nominations would discourage participation because it requires too much thought; there's a fair bit of discussion earlier in this thread and in the previous thread.

#738 ::: J Thomas ::: (view all by) ::: May 02, 2015, 05:11 AM:

@731 Greg Q

The problem with SAV / PAV / RAV is that they utterly ignore the issues of preference. I nominated 5 novels, unranked. But I most certainly did have a ranking for them. If my #5 novel gets a nomination, and that causes my #1 novel NOT to get nominated, then the voting system sucks.

Interesting. We stopped thinking about ranked voting systems and I don't quite remember why. I think they're more complicated to analyse, to figure out the consequences.

That's worth a fresh look.

Consider #127 above.

With reweighted approval voting, using the correct weights, you can prove mathematically that the list of "winners" (nominees in this case) is the list with the largest number of voters approving of at least one winner.

Assuming he's right, does this matter? What if there's less than 50% chance that any of your choices make it onto the ballot? If you rank them, it might be the 5th one. If you don't rank them, it might be the one you least wanted. If you could balance the value of ranking them against a reduced chance that any win, how much would you give up for ranking?

#739 ::: David Harmon ::: (view all by) ::: May 02, 2015, 06:43 AM:

JT: As felice notes, mandatory ranking got ejected from the discussion a while back earlier as too much cognitive overload on the nominators.

ISTM that "sequence on the ballot" can still serve as a last-ditch tiebreaker, especially if it's not otherwise used. But we wouldn't want it to be more powerful than that, since it's not otherwise supposed to be significant.

#740 ::: J Thomas ::: (view all by) ::: May 02, 2015, 08:51 AM:

@734 felice

The ten works eliminated first are all nearly twice as popular as the final five with simultaneous elimination.

He has already explained his position on this. Since he basicly defines "popularity" as "what SDV-LPE selects", if a work is eliminated under SDV-LPE then it is less popular than anything which has not been eliminated yet.

... the results of simultaneous elimination can potentially be really bad.

They can have some really good results too. But I'm pretty much convinced that the rare bad results trump the (less rare with slates) good results. We can do without them, provided people will agree it's fair.

If two works tie on everything to the point that we eliminate the one that was nominated later, will people agree it's fair to do that?

The longlist problems surprised me. You suggested a longlist in #105. We went 500 posts before the important objections came. What if we got a new voting system approved and two years later a whole lot of fans got upset about the treatment of ties? I don't see any way to prevent that. We can't get very many people to pay attention, and at any time later when it's hard to change things a bunch of people might suddenly wake up and be outraged, no matter what got agreed to before.

#741 ::: J Thomas ::: (view all by) ::: May 02, 2015, 09:16 AM:

@739 David Harmon

As felice notes, mandatory ranking got ejected from the discussion a while back earlier as too much cognitive overload on the nominators.

Too bad. I guess we must accept that Greg Q will oppose any new voting system.

He's right about the strategic voting, most of the strategic voting schemes I've looked at involve not nominating things you don't like as much, to help out the one you like the most. (And then there's not voting for the one you think is most popular so you can give more help to the ones you think need more help.) A system which did them in order of rank would prevent the first kind. And partly the second -- if you don't want the most popular to hurt your vote for others, just rank it last.

If complexity was not an issue, I'd want to look at optional ranking. Let voters rank if they want to, and honor their wishes, and get the best of both worlds. But I'm afraid that would be too hard to explain.

What if that got grafted onto SDV-LPE? If you rank your votes, then the first item gets the whole vote until it is eliminated, and no other choice gets anything. Then when it does get eliminated, the next surviving vote gets the whole thing. If any of your choices get eliminated before they get any of your vote, then too bad, that's what you chose.

For strategic purposes, a voter would want to rank his votes in reverse order of how popular he thinks they are, so that his vote helps each one as much as it can until that nominee falls.

Or if it looks like too much trouble to think about all that, don't rank them.

It isn't that complicated, really. And not that complicated to handle once the votes are in a form a program can compute.

But I'm afraid it might still be too complicated to propose.

#742 ::: J Thomas ::: (view all by) ::: May 02, 2015, 09:44 AM:

@172 Buddha Buck

About contacting shortlist winners:

Worse than someone going "if the balloting had been rerun without Tom, who declined, I would have been nominated" is the unenviable phone call of "Um, yeah, you know how we said you were the #2 nominee? Well, the #5 guy dropped out, and re we re-ran the nomination software without him, and, um, you came in 6th..."

Is there anything wrong with calling them and saying "There's a possibility that your work will be one of the top five nominees in category X of the Hugos, and if it happened would you accept the nomination? Yes? Thank you. Please don't tell people about it, since after all it still might fall through. We'll try to let you know for sure as soon as we know, and of course it's customary for you not to tell anybody about it then until after the announcement."

Then if somebody declines, just rerun the vote and call whatever new people you need to call. Don't tell people who are now off the list that they're off the list until you actually get all the acceptances, because if another is declined they might be back on the list.


#743 ::: Keith "Kilo" Watt ::: (view all by) ::: May 02, 2015, 10:48 AM:

JT@740:

He has already explained his position on this. Since he basicly defines "popularity" as "what SDV-LPE selects", if a work is eliminated under SDV-LPE then it is less popular than anything which has not been eliminated yet.

That's an ... interesting ... way of stating that, and for the record it's not true, but I don't think I'll engage with that.


All: This is a busy weekend for us, so I won't be on much. Let me reiterate, though, that I don't really have any strong feelings about how ties are handled. Anything the community decides on is fine with me. We should keep in mind, however, that any tie-breaking system must -also- be explained at the business meeting, so we should avoid complexity for its own sake.

Kilo

#744 ::: J Thomas ::: (view all by) ::: May 02, 2015, 11:21 AM:

@743 Keith "Kilo" Watt

for the record it's not true

Popularity has two aspects: How strongly the voters felt about a work they nominated (fewer works nominated means they value those works they do nominate more -- so they get more points) and how many voters considered the the work worthy at all (and so get more votes for nomination).

It looks to me like you defined it to fit the way the voting is done. This is not a bad thing. People who agree with a voting system do that. It just means there's no point in arguing with you about that. If somebody says "Look, the ones with the most votes can sometimes lose, how strange", it isn't strange to you because you think it's the right way. I'm not sure whether we can do better, juggling the various goals and constraints. It looks pretty good to me. I want to do better, partly as Felice says:

@729

As I pointed out in #675, in 2013 it would only have taken a bloc of 153 slate voters to take four of the short story slots, even with SDV-LPE. (....) This isn't just not astronomically hard, it's downright probable. And a slate getting four of the spots on the final ballot is virtually as bad as five; a victory for the lone non-slate finalist would be pretty hollow.

I would prefer a system that gives a slate at most one nomination. So far, every approach which has done that has had flaws.

#745 ::: P J Evans ::: (view all by) ::: May 02, 2015, 11:48 AM:

Data will be arriving this week. I haven't seen what's in the digital files, but I believe anonymized nominations are in there.

#746 ::: jameson quinn ::: (view all by) ::: May 02, 2015, 05:37 PM:

@735: " Personally, I'll strongly oppose any *AV voting proposal that doesn't include ranking."

Speaking of pushing one's buttons...

This attiude, "I'd strfonglyh oppose anything but my favorite system", is asdfghyjkl. Meaning, it's exaspersatingly against the spirit of better voting systems. Furthermore, the actual argumentg is badly flawed.more intricate ballots are not the way to avoid tricky strategies. Overall, ranked systems tend to lead to lower voter satisfaction on average.

#747 ::: felice ::: (view all by) ::: May 02, 2015, 05:41 PM:

Keith "Kilo" Watt @743: "We should keep in mind, however, that any tie-breaking system must -also- be explained at the business meeting, so we should avoid complexity for its own sake."

Would something like "only one work may be eliminated each round, and any appropriate aspect of nomination data may be used to break ties at the discretion of the admins" suffice for the actual constitution? Or would the vagueness cause more fuss at the business meeting than providing a detailed description of the tie-breaking process?


J Thomas @744: "I would prefer a system that gives a slate at most one nomination. So far, every approach which has done that has had flaws."

I'm pretty confident that's not possible without changing how people choose works for nomination, for categories where there's a wide range of candidates. An organised voting bloc can work around pretty much any system that seeks to make rules prohibiting slates or punish them disproportionately; even the Fannish Inquisition would have to make decisions that could look pretty dubious. I don't think we're likely to be able to do significantly better than SDV-LPE for the counting stage.

#748 ::: Greg Q ::: (view all by) ::: May 02, 2015, 06:33 PM:

#746 "This attiude, 'I'd strfonglyh oppose anything but my favorite system',"

I don't believe that's an even remotely fair way to describe my position. I've got a favorite system, I've offered it up. If no one else likes it? Shrug.

Then there are those systems that I think are worse than the current system. A system where my last choice getting nominated -> my top choice doesn't get nominated is a non-starter for me.

Yes, I understand the claim that it makes it harder to nominate if you have to rank. I don't, however, know that I actually believe it.

First, you have to do exactly that in order to vote for the Hugos, and we routinely get, what, twice as many people voting as nominating, despite having a far larger pool available for nominating than for voting?

Second, you could just as easily make ranking optional. For example, everyone starts out with 1 (or 5, it really doesn't matter) "vote points". If you have your choices ranked, the top choice gets all your points. If you don't have them ranked, each choice gets pts / X, where X is the number of choices. You can go one from there.

Bottom line: It appears you HAVE to game a *PV voting system without ranking, if you, the voter, want to avoid bad results. And that, IMHO, makes for an inferior voting system.

#749 ::: J Thomas ::: (view all by) ::: May 02, 2015, 07:06 PM:

@747 felice

I'm pretty confident that's not possible without changing how people choose works for nomination, for categories where there's a wide range of candidates. An organised voting bloc can work around pretty much any system that seeks to make rules prohibiting slates or punish them disproportionately

You could be right. I haven't seen a proof, and I haven't seen a great example, so I don't know.

SDV-LPE works by exposing works with full ballots to early duels. The slate works tend to eliminate each other. But unless there are enough strong nonslate contenders, the slate works won't have time to eliminate very many of their fellows.

Unless you have a nonslate work whose score is more than the scores of the lowest two slate works at the beginning, the slate will kill off everything and get 5 wins.

Unless you have two nonslate works whose scores are more than the lowest two remaining slate scores after the first slate work loses, then 4 slates will win.

Unless you have three nonslate works whose scores are more than the lowest two slate scores after the second slate work loses, then 3 slates will win.

You need four whose scores are more than both slate scores when there are two slate works left, or two will remain.

Any voting system that throws out your vote after you have one win, will keep slates from getting more than one win. That is a good thing. There are various ranked voting systems that do that. As soon as the first slate work gets in, all the slate ballots that included that work are discarded. Mostly, no more slate.

We've had good reasons to avoid ranking systems, but this is one good reason to reconsider.

Also, there might be other ways to get that result.

#750 ::: J Thomas ::: (view all by) ::: May 02, 2015, 07:27 PM:

@748 Greg Q

"This attiude, 'I'd strfonglyh oppose anything but my favorite system',"

I don't believe that's an even remotely fair way to describe my position.

You did say that you would oppose anything that didn't fit into your favorite class of systems. I think his description is not completely wrong but it isn't exactly fair either.

Some voters prefer to rank their votes and others don't. Either way is bad for the ones who don't like it.

Then there are the results. I like the goal of getting winning votes for as many voters as we can. We have to balance that against other goals, but I like that to be one of them. Without ranking that's pretty simple. With ranking, how much better is it for your first choice to win than your fifth choice? I could come up with arbitrary values to score for those. It makes my thinking all complicated when I'm trying to decide how good the system is. So I prefer something that I can work with easier.

Second, you could just as easily make ranking optional.

I like that idea. But it makes it even harder to tell how well a voting system is doing at achieving our goals for it. It's likely that some ways it gives us the advantages of both systems, and other ways it gives us the flaws of both systems. It's a mess to think about. And it's one more thing to explain to the Business meeting and the fans. If strategic voters decide that one way gives them more of an advantage, they'll probably all do it that way and after awhile so will everybody else, and we just wasted our time providing both. I want to try it out because it might help solve our problems, but my hopes and doubts are pretty much balanced at this point.

#751 ::: felice ::: (view all by) ::: May 02, 2015, 07:54 PM:

J Thomas @749: "Any voting system that throws out your vote after you have one win, will keep slates from getting more than one win. That is a good thing. There are various ranked voting systems that do that. As soon as the first slate work gets in, all the slate ballots that included that work are discarded. Mostly, no more slate."

Unless the slate divides up into five groups each bullet voting one work, in which case the slate will win all the slots after the first work they beat (eg a slate of 195 gets four short story slots in 2013; possibly a significantly smaller slate, since some of the 107 ballots who nominated the most popular story would likely also have contained nominations for the second placed work which get discarded. The more widely popular the top work is, the fewer nominations left over for lower placed works. Under this system, the slate should probably concede first place and divide its efforts four ways (156 bloc voters to win all four slots in 2013). Or possibly nominating equally distributed pairs of slate works would maximise their chances, improving their chances of one win while minimising the loss to other candidates?

#752 ::: jameson quinn ::: (view all by) ::: May 02, 2015, 08:07 PM:

@748: you're right I'm not being entirel fair. Hard to,from a phonne. I. Sgtill disagree on several grfounds, I'll try to respond more fully when I get a keyb ard.

#753 ::: J Thomas ::: (view all by) ::: May 02, 2015, 08:20 PM:

@751 felice

"Any voting system that throws out your vote after you have one win, will keep slates from getting more than one win. That is a good thing. There are various ranked voting systems that do that. As soon as the first slate work gets in, all the slate ballots that included that work are discarded. Mostly, no more slate."

Unless the slate divides up into five groups each bullet voting one work, in which case the slate will win all the slots after the first work they beat (eg a slate of 195 gets four short story slots in 2013

If a slate has 200 members and they bullet-vote 5 nominations with 40 votes each, and they win, I don't see anything you can possibly do about it. They are like any other bunch of bullet-voters. Possibly the Fannish Inquision might figure out who they are and hunt them down and kill them refund their membership fees.

If you have turned a slate with five nominations that each get 200 votes into five nominations that each get 40 votes, that's about as good as you can possibly do.

#754 ::: felice ::: (view all by) ::: May 02, 2015, 08:36 PM:

J Thomas @753: "If you have turned a slate with five nominations that each get 200 votes into five nominations that each get 40 votes, that's about as good as you can possibly do."

Or one that gets 60 votes, one 52, one 48, and one 40? Good, but not good enough, even before you start counting the effect this has on genuine nominations. If one widely acclaimed work is nominated by 90% of the non-slate voters, so few non-slate ballots will be left that the slate is guaranteed the other four spots (and even without a slate, the remaining works are likely to be eccentric). Or if nobody agrees on what's best the non-slate nominations are widely distributed, you get the current short story category or worse. And there's a huge incentive to avoid nominating anything you think is certain to win without your vote (which could mean nobody nominates the most popular works).

#755 ::: J Thomas ::: (view all by) ::: May 02, 2015, 09:08 PM:

@754 felice

"If you have turned a slate with five nominations that each get 200 votes into five nominations that each get 40 votes, that's about as good as you can possibly do."

Or one that gets 60 votes, one 52, one 48, and one 40? Good, but not good enough, even before you start counting the effect this has on genuine nominations.

There's nothing legal you can do about that, unless you find a voting system that punishes bullet-voting. If they're a slate and the voting system gives them big power for 5 works because 5 votes on a ballot do as much good as 5 bullet votes? (The current system.) Slates are 5 times as good as bullet-voting for them. But if they're strong enough to win with bullet-voting, what can you do about that short of disqualifying them for being puppies? "On the internet nobody knows you're a dog...."

I can imagine a voting system that punishes slates and bullet-votes both. I'm pretty sure you wouldn't like it. ;-)

If one widely acclaimed work is nominated by 90% of the non-slate voters, so few non-slate ballots will be left that the slate is guaranteed the other four spots

Earlier we looked at ways to pick the one with the most votes, and then take just as many votes as that one needed, and distribute fractional votes to all the others on its ballots. That solves this problem with the problem that it brings the slates back too.

We might do better to approach it from the other end. Pick the winner that has the least votes, so after they're discarded you have more ballots left to pick the other winners from. But when I suggested that approach a long time ago it got shouted down.

You've pointed out a real concern and there might be some way to handle it that will look obvious and fair.

#756 ::: felice ::: (view all by) ::: May 02, 2015, 09:22 PM:

J Thomas @755: "There's nothing legal you can do about that, unless you find a voting system that punishes bullet-voting."

My point is I don't think any tallying system is sufficient on its own to combat organised bloc voting; there's nothing to be gained by rehashing all the other possibilities in detail. We can only really beat bloc voting by changing how people nominate as well, eg Option 5b (or democratic counter-slates; or allowing more than five nominations - though that only helps if lots of voters have long lists of potential nominations in each category that they're currently unable to include on their ballots, which I doubt is the case; or...?).

#757 ::: David Harmon ::: (view all by) ::: May 02, 2015, 10:07 PM:

If a given slate (or bloc in general) has enough voters and can organize them well enough, then they win. That's pretty much fundamental to this whole "voting" thing.

It sounds to me like the current proposal at least makes a slate sweep somewhat more difficult, and requiring a lot more effort and coordination on the part of their voters (not just "everybody vote for this list"). I'd say that's about as good as we're likely to get.

#758 ::: Jameson Quinn ::: (view all by) ::: May 02, 2015, 10:28 PM:

OK. I'm back at a keyboard.

Greg, my abbreviated characterization of your position wasn't entirely fair, as I said. But yes, you are saying you'd "strongly oppose" entire classes of systems that I regard as unquestionably improvements over the status quo.

In fact, your proposed system is quite unworkable. As stated, it doesn't even have any bottom-up elimination. So the only way your second vote will ever even get seen, is if your first choice is elected. That's crazy.

And if you added elimination, it would be STV with a strange unpredictable and shifting quota.

Why do I think approval-based systems are best in this case? It's not just the easier voting that others have mentioned. It's that ranked systems open up a whole pandoras box of strategies and even honest pathologies. There's the strategy of voting some nonentity candidate as your top choice, to stop your vote from being deweighted in case your favorite gets elected in the first round. There's the strategy of voting for your favorites in reverse order of viability, independent of the order you prefer them. There's the nonstrategic pathology where a well-liked work that is everyone's second choice is eliminated early on for lack of first-choice support; a pathology which can easily be worsened by various strategies.

And it should be no surprise that a ballot format that allows the voter to make more choices, should allow more strategy.

Against this, you argue that the problem of a voter's 5th choice possibly knocking their 1st choice off the final list makes all unranked systems unacceptable. First off, that would be rare, especially with SDV-LPE. Second, that is avoidable; if you really really don't want that to happen, don't include the 5th choice on the ballot. And third, when that happens, it happens for a reason: the 5th choice actually has more support than the 1st choice; in other words, by making this one voter slightly less satisfied, this result makes other voters markedly more satisfied, so the so-called "unacceptable" result is actually almost certainly improving average satisfaction.

Yes, there are upsides and downsides to both ranking and rating. In this message, I'm taking one side. If you'd see that this is a two-sided issue, I'd be happy to help you explore both of those sides. But if you think that "your system isn't perfect" should mean "I'll threaten to strongly oppose fixing the huge problems with the status quo", then yes, I'm going to continue to stress the upsides of rating and the downsides of ranking. I recommended approval ballot systems for this case having weighed both sides.

#759 ::: Jameson Quinn ::: (view all by) ::: May 02, 2015, 10:42 PM:

Further: several times in this thread, people have advocated systems which are basically STV, except ballots are exhausted the first time they help elect a candidate, with no transfer of overvotes. This is a particularly clear case where strategic backlash is a problem. This kind of system is designed to punish slates and keep them to one nominee only; but it would only create massive strategic incentives, while slates capable of some simple vote management could easily make an end run around these mechanisms.

#760 ::: J Thomas ::: (view all by) ::: May 02, 2015, 11:18 PM:

@756 felice

.My point is I don't think any tallying system is sufficient on its own to combat organised bloc voting

If you have 200 voters and they come in with 300 voters, there's nothing that a voting system *should* do about that.

If you never have more than 40 votes for anything, and they can get 41 bullet votes for each of theirs, what can possibly be done?

But it's possible to hinder slates without crippling regular voters who want to make 5 nominations. I hope we can do better at that and I want to try. If you're tired of trying at that and want to settle for this one voting system, I'll help refine and document this one, and if somebody wants to work with me on others I'll do that too. Possibly elsewhere, if it looks too disruptive here.

We can only really beat bloc voting by changing how people nominate as well, eg Option 5b

Which I think will be strongly rejected. Probably too many are dead-set against it.

or democratic counter-slates;

The same people are strongly against that, but I don't see how they can stop it from happening if other people want to do it, and it works. Social shaming might go a long way, or maybe not.

or allowing more than five nominations - though that only helps if lots of voters have long lists of potential nominations in each category that they're currently unable to include on their ballots, which I doubt is the case

I doubt it's the case too. If people decide it's the only moral way to defeat slates they might do it for awhile until the SP slates go away. If the other slates never get started or quit when the SP slates do, things could get back to normal.

or...?

In the short run they might get lots of new people paying the fee and nominating. I doubt that's sustainable, but if the SPs go away then it doesn't have to last that long.

More people making nominations on the smaller awards, which they will probably do at least in the short run. Or close down the small awards. People who start nominating for small awards might keep on, once they get over the idea that they're supposed to be experts. So it's possible if you can keep slates from overwhelming the big awards, it might work reasonably well for smaller ones too.

Anyway, we have a nominating system that could help some, if they accept it, and we might find one that can help close to twice as much if they'd accept that. We're starting to find out about the constraints.

#761 ::: felice ::: (view all by) ::: May 03, 2015, 12:55 AM:

J Thomas @760: "If you have 200 voters and they come in with 300 voters, there's nothing that a voting system *should* do about that."

Sure, but if you have 2,000 voters spreading their nominations around 100 different works, and they come in with 300 organised voters and sweep the ballots, something definitely should be done (but not just at the tallying stage).

In the ideal system, everyone would read every eligible work in each category and rank them all, and we'd take the five condorcet winners or whatever the closest multiple-winner equivalent is. Obviously that's not even remotely practical, but we want the system we actually use to get as close to the same outcome as possible. The problem with the way we nominate now is that most voters haven't expressed a preference in any given pairing, so it doesn't take many banding together saying "we all prefer these five over any others" to swamp the voters who have expressed a preference for any specific competitor to the slate. It's a problem even without a slate, because most voters haven't expressed a preference between the works that make the ballot and other popular works; only a small subset of voters have their preferences considered.

#762 ::: J Thomas ::: (view all by) ::: May 03, 2015, 07:21 AM:

Sure, but if you have 2,000 voters spreading their nominations around 100 different works, and they come in with 300 organised voters and sweep the ballots, something definitely should be done (but not just at the tallying stage).

I think there's a strong sentiment among fans that Hugo voters should not get organized. I think that's probably strong enough to block proposals to help fans organize to choose electable candidates.

So that's two related goals.

1. Minorities of organized fans should not have control.

2. Fans should stay disorganized and still mostly win.

Add a third,

3. We should be fair to organized fans too.

From those I say that big enough slates should be allowed one win.

A slate that approaches 40% of the membership should be allowed two wins.

But there's a fourth goal.

4. We should not encourage strategic voting of any kind.

The fifth goal shown by the existing 5% rule:

5. Each winner should get some minimum support.

I think a voting system which furthered these goals, which was simple enough and well-explained, might pass. I think it would be considered more important that a slate with 20% of the membership should not get two nominations, than to make sure a slate with 40% of the membership did get two. When there's a slate with 40% of fans, the Hugos will be a different ball game.

I think SDV-LPE is excellent for #3 and #5, and far better than acceptance voting for #1 and #2 but could be better. People say it has some problems for strategic voting, but I believe those will turn out mostly OK.

#763 ::: J Thomas ::: (view all by) ::: May 03, 2015, 08:11 AM:

Here is an argument why SDV-LPE will not significantly encourage bullet-voting.

First terminology. When a nominee has a number of votes that are split between it and other nominees, I'll say the nominee is split. If that's a whole lot I'll call it heavily split.


Rank the nominees by total vote. Look at nominee N. For him to get to position N-1, the following has to happen: sometime before he is eliminated, two of the N-1 nominees ahead of him must be split so heavily they have the lowest scores, and one of them eliminates the other.

For N to win, this has to happen N-5 times.

Think about that for #15. For him to win, 11 of the ones ahead of him have to be so heavily split that they score below him, and they have to stay split enough to score below him until 10 of them have been eliminated.

For any reasonably large N, this is very very unlikely. So if your nominee is not fairly popular, he will win so seldom that even if you bullet-vote for him it just won't matter.

I predict that under reasonable conditions, this will be mostly true for N=9. It might improve things a little bit, but not enough to matter.

It might give a significant improvement for F, #6 though. I don't know for sure.

It could give some improvement for E, #5. Of the top 5, E is the most likely to be removed. But if every one of E's supporters bullet-vote, E will definitely win. Depending on how the numbers come out, this may not look very important. If the chance for F to win goes from 2% to 3%, that's a big change. If the chance for E to win goes from 95% to 100%, that doesn't seem as big.

I predict that in practice, the only time bullet-voting will matter enough to care about is when the nominee you want the most is right on the edge between 5th and 6th place. And you don't really know when that is. You might have a reasonably good idea who's in first place, but by the time it gets to 5th place you could be pretty far off. So it just isn't worth bothering about.

We could kind of test this. Assemble ballots in reasonable distributions, change some of the votes to reflect bullet-voting, and see how much difference it makes. We will never have enough real Hugo data to test it much, but if we generate fake data that's reasonable enough it would say something. Hidden assumptions built into the way we chose the ballots would have an effect, but the actual voting system would have a big effect too.

#764 ::: Jameson Quinn ::: (view all by) ::: May 03, 2015, 08:22 AM:

@762: I think that's well put.

The only way to do better at #1 (no minority takeover) without violating #1 (fans can stay disorganized) is to somehow enable and encourage non-slate fans to vote for more candidates, and listen to those votes.

That means, first of all, lifting the 5-per-voter limit.

Second, it might mean some formal or informal means for fans (who may not all hang out in the same communities) to be reminded of what works are eligible and perhaps worthy. That was the spirit in which I offered my "suggestions" suggestion @686. The intent was NOT to cause favorite betrayal strategy (which I think is what J Thomas meant by "compromise" in #704, and I think is most of what pushes the "HELL NO" button for Cassy B and Fade Manley); that is, I do not want people to change from voting ABCDE to ADEHI. The intent was to promote actual compromise; that is, changing from ABCDE to ABCDEFGHI or from ABC to ABCDE. I still think that there may be a way to promote this kind of breadth that won't get fans' backs up. For instance: some list of eligible nominees, built in real time from all nominees appearing on any ballots, but with some wiki-like mechanism for crowdsourced vetting for ineligible works and duplicates.

I'm not sure, but I suspect that Greg Q's strong preference for ranked systems may partly be motivated on this basis. Whether or not he's thinking this, the line of reasoning might go: "(a) SDV-LPE can sometimes reward a narrow bullet-voting strategy, so (b) the ballots in a rated system might end up being broader, so (c) a rated proportional system would be better able to prevent slates getting 4 of 5 slots." This logic is simply wrong. It's not that (a) or (b) is incorrect; I believe these effects would be insignificant, but I can't rule them out. It's that (a) and (b) don't lead to (c). Because STV and similar ranked systems don't even look at lower choices until higher choices are eliminated, extra breadth in the lower choices does not do anything to help a disorganized majority beat an organized minority.

Note that there are other voting systems that might help in this regard. I'm thinking in particular of Bucklin Transferrable Voting (BTV), in which each ballot has a list of "grades" like "peerless, outstanding, deserving", and the voter can list any number of works beside each grade, and then the system looks for the highest grade where 5 works all pass some threshold number of voters placing them at or above that grade (in a greedy way, so that the set of winners will probably include works with the most "peerless" grades as well as the ones with the broadest support at or above "deserving".) Though this is a good system from a theoretical point of view, honestly I think the extra complexity is just not worth it, so I won't even bother explaining the full algorithm.

#765 ::: Jameson Quinn ::: (view all by) ::: May 03, 2015, 08:29 AM:

@763: I think that's well put too.

I've programmed this sort of simulation to explore strategic sensitivity for single-winner systems (https://github.com/The-Center-for-Election-Science/vse-sim). With that experience, I think the results for the sim you propose would show SDV-LPE in a favorable light. I'm not sure it's worth the work involved if we don't have SDV-LPE skeptics involved from the beginning, though; if the system's supporters make the sim, we could easily be accused of biasing the results, because there are plenty of places a programmer has to make judgment calls, and a biased one could be putting their finger on the scales.

#766 ::: J Thomas ::: (view all by) ::: May 03, 2015, 10:17 AM:

I'm not sure it's worth the work involved if we don't have SDV-LPE skeptics involved from the beginning, though; if the system's supporters make the sim, we could easily be accused of biasing the results, because there are plenty of places a programmer has to make judgment calls, and a biased one could be putting their finger on the scales.

If the software is made public, then at any time skeptics with the skills could bias the inputs their own way and see what results they get.

It doesn't look like a whole lot of work (says the man who hasn't done it yet). There's the voting system which Kilo Watt has already done. (And so have I, but mine still eliminates ties.) There's the system to generate ballots. That's where most of the work is, trying to guess what distribution of ballots is good. And then there's the statistical stuff collecting and displaying the results, which is easy. You probably don't need statistics. If you assume that the people who most like one particular nominee all do bullet voting and nobody else does (beyond normal statistical variation), and with that change there's an extra win for 30 out of 1000 tries, that's a lot of people abandoning their other choices to win 3% of the time -- in the best case.

The hard part is deciding what statistical distribution to draw the votes from. For the moment I'm using a simple power law that looks kind of similar to some of the Hugo data by eyeball. I have theoretical reasons to think that isn't quite right, but I don't have data that really justifies refining it more.

Since programming is faster and easier than designing correct models, I do better to program with what I have and use it to help me think. Some of today's routines will still be good tomorrow, so that's another plus.

#767 ::: Jameson Quinn ::: (view all by) ::: May 03, 2015, 10:25 AM:

If there are any Sasquan organizers or other interested trufans here who would like me to attend the business meeting this year as an outside expert, please contact me: first name dot last name, using google's email service.

#768 ::: Jameson Quinn ::: (view all by) ::: May 03, 2015, 10:51 AM:
The hard part is deciding what statistical distribution to draw the votes from.

Yes, exactly. Except to get results you can trust, you have to try various distributions and various strategic models, and see that the thing you are interested in is relatively robust. ("Sensitivity analysis"). I've done this before, and like any programming, you end up doing more work than you thought.

I'm not saying it's not worth doing. I'm saying that if you're doing it to answer somebody's objections, you should get them to agree on how it should be done first, so they aren't so tempted to move the goalposts afterwards.

#769 ::: Jameson Quinn ::: (view all by) ::: May 03, 2015, 10:57 AM:

@764: oops, the second #1 should have been a #2.

#770 ::: J Thomas ::: (view all by) ::: May 03, 2015, 11:16 AM:

@764 Jameson Quinn

The only way to do better at #1 (no minority takeover) without violating #1 (fans can stay disorganized) is to somehow enable and encourage non-slate fans to vote for more candidates, and listen to those votes.

That's one way. Another way would be to listen less to slate ballots. There are various ways to only give one vote to each slate ballot. So far each of them has been rejected, but we might find a good one.

That means, first of all, lifting the 5-per-voter limit.

I don't see how that could hurt. Of course slates could nominate more choices, but so far the problem has been not eliminating enough of the last ones. SDV-LPE ought to work fine at eliminating more of them early.

Second, it might mean some formal or informal means for fans (who may not all hang out in the same communities) to be reminded of what works are eligible and perhaps worthy.

The consensus seems to be that this should not in any way favor any works over others. It must not favor already-popular works over less-popular ones.

The intent was NOT to cause favorite betrayal strategy (which I think is what J Thomas meant by "compromise" in #704

http://en.wikipedia.org/wiki/Tactical_voting#Compromising

I'm not sure, but I suspect that Greg Q's strong preference for ranked systems may partly be motivated on this basis.

What I heard him say was that he prefers some choices over others, and with any system where he doesn't get to rank them, he has to delete his later choices so they won't hurt the chances for the one he most wants.

This is a valid concern, though I don't know how much difference it makes quantitatively. (My guess is that with SDV-LPE it's far worse to include works that are more popular than your favorite, than works that are less popular.)

Of course, the other way people who want all their picks equally have to choose which to rank last when they didn't want to do that.

I'm thinking in particular of Bucklin Transferrable Voting (BTV), in which each ballot has a list of "grades" like "peerless, outstanding, deserving", and the voter can list any number of works beside each grade

I just invented that part. If you can optionally rank or accept them, the next choice (with a limit of 5 nominees) is to either accept all 5 by putting them all on one line, or rank all 5 by listing them vertically, or putting as many of them as you like on each line. SDV-LPE can handle that as easily as it can handle giving people a choice between straight ranking and straight acceptance.

#771 ::: Jameson Quinn ::: (view all by) ::: May 03, 2015, 01:35 PM:

@770:

Any way of listening less to slate ballots will either be possible to circumvent for an organized group of slate voters, or be problematic for some non-slate voters, or both. I strongly doubt we'll find any mechanism that doesn't cause more troubles than it resolves.

...

"The consensus seems to be that this should not in any way favor any works over others. It must not favor already-popular works over less-popular ones."

I agree. There may still be a way to do this. The point is to remind voters of what's eligible to prompt them to vote for more works (insofar as they deem those extra works eligible.

...

The Wikipedia voting strategy page is slightly problematic; it basically only considers strategies under strict ranking, which are always dishonest strategies. In systems where strict ranking is not mandatory, each strategy comes in both a dishonest and a semi-honest version. Generally, the dishonest version of what wikipedia calls "compromising" is called "betrayal", while the term "compromising" applies more properly to the semi-honest version of the strategy only. At least, that's how I use the terminology. (Similarly, "burial" is the dishonest strategy, while the corresponding semi-honest strategy is usually called "truncation".)

....

"(My guess is that with SDV-LPE it's far worse to include works that are more popular than your favorite, than works that are less popular.)"

Voting for additional works that are less popular than your earlier preferences will generally not hurt your earlier preferences at all. Voting for additional works that are more popular — in the alphabet example, voting ABCDE instead of just ABCD — will generally not hurt your earlier preferences unless one of your earlier preferences turns out to more correlated with the more popular candidates than its closest rivals are. In the letter example, E's strongest correlations are with T and H (because of "THE" voters), so voting ABCDE shouldn't hurt ABCD, but voting THE rather than TH could well hurt T and H.

....

"I just invented..."

I'm not sure what you're saying here. BTV is a complex system. You may have reinvented it — it certainly wouldn't be the first time somebody independently reinvented a voting system — but if you have, you haven't described it here. If people are actually interested in BTV I can describe it, but I don't think it's a better system than SDV-LPE for this case.

#772 ::: Greg Q ::: (view all by) ::: May 03, 2015, 09:46 PM:

J Thomas @749: "Any voting system that throws out your vote after you have one win, will keep slates from getting more than one win. That is a good thing.

May I attempt a translation here? Because what I get from that is "If you like popular works, we don't value your opinion, or your vote."

You're free to have that opinion. I'm free to oppose any voting system put forward to advance that opinion.


I will just note here that, in your eagerness to block "slates", you've greatly increased the power of "one trick ponies."

"Hi, I'd like all my fans to nominate me for Best Novel this year" becomes a lot easier when having one success on your ballot eliminates all your other votes in that category. Can someone explain to me how it's superior to have a voting system where the normal result is each category ends up with one, maybe two, actually popular works, and three or four "campaigning author" works? Because that seems the inevitable result of these type of systems.

#773 ::: P J Evans ::: (view all by) ::: May 03, 2015, 09:51 PM:

772
We ruled out preferential ratings on nominations early in this discussion (there's a previous thread). But perhaps you missed that part?

#774 ::: J Thomas ::: (view all by) ::: May 03, 2015, 09:51 PM:

@771 Jameson Quinn

Any way of listening less to slate ballots will either be possible to circumvent for an organized group of slate voters, or be problematic for some non-slate voters, or both.

You could be right. I haven't seen a proof. It may be that the goals here are sufficiently unusual that the usual standards won't apply. For example, the goal of diversity -- representing many different voters -- in this case trumps the goal of getting more power for more votes. It's fine for smaller nonslate works to beat slate works that have many more votes, provide the nonslate works aren't *too* small. Usually voting systems that try to get representation are looking for *some* representation for minorities, not *lots* of representation for a scattered majority.

There could possibly be an opportunity here that will not generalize well.

Voting for additional works that are more popular — in the alphabet example, voting ABCDE instead of just ABCD — will generally not hurt your earlier preferences unless one of your earlier preferences turns out to more correlated with the more popular candidates than its closest rivals are.

Let's stipulate that I can make examples where a vote for the more popular work causes your preferred less-popular work to fail.

That can happen and still you can be right that it *generally* will not hurt your earlier preferences much.

I can argue that it *will* hurt them. When you vote for something that gets more votes than your favorite, it's better-than-even chances your favorite's score will be reduced for its whole lifetime. So it will face more challenges. The more popular one's score increases so it will face fewer challenges. Your favorite does better if the more popular ones have more challenges since that increases the chance they will knock each other out, and it does better if it has fewer challenges itself.

But the effect of your vote will be small.

Similarly, if you also vote for something that gets fewer votes than your favorite, for awhile your favorite will have its score reduced. If the other one you vote for temporarily has a higher score, there is a chance that your favorite will have a bottom score during that time and can be eliminated. Your vote raises that one's score and reduces your favorite's score. The effect will be small, and smaller than the effect of a more popular one.

Agreed?

#775 ::: J Thomas ::: (view all by) ::: May 03, 2015, 10:15 PM:

@772 Greg Q

"Any voting system that throws out your vote after you have one win, will keep slates from getting more than one win. That is a good thing."

May I attempt a translation here? Because what I get from that is "If you like popular works, we don't value your opinion, or your vote."

No, that isn't what I said, though it's understandable that you'd see it that way.

Let me put it this way. A voting system that throws out slate votes after one slate work has been selected, will keep a slate from getting more than one win. I consider that a good thing.

Now, if the system also chooses the most popular nonslate work, and then throws out every ballot that has a vote for it, I consider that a bad thing. If you vote for anything popular then none of your other votes count at all. But it might be possible to do it some other way. To keep the part about the slates true, but not have the rest.

What if we could have a voting system where the following statement was true: "You can vote for multiple works, and your vote will count for one of them. It will count for the least popular work that can win."

So if your vote was counted for a less popular work it would be wasted because that work would still not win. And if your vote was counted for a more popular work, that work would win with a bigger margin than the one it was counted for.

Would that be OK?

I can imagine you might prefer the following:

Your vote counts for your favorite, and if your favorite fails then it counts for your next-favorite but if your favorite wins then your vote is thrown away and none of your lesser choices is counted.

That would also give a slate one win only.

I could go with either of those, depending on how everything else worked out. We need to balance various goals, and getting one thing done very well might give us something unacceptable for another goal.

I agree we should not just count your vote toward the most-popular work on it, and throw away all your other choices. And ideally a slate with less than 20% of the votes should not have more than one win.

#776 ::: Greg Q ::: (view all by) ::: May 03, 2015, 10:17 PM:

#758 ::: Jameson Quinn

Greg, my abbreviated characterization of your position wasn't entirely fair, as I said. But yes, you are saying you'd "strongly oppose" entire classes of systems that I regard as unquestionably improvements over the status quo.

See, the point is I don't find them improvements over the status quo.

I thought about it a bit, and here's the problem:
In the current situation, it could be that my #1 and #1 choices are in competition for last place. And that by voting fro both of them, I cause #5 to beat #1. That's a bummer.

In the *PV systems, by voting for #1 and #5, #5 winning (even without my vote being the margin of difference) could cause #1 losing to a work I don't like. That is more than a bummer, that's "I think this voting system sucks."


In fact, your proposed system is quite unworkable.

That's fine, I'll note I specified it was a "flawed" system. It was a very rough draft. I'm very much open to improvements.

But as I noted in my previous comment, a system that "blocks" slates by rewarding authors who pimp their works, doesn't strike me as a significant improvement. Which is why I like the idea of any "voting penalty" being inversely proportional to the size of the gap between the top finisher and the next one.

IOW: throwing out all the votes of anyone who voting for a really popular work strikes me as spite, not the basis for a good voting system.

Why do I think approval-based systems are best in this case? It's not just the easier voting that others have mentioned. It's that ranked systems open up a whole pandoras box of strategies and even honest pathologies.

My point is that the unranked *AP systems do the same.

Against this, you argue that the problem of a voter's 5th choice possibly knocking their 1st choice off the final list makes all unranked systems unacceptable. First off, that would be rare, especially with SDV-LPE. Second, that is avoidable; if you really really don't want that to happen, don't include the 5th choice on the ballot.

Again, we're back to forcing me to game the system, and not vote for works I value, simply because of fear that the voting system will screw me over for having done so.

And third, when that happens, it happens for a reason: the 5th choice actually has more support than the 1st choice; in other words, by making this one voter slightly less satisfied, this result makes other voters markedly more satisfied, so the so-called "unacceptable" result is actually almost certainly improving average satisfaction.

There I think you are wrong. My #1 choice was also more popular than the choice it lost to under your voting system, it's just that it lost strength because #5 won.

I have one other response to this, but I think it deserves it's own post.

#777 ::: Greg Q ::: (view all by) ::: May 03, 2015, 10:51 PM:

A thought that crystallized because of something Jameson said:

What is the purpose of the Hugo Nomination process?

1: Is it to find the best X works in each category?

2: Or is it to make the most number of nominators "happy", with happiness being defined as a non-linear function on "number of favorite works nominated", where additional happiness drops off with each nominee approved?

Because the claim for Hugo Nominee, or Hugo Winner, being a meaningful award relies on it being #1. But the justification for the preferred voting systems seems to be #2.

No?

#778 ::: P J Evans ::: (view all by) ::: May 03, 2015, 11:01 PM:

777
Some of the more prolific commenters here seem to confuse the two. We've made it really clear, I hope, that the nomination process doesn't not, and will not, involve ranking things by preference: that's not what it's for; it's just tossing the names of stuff we think is worthy of a rocket into the pot, one pot per category, and then the approximately-five with the most appearances in the pot get on the actual rank-by-preference ballot. (Note that that usually means that those are liked by a lot of people.)

#779 ::: felice ::: (view all by) ::: May 03, 2015, 11:13 PM:

Greg Q @776: "In the *PV systems, by voting for #1 and #5, #5 winning (even without my vote being the margin of difference) could cause #1 losing to a work I don't like. That is more than a bummer, that's "I think this voting system sucks.""

That can't usually happen. The eliminated work is whichever one has the fewest total nominations, irrespective of what else its supporters may have nominated; votes are only split for purposes of ranking. Technically speaking, it could theoretically happen if and only if your extra half-a-vote is enough to bring your #1 up from 5th to 4th in the rankings, so it never gets compared for elimination against a 6th ranked work with more total nominations than your #1, but the chances of that are pretty tiny (4th and 5th ranked works must differ by less than half a vote), it's not clear your #1 deserves to win a spot (more people nominated the one that would beat it!), and you've already got a work you thought was Hugo-worthy on the ballot, even if it wasn't your first choice.


Greg Q @777: "What is the purpose of the Hugo Nomination process?
1: Is it to find the best X works in each category?
2: Or is it to make the most number of nominators "happy", with happiness being defined as a non-linear function on "number of favorite works nominated", where additional happiness drops off with each nominee approved?
Because the claim for Hugo Nominee, or Hugo Winner, being a meaningful award relies on it being #1. But the justification for the preferred voting systems seems to be #2."

#1 isn't possible; "best" is subjective, for a start, and few if any nominators will have read all the works considered by other nominators to be Hugo-worthy, so they can't even venture an informed opinion on which is best. Happiness in #2 could be better defined as a function on "number of works nominated that they consider to be amongst the best", ie the goal is to get a selection of works in each category that as many people as possible believe contains at least one of the best eligible works.

#780 ::: Greg Q ::: (view all by) ::: May 03, 2015, 11:21 PM:

On further contemplation, I think I prefer the following voting system (I'm sure it's been mentioned before):

Two rounds of nominations. Both rounds, each person gets five nominations.

First round you can nominate any appropriate work / person. Take the top 20 (15?) in each category from the first round, with a 5% of voters minimum required (so if 1000 people nominate with 4500 nominations in a category, moving to the second round requires at least 50 nominations).

Final Nominees are the top 5 from the second round.

Because, when all is said and done, the real problem isn't that ~360 people could dominate the Hugo nominations. The real problem is that in a typical year, it generally only took 40 - 50 people, 100 at the most, to get something a Hugo nomination.

When your nominating pool is > 10,000 people (last Worldcon, this Worldcon, and next Worldcon), and > 50,000 potential votes, those numbers are the real joke.

#781 ::: felice ::: (view all by) ::: May 03, 2015, 11:48 PM:

Greg Q @780: "On further contemplation, I think I prefer the following voting system (I'm sure it's been mentioned before): Two rounds of nominations."

That's basically Option 5b; there has been discussion, and will be more. It works best in combination with SDV-LPE, though; it's not in competition with it.

"Take the top 20 (15?) in each category from the first round, with a 5% of voters minimum required"

The 5% threshold is a problem even for selecting the top 5 in some categories; it's much too high a bar for a top 15-20.

#782 ::: J Thomas ::: (view all by) ::: May 03, 2015, 11:55 PM:

@780 Greg Q

On further contemplation, I think I prefer the following voting system (I'm sure it's been mentioned before):

Two rounds of nominations. Both rounds, each person gets five nominations.

Yes, that has been proposed at length. We got an intense reaction from some people -- they don't want it. They're reconciled to having a final election where you rank 5 choices, but they are not willing to have an intermediate step where you choose among the most popular instead of getting to choose whatever you want. That would remove the charming chaos of the existing system and replace it with a popularity contest like other awards. People should choose whatever they like most, with no regard whatsoever for what other people think.

It's good to see somebody on the other side of that issue. If enough people can be persuaded that it isn't bad to have a sort of runoff election, that would make a giant difference. As it is, I'm afraid the idea may be so controversial it won't be accepted at the business meeting.

One practical concern came up, that if we had to have the runoff halfway through the nominations period, that would create havoc in the scheduling because in all the confusion of running the elections, it helps that they don't have to announce the end of the nominations period until near the end. If they have to announce at the halfway point when the end will be, that causes a lot of problems.

At the time I didn't suggest scheduling the runoff for X weeks after the beginning of the nominations, and let the rest of the schedule float. Because it was a practical problem about how to do it, when the big problem was that they thought it should not be done at all.

#783 ::: P J Evans ::: (view all by) ::: May 04, 2015, 12:45 AM:

That would remove the charming chaos of the existing system and replace it with a popularity contest like other awards

It's explicitly been said, several times, that it increases the workload on the administrators, who are doing this in their free time, and aren't paid for it.

So you're not getting to run it your preferred way: start your own awards, if you want to do that.
The Hugos belong to Worldcon, and rules changes have to go past the business meeting - twice. And I don't think that adding another ballot, and making nominations harder or preferential, are going to fly.

#784 ::: Jameson Quinn ::: (view all by) ::: May 04, 2015, 02:03 AM:

Greg Q: I'd like to see you make a ballot set where the problem you fear happens in SDV-LPE. I bet it would take you at least 8 adjustments to get it right. In other words: yes, that's possible, just as it's possible your airplane to Spokane will run into a helicopter.

#785 ::: J Thomas ::: (view all by) ::: May 04, 2015, 02:47 AM:

@783 ::: P J Evans

"That would remove the charming chaos of the existing system and replace it with a popularity contest like other awards"

It's explicitly been said, several times, that it increases the workload on the administrators, who are doing this in their free time, and aren't paid for it.

Yes, that has been explicitly said several times.

However that's a yesbut. The gotcha is that it looks like a lot of people are deadset against it and so it can't pass the business meeting.

If that turns out not to be so, then we should talk to people who have administered the award recently, and people who are about to administer the award, and see what they need. If they think it's just too hard to do then we drop it, if they need specific changes then we do what they need.

It doesn't require nominators to vote again. It doesn't require administrators to clean up the nominations early. (Though making all the existing nominations available early could help a lot with that.) It doesn't require administrators to handle all the paper ballots early. It doesn't require administrators to announce any later schedules.

But there's no point working with them about it if it cannot pass because too many people just don't want it.

#786 ::: Greg Q ::: (view all by) ::: May 04, 2015, 03:10 AM:

#782 ::: J Thomas

They're reconciled to having a final election where you rank 5 choices, but they are not willing to have an intermediate step where you choose among the most popular instead of getting to choose whatever you want. That would remove the charming chaos of the existing system and replace it with a popularity contest like other awards.

You know, that suspiciously sounds like "we want to be able to continue to game the system, we just want to stop others from being able to game it better than us."

Because the whole point of a voting democracy is to have a "popularity contest". No?

From this year's Hugo nominations: A total of 2122 valid nomination forms were received (2119 online and 3 paper).

Given that, having two rounds of nominations should be trivial, because it takes almost 0 data entry, and a quick run of a computer program to get the results of round 1.


#781 ::: felice

Greg Q @780: "On further contemplation, I think I prefer the following voting system (I'm sure it's been mentioned before): Two rounds of nominations."

That's basically Option 5b; there has been discussion, and will be more. It works best in combination with SDV-LPE, though; it's not in competition with it.

No, it's entirely different from any system where your vote gets downgraded after one of your picks gets in.

The 5% threshold is a problem even for selecting the top 5 in some categories; it's much too high a bar for a top 15-20.

Not when it's 5% of voters, not 5% of ballots.

Besides, the point is to have a system that doesn't get swamped by slates. Top 20 + 5% means that even if you have two or three slates, the five most popular with everyone else should get through the first round. Then in the second round, one or two thousand non-slate voters focused on a smaller number of items means we won't get 360 people forcing crap on everyone else.

But we also won't get 50 - 100 people forcing crap on everyone else, which is what the current system allows, and which any sort of "delete / downgrade your ballot after one item makes it" voting system also allows.

#787 ::: J Thomas ::: (view all by) ::: May 04, 2015, 07:30 AM:

@786 Greg Q

You know, that suspiciously sounds like "we want to be able to continue to game the system, we just want to stop others from being able to game it better than us."

That is uncharitable. This is entirely a side issue, but somebody did a tiny bit of analysis:

http://scienceblogs.com/principles/2015/04/09/on-hugo-voting-slates-and-clustering/

He figured that if we assume a slate that nominated 5 works which won the novel award (which is where the money is) the maximum size of that slate would be the number of votes for 5th place.

What if there was a small secret slate, how much could it influence the nominations? Even a small slate could have some effect.

2009 2010 2011 2012 2013
20 1 2 3 1 1
30 1 2 3 2 1
40 4 3 3 2 1

A single very small slate of 20 to 30 nominators would not be very effective. The larger it was, the harder it would be to keep secret. (2011 was special because the 5 runner-ups got more votes than usual compared to the top 5 so a smaller slate could have made more difference than usual.)

Out of all fandom, I can imagine a group of 40 fans who make a secret slate and who don't let anybody know because they are ashamed. But I think the feelings that have been expressed about it are real, and not secret conspirators speaking out trying to save their privilege.

#788 ::: nathanbp ::: (view all by) ::: May 04, 2015, 08:12 AM:

@786 Greg Q
But we also won't get 50 - 100 people forcing crap on everyone else, which is what the current system allows, and which any sort of "delete / downgrade your ballot after one item makes it" voting system also allows.

After reading your posts here, I still don't understand why you're concerned about this. Under the current system for the past decades any reasonably popular author or group that could organize 50-100 people could get one work on the ballot, especially in the shorter fiction categories. No one seems to have been particularly concerned about that. We can't tell exactly without seeing a set of ballots, but SDV (or RAV, or any of the other systems under discussion here), are unlikely to lower the threshold of how many votes are required to be nominated by a large amount, because votes are typically spread over a large number of works and are probably not all that correlated (except for slate voters). On the other hand, without limiting the number of effective votes allowed to each voter, slate voting will continue to dominate unless there is an impossible increase in the number of nominators.

As a side note, I'd suggest that any 3 round voting system continue to allow write-in candidates for the second round.

#789 ::: J Thomas ::: (view all by) ::: May 04, 2015, 08:50 AM:

@786 Greg Q

Besides, the point is to have a system that doesn't get swamped by slates. Top 20 + 5% means that even if you have two or three slates, the five most popular with everyone else should get through the first round. Then in the second round, one or two thousand non-slate voters focused on a smaller number of items means we won't get 360 people forcing crap on everyone else.

You're right about the problem, and something like that approach would probably work.

The problems are these:

1. A bunch of fans don't want it and won't put up with it.

2. Depending on how it's done, it could be a lot of work for the administrators.

If you figure that the most important thing is to get a bunch of popular works out there so fans have an alternative to the slate, then it isn't much work at all.

But if you figure that the administrators' honor requires that they be 100% fair to everybody, then it's a tremendous amount to do. Any work that gets left off the longlist that should be on it, is damaged. Any tiny oversight could mean that the rightful last-place winner would lose. If they accept an obligation to get absolutely everything completely perfect twice in the same time they used to do it once, that's a big thing. It's more than twice as much work, because for example contacting 15-20 authors to see if they want to withdraw is harder than contacting 5. Plus the votes might change while they're making the calls.

It's easy to do it, but it's hard to do it perfectly.

#790 ::: Greg Q ::: (view all by) ::: May 04, 2015, 12:06 PM:

#788 ::: nathanbp

@786 Greg Q
But we also won't get 50 - 100 people forcing crap on everyone else, which is what the current system allows, and which any sort of "delete / downgrade your ballot after one item makes it" voting system also allows.

After reading your posts here, I still don't understand why you're concerned about this. Under the current system for the past decades any reasonably popular author or group that could organize 50-100 people could get one work on the ballot, especially in the shorter fiction categories. No one seems to have been particularly concerned about that.

No one in your circle of friends, perhaps. Plenty in mine.

We are at a bifurcation point. What are the Hugo Awards?

Are they the most prestigious award in Science Fiction and Fantasy?

Or are they the quirky awards of a quirky little group?

Are they the judgement of fandom?

Or are they the province of anyone in the in crowd who can get 50 - 100 people to go along with them?


I can't make that decision for the Worldcon. If you prefer option B, and your only objection to the Sad Puppies is that it's the "wrong" people gaming the system, oh, and damn them for showing off how easy it actually is to game the system, then you're going to chose whatever you want, and I'm going to be on the opposite side.

I don't see any real difference between 50 people getting one undeserving work / person on the ballot, and 360 people getting five such works there. If we're going to fix the system, then let's actually fix it.

#791 ::: nathanbp ::: (view all by) ::: May 04, 2015, 12:52 PM:

@790 Greg Q

I see those as 2 orthogonal concerns. I don't think changing the voting system does much to help or harm people determined to get a single work on the ballot. Something like 5b would make it more difficult by concentrating nomination votes. I do see it as a lesser concern, since if a single truly undeserving work makes the ballot it presumably won't win.

#792 ::: felice ::: (view all by) ::: May 04, 2015, 04:55 PM:

nathanbp @788: "As a side note, I'd suggest that any 3 round voting system continue to allow write-in candidates for the second round."

Option 5b does that, and I agree it's a good thing, but some people seem strongly averse to the idea, in that write-in candidates are strongly disadvantaged (even though they'd be much more disadvantaged by an exclusive second round). Actually, that's a point - allowing write-ins for the final voting round would be an effective anti-slate tactic, though I expect really bad from the administrators' point of view.


J Thomas @789: "1. A bunch of fans don't want it and won't put up with it."

A very small bunch are vocally opposed; we don't know if they represent an opinion widely-held enough to defeat it at the Business Meeting, and the only way to find out is to try. We should take their concerns on board in rewriting the draft proposal commentary.

"Any work that gets left off the longlist that should be on it, is damaged. Any tiny oversight could mean that the rightful last-place winner would lose. If they accept an obligation to get absolutely everything completely perfect twice in the same time they used to do it once, that's a big thing. It's more than twice as much work, because for example contacting 15-20 authors to see if they want to withdraw is harder than contacting 5. Plus the votes might change while they're making the calls."

The way I'd suggest handling typos etc would be for the system to generate a list of all new unique nominations in each category every day, so corrections can be made on an ongoing basis rather than having a big job at the end. Ideally a different volunteer for each category, to minimise the work for each one, and maximise their knowledge of the category (eg remembering what's already been nominated so anything familiar but misspelled will leap out at them). They wouldn't need to see how many nominations each work gets. And there's definitely no need to contact the authors on the longlist - they're just potential finalists, not actual finalists. If they contact us to withdraw at the longlist stage, great; if not, and they make the final ballot, the process is the same as it is now. The proposal specifies taking a copy of the data, so votes changing isn't an issue. Yes, a work that gets left off the longlist is damaged, but if it's not in the top 5, it would have been left off the shortlist anyway, so it's no worse off; and it's only works around 14-15 that are at all likely to miss out incorrectly.

#793 ::: J Thomas ::: (view all by) ::: May 04, 2015, 06:05 PM:

@792 felice

A very small bunch are vocally opposed; we don't know if they represent an opinion widely-held enough to defeat it at the Business Meeting, and the only way to find out is to try.

When it's in good shape, we should probably start trying to lobby for it early. I think it's the sort of thing that has a better chance if everybody at the meeting has already had a chance to study it, as opposed to the sort whose chance is best if the opposition doesn't have time to organize.

We should take their concerns on board in rewriting the draft proposal commentary.

Yes.

The way I'd suggest handling typos etc would be for the system to generate a list of all new unique nominations in each category every day, so corrections can be made on an ongoing basis rather than having a big job at the end.

That sounds good. We should communicate with members of the upcoming vote administration, and members of the outgoing one. If it's possible for the coming year's group to do it without an order to -- perhaps the business meeting could declare that they don't think it's illegal for them to -- then they might want to do it. If not, they might testify to the business meeting whether they think it would be practical. The first one that the business meeting could make a rule for, will not have been chosen by then, right?

The people who are close to the work are the best to say what's possible.

And there's definitely no need to contact the authors on the longlist - they're just potential finalists, not actual finalists.

Once in Paris I visited a friend -- he'd married a French Communist english teacher. My nose had cleared up there for the first time, and all the smells were clear. I remember we sat in her kitchen and I smelled the vegetables cooking, and she smelled of some hair care product I've never smelled since and human milk since she was still breastfeeding their youngest child. She was beautiful, and she explained to me how to stop something from happening in a group. You insist that it had to be fair to everyone. Not just your group and the people you're negotiating with, but everybody. Think of people who are not present and come up with ways it is not fair to them. Come up with other people for whom anything that's fair to the first group has to be unfair to the second. Continue finding ways it's unfair to third parties who can't speak for themselves, until they give up.

I don't think anybody here is deliberately trying that strategy. But some of it does amount to that. Anything that happens to encourage people to vote for some works, is unfair to all the works that are not being encouraged. And of course our intention in saying which works have some chance to beat the slates, is that people would vote for some subset of them.

The backup position from there, is to say that it is unfair to works that ought to be on the list but are not. Insist that any attempt to make and publish a longlist must be done so perfectly, that it is unfair to the nominating administration to ask them to handle such an incredibly heavy workload.

You argue that something in 20th place would be unlikely to get to 5th place even if there was no list. But still, the awards administration is *doing* something that makes it harder, when before it was due to natural forces nobody was responsible for.

And yet it's even more unfair for everyone if a slate takes 3 to 5 of the shortlist positions....

So anyway, if a longlist is announced and then an author withdraws, then the list has only 14 positions and not 15. It isn't fair to someone -- like the work that would have been on the list at the same time as the others if we had known ahead of time. The only way to make it fair to everybody is to contact authors with works on the list ahead of time until there are 15 who promise not to withdraw.

And yet, is it so bad for the administration to do the best they can, and not worry too much if it turns out there are only 14 or 13, or to add more to the longlist as they find out?

I believe your approach is likely to be the best single move to handle the slate problem. Better that without SDV-LPE than SDV-LPE without that. Better to have both. Possibly a better voting system could be found that would be adequate without a longlist, but we don't have it yet.

So it's worth a serious attempt. We need people who have recently run the voting to word the proposal in a way they say will be feasible, and you are making a good start at arguments that it is fair enough.

#794 ::: felice ::: (view all by) ::: May 04, 2015, 07:12 PM:

J Thomas @793: "And yet, is it so bad for the administration to do the best they can, and not worry too much if it turns out there are only 14 or 13, or to add more to the longlist as they find out?"

Adding to the longlist if necessary is in the current version of the proposal.

#795 ::: P J Evans ::: (view all by) ::: May 04, 2015, 07:45 PM:

Aaaaannnnnd....
we have real data now available from 1984.
This is (a) fixed-format text files, as received (with all the misspellings left in)
and (b) files sorted by category (misspellings corrected -I hope!)

The files I received had the extension .BAL, but I've saved them as .TXT to make life easier. There's also a file with necessary data on the contents. It's a 602K zip file.

available at (ROT-13) cw.rinaf88@tznvy.pbz

#796 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 01:46 AM:

Long post warning!

Thanks to PJ, I've now run the 1984 Hugos under the SDV-LPE code. Please keep in mind that this is the original version of the code that eliminates all ties. So far, I haven't seen the community converge to a different tie-breaking system yet, but I will implement that when they have.

As you can see, the results are identical, except for:

- Best Novella
- Best Fan Writer
- Campbell Award

Note that in 1984, Starship declined the nomination, but they wouldn't have made it as a finalist under SDV-LPE. Interestingly, the group that SDV-LPE predicted did end up being the finalists.

SDV-LPE predicted "Her Habiline" for Best Novella instead of "Hurricane Claude".

Also, SDV-LPE predicted Locus as one of the Best Fanzine finalists (based on the actual nomination ballots), but I believe they were actually removed from that competition since obviously they aren't a fanzine. The sixth place choice predicted by SDV-LPE ended up being the real-world fifth choice.

For best fan writer, SDV-LPE predicted TNH to be 6th; she actually was a finalist.

Interestingly, for the Campbell Award, SDV-LPE also predicts six nominations on the final ballot, but selects Zahn and Hambly instead of Tepper and Rosenberg

Discussions and analysis welcome...

Kilo

----------
Best Novel
Winner: Startide Rising by David Brin [Bantam, 1983]
2 Tea with the Black Dragon by R. A. MacAvoy [Bantam, 1983]
3 Millennium by John Varley [Berkley, 1983]
4 Moreta: Dragonlady of Pern by Anne McCaffrey [Ballantine Del Rey, 1983]
5 The Robots of Dawn by Isaac Asimov [Doubleday, 1983]

SDV-LPE
1. robots of dawn the Points: 53 Nominations: 75
2. tea with the black dragon Points: 40.1666666666667 Nominations: 55
3. startide rising Points: 106 Nominations: 137
4. millennium Points: 33.8333333333333 Nominations: 52
5. moreta: dragonlady of pern Points: 42 Nominations: 54

--------------
Best Novella
Winner: "Cascade Point" by Timothy Zahn [Analog Dec 1983]
2 "Hardfought" by Greg Bear [Asimov's Feb 1983]
3 "In the Face of My Enemy" by Joseph H. Delaney [Analog Apr 1983]
4 "Seeking" by David R. Palmer [Analog Feb 1983]
5 "Hurricane Claude" by Hilbert Schenck [F&SF Apr 1983]

SDV-LPE
1. her habiline husband Points: 24.4166666666667 Nominations: 40
2. hardfought Points: 39.5833333333333 Nominations: 61
3. seeking Points: 45.25 Nominations: 80
4. cascade point Points: 39.0833333333334 Nominations: 71
5. in the face of my enemy Points: 21.6666666666667 Nominations: 43

---------------
Best Novelette
Winner: "Blood Music" by Greg Bear [Analog Jun 1983]
2 "The Monkey Treatment" by George R. R. Martin [F&SF Jul 1983]
3 "The Sidon in the Mirror" by Connie Willis [Asimov's Apr 1983]
4 "Slow Birds" by Ian Watson [F&SF Jun 1983]
5 "Black Air" by Kim Stanley Robinson [F&SF Mar 1983]

SDV-LPE
1. black air Points: 25.4166666666667 Nominations: 37
2. blood music Points: 33.1666666666667 Nominations: 44
3. monkey treatment the Points: 24.25 Nominations: 35
4. slow birds Points: 20 Nominations: 32
5. sidon in the mirror the Points: 24.1666666666667 Nominations: 34

----------------
Best Short Story
Winner: "Speech Sounds" by Octavia E. Butler [Asimov's mid-Dec 1983]
2 "Servant of the People" by Frederik Pohl [Analog Feb 1983]
3 "The Geometry of Narrative" by Hilbert Schenck [Analog Aug 1983]
4 "The Peacemaker" by Gardner Dozois [Asimov's Aug 1983]
5 "Wong's Lost and Found Emporium" by William F. Wu [Amazing May 1983]

SDV-LPE
1. wong's lost and found emporium Points: 25.8333333333333 Nominations: 32
2. speech sounds Points: 29.5 Nominations: 37
3. geometry of narrative the Points: 24.5 Nominations: 33
4. servant of the people Points: 21.8333333333333 Nominations: 27
5. peacemaker the Points: 25.3333333333333 Nominations: 31

---------------------
Best Non-Fiction Book
Winner: The Encyclopedia of Science Fiction and Fantasy through 1968, Vol 3: Miscellaneous by Donald H. Tuck [Advent, 1983]
2 The High Kings by Joy Chant, Ian Ballantine, Betty Ballantine, George Sharp, and David Larkin [Bantam, 1983]
3 Dream Makers, Volume II by Charles Platt [Berkley, 1983]
4 Staying Alive: A Writer's Guide by Norman Spinrad [Donning, 1983]
5 The Fantastic Art of Rowena by Rowena Morrill [Pocket, 1983]

SDV-LPE
1. dream makers volume ii Points: 28.7 Nominations: 45
2. high kings the Points: 29.6166666666667 Nominations: 45
3. encyclopedia of sf & f volume 3 the Points: 9.78333333333333 Nominations: 19
4. fantastic art of rowena the Points: 18.5333333333333 Nominations: 29
5. staying alive: a writer's guide Points: 15.3666666666667 Nominations: 26

---------------------------
Best Dramatic Presentation
Winner: Return of the Jedi (1983) [Lucasfilm] Written by Lawrence Kasdan and George Lucas; Story by George Lucas; Directed by Richard Marquand
2 The Right Stuff (1983) [The Ladd Company] Screenplay by Philip Kaufmann; Directed by Philip Kaufmann; based on the book by Tom Wolfe
3 WarGames (1983) [MGM] Written by Lawrence Lasker & Walter F. Parkes; Directed by John Badham
4 Brainstorm (1983) [MGM] Screenplay by Philip Frank Messina and Robert Stitzel; Story by Bruce Joel Rubin; Directed by Douglas Trumbull
5 Something Wicked This Way Comes (1983) [Bryna/Disney] Written by Ray Bradbury; Directed by Jack Clayton; based on the novel by Ray Bradbury

SDV-LPE
1. wargames Points: 42.4166666666667 Nominations: 85
2. return of the jedi Points: 146.333333333333 Nominations: 226
3. brainstorm Points: 35.25 Nominations: 74
4. right stuff the Points: 47.4166666666667 Nominations: 87
5. something wicked this way comes Points: 20.5833333333333 Nominations: 50

------------------------
Best Professional Editor
Winner: Shawna McCarthy
2 David G. Hartwell
3 Terry Carr
4 Edward L. Ferman
5 Stanley Schmidt

SDV-LPE
1. hartwell david Points: 41.6666666666667 Nominations: 75
2. ferman ed Points: 39.5 Nominations: 73
3. carr terry Points: 44.8333333333333 Nominations: 83
4. mccarthy shawna Points: 54.25 Nominations: 96
5. schmidt stanley Points: 48.75 Nominations: 82

------------------------
Best Professional Artist
Winner: Michael Whelan
2 Rowena Morrill
3 Don Maitz
4 Barclay Shaw
5 Val Lakey Lindahn

SDV-LPE
1. maitz don Points: 32.9166666666667 Nominations: 59
2. shaw barclay Points: 28 Nominations: 49
3. morrill rowena Points: 40.5 Nominations: 73
4. whelan michael Points: 88.5 Nominations: 123
5. lindahn val lakey Points: 24.0833333333333 Nominations: 31

----------------
Best Semiprozine
Winner: Locus ed. by Charles N. Brown
2 Science Fiction Chronicle ed. by Andrew I. Porter
3 Science Fiction Review ed. by Richard E. Geis
4 Fantasy Newsletter/Fantasy Review ed. by Robert A. Collins
5 Whispers ed. by Stuart David Schiff
Withdrawn - Nomination Declined:

Starship ed. by Andrew I. Porter

SDV-LPE
1. sf chronicle Points: 43.3666666666667 Nominations: 90
2. sf review Points: 29.5333333333333 Nominations: 60
3. locus Points: 112.366666666667 Nominations: 167
4. fantasy newsletter Points: 15.45 Nominations: 32
5. whispers Points: 9.28333333333333 Nominations: 19

--------------
Best Fanzine
Winner: File 770 ed. by Mike Glyer
2 Ansible ed. by Dave Langford
3 The Philk Fee-Nom-Ee-Non ed. by Paul J. Willett
4 Izzard ed. by Patrick Nielsen Hayden and Teresa Nielsen Hayden
5 Holier Than Thou ed. by Marty Cantor and Robbie Cantor

SDV-LPE
1. izzard Points: 18.1666666666667 Nominations: 22
2. file 770 Points: 73.1666666666667 Nominations: 84
3. holier than thou Points: 16.1666666666667 Nominations: 25
4. philk fee-nom-ee-non the Points: 18.5 Nominations: 20
5. locus Points: 19 Nominations: 19

(NOTE 6th Place: ansible Points: 7.08333333333333 Nominations: 15)

---------------
Best Fan Writer
Winner: Mike Glyer
2 Richard E. Geis
3 Dave Langford
4 Teresa Nielsen Hayden
5 Arthur D. Hlavaty

SDV-LPE
1. langford dave Points: 16.8333333333333 Nominations: 26
2. glyer mike Points: 44.1666666666667 Nominations: 58
3. white ted Points: 8.33333333333333 Nominations: 12
4. geis dick Points: 18.8333333333333 Nominations: 24
5. hlavaty arthur Points: 20.8333333333333 Nominations: 28

(NOTE 6th Place: nielsen-hayden teresa Points: 6.08333333333333 Nominations: 10)

---------------
Best Fan Artist
Winner: Alexis Gilliland
2 William Rotsler
3 Joan Hanke-Woods
4 Brad W. Foster
5 Stu Shiffman

SDV-LPE
1. shiffman stu Points: 20 Nominations: 33
2. gilliland alexis Points: 18.75 Nominations: 34
3. rotsler william Points: 24.5833333333333 Nominations: 37
4. foster brad Points: 17.0833333333333 Nominations: 26
5. hanke-woods joan Points: 16.5833333333333 Nominations: 28

---------------------------------------
John W. Campbell Award for Best New Writer
Winner: R. A. MacAvoy
2 Joseph H. Delaney
3 Lisa Goldstein
4 Warren Norwood
5 Sheri S. Tepper
6 Joel Rosenberg

SDV-LPE
1. macavoy r. a. Points: 102.25 Nominations: 119
2. goldstein lisa Points: 11.4166666666667 Nominations: 17
3. hambly barbara Points: 10.5 Nominations: 14
4. norwood warren Points: 11.6666666666667 Nominations: 16
5. delaney joseph Points: 8.16666666666667 Nominations: 14
6. zahn timothy Points: 12 Nominations: 18


#797 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 01:55 AM:

Oh, one other tidbit, it's worth pointing out that in no case would the actual Hugo winner have changed under SDV-LPE, so that's something to keep in mind.

Kilo

#798 ::: felice ::: (view all by) ::: May 05, 2015, 02:38 AM:

Keith "Kilo" Watt @797: "Oh, one other tidbit, it's worth pointing out that in no case would the actual Hugo winner have changed under SDV-LPE, so that's something to keep in mind."

Well, we can't actually be sure of that; what if say "Her Habiline Husband" was more popular with voters than with nominators? There's no way to be sure it wouldn't have beaten "Cascade Point" given the chance. Of course, it's hard to object to a system for giving a better work a spot on the ballot :) I assume you meant no actual winner was denied a spot on the ballot, which is an important result.

#799 ::: felice ::: (view all by) ::: May 05, 2015, 05:47 AM:

I've had a brief look at how the nomination data changed over time; one month before closing, "startide rising", "robots of dawn", and "moreta: dragonlady of pern" were the top three in best novel, "millenium" was 7th, and "tea with the black dragon" was 14th; ie all the finalists were in the top 15. At two weeks, with a bit over a third of the nominations in, "millenium" was up to 5th, with only "tea" still down at 15th. Aside from the late surge from "tea", the finalists were all right near the top from very early on.

For non-fiction book (the category with the fewest nominations), the top five were set three weeks out; go back much further than that, and there aren't enough ballots to be meaningful.

I'd consider this to be weak evidence that the works that would be finalists under the current system are unlikely to be disadvantaged by publication of a longlist.

#800 ::: David Harmon ::: (view all by) ::: May 05, 2015, 06:10 AM:

Keith "Kilo" Watt #796: Please keep in mind that this is the original version of the code that eliminates all ties. So far, I haven't seen the community converge to a different tie-breaking system yet, but I will implement that when they have.

Can you try implementing various tie-breaking methods and note what if any changes tey produce in that list?

#801 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 09:46 AM:

David@800: Sure can, though some are easier to code than others. Do you have a preference?

Kilo

#802 ::: David Harmon ::: (view all by) ::: May 05, 2015, 09:57 AM:

Keith "Kilo" Watt #801: No, I just note there's been a bit of argument between different methods, and I figure some data would help -- either showing advantages to one or another, or indicating whether it actually matters.

#803 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 10:51 AM:

@802: I suspect that in the data as is, ties are relatively rare, and significant ties even more so. That's why I've emailed PJ asking for the data so I can do some bootstrap (that is, resampling the ballots with replacement), so as to see how frequent ties would be in the universe of "elections mostly like the 1984 Hugo Elections". (Note for non-statisticians: Bootstrapping was the hot new fad in statistics for about a decade recently, so there's reams of papers verifying its theoretical validity).

Also: Tea With the Black Dragon? Startide Rising? Well is it said that the golden age of SF is 12. (With a couple years for me to have gotten my hands on these in paperback...)

#804 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 11:08 AM:

A cursory glance at the round-by-round results last night showed a few ties, but as Jameson said, not that many. I'll post more details when I get home this afternoon.

Kilo

#805 ::: JonW ::: (view all by) ::: May 05, 2015, 11:08 AM:

@#803: I was twice twelve in 1983, and I *really* liked Tea With the Black Dragon.

@#792: Only a "very small bunch" of people have expressed opposition on this thread to the longlist proposal. But fwiw the people who have expressed support for it are a much smaller group still. I haven't participated in this thread for a while, but I too think the longlist is a bad idea.

#806 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 11:14 AM:

@803/805:

You guys are making me feel very, very old right now...

K

#807 ::: J Thomas ::: (view all by) ::: May 05, 2015, 11:38 AM:

@786 GregQ

Because the whole point of a voting democracy is to have a "popularity contest". No?

Well, no.

Some people sort of think that the purpose is to pick the very best work produced that year. We cannot expect to do that, because nobody can read everything, lots of things get read by a small fraction, and very often the one that would be considered the best if everybody read it will not get much chance.

What the nominations should do is to first provide a large list of things that (a large sample of) fans like. Unfortunately the large list is usually discarded, but we now have it for 1984. Usually a longlist of 16 or more is published in a form that can be found if you're willing to dig for it.

Second, it should provide a short list of works that a diverse group of fans think deserve an award. (There is not a consensus that fans ought to like the great works. Some people will vote for things they think are great that they in fact did not enjoy. That's Culture.) Lots of people will read everything on the short list. Somebody who wanted to get a sense of what SF is about could read a lot of things from the list over the years and get a sense of what fans have thought was good.

Third, it should provide a single winner from the short list. The authors on the short list might get some sort of financial benefit so it's important to them; the single winner presumably gets more. It's fun for people to vote for a single winner, and this can be considered the best among the shortlist.

If you nominate while thinking "I have three in mind that deserve the Hugo award, but I will do everything I can to make sure that one particular one of them wins" then you are not acting in the spirit of the thing. Just list the ones you think deserve it, up to five, and let it go at that.

On the other hand, if there are people who are trying to game the system so that the will of the people is subverted, we ought to try to arrange things so they will fail.

The dominant voting system we're looking at now, goes some way to stop that. It's possible that the large majority of nominators will decide that they care more about getting their favorite on the shortlist than about nominating any others. If everybody bullet-votes then it will be just the ones who get the most votes that win. That at least keeps slates from getting too much power.

In almost any acceptance voting system, people who care more about getting their favorite on the ballot than anything else, will bullet-vote. In systems where they are punished for bullet-voting they will vote for additional works they think will not be any competition. I don't think there's anything we can do about people like that. Giving them ranked voting so they can play the game we don't want them to play, is probably not going to help much.

Unless it were to turn out that a ranked voting system did have other big advantages to balance out its negatives. I haven't seen that yet, but if I do see one that looks good I'll try to get people to agree to it.


#808 ::: P J Evans ::: (view all by) ::: May 05, 2015, 11:42 AM:

A note on the data: some things were nominated in more than one category, which requires correction. And the misspellings....
The results as I have them (which have no names attached, I have to figure that out):
novel: 137, 75, 55, 54, 52 \ 49, 47, 44, 44, 42; min to qualify 22
novella: 80, 71, 62, 47, 42 \ 41, 34, 33, 32, 31; 12
novelette: 46, 39, 38, 36, 33 \ 31, 30, 25, 23, 21; 13
short story: 37, 34, 32, 31, 28 \ 22, 21, 21, 17, 16; 15
non-fiction: 45, 45, 29, 26, 19 \ 16, 12, 12, 11, 11; 9
dramatic: 226, 87, 86, 74, 51 \ 39, 27, 24, 23, 20; 17
pro editor: 129, 96, 83, 83, 82 \ 60, 32, 25, 23, 15; 15
pro artist: 123, 73, 59, 49, 31 \ 30, 30, 29, 29, 29; 17
semi-prozine: 185, 94, 70, 49, 23 \ 17, 15, 14; 12
fanzine: 84, 25, 22, 20, 15 \ 14, 13; 12
fan writer: 58, 36, 33, 28, 18 \ 12, 11; 10
fan artist; 37, 36, 34, 32, 28 \ 21, 19, 11, 10; 10
Campbell Award: 123, 19, 19, 18, 18*, 17, 17 \ 15; 13
* not eligible
\ is the break between on the ballot and not on it

#809 ::: Steve Halter ::: (view all by) ::: May 05, 2015, 12:23 PM:

felice@792:It looks like a total of 76 people have commented in this thread. Of those, 3 appear to have expressed approval for the longlist idea while 11 have expressly objected.
26 people have only commented once while 5 have commented 50 or more time.

#810 ::: JonW ::: (view all by) ::: May 05, 2015, 02:51 PM:

Keith "Kilo" Watt @806 --
I was 24 when Tea With the Black Dragon came out, so I'm 56 now (having read a bunch of R.A. MacAvoy in the years following, not all of it to my liking, on the strength of that book and Lens of the World). Are you older than I think you are?

#811 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 04:33 PM:

JonW@810

Ah, I missed the "twice" 12 -- we're about the same age, actually. You've got a couple of years or so on me, though. :)

K

#812 ::: J Thomas ::: (view all by) ::: May 05, 2015, 05:22 PM:

@809 Steve Halter

It looks like a total of 76 people have commented in this thread. Of those, 3 appear to have expressed approval for the longlist idea while 11 have expressly objected.

That looks like a lot of work! Thank you.

If it's easy for you to do, can you tell how many of the objectors had only technical concerns -- about how hart it was to do -- versus existential objections -- that it shouldn't be done?

The technical concerns might be handled, but the fundamental issues would be harder.

It might be time to take this to other blogs etc, and get a sense whether it can get a lot of support or whether the opposition is truly too strong.

It's likely one of the most effective things we can do to oppose slates, far more workable than just getting more people to vote. But only if we can do it. To pass at the business meeting, most of the people at the business meeting would need to already have made their choice. I don't think it can pass depending on people who're undecided when they go in. So maybe now is the time to see how many are open to it.

#813 ::: felice ::: (view all by) ::: May 05, 2015, 05:51 PM:

JonW @805: "I haven't participated in this thread for a while, but I too think the longlist is a bad idea."

Worse than letting slates continue to get multiple slots on the ballot in many categories? There is no way to beat bloc voting just by changing the counting method when nominations are so diverse.

Just 56 bloc voters could have taken three of the best novel slots in 1984 under the current system; 168 could have taken three under any system (probably a lot fewer, depending on how the system hurts coincidental overlap - 30% of people who nominated a finalist also nominated at least one other; 8% of people who nominated 3, 4, or 5 nominated more than one of those).

46 bloc voters could have taken all of non-fiction in 1984 under the current system; 77 could have taken three slots under any system (probably a lot fewer, depending on how the system hurts coincidental overlap - 40% of people who nominated a finalist also nominated at least one other).

38 bloc voters could have taken all of short-story; 93 taken three under any system; 25% of people who nominated a short story finalist also nominated at least one other.

#814 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 06:30 PM:

felice@813:
Just 56 bloc voters could have taken three of the best novel slots in 1984 under the current system; 168 could have taken three under any system

I just ran a sim with the '84 Best Novel ballots and added 150 slate ballots for their own 5 works. If all 150 are identical, then -none- of the slate works made the finals due to the way all ties are eliminated in the current version of the code. That's not likely to happen, of course. So, I re-ran it making sure that none of the slate works had exactly the same number of nominations and the results were the top 3 slate works plus Robots of Dawn and Startide Rising -- which ended up wining, of course.

But let's think about what that means here. There were only 427 nomination ballots cast for Best Novel in '84. The 150 slate ballots that I added puts that number to 577. If the slate puts up 150 nomination ballots, well, then they should get 3.8 nomination slots on average. In other words, any system that denies them those slots can be said to be inherently unfair, can't it?

Perhaps I'm overthinking it. But I do think that you're making the situation sound a bit worse than it actually is. At some point we have to decide whether we simply want to punish slates because they are slates, or if we just want to make sure that they don't claim more than their fair share of the final ballot slots. I'm firmly in the latter camp, but I may be in the minority, I don't know.

Kilo

#815 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 06:38 PM:

Bah...

And of course, I can't do math while running kids around.

It is a bit worse that I was thinking. I divided 577 by 150, which is silly.

150/577 = 26%, so they should get 1.4 slots. So they got basically one extra slot by virtue of coordination. Apologies.

But I still think we do need to decided what our goals are: punish slates, or keep them to their fair share?

Kilo

#816 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 06:44 PM:

For the curious, here were the final results under the slate-enhanced '84 Best Novel:

SLATE A Points: 50.84 Nominations: 150
SLATE B Points: 49.84 Nominations: 149
SLATE C Points: 49.34 Nominations: 148
robots of dawn the Points: 64 Nominations: 75
startide rising Points: 126 Nominations: 137

As you can see, there was a pretty big gap in points between Startide and the others. And Robots came out ahead on points, even though it was way behind in number of nominations, relative the slates.

Kilo

#817 ::: J Thomas ::: (view all by) ::: May 05, 2015, 06:59 PM:

@815 Keith "Kilo" Watt

150/577 = 26%, so they should get 1.4 slots. So they got basically one extra slot by virtue of coordination.

They got twice the slots they deserved.

But I still think we do need to decided what our goals are: punish slates, or keep them to their fair share?

I say, fair share on average. We don't necessarily need to have winners be directly proportional to the number of votes they get, we have to balance that against other goals. But too much punishment for being slates is bad. If we can keep strategic voters from getting away with things, we don't have to punish them for trying and failing.

#818 ::: J Thomas ::: (view all by) ::: May 05, 2015, 07:09 PM:

SLATE A Points: 50.84 Nominations: 150
SLATE B Points: 49.84 Nominations: 149
SLATE C Points: 49.34 Nominations: 148
robots of dawn the Points: 64 Nominations: 75
startide rising Points: 126 Nominations: 137

As you can see, there was a pretty big gap in points between Startide and the others. And Robots came out ahead on points, even though it was way behind in number of nominations, relative the slates.

This is almost a perfect example of what I was saying earlier. It had slightly more than half the votes of two that it beat out.

This is the way the system is designed and there's nothing wrong with it being that way. We just need to understand it and explain it.

#819 ::: felice ::: (view all by) ::: May 05, 2015, 07:12 PM:

Keith "Kilo" Watt @814, 815, 816: "At some point we have to decide whether we simply want to punish slates because they are slates, or if we just want to make sure that they don't claim more than their fair share of the final ballot slots"

Definitely the latter.

"150/577 = 26%, so they should get 1.4 slots. So they got basically one extra slot by virtue of coordination."

1.6 extra slots, or rounding to the nearest whole number, three times as many as they should get. 26% gets three means 74% gets two (assuming everyone who nominated at all would have preferred most of the actual finalists over the slate).

"For the curious, here were the final results under the slate-enhanced '84 Best Novel:"

What was the score for #6 when it got eliminated? I calculate "tea with the black dragon" would have had 43.5 points (overlap with startide and robots), so 133 bloc voters should be enough. 88 would get two slots, when they've got less than 20% of the vote. And best novel is one of the least vulnerable categories. How does short story fare under SDV-LPE with a slate added?

#820 ::: J Thomas ::: (view all by) ::: May 05, 2015, 07:25 PM:

Here is an example:

Ballots:
ABCD 52
ABCDE 49
ABCEF 49
ABCFG 48
ABCGH 47
ABCHI 46
ABCIJ 45
ABCJK 44
ABCKL 43
ABCLM 42
ABCMN 41
ABCNO 40
ABCOP 39
ABCPQ 38
ABCQR 37
ABCRS 36
ABCST 35
ABCTU 34
ABCUV 33
ABCVW 32
ABCWX 31
ABCXY 30
Z 24

Votes:
A 862
B 862
C 862
D 101
E 99
F 97
G 95
H 93

W 63
X 61
Y 30
Z 23

The starting scores are:

A-C very high
Z 23
D 22.8
E 19.8
F 19.6

V 12.6
W 12.4
X 12.2
Y 6

X and Y compete, Y is eliminated
The new changed score is:
X 13.7
V and W compete, W is eliminated
The new score is:
V 14.6

You can see where this is going. Each winner has scores from two kinds of ballots, N/5 + (N-1)/4. Always less than 23.

The last time, it's

D 22.8
E 22.25

D and E compete, D wins.

The final winners are ABCDZ. The votes are 892, 892, 892, 102, 23.

Z had the lowest score and would have been defeated if it was ever matched up against anything. But it never was. It was a competition for “last 5 men standing” and all but 4 of the others defeated each other, before Z got a chance to be defeated.

#821 ::: JonW ::: (view all by) ::: May 05, 2015, 07:50 PM:

Felice@813,

I'm not sure why you say 77 bloc voters could have taken three nonfiction slots under any system. The third-place finisher got 29, so the threshold for a disciplined slate getting three slots under any system is 90. But again, there were only about 180 votes cast in that category. So if the slate brings in at least 90 votes, then it *should* be getting a couple of slots -- the problem is that it's getting one too many.

Weigh against that the deficiencies of the longlist system. If that system has any effect at all in combating slates, it's because a bunch of voters will see works on the longlist and say, "I didn't think enough of those works to nominate them on their merits -- otherwise I would have, longlist or no longlist -- but I'm going nominate a bunch of them anyway, because they're Not On The Slate I Oppose." That's just strategic voting and fighting slates with slates; I'd rather see some bloc get an extra slot.

#822 ::: P J Evans ::: (view all by) ::: May 05, 2015, 07:53 PM:

In fact, there was an attempt at stuffing the ballot box that year.

#823 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 08:17 PM:

OK, I've done bootstrapping with the 1984 data. I don't have my results all nice and pretty, but...

First, disclaimers:

-I did rough string matching to deal with typos. This was imperfect — for instance, it counted the story "cat, the" as being the same as "catch, the", while "macavoy, r. a." was different from "macavoy, r. a. (roberta)" — but I left it as is, as being good enough for my purposes.

-Bootstrapping the full ballot sets for each category turned out to be futile; with so many votes, the results were too stable, so all the bootstrap samples came out the same. So I used smaller bootstrap samples of n ballots. For the numbers below, I used n=600.

-Because bootstrapping involves sampling with replacement, the correlation between different voters is higher in the bootstrap world than in the real world. This should tend to exaggerate the difference between SDV-LPE and the old system.

So, here are the numbers, by category. I did 100 bootstrap samples for each category. Each line contains:

-"[1]", an artifact of the programming language I used (R).
- The average number of candidates in common between the SDV-LPE winner sets and the traditional (plurality) winner sets
- The average number of candidates in the SDV-LPE winner sets.
- The average number of candidates in the traditional (plurality) winner sets. (This is elevated, because with small bootstrap samples, ties are not unlikely).
- The average number of ties that showed up in the last 4 elimination rounds of SDV-LPE. (This is a ceiling on how important the tiebreaker is. I used the "kill them all" version of SDV-LPE).

So, here are the numbers:


[1] 4.12 5.00 5.41 0.00
[1] 4.49 5.00 5.16 0.01
[1] 4.19 5.00 5.23 0.00
[1] 4.30 5.00 5.25 0.00
[1] 4.68 5.00 5.05 0.01
[1] 4.79 5.00 5.00 0.00
[1] 4.73 5.00 5.08 0.04
[1] 4.31 5.00 5.22 0.16
[1] 4.67 5.00 5.10 0.00
[1] 4.37 5.00 5.17 0.00
[1] 4.89 5.00 5.07 0.34
[1] 4.79 5.00 5.04 0.00
[1] 4.03 5.07 5.34 0.35

As you can see, the average overlap has a low of 4.03 and a high of 4.89. If I increase n, this tends towards 5 in all categories except the last one, where it tends towards 4. Note also that if I increased n to realistic values, the number of SDV-LPE ties would go to very small numbers (less than 0.05)

So, all in all, I'd say this demonstrates:

- SDV-LPE is usually pretty similar to the current system without slate voters; but not always indistinguishable.

- Ties are pretty rare with SDV-LPE, so it's probably not worth worrying too much about the tiebreaker.

I wrote my code in R and I could throw it up on github if anybody wants to play with it.

#824 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 08:22 PM:

By roughly eyeballing it, I'd say that my string matching got comfortably above 95% of the cases right. (I used Jaro distance, with a threshold of 0.14 for categories 1-6 and of 0.09 for categories 7-13.)

#825 ::: J Thomas ::: (view all by) ::: May 05, 2015, 08:23 PM:

@822 P J Evans

In fact, there was an attempt at stuffing the ballot box that year.

Was it in some way an unusual attempt? Does it make the year atypical? Perhaps there is a way to "correct" the data to make it more representative?

#826 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 08:26 PM:

Note that the "r.a. macavoy" string matching failure only matters for "best new author", not for "best novel", because I matched on title only. There were only I think 3 voters who said "roberta" so it's not a huge deal.

#827 ::: JonW ::: (view all by) ::: May 05, 2015, 08:30 PM:

Felice @813 again,

Oh I see, I think -- you're getting to 77 by adding 30+26+20. But that won't work. The way to combat the simplest slate-unfriendly voting system (the "discard every ballot after one win" system) is for each of the slate's voters to bullet-vote a specified work. If SlateA got 30, SlateB got 26, and SlateC got 20, then under that voting system the nominees in 1984 would have been The Encyclopedia of Science Fiction and Fantasy, The High Kings, SlateA, Dream Makers, and SlateB. I believe the results under the more complicated system under discussion here would have been the same.

#828 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 08:30 PM:

If anyone were really interested, it would be pretty easy to do the bootstrap resampling with unequal weights, to make it skew towards early voters or late voters. I doubt that it would make a qualitative difference in how similar SDV-LPE is to the status quo (giving each system the same bootstrapped ballots, of course), though it might shift the actual winners significantly.

#829 ::: felice ::: (view all by) ::: May 05, 2015, 08:55 PM:

JonW @821: "I'm not sure why you say 77 bloc voters could have taken three nonfiction slots under any system. The third-place finisher got 29, so the threshold for a disciplined slate getting three slots under any system is 90. But again, there were only about 180 votes cast in that category. So if the slate brings in at least 90 votes, then it *should* be getting a couple of slots -- the problem is that it's getting one too many."

Oops, you're right, I counted beating each of 3, 4, and 5, rather than beating 3 three times, so 90 for three slots, and 54 for two, as the upper limit (fewer would probably suffice since any method to reduce the effect of slates also harms coincidental overlap). And not enough people nominating is a big part of the problem.

"Weigh against that the deficiencies of the longlist system. If that system has any effect at all in combating slates, it's because a bunch of voters will see works on the longlist and say, "I didn't think enough of those works to nominate them on their merits -- otherwise I would have, longlist or no longlist -- but I'm going nominate a bunch of them anyway, because they're Not On The Slate I Oppose." That's just strategic voting and fighting slates with slates; I'd rather see some bloc get an extra slot."

We disagree on whether strategic voting against slates is a good idea, but that aside, I think you're mischaracterising the effect of the longlist. If you were right, all the people who didn't nominate the finalists would vote No Award, because none of the works merited nomination in the first place; and since only a minority generally nominates even the most popular work, No Award would tend to win most categories. It seems more likely to me that people will say "well, it wasn't my first choice, but that's certainly a very good work and I'll happily nominate it since my favourites don't have a chance", or even "I nominated a bunch of more obscure works because they needed the support more, but since most of them haven't made it, I'll nominate these equally good but more mainstream works instead". Or "I didn't nominate before because picking from everything published last year was too overwhelming, but I did read that and that and that, and I agree they're Hugo-worthy, so I'll nominate them now". Or "I hadn't heard of those, but they look really interesting; I'll read what I can over the next few weeks and nominate any I like". Or "I can't believe I forgot to nominate X! I'd better do so now". I'm confident the longlist would generate a lot more genuine nominations for the most popular works, which is a good thing even without a slate. It might generate some purely strategic voting as well when there is a known slate, but I don't see that as making the situation any worse.

And does a "slate" consisting of the works that would have been the finalists in the absence of a slate (plus some runner-ups to give you more choice), as calculated from the actual nominations, really count as a counter-slate?

#830 ::: Keith "Kilo" Watt ::: (view all by) ::: May 05, 2015, 09:02 PM:

JT@818:

This is the way the system is designed and there's nothing wrong with it being that way. We just need to understand it and explain it.

You're right, this is how it's supposed to work. If you prefer the work with the most nominations to win, then by definition you're okay with slates always winning -- they will generally always have the most number of nominations.

You say there's nothing wrong with it being that way, but your next example (in which Z does not get eliminated because the other works had fewer points than it did) implies to me that you think it's a bad or at least odd thing. Maybe I'm just misinterpreting, though.

Kilo

#831 ::: felice ::: (view all by) ::: May 05, 2015, 09:07 PM:

J Thomas @820: "The final winners are ABCDZ. The votes are 892, 892, 892, 102, 23. Z had the lowest score and would have been defeated if it was ever matched up against anything. But it never was. It was a competition for “last 5 men standing” and all but 4 of the others defeated each other, before Z got a chance to be defeated."

I don't see that as a problem. It's an incredibly unlikely distribution of votes; if just one supporter of D hadn't liked B and C, Z wouldn't win (and many similar tiny variations would similarly kill Z). And virtually everyone likes A, B, and C, while comparatively hardly anyone likes any of the alternatives (not just Z). Z getting a slot as representative of a faction with disparate tastes from the rest of fandom isn't unreasonable in this specific situation.

#832 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 09:42 PM:

I agree with Felice@829 that the fact that "No Award" generally does pretty poorly is evidence that a longlist (or a individually-sampled quasi-longlist) would add lots of sincere votes (especially if combined with lifting the 5-per-category vote limit). So I think there should be two separate proposals:

- SDV-LPE
- Quasi-longlist (as I proposed @686) and lifting the 5-per-ballot limit.

These are two fully independent proposals; each would help alone, and together they would not get in each others' way.

Why do I suggest @686 rather than a longlist, when I'm the only one who has favored @686? Several reasons:

-@686 involves only programming work, no extra administrative burden. That's important.
-@686 does favor popular works slightly (they're more likely to come up as suggestions), but even unpopular works get a chance. The point is to encourage broader voting in general, because the cream will rise naturally to the top; the idea is not to promote strategic voting or to say that after a certain date no new works have a chance.

I realize that the @686 proposal would have more of an uphill battle in the BM than the SDV-LPE one. But it's worth a try to do both. The associated FAQ should definitely make the points that felice@829 makes.

I expect that with both SDV-LPE and @686/approval, no slate could get more than 2 slots. With just SDV-LPE, I wouldn't guarantee that slates wouldn't get 3 slots, or perhaps even 4 if the slate candidates got enough non-slate votes. With just @686/approval, I couldn't be sure that slates wouldn't sweep, though it would help reduce the chances of that at least.

#833 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 09:50 PM:

Possibly the @686/approval proposal should state that voters may only add new works to their ballot as time went on, not remove old ones. This would help make it clear that the point is to keep ballots semi-honest, not to encourage any dishonest strategy. (At least, that would make it clearer to somebody like me, who has dreams about Arrow's theorem... would it help for a more typical voter? I can't say.)

#834 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 09:50 PM:

Possibly the @686/approval proposal should state that voters may only add new works to their ballot as time went on, not remove old ones. This would help make it clear that the point is to keep ballots semi-honest, not to encourage any dishonest strategy. (At least, that would make it clearer to somebody like me, who has dreams about Arrow's theorem... would it help for a more typical voter? I can't say.)

#835 ::: J Thomas ::: (view all by) ::: May 05, 2015, 10:35 PM:

@830 Keith "Kilo" Watt

You say there's nothing wrong with it being that way, but your next example (in which Z does not get eliminated because the other works had fewer points than it did) implies to me that you think it's a bad or at least odd thing. Maybe I'm just misinterpreting, though.

It doesn't match people's usual expectations. We will need to explain it carefully so that people don't think there's something wrong.

My explanation that it's like a medieval tournament -- sequential single-elimination -- matches up exactly to how it works. The number of votes is like a knight's strength and skill -- it determines who wins in any pairwise combat. The points influence the order of the list -- who fights who.

If a lot of the ballots for one work are split, then it gets called to fight early, and it will usually fight others whose strength is lower. But there's the chance that it will fight one that's stronger whose split ballots have brought its points just as low, and lose.

The only way for a work other than the five top strongest to win, is if at least two of the top five have scores so low that they fight each other so that at least one of them loses. Without that, the top five in strength will be the winners.

Is there something unsatisfying about this explanation?

I get the feeling that you think I'm against this voting scheme and I'm trying to make it look bad. But I can multi-task. I can work to further this one while I still look for alternatives. If this one has so much momentum among the three other people who are working on it that no alternative has a chance, OK, too bad. I will work to assist this one and look for alternatives both, regardless.

#836 ::: felice ::: (view all by) ::: May 05, 2015, 10:42 PM:

Jameson Quinn @832: "Quasi-longlist (as I proposed @686) and lifting the 5-per-ballot limit."

That could help, yeah. Removing the limit is essential for this option; people aren't likely to drop their first choices for other suggestions if they don't know their first choices (probably) can't win while the suggestions have a much better chance. But even without the limit, I think it's a much weaker anti-slate measure than Option 5b; there's a lot less motivation to check out the suggestions at all - "someone liked this" is less compelling than "lots of people liked this", and there's no reason to be dissatisfied with what they're already nominated.

#837 ::: J Thomas ::: (view all by) ::: May 05, 2015, 10:45 PM:

@834 Jameson Quinn

Possibly the @686/approval proposal should state that voters may only add new works to their ballot as time went on, not remove old ones.

I like that. As I understand it, the way it is now voters with internet ballots can change their votes at any time until the nominations close, but they can never have more than 5 at a time. The rules don't say it can or should be that way, but it is that way and that's accepted.

It would be a change to allow more than 5 votes on one ballot. Again as I understand it, the way it is now you never get a look at any of the other nominations until after the nominations have closed. So a minimal change would be to say that you can make only 5 original nominations, and then you can vote for additional nominations that are already on the list. Presumably somebody new who hasn't voted before could add new nominations even after the longlist was published, but people who have already registered their ballot could only add votes to things which have been nominated by somebody else.

It could be argued that this is just part of having a list of nominations with also a longlist -- people couldn't vote for other nominations before because they were all secret. So voting for other people's nominations is not breaking the rule that you can only nominate 5 yourself.

#838 ::: P J Evans ::: (view all by) ::: May 05, 2015, 10:49 PM:

837
It's five because that's the rule: you can nominate up to five in a category. That's why there's a motion to change that to 4 coming up, because of the juvenile canines.

#839 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 10:57 PM:

@836: How about: Sample all votes until you get 2 votes each for 5 different suggestions. Or just: keep a list of all candidates with 3 or more suggestions, and put out a web page with the up-to-date list for a category as soon as there were X or more candidates on the list (with X as 5 or 10 or so). You'd need to auto-check for typos — for instance, whenever somebody votes for something that, based on string similarity, seems like a possible duplicate, ask them "is this a duplicate". There could still be ineligible works on the list, and you should warn people about that. But it would be pretty clear that the point of such a list would be to remind people what's out there and gently encourage broader voting, not to give people enough information to be strategic.

#840 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 10:58 PM:

@836: How about: Sample all votes until you get 2 votes each for 5 different suggestions. Or just: keep a list of all candidates with 3 or more suggestions, and put out a web page with the up-to-date list for a category as soon as there were X or more candidates on the list (with X as 5 or 10 or so). You'd need to auto-check for typos — for instance, whenever somebody votes for something that, based on string similarity, seems like a possible duplicate, ask them "is this a duplicate". There could still be ineligible works on the list, and you should warn people about that. But it would be pretty clear that the point of such a list would be to remind people what's out there and gently encourage broader voting, not to give people enough information to be strategic.

#841 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 10:59 PM:

aak, sorry about the double-comments. I'll be more careful. (moderators: please delete this if you delete the dups.)

#842 ::: Jameson Quinn ::: (view all by) ::: May 05, 2015, 11:05 PM:

Actually, asking people "is this a duplicate" (when it seems like that might be probable) seems like a good idea for reducing admin burden regardless of anything else.

#843 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 02:51 AM:

All:

At David@800's request, I coded a sample tie-breaking scheme and re-ran all the '84 ballots. The basic idea is that if there was a tie for number of nominations, then the work with the lowest point total was eliminated. The details of the scheme I ran go like this:

As always, take the works with the two lowest point totals and eliminate the work with the least number of nominations.

- If there is a tie in number of nominations for works with the lowest point total, eliminate them both.

- If there is a tie in number of nominations among works with the lowest -and- with the second-lowest point totals, eliminate the lowest point total work (there will never be more than one, due to the rule above).

As a result, the only time we eliminate more than one work in a round is when there are two (or more) works that are tied with the lowest point total and also are tied for number of nominations.

For the one case that remains where we eliminate more than one work, I thought about Cherdenine@677's idea of looking back into the previous rounds to see which had fewer points in those rounds and eliminating that one. I wasn't able to find a case in the '84 ballots where that ever mattered -- by the time a tie needed to be resolved, they had possessed the same number of points and nominations from the beginning. As such, there was no way to test such a case with these ballots.


In all cases but one, the results came out exactly as with the "kill 'em all" tie breaking system that I coded before. The only difference was that it took a few more rounds to reach the end. The one exception was the Campbell Award. The new tie-breaking system knocked Joseph Delaney out of the 6-way tie that existed originally and just left five authors on the final ballot.

In all but three cases (that I noted in scanning the results files), the multiple eliminations only occurred when the works in question only had one or two nominations. There were two cases where the works had three nominations and one case where they had four nominations. It seems that the more nominations they have, the more rare ties become.

The system I coded here has some significant advantages, however. From a computational standpoint, it gets rid of the minor works (those that get only 1 or 2 nominations) very quickly. The size of the pool of works drops dramatically in the first few rounds. This is good, because those works had no chance of getting on the final ballot anyway.

A second (and much smaller) advantage is that if slate manage perfect coordination (so that everyone votes for the exact slate) then the ties that result remove -all- slate works from the final ballot. Of course, as I mentioned, having the slate works differ by a single vote removes this effect, but still, it's helpful.

What do you think? Could this be a valid tie-breaking system?

Kilo

#844 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 02:57 AM:

Clarification: Delaney was not in a 6-way tie for the Campbell Award, the final ballot had six names on it.

K

#845 ::: felice ::: (view all by) ::: May 06, 2015, 06:03 AM:

Keith "Kilo" Watt @843: " If there is a tie in number of nominations for works with the lowest point total, eliminate them both... "

I'm strongly opposed to ever eliminating two works at once (except an initial discarding of works with very few nominations, which can't do any harm; eg dropping everything with fewer than 5 nominations would have eliminated 70% of the best novel candidates in 1984).

"The new tie-breaking system knocked Joseph Delaney out of the 6-way tie that existed originally and just left five authors on the final ballot."

I don't think we wanted to do tie-breaking on ties involving 5th place, did we? I'm assuming that is where Delaney was eliminated - if he was knocked off earlier, more detail on how that happened would be good.

"A second (and much smaller) advantage is that if slate manage perfect coordination (so that everyone votes for the exact slate) then the ties that result remove -all- slate works from the final ballot. Of course, as I mentioned, having the slate works differ by a single vote removes this effect, but still, it's helpful."

It's extremely unlikely to help against slates in practice; a disciplined slate will act to ensure it doesn't hurt them, and an undisciplined slate more or less by definition won't have exact ties. And it could knock out two genuinely popular works with significant overlapping support, if they coincidentally tie. Such ties would be very rare, but not impossible, and the possibility far outweighs the computational savings.

#846 ::: J Thomas ::: (view all by) ::: May 06, 2015, 06:38 AM:

@843

The basic idea is that if there was a tie for number of nominations, then the work with the lowest point total was eliminated.

We considered a variety of systems. None of the "good" ones was guaranteed to work all the time. Order of nomination works all the time but it's kind of arbitrary.

I'd prefer to use just one "good" approach and then use order of nomination as a backup if we want to never keep a tie. Using a laundry list of methods to break ties somehow feels more arbitrary to me than using just one and then the guaranteed way.

Other things equal, I'd prefer the "good" way that should statistically break the most ties. Apart from that I like the one you chose best.

#847 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 07:54 AM:

Evidence regarding my idea @840 of automatically publishing a "superlonglist" consisting of every work with 2 or more nominations, starting as soon as that list has 5 or more members:

In 1984:

1 - novel had 89 works with 2 or more votes.
2 - novella had 19 works with 2 or more votes.
3 - novelette had 73 works with 2 or more votes.
4 - short story had 84 works with 2 or more votes.
5 - non-fiction book had 25 works with 2 or more votes.
6 - dramatic presentation had 38 works with 2 or more votes.
7 - pro editor had 29 works with 2 or more votes.
8 - pro artist had 56 works with 2 or more votes.
9 - semiprozine had 24 works with 2 or more votes.
10 - fanzine had 38 works with 2 or more votes.
11 - fan writer had 42 works with 2 or more votes.
12 - fan artist had 43 works with 2 or more votes.

Those are big numbers. Nobody's going to read everything on that list. But if the list were available, it would not be hard to at least read the list, to "jog your memory" for eligible works you may have forgotten, and thus gently nudge you to vote for more works overall.

#848 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 07:57 AM:

Here's the 3-or-more data from 1984:


Category 1 - novel - had 70 works with 3 or more votes.
Category 2 - novella - had 18 works with 3 or more votes.
Category 3 - novelette - had 59 works with 3 or more votes.
Category 4 - short story - had 63 works with 3 or more votes.
Category 5 - non-fiction book - had 22 works with 3 or more votes.
Category 6 - dramatic presentation - had 31 works with 3 or more votes.
Category 7 - pro editor - had 25 works with 3 or more votes.
Category 8 - pro artist - had 45 works with 3 or more votes.
Category 9 - semiprozine - had 19 works with 3 or more votes.
Category 10 - fanzine - had 29 works with 3 or more votes.
Category 11 - fan writer - had 26 works with 3 or more votes.
Category 12 - fan artist - had 26 works with 3 or more votes.
Category 13 - Campbell Award (not a Hugo) - had 35 works with 3 or more votes.

#849 ::: J Thomas ::: (view all by) ::: May 06, 2015, 08:56 AM:

If anyone would like to contact me privately,

http://alt.free.newsservers.narkive.com/o7Wq4Ypt/sci-physics-last-2-months-20-thousand-posts-news-shared-secrets-com

my email address is #296 on this list. cut-and-paste the address, search for 296.

@847 I would like it if the nominating software presents people with a list of existing names, and lets them click on those to add to an existing nomination. That potentially saves some effort for voters who're nominating, and it potentially saves some effort for administrators.

An administrator has to review each new nomination before it gets posted, or some people will scribble in random junk that has to be deleted. That could be done daily, perhaps?

This is potentially a convenience for almost everyone. One task that previously was unscheduled turns scheduled, and somebody has to maintain a database that presumably wouldn't need as much maintenance if it was kept offline.

Is this something that the Business Meeting would have to approve? Or could a voting administration just do it if they wanted to? I see nothing in the Constitution about letting people change their nominations at any time, but the software currently allows that. If it were to let people nominate off a list or add their write-in, would that take a constitutional change?

It does say in 3.7.1 that nominations are limited to 5 equally-weighted choices.


Anyway to help nominators create enough consensus to get a considerable number of votes without a slate, we would need to provide voting numbers or something like that. And helping nominators create a consensus is exactly what opponents are opposing.

Without that, we need a voting system which has no compunction about selecting works with fewer votes over works with more votes. SDV-LPE is a good start toward that. In each elimination it eliminates the one with fewer votes, but it provides opportunities for slate works to eliminate each other before all of the works with fewer votes have been eliminated.

#850 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 09:32 AM:

felice@845:
I'm strongly opposed to ever eliminating two works at once (except an initial discarding of works with very few nominations, which can't do any harm; eg dropping everything with fewer than 5 nominations would have eliminated 70% of the best novel candidates in 1984).

Keep in mind that in this case, that was exactly what happened: The only time we eliminated more than one work at a time was when the eliminated works had not more than 4 nominations. Still, I'm sure we could manufacture a case where it happened later. I'm not sure how likely that would be to come up in reality, though.

How would you resolve these ties with works having the lowest point total? In every case here (that I found -- I was manually scanning the result files), they were tied for points and nominations from the very first round, so there's no real way to eliminate them without being arbitrary (which I oppose about as strongly as you oppose eliminating more than one at a time).

Kilo


#851 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 09:39 AM:

felice@845:

Delaney was tied with Hambly for total nominations in the last round (14 each). The original code kept both. Hambly, however, had 10.5 points whereas Delaney had 8.17, so the new code eliminated Delaney.

Kilo

#852 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 09:42 AM:

@849:

Anyway to help nominators create enough consensus to get a considerable number of votes without a slate, we would need to provide voting numbers or something like that. And helping nominators create a consensus is exactly what opponents are opposing.

I disagree. The point of my proposal is that simply making it easy and salient for people to vote for more works per ballot will naturally lead to more votes per work, and thus increase resistance to slates, whose number of votes per work is fixed.

So, my proposal is:

1. Lift the 5-votes-per-cat-per-ballot limit.
2. Allow adding votes as the nomination period goes on, but don't allow taking them away (barring special circumstances?)
3. Keep a list of works with 2 or more nominations, and make it easy to add votes for those (by clicking).
4. Any time you vote for a new work, a fuzzy string match is run against all works with existing nominations. If a near match is found, you are asked "Did you mean...?". Some reasonable system decides which spelling is canonical for each work.

I believe this has the principal advantages of option 5b — encouraging more votes for the best non-slate works (by encouraging more votes for all of them) — without the nasty taste of encouraging dishonest strategy. I may be wrong, but for those who oppose a longest, I would be very interested to hear what you think of this proposal.

#853 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 10:13 AM:

grr, autocorrect. I mean, I'd like to hear what people who oppose a _longlist_ feel about my proposal, not "longest".

#854 ::: Steve Halter ::: (view all by) ::: May 06, 2015, 10:20 AM:

J Thomas@812:From memory, it seemed to be fairly equal between people objecting due to administrative difficulty (hardness to implement) and due to objection to the concept of the top 15 list.

#855 ::: Steve Halter ::: (view all by) ::: May 06, 2015, 10:27 AM:

Jameson Quinn@852:That's an interesting proposal. It does seem like it could be mostly automated, so it wouldn't be that much work to maintain.
I am a little hesitant about there being no limit to the number of works you can nominate. In some ways having to think about what the top five works that you read were is a good process for forcing you to think about the quality of the works.
On the other hand, five (or any number) is an arbitrary limit.
So, interesting...

#856 ::: JonW ::: (view all by) ::: May 06, 2015, 10:32 AM:

felice and Jameson @829 & @832:

When I nominate a work, I’m saying, “I think, based on my own tastes and preferences, that this is one of the best works published all year.” When place a work above No Award in the final selection, I’m saying, “I think, taking into account the facts that different voters have different tastes and preferences and that Hugo winners have been of varying quality over the years, that this work is not so bad that its receiving a Hugo would embarrass Worldcon.” They’re different standards.

Under felice’s proposal, which standard would you have the nominator use after seeing the longlist? Not the first, or she’d make almost no nominations beyond the ones she's made already. Not the second, or she’d add nominations for nearly everything on the longlist. In the 1984 balloting, only 70 people (16%) nominated a novel *without* including one of the eventual top fifteen (and of those 70, ten were sequential bullet votes for a single work).* So there won’t be many people saying “gee, everything I nominated is out of the running.” In order to achieve a significant number of new nominations (which is the point, after all), you’d need a lot of people who already have at least one nomination in the longlist to say something like “here’s my chance to nominate a whole bunch of additional works I didn’t like enough to nominate last time round, but I’m going to include them now because they were fine, I guess, and anyway they're popular, so I'm going to help them push out the works I liked better . . . ?” (An alternative would be if people treated the longlist as a reading list and read new works and nominated the ones they really liked, which would be fine. But I think the list is too long and that task too burdensome for that to generate more than a few additional votes.)

I have no problem at all with Jameson’s suggestion @852, which seems to me sound.

- - -

* I don't code, so I hand-counted. 70 may be a smidge too high, to the extent I missed a top fifteen work in individual ballots.

#857 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 10:34 AM:

Jameson@852:

Quick question for you: Under your proposal, most people would (ideally) nominate more than five works. But if there are only five works on the final ballot, and if a slate has the discipline to restrict itself to five nominations, doesn't that increase their power relative to the non-slate voters who picked more than five? I'm not certain, hence the question.

Thanks,
K

#858 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 10:42 AM:

@856: Interesting, thank you.

In response to the question: "what standard should you use?", I'd answer, "whatever standard you want to, as long as it's consistent. However, given the (relatively low) amount of natural overlap, the process would probably work best if the average number of votes per ballot per category was in the neighborhood of 10. I realize that it's unlikely to be that high in practice, so I'd just say, vote for as many as you can in good conscience; a good rule of thumb would be, vote for anything that's better than the median nominee from last year."

#859 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 10:43 AM:

@858: I should have said "at least as good as", not "better than".

#860 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 10:48 AM:

@857: No. In SDV-LPE, by the time the last eliminations are being run — the most decisive ones — no ballot should have more than 5 votes still in the running, and in fact I'd guess the average non-slate ballot should ideally have between 1 and 2 still in the running. The problem is, currently, that average is less than 1, which means too many voters are getting no slate-fighting power at all by that point.

#861 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 10:53 AM:

Additions to my suggestion @852:
5. Fuzzy string matching should be run against all categories, and matches in other categories should be reported. "Are you sure you didn't mean to vote for this in category X?"
6. There should be a way for admistrators to OPTIONALLY mark a given work as ineligible for a given category while votes are still ongoing. I doubt there would be a need for this in general, but it's good to have the option in case circumstances warrant it.

#862 ::: P J Evans ::: (view all by) ::: May 06, 2015, 11:04 AM:

To the people who keep proposing a longlist, particularly one of everything that's eligible:
can you also come up with 36-hour days for the administrators, who are doing this in their copious spare time?

#863 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 11:06 AM:

Jameson@860: That's a good explanation, thanks. That makes sense to me.

Kilo

#864 ::: Cassy B. ::: (view all by) ::: May 06, 2015, 11:15 AM:

Jameson Quinn @858, I'm not sure which Hugo nominators you think will nominate ten works... but I typically nominated one, maybe two, POSSIBLY three, works in a category. I can't see myself ever nominating the full five possibilities, even if I had a longlist suggestion list; I simply don't have the time to read that much. I don't know if I'm typical or atypical, but I certainly am a data point to consider.

#865 ::: Cassy B. ::: (view all by) ::: May 06, 2015, 11:17 AM:

Addendum: I don't know if this is the sort of data preserved, but is there any way to see what proportion of Hugo nominators have nominated the full five possibilities in each (or any) category? If it's 90%, then obviously I'm an outlier and can be disregarded. If it's 10%, then I'm probably pretty typical....

#866 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 11:22 AM:

@862: I don't know if you're talking to me, but if you are: my proposal is that the computers would do this automatically. Yes, somebody has to program the computers, but it's not that much harder than the programming that already needs to exist, and it actually saves work for the administrators manually cleaning up the ballots afterwards.

#867 ::: P J Evans ::: (view all by) ::: May 06, 2015, 11:37 AM:

866
I think you're underestimating the problems as well as the amount of time involved. The human brain is a lot better at dealing with the fuzzies - you saw all the notes in the file, I hope - than a computer. (I was a CS major; I've been dealing with databases and quality control of data for the last 30 years. It isn't that easy.)

#868 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 11:38 AM:

@864: The number 10 was an ideal; as I said, I realize it's impractical. The idea is just, as many as possible.

If I had a robot that was already reading and having opinions on SF works, and following the Hugo process, here's how I'd program it to vote, so that the outcome of a lot of robots voting using my program would be close to ideal (average of 1-2 nominees supported per ballot):

First year: vote for anything you read in the category that's above the median of the things you read in that category that year, or 15 works, whichever is less.

When the nominees come out in the first year: before you go read or reread any of the nominees, record your current rating for all of them. For any you haven't read, record a 0. For any that you have read, but you regret not voting for (that is, you forgot to consider voting for), also record a 0. Now, take the second highest rating in each category, if it's not a 0.

Second and later years: vote for anything that's better than the average (or median...) of your recorded second highest ratings in that category over all past years. If it's the same, flip a coin to see whether to vote for it. (If I were really programming the robot, I'd actually do a hierarchical model to borrow some data across categories and to take into account the years when you didn't read two nominated works in a category; but that's too much math to write here.)

Obviously, I don't expect real humans to follow that program exactly. But it's roughly feasible, and that kind of thing works as long as you tend to have read at least two nominees in the categories you care about at the moment when the nominees are announced. Insofar as you read less than this, you should probably lower your standards for what to vote for, but of course never vote for something that you'd put below "no award".

....

OK, that came out really technical. Here's the tld̦r: look at past years nominee lists and vote for anything that you think would have belonged there, keeping in mind that you should err on the side of voting for as many as possible.

#869 ::: Buddha Buck ::: (view all by) ::: May 06, 2015, 11:42 AM:

PJ Evans @867:

Are you saying Quinn's proposal is a SMOP for the SMOFs?

#870 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 11:48 AM:

@867: I think you're overestimating the precision needed. If a work shows up on the list under two different names, it's not the end of the world. And if it even remotely has a chance, it will show up at least once.

Note that I actually did program an auto-matching algorithm for the 1984 data yesterday, and as I said, I think it got well over 95% of the matches right, even though it did have errors in both directions. 95% is more than good enough to be used with my proposal; while you want essentially 100% when you're determining the final nominee list, during the voting period one or two duplicates on a list of dozens of works is not a problem.

(of course, my auto-matching yesterday took advantage of the fact that R already has good packages for fuzzy string matching using various algorithms, and I just had to pick the algorithm that worked best. But now I did that, I can report that the Jaro distance, aka default Jaro-Winkler distance, works pretty well. That is: 1 − (1/3)(w1m/|a| + w2m/|b| + w3(m − t)/m). Here,|a| indicates the number of characters in a, m is the number of character matches and t the number of transpositions of matching characters. The wi are weights associated with the characters in a, characters in b and with transpositions. A character c of a matches a character from b when c occurs in b, and the index of c in a differs less than max(|a|, |b|)/2 − 1 (where we use integer division) from the index of c in b. Two matching characters are transposed when they are matched but they occur in different order in string a and b.)

#871 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 11:56 AM:

Note that, using my "megalist" proposal, in the categories where the nominees are people and not works, people who have nicknames or optional initials could be at a very slight disadvantage. That is, if the string matching didn't work for them — which failure, in the 1984 data, happened less than 10% of the time, even for authors who had multiple versions of their name like this — they might take as much as 50% longer to show up in the list. That said, this should not be a problem in practice, because any person who has a chance of winning will show up very early in the voting process. 50% longer than a few days is still a small fraction of the full nomination period.

#872 ::: Steve Halter ::: (view all by) ::: May 06, 2015, 12:01 PM:

Jameson Quinn@870:Cool. I agree that the nomination list does not need to be 100% accurate as far as matching goes.
If the idea actually goes to a proposal, it would probably be quite helpful to put supporting code into some useful place like github.

That probably applies to any of the proposals that are relying on software support. Showing people the code they can reuse is already there provides for a better argument than saying it probably is not much work.

#873 ::: Jon Lennox ::: (view all by) ::: May 06, 2015, 12:11 PM:

Are the various longlist proposals different, at a high level, than the Worldcon hosting a public Hugo nominations brainstorming Wiki?

If a Worldcon wanted to do that, I don't see anything in the rules that would stop them.

For that matter, if anyone else wanted to, nothing would stop them either. (Anyone else wouldn't be able to restrict editing privileges to eligible nominators, but whatever.)

#874 ::: Nancy Lebovitz ::: (view all by) ::: May 06, 2015, 12:18 PM:

If the longlist is public, you have people looking at it who might notice duplicates (this will probably include the highly incentivized author) as well as your fuzzy matching program.

#875 ::: Buddha Buck ::: (view all by) ::: May 06, 2015, 12:33 PM:

Nancy @874:

Quinn's longlist proposal would be to make every entry with two or more nominators visible. Highly incentivised authors would still find it difficult to find low-frequency duplicates.

Sure, they could find "Star Trek Into Darkness" and "Star Trek into Darkness" (even if the fuzzy matches didn't catch it) because both are going to have a decent number of nominators. But only Randall Munroe is going to nominate "~-~StAr TrEk InTo DaRkNeSs~-~"

#876 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 12:37 PM:

@872: You'd have to be insane to try to program a web app in R, so my actual code isn't much good. But I just googled "Jaro-Winkler" along with popular web programming languages like python, java, ruby, node, and PHP, and there appear to be packages for all of them.

#877 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 12:48 PM:

Actually, using the thresholds I'd set, "~-~StAr TrEk InTo DaRkNeSs~-~" does match either of the others. My matching was routinely handling things like that (as well as Dave/David, missing "the", extra initials or "Jr." in a name, etc.)

Furthermore, as long as there are no more than two non-matching versions of a given name — which, given the looseness of my fuzzy matching, is a reasonable assumption — the worst that could happen is that the two are equally likely, and it takes an average of 2.75 nominations to get listed instead of 2. That's not the end of the world; for a work with a chance of making the final list, it is probably something like the difference between appearing after 40% of the nominations are in, instead of 30%. I very highly doubt any actual potential Hugo winners would miss being nominated because of such a small change.

#878 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 12:51 PM:

In fact, until I tuned the threshold, my fuzzy matching thought "the first book of swords" and "the sf book of lists" were the same. I'd suggest using such a loose threshold for flagging potential matches, because obviously once it's flagged the voter could easily tell that those two are not the same.

#879 ::: P J Evans ::: (view all by) ::: May 06, 2015, 01:05 PM:

865
The ballots, as entered, actually do have that information - they have a field that indexes the item/line in the category.

#880 ::: Steve Halter ::: (view all by) ::: May 06, 2015, 02:19 PM:

Jameson Quinn@876:Sure. I just mean that there should be supporting code available with the proposal--not what the particular code should be.

#881 ::: Duncan J Macdonald ::: (view all by) ::: May 06, 2015, 03:12 PM:

Jameson Quinn@852-853:

As a dyed-in-the-wool anti-longlist-on-a-visceral-level person I'll respond this way:

Any list of nominated works released between the 'Nominate your choices in each category' and 'Here is the final ballot of top five in each category' has the appearance of a slate of its own -- it doesn't pass the duck test.

Specifically:

1. Lift the 5-votes-per-cat-per-ballot limit.
Why? I see no valid reason to increase a limit that in a lot of cases may never be reached anyway.

2. Allow adding votes as the nomination period goes on, but don't allow taking them away (barring special circumstances?)
Are you saying I can't change my mind? I can do that now (at least electronically), so why am I being penalized?

3. Keep a list of works with 2 or more nominations, and make it easy to add votes for those (by clicking).
So if I am Evil Slate Person, I keep a set of minions who have not yet voted, and, watching results in Real-Time (tm) I coordinate their votes to bring my selected work(s) to the top of the list.

4. Any time you vote for a new work, a fuzzy string match is run against all works with existing nominations. If a near match is found, you are asked "Did you mean...?". Some reasonable system decides which spelling is canonical for each work.
No particular problem with this, except in the larger sense of it being a slate in and of itself.

#882 ::: felice ::: (view all by) ::: May 06, 2015, 03:34 PM:

Cassy B. @865: "is there any way to see what proportion of Hugo nominators have nominated the full five possibilities in each (or any) category? If it's 90%, then obviously I'm an outlier and can be disregarded. If it's 10%, then I'm probably pretty typical...."

In best novel, one nominated nine works, including "startide rising" twice (don't ask me!); 192 nominated five, 48 four, 57 three, 65 two, and 63 one.

In short story: one nominated six; 100 five, 26, 37, 38, 80.

In non-fiction: 24, 10, 25, 32, 87

#883 ::: Michael I ::: (view all by) ::: May 06, 2015, 03:40 PM:

Duncan McDonald@881

I interpreted #3 as an UNRANKED list of all works with at least two nominations. With no actual indication of the number of nominations the works received (aside from getting at least two).

#884 ::: P J Evans ::: (view all by) ::: May 06, 2015, 03:52 PM:

882
It isn't possible for anyone to have nominated 6 in a category - there were only five lines per category on the ballot.
I think you need to check your results.

#885 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 03:55 PM:

882/884: I saw that in the data files as well. I think it's just a mistake in the membership ID number that was reported, so I just assigned the extras to an unused membership ID.

Kilo

#886 ::: felice ::: (view all by) ::: May 06, 2015, 04:38 PM:

Keith "Kilo" Watt @850: "Keep in mind that in this case, that was exactly what happened: The only time we eliminated more than one work at a time was when the eliminated works had not more than 4 nominations. Still, I'm sure we could manufacture a case where it happened later. I'm not sure how likely that would be to come up in reality, though."

I expect it would be rare, but it's not impossible. See #734 for an extreme example.

"How would you resolve these ties with works having the lowest point total? In every case here (that I found -- I was manually scanning the result files), they were tied for points and nominations from the very first round, so there's no real way to eliminate them without being arbitrary (which I oppose about as strongly as you oppose eliminating more than one at a time)."

Ties where not eliminating simultaneously matters are generally ones where there's overlap (and usually a lot of overlap) between supporters of the two works, so in this specific case, I don't think arbitrary is a problem. Whichever work wins, people who said the eliminated work was superior are in a minority. In the event of a tie for entirely non-overlapping works, it shouldn't matter because the winner won't gain any points from the victory and will presumably be eliminated in the next round anyway. Hmmm... I suppose if they were tied for 7th, and both had more nominations than the 5th and 6th, that would lead to an arbitrary finalist...

So, I guess we do need to always compare the works with lowest two point totals for elimination, irrespective of where any ties may occur. Eg F: 6.5 points, G: 6 points, H: 6 points - compare all three and drop the lowest. F 6.5, G 6.5, H 6 - compare all three and drop the lowest. F 6, G 6, H 6, I 4, J 4, K 4, L 4 - compare all seven and drop the lowest. That way, a set of tied works can make its way up the list to 5th place if it needs to. Though that would mean a perfect slate with enough support would get all its works on the final ballot; not the end of the world, since it would be an oversized set of finalists rather than the slate pushing off all the deserving works, and voters could just ignore the slate finalists if they wished; and it's pretty unlikely, since we haven't seen a voting bloc with perfect discipline yet and it also relies on nobody outside the voting bloc nominating any of the slate works. So it depends whether you see arbitrary elimination or having more than five finalists as the bigger problem.

When works are tied on both points and nominations, and aren't in fifth place (in which case we extend the ballot), keep the ones that were nominated first. Yes, it's arbitrary, but it's consistent, and better than eliminating both works when either one could be a finalist without the other.

#887 ::: felice ::: (view all by) ::: May 06, 2015, 05:24 PM:

JonW @856: "Under felice’s proposal, which standard would you have the nominator use after seeing the longlist? Not the first, or she’d make almost no nominations beyond the ones she's made already. Not the second, or she’d add nominations for nearly everything on the longlist."

My proposal sticks with the five nominations limit, so at most they'd pick the five they like best out of the fifteen. So the standard is "these are the works I consider to be the best out of the ones which have a chance of making the final ballot" or "out of the works which have a chance of making the final ballot, I've read and liked these".

"In the 1984 balloting, only 70 people (16%) nominated a novel *without* including one of the eventual top fifteen"

And if just 20 of them added nominations for each of the finalists, that raises the bar for 5th place from 52 to 72; not insignificant! And I think it's safe to assume that at least some people who'd already nominated some top 15 works would nominate at least one more. Plus there are the people who hadn't nominated at all before seeing the longlist.

"(An alternative would be if people treated the longlist as a reading list and read new works and nominated the ones they really liked, which would be fine. But I think the list is too long and that task too burdensome for that to generate more than a few additional votes.)"

If 150 people each read one work on the longlist, like it, and nominate it, that's at least another 10 nominations for everything on the longlist (maybe 20 for some, none for others - which one they read won't be evenly distributed). Or if 300 read one work and half of them like it enough to nominate it. Or 60 people read five works and nominate half of them.

All up, I'm confident the longlist could double the number of nominations for the fifth-placed work in each category.

#888 ::: JonW ::: (view all by) ::: May 06, 2015, 05:42 PM:

Felice @887 —

I wrote: “In the 1984 balloting, only 70 people (16%) nominated a novel *without* including one of the eventual top fifteen.”

And you wrote: “if just 20 of them added nominations for each of the finalists, that raises the bar for 5th place from 52 to 72; not insignificant!”

Hold on. That’s true only if those 20 people each withdraw their old nominations and substitute nominations for the five works that will later turn out to be the finalists. But that would be an amazing coincidence; they don’t know at that point which five those will be. If those 20 each submit two additional nominations spread across the longlist, that would raise the bar for 5th place from 52 to 54.

#889 ::: felice ::: (view all by) ::: May 06, 2015, 06:12 PM:

JonW @888: "Hold on. That’s true only if those 20 people each withdraw their old nominations and substitute nominations for the five works that will later turn out to be the finalists. But that would be an amazing coincidence; they don’t know at that point which five those will be. If those 20 each submit two additional nominations spread across the longlist, that would raise the bar for 5th place from 52 to 54."

20 of the 70 add nominations for the five finalists, 40 more add nominations for the ten works that aren't finalists? But the distribution isn't random; the finalists become finalists because more people like them, so extra nominations are more likely to be nominations for finalists. And it doesn't need to be the same 20 nominating each finalist; fifty people nominating an average of two finalists each doesn't seem implausible; it's a long way short of all 70 people nominating all five finalists.

#890 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 07:02 PM:

@881:

First off, thanks for responding. I'm going to argue back at you, but I respect that you're discussing in good faith, and that there are probably other people who feel like you.

...

Any list of nominated works released between the 'Nominate your choices in each category' and 'Here is the final ballot of top five in each category' has the appearance of a slate of its own -- it doesn't pass the duck test.

What if you could have a comprehensive list of all the eligible works? I realize that's impossible — who's to say that "Animorphs #241" or "The Sex in the City Cookbook" or "110%: Out-of-the-box secrets of B-list billionaires" aren't eligible if they're published in the right time period — but let's pretend it were possible. Would that still sound like a duck to you?

1. Lift the 5-votes-per-cat-per-ballot limit. Why? I see no valid reason to increase a limit that in a lot of cases may never be reached anyway.

We want to encourage people to vote broadly but honestly. If you limit the number of votes per ballot, you create an inherent dilemma for some people: should they "wast their 5 votes" on their favorite candidates, or vote for the "lesser evils" that are more viable? Yes, it's true, many never reach 5 anyway. But if you are so opposed to strategic voting, why don't you want to lift a limit which is promoting dishonest strategy in every single case where it's not irrelevant?

2. Allow adding votes as the nomination period goes on, but don't allow taking them away (barring special circumstances?) Are you saying I can't change my mind? I can do that now (at least electronically), so why am I being penalized?

Why would you want to change your mind, except for strategic reasons? If a work seems nomination-worthy to you in January, why wouldn't it in March?

The reason I suggest this as part of the proposal is to emphasize that the idea is not to cause dishonest strategy. Each work should be evaluated on its own merits, so adding further votes shouldn't go hand-in-hand with removing old ones. But I'm not attached to this as part of the proposal; if it would be less objectionable as a suggestion rather than a rule, that's fine.

3. Keep a list of works with 2 or more nominations, and make it easy to add votes for those (by clicking). So if I am Evil Slate Person, I keep a set of minions who have not yet voted, and, watching results in Real-Time (tm) I coordinate their votes to bring my selected work(s) to the top of the list.

I think you've misunderstood. The list is unordered. All you know is that a work exists and has some minimal level of support. That is not strategically-valuable information.

(Technically, if you carefully watched when works showed up on the list, you could get some weak information about which ones weren't viable. But though viable works would almost always show up early, nonviable ones could show up either early or late; all it takes are two votes, who could easily even come from the same household. So this information would also have almost no strategic value.)

4. Any time you vote for a new work, a fuzzy string match is run against all works with existing nominations. If a near match is found, you are asked "Did you mean...?". Some reasonable system decides which spelling is canonical for each work. No particular problem with this, except in the larger sense of it being a slate in and of itself.

Again, I wonder if a list of all valid works would still seem like a slate to you. Because I feel that this list of anything with 2 or more nominations is a lot closer to "all valid works" than it is to being "a suggested ballot that some people will thoughtlessly vote for as if it were a slate".

#891 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 07:05 PM:

@887:

All up, I'm confident the longlist could double the number of nominations for the fifth-placed work in each category.

I'd say that's a little optimistic. But more than 50% more? Easily. That would mean that slates would get about 50% fewer slots on average... it could easily be the difference between 5 and 3 slate slots, or even 4 and 2.

#892 ::: P J Evans ::: (view all by) ::: May 06, 2015, 07:18 PM:

Y'all are going to have to come up with really good reasons to allow more than five nominations per category, and so far - I'm still waiting.
Y'all are also going to have to come up with really good reasons for a long-list, and so far I'm not seeing those, either.

I think the proposal to limit nominations to four per category makes more sense - most people don't use all five now. The final ballot will still have five nominees, or even six, depending on the next two business meetings.

Remember that anything that gets proposed has to be explained to the voters, who aren't into these very-theoretical details, it has to be easily understandable, and it has to pass through two business meetings.

#893 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 07:41 PM:

@892: The reason is: the more honest people vote, the more votes things will get. This does not apply to slates, because people can't vote twice for the same thing. So it helps honest candidates beat slates. It combines and reinforces a proportional system. Either of those steps alone helps, but putting them together is the only reliable way to get a truly fair result, where 40% of organized voters can't get more than 40% of the slots.

Honest question: does this reason seem too confusing to you? Or not important enough? Or what?

#894 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 07:43 PM:

@891: Sorry, math error. 50% more non-slate votes corresponds to 33% fewer slate slots, not 50% fewer. Percents are just funny that way.

#895 ::: felice ::: (view all by) ::: May 06, 2015, 08:07 PM:

P J Evans @892: "I think the proposal to limit nominations to four per category makes more sense - most people don't use all five now. The final ballot will still have five nominees, or even six, depending on the next two business meetings."

Do you really think a final ballot with four slate works and one or two non-slate works in most categories is an adequate solution? I'm not keen on only having two genuine nominations in each category to choose from. And that assumes the slate isn't strong and smart enough to win all six slots by coordinating how they divide up the works; a disciplined bloc of 135 could have taken all six novelette slots in 2013 under this system; less than 300 could take all of best novel.


Keith "Kilo" Watt @850: "The only time we eliminated more than one work at a time was when the eliminated works had not more than 4 nominations. Still, I'm sure we could manufacture a case where it happened later. I'm not sure how likely that would be to come up in reality, though."

As an example based on the 1984 data, 4 people who nominated "tea with the black dragon" also nominated "moreta: dragonlady of pern". What if just one more person had added "moreta" to their nomination ballot? Then they'd be tied on 55 nominations. Change the overlap with other works just a little, and it's not implausible that they could also have tied for points below five other works. Eliminate either one, and the other makes it to the final ballot; eliminating both seems blatantly unfair to me. It's reasonable for a "tea" supporter adding a nomination for "moreta" to come with the risk of "moreta" beating "tea" as a consequence, but not for the extra nomination to hurt both.

#896 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 09:45 PM:

@895: "Change the overlap with other works just a little, and it's not implausible that they could also have tied for points below five other works."... I think you'd find that it takes clearly more than a "little" to get this to work. But I could be wrong; I'm ready to eat my words if you accomplish this without major changes (say, editing more than half of the ballots for any of the 7 top works). If you can show it's a problem, I'll worry; if you're just going on intuition, my intuition disagrees.

#897 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 09:46 PM:

felice@895:

I'm not keen on only having two genuine nominations in each category to choose from

Let's be careful here. Just because a nomination comes from a slate doesn't mean that it isn't genuine. They have as much right to nominate as anyone else. I think you mean (and I agree if so) that a slate shouldn't get more than its fair share, but that's not actually what you said.


Regarding ties: We need to acknowledge that -any- tie-breaking system is going to give a different final result from if there were no ties in a slightly different situation. It is also going to give a different final result from another tie-breaking system. So, in my mind, arguing that you got a different result isn't really all that persuasive. There isn't a "fated" result, and I disagree that nomination count alone (without any estimate of how strongly you felt about each nomination) is a clear measure of popularity. If you disagree, that's fine, but I don't feel I can accept your contention as a given.

But as I said, I'm not married to any particular tie-breaking system, so I'm happy to entertain anything that people can come up with subject to one proviso: I do not think there should be any randomness or arbitrariness anywhere in the nomination system. It's a non-starter for me, as well as for a number of people I've spoken with. We can accept that we will get a different final ballot as result of the rules of a tie-break, so long as those rules are deterministic. What we can't accept is a different final ballot as a result of a coin flip (and the chronological order of nomination is also essentially a coin flip as well). I don't think a system that involves randomness in any way has a prayer of passing.

So, again, I'm open to other ideas. I do recognize that the different ways to break a tie give different results and some may be more palatable than others. I'm cool with that idea. But so far, I haven't seen a deterministic way to deal with ties that doesn't involve eliminating more than one work at -some- point in the process. I would definitely welcome one if someone can think of it, though.

Just my opinion,
Kilo

#898 ::: J Thomas ::: (view all by) ::: May 06, 2015, 09:48 PM:


Here are average votes per ballot for the top five novels, and for the top 15 novels for 2009 through 2013. I figure 2014 was somewhat contaminated by a slate, and the older we get the less representative of recent years.

2009 0.676056 1.29108
2010 0.5625 1.08565
2011 0.595438 1.32773
2012 0.647182 1.26931
2013 0.644205 1.30009

So in 2009,about there were about 2 votes for something in the top 5 for every three ballots, and about 4 votes for something in the top 15 for every three ballots.

I can't tell anything about how those votes were distributed among the ballots, but this is enough to do some simple arithmetic. Suppose that for every N normal fans we add N/6 or N/5 slate voters. (N/5 is probably about as many as we could hope to handle well.)

So with N/6, we would have on average 5N/6 votes for slate works, and we'd have 4N/6 votes spread among the top 5 nonslate works.

With N/5, we'd have on average N votes for slate works, and 2/3 N votes spread among the top five others.

With N/5, is it any wonder that they would get 3 wins with acceptance voting? They would have 3/5 of the votes between their candidates and the top 5 nonslates. If 2 of the top 5 had more than N/5 votes each, then the other three wou ld not have that much left among them. You could squeeze out one more win if the last two had very little.

Even with N/6, the slate has the majority of acceptance votes. Is it surprsing they might take the majority of wins?

With acceptance voting, to get the N/5 case down to one slate win, where it belongs, you would need to get more than 4N votes for the top 4 (plus however many the 5th gets) instead of 2N/3. That's 6 times as many as they had in 2013.

You could do that if each nominator voted for six times as many works as in 2013. In 2013 (but not other years) we know how many votes there were during the nominations, and it averaged 3.447439353 votes per ballot. We don't know much about the distribution of votes, but we do have that average. With an average of 20 votes per nonslate nominator we could balance the slate. Alternatively, we could get 6 times as many voters.

SDV-LPE is more complicated, because the distribution of votes matters. I'll think about it later.

#899 ::: P J Evans ::: (view all by) ::: May 06, 2015, 09:59 PM:

895
Do you think that we can't handle that possibility? It isn't like it's new.

Remember, you're talking to voters here. We can accept that some slate candidates get through - the system as it is has had the ballot box stuffed at least twice, successfully, that I know of, not counting the juvenile canines. So we're not looking for perfect unstuffable methods of nominating, we're looking for more difficult to stuff.

Also remember that this is going to be done by people, so it has to be understandable by non-computer and non-poli-sci people, and it needs to be doable by people who don't have a lot of time to write programs or fix things that break. 'KISS' is the principle you want to use. (Why do you think things have had minimal changes over the last mumble years? Not breaking systems that work is a good idea.)

#900 ::: P J Evans ::: (view all by) ::: May 06, 2015, 10:01 PM:

895
Do you think that we can't handle that possibility? It isn't like it's new.

Remember, you're talking to voters here. We can accept that some slate candidates get through - the system as it is has had the ballot box stuffed at least twice, successfully, that I know of, not counting the juvenile canines. So we're not looking for perfect unstuffable methods of nominating, we're looking for more difficult to stuff.

Also remember that this is going to be done by people, so it has to be understandable by non-computer and non-poli-sci people, and it needs to be doable by people who don't have a lot of time to write programs or fix things that break. 'KISS' is the principle you want to use. (Why do you think things have had minimal changes over the last mumble years? Not breaking systems that work is a good idea.)

#901 ::: J Thomas ::: (view all by) ::: May 06, 2015, 10:27 PM:

But so far, I haven't seen a deterministic way to deal with ties that doesn't involve eliminating more than one work at -some- point in the process. I would definitely welcome one if someone can think of it, though.

Here's an imperfect idea -- it's OK to eliminate the bottom two of a three-way tie, isn't it? Both of them have individually lost to the one that wins. So if you have no way to tell them apart, you could keep them both until either something eliminates them, or they eliminate something that changes one of their scores so they are no longer tied for score, or they reach fifth place (or higher) and get accepted.

What I don't like about this is that it makes slate ties win unless some slate voters mess up, and copying a slate to a ballot is one of the easiest possible strategies.

When slates have the most votes total, with SDE-LPE the nonslate can win only if slate works get eliminated by other slate works. When slate works tie and can't eliminate each other,that helps them win. They are by far the most likely to tie.

#902 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 10:36 PM:

Just a note on tied "scores" (that is, more than 2 up for elimination). When I was programming, I sorted the "scores" array, set "numInShowdown" to 2, then while scores[numInShowdown] == scores[numInShowdown + 1], added 1 to numInShowdown. This would prevent a "perfect slate" from keeping all its members. Looking at the "lowest two distinct scores" wouldn't. So I have no strong opinion on how to break ties in number of ballots, but as for scores, numbers like 10, 10, 12, 13 should result in just the 10s facing off, not the 12.

#903 ::: felice ::: (view all by) ::: May 06, 2015, 10:43 PM:

P J Evans @900: "Do you think that we can't handle that possibility? It isn't like it's new."

What do you mean it's not new? We haven't had a slate sweeping most of the finalist slots before, a lot of people are understandably not happy about it, and the 4/6 proposal does virtually nothing to prevent it continuing to happen. Not breaking systems that work is a good idea, but it's been demonstrated quite effectively that the current system doesn't work.

#904 ::: Jameson Quinn ::: (view all by) ::: May 06, 2015, 10:49 PM:

@898: "With acceptance voting, to get the N/5 case down to one slate win, where it belongs, you would need to get more than 4N votes for the top 4 (plus however many the 5th gets) instead of 2N/3. That's 6 times as many as they had in 2013."...

If they're perfectly distributed, you only need 2N (enough to beat the slate when divided in 2.) In practice, given uneven distribution, 3N is probably enough. Still, you're right, that means over 3 times as many votes per ballot — that's a bit above the number 10 I guessed above, but not too far off.

I think an average of 6 votes per ballot is achievable in the headline categories. That means some people voting for 3 or 1 and some voting for 9 or 11. That would be just enough to keep your N/5 slate down to 2 nominees — more than they strictly deserve, but not enough to really taint the eventual winner's victory.

#905 ::: J Thomas ::: (view all by) ::: May 06, 2015, 10:51 PM:

So I have no strong opinion on how to break ties in number of ballots, but as for scores, numbers like 10, 10, 12, 13 should result in just the 10s facing off, not the 12.

Say the scores are 10, 10, 12, 13.

As a possibility, I suggest first compare 10, 10. If they tie and you see no fair way to break the tie, perhaps then compare 10, 10, 12. Then if 12 wins, eliminate both 10's. Otherwise eliminate 12.

Now with 12 gone, it's possible the two 10's won't be tied after all. If they aren't a tie then that tie is resolved.

If they are still tied, treat them like a single unit until they aren't tied any more or they win 5th place. While they're tied, what happens to one happens to both. But they don't eliminate *each other*.

What I don't like about this is it makes slate members eliminate each other less.

#906 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 11:11 PM:

Jameson@902:

numbers like 10, 10, 12, 13 should result in just the 10s facing off, not the 12.

Currently, the code would look at 10,10, & 12 for elimination. There are four possibilities:

- If 12 had the fewest nominations, then it is eliminated.

- If one of the 10's had the fewest nominations, then it is eliminated.

- If the 12 and one of the 10's are tied for fewest nominations, the 10 is eliminated.

- If both 10's had the same number of nominations, then they are both eliminated.

If I understand you correctly, you're say the first and third cases shouldn't happen, correct? I can do that, but can you explain the advantages in simple terms?

K

#907 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 11:14 PM:

JT@905:

As a possibility, I suggest first compare 10, 10. If they tie and you see no fair way to break the tie, perhaps then compare 10, 10, 12. Then if 12 wins, eliminate both 10's. Otherwise eliminate 12.

This is precisely how the code works now.

K

#908 ::: felice ::: (view all by) ::: May 06, 2015, 11:19 PM:

Jameson Quinn @896: "... I think you'd find that it takes clearly more than a "little" to get this to work. But I could be wrong; I'm ready to eat my words if you accomplish this without major changes (say, editing more than half of the ballots for any of the 7 top works). If you can show it's a problem, I'll worry; if you're just going on intuition, my intuition disagrees."

Of those who nominated "citadel of the autarch", 16 also nominated "startide rising". Of those who nominated "helliconia summer", 19 also nominated "startide rising". What would deleting those "startide" nominations do to the points? "Startide" would still be well ahead. If that's not enough, add some "startide" nominations to "moreta" and "tea" ballots to bring their points down and push "startide" back up to where it was?

With the actual data, what are the points for each novel in the round with 7 left?


J Thomas @901: "Here's an imperfect idea -- it's OK to eliminate the bottom two of a three-way tie, isn't it? Both of them have individually lost to the one that wins. So if you have no way to tell them apart, you could keep them both until either something eliminates them, or they eliminate something that changes one of their scores so they are no longer tied for score, or they reach fifth place (or higher) and get accepted."

Yes, that works for me.

"What I don't like about this is that it makes slate ties win unless some slate voters mess up, and copying a slate to a ballot is one of the easiest possible strategies."

Not that easy; I'm sure the puppies wanted all their supporters to nominate all the works they put on their slates this time, but it didn't happen. Unless you keep your slate top secret and only tell trusted nominators what's on it (which reduces the size of the voting bloc), and only pick works you're confident not even one other person is going to nominate independently (limiting your options quite a bit, and putting off potential bloc members who want to tell themselves they're nominating work that deserves to win)...

#909 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 11:25 PM:

Felice @908: Assuming I'm parsing the second half of JT's 905 correctly (which is the same, I believe, as his 901), then this is how the current code works. Are you OK with that?

Kilo

#910 ::: Kevin Standlee ::: (view all by) ::: May 06, 2015, 11:47 PM:

If I understand it correctly, the idea of 4/6 and all its variants is that even if slate-voters can forced up to N works onto the ballot, a few years of No Awarding any obviously slated finalists will discourage such tactics, as would-be slate organizers decide that there's no payback in doing it, and would-be slate voters get tired of "wasting" their votes on works that they know will get down-voted by the majority of the members.

#911 ::: Keith "Kilo" Watt ::: (view all by) ::: May 06, 2015, 11:54 PM:

Kevin @910:

As an interesting aside, there's nothing to stop doing something like 4/6 with SDV-LPE, either, though I'm not sure what the effects would be. I imagine some here would prefer more works nominated than fewer, though.

Kilo

#912 ::: felice ::: (view all by) ::: May 07, 2015, 12:30 AM:

Keith "Kilo" Watt @909: "Felice @908: Assuming I'm parsing the second half of JT's 905 correctly (which is the same, I believe, as his 901), then this is how the current code works. Are you OK with that?"

If tied works aren't eliminating each other, yes, that's fine; no problem there.


Kevin Standlee @910: "If I understand it correctly, the idea of 4/6 and all its variants is that even if slate-voters can forced up to N works onto the ballot, a few years of No Awarding any obviously slated finalists will discourage such tactics"

Maybe, but I'd rather not have years of slate-dominated Hugos before the bloc voters get bored and go away, and I'm not sure the Hugos could survive it with any credibility left intact. Can we even be sure that the non-bloc voters wouldn't give up first?

#913 ::: Keith "Kilo" Watt ::: (view all by) ::: May 07, 2015, 12:48 AM:

felice@912:

Just to be clear, under what JT was describing, they can get eliminated at the same time, but only if they have fewer nominations than the next lowest point candidate. So they aren't eliminating each other, that third work is eliminating them. But that's the same situation we've been talking about all along, so I wasn't sure you agreed with it.

Kilo

#914 ::: felice ::: (view all by) ::: May 07, 2015, 01:26 AM:

Keith "Kilo" Watt @913: "Just to be clear, under what JT was describing, they can get eliminated at the same time, but only if they have fewer nominations than the next lowest point candidate. So they aren't eliminating each other, that third work is eliminating them. But that's the same situation we've been talking about all along, so I wasn't sure you agreed with it."

Urgh, no, sorry, I don't agree. My brain's melting a bit here; ignore #912. They both have fewer nominations than the third work, but if you eliminate one, then the other might not ever need to face the third work, if the elimination gives it enough extra points. That's how SDV-LPE is supposed to work, and the tie is interfering with it.

#915 ::: J Thomas ::: (view all by) ::: May 07, 2015, 06:39 AM:

"As a possibility, I suggest first compare 10, 10. If they tie and you see no fair way to break the tie, perhaps then compare 10, 10, 12. Then if 12 wins, eliminate both 10's. Otherwise eliminate 12."

@907 Keith "Kilo" Watt

This is precisely how the code works now.

I didn't know that. The way I did it, which was what I thought we agreed on before, a tie at 10,10 meant 12 was not involved and they eliminated each other whatever 12 would be.

Your way may be better, but it depends on the goals.

@914 Felice

They both have fewer nominations than the third work, but if you eliminate one, then the other might not ever need to face the third work, if the elimination gives it enough extra points. That's how SDV-LPE is supposed to work, and the tie is interfering with it.

If you eliminate one 10, then the other 10 might not need to face the 12. And if you eliminate both, the 12 might not need to face either of them

What's the fair way to decide who gets the chance to avoid single combat? It depends mostly on the luck of the details of how the ballots are distributed. Change the ballots a little -- one ballots gets three high-vote works and two others are all losers, versus they all get one winner -- and the results may change.

Ideally we'd like to be able to point at something that makes the one that doesn't get eliminated the better choice to stay in the game. It was nominated 15 minutes earlier? If we really need to eliminate something, that at least will always tell us which one. If we eliminate both, that's good for all the remaining survivors, and we aren't treating one of them unfairly wrt the other.

And if these two are among the top five votes, two others will be squeezed out unless these two either eliminate themselves or get eliminated by one of the zero-to-three others that have more votes.

Why *not* let them eliminate each other? Is it somehow more unfair to them, than letting them stay is unfair to the ones that would win if they were eliminated?

#916 ::: J Thomas ::: (view all by) ::: May 07, 2015, 07:29 AM:

@914 Felice

Urgh, no, sorry, I don't agree. My brain's melting a bit here; ignore #912.

This is going to happen when we aren't clear about the goals.

Two of our goals are to make it fair, and make it look fair. The procedure is to eliminate the worst one at this moment, repeatedly until there are 5 left.

The special case is that we have two that are tied for worst, and they are so much the same that we have no fair way to decide one is really worse than the other.

Two of our goals conflict. We want to eliminate the worst, and we don't want to eliminate more than one at a time.

We can violate the rule that says not to eliminate more than one. We can make that look as fair as anything else -- they tied for last score, so we compared them, and then they tied for votes too. We're following the same rules we always do, and they tied for last, eliminate them both.

We can violate the rule that says we don't eliminate one unless it's the worst. If we keep one and not the other, that looks bad. Flipping a coin looks worse than going by nomination date, which still doesn't look good.

We can violate the rule that says we eliminate the worst, and keep both of them. They then eliminate as many others as they run into that have lower votes, changing around the scores each time, until something eliminates them or they win.

Or we can violate the rule that says we compare only two unless we have no choice. Take everything which has one of the two lowest scores, and do a melee, and eliminate the worst. So if it's a 3-way tie for lowest and a 2-way tie for second-lowest, compare all 5. Even if they all have the same number of votes, we can eliminate the three with the lower score. You always have somebody to eliminate if there are always two scores.

One of the things I don't like about that is the possibility of two slate members that tie. It won't always happen because they can't coordinate perfectly, but it's possible. If you get two slate members that tie and have the highest votes, with this system they will run the table, eliminating everything ahead of them until they win.

I would prefer that they eliminate each other, and I see no reason why it's unfair for them to do that. The rule is eliminate the lowest, and they tied for lowest place.

#917 ::: David Harmon ::: (view all by) ::: May 07, 2015, 07:45 AM:

felice #903: the 4/6 proposal does virtually nothing to prevent it [sweeping a slot] continuing to happen.

Um, sure it does. It's not related to the tiebreaking issue, but we established a while back that it sharply increases the number of voters, and the amount of coordination, that a slate needs to pull off a sweep. Obviously not an absolute barrier, but definitely an impediment.

#918 ::: J Thomas ::: (view all by) ::: May 07, 2015, 07:56 AM:

@910 Kevin Standlee

If I understand it correctly, the idea of 4/6 and all its variants is that even if slate-voters can forced up to N works onto the ballot, a few years of No Awarding any obviously slated finalists will discourage such tactics ....

That might work on SPs, many of whom will probably go away in a few years regardless, claiming that they have successfully discredited the Hugo awards.

It will discourage others who might make slates because they will feel that the whole approach is considered socially unacceptable.

But when fans do create things that are very close to slates, will there actually be a big push for No Award then? Slates work, except that some fans disapprove. If popular fans do something similar that they can argue is not really a slate, how well would this solution work?

#919 ::: Jameson Quinn ::: (view all by) ::: May 07, 2015, 08:30 AM:

I agree with @916.

re: @917: If SDV-LPE passes, then I would oppose also passing 4/6. But if nothing else passes, then, much as I find it distasteful, 4/6 would help a bit, so in that case I would not oppose it. Therefore, I'd favor putting language in the SDV-LPE proposal that says "If this proposal gets more votes then 4/6, then it supersedes 4/6, and vice versa."

#920 ::: J Thomas ::: (view all by) ::: May 07, 2015, 09:59 AM:

@864 Cassy B.

I'm not sure which Hugo nominators you think will nominate ten works... but I typically nominated one, maybe two, POSSIBLY three, works in a category. I can't see myself ever nominating the full five possibilities, even if I had a longlist suggestion list; I simply don't have the time to read that much. I don't know if I'm typical or atypical, but I certainly am a data point to consider.

For 2013, the only year I have this number for that isn't somewhat contaminated by SP slates, there were an average of about 3.5 votes per ballot. By that standard yes, you're a bit on the low side.

Not that there's anything wrong with that. ;)

I'll make a tenuous chain of reasoning. Suppose that people get persuaded to nominate more works. Before, when they on average did 3.5 nominations and on average 2/3 of one of those nominations was for something in the top 5, what will they do when they nominate more?

In general, the top 5 will be things that got big print runs and advertising etc, things that a lot of people have heard of. Despite the exposure, 1/3 of the nominators did not nominate any of them. If they decide to nominate more, will it be those or something more obscure they remember when they think of it?

Put that aside and assume they vote in the same proportions they do already. Assume slate voters are 1/5 as many as fan voters, which is low for most categories. They will produce N votes for their top 5, and fans will produce 2/3 N for their top 5. If we start with the naive idea that they should get nominations in proportion to their votes, they deserve 3 out of 5 nominations. (And they're likely to get more.)

For fans to deserve to win 3/5 by that standard, we would need to produce 9/4 as many votes for the top 5 as before.

To deserve to win 4/5 we'd need 6 times as many.

I don't think this approach will get very far. But there's nothing wrong with allowing people to nominate more (except that we'd be facing more slate candidate). And there's nothing wrong with encouraging people to nominate more.

#921 ::: Jameson Quinn ::: (view all by) ::: May 07, 2015, 10:29 AM:

OK, here's another attempt at pitching my "encourage broader votes" idea.

Goals:

It should be easy to administrate and count.

It should be easy to vote.

It should be easy to vote effectively. Though it is impossible to create a system with no possibility of strategic advantage, strategies should be minimally beneficial, hard to plan, and/or rarely effective.

In particular, coordinated strategies which allow a minority to control a majority or all of the nominees (through slates or other strategies) should be difficult or impossible.

Insofar as possible, the list of 5 nominees should have something for everyone to be enthusiastic about.

Insofar as possible, the list of 5 nominees should have broad support; most voters should consider most candidates not unworthy of a Hugo.

....

In order to make it easy to count, it is good if people use the same name for the same work. This is possible if there is a public list of "reasonable candidates". It would be impossible to list everything that might be eligible, but it should be possible to strike some kind of a balance, creating a list which has anything with even the slightest chance of winning, but which doesn't list works that are clearly not appropriate for the Hugos. In creating such a list, it's probably better to err on the side of breadth, but that doesn't mean there is no reasonable stopping point.

What better way to create the list, than to use the ballots themselves? Thus, my proposal for an unordered public list of all candidates with 2 or more votes (as soon as there are more than 4 such candidates).

This list helps make voting easier. It helps remind voters of works they may have liked when they read them but not otherwise have thought of for voting. It helps avoid typos and other non-identical votes, thus easing the counting process. It is not an administrative burden to maintain; while administrators should have the power to remove "Mein Kampf" if it shows up, frankly an occasional flaw in the list is not a problem, so admins can be pretty hands-off.

Once you have this list, how can you achieve the other goals — strategy resistance and especially slate-resistance, enthusiastic support, and broad support?

Let's take a tough case: in one of the shorter fiction categories, a slate of 300 divides into 75 votes each for 4 separate candidates, and each slate voter also votes for 0-2 other unique, completely nonviable candidates. In that case, even if the slate is known to exist, there is no way to prove that a given ballot comes from a slate voter; an oblivious honest voter could have easily submitted a valid ballot that happened to look like a slate ballot.

So the only way to stop the slate from capturing 4/5 of the nominations is to have at least two non-slate works with over 75 (largely non-overlapping) votes each. In 2013, the second-, third-, and fourth-place eligible short stories got only 38, 34, and 28 votes. So to keep this hypothetical slate to 3, 2, or 1 slots (where 1 is "fair") has to mean doubling or tripling these totals. That means some combination of the following:

- Increasing the number of non-slate voters, without also increasing the number of slate voters. This is nice to dream about, but hard to ensure.

- Increasing the number of votes per voter.

- Keeping the total number of votes the same, but making it so that these votes are more concentrated on the top 5 works. To me, this is exactly what we want to avoid; becoming a slate in order to beat a slate.

Since the first possibility is unachievable and the third is undesirable, that leaves only the second. In order to achieve the second without the third, we need to:
- Allow voters to vote for more than 5.
- Make it as easy as possible for them to do so.
- Help remind them of things they may want to vote for, without trying to skew their judgement of what deserves it.
- Encourage them to vote broadly; not to set their standards unreasonably high (though of course they should and will still have real, meaningful standards).

Everything about my proposal relates to the above 4 subgoals.

Once again: the point is NOT to skew who a voter will vote for, but to nudge them to broaden it. Some voters will not want to broaden their vote, and that is TOTALLY OK. If a third of the voters each vote the same number, twice as many, and 3 times as many, then that is a doubling of the total number of votes; if vote distributions across works stay approximately the same (which there is every reason to believe will be the case, based on the relevant social science and statistical literature), that would be enough to ensure at least 2 and probably 3 non-slate wins against even the tough-to-beat slate I've hypothesized.

tld̦r: for some slates, more votes per ballot on average is the only reasonable way to beat them. My proposal is the simple, logical way to get more votes per ballot on average without biasing the voters towards particular works (that is, without creating a "counter-slate").

#922 ::: Jameson Quinn ::: (view all by) ::: May 07, 2015, 10:30 AM:

OK, here's another attempt at pitching my "encourage broader votes" idea.

Goals:

It should be easy to administrate and count.

It should be easy to vote.

It should be easy to vote effectively. Though it is impossible to create a system with no possibility of strategic advantage, strategies should be minimally beneficial, hard to plan, and/or rarely effective.

In particular, coordinated strategies which allow a minority to control a majority or all of the nominees (through slates or other strategies) should be difficult or impossible.

Insofar as possible, the list of 5 nominees should have something for everyone to be enthusiastic about.

Insofar as possible, the list of 5 nominees should have broad support; most voters should consider most candidates not unworthy of a Hugo.

....

In order to make it easy to count, it is good if people use the same name for the same work. This is possible if there is a public list of "reasonable candidates". It would be impossible to list everything that might be eligible, but it should be possible to strike some kind of a balance, creating a list which has anything with even the slightest chance of winning, but which doesn't list works that are clearly not appropriate for the Hugos. In creating such a list, it's probably better to err on the side of breadth, but that doesn't mean there is no reasonable stopping point.

What better way to create the list, than to use the ballots themselves? Thus, my proposal for an unordered public list of all candidates with 2 or more votes (as soon as there are more than 4 such candidates).

This list helps make voting easier. It helps remind voters of works they may have liked when they read them but not otherwise have thought of for voting. It helps avoid typos and other non-identical votes, thus easing the counting process. It is not an administrative burden to maintain; while administrators should have the power to remove "Mein Kampf" if it shows up, frankly an occasional flaw in the list is not a problem, so admins can be pretty hands-off.

Once you have this list, how can you achieve the other goals — strategy resistance and especially slate-resistance, enthusiastic support, and broad support?

Let's take a tough case: in one of the shorter fiction categories, a slate of 300 divides into 75 votes each for 4 separate candidates, and each slate voter also votes for 0-2 other unique, completely nonviable candidates. In that case, even if the slate is known to exist, there is no way to prove that a given ballot comes from a slate voter; an oblivious honest voter could have easily submitted a valid ballot that happened to look like a slate ballot.

So the only way to stop the slate from capturing 4/5 of the nominations is to have at least two non-slate works with over 75 (largely non-overlapping) votes each. In 2013, the second-, third-, and fourth-place eligible short stories got only 38, 34, and 28 votes. So to keep this hypothetical slate to 3, 2, or 1 slots (where 1 is "fair") has to mean doubling or tripling these totals. That means some combination of the following:

- Increasing the number of non-slate voters, without also increasing the number of slate voters. This is nice to dream about, but hard to ensure.

- Increasing the number of votes per voter.

- Keeping the total number of votes the same, but making it so that these votes are more concentrated on the top 5 works. To me, this is exactly what we want to avoid; becoming a slate in order to beat a slate.

Since the first possibility is unachievable and the third is undesirable, that leaves only the second. In order to achieve the second without the third, we need to:
- Allow voters to vote for more than 5.
- Make it as easy as possible for them to do so.
- Help remind them of things they may want to vote for, without trying to skew their judgement of what deserves it.
- Encourage them to vote broadly; not to set their standards unreasonably high (though of course they should and will still have real, meaningful standards).

Everything about my proposal relates to the above 4 subgoals.

Once again: the point is NOT to skew who a voter will vote for, but to nudge them to broaden it. Some voters will not want to broaden their vote, and that is TOTALLY OK. If a third of the voters each vote the same number, twice as many, and 3 times as many, then that is a doubling of the total number of votes; if vote distributions across works stay approximately the same (which there is every reason to believe will be the case, based on the relevant social science and statistical literature), that would be enough to ensure at least 2 and probably 3 non-slate wins against even the tough-to-beat slate I've hypothesized.

tld̦r: for some slates, more votes per ballot on average is the only reasonable way to beat them. My proposal is the simple, logical way to get more votes per ballot on average without biasing the voters towards particular works (that is, without creating a "counter-slate").

#923 ::: Jameson Quinn ::: (view all by) ::: May 07, 2015, 10:47 AM:

@920: I strongly agree with your basic argument here, but I have a couple of quibbles.

1. In order to "deserve" exactly 4/5 of the slots in your example, you're right that the non-slate voters would need 6 times the votes. But in order to deserve more than 3 slots, so that the slate slots get "rounded down" to 1, the non-slaters only need 3 times the votes.

2. "(except that we'd be facing more slate candidate)" — that's just wrong. If slate voters tried to "take advantage of" lifting the limit on votes per ballot — say, by running a slate of 10 instead of a slate of 5 — they'd still get exactly the same number slots as with the slate of 5. So there really is no downside to encouraging broader votes.

...

I understand the resistance to the 15-candidate longlist. A proposal like that really would skew people's judgement, and I can see how that would make people hear the quacking of a counter-slate. But the "everything with 2 nominations" longlist would not skew people's judgement, it would just broaden it. It has really nothing in common with the idea of counter-slates.

How many extra votes would it encourage? Some. And since those extra votes would be using judgment as unbiased as the current votes, this is only to the good. As I've said, I think this could reasonably be expected to help keep the largest slates to getting one or two fewer slots, thus ensuring several competitive alternatives among the nominees. That is significant and I think worthwhile.

#924 ::: P J Evans ::: (view all by) ::: May 07, 2015, 10:55 AM:

920
Some people will nominate five in every category, some will nominate one in one, and the average is probably about 2 or 3, in about half of them.

You can try that, or you can figure that people do what they do, which is why there's a limit on how many you can list. (It's a system that grew, instead of being put into place completely planned, but it does generally work.)

#925 ::: Buddha Buck ::: (view all by) ::: May 07, 2015, 11:28 AM:

Jameson Quinn:

Would your fuzzy-match corrector match on entries with only 1 nomination? And can it be made to prefer canonical or otherwise accurate entries without frequent grooming by the administrators?

For instance, if I nominate "Wizard of Earthsea" and am offered the correction "the wizard of earthsea" (because that's what the previous nominator typed in) I'm likely to go "Oh, I forgot about the article, and the capitalization is wonky, but that's the book I mean" and hit "yes".

Of course the books title is "A Wizard of Earthsea", so I reinforced the erroneous nomination, and (if the long-list proposal is adopted) pushed the mistake onto the long-list.

#926 ::: Pfusand ::: (view all by) ::: May 07, 2015, 11:57 AM:

On a different part of the SDV-LPE, now that we have real data:

I like the idea of returning a fractional point to an individual's vote for an item that has been eliminated in a category . I don't like the idea of returning a fractional point to an individual's vote for an item that has been put on the final ballot.

(I may have misunderstood that part.)

Would it be possible to run the data with fractional points not being re-distributed after a work goes on the final ballot? Then, if it doesn't change things, put in 20% slate ballots and run it again?

#927 ::: Jameson Quinn ::: (view all by) ::: May 07, 2015, 12:12 PM:

@926: I think you've misunderstood. In SDV-LPE, works are never "put on the final ballot". Whatever's left over at the end is the final ballot, but "put on the final ballot" is not an explicit step in the process.

@925: I believe that there are good solutions to that problem, but they should be found by programmers, not written into the rules by rules-lawyers. For instance: when you get a fuzzy match "hit", you are asked "Which spelling is better?"; and then, there's also a small button for "something is misspelled in the list", which lets people vote on which spelling is best. It wouldn't take more than 3 votes in almost all cases; there might be the occasional fight over something like "Star Trek: Into Darkness", but in that case, either of the top-voted titles would be acceptable.

But in the end, if the programmers decided not to have voting because it would be too hard and the occasional misspelling is OK, then fine, let them decide that.

@924: Right, some people nominate 5, or 3, or 1, and so the current system seems to have worked OK most of the time. But as long as voters are using their honest judgment, if some nominate 7, it's a purely good thing. I can see why you might not expect allowing more to improve things by very much, but I really can't see why you're opposing the change, unless you're just anti-change in general.

#928 ::: P J Evans ::: (view all by) ::: May 07, 2015, 12:29 PM:

927
The number is limited to keep it manageable, as well as to make people think about choices. Increasing the number of slots-per-category makes it harder to administer - that's more work.
(There isn't any reason to do that, anyway: if most people aren't using all five, increasing it to six or seven isn't going to be an improvement. Number of nominators is not related to how many slots are available.)

#929 ::: Duncan J Macdonald ::: (view all by) ::: May 07, 2015, 12:41 PM:

Jameson Quinn @890:

First off, thanks for responding. I'm going to argue back at you, but I respect that you're discussing in good faith, and that there are probably other people who feel like you.
Roger that. I may or may not be an outlier in the data, but as you say, there are probably others.

I will attempt to keep the flow of discussion going without having to retype/cut'n'paste large chunks of text by using boiled-down headers. I appologize in advance if that becomes confusing.

Comprehensive list of all eligible works
That would not be a duck, as it encompasses the entire space of possible nominatible works. Likewise any comprehensive list broken down by category. Why isn't that a duck? Because no known ballots have been cast yet to begin the filtering process.

We want to encourage people to vote broadly but honestly
Here is the first sticking point that I can see. I do not want to encourage people to vote broadly -- just honestly. To me, honestly means that I've read/seen the work in question to the extent that I need to make a decision on whether I want to nominate it. Added to this is the fact that I (and everyone else) have a Life(tm) and there exists less free time to read/view the current limit of 5 works per category than is needed to honestly judge them.

Adding new nominations and possibly changing current nominations
There are other reasons to change one's mind than strategic voting. My to-be-read/viewed pile may or may not include eligible works that I haven't gotten to when nominations open, but that I do get to before nominations close. Since any ballot includes an intrinsic ranking -- the order in which I place them -- that order is subject to change as I evaluate new eligible works.

List of nominated works, two nominations minimum
I believe that such a list is ordered -- by having at least two nominations. Regardless, this is where the duck starts waddling. I feel that any winnowing of the set of nominatible works by anyone who is not me* impinges on the honestly judge requirement.

Why is this different from the final short-list voting ballot? Because that final ballot is made up of nominations made by people who honestly evaluated the works they read/viewed.

Is a non-ordered list of works with a minumum of two nominations closer to the set of all possible nominatible works, or closer to a defined slate?
I don't have the numbers readily at hand, but in all the runs that have been done on the data, what is the ratio of individula nominated works to the set of all possible nominatible works? My gut** says that the number nominated is far small than the number eligible, hence my likening of the long-list to a slate.***

-----

* Also known as "other people"
** which has unfortunatley grown larger over the years
*** On a related note, I am one of those contrarians who, when presented with a list of "Other people who bought X also bought Q,R,S,T,U, and Y" by websites named after large South American Rivers instantly refuse to buy any of them, regardless of a pre-existing need. Irrational, but true non-the-less

#930 ::: P J Evans ::: (view all by) ::: May 07, 2015, 01:13 PM:

929
The one-or-two-nominations list is interesting, just because of what shows up on it. Not everything is eligible, though (in 1984, someone nominated Dune).

#931 ::: J Thomas ::: (view all by) ::: May 07, 2015, 01:43 PM:

Let's take a tough case: in one of the shorter fiction categories, a slate of 300 divides into 75 votes each for 4 separate candidates, and each slate voter also votes for 0-2 other unique, completely nonviable candidates.

That's definitely a tough case. For short stories in 2013, there were 662 ballots voting for 568 different short stories. 35.8% of those ballots included one of the eventual top 5. (They only put 3 on the final ballot because the others just didn't have enough votes.) The average nominator had 3 votes on his ballot.

We're talking about a disciplined army of 300 attacking a totally disorganized group of a little more than twice their number. Of course it's a slaughter.

If it had happened that way in 2013, the top nonslate short story would have won because it had 107 votes. To get a second win you'd need to double the 2nd place winner's 38. To get three, a bit more than double.

Leaving them uncoordinated, you can't double 3 votes per ballot to 6 votes per ballot without changing the rule that limits it to 5! Worse, if SDV-LPE passes, your ballots with 6 will tend to eliminate each other while the slate ballots with 1 stand aloof.

Maybe with a voting system that penalizes bullet voting? But then they go back to slates....

At some point you just have to accept defeat. An organized force will usually beat a group twice the size that refuses any attempt to organize.

#932 ::: Jameson Quinn ::: (view all by) ::: May 07, 2015, 03:00 PM:

Jameson Quinn @890:

Thanks for responding again. Good faith, etc.

Comprehensive list of all eligible works That would not be a duck, as it encompasses the entire space of possible nominatible works. Likewise any comprehensive list broken down by category. Why isn't that a duck? Because no known ballots have been cast yet to begin the filtering process.

OK. So, let's try to think of a way to create a list of all eligible works. First, you have to make sure only valid voters can add to the list, to prevent spam. Second, you have to make sure they're right about it actually being eligible, and that they have the spelling right, and similar concerns of accuracy — better wait until two of them agree on something. Third, these people have lives, so insofar as making the list can not be a burdensome task, that's good; ideally, it would be an offshoot of something they're doing already.

I'm sure you can see where I'm going with this. The list of works that have two votes is the closest to a list of all eligible works that's feasible. That's exactly why I proposed using it.

We want to encourage people to vote broadly but honestly Here is the first sticking point that I can see. I do not want to encourage people to vote broadly -- just honestly. To me, honestly means that I've read/seen the work in question to the extent that I need to make a decision on whether I want to nominate it. Added to this is the fact that I (and everyone else) have a Life(tm) and there exists less free time to read/view the current limit of 5 works per category than is needed to honestly judge them.

If you don't want to vote for more candidates than you are already, that's perfectly OK. But why wouldn't you want to encourage others to do so, if they're amenable? Remember, the more people vote broadly, the better the system can resist slates.

Adding new nominations and possibly changing current nominations There are other reasons to change one's mind than strategic voting. My to-be-read/viewed pile may or may not include eligible works that I haven't gotten to when nominations open, but that I do get to before nominations close. Since any ballot includes an intrinsic ranking -- the order in which I place them -- that order is subject to change as I evaluate new eligible works.

So you're saying you might think, "I voted for X earlier, but now that I've read Y and Z, I realize that there are some especially good options this year, and in such a tough field, I don't think that X really makes the grade."

I'd respond: if X is good enough to vote for, it's good enough to vote for. Yes, Y and Z may be better. But if you trust the other voters, there's a good chance that their votes will help Y and Z beat X. And if that doesn't happen, then Y and Z probably wouldn't have made it anyway, so in that case, your reasons for dropping the vote for X turned out to be invalid. So, you should just leave your vote for X intact, and add Y and Z. No problem.

List of nominated works, two nominations minimum I believe that such a list is ordered -- by having at least two nominations. Regardless, this is where the duck starts waddling. I feel that any winnowing of the set of nominatible works by anyone who is not me* impinges on the honestly judge requirement.

But if you hate the list so much, you can just not look at it. In any case, you're encouraged to vote for stuff that's not on the list, if you feel it deserves it. The only reasons the list exists are:
1. To help remind you of what's out there.
2. To make it easier to vote.
3. To help avoid typos.

It's not about "only vote for what's on the list". At all.

Why is this different from the final short-list voting ballot? Because that final ballot is made up of nominations made by people who honestly evaluated the works they read/viewed.

Um.... that's true here too. All votes should be based on an honest reading and evaluation.

Is a non-ordered list of works with a minumum of two nominations closer to the set of all possible nominatible works, or closer to a defined slate? I don't have the numbers readily at hand, but in all the runs that have been done on the data, what is the ratio of individula nominated works to the set of all possible nominatible works? My gut** says that the number nominated is far small than the number eligible, hence my likening of the long-list to a slate.***

There are probably thousands of "eligible" works — that is, works that would not be disqualified by the judges if by some miracle they got enough votes to be nominated. So yes, a list of dozens is not close to that number. But it has more in common with "the biggest clean list of things you might vote for that we can practically make" than with "here's a list of works so you can vote for all of them so that we our secret cabal can take over the world."

#933 ::: JonW ::: (view all by) ::: May 07, 2015, 05:24 PM:

felice@889,
None of us has any solid basis for quantitative prediction as to how people would react to mid-voting publication of a longlist; we’re reduced to pulling numbers out of the air. (That’s not a good recommendation for a proposal, natch.) Nonetheless, here’s my best-case analysis: The most likely folks to cast additional votes are the people with no favorites in the top 15. Not all of them will cast additional votes, though. Some of them may never see the longlist; some may not be interested in voting for anything in the top 15; some may not even understand that they can cast additional votes, or why it would be desirable to do so. Let’s say — and from my standpoint this is a high-end estimate — that 2/3 of them each cast 1.5 additional votes (on average) after publication of the longlist. So on 1984 best-novel numbers, that’s 70 additional votes. Then there are the people who do have at least one favorite in the top 15. They’re disincentivized from casting additional votes, because additional noms from them compete against the works they’ve already nominated. But let’s say that half of them are either reminded of one other work they really liked, or are inspired to read additional works and find one more they really like. So that’s another 185 votes, for a total of 255. In 1984 the 3d, 4th and 5th place works each got 7% of the total nominations for top-15 works, so we can figure on these assumptions that all this would boost those works by .07*255 = 18 votes. Maybe some of my assumptions are still too low, but that's the ballpark I figure we're playing in. It certainly doesn't double the number of votes, and it's very little compared to SDV-LPE, which raises the number of disciplined members needed for a slate taking three slots well into the three figures.

#934 ::: J Thomas ::: (view all by) ::: May 07, 2015, 08:18 PM:

@933 JonW

Maybe some of my assumptions are still too low, but that's the ballpark I figure we're playing in.

Consider how it worked this year. Last year people saw the slate was ineffective. This year suddenly they found out it had won a whole lot.

If there had been a longlist, they could have seen the five slate items on each longlist and seen that in some categories there were only three nonslate nominations because not enough had gotten 5%. That would have been a wakeup call. There might have been a significant number of votes for the 3-10 nonslate items.

We can argue that this amounts to another slate which is 100% as bad as the original slate. I figure it's no more than 50% as bad as the original slate, and it only happens when people get upset about the slate early enough to do something about it. It isn't a slate that happens every year, and it doesn't happen just because people hear that somebody else is running a slate, before they find out whether it's serious or not.

Maybe we should be careful not to fight fire with fire, careful not to do anything to get people to vote for nominees that can win. This looks to me like a potentially effective approach to blocking slates, but if it's too evil to use we can still try other approaches. If we find out early that it is very unlikely to be voted in then it's better not to propose it at the business meeting. I'm not sure how to find that out, beyond look for more people who will be at the business meeting to find out what they think.

#935 ::: P J Evans ::: (view all by) ::: May 07, 2015, 08:24 PM:

934
We're not expecting perfection, but something that's workable. No system is ungameable. (Or: it's impossible to idiot-proof things, because idiots are so ingenious.)

#936 ::: J Thomas ::: (view all by) ::: May 07, 2015, 08:27 PM:

@926 Pfusand

On a different part of the SDV-LPE, now that we have real data:

....

(I may have misunderstood that part.)

Yes, no shame in that. Here is an attempt at an explanation how it works. I would appreciate it if you look at it and say how clear it is, and maybe give suggestions to make it clearer.

SDV-LPE is like a medieval tournament. We do pairwise jousts, and the loser is eliminated, and that happens over and over until there are only five knights standing.
The winner of each joust is the one with the most votes. Having the most votes is like being strong and well-trained, it means you win against your opponent.
The jousts happen one at a time, we finish one before we start the next. Here is how we pick their order:
For each knight, we look at his ballots. We count 1 point for every ballot that has only his name on it, half a point for every ballot he shares with another surviving knight, down to one fifth of a point for a ballot he shares with four others. After the points are added up, the two knights with the lowest scores are chosen to battle.
It's kind of like the weakest knight fights the second-weakest knight, and then the third-weakest and so on until he loses. But not quite! Because the score is changed by the shared votes.
Say that one knight has all his ballots split five ways. He's likely to fight knights that are 1/5 his strength, and he'll cream them.
But every time he beats a knight that was on some of his ballots, he will share those ballots with fewer other knights and his chance to miss the next joust goes up. (And of course everybody else whose ballots had that knight on them watches their scores go up too.) Once he clobbers enough of the knights on his ballots then he can take a rest.
This is exactly how the system works. We figure the scores this way, and the lowest scores fight, and the highest votes win.
Think of the five strongest knights. The strongest one, Sir Absolute, can clobber anybody. His voters shared the vote with lots of people, so he's down there among the also-rans, beating anybody he comes up against.
And the second-strongest knight, Sir Brawny, can clobber anybody but the strongest. He's down there too, clobbering people, but if he ever comes up against Sir Absolute he's had it.
The third knight, Sir Claymore, is clobbering everybody but Absolute and Brawny. And so on.
IF -- through the luck of the draw -- Absolute, Brawny, Claymore, Destrier, and Exploder fight each other, then others can slip in and win. Otherwise, no chance.
A knight still has to be pretty good to win, even if his strategy is to win by avoiding combat. If he isn't strong enough then his score will be low and he will have to fight and lose. But if Sir Sneaky's voters vote only for him, he has a chance to avoid combat with knights who are nearly 5 times as strong. Maybe they'll knock each other out before they get to him. Just maybe.
And if he's more than half as strong as Sir Exploder, he has a chance that Sir Exploder will still be mostly fighting weaker knights when Sir Absolute eliminates him. Sir Sneaky might squeak by, if the luck of the draw never forces him to fight anybody stronger than he is.
And he doesn't *have* to have voters who vote only for him. He could luck out and have voters who load up his ballots with such weaklings that they all get eliminated before he's in danger. He gets all the advantages of voters who really really want him, without actually having such voters.
When you get right down to it, it's the luck of the draw. We can expect that usually the strongest will still win, the way they used to. But every now and then Sir Sneaky has his day.
What if Sir Awful, Sir Brute, Sir Crazy, Sir Dreadful, and Sir Excrement all have more votes than anybody else, but they share all the same ballots? They'll be down there fighting the weaklings, getting plenty of opportunities to possibly fight each other. Probably two or three -- or even four-- of them will drop, even if they are the strongest.
Nobody can be sure how the tournament will go. Nobody knows who each knight will be matched up against, or (unless he's weak) whether he will have to fight at all. It isn't really chaos, it's completely determined by the combination of all the votes, but it looks like chaos to anybody who watches. Still, the strong tend to win, the sneaky have a chance, and the slates fight themselves so fewer of them win.

Sasquan
John Lorentz
Ruth Sachter
Ronald Oakes web application

Midamericon
Dave McCarty
Will Frank

#937 ::: Steven desJardins ::: (view all by) ::: May 08, 2015, 12:34 AM:

That explanation looks dreadful to me: it doesn't clearly explain the mechanics, and it emphasizes the role chance plays in the outcome. I'd take out the reference to pairwise jousts and replace it with a simple statement that the works with the least support are gradually eliminated. (Anyone who really wants to know the mechanics can read the detailed proposal.)

You want to explain SDV in a way that's comprehensible and emphasizes fairness. Otherwise, people will vote it down without even making the attempt to understand how it works. (Remember, most people who attend the business meeting are probably seeing these proposals for the first time there. You don't have a lot of time to win them over.)

I'd say something like, "Everyone gets a single vote in each category, divided among everything they nominate. The works with the least support are gradually eliminated, and the votes for the works that are eliminated are redistributed to whatever's left on those ballots. So if you nominated A B C D E, each of those would get 1/5 of a vote. If E was eliminated, then A B C and D would each get 1/4 of a vote in the next stage of counting. If C is eliminated next, A B and D would each get 1/3 of a vote. The administrators continue eliminating works with the least support and redistributing votes until there are only five works left, which are the final ballot.

"The advantage of this method is that everyone has an equal say in determining what gets on the final ballot. A cohesive minority voting in concert can easily get a single work on the ballot, or perhaps even two or three, but they can't easily crowd everything else off the ballot.

"As with the STV system used on the final Hugo ballot, it is theoretically possible for the order in which works are eliminated to affect the final outcome, but the system has been designed to eliminate as much as possible the role played by chance."

#938 ::: P J Evans ::: (view all by) ::: May 08, 2015, 01:06 AM:

937
That's a good explanation. I think someone much further upthread explained it also, in about those terms, and it's how I understand the idea.

It isn't overly technical, and it gets the idea across, as well as being actually not a huge change (if you're nominating, you don't have to change the way you do it).

#939 ::: Cubist ::: (view all by) ::: May 08, 2015, 04:56 AM:

J Thomas @936: Again with the "medieval jousting" metaphor? Even after it's been explained to you how that metaphor really isn't anything close to an accurate way to portray the voting? Give it a rest, dude. Please.

#940 ::: J Thomas ::: (view all by) ::: May 08, 2015, 06:00 AM:

@939 Cubist

Again with the "medieval jousting" metaphor? Even after it's been explained to you how that metaphor really isn't anything close to an accurate way to portray the voting? Give it a rest, dude. Please.

Cubist, I do not understand your objection. The claim that this was not an accurate way to portray it was simply wrong. This describes exactly how it works, except that it leaves out what happens in the case of ties. If you think this is not an accurate description, then you do not understand how it works and you have no business deciding how accurate a description it is.

@937 Steven desJardins

That explanation looks dreadful to me: it doesn't clearly explain the mechanics, and it emphasizes the role chance plays in the outcome.

I don't understand why you say it doesn't clearly explain the mechanics, since it describes the mechanics precisely. It points out that chance actually plays no role in the outcome, except for the chance that voters will present precisely the ballots they do.

(Remember, most people who attend the business meeting are probably seeing these proposals for the first time there. You don't have a lot of time to win them over.)

http://www.thehugoawards.org/changing-the-rules/
"Realistically, if you are not already a regular participant in the WSFS Business Meeting, it will be very helpful for you to bring your proposal up informally as early as possible. Surprises at Worldcon, especially controversial or complicated surprises, are essentially never adopted."

Probably, discussion at the meeting will be limited to no more than 20 minutes if it is not rejected at first sight. You will want as many attendees as possible to be already familiar with it.

I'd say something like, "Everyone gets a single vote in each category, divided among everything they nominate. The works with the least support are gradually eliminated, and the votes for the works that are eliminated are redistributed to whatever's left on those ballots. So if you nominated A B C D E, each of those would get 1/5 of a vote. If E was eliminated, then A B C and D would each get 1/4 of a vote in the next stage of counting. If C is eliminated next, A B and D would each get 1/3 of a vote. The administrators continue eliminating works with the least support and redistributing votes until there are only five works left, which are the final ballot.

That is in fact wrong, though it has the advantage of being short.

"The advantage of this method is that everyone has an equal say in determining what gets on the final ballot.

I have absolutely no idea what that means, but it sounds good.

A cohesive minority voting in concert can easily get a single work on the ballot, or perhaps even two or three, but they can't easily crowd everything else off the ballot.

Yes, that's good.

"As with the STV system used on the final Hugo ballot, it is theoretically possible for the order in which works are eliminated to affect the final outcome, but the system has been designed to eliminate as much as possible the role played by chance."

There is no role for chance at all, as I explained. It only looks like chance to voters because they can't see the big picture.

If you vote for the work with the second most number of votes and four others that each get more than one fifth the votes of the popular one, it's possible (but unlikely) that the popular work will be eliminated immediately. You have no way to know, because you don't know how many votes any of them will get, and you don't know how crowded each ballot will be.

Saying possible but unlikely is a description of what you don't know -- in reality, either it certainly will happen or it cannot happen because there is no chance involved once the votes are in.

I will rewrite that part to make it more obvious that there is no chance involved.

I have gotten objections from three people and approval from none. I am more concerned about the latter since the objections are uninformed.

#941 ::: Jameson Quinn ::: (view all by) ::: May 08, 2015, 09:29 AM:

If the metaphor is jousting, then the jousts are between the two knights with the fewest ladies' handkerchiefs tied to their pennant. The winner of each joust is the one with the most ladies cheering for them. Each joust is between just two knights, so the ladies can cheer for whichever one they prefer, at full strength; but each lady has only 60 handkerchiefs, which they have to divide among all their favorite knights who are still in the tournament. They're also not allowed to give one knight more handkerchiefs to Every time a knight is defeated, the ladies can redistribute the handkerchiefs to other knights. Rainbow knights with lots of colors of handkerchiefs will gradually get more of them as the other weaker knights are defeated, so a popular enough rainbow knight might make it into the top 5 without ever jousting. Monochrome knights who share mostly the same color handkerchiefs (from the same family of ladies) might occasionally get an extra handkerchief or two, but those extras will not be proportional to the number they already have, so they will tend to fall to the bottom of the lists, and have to do a lot of jousting. Eventually, two knights of the same color will have to face each other, and then one will be eliminated. Thus, the winners will tend to represent all colors of handkerchiefs; that is, there will probably be several rainbow knights and at most one or two monochrome knights.

....

OK, that was an incredibly strained metaphor, especially the part about "60 handkerchiefs". But I hope it was helpful in understanding what's going on.

#942 ::: Jameson Quinn ::: (view all by) ::: May 08, 2015, 09:31 AM:

@940: "I have gotten objections from three people and approval from none. I am more concerned about the latter since the objections are uninformed."

Please do not interpret my addition to your metaphor as "approval". I think you're veering back towards the territory Teresa warned you against.

#943 ::: Keith "Kilo" Watt ::: (view all by) ::: May 08, 2015, 09:47 AM:

JT@940:

We talked about this as some length off-list, and I tried to explain why your metaphor is fundamentally flawed and at best misleading. As I mentioned then, the only purpose of an analogy is its explanatory power, and yours actually has none, or at least not enough to overcome its many, many limitations. You say you've received objections from three people and approval from none. Maybe instead of dismissing those objections as you have done, that should tell you something? I am not an expert on SDV-LPE or voting systems in general by any stretch of the imagination, but I don't think it's accurate to say I'm uninformed.

As an intellectual exercise, I think it's great that you want to explore this for yourself, but you have to understand that I think the chances of using such a flawed analogy to help pass the proposal -- when it is much more likely to make it fail (which I'm hoping is not your actual intent) due to confusion and misleading explanations -- is near zero, as far as I'm concerned at least.

As I told you off-list, if you want to come up with a different analogy, that's great, but frankly I don't see the usefulness of such an analogy as anything other than a distraction from actually trying to get this proposal done.

Kilo

#944 ::: Buddha Buck ::: (view all by) ::: May 08, 2015, 10:33 AM:

J Thomas @940:

I'm inclined to say that if three different people have misunderstood what you are saying, then perhaps the communication problem isn't with them.

Based on my understanding of SDV-LPE, your "jousting" metaphor accurately captures the mechanics, but apparently communicates them poorly. Perhaps there are other aspects of the "jousting" metaphor which are distracting?

For instance, in an actual joust (and any sporting event, really), the skills of the competitors matter, and there still a tremendous amount of luck in the outcome of a particular bout. You also emphasized repeatedly how a knight could benefit from the "luck of the draw", or "luck out". Even though the SDV-LPE process is completely deterministic (modulo certain tie-breaking methods), your metaphor explicitly and implicitly highlights luck and randomness.

Similarly, in SDV-LPE, the elimination of a candidate is based solely on the number of votes the candidate received (and that of the other candidate up for elimination), not on any measure of value of the candidate itself. In a joust (or other sporting event), the outcome is ideally determined by the skill of the competitors, and not by the number of fans for each side. Although you state that the stronger knight will always beat the weaker knight, you don't emphasise that knight strength is solely determined by the size of the fan-base, and the sports-metaphor leads people to think the elimination "bout" is based on the merit of the works.

#945 ::: Duncan J Macdonald ::: (view all by) ::: May 08, 2015, 10:42 AM:

Jameson Quinn @932:

I am beginning to think that we are speaking past each other, not to each other.

I reject the idea of such a list of eligible works created out of nominations. By its very nature it is a severely restrictive list -- one created by other people -- and is therefore indistinguishable from a slate.

@932 If you don't want to vote for more candidates than you are already, that's perfectly OK. But why wouldn't you want to encourage others to do so, if they're amenable? Remember, the more people vote broadly, the better the system can resist slates.

Because I view the list that you propose to provide as a slate. And slates reduce freedom of choice.


@932 So you're saying you might think, "I voted for X earlier, but now that I've read Y and Z, I realize that there are some especially good options this year, and in such a tough field, I don't think that X really makes the grade."
Correct. And if I read/view enough additional eligible works that X slides right out of the top five that I can nominate, then it goes away. Please note that at no time during my vetting process do I ask the question: "Well, what do other nominators think of this?" What other people think of any given work is not germane to my evaluation of it -- to the point that if I were to think, "Hey, maybe Fred and Joey would really like this, if they get around to reading/viewing it, so I'll just nominate it for them." then my nomination is no longer honest.

TL;DR -- Strategic Voting Isn't Honest Voting


@932 But if you hate the list so much, you can just not look at it. In any case, you're encouraged to vote for stuff that's not on the list, if you feel it deserves it. The only reasons the list exists are:
1. To help remind you of what's out there.
2. To make it easier to vote.
3. To help avoid typos.

Your number 2 above is the killer. It's not about "only vote for what's on the list". At all. But it can be construed that way. By presenting a list "To make it easier to vote" you are artificially prescribing the size of the nominatible field of works.

You see your list as a reasonable reminder of works that might be something someone somewhere might also like to consider. I see it as a slate and anathema to honest nomination.

#946 ::: J Thomas ::: (view all by) ::: May 08, 2015, 11:53 AM:

SDV-LPE is a voting system that uses sequential pairwise elimination.

There are two repeated steps. In the first step, two or more contestants with the smallest scores are selected. In the second step, one or more that have the fewest votes are eliminated. This is repeated until there are only five left. (Or with ties at the last round, perhaps more.)

The scores are determined as follows: for each ballot, the surviving contestants on that ballot each get a fraction of a point. If you vote for two, they each get half a point. If you vote for five, they each get one fifth of a point. So a contestant may have a low score even with many votes, if those votes are shared with many others. However, note that in the elimination step, that contestant gets the full count for every vote. So if your ballot has votes for five works, and for one of your works every other ballot also has votes for four other works, it could have the lowest score when another work that has one fifth as many votes has the second lowest score. Yours will win the elimination because it has more votes.

And if the loser shared some of your work's ballots? Those ballots will now have one less surviving competitor on them. Your work's score will go up. It may no longer be the lowest.

This gives an important different result from just taking the five with the most votes. (If we chose the scores by the number of votes, instead of the fractional number, we would in fact wind up with the five with the most votes.) The difference is that works which have a lot of votes but low scores, because their ballots have votes for other works on them, may wind up at the bottom and get eliminated by other works that have even more votes but low scores. So sometimes the works with the top five votes will not win.

This is particularly true for slates. Imagine a slate with five works. Each of the five has more votes than any single work that isn't on the slate. But their scores will be only a fifth as big as their votes. So first the works with very low scores will be removed. Then the members of the slate will compete against each other, and at least one will be removed. When that happens their scores go up because all of their ballots included the one that's gone. Works with scores less than a quarter the number of slate votes will be removed, and then it is the slate's turn again. They will lose one or more, and their scores will go up. And so on.

If there are four nonslate works whose scores are higher than half of the slate votes, then only one slate work will win. If there are three works whose scores are higher than one third of the slate votes, then only two slate works will win. If there are two works whose scores are higher than a quarter of the slate votes, then only three slate works will win. And if there is even one work whose score is higher than one fifth of the slate votes, then that one will be among the winners. A slate does no better than it would if it split into five equal groups that each voted for one single work. With this system slates are useless.

There are always trade-offs. Suppose that you think several candidates are good enough to get a Hugo, but you want one of them the most. Depending on the details, you do better to vote only for that one. This is mathematically proven to be true in any system which does not have any randomness in it. (This system has no randomness, any collection of ballots will come out the same way every time.) The penalty for voting for more than one is a little bigger with SDV-LPE than some others. There is less penalty if your favorite is more popular than the other work, than when it is less popular. In exchange for this small flaw, slates lose their power.

#947 ::: P J Evans ::: (view all by) ::: May 08, 2015, 11:53 AM:

945
And I suspect that they're considering the longlist proposal without remembering that there would have to be one for every category, which multiples the work for the administrators by quite a bit.
If it takes an hour a week to maintain one, how many hours does it take for seventeen, given that the people doing the work are probably also working full time as well as having a life?

#948 ::: J Thomas ::: (view all by) ::: May 08, 2015, 11:57 AM:

@944 Buddha Buck

I'm inclined to say that if three different people have misunderstood what you are saying, then perhaps the communication problem isn't with them.

In any group there are likely to be three people who criticize without reading. But when nobody liked it, that implies a problem.

Based on my understanding of SDV-LPE, your "jousting" metaphor accurately captures the mechanics,

Thank you!

but apparently communicates them poorly.

It looks like an esthetic issue, so I tried again. Esthetics is important.

Thank you for the feedback.

#949 ::: P J Evans ::: (view all by) ::: May 08, 2015, 11:58 AM:

946
That's still an explanation that isn't going to work. You're not explaining it to a class in voting systems; you're talking to a group of people between roughly 12 and 80 who are interested in how much it's going to fuck up the system they're familiar with, and will be voting on it with a simple yes or no.

Too long, too complicated, and doesn't clarify what the people at the business meeting want to know. Look at the one Steve wrote: that's the kind of thing that people can follow.

#950 ::: Cubist ::: (view all by) ::: May 08, 2015, 12:00 PM:

J Thomas, your "jousting" metaphor portrays the voting process as a series of one-against-one battles. However, the voting process is actually more like one big free-for-all in which everybody is competing against everybody, all at the same time. 'Nuff Said?

#951 ::: J Thomas ::: (view all by) ::: May 08, 2015, 12:26 PM:

@945 Duncan J Macdonald

I reject the idea of such a list of eligible works created out of nominations. By its very nature it is a severely restrictive list -- one created by other people -- and is therefore indistinguishable from a slate.

Oh, this is unfortunate.

I figured that this would be hardly any help against slates, or as a slate, but it might cut down the work for administrators. Or maybe not. But the Hugo administrators could decide whether it would.

At some point we'd get somebody who will actually be doing the work to look at it and see what he thinks, and see if he has suggestions. If he likes it he could ask for volunteers to do coding or something. And at the business meeting, if he stood up and said "I'm the one who's going to be doing the work and I approve (or disapprove)" that would carry a whole lot of weight. If he said he disapproved before it got that far I sure wouldn't push it any farther.

But now the slate stuff has gotten intertwined in it. Something that didn't need to be political, that could have been a sheer administrative thing, has turned into....

Are we agreed that a list of things that have been nominated so far doesn't encourage people to vote more for one thing on the list than for another? At least there's that.

And if you want to nominate something and it isn't on the list, you can put it on the list. You ought to put it on the list.

If I had a list like that, and there were a lot of names on it, I'd probably think of what I wanted to nominate and then do a search on author's last name, or some part of the title, and if it was there then I'd vote for it. If not then I'd nominate it fresh.

I doubt the list would persuade me not to nominate what I wanted to, and it probably wouldn't get me to vote for something I wouldn't otherwise. But it might have that effect for some people. It seems to me like a small effect, but I can understand that you wouldn't want to have any of that at all.

It's a side issue for me. I want fair things that will help stop slates. The longlist I thought would help, is already something you oppose strongly enough I don't have a lot of hope for it. This one I don't much care about, except that if you're against even this then I can't imagine anything at all in that direction that you'd accept.

For this one, if the guys who actually do the work decide they want it, would you listen to them say why? If it turns out to be valuable to them, could that weigh in the balance of your concerns?

#952 ::: J Thomas ::: (view all by) ::: May 08, 2015, 12:37 PM:

@950 Cubist

J Thomas, your "jousting" metaphor portrays the voting process as a series of one-against-one battles. However, the voting process is actually more like one big free-for-all in which everybody is competing against everybody, all at the same time. 'Nuff Said?

Cubist, you are simply wrong. The SDV-LPE voting process is in fact a series of one-against-one battles. Except occasionally when due to ties in the scoring, several are chosen to compete at once.

I don't want to be rude, but the truth is that you simply do not know what you are talking about.

I would not usually be so blunt, but I've been getting a whole lot of negative reinforcement for awhile now and I don't feel cheerful.

#953 ::: J Thomas ::: (view all by) ::: May 08, 2015, 12:49 PM:

949

That's still an explanation that isn't going to work. You're not explaining it to a class in voting systems; you're talking to a group of people between roughly 12 and 80 who are interested in how much it's going to fuck up the system they're familiar with, and will be voting on it with a simple yes or no.

Too long, too complicated, and doesn't clarify what the people at the business meeting want to know. Look at the one Steve wrote: that's the kind of thing that people can follow.

I think I see. OK, here's the revised version:


SDV-LPE is a voting system designed and extensively tested by Jameson Quinn, an expert on voting systems. It will reduce the effect of a slate of five nominations to the same effect we would get with five smaller sets of votes with one nomination each.

It requires no change in how anyone votes. If you have been voting for the five or fewer that you think are the best Hugo nominations, then keep doing that with no change.

SDV-LPE is easy to code and it does not require excessive computing resources.

The gory details are available in Appendix A, for anyone who is that interested.

#954 ::: Fade Manley ::: (view all by) ::: May 08, 2015, 12:59 PM:

J Thomas @953: I'm afraid the most recent explanation wouldn't work at all for me, as a voter. It would, in fact, look a lot like an attempt to hide details I wouldn't like from me by assuring me that I don't need to examine them, with the side implication that since I wouldn't understand them, I shouldn't try. When it says it was extensively designed and tested by a single person, and notes he's an expert, my reaction is that someone who's clever with voting systems is trying to get something past the rest of us who aren't.

And I'm not clever with voting systems! I'm really not. I barely understand a lot of what's being discussed in this thread. But I keep following along because I care about the results, and want to give feedback based on what a "not clever with voting systems" voter might react like.

Furthermore, "It will reduce the effect of a slate of five nominations to the same effect we would get with five smaller sets of votes with one nomination each" makes absolutely no sense to me. I don't even know what that's trying to convey. It says it reduces the effect of slates--good!--but says it does that by doing a thing I don't understand.

#955 ::: J Thomas ::: (view all by) ::: May 08, 2015, 01:12 PM:

@954 Fade Manley

Thank you, I'll try to do better. I'm not at all sure where to strike the balance between leaving out the details versus putting in just enough useless detail to be misleading. Maybe the details can go in the Appendix?

-----------

SDV-LPE is a voting system designed and extensively tested by Jameson Quinn, an expert on voting systems.

If there is a slate that under today's rules delivers 300 votes for each of 5 nominations, under SDV-LPE it will do no better than 5 nominations that get 60 votes each.

It requires no change in how anyone votes. If you have been voting for the five or fewer that you think are the best Hugo nominations, then keep doing that with no change.

SDV-LPE is easy to code and it does not require excessive computing resources.

The details are available in Appendix A, for anyone who is interested.

#956 ::: P J Evans ::: (view all by) ::: May 08, 2015, 01:37 PM:

Read the explanation in 937: that's the one that will sell it, or not.

(You're still explaining it as if you were talking to experts.)

#957 ::: Fade Manley ::: (view all by) ::: May 08, 2015, 01:38 PM:

J Thomas @955: I think you'll need to put into the main description a basic explanation of how it works, not just its effects. I'm not sure if there's any simple way to explain it; that is, I suppose, a great deal of the challenge there.

#958 ::: Keith "Kilo" Watt ::: (view all by) ::: May 08, 2015, 01:48 PM:

Writing voting explanations sounds like a much more fun game than grading these papers, so I'll take a turn. Maybe if we get enough contributions we can converge to something that makes sense to everyone. I do think that the "Commentary/FAQ style" that I used in my first proposal draft (see #458) is a good way to go, though.

Here's a first attempt:

Least Popular Elimination (formally called "single divisible vote with least popular eliminated" or SDV-LPE for short) is very simple and straightforward.


- You have one nomination "vote", which we'll call one "point" to avoid confusion.

- You can distribute that nomination "vote" among as many works as you feel are Hugo-worthy and it will get divided among them equally. So, if you nominate two works, each gets half a point, if you nominate three works, each gets one third of a point, etc.

- All the points for each work from all the ballots submitted are added together, and the two works that got the least number of points are compared with each other. One of these works is the least popular and will be eliminated.

- For those works that are eligible to be eliminated, we compare the total number of nominations they each received (that is, the total number of times that work appeared on anyone's nomination ballot). The work that received the fewest number of nominations is the least popular and now completely vanishes from the nomination process as though it never existed.

- We start over for the next round, and repeat the process, however, if one of your nominations was eliminated, you now have fewer works on your nomination ballot -- so each one gets more points since you aren't dividing your vote among as many works.


Why does this help? In the past we have strictly counted the number of nominations and the top five works were put on the final ballot. Because SF fandom typically nominates a diverse range of works, it was easy for an organized slate to make it so that no other works made the final ballot.

Using this sytem, fandom isn't penalized for nominating a wide variety of works. If you nominate something that ends up not having a chance to make the final ballot, then your remaining choices automatically get more of your support instead of just being wasted. In other words, you can safely nominate -anything- you feel is Hugo-worthy. If enough people agree with you, it will make the final ballot. If they don't, that's okay - when that work is eliminated, your other choices will have a greater chance of making the final ballot. In this way, by eliminating the least popular candidates, fandom slowly converges to a concensus as to which works should be voted on to be the final Hugo winner.

How does this eliminate slates? It doesn't, not completely -- nor should it, as slate proponents have as much right to representation as any other voter. However, by nominating a full slate of works, they have weakened their votes by spreading their points out among the five works on the slate. Since the rest of fandom is slowly increasing the points given to their choices, it's not possible for slates to control the -entire- ballot. It is very likely, however, that they will get some of the final ballots slots -- and again, this is entirely fair and appropriate, since a large number of people are supporting those works. In the end, it is the final ballot that will determine the winner -- and the voting system for the final winner is completely unchanged by this system. With this system, a significant fraction of the final ballot will have been determined by independent members of fandom.


Fade, I'm particularly interested in your criticism and suggestions...

Thanks,
Kilo

#959 ::: P J Evans ::: (view all by) ::: May 08, 2015, 02:57 PM:

958
Keith, that works for me.
It's not so technical that people will get lost in it, and it's clear that it's about changing the way people need to think about their choices. It's just another way of counting, one that will limit the effect of slates on the results.

#960 ::: Jameson Quinn ::: (view all by) ::: May 08, 2015, 03:01 PM:

OK, we're calling it LPE now, not SDV-LPE. That sounds fine.

Here's my attempt to explain it.

This system uses the same ballots as the current system; you just vote for all the works that you think may deserve a Hugo, up to a maximum of 5.¹

When counting the votes, the system eliminates one candidate at a time, until there are 5 left. At each step, it is designed to look for the two candidates who do the least to increase the "representativeness" (or "diversity") of the candidate pool, and eliminate whichever of those two has the fewest supporters.

Thus, when you add a candidate to your ballot, you are essentially doing two things. First, you have 1 "representativeness point" to spread among the candidates, to protect them from even being considered for elimination; and so adding a candidate to your ballot redistributes that point. This means that if a group of voters all vote for the same 5 works, their "points" will all be spread five ways, and so those 5 works will probably end up eliminating each other. But second, once two candidates are selected for possible elimination based on representativeness, your ballot will count fully for whichever of them you supported (if any). This means that, if you voted independently and not as part of a slate, it is unlikely that adding an additional candidate to your ballot will cause any of the candidates already on there to be eliminated if it wouldn't have been otherwise.²

In other words: While it is theoretically possible that you'd prefer the results of voting "narrowly" — for just one or two of your favorite candidates — rather than "broadly", in practice it is very likely to be safe to vote "broadly", and simply vote for all candidates whom you think may deserve a Hugo.³

This system has been tested against old Hugo voting data, and on novel scenarios constructed by "bootstrap" resampling that old data, by at least two people. Both of them found that, in the absence of slate voting, LPE would tend to give similar results to the current system; on average, over 4.5 of the 5 nominees are the same.

¹ This system would work just fine if you removed the maximum of 5 votes per ballot per category, but to avoid overcomplicating things, we have not included that in this proposal.

² It is possible to construct various artificial scenarios where adding a candidate X to your ballot ends up eliminating one of the candidates Y who was already there. However, all such scenarios are at best unlikely; there is not a big chance that a fraction of a point is going to make a difference. In fact, only two such scenarios are even remotely plausible.

Scenario 1 is when X is the one that beats Y. In this case, both X and Y must be just on borderline of getting elected, and the fractional point is not what makes a difference, but rather the lack of decisive support for Y in the showdown. Thus, this is essentially the same thing as what can already happen using the current system.

Scenario 2 is when adding X happens to move Y just below a set of slate candidates, so that the slate candidates eliminate Y just before they eliminate one of their own. Of course, hopefully, the fact that LPE resists slates would mean that slate voters wouldn't bother, so that this scenario wouldn't happen.

³ In fact, any possible voting system will sometimes reward voting strategy. If it doesn't reward "narrow" over "broad" voting, it must reward some strategy that's even more dishonest and potentially harmful. However, this system is designed to give no reward at all to (any feasible version of) dishonest voting, and the slimmest possible reward to (strategic but still "semi-honest") "narrow" voting.

#961 ::: J Thomas ::: (view all by) ::: May 08, 2015, 03:17 PM:

@959 P J Evans

Keith, that works for me.

Then I will consider my part in this done.

#962 ::: Seth Gordon ::: (view all by) ::: May 08, 2015, 03:38 PM:

I’m not sure I can write a complete description of LPE that would “sell” it to a skeptical Business Meeting voter, but let me suggest this as a lede:

“Under the LPE system, if a hundred people nominate the same novel for a Hugo, but have no other preferences in common, they will have more influence over the results than a hundred people who vote for an identical list of five nominees. This insulates the Hugo nomination process against both formal bloc-voting campaigns and informal log-rolling, so that each choice on the final Hugo ballot represents a large and diverse subset of fans.”

#963 ::: Cubist ::: (view all by) ::: May 08, 2015, 03:40 PM:

sez j thomas @952: "Cubist, you are simply wrong. The SDV-LPE voting process is in fact a series of one-against-one battles. Except occasionally when due to ties in the scoring, several are chosen to compete at once."
Rubbish. If SDV-LPE were a series of one-against-one battles, it would present the voters with a series of "do you prefer candidate X or candidate Y?" questions, with the results of past questions being used to determine which candidate-pairs will be presented in future questions. In reality, SDV-LPE presents every friggin' candidate, all at the same time, and asks the voters to register their preference among all the friggin' candidates at once. That's not a series of one-against-one battles; that's a grand melee free-for-all.

Given that you appear to be unshakably convinced that everybody else on the dance floor is out of step, there's a good chance that I won't bother to respond to any further verbiage from you on this topic.

#964 ::: Duncan J Macdonald ::: (view all by) ::: May 08, 2015, 03:42 PM:

J Thomas @951: Are we agreed that a list of things that have been nominated so far doesn't encourage people to vote more for one thing on the list than for another? At least there's that.

Umm, no. The presence of any list at all will (IMHO) skew some nominators to consider those works as already properly vetted and therefore safe to nominate on their own ballot without any additional effort on their part. You said as much with the following: If I had a list like that, and there were a lot of names on it, I'd probably think of what I wanted to nominate and then do a search on author's last name, or some part of the title, and if it was there then I'd vote for it.

I realize that you would have likely already read/viewed that particular work, and its presence on the list would merely be positive reinforcement that you were correct in your evaluation. I'm worried about the people who take the list as a reason to not do their own work.

For this one, if the guys who actually do the work decide they want it, would you listen to them say why? If it turns out to be valuable to them, could that weigh in the balance of your concerns?

I will listen. As a tool to be used to help the administrator's workload of determining eligibility and variant spellings, I can see utility. Out of the laboratory and into the wild is where the issues lie.

#965 ::: Pfusand ::: (view all by) ::: May 08, 2015, 03:42 PM:

I think it would be helpful to first-time readers of the system if you defined your terms first:

There are 17 "categories" on the Hugo ballot (e.g., Best Novel, Best Short Dramatic Presentation, John W. Campbell Award).

You are entitled to nominate up to 5 "works" in each category (novels, people, etc.).

Under the SDV-LPE system, as now, each nomination constitutes one "vote." The primary change is that you are given one "point" for each category.

If you nominate four works in a category, then you are considered to have cast four votes in that category, and each "vote" is considered to be one-fourth (1/4, 0.25) "point." If you nominate only one work, it is one vote and one point.

Elimination is done by adding all the "points" for each work nominated. The works with the two lowest totals of points are compared...

and we go from there.

#966 ::: Steven desJardins ::: (view all by) ::: May 08, 2015, 03:43 PM:

I think that's a pretty good explanation, Keith. I'd suggest adding an analogy to the ranked STV on the final Hugo ballot: as items are eliminated, their support is redistributed to other items on the ballot, in accordance with voters' expressed preferences. So, as with STV, it generally doesn't hurt to vote for something that you think isn't likely to win; once an item goes away, its points are transferred to the other, not-yet-eliminated items you also voted for.

#967 ::: P J Evans ::: (view all by) ::: May 08, 2015, 03:52 PM:

965
You don't need the quotes around category and works - those are the terms used (for the categories where the rockets aren't going to people: editors, artists, writers - and the Campbell Award).

#968 ::: Jameson Quinn ::: (view all by) ::: May 08, 2015, 03:54 PM:

My last comment was an attempt at KISS, so in this one I get to geek out with impractical theory. Not intended for the actual proposal.

Actually, LPE doesn't have to divide the same fixed voting power (that is, use D'Hondt divisors for dividing "points"). You could use Saint-Lagüe divisors, and it would still be proportional. You could even use exponential divisors, and while that technically wouldn't be proportional, it would probably be better for this use case— 1 slot to a 20% slate, 2 slots to 40%, but still 2 slots at 51% and getting 4 slots would require over 90%.

I realize that that would make the explanation harder, so I'm NOT proposing it seriously. But it would make the actual outcome better.

Extra-super-geeky: you could phrase a proposal like this in terms of a flat "transaction tax" of 0.1 votes on the amount of voting power you can give to each candidate. I'm sure phrasing it that way would _totally_ make it likely to pass, because nothing's more intuitive than death and taxes.

#969 ::: Fade Manley ::: (view all by) ::: May 08, 2015, 04:14 PM:

Keith "Kilo" Watt @958:

That...actually makes sense! I found it fairly easy to follow, and it looks fair. As explained, the only potentially inconvenient side-effect that springs to mind is that it makes me less likely to nominate as many works as usual; I'd often nominate all five slots in novel, and now I'm going to put more thought into nominating more than two, because they don't "count" for as much that way. But since having things fall off just makes my other noms count as if I had only listed them, it doesn't make me worried about nominating things other people don't like--or, conversely, about accidentally voting too many similar things as other people.

Jameson Quinn @960: I was following your explanation at first, but it lost me at:

"When counting the votes, the system eliminates one candidate at a time, until there are 5 left. At each step, it is designed to look for the two candidates who do the least to increase the "representativeness" (or "diversity") of the candidate pool, and eliminate whichever of those two has the fewest supporters."

I paused and went "How are things being eliminated?" first; then I realized it was explaining, and moved on. But it wasn't explaining how, it was explaining a goal--diversity--which makes me pause, because now I'm distracted from the mechanical explanation (how this works) by thoughts about the purpose of this voting system, and what "diversity" is, and who decides what it is, and so forth.

Then when you get to this part:

"This means that if a group of voters all vote for the same 5 works, their "points" will all be spread five ways, and so those 5 works will probably end up eliminating each other."

I don't understand how these things are eliminating each other! I was following the point-splitting, because it sounded like it was a "vote for what you REALLY want, or more if you must" explanation, but suddenly things are competing against things on their own ballot? Or other ballots? I don't get how these eliminate each other. It makes me worry that if I accidentally vote the same as a friend in a category, we're actually competing somehow, instead of voting for things we both like and equally supporting them. So now I'm confused and distracted again.

"In other words: While it is theoretically possible that you'd prefer the results of voting "narrowly" — for just one or two of your favorite candidates — rather than "broadly", in practice it is very likely to be safe to vote "broadly", and simply vote for all candidates whom you think may deserve a Hugo."

This ends up being sort of confusing and upsetting again. I hadn't been worried about the safety of nominating several things--I thought I was choosing based on what I liked--but the reassurance that it's uuuuuusually okay to nominate as many and it's proooooobably "safe" is now telling me that it is, in fact, potentially "unsafe", and that this voting system is complicated enough that by nominating things, I could be hurting the things I nominate. Somehow.

I'm not sure if I was supposed to count the footnotes as part of the explanation or not, so I didn't follow through them.

Seth Gordon @962:

This strikes the same "...but how?" question for me, while it also makes me a little dubious. I wouldn't expect five hundred people to nominate identically, but I'd be downright shocked if 500 Hugo voters that all had one shared book on their nomination list didn't have several different overlapping shared interests in various subsets. As written, it sounds like the only safe way to nominate is to make sure you never share more than one thing in common with anyone else who's voting.

#

This is certainly enlightening to me, in that it's showing me just how important how things are described, not just what's being described. I am really trying to read these explanations with an equally open mind; the first one gave me a response of "Oh, that's a little odd but fair, I guess I can work with that" and the second one made me feel like I was being given cryptic warnings and suggestions about strategy that I just didn't understand, while an obscure goal I also didn't understand was being pursued; and the third told me nothing about process, but sounded like I had to start comparing my nominations to those of friends to avoid somehow undercutting my own choices.

I am not sure I would've realized these were all describing the same mechanical system, if they were presented to me as three explanations of possible voting system changes.

Also, I have been reading along, and that has been coloring both my reactions and my understanding of these explanations, much as I'm trying to look at them as I would a completely new thing. So YMMV, might want to run these things by several people who have not been reading this thread, etc.

#970 ::: Steve Halter ::: (view all by) ::: May 08, 2015, 04:59 PM:

Keith "Kilo" Watt @958:I liked that one also.

#971 ::: Seth Gordon ::: (view all by) ::: May 08, 2015, 05:08 PM:

@Fade: That’s why I described my contribution as just a lede. The answer to “but how?” would go in the following paragraphs.

#972 ::: Fade Manley ::: (view all by) ::: May 08, 2015, 05:24 PM:

@Seth Gordon: Ah, I hadn't quite parsed that meaning of "lede". Thanks for the clarification!

#973 ::: felice ::: (view all by) ::: May 08, 2015, 07:10 PM:

Keith "Kilo" Watt @958: "Least Popular Elimination (formally called "single divisible vote with least popular eliminated" or SDV-LPE for short)"

I think it would make more sense to just call it Single Divisible Vote if we need to shorten the name, or "Single Divisible Vote (LPE)" if needed for disambiguation; the LPE part is basically identical to the current system (works with fewest nominations get eliminated until there are five left), it's the addition of SDV ranking that makes the difference. And "Least Popular Elimination" on its own strikes me as a somewhat negative-sounding name, and things like that can have a subtle influence on people's attitudes.

Or perhaps something like "Hybrid Divisible Vote" or "Quantum Divisible Vote" to suggest that it takes into account both the divided and the undivided vote (the key distinguishing feature, I think)?

Would "SDV-LPE increases the chance that one of the works you nominate will make the final ballot, but decreases the chance that more than one will?" be a good way to succinctly summarise the effect (rather than the process) of SDV-LPE?

"- For those works that are eligible to be eliminated, we compare the total number of nominations they each received (that is, the total number of times that work appeared on anyone's nomination ballot). The work that received the fewest number of nominations is the least popular and now completely vanishes from the nomination process as though it never existed."

Perhaps add something like "...anyone's nomination ballot, irrespective of how many other works are on each ballot)"? Or for a more extensive rewrite: "For determining which of the two works is least popular, they are compared as if they were the only two works nominated; each nomination counts as a full point, rather than being divided amongst all the works on that ballot. The work that received the fewest..."? I think it's important to emphasise that the vote isn't divided when picking which work to eliminate.

"it's not possible for slates to control the -entire- ballot"

Not entirely true, if the non-slate opinion is too divided; it's entirely possible for no single non-slate work to be nominated by more than a fifth as many people as the slate voting bloc.

#974 ::: J Thomas ::: (view all by) ::: May 08, 2015, 07:39 PM:

@964 Duncan J Macdonald

The presence of any list at all will (IMHO) skew some nominators to consider those works as already properly vetted and therefore safe to nominate on their own ballot without any additional effort on their part. You said as much with the following: "If I had a list like that, and there were a lot of names on it, I'd probably think of what I wanted to nominate and then do a search on author's last name, or some part of the title, and if it was there then I'd vote for it."

My meaning was that instead of typing it in again (so that if there was a typo somebody would need to check whether it was the same as somebody else's nomination) I'd check whether it was already there and if so, use the same text.

I realize that you would have likely already read/viewed that particular work, and its presence on the list would merely be positive reinforcement that you were correct in your evaluation.

I don't consider it positive reinforcement to nominate something I'd already decided to nominate. Just, I'd rather click on one that's already there than make a new one in addition. If it wasn't there, then I'd nominate it myself. Which would be extra work, I'd have to look it up and copy-and-paste the name and author etc.

But there likely are people who would see something on the list and impulse-nominate. I see it making things easier for me so I like it, but maybe if we make it too easy some people will mis-use it who wouldn't otherwise.

I could imagine that for everybody who did that, there might be somebody else who went "What? I didn't remember that was this year! I'm glad I saw it, I might have forgotten to vote for it." That is, what to me is a perfectly legitimate memory-jogging. But I don't know. I'm not even sure how to tell how much difference it would make.

Actually, if we wanted to tell what difference it makes, one obvious thing to try would be to do it one year. Show the list to half the voters who vote online, and not the other half. See what differences we can find in their nominations. But it would be hard to do that double-blind, because voters might talk. Married voters might even show each other their ballots in the privacy of their own rooms....

#975 ::: felice ::: (view all by) ::: May 08, 2015, 09:24 PM:

JonW @933: "here’s my best-case analysis:... 18 votes. Maybe some of my assumptions are still too low, but that's the ballpark I figure we're playing in."

Ok, 18 extra votes from 427 ballots for best novel in 1984. That's equivalent to 44 extra in 2013 with 1113 novel ballots. Considering that the top five got between 118 and 193 nominations in 2013, 44 extra is not insignificant! And the effect could be stronger in other categories; eg the short fiction categories get a lot fewer nominations than novel, and people are more likely to have time to read more of them before nominations close, encouraging people to nominate in those categories who otherwise wouldn't, and increasing the chances that those who already nominated will find other candidates they like.

And I think your ballpark might be reasonable in the absence of a slate, but when nominators are strongly motivated to maximise non-slate nominations, as they are likely to be next year and as long as the puppies keep trying, I think we could expect people to nominate a lot more longlist works.

"It certainly doesn't double the number of votes, and it's very little compared to SDV-LPE, which raises the number of disciplined members needed for a slate taking three slots well into the three figures."

For best novel, yes - 81 could have taken two slots, though. And a bloc of 77 could have taken three short story slots under SDV-LPE in 1984. It seems a safe assumption that we've got proportionately that many puppies now. 104 could have taken four slots, and I'm not confident the puppies don't have equivalent numbers today. There's also the 5% threshold - in 2013, the second placed work was only on 5.74% of the ballots, and the addition of a voting bloc could easily have pushed it below 5%, resulting in a final ballot of three slate works and just one non-slate. I think we can rely on SDV-LPE to reduce the number of slate works on the ballot to 2-4 per category, but SDV-LPE in combination with a longlist would do better, and we need to do better.

David Harmon @917: "felice #903: the 4/6 proposal does virtually nothing to prevent it [sweeping a slot] continuing to happen.

Um, sure it does. It's not related to the tiebreaking issue, but we established a while back that it sharply increases the number of voters, and the amount of coordination, that a slate needs to pull off a sweep. Obviously not an absolute barrier, but definitely an impediment."

"[sweeping a slot]" isn't an accurate paraphrase of what I said. This year, the puppies swept most of the slots, unevenly distributed between categories. The 4/6 proposal will let the slate continue to get up to 2/3 of the slots overall, though it will probably stop them sweeping entire categories. Though if they are at all coordinated, it only raises the bar for sweeping short story by 50% - eg a bloc of 120 dividing their nominations equally between six slate works could have won all of short story in 2014, and it seems safe to assume that there are more than 120 puppies.

J Thomas @915: "If you eliminate one 10, then the other 10 might not need to face the 12. And if you eliminate both, the 12 might not need to face either of them. What's the fair way to decide who gets the chance to avoid single combat?"

In the ties that matter, there's overlap in support between the two 10s, so letting either one avoid combat is fair, because whichever is picked, more people nominated the one that wins than didn't nominate it. If there's no overlap, whichever isn't eliminated first gains no points and loses to the 12 next round.

"Why *not* let them eliminate each other? Is it somehow more unfair to them, than letting them stay is unfair to the ones that would win if they were eliminated?"

It's unfair to anyone who nominated both works, and misses out on getting their points reallocated to the other when one is eliminated. The reallocation of points after each elimination is a vital part of SDV-LPE, and these rare ties bypass it.

#976 ::: J Thomas ::: (view all by) ::: May 08, 2015, 10:07 PM:

@975 felice

915: "If you eliminate one 10, then the other 10 might not need to face the 12. And if you eliminate both, the 12 might not need to face either of them. What's the fair way to decide who gets the chance to avoid single combat?"

In the ties that matter, there's overlap in support between the two 10s, so letting either one avoid combat is fair, because whichever is picked, more people nominated the one that wins than didn't nominate it. If there's no overlap, whichever isn't eliminated first gains no points and loses to the 12 next round.

Well, but if the one that loses overlaps with the 12, the 12 might avoid combat next time and the remaining 10 might then win against the 8 that replaces that 12.

There are lots of ways it can go.

My thought is that if you can't find some good reason to say that the 10 you eliminate deserves it and the 10 you don't eliminate deserves to stay significantly more than the other, then it won't look fair.

So they should survive together or fall together. It looks to me completely arbitrary which you choose.

"Why *not* let them eliminate each other? Is it somehow more unfair to them, than letting them stay is unfair to the ones that would win if they were eliminated?"

It's unfair to anyone who nominated both works, and misses out on getting their points reallocated to the other when one is eliminated.

If they nominated a third work it will get all the points. But if they nominate two losers and they both lose, there's nothing unfair about that. In 2013 novels there were votes for 475 candidates and 470 of them lost. More than 35% of ballots had no winner on them -- more than that had to lose if there were some ballots with two winners. It's the luck of the draw, it's no more unfair than the rest of the system.

The reallocation of points after each elimination is a vital part of SDV-LPE, and these rare ties bypass it.

I don't see that there's a principle each competition has to have a winner. Points get reallocated for everybody else after each elimination, whether there's a winner or not.

Truly, I see nothing more fair about sparing both 10's when there's nothing to say one of them is better, than eliminating them both. If we keep them both, that's bad for every work they eliminate before they get removed. Why should we be nice to the twins and not to the rest?

While eliminating one is obviously bad.

And anyway, the ties might not be rare at all for slates. We talk like they'll inevitably mess up and get their votes misaligned, but their screwups are not dependable. It isn't that hard to get a twin or two out of five nominees that were intended to be the same.

#977 ::: felice ::: (view all by) ::: May 08, 2015, 11:09 PM:

J Thomas @976: "If they nominated a third work it will get all the points. But if they nominate two losers and they both lose, there's nothing unfair about that."

There is if they only lose because both were nominated, and either would have won if the other hadn't interfered. These could be two works that would both make the final ballot under the current system, and under normal circumstances SDV-LPE would put one on the ballot and eliminate the other because they share a lot of the same nominators.

It could disenfranchise an entire subset of nominators. Eg suppose under the current system an epic fantasy gets the most nominations, there are two popular hard sf novels that come 2nd and 3rd, two popular space operas that come in 4th and 5th, and the most popular urban fantasy and most popular cyberpunk novel at 6th and 7th. Under normal circumstances, SDV-LPE eliminates one of the space operas and lets the other win, along with putting the urban fantasy on the ballot (a more diverse set of finalists, which is what we want to happen); but if they happen to tie, they could both be eliminated by one of the hard sf works, and we get a final ballot with a cyberpunk novel and no space opera, even though there are more space opera fans than cyberpunk fans.


"And anyway, the ties might not be rare at all for slates"

It might disproportionately affect slates on a more or less random basis, and might randomly affect coincidental non-slate ties. Randomness is not a desirable trait in a voting system, even if the randomness is somewhat more likely to have a positive effect than a negative one.

#978 ::: J Thomas ::: (view all by) ::: May 08, 2015, 11:22 PM:

@977 felice

"If they nominated a third work it will get all the points. But if they nominate two losers and they both lose, there's nothing unfair about that."

There is if they only lose because both were nominated, and either would have won if the other hadn't interfered.

So what?

I mean, so what? Depending on circumstance, either or both of them could lose because of one changed vote for some other nominee that looks totally unrelated. That's just how the system works. There's nothing more unfair about it this time than any of the other times.

It could disenfranchise an entire subset of nominators. Eg suppose under the current system an epic fantasy gets the most nominations, there are two popular hard sf novels that come 2nd and 3rd, two popular space operas that come in 4th and 5th, and the most popular urban fantasy and most popular cyberpunk novel at 6th and 7th. Under normal circumstances, SDV-LPE eliminates one of the space operas and lets the other win, along with putting the urban fantasy on the ballot (a more diverse set of finalists, which is what we want to happen); but if they happen to tie, they could both be eliminated by one of the hard sf works, and we get a final ballot with a cyberpunk novel and no space opera, even though there are more space opera fans than cyberpunk fans.

So what? You could just as easily switch the names around, it depends only on the number of votes and not at all on subgenre. We could get the space operas at 4th and 7th and then with ties etc it's the 7th space opera that wins. There's nothing wrong with that.

Which of the fairness rules does this violate?
You seem to have a feeling that SDV-LPE is more fair without double-elimination than with it, but I don't see where that comes from.

#979 ::: J Thomas ::: (view all by) ::: May 08, 2015, 11:31 PM:

"And anyway, the ties might not be rare at all for slates"

It might disproportionately affect slates on a more or less random basis, and might randomly affect coincidental non-slate ties. Randomness is not a desirable trait in a voting system, even if the randomness is somewhat more likely to have a positive effect than a negative one.

We have been over that. There is nothing random about it. Once we choose the rules, then for any set of ballots there is only one way it can come out. No randomness whatsoever.

It seems random to you because you notice that it's unpredictable. But it was not designed to be predictable.

One vote change can change the results. That is not a bug, that's true of pretty much any voting system. Usually it's the tie or one vote away from a tie that's the change, and it looks like it's intimately connected to the nominees that are affected. With SDE-LPE it can be a changed vote that affect's some other nominee's score, and that has effects that cascade through the system. And that's OK.

#980 ::: P J Evans ::: (view all by) ::: May 09, 2015, 12:44 AM:

Why are y'all trying to bring sub-genre into it? That's not where we want to go.

Go back to the top of the post and read what the criteria are. Y'all seem to have forgotten them.

#981 ::: felice ::: (view all by) ::: May 09, 2015, 02:05 AM:

P J Evans @980: "Why are y'all trying to bring sub-genre into it? That's not where we want to go."

Just as a simple, easy-to-understand example of how simultaneous elimination can result in outcomes contrary to the spirit of SDV-LPE. The actual patterns of how nominations overlap would be a lot more complex than grouping by subgenre, but the point remains valid.


As to the unfairness of arbitrary tie-breaking, how about we look at it from the point of view of the people actually affected? How would most voters respond to "One of the works you nominated is tied for elimination with a work you didn't nominate; would you rather toss a coin to see which is eliminated (50% chance your nomination is eliminated), or eliminate them both (100% chance your nomination is eliminated)?" or "Two of the works you nominated are tied for elimination; would you rather toss a coin to see which is eliminated and which has a chance to be a finalist, or eliminate them both?"?

#982 ::: Keith "Kilo" Watt ::: (view all by) ::: May 09, 2015, 02:17 AM:

Felice@981:
As to the unfairness of arbitrary tie-breaking, how about we look at it from the point of view of the people actually affected? How would most voters respond to "One of the works you nominated is tied for elimination with a work you didn't nominate; would you rather toss a coin to see which is eliminated (50% chance your nomination is eliminated), or eliminate them both (100% chance your nomination is eliminated)?" or "Two of the works you nominated are tied for elimination; would you rather toss a coin to see which is eliminated and which has a chance to be a finalist, or eliminate them both?"?

I can't speak to how most voters would respond, so I can only give a data point. But everyone I asked here said of those two choices, they'd eliminate both. The most concise statement of why was that in order to ensure the integrity of the process, you'd have to be able to re-run it later and get the exact same results, else you could never be sure that a mistake wasn't made.

Kilo

#983 ::: J Thomas ::: (view all by) ::: May 09, 2015, 02:20 AM:

Felice, are you arguing in favor of random outcomes now? I thought we had an agreement we didn't want to go there. Have you changed your mind?

How would most voters respond to "One of the works you nominated is tied for elimination with a work you didn't nominate; would you rather toss a coin to see which is eliminated (50% chance your nomination is eliminated), or eliminate them both (100% chance your nomination is eliminated)?"

How would most voters respond to "A couple of works nominated by two other guys are tied for elimination; would you rather toss a coin to see which is eliminated or eliminate them both?"?

I don't think appealing to voters' perceived self-interest is the best way to resolve this.

#984 ::: felice ::: (view all by) ::: May 09, 2015, 03:03 AM:

Keith "Kilo" Watt @982: "I can't speak to how most voters would respond, so I can only give a data point. But everyone I asked here said of those two choices, they'd eliminate both. The most concise statement of why was that in order to ensure the integrity of the process, you'd have to be able to re-run it later and get the exact same results, else you could never be sure that a mistake wasn't made."

You said earlier "chronological order of nomination is also essentially a coin flip as well", and you can re-run that as many times as you like and get the same results. Or use a public pre-generated sequence of coin tosses; nobody can possibly know during the nomination period where ties will end up occurring, so advance knowledge of the sequence will do nothing to influence people's choices, but it's also repeatable.


J Thomas @983: "Felice, are you arguing in favor of random outcomes now? I thought we had an agreement we didn't want to go there. Have you changed your mind?"

Randomness in whether a rule is applied or not is worse than randomness in picking between two indistinguishable possibilities. And yes, technically the set of ballots is unpredictable rather than random, but I don't think the distinction matters here.

"How would most voters respond to "A couple of works nominated by two other guys are tied for elimination; would you rather toss a coin to see which is eliminated or eliminate them both?"?"

What difference does it make to them? It doesn't affect them, so why ask them? Whereas it matters a lot to the people who are affected by the tie, and there's no action they can take to avoid the risk of causing a tie (if they want to nominate both A and B, either including both or leaving one off could cause a tie, depending on the unknown factor of how many other people nominate A or B).

#985 ::: J Thomas ::: (view all by) ::: May 09, 2015, 08:44 AM:

"How would most voters respond to "A couple of works nominated by two other guys are tied for elimination; would you rather toss a coin to see which is eliminated or eliminate them both?"?"

What difference does it make to them? It doesn't affect them, so why ask them?

It does affect them. Each nominee that *other* people chose, that isn't one of theirs, is a threat to their own nominees' success. Of course it's only a big threat when it has more votes, but the general rule is "My nominee has a better chance when your nominee fails."

Again, I don't think we should resolve the fairness issue by appealing to individual voters' self-interest. Find some other reason to say it's more fair to eliminate one of an identical pair, than to eliminate both or neither.

Whereas it matters a lot to the people who are affected by the tie, and there's no action they can take to avoid the risk of causing a tie (if they want to nominate both A and B, either including both or leaving one off could cause a tie, depending on the unknown factor of how many other people nominate A or B).

If there was a strategy people could use to keep their nominees from being eliminated, they would use it.

We don't want to have a strategy people can use to keep their nominees from being eliminated. Avoiding such strategies is one of the goals.

For every situation where your nominee could be eliminated in a tie, with a difference of one extra vote for the opponent, it would not be a tie. Yours would simply lose. And it can happen *because* the opponent had more votes but they were shared votes so its score is low, and by luck it wound up paired off against yours. If a few votes were different yours could have avoided that fate. That's just how the system works. A nominee that's in the wrong place at the wrong time will be eliminated. Because of the distribution of votes to other nominees, in ways that appear to have nothing to do with your nominee.

And there's nothing wrong with that. It's completely fair. Right?

#986 ::: J Thomas ::: (view all by) ::: May 09, 2015, 08:50 AM:

Randomness in whether a rule is applied or not is worse than randomness in picking between two indistinguishable possibilities.

No one has suggested applying the rules at random. For each precisely-worded set of rules, we know exactly how to apply them.

I have the idea that what you mean is, that SDE-LPE is the right way, the fair way, so we should use it all the time and not just sometimes. The trouble with that is that SDE-LPE does not say what to do with ties. So we have to choose what to do in that case.

#987 ::: Jameson Quinn ::: (view all by) ::: May 09, 2015, 12:06 PM:

OK, I'm going to try again with a simple explanation of LPE (including the tiebreaker I'd suggest), then attempt a few FAQ questions about it, then propose a modification. I'm interested in seeing how people react to the proposed modification, and I personally think it's an improvement, but I'm ready to drop it if people don't like it.

Explanation:

What (Terms and definitions)

This voting system is called Least Popular Elimination, or LPE for short. In this system, each voter votes for 1-5 candidates, which will be referred to below as "works" (though in some categories they are actually people).

LPE alternates between two steps, which count the votes in different ways, so we will use two different terms to refer to the votes depending on how we are counting them. When we count the votes divided in fractions between the different works on a ballot, we refer to them as "points". When we count each vote on each ballot as one, we refer to them as "approvals".

LPE works by eliminating one work at a time. Works which have not been eliminated are referred to as being "remaining". When the process ends, they become "nominees".

How (Process)

The process of voting is the same as it is currently. That is, for each category, you simply list 1-5 works that you believe may deserve a Hugo. (This system would work well without the limit of 5, too; but for simplicity, we do not propose to change that here.) The order in which you list the works does not matter.

This system eliminates one work at a time, until there are 5 remaining. If there is a tie in the last elimination step, both of the tied works are kept, so there may occasionally be more than 5 works nominated. Each elimination has two steps, a "matchup" and a "showdown".

In the matchup step, we use points to look for the two weakest candidates. Each ballot is worth 1 point total, and this point is divided evenly between the remaining candidates which that ballot supports. The two works with the least points are sent to the showdown step. If there are any other works with the same number of points as the second-lowest, they are also sent to the showdown.

In the showdown step, whichever of the matched-up works has the fewest approvals is eliminated. If there is a tie, and only 4 untied works remain, then no elimination occurs, and the remaining works are the final nominees. If a tie happens when more works remain, then the tie is broken, first using the number of points each work had in the current matchup step, then using the number of points from the preceding matchup step, and so on. If the points for two or more works have been tied at each step since the beginning of the process, then both (or all) of them are eliminated.

Why (FAQ)

Q. Why was this system designed?

A. In the past, when not distorted by organized minorities, the Hugo process has worked well. However, it leaves open the possibility for a minority of voters to to control a supermajority of nomination slots by organized voting for "slates" of candidates. This has been alleged to have happened at various times, and is generally agreed to have happened at least once, in 2015, when two related subfactions called "sad puppies" and "rabid puppies", together constituting a minority of voters, promoted overlapping slates. LPE is designed to prevent that from happening, while preserving the good aspects of the current system when organized strategic voting is not a factor.

Q. So the point of LPE is to punish slate voting?

A. No. Even if you think that slate voting is evil, it's generally a bad idea to design a voting system to punish something. Punishing one group means rewarding the opposite group; and since people are smart, they can usually figure out strategic votes which will make them appear to be in the group that deserves reward, distorting the results yet further. In any case, that was never the design of LPE. The idea is that every group of voters should get their fair share of influence, so about 20% of the voters should control about 20% of the results.

Q. Why are there two steps (matchup and showdown) in each elimination?

A. Voting systems involve tradeoffs between competing goals, and often it's best to focus on them one at a time.

Q. What are the goals of the matchup step?

The matchup step is about the goal of breadth of representation; generally speaking, most of the voters should like at least one of the final nominees. Thus, the point assignment reflects the fact that a voter who has only one of their choices remaining would probably be more upset when that work was eliminated, than a voter who had several of their choices remaining.

Q. What are the goals of the showdown step?

The showdown step has several related goals. First, there is the obvious: the nominees should be works with a lot of votes (approvals); and in particular, the work with the most approvals should always be among them. Second, there is the fact that the results should be similar to the historical results from years without significant organized voting.

....(The third goal is a bit more complex. Any democratic voting system has the possibility of strategy; and any proportional voting system like this one has the possibility of "bullet" or "truncation" strategy. The showdown step is designed to minimize the incentives for truncation, and encourage broad voting)........

(to be continued. I have to go for now, but I'd like to post this unfinished.)

#988 ::: J Thomas ::: (view all by) ::: May 09, 2015, 12:44 PM:

@987 Jameson Quinn

This looks good to me, at least as the full explanation to go into the appendix and as the start of a FAQ. It's far too complicated to be an early page of a proposal.

This voting system is called Least Popular Elimination, or LPE for short.

Someone already suggested that this sounds bad. I would suggest renaming it to Remove The Worst. Maybe someone else will have a better suggestion for a name.

Q. What are the goals of the matchup step?

The matchup step is about the goal of breadth of representation; generally speaking, most of the voters should like at least one of the final nominees. Thus, the point assignment reflects the fact that a voter who has only one of their choices remaining would probably be more upset when that work was eliminated, than a voter who had several of their choices remaining.

This is excellent!

Q. What are the goals of the showdown step?

The showdown step has several related goals. First, there is the obvious: the nominees should be works with a lot of votes (approvals); and in particular, the work with the most approvals should always be among them. Second, there is the fact that the results should be similar to the historical results from years without significant organized voting.

Also excellent! You have come up with reasons why people might want these steps for themselves, not just because they provide a way to oppose slates. You want to choose the two that people care the least for, and eliminate at least one of them. So the average goes up. Great rationale!

#989 ::: P J Evans ::: (view all by) ::: May 09, 2015, 12:52 PM:

I'd like to see that explanation tried on the business meeting. It still sounds like something intended for people who have taken a class - or a seminar - in voting systems, not for people who are looking for a way to keep the nomination system from being gamed so thoroughly.

Go back and read the beginning of this post. It doesn't seem to be registering correctly.
Read the explanations that we-the-fans have said work well.
Then look at yours again.

#990 ::: J Thomas ::: (view all by) ::: May 09, 2015, 02:31 PM:

@987 Jameson Quinn

When I thought about your explanation and looked for simple language to describe it, I found myself with a variant voting system I would like you to at least briefly consider.

What if, for the matchup step, instead of counting up all the partial votes you take their average. That is, for a work with 3 votes, instead of counting 1/2 vote for a ballot with 2 items and 1/5 vote for a ballot with 5 items and 1 vote for a ballot with just 1 item, resulting in a score of 1.7, instead we take the score and divide it by the number of ballots which in this case would give a rating of 1.7/3 = .5666,

The result would be that a slate with 5 items would have a rating of 1/5 = .2, the lowest possible. One of them would be lost almost immediately.

Then nonslate works with a rating less than 1/4 would be removed, until a second slate work was removed.

Then nonslate works with a rating less than 1/3 would be removed, until a third slate work was removed.

If we wound up with more than three nonslate works with a rating more than 1/2, the fourth slate work would also be removed.

I have some doubts about this approach. We could lose popular works because they were on too many ballots with each other, and got eliminated by slate works, leaving behind less popular works that were bullet-voted. (That can happen now but not as much.)

But it's about as simple as LPE (or RTW) and easy to test. Your explanation about why LPE is good would fit this system better than it fits LPE.

#991 ::: Jameson Quinn ::: (view all by) ::: May 09, 2015, 02:46 PM:

@988: "Remove The Worst"... I see what you're going for, but I'm not sold. "W" is horrible in an acronym; best are acronyms that are pronounceable.

Hmmm...

"Bottom Up Matched Pairs Ylimination" (BUMPY) voting?

@989: I'm not sure what to do with that critique. "Just look at what you wrote ... it's bad."

Part of the problem is that there are some people who react badly when you leave out any details, because you must be trying to hide something; and others who react badly when you put in the details, because it sounds too complicated.

I think that defining my terms at the start, before saying what those terms were really about, didn't work too well.

Should I move all the explanations of the tiebreakers and special cases to the end? It feels a bit dishonest for me to say "You do X, period" and then later say "But occasionally, if Y is true, you do Z instead of X." But maybe it would work better that way, because the simple, no-tiebreaker process is what would happen over 90% of the time.

....

So I still have to finish the FAQ, but I think I'll wait for more comments before I do that. Meanwhile, I have another proposal to mention.

What if, instead of spreading 1 point per voter among all the remaining works, we reduced the total by 0.2 for every work after the first? So when only 1 of your choices remained, you'd give it 1 point; when 2 remained, you'd give them 0.4 points each (0.8 total); when 3 remained, you'd give them 0.2 points each (0.6 total); when 4 remained, you'd give them 0.1 points each (0.4 total); and when 5 remained, you'd give them 0.04 points each (0.2 total).

This is harder to explain, but it's a good approximation of the Saint-Lagüe weights, which give better resistance to slates. If a category had 100, 50, 35, and 30 votes for the top 4 non-slate candidates, getting 4, 3, or 2 slots would take 1000, 175, or 75 straight-slate voters, rather than 200, 105, or 60 in SDV-LPE (BUMPY, whatever). For regular voters, it wouldn't change much compared to regular SDV-LPE; rarely do voters end up supporting more than 2 of the eventual finalists.

Is this even worth pursuing?

#992 ::: J Thomas ::: (view all by) ::: May 09, 2015, 03:18 PM:

@991 Jameson Quinn

What if, instead of spreading 1 point per voter among all the remaining works, we reduced the total by 0.2 for every work after the first?

Is this even worth pursuing?

I think it's worth some simulation etc. The immediate problems are that first, it's harder to explain. When you say to just split the vote, that sounds inherently fair though there's no real reason it ought to be. When you throw away an arbitrary-looking part of the votes, it looks somehow less fair even though there's no real reason it shouldn't go that way.

I'm guessing that this would give more reason to bullet-vote. When you vote for 5, all of them look like they'd be more susceptible to random elimination -- even the one you think is most popular. If so, that's another concern to balance against the (hoped) advantages.

But we won't know how it actually does unless we try.

#993 ::: Jameson Quinn ::: (view all by) ::: May 09, 2015, 07:30 PM:

@992: "Arbitrary-looking": yes, but of course there actually is a good amount of theory involved.

Yes, it would be slightly more encouragement to bullet vote. But in a non-slate environment, that is the difference between "almost none" and "very little" strategic incentive. In an environment where slates are a thing but you're not one of them, that goes up to "a bit", and of course if you are a slate voter the incentive for managed bullet votes rather than simple slate voting goes up to "extreme".

I'll do sims on this tonight.

#994 ::: P J Evans ::: (view all by) ::: May 09, 2015, 07:58 PM:

I guess y'all are happy talking to yourselves, even though it's not going to get you anywhere.

#995 ::: felice ::: (view all by) ::: May 09, 2015, 08:08 PM:

J Thomas @985" "It does affect them. Each nominee that *other* people chose, that isn't one of theirs, is a threat to their own nominees' success. Of course it's only a big threat when it has more votes, but the general rule is "My nominee has a better chance when your nominee fails.""

Unless eliminating both others gives extra points to their nominee's rival, and results in their nominee facing and losing an elimination contest they'd otherwise have avoided. There's no way for them to know if dual or single elimination would better serve their interests, so there's no reason for them to have an opinion.

"Again, I don't think we should resolve the fairness issue by appealing to individual voters' self-interest. Find some other reason to say it's more fair to eliminate one of an identical pair, than to eliminate both or neither."

There is no other reason, by definition - it's a tie!

"For every situation where your nominee could be eliminated in a tie, with a difference of one extra vote for the opponent, it would not be a tie. Yours would simply lose."

In the ties that matter, there are people who nominated both of the tied works; With a difference of one extra vote or one less vote for either work, one of their nominees wins and the other looses; the tie means they both lose.


J Thomas @986: "I have the idea that what you mean is, that SDE-LPE is the right way, the fair way, so we should use it all the time and not just sometimes. The trouble with that is that SDE-LPE does not say what to do with ties. So we have to choose what to do in that case."

What we shouldn't do is throw out the bit that says "if one of the works on your ballot is eliminated, your vote is redistributed to the remaining works to increase their rankings".

Jameson Quinn @987: "If a tie happens when more works remain, then the tie is broken, first using the number of points each work had in the current matchup step, then using the number of points from the preceding matchup step, and so on."

Going back through the preceding steps didn't help when Kilo looked at the ties in the '84 ballots, and it seems likely that it will seldom be useful for the Hugos. It makes sense for STV with a limited number of candidates, but probably isn't worth including in the Hugo rules.

#996 ::: Keith "Kilo" Watt ::: (view all by) ::: May 09, 2015, 09:01 PM:

PJ@994:

For myself, I'd really like to finish the proposal we're working on before we start another -- and for the record, I'm not opposed to another, but if we don't focus a little bit, I think we're going to wander (all who wander are not lost, but they sure take a long time to get where they're going...).

As a quick status check, am I correct that the only missing pieces on the SDV-LPE proposal are:

- How to handle ties
- How to handle withdrawn nominations
- Best way to present at the business meeting
- The final formal proposal language itself

For the second, can we agree to just take the nomination(s) from the previous round if someone withdraws?

For the the third, I think one way to look at it is to put the more non-specialist description (maybe a modification of what I wrote, if that's acceptable) in the proposal, but then have the more technical descriptions available for any Q&A that come up at the business meeting. Does that seem reasonable?

- For the fourth, I'd be willing to write a second draft based on what has been discussed so far. My hope is that would just be editing, assuming all the issues have been resolved.

Which leaves ties as a sticking point. Respect to felice, but I'm not sure a proposal with arbitrary factors in it is going to go over well at the business meeting. That is just my sense from talking to people locally, so this is not a large enough sample to say for sure. It's certainly possible, but I think it makes the job way harder. As mentioned, I'd be open to any non-arbitrary tie-breaking system but I haven't come up with anything myself (and I've been thinking about it!) for a tie in the lowest point candidates situation, nor have I heard anything from anyone else.

If we could quickly give a "yea" or "nay" to the SDV-LPE itself and the last three items, I think that would help. Then if we could focus on the ties for a bit and make a decision, I think we could get this proposal ready and then happliy move on to others if we desire.

Does that seem like an acceptable path to everyone? As someone pointed out, a very small group has made most of the posts here, but I know others are interested and following. If you wouldn't mind just piping up with a quick opinon on the above, I'd really appreciate it. I think it'd be very helpful to just take the pulse of the community right quick.

Many thanks!
Kilo

#997 ::: P J Evans ::: (view all by) ::: May 09, 2015, 09:21 PM:

Keith, you're doing fine (you're writing clearly and concisely). It's the other three....

And a bit of weediness, average nominees per category per list in 1984:
novel ... 3.01
novella ... 1.21
novelette ... 1.65
short story ... 1.73
non-fiction ... 0.77
dramatic presentation ... 1.68
pro editor ... 1.64
pro artist ... 1.95
semiprozine ... 1.04
fanzine ... 1.11
fan writer ... 0.95
fan artist ... 1.07
Campbell Award ... 1.03

(average 18.83 nominations per list)

#998 ::: Jameson Quinn ::: (view all by) ::: May 09, 2015, 09:40 PM:

PJ... I did my best responding to your comments to me, and I do appreciate them. There's less to appreciate and it's harder to respond, though, when you make negative comments in the third person.

#999 ::: felice ::: (view all by) ::: May 10, 2015, 01:28 AM:

Keith "Kilo" Watt @996: "As a quick status check, am I correct that the only missing pieces on the SDV-LPE proposal are:
- How to handle ties
- How to handle withdrawn nominations
- Best way to present at the business meeting
- The final formal proposal language itself"

What to call it is also in question; sticking with "SDV-LPE" works for me, but alternatives have been suggested.

"For the second, can we agree to just take the nomination(s) from the previous round if someone withdraws?"

I'd prefer something like "If a finalist declines nomination or is found to be ineligible, the tallying is rerun without the withdrawn work, and the vacant spot on the ballot is filled by highest ranked work that isn't already a finalist." The nomination from the previous round is simpler but less fair; rerunning and taking the new top five is technically best but would be too problematic in practise.

"For the the third, I think one way to look at it is to put the more non-specialist description (maybe a modification of what I wrote, if that's acceptable) in the proposal, but then have the more technical descriptions available for any Q&A that come up at the business meeting. Does that seem reasonable?

I assume the proposal has to include the specific wording for the changes to the WSFS constitution?

A suggestion - would it be helpful to say a ballot gets 60 points in each category, rather than one point? That way the points can be divided up between works without worrying about fractions, and it will help clarify the difference between points and nominations (currently, a ballot with one remaining work contributes both one point and one nomination, which blurs the distinction).

"Which leaves ties as a sticking point. Respect to felice, but I'm not sure a proposal with arbitrary factors in it is going to go over well at the business meeting"

Does my objection to the simultaneous elimination make sense to you, though, even if you don't like my alternative?


#796 - for the Campbell Award, the ballots for Tepper are quite mixed - 10 for "sheri s.", 7 for "sheri", and 1 for "sherri" - did you correct for that when running it through SDV-LPE? If not, could you run the Campbell Award again with the Teppers consolidated and see what results we get? Also for fan writer - there's a mixture of "teresa" and "theresa", so she should probably be a SDV-LPE finalist too. And there are a bunch of nominations for "hurricane claude" under novelette, so it should also probably have beaten "habiline husband". Ie, it looks like most of the few differences between the SDV-LPE results and the actual results for 1984 are due to inconsistencies on the ballots, rather than genuinely different outcomes.

#1000 ::: J Thomas ::: (view all by) ::: May 10, 2015, 07:35 AM:

For the second, can we agree to just take the nomination(s) from the previous round if someone withdraws?

I would like us to reconsider that. What are the disadvantages of just running it again? Doing the vote correctly is the right thing to do, if it's practical.

Technically it's trivial. The sticking point seems to be that other nominations might change. That could be palliated. Encourage authors who want to decline to be pro-active and decline before they are asked, as some have already done this year.

When calling authors to ask if they would accept, stress that results are not final and they could still lose, and that reduces their incentive to announce early which we don't want them to do.

The big issue I see is the possibility that it takes awhile to find one author who then declines, the new tally produces several new authors, one of them is hard to reach and declines late, the new tally produces several new authors, one of them is hard to reach and declines late, and you're behind schedule. It's obviously better to rerun the voting but not with that result. Is there a way to reduce that risk?

Would it be OK if I asked some upcoming Hugo voting administrators what they think about it?

For the the third, I think one way to look at it is to put the more non-specialist description (maybe a modification of what I wrote, if that's acceptable) in the proposal, but then have the more technical descriptions available for any Q&A that come up at the business meeting. Does that seem reasonable?

Something like that is necessary. I think we need a full description, probably with pictures and maybe moving pictures or a simulation or something on a website, and provide a link in the proposal. Have a good short description there too, of course. The more members who have it all worked out ahead of time, the better.

- For the fourth, I'd be willing to write a second draft based on what has been discussed so far. My hope is that would just be editing, assuming all the issues have been resolved.

They have not, but I hope the writing can be modular enough that changes in one part won't ripple out into the rest.

I would like it if you could incorporate into something like your #958, some simple version of the most vital part of Jameson's FAQ, where he explains why the two steps are fair. The first step encourages works with broad support, while the second step encourages works with lots of votes. Make completely clear the rationale for why fairness is built into the system.

Which leaves ties as a sticking point. Respect to felice, but I'm not sure a proposal with arbitrary factors in it is going to go over well at the business meeting.

I see two main approaches to resolve this. One is to look at what's fair for everybody. I think that's the right thing to do, but a friend once told me that that's a great tactic to keep anything from being decided.

A second is to start contacting people who will be at the business meeting and ask them what they think. The ones who think about it and get their way, are likely to be favorably disposed to the proposal when it comes up.

#1001 ::: J Thomas ::: (view all by) ::: May 10, 2015, 07:44 AM:

I finally saw clearly one of my reservations about this voting system.

Imagine that we have handled the ballots and the result is ABCDE. But A declines, and we run the results again. This time the winners are FGHIJ.

That can happen, though it's probably unlikely. If B comes in second when we count A, but it isn't in the top five when we don't, isn't there something wrong? B should be better than J regardless of A.

Arrow's theorem lists several reasonable ideas about fairness and says we can't have all of them, we have to give up something. This is one of Arrow's fairness ideas we've given up.

It isn't that big a deal since we have to give up one, it's just sort of jarring when I notice it happening.

#1002 ::: J Thomas ::: (view all by) ::: May 10, 2015, 07:47 AM:

@999 Felice

"For the second, can we agree to just take the nomination(s) from the previous round if someone withdraws?"

I'd prefer something like "If a finalist declines nomination or is found to be ineligible, the tallying is rerun without the withdrawn work, and the vacant spot on the ballot is filled by highest ranked work that isn't already a finalist." The nomination from the previous round is simpler but less fair; rerunning and taking the new top five is technically best but would be too problematic in practise.

That looks like the best if we can't rerun it. Thank you! That's a very good idea.

#1003 ::: J Thomas ::: (view all by) ::: May 10, 2015, 08:04 AM:

PJ... I did my best responding to your comments to me, and I do appreciate them. There's less to appreciate and it's harder to respond, though, when you make negative comments in the third person.

I think he's frustrated. We've put a lot of work into something that he sees would be an improvement, and he's our expert on how to get it passed, and we are not doing very well at following his advice.

My response at this point is to let Keith "Kilo" write it, and make suggestions. I could possibly contribute to a technical description that would go into the tl;dr section.

Though it's possible that we'd do better to publish documented code for that, than write it out in english. The code might be more readable, and would probably be less ambiguous.

#1004 ::: Jameson Quinn ::: (view all by) ::: May 10, 2015, 08:51 AM:

The issues are:
- Name of the system
- How to handle ties
- How to handle withdrawn nominations
- Best way to present at the business meeting
- The final formal proposal language itself

One at a time:


- Name of the system

I think we should do score voting. That is, all rate the options 0-10, and whichever gets the highest average rating wins. I'm going to explicitly choose to rate everything above 5 to show that I don't really care and they all look good to me. Here are the names I've seen; if I've missed any, or there are other suggestions, I'll add in votes for the others.

vote: name

5: SDV-LP
9: SDV-LPE
5: SDV
8: LPE
5: Remove the worst (RTW)
6: BUMPE (pronounced "bumpy"; bottom up matched pairs elimination; where the "matched" is really just to make the acronym pronounceable)
7: BUPE (as above, without "matched". Pronounced "boo-pay"?)
10: BUE (as above, without "pairwise". Pronounced "byoo"?)

- How to handle ties

My proposal was, break ties using current score, then past scores; then mass-eliminate any remaining ties. There are only a few ways mass-elimination could happen in that system. In order of decreasing probability:

Works appearing on just a few ballots, eliminated early. This is not a problem; these works could not possibly win.

Works appearing (almost) exclusively as bullet votes. This is slightly problematic, but easy for voters to avoid.

Works appearing exclusively on slate ballots. I have no problem mass-eliminating these.

Other cases. I didn't come up with even one case of this happening in 1300 bootstrap samples using 1984 data (100 for each category), so I think we can safely ignore this possibility.

- How to handle withdrawn nominations

Again, I think we're far enough along to vote on this.

0: rerun everything from scratch
5: rerun and take the first new nomination
10: take the next candidate from the original run

(Yes, it's not perfect. But it's simple. And the only likely case where it doesn't give the same result, is where a slate candidate declines, and the slate ends up disenfranchised. Not a concern for me; they should have gotten the candidate's consent first.)

- Best way to present at the business meeting
- The final formal proposal language itself

We'll work on these once the above are resolved.

#1005 ::: Jameson Quinn ::: (view all by) ::: May 10, 2015, 09:01 AM:

Obviously, score voting is not legitimate unless we get several votes. Also, since we're now in the 4 figures of comments, a new thread is due. So I suggest that we, at least the heavy commenters, come to a consensus on whether voting is a good idea, then approach TNH with a request for a new thread summarizing the discussion so far, the state of the proposals, and asking for votes on outstanding issues.

As for my other "deduct .2" proposal: although I honestly think it's almost certainly the best we'll get in terms of better slate resistance in a long-tail context, without much more bullet voting incentive or complexity... it's still too complex, so I'm dropping that whole line of thought, including 5b and my "mega-longlist" proposal.

#1006 ::: Jameson Quinn ::: (view all by) ::: May 10, 2015, 09:18 AM:

@1005: Just to make extra-clear: I'm not at all suggesting using score voting for the Hugos themselves, but rather just using an informal score poll here to decide on the simpler of the outstanding issues.

#1007 ::: Keith "Kilo" Watt ::: (view all by) ::: May 10, 2015, 10:44 AM:

Mother's Day, so we're off to get doughnuts for my wife. I did run one sim, however, and randomly removed one of the 84 Hugo Best Novel finalists (Tea with the Black Dragon). The results came out identically, except that Citadel of the Autarch, which had been eliminated in the next to last round originally, got moved up onto the final ballot.

That's just one data point. I'll try to run some more, and of course I'm sure we can maufacture cases where it matters. But I'm not sure SDV-LPE is quite as sensitive to events in the elimination process as you may think. Every time I've run realistic data, I've not come up with weird or counter-intuitive results. That doesn't mean the weird cases won't come up, of course, so we should definitely look at them.

Kilo

#1008 ::: J Thomas ::: (view all by) ::: May 10, 2015, 11:31 AM:

1007 ::: Keith "Kilo" Watt

I did run one sim, however, and randomly removed one of the 84 Hugo Best Novel finalists (Tea with the Black Dragon). The results came out identically, except that Citadel of the Autarch, which had been eliminated in the next to last round originally, got moved up onto the final ballot.

That's a plausible outcome, and here's an argument for it:

Jameson says that 90% of the time, we get the same result as approval voting. I see no reason to expect that authors would decline more under circumstances that LPE gets unusual results. So we can expect the same outcome 90% of the time. And often when the result is not the same it would be similar. If there's only *one* new winner, that's fine regardless if the order of winners changes.

So I would expect it not to be a problem in practice to just re-run the whole process. It will almost always be no harder to handle than acceptance voting.

One possibility for a belt-and-suspenders type administrator: If he's concerned about later time pressures, rerun the data five times, assuming that each different winner has declined. That may usually get one extra name, rarely two, possibly three. Add those to the list of people to call. Tell each of them, "We don't know yet who will be on the Hugo shortlist, it depends on who declines. Would you decline?" Then if somebody does decline, it's already handled before the deadline.

If it looks like it wouldn't be such a big deal to re-run the whole vote-counting, why not do that? I think a long time ago somebody assumed it would be a big deal. It's possible for it to blow up, in theory. Careful planning could help to minimize that possibility.

#1009 ::: Jameson Quinn ::: (view all by) ::: May 10, 2015, 11:40 AM:

I suggest that the people still in this thread are a nonon-representative sample. Our job now is to agree on a summary of the outstanding issues to post at the top of a new thread, not to decide among ourselves. I will be gone for the rest of the day and I suspect I'm not the only one. I approve whatever others agree on but I suspect this will be resolved hasta mañana.

#1010 ::: Jameson Quinn ::: (view all by) ::: May 10, 2015, 01:08 PM:

I just did a bootstrap to see how often dropping one winner would change any existing winners. 325 samples of 100 votes drawn from 1984 data (25 each category), and only 3 examples where it changes. I think the moral is, we should just do the simple thing — when somebody declines, add the last to be eliminated using original data — because in literally over 99% of the cases it is the same as the more complicated things.

(Note that by sampling only 100 ballots, I deliberately increased the chances of any strange results by at least a factor of 3 over reality.)

#1011 ::: J Thomas ::: (view all by) ::: May 10, 2015, 02:18 PM:

@1010 Jameson Quinn

I think the moral is, we should just do the simple thing — when somebody declines, add the last to be eliminated using original data — because in literally over 99% of the cases it is the same as the more complicated things.

I would argue the alternate moral -- when there's hardly ever any bad consequence to doing the right thing, go ahead and do the right thing.

There's essentially no effort to running the data through one more time, and then you can say that you counted the votes and got the right answer.

Getting the right answer is a positive good. It costs you a few keystrokes compared to getting an answer that's probably right.

It won't make much practical difference, but opportunities to do it right and not get punished for it come so rarely I hate to waste them.

#1012 ::: Jameson Quinn ::: (view all by) ::: May 10, 2015, 02:36 PM:

@1011: I'd say we should do whichever thing is more likely to pass the BM. Which I'd guess is the simple thing. But if we disagree on that, that should be part of what we should put in the questions at the top of the new post; because we need more voices than just the few people willing to wade through 1000+ comments.

#1013 ::: abi ::: (view all by) ::: May 10, 2015, 02:42 PM:

And you anticipate me! We do indeed need a new post, and I'd very much appreciate some suggestions of what to put in it.

What I would suggest is that anyone who wants to have input into the next OP post what they'd like to say. I'll then assemble them into such a form.

The deadline for this is twenty-four hours from this very moment, at which point I will take the last comment from each person who posts between now and then and indicated that they want to be quoted in the OP.

Unless someone has a different idea how to do this?

#1014 ::: felice ::: (view all by) ::: May 10, 2015, 05:30 PM:

J Thomas @1001: "I finally saw clearly one of my reservations about this voting system. Imagine that we have handled the ballots and the result is ABCDE. But A declines, and we run the results again. This time the winners are FGHIJ. That can happen, though it's probably unlikely. If B comes in second when we count A, but it isn't in the top five when we don't, isn't there something wrong? B should be better than J regardless of A."

No, nothing is wrong. The people who voted A prefer FGHIJ over BCDE, and either there are a lot of them or FGHIJ and BCDE were very closely matched to begin with. Under the current system, the original finalists would probably have been AFGHI, and the SDV-LPE result of ABCDE is better. Voting doesn't measure "better" in any objective sense; in this example, people who don't like A prefer B over J, people who do like A prefer J over B. But such a radical change is highly unlikely in practice, because lots of A voters will have also liked BCDE, or A was their last remaining choice, or most A voters preferred J over F, G, H, and I. The difference between "take the existing runner up" and "rerun" is more likely to be BCDEF vs BCDEJ. I think "rerun and add the first new" is likely to get the same result as "rerun and replace all" in most cases.

Jameson Quinn @1004: "Here are the names I've seen; if I've missed any, or there are other suggestions, I'll add in votes for the others."

You missed "hybrid divisible vote" and "quantum divisible vote", intended to suggest that both divided and undivided vote are used.

"- How to handle withdrawn nominations"

My vote:

0: rerun everything from scratch
9: rerun and take the first new nomination
3: take the next candidate from the original run

Keith "Kilo" Watt @1007: "I did run one sim, however, and randomly removed one of the 84 Hugo Best Novel finalists (Tea with the Black Dragon). The results came out identically, except that Citadel of the Autarch, which had been eliminated in the next to last round originally, got moved up onto the final ballot. That's just one data point. I'll try to run some more, and of course I'm sure we can maufacture cases where it matters. But I'm not sure SDV-LPE is quite as sensitive to events in the elimination process as you may think."

Try taking out "startide rising" - that's most likely to upset the other results. 2013 Novelette is a good example of where we might run into differences: in the actual result, there were two Seanan McGuire stories; under SDV-LPE, there would probably only have been one, with "the waves" replacing it on the ballot, and "astrophilia" as runnerup. But if the first Seanan story had been withdrawn for whatever reason (say it was found ineligible), the second would probably pop up above "astrophilia" if the tallying was rerun, and would I think be indisputably the more appropriate replacement finalist. (If it was declined rather than ineligible, all Seanan stories would probably need to be removed before rerunning, but there'd sometimes be similar correlations between works by different authors.)

#1015 ::: J Thomas ::: (view all by) ::: May 10, 2015, 06:40 PM:

@10140 Felice

"I finally saw clearly one of my reservations about this voting system. Imagine that we have handled the ballots and the result is ABCDE. But A declines, and we run the results again. This time the winners are FGHIJ. That can happen, though it's probably unlikely. If B comes in second when we count A, but it isn't in the top five when we don't, isn't there something wrong? B should be better than J regardless of A."

No, nothing is wrong. The people who voted A prefer FGHIJ over BCDE, and either there are a lot of them or FGHIJ and BCDE were very closely matched to begin with.

This is an academic issue, but we're at the point that nobody but us is listening so I'll explain anyway.

You just don't get it, which is OK. The point is that in a fair system, the presence of a spoiler should not matter. If A wins over B, but then C runs and that means B wins over A, that isn't fair. When people prefer A over B, the voting system should show that they prefer A over B regardless. It's a basic principle of fair voting systems.

We have violated that, and it's OK. Arrow's theorem (you can look it up easily) says that several obvious basic principles of fair voting conflict, and we can't require them to all be true at the same time. We have violated this one and it has to be OK. It bothered me until I noticed that.

#1016 ::: felice ::: (view all by) ::: May 10, 2015, 08:20 PM:

J Thomas @1015: "You just don't get it, which is OK. The point is that in a fair system, the presence of a spoiler should not matter. If A wins over B, but then C runs and that means B wins over A, that isn't fair. When people prefer A over B, the voting system should show that they prefer A over B regardless. It's a basic principle of fair voting systems."

I get it, but I don't think it applies to multi-winner systems. What we've got here is A+D wins over B+E, but if C runs, C+B wins over A+D+E - it's not meaningful to say one specific subset of the finalists wins over another specific subset of the non-finalists. And C isn't a spoiler, it's a winner; I don't think the presence or absence of any particular non-finalist can make a difference to the outcome, can it? SDV-LPE isn't a rank order voting system; while it does make use of two different types of ranking, the output is a set of five winners, order irrelevant and debatable (which is "first", A with 10 points and 15 nominations, or B with 9 points and 16 nominations?)

#1017 ::: Tammy Coxen ::: (view all by) ::: May 10, 2015, 08:40 PM:

@989 - I'm a member of the SMOFS list and there are a lot of business meeting attendees on there (it's a mailing list for conrunners, especially Worldcon conrunners). When you think you have a proposal/ description that's ready for prime time, I can post it over there and report back on the feedback.

#1018 ::: Tammy Coxen ::: (view all by) ::: May 10, 2015, 08:41 PM:

@987, 989 and etc - I'm a member of the SMOFS list and there are a lot of business meeting attendees on there (it's a mailing list for conrunners, especially Worldcon conrunners). When you think you have a proposal/ description that's ready for prime time, I can post it over there and report back on the feedback.

#1019 ::: P J Evans ::: (view all by) ::: May 10, 2015, 08:59 PM:

There's a reasonably good explanation at 958. It isn't technical, but it's clear.

#1020 ::: Keith "Kilo" Watt ::: (view all by) ::: May 10, 2015, 09:12 PM:

I've been taking fairly extensive notes throughout all of this. I've got plans this evening, but I'll type up a summary tomorrow morning before the deadline.

Kilo

#1021 ::: J Thomas ::: (view all by) ::: May 10, 2015, 11:15 PM:

"You just don't get it, which is OK. The point is that in a fair system, the presence of a spoiler should not matter. If A wins over B, but then C runs and that means B wins over A, that isn't fair. When people prefer A over B, the voting system should show that they prefer A over B regardless. It's a basic principle of fair voting systems."

I get it, but I don't think it applies to multi-winner systems. What we've got here is A+D wins over B+E, but if C runs, C+B wins over A+D+E - it's not meaningful to say one specific subset of the finalists wins over another specific subset of the non-finalists.

Consider the possible situation I described, where ABCDE wins, but without A running, FGHIJ wins. (I think it's possible. I know something similar is possible.) Every one of BCDE was ranked about FGHIJ with A, but ranked behind FGHIJ without A. I think it's meaningful to say that one specific subset of the finalists wins over another specific subset of the non-finalists in this case.

It could be claimed that for one to replace another they must be "close" according to the particular system that decides which wins. But they don't have to be "close" in votes at all. And for both, the scores can change throughout the elimination process. They don't have to be "close" in scores, ever. That claim sounds good said in English, but I'm not sure it means anything.

Anyway, none of this matters. I say it doesn't matter that we give up this kind of fairness -- many voting systems give it up. If we chose some alternative probably that would be unfair the same way.

And it's certain that this is the voting system that will be presented to WorldCon, and the only system that can be presented this year. It bothers me some that several alternatives which might be better were rejected with no analysis, but them's the breaks. This one is probably good enough.

#1022 ::: P J Evans ::: (view all by) ::: May 10, 2015, 11:46 PM:

1021
You're still thinking that nominations involve ranking the nominees.

NO.

That is not part of nominating. The final ballot ranks nominees, but the nominations are not ranked.

#1023 ::: J Thomas ::: (view all by) ::: May 11, 2015, 12:21 AM:

@1022 P J Evans

1021
You're still thinking that nominations involve ranking the nominees.

No, of course not. Nominations result in a group of winners, and a group of losers. The winners are ranked as "better" than the losers. The losers are eliminated (mostly) one at a time with the implication that one that was eliminated earlier is worse than one that was eliminated later. It makes a certain sense to say that the (usually) five winners are ranked by the number of votes for them, and in fact they are required to later be published in that order, and with those numbers.

But the ballots do not rank the nominees at all, now or in the only ML proposal for change, except that there has been a suggestion to use the ranking on ballots to decide ties. That suggestion has not gotten much support.

#1024 ::: Steve Halter ::: (view all by) ::: May 11, 2015, 10:20 AM:

"- How to handle withdrawn nominations"

For the final list of nominations, I don't see how there can be withdrawn items. The final run of nominations -> ballot is not done until the nomination period is closed. At that point we have a fixed list.

The production of some form of ongoing "long list" would be a separate topic. I would prefer not to muddle the two.

#1025 ::: Keith "Kilo" Watt ::: (view all by) ::: May 11, 2015, 10:31 AM:

Steve@1024:
For the final list of nominations, I don't see how there can be withdrawn items.

That may not be clearly stated. What I mean is, suppose nominations are closed, and we have made the final ballot. The administrators then call all of the people of the final list. It's possible (and has happened) that someone on the final ballot would decline the nomination. We have to decide how to handle that case. This has to happen no matter which nomination system is used, actually. With the current system, they just take the next person on the list.

Kilo

#1026 ::: P J Evans ::: (view all by) ::: May 11, 2015, 11:01 AM:

1023
No, they aren't. I don't think you grok the process.

#1027 ::: Buddha Buck ::: (view all by) ::: May 11, 2015, 11:22 AM:

P J Evans @1026:

I'm not sure what you are referring to, specifically, with "No, they aren't". The antecedent to your only pronoun isn't clear.

(Abi: feel free to ignore this post when summarizing the final 24 hours of this thread)

#1028 ::: Steve Halter ::: (view all by) ::: May 11, 2015, 11:45 AM:

Keith@1025:Thanks, that makes sense. In that case, I would tend to go:

0: rerun everything from scratch
5: rerun and take the first new nomination
10: take the next candidate from the original run

#1029 ::: J Thomas ::: (view all by) ::: May 11, 2015, 12:28 PM:

Why vote now, instead of wait for the new discussion?

#1030 ::: Stuart Hall ::: (view all by) ::: May 11, 2015, 12:50 PM:

All of the discussions so far have been about changing the nomination system. I have been thinking about supplementing it.
A lot of the nominated short fiction each year comes from magazines. Some of these magazines run a reader survey that picks out the most highly rated piece published in the previous year. The winner from each of these surveys could be added to the ballot in the appropriate category (novella, novelette or short story) if it has not already been nominated. The additional works would all be short (it is possible a serialized novel could win a survey, but unlikely) and should be worthy of consideration. Obviously this would only work for the short fiction categories, but these are the most problematic. The additional reading required before voting should not be too onerous and the varied demographics of the magazine readerships would ensure that everyone could find something on the ballot to suit their taste. Of course there would have to be some rules governing which magazines would be eligible. The big advantage would be ensuring the presence on the ballot of works that could not be slated.

#1031 ::: Jameson Quinn ::: (view all by) ::: May 11, 2015, 01:28 PM:

@1030: I think that's a good idea, but it could just as well happen outside official channels. That is, each magazine could time its poll so that the results were available before Hugo nominations, and then release the winning work to all Hugo voters, as a sort of "pre-voters' packet". Surely some would decry the evil attempts to manipulate, but as long as nominators continued to use their honest best judgement as to what to nominate, I personally think it would be legit.

However, I think that this is probably not the best place or time to get too deeply into this discussion.

#1032 ::: Cassy B. ::: (view all by) ::: May 11, 2015, 01:31 PM:

Stuart Hall @1030, interesting idea, but who decides which magazines qualify to have their "reader-chosen best" automatically nominated? (I hear the cry "conspiracy" echoing to the rafters...) Would you decide to include by circulation, or some other criteria? And how many magazines are we talking about? Could we end up having to choose from 20 entries (five normally nominated; fifteen "best-of-magazine" nominated)? Would this be for all three short-fiction categories or just for short stories?

I'm not dismissing your idea out of hand, but I'd very much like to hear more about how it would be implemented fairly.

Cassy

#1033 ::: Jameson Quinn ::: (view all by) ::: May 11, 2015, 01:58 PM:

My contribution to the new post:

I've created a public Google doc to hammer out a proposal text and FAQ for SDV-LPE.

There have been numerous proposals in this thread which do not fall under the SDV-LPE umbrella. In my opinion, they have turned out to be either off-topic or dead ends. This is not a criticism; several of the proposals were my own, and I thought and still think that they would have been good ideas. But it seems the community is very reluctant to experiment with anything that goes much beyond fixing the immediate problem, and I can respect that.

So I think the remaining order of business is to wrap up the SDV-LPE proposal, either using the doc above, or through other means.

#1034 ::: Keith "Kilo" Watt ::: (view all by) ::: May 11, 2015, 02:31 PM:

Abi: I think there are four sections that need to be preserved in the next thread. I apologize in advance for formatting and typos. I just cannot seem to edit effectively in the tiny font of the preview screen!

All: Please note that I'm just reporting the current state of the code and proposal; see the To-Do list for items that still need to be taken care of or changed.

=================================================
[Remaining Issues “To-Do” List:]
- Name of the system
- How to handle ties
- How to handle withdrawn nominations
- Best way to present at the business meeting
- The final formal proposal language itself

=========================================
[Plain-Language Explanation of SDV-LPE]

Least Popular Elimination (formally called "single divisible vote with least popular eliminated" or SDV-LPE for short) is very simple and straightforward.

- You have one nomination "vote", which we'll call one "point" to avoid confusion.

- You can distribute that nomination "vote" among as many works as you feel are Hugo-worthy, and it will get divided among them equally. So, if you nominate two works, each gets half a point, if you nominate three works, each gets one third of a point, etc.

- All the points for each work from all the ballots submitted are added together, and the two works that got the least number of points are compared with each other. One of these works is the least popular and will be eliminated.

- For those works that are eligible to be eliminated, we compare the total number of nominations they each received (that is, the total number of times that work appeared on anyone's nomination ballot). The work that received the fewest number of nominations is the least popular and now completely vanishes from the nomination process as though it never existed.

- We start over for the next round, and repeat the process, however, if one of your nominations was eliminated, you now have fewer works on your nomination ballot -- so each one gets more points since you aren't dividing your vote among as many works.

Ties (in Points):
- This isn’t really anything different. Basically, if there is a tie for least or second-least number of points, all of those works are eligible to be eliminated.

Ties (in Number of Nominations):
- If two or more works are tied for appearing on the fewest number of ballots, the tied works with the lowest point total will be eliminated. (In other words, if we’re tied for number of nominations, we’ll go back to comparing points for this case.)

- If there is a tie for appearing on the fewest number of ballots as well as for lowest point total, then all members of the tie will be eliminated, unless that leaves the final ballot with fewer than five works; in that case, keep all the members of the tie. (In other words, if we’re tied in both comparisons, in general we’ll just eliminate all the tied works.)

Why does this help? In the past we have strictly counted the number of nominations and the top five works were put on the final ballot. Because SF fandom typically nominates a diverse range of works, it was easy for an organized slate to make it so that no other works made the final ballot.

Using this system, fandom isn't penalized for nominating a wide variety of works. If you nominate something that ends up not having a chance to make the final ballot, then your remaining choices automatically get more of your support instead of just being wasted. In other words, you can safely nominate -anything- you feel is Hugo-worthy. If enough people agree with you, it will make the final ballot. If they don't, that's okay - when that work is eliminated, your other choices will have a greater chance of making the final ballot. In this way, by eliminating the least popular candidates, fandom slowly converges to a consensus as to which works should be voted on to be the final Hugo winner.

How does this eliminate slates? It doesn't, not completely -- nor should it, as slate proponents have as much right to representation as any other voter. However, by nominating a full slate of works, they have weakened their votes by spreading their points out among the five works on the slate. Since the rest of fandom is slowly increasing the points given to their choices, it's not possible for slates to control the -entire- ballot. It is very likely, however, that they will get some of the final ballots slots -- and again, this is entirely fair and appropriate, since a large number of people are supporting those works. In the end, it is the final ballot that will determine the winner -- and the voting system for the final winner is completely unchanged by this system. With this system, a significant fraction of the final ballot will have been determined by independent members of fandom.

=================================================
[FAQ’s]
1. How does this system eliminate slate or bloc voting?
It doesn't, exactly, nor should a work be automatically eliminated just because it appears on a slate. On the other hand, any slate which nominates a full set of five works will find that each of its nominations only count 1/5 as much. With a large enough support behind the slate (five times as much), the slate may still sweep a category; however, if that many voters support the slate, they arguably deserve to win, and no fair and unbiased system of nomination will prevent that. The answer is, simply, to increase the general pool of voters. Regardless, with SDV-LPE, slates will never receive a disproportionate share of the final ballot, as occurred in the 2015 Hugos.

2. What if there are multiple slates (slate wars, “parties”, etc.)?
As with a single slate, the more works that anyone nominates, the less their votes count for each work. The end result is that even multiple slates are unable to sweep the nominations.

3. What happens if a genuinely popular work is nominated by a group of unrelated people?
If it is genuinely popular, the system will still select that work for the final ballot.

4. What happens if a genuinely popular work also appears on a slate?
Even if it is on a slate, if the work garners support from individuals – particularly if they list it as their only nomination, or with just a few nominations – then the system will select that work for the final ballot.

5. Isn't it true that any voting system can be gamed (or strategized, etc.)?
Yes, there is a theorem which proves that all voting systems must have inherent flaws. The objective is to choose a system whose flaws are not in an area of concern to the electorate.

6. What are SDV-LPE’s flaws?
In very rare cases, it is possible for eliminating both members of a tie to change the final ballot slightly from what it would be if the tie were broken so that only one member was eliminated. This situation was extremely rare using realistic data, so the statistical probability of how ties are handled affecting the results is extremely small.

7. What are SDV-LPE’s benefits?
Simply put, it reduces the power of bloc voting without eliminating the chance that works appearing on slates will make it to the final ballot. Conversely, it makes it very difficult for slates to prevent non-slate works from appearing on the ballot.

8. Couldn't slates just recommend a single work for a candidate, and it will automatically appear on the final ballot?
Yes, that is certainly a viable possibility – it’s also completely fair. It does not force any other works off of the ballot, and the final Hugo winner is determined by the same voting process we have always had. Just appearing on the ballot isn't a guarantee of winning a ballot. However, if a large section of fandom strongly believes that a work deserves a Hugo nomination, then it should, in fact, be represented on the ballot.

9. What happens with a large field with no stand-out favorites when a slate votes?
Even in this case, we were unable to find a simulation in which no non-slate works appeared on the final ballot. Slate works did receive a larger proportion of nomination slots than they did otherwise, however, again, this could be considered a fair and valid result. If there was no general favorite, then voters really had no collective preference.

10. Isn't this system too complicated for the average voter to understand?
No, it’s actually quite simple and straightforward, both in terms of voter instructions and in how the system operates. Essentially, the total number of points for each work are totaled (and this will usually be the sum of fractional points). These points are used to determine our two candidates for elimination, since the voters felt the least strongly about them. We then look at the number times the two works appeared on any ballot. The work that appeared the least number of times must be the less popular of the two, so is eliminated. This process continues until the five finalists remain. Note that this is not that different from the STV process used in determining the final Hugo winner once the finalists have been selected.

11. I think we should just increase the number of nomination slots on the final ballot to (for example 6), and decrease the number of slots a voter can vote for to a smaller number (for example, 4). Wouldn't that be simpler and easier?
Unfortunately, this simply means that the largest slate will receive four of the nominations and the next largest will receive the remaining two. It doesn't solve the problem of forcing works off the ballot that had a chance to win the final election. Keep in mind that SDV-LPE will work with this (or most any other) change as well, so one does not preclude the other.

12. I think we should set up a committee to handle these situations as they occur. The committee would be empowered to add nomination slots or throw out slate-influenced ballots as required.
This could work. The problem is that now you have a small group of people who serve as literal gatekeepers to the Hugo Awards. In spite of the word on the Internet, this has never been the case in the past. Establishing it now means that those groups who believed it existed in the past will now be correct. Ultimately, human judgement is fallible. The fairness of a committee’s decisions will forever be subject to opinion. The end result is that the prestige of the Hugo Awards will forever be tarnished.

13. I think we should use [insert other mathematical voting system].
We considered essentially every applicable type of voting system currently in the literature, guided by two experts in the field. It should be kept in mind, however, that the goals and requirements for choosing a set of representatives in a political situation are different from those for choosing a set of Hugo finalists. Some of these systems do, in fact, have positive properties that speak for them. None of them were as simple or as intuitive as SDV-LPE, yet SDV-LPE meets all of the stated goals for a Hugo nomination system.

14. Won’t SDV-LPE be complicated to code and implement?
Actually, no. One of our non-experts coded a full simulator for the system in a matter of days. A full web-based app would not be much more difficult to handle.

15. Wasn't this system just designed by Social Justice Warriors to block the Good Stuff?
It is true that much of the discussion for this system occurred on Teresa and Patrick Nielsen Hayden’s “Making Light” discussion board, and it is also true that groups such as the Sad Puppies and the Rabid Puppies consider TNH and PNH to be The Enemy, and therefore completely biased and not to be trusted. Other than serving as moderators, TNH and PNH had no real input in the discussions of the system, however. Those of us who worked on the system were very clear that our goal was not to keep the Sad/Rabid Puppies off of the Hugo ballot and that any system which specifically targets any type of work is inherently wrong and unfair. One of the members of the group is a retired US Naval officer, a combat veteran, a certified Navy marksman, a Christian, and considers Robert Heinlein to be the greatest science fiction author who has ever lived. In short, he is exactly the Puppies’ demographic. But any slate, of any sort, be it a Sad Puppy or a Happy Kitten of Social Justice, breaks the Hugo Award because a small percentage of voters can effectively prevent any other work from appearing on the final ballot. This is a major flaw in the Hugo nomination system, and it is a flaw that must be fixed if the integrity of the award is to be maintained. Politics should play no role whatsoever in whether a work is Hugo-worthy or not.
=================================================

[Current Draft of Proposal Language]
Short Title: Change Nominations to Single Divisible Vote, Least Popular Elimination

Moved, to amend section 3.8 (Tallying of Nominations) as follows:

Section 3.8: Tallying of Nominations.

3.8.1: Except as provided below, the final Award ballots shall list in each category the five eligible nominees receiving the most nominations. If there is a tie including fifth place, all the tied eligible nominees shall be listed.

3.8.2: The Worldcon Committee shall determine the eligibility of nominees and assignment to the proper category of works nominated in more than one category.

3.8.3: Any nominations for “No Award” shall be disregarded.

3.8.4: If a nominee appears on a nomination ballot more than once in any one category, only one nomination shall be counted in that category.

3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.

3.8.6: The Committee shall move a nomination from another category to the work’s default category only if the member has made fewer than five (5) nominations in the default category.

3.8.7: If a work receives a nomination in its default category, and if the Committee relocates the work under its authority under subsection 3.2.9 or 3.2.10, the Committee shall count the nomination even if the member already has made five (5) nominations in the more-appropriate category.

3.8.8 The final Award ballots shall list in each category the five eligible nominees as determined by successive rounds of elimination, where each nominator gets a single vote, divided equally among their nominations. For example, if only one work is nominated in the category, that nomination gets a full vote. If two works are nominated in a category, each nomination gets 1/2 of a vote. If three works are nominated in a category, each nomination gets 1/3 of a vote, etc.

3.8.8.1 In each round, the two works (or more, in the case of a tie) with the least total number of (fractional) votes will be compared. Of those works, the one(s) that appear on the fewest number of ballots will be removed from all nomination ballots.

3.8.8.2 In the event that two or more works are tied for appearing on the fewest number of ballots, the tied works with the lowest point total will be eliminated. If there is a tie for appearing on the fewest number of ballots as well as for lowest point total, then all members of that tie will be eliminated.

3.8.8.3 If (due to a tie) elimination would reduce the number of ballots to fewer than 5, then instead none of the works should be eliminated and all remaining works appear on the final ballot.

3.8.8.4 For the next round, votes are reassigned to each work as in 3.8.8. (Example: If a ballot contained five nominations originally and one of those nominations is eliminated, each of the four remaining nomination now gets a 1/4 of a vote)

Section 3.11 Tallying of Votes

3.11.4: The complete numerical vote totals, including all preliminary tallies for first, second, … places, shall be made public by the Worldcon Committee within ninety (90) days after the Worldcon. During the same period the nomination voting totals shall also be published, including in each category the vote counts for at least the fifteen highest vote-getters and any other candidate receiving a number of votes equal to at least five percent (5%) of the nomination ballots cast in that category, but not including any candidate receiving fewer than five votes. During the same period a record of at least the last fifteen rounds of the selection process for each category shall also be published.

Submitted by: (insert members here, ideally including at least one attending member who will be at all the relevant WSFS meetings. A proposing member gets the privilege of speaking first to the pro side of the proposal)

Commentary: In the past we have strictly counted the number of nominations and the top five works were put on the final ballot. Because SF fandom typically nominates a diverse range of works, it was easy for an organized slate to make it so that no other works made the final ballot.

Using this system, fandom isn't penalized for nominating a wide variety of works. If you nominate something that ends up not having a chance to make the final ballot, then your remaining choices automatically get more of your support instead of just being wasted. In other words, you can safely nominate -anything- you feel is Hugo-worthy. If enough people agree with you, it will make the final ballot. If they don't, that's okay - when that work is eliminated, your other choices will have a greater chance of making the final ballot. In this way, by eliminating the least popular candidates, fandom slowly converges to a consensus as to which works should be voted on to be the final Hugo winner.

#1035 ::: Joshua Kronengold ::: (view all by) ::: May 12, 2015, 02:25 AM:

I'm pretty happy with where things are going at the moment. Kilo's last post in 1034 had a good statement of issues, and I didn't catch any red flags in his hugo proposal.

Re name, if we want an acronym that means something, how about:

One Notional Equally-divided Vote, with Oppositional Test Eliminations?

(ie, ONEVOTE)

I've always favored "rerun the vote with declined candidates eliminated, but only fill as needed" when people declined as a rule, since it ends up with the fairest overall result (and withdrawing Hugo nominations is a can of worms we never need open). I'd be fine (5 vs 10) with just taking the next item on the list.

Re tiebreakers: I'm reasonably fine with no tiebreaker (as the system was originally conceived), but also entirely fine with the current "just score" version, or using rank as a not-really-random (but close enough while being verifiable) tiebreaker. It's probably best to just have "just score" as tiebreak despite potential issues, because it makes things much easier to independently verify and keeps things simple.

Based on where things are going here, I'd be perfectly happy to propose or cosign a resulting ammendment (and will be at the BM). I'm a bit of a troublemaker, and not the best at public speaking, so if we have someone who is going to be there who is better with the opening speaches, they might be better at being the first speaker in favor.

#1036 ::: Idumea Arbacoochee, Playing Atropos ::: (view all by) ::: May 12, 2015, 03:04 AM:

OK, I've posted, and will be closing this thread.

Flag me if I've got something wrong, missed a comment you want included, or whatever?

Choose:
Smaller type (our default)
Larger type
Even larger type, with serifs

Dire legal notice
Making Light copyright 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 by Patrick & Teresa Nielsen Hayden. All rights reserved.