We Could Use a Tiny Bit of Help With Our Feedback Tool.

[Aaron Stanton]

I have a favor to ask.  Dan, one of our statisticians, has been going over our the results of our Feedback Tool.  The Feedback Tool is designed to let humans rate a paragraph's score on action, density, pacing, dialog, and description, which we then analyze to help improve our formulas.  This helps us make sure that our "action" being measured by the computer is the same thing that most people consider to be "action" in reality.

We have a fair number of feedback results in that tool already - roughly about 3,000 ratings.  But we recently updated the code to track a new metric, something that we were not tracking before.  So we need more data points now, and we need it in a fairly short period of time.  As in, two weeks, if possible!  Not much time at all.

He asked me if I'd put a request out for some help, so here I am. 

This is just a really quick request - if you're a fan of the project and have a few seconds, drop by the Feedback Tool (http://beta.booklamp.org/feedback/), sign in, and rate a few paragraphs for their different metrics.  It's easy, and extremely useful to us.  You don't have to do all of them (there's lots), and every paragraph is from an out-of-copyright source.  Rate what you have time for, and that will be simply brilliant. 

Thanks again!

Aaron

[Stephen Rollins]

Ooh, another chance to help out! 

[Kyle Gruel]

Is there a point in which we ought to stop?  I hit 300 points and realized it seemed to be recycling alot.

[Daniel Bowen]

Short Answer: No

There are are only meant to be 50 passages in the survey, so yes it was recycling a lot.  Thank you for that input though, both here and the survey.  It may actually be time for a version two, depends on how we might want to improve on it outside selecting new (?longer?) passages.

I have to ask you about the survey, if you don't mind.  Taking the survey for 300 rounds has to have left an impression of some kind.  Do you have any suggestions? ... interface ... text passages ... more stars... maybe genre specific surveys, ... yeah, I like that idea.

What are your thoughts?

Dan

[Aaron Stanton]

Kyle,

Thanks for putting so much time in.  As Dan mentioned, it only has about 50 snippets, but you're rating them for different things - sometimes density, sometimes pacing, etc.  After 300, you're going to have hit most of them.  As far as I'm concerned, you've done your duty above and beyond. 

I think Dan's comment about time for a refresh of the tool, meaning more snippets, might not be a bad idea.

Aaron

[Stephen Rollins]

I was in the same boat as Kyle when I took a shot at it.  Filled the bar all the way up (what was with the weird messages that happened sometimes?  o.O) and the system kept on giving me more passages to rate... XD

[Therin]

Having done several hundred points worth myself - it's a bit like tetris you just keep going expecting it to finish  - it is kind of relieving to be told that there isn't an end in sight - yeah, I know, perverse.  But when you were sort of pushing through to get to the done and dusted stage and it just kept handing them out...

As to suggestions on improving - why does it keep score if there's no finish line? Are you manipulating the Alpha personalities out there?     I know it sucked me in.

Ah well.  I did really like the continual description of what I was supposed to be measuring - "Density is..."  That was an excellent idea.  The repetition was amazing but if nothing else it has strengthened my knowledge of the classics?  If I read one more section on Moby Dick!  Still, the Alice in Wonderland was enjoyable - I never have got around to reading the book.

But have you considered comparisons?  As a method of fine tuning.  You're asking the reader currently to maintain an internal standard on a variety of measures.  This is harder than merely asking them to measure 14 passages on density and then 14 passages on dialogue etc...  They are continuously recalibrating the internal scale and it will be less accurate.   If you were to specifically ask, as between two passages, "which contains more dialogue?",  "Which has a greater density?" or "Which passage is most descriptive?"  you may get more reliable feedback than asking the reader to judge each passage alone.  An intelligent system could also build on their knowledge; compare what they said was a three with an unknown - get a higher rating and realise that it must be four or five? 

Just a suggestion.    It's just I found myself thinking back to previous answers to justify newer ones and a comparitive measurement system would simply ask me to choose between two or three quotes.

[Daniel Bowen]

Therin,

... "which contains more dialog?" ... That's an interesting thought, and I like it.  I will certainly think about what precisely we could find from it, but that's great idea.

Thanks for sharing your thoughts.  I think you may be on to something.

Dan