To rate or not to rate

17 Comments

It feels like it’s time to come up with another blog post on rating and commenting on jovoto.

The jovoto system is based on this two way mechanism of giving feedback and deciding which ideas are the best ones. It’s therefore essential that everybody understands the importance of ratings and comments and has the chance to express feedback – be it positive or negative.

To cover some principles:

  • every community member can rate and comment as much as he/she wants to
  • the goal is to rate and comment as many ideas as possible as fair and objective as possible
  • ratings aren’t publicly displayed, only ratings of 1 and 2 as well as 9 and 10 stars are displayed under „other ideas“ in the user profile
  • submitted ratings can be changed at all times until the end of a contest
  • ratings of the jovoto team do not count

More information can be found under Votingsystem and FAQs.

Vote3

Luckily rating isn’t that complicated on jovoto!

Picture by John C Abell

What we’ve witnessed in the past was ongoing discussion about some of the core features related to rating and commenting.

  • Should it be mandatory that a comment must be made when a bad rating is given?
  • Should authors see in detail what ratings their idea got?
  • Should rating be transparent?
  • Should community members who participate in a contest not have the right to rate in that contest?

We do get different feedback on these questions. There are almost 4.000 community members, so obviously there is not one single solution that pleases everybody. All we can do is to evaluate the community’s feedback and base the decisions we have to take on this ground.

Another discussion that comes up regularly is about so called „fake user“, „basher“ and „micro communities“. We absolutely understand worries about such phenomena – we have them as well. That’s why we do everything that’s in our power to prevent and punish such violations of the rules. We check all ratings and EVERY alarming behaviour is followed up on. Our system proved to be very effective and manipulations of the rating system have become almost impossible.

However it seems to have become some kind of trend to throw in such accusations whenever one doesn’t like the result of a contest. We appreciate every “report misuse“ message and take them seriously, but we can’t understand why discussions are constantly started under ideas of users that are not even the accused ones. This kind of behaviour can only be understood as willfully disrupting.

Moreover, all these questions can more or less be solved by one simple recipe: more rating and commenting.  More ratings stabilize the entire system, making manipulation efforts futile (Of course backed up by our constantly improving rating monitoring) . Instead of mistrust and accusations, more ratings should be the solution.

We’re curious to get some feedback on this whole issue.

Vote2Picture by Dean Terry

ABOUT THE AUTHOR
Olof
Olof

Olof Ekman is Head of Community Management at jovoto and spends his time making the the jovoto experience as fun & pleasant as possible for it's community members!

Tags

17 Comments

    Ok i start –

    Should it be mandatory that a comment must be made when a bad rating is given?
    I don´t think so. It would be eve more negative for the idea and the user, that submitted it.

    Should authors see in detail what ratings their idea got?
    Hm. Well – i think we trust you ! : )

    Should rating be transparent?
    No. This would produce a loooong discusiion betwenn hundrets of users about who how & why!
    I´m still a fan of secret elections.

    Should community members who participate in a contest not have the right to rate in that contest?
    Sounds logical. But the contributors are highly involved and might have a good overview of the submissions. And if i contribute in a contest i appreciate it to have a little influence on therating as well.

    *SOME ADDITIONAL IDEAS*

    As i am becoming o fan of “THEFUNTHEORY”, i think you could spend some thoughts on motivation by fun.

    Can you get more poeple to do ratings by making it fun to do ?
    – What about some funny sound-feedback for ratings, that you can turn on or off ?
    (A rocket-start, opening a beer, a dog, church-bells, …?)

    What about Airberlin paying an (extra) award in (stand by) MILES in the upcoming contest ?

    Good luck with your good work ! = )

    its great you make a blog on this topic. thank you.

    hope some more will give feedback… in english is hard for me, but i try;)

    its very helpful to repeat again and again how important it is to give voting to every idea, not only to your preferred and favourite buddies. this voting should be as objective and fair as it could be, without being manipulated be any other comments.

    but the most important is to look at the BRIEFING before you vote 😉

    fairness means, that everyone has spent time and energy to develop his idea. if anyone don’t like it, no problem, but please comment it in a most sensible (sensitive?) not aggressive way….

    i prefer critical (professional) comments and don’t want to be ignored. the most bad thing for me is an abslutely ingnorated idea.

    i would like to have the community-members votings count also. why not. it woud be a professional voting further…

    maybe the client could give one voting as well? its only on more voice….(don’ kill me for this)

    let us think about motivation to vote. not just fun but …. maybe we should start a contest about this….?

    Es ist höchste Zeit das dieses Thema ausführlich besprochen wird.

    Ich befürworte ein transparentes Bewertungssystem. Ist ein bewertendes Community Mitglied von der unzureichenden Qualität einer Arbeit überzeugt ist er auch in der Lage seine Bewertung zu begründen und dazu in einem Kommentar Stellung zu nehmen.
    Denn bei Veranstaltungen dieser Art ist es selbstredend Arbeiten konstruktiv(!) zu kritisieren. Es wird viel kommentiert aber nur selten konstruktiv.
    Um hilfreiche Kritik voranzutreiben müsste man sie zur Not forcieren. Denn Vorgaben wie „schwache Idee“ fördern das bequeme und schnelle Bewerten führen aber oft zu voreiligen Entscheidungen. Deshalb sollte die Bewertung eines jeden Community Mitgliedes begründet werden.

    Um eigennütziges Fehlverhalten zu unterbinden müssen die Mitglieder kontrolliert werden. Da führt kein Weg dran vorbei. Und ein transparentes Bewertungssystem ist womöglich die demokratischste Art.

    Sinnvoll ist es auch denen die ihre Arbeiten einreichen das Abstimmen zu verbieten. Mit dem hohen Preisgeld vor Augen kann eine objektive Bewertung kaum stattfinden. Man hat das eigene Glück vor Augen und bewertet keineswegs so selbstlos wie wenn man nicht mit einer eigenen Arbeit im Wettbewerb vertreten ist.

    I find this discussion extremely important thats why theres an englisch version as well.

    I am in support of a transparent rating system. A community member should be able to justify his / her vote and give constructive feedback. Unfortunately comments seldom push new ideas.
    Given formulations like „schwache idee“ (lousy idea) stimulate hasty and unthought of actions and perhaps even laziness. That‘s why everyone willing to rate must explain his / her decision.
    In addition a transparent rating system probably is the most democratic and easiest way to check on suspicious behavior.
    Last but not least: in order to prevent selfish acts members who are submitting ideas must be excluded from rating. The awards in mind it is impossible to stay fully objective when rating the ideas of other jovoto members.

    i like it very much here and think you are doing a GREAT job!
    thnak you for that. averyone is very freindly here and i like how much everyone is participating in the contests and in term sof commenting and giving suggestions and stuff.

    …i read jovoto is using sort of complicatet uuuh algorism vor the votes…..
    but what happens if….
    …. an entry gets 10 votes with 9 and 10 votes with a 5 rating wich would get it an overall 7
    an another contest gets only 3 votes with a 9 ???
    the one voted three times wins? doesnt it?
    just a thought and just an (a bit extreme) example

    because i wont vote anymore “bad” or “contra” votes i did in the past yes, but i see no use in it anymore.
    i comment some instead…. or should i really vote everything even if i dont like it at all??? dont wanna do that

    Mal von der Qualität der Votes abgesehen (die man nie 100% kontrolieren können wird), ist es momentan auch mit der Quantität etwas im argen. Wenn ich mir meine Einreichungen mal anschaue, voten nicht einmal 5% der Betrachter einer Arbeit. Ich denke bei anderen ist es ähnlich. – Im übrigen finde ich die Idee zwar prinzipiel gut das man nicht bewerten darf, wenn man selber Arbeiten eingereicht hat. Allerdings würde man damit die aktiven Mitglieder bestrafen. Alles in allem halte ich es frei nach Churchill: “Die Demokratie ist die schlechteste aller Möglichkeiten, abgesehn von allen anderen.” — Apart of the quality of the votes, i like to menton that the quantity is some kind of problem too. Only aprox. 5% of the viewers of a projekt are willing to vote. I`ve been told that jovoto will come out with a “karma level” – this schould be a great idea. Anway, i like the idea that you can not vote if you submit any work to a competion, but this would penalize useres who are highly involved. So, its kind of difficult, but the system jovoto has is not perfect but its the best you can get.

    Thanks for all your comments!

    So far there are two things it seems like everybody agrees on:
    1) That there should be more ratings and constructive criticissm
    2) that rating and commenting should be made more fun and that some kind of reward should be given out for “outstanding performances in voting and commenting.

    Well, as hiro42 said, we’re working on a new karma system that also takes rating activity and constructive feedback (Useful comment) into account. What would you as community members like to have as remuneration for high karma?

    What would make rating and commenting more attractive to you?

    What people don’t agree on is if users shouldn’t be allowed to rate in a contest they participate in. I think two of the most important arguements for NOT taking the right to vote away if participating in a contest have been pointed out: you penalize the active users who make the most input and you shut out those people who dealt with the topic at hand most intensely. Concerning Tyler’s arguement that it’d be “impossible to stay fully objective when rating the ideas” of competitors, I have to once again point out that MORE RATING makes manipulation efforts futile. So even IF a competitor is not rating a 100% fair and obejective, the sheer number of votes evens this out. Moreover, our monitoring system punishes extreme forms of such behaviour and manipulative votes are being deleted.

    @hannes dolde: the model you chose is a great example of how the system does NOT work. The algorithm is there to countervail exactly such events. See details here https://www.jovoto.com/de/kreative/gut-zu-wissen/votingsystem/

    @hiro42: the number “angesehen” counts in EACH time a community member looked at your idea, so e.g. maybe I checked up on your idea 35 times, the 35 times are counted in the total number displayed. So the 5% calculation doesn’t come out even.

    I am curious to here more opinions and proposals! And I am curious to see whether this discussion helps to bring the importance of rating and commenting on people’s minds.

    good discussion in here. Although my votes (as a jovoto team member) don’t count at all I will get down and vote now 😉

    Den Vorrednern möchte ich mich teilweise anschließen. In dem Punkt dass eine Punkteverteiltung nicht transparenter sein sollte stimme ich Jesko zu. Hier geht es ja auch um Demokratie, die am besten angesehene Idee gewinnt usw. und Wahlen laufen im Allgemeinen im Geheimen ab … und das jovoto-System ist ja ähnlich angelegt.

    Wenn das System total transparent sein würde, liefe man möglicherweise Gefahr dass die Votings an sich nicht nur weniger werden könnten, sondern dass es auch nur noch “Kuschelvotings” geben könnte, da man ja selber nicht schlecht bewertet werden möchte.

    Das aktuelle Votingsystem finde ich in im Grunde gut. Man sieht die Lieblingsideen und die als schlecht bewertete Ideen. Jede einzelne Bewertung zu begründen (und zu sehen) halte ich für übertrieben. Eine optische “Massenbegründungsbewertung” würde den Betrachter überfordern, wer hat schon Zeit und Lust 300 Votings in einem Profil nachzulesen?

    Hingegen finde ich es sinnvoll (da gehe ich teilweise mit “tyler” überein) dass als schlecht bewertete Ideen einer kurzen Begründung bedürfen sollten, die man dann logischerweise auch beim “Basher” nachlesen kann.

    Wenn man an einem Contest teilnimmt, sollte man auch selber abstimmen dürfen. Wäre dies nicht so, hätte man optisch viel eher ein “gegeneinander” als ein “miteinander” in einem Contest. Sowohl in den Kommentaren als auch in der Bewertung untereinander.

    Was ich persönlich sehr schön finde, ist dass man seit längerer Zeit seine Bewertung ändern kann, je nachdem wohin sich eine Idee entwickelt.

    mhh hab mir den algorythmus schonmal angesehn…
    dennoch hab ich das gefühl, das arbeiten, die lange dabei sind und entsprechend viele votings haben etwas festgefahren sind. werden arbeiten im verhältniss zueinander bewertet? konkret: meine drei arbeiten in ihrer wertung extrem eingefallen,
    ich kann mir kaum vorstellen , dass bei allen drei arbeiten so viele negative bewertungen bekommen habe.

    …ich würde im dashboard gerne auch über schlechte bewertungen (1 und 2) informiert werden… nur so

    ….hofft…

    Hey Dizzy,
    the more ratings an idea has the less vulnerable it is for sudden changes. Obviously the 11th and 12sth ratings do have more impact than rating number 61 and 62.
    Each submitted rating has effects on all ideas – because each idea starts with the average rating of the contest. From their on it fights its own way, the initial (start) rating gradually looses it’s influenec. So in a way ideas are rated proportinally – an idea starts with a higher average rating if the average rating in a contest is high/ the ideas in a contest are of high quality. An idea is then judged upon in perspective to the other ideas in a contest and rated accordingly.

    What exactly do you want to be informed about on your dasboard? We do not want to pillory a community member by sending messages like “xxx dislikes your idea” because it also should be ok to give a low rating – if an idea is not good. So would you like to get a message simply saying “your idea was rated with 2 stars”? In effect, what would be your benefit?

    Really interested in all you guys opinion! Please don’t stop commenting!

    Something I’ve come across while rating is when you see a design that is clearly breaking the brief but people are giving it high scores. I think it has something to do with not wanting to put down their effort. I’ve seen really great work that doesn’t fit the brief but I don’t know what to rate it. The “description” text that goes along with a 0 is something like “trash it, start over” Well, you don’t want them to “trash” that work because it’s good. What you want to say is, this shouldn’t win because you didn’t follow the brief, but your art is really great.

    Maybe just a button that says “doesn’t follow brief” and then you can rate it for what it deserves, which might mean you give a 10. Or maybe 2 scores than influence the main score “how well does this follow the brief?”

    The redesign does not show the amount of loved and bashed ideas anymore. It was a quite good feature to get an idea of somebodys rating behaviour, if you see the proportion of loved and bashed ideas.

    true

    So would you like to get a message simply saying “your idea was rated with 2 stars”? In effect, what would be your benefit?

    yes i would like to have this.
    your right its no problem for someone to dislike something if he feels that way no problem.
    but it would make things more transparent and …let me give it the name “strategic voting”
    could be more easily spotted.
    …sometimes i wonder…

    Well for me, i would not like to see the bashed ideas in my dashboard. Just personally.

    My 2 Cents….

    Hello everyone,

    I am fairly new here…I joined because a friend directed me to one of the contest being currently hosted in this site. I have joined many contest before and this sort of “rating” system is pretty new to me as well.

    There are a ton of problems with it. I assume that the main reason people have it is because it seems logically a fair way to asset design work here, since it is indeed, rather hard to join this site.

    Someone mentioned above that some designs make it to the top and we don’t know why, when they don’t follow the brief. People are forgetting that ultimately we are working for a client not the community. You don’t make something to please the community, its not pride or ego at stake, its the client request….so rating should be according to the brief.

    This is why we have comments, to tell the person they did a good job. Apllied them self but it doesn’t fit. Just like we tell the people lower in the pool, “your design has potential but its missing something. ”

    Artist take things to much to heart when it comes to contest. I’ve watched some designs that deserve placement and I’ve seen the light of day…and why?

    This so called rating system. I’ve been watching it. Not everyone votes. Not everyone votes according to design. It wouldn’t surprise me if some people even posted a designs and it didn’t get far so they just stop checking the contest. When i entered the contest, the top players had 30-40 ish votes already. While the average of person’s voting was about 10-19.

    You can’t tell me that community voted fairly here….

    Also you want to speak about design and voting for it fairly? I’ve only see a few people here and there in the contest I entered, actually give artistic criticism. Not to mention that I’ve seen things get omited from the contest, people still submitting work, and people even submitting more than the max designs requested…..

    So you say you want fair?

    First you need to make things more anonymous….This whole Likes and Dislikes thing is bad…There were designs I liked- a lot- but i didn’t feel like they should get a 9 or 10., but they get no recognition in my profile, their dashboard or the community dashboard. And there there designs I did not like, but because they broke the brief or common sense rules like “copy written stuff” and they got “bashed” as people call it here. It also makes people so fussy about it.

    Then because artist takes things to heart, they know if someone didn’t like their work and thus, they may change how they vote for someone else….

    I know this is comment is really long..people may or may not want to read it all the way…I could go on but in reality i don’t know how to fix the problem because your asking a massive group of people to decide what best fits the breif, when normally its a small room with like maybe 4-5 people max [with tons of industry experience] or even just one guy saying “This works” or “This doesnt work, start over”.

    Hi Cheshire,

    Thanks for your thoughts here. I think jovoto is quite a unique place. our contest model bases on the idea of the community-driven prize as well as orientation towards the needs of the client. We usually encourage people NOT to trash ‘outlier’ ideas which are cool but not 100% on brief, but rather pesent them to the community and see how they go.
    The unique part of our platform is that the clients has no say in distributing the community prize. After the contest has closed, they’re free to choose any design from the pool of designs, it does not need to be the winner.

    Discussions about the fairness of the system and possble improvements come up all the time. Read also here https://www.jovoto.com/2009/10/more-about-rating-criteria/ or here https://www.jovoto.com/de/2009/07/voting-transparency-on-jovoto/, if you’re inetersted.

    We’re lagging behind in evaluating and testing all the valuable feedback we have received from community members during these discussions, but… it’s coming! Thrdr ideas range from making rating even more transparent to ditching the ideas of negative votes altogether, or implementing a more differentiated rating score composed of serveral other values.

    Szay tuned. I think we will be testing several new rating models on jovoto early next year.

newsletter