Updates to Point System (Round 2)

I am definitely not as accomplished as you, Jackie or a number of others who have been naming much longer. But I would not have accomplished what I have on SH if I hadn’t continually submitted a large number of entries and worked with the feedback.
The best solutions are those that produce the ideal names, whether they come with 2 tries or 40 tries. That’s the whole point of the site.

5 Likes

How does this system affect newer creatives? Let us say Joe newbie joined a week ago, submitted 10 names and got a 4 star in one of them, 3 star in one and 8 unrated. Does he get to submit 30 entries?

Does this system favor beginner’s luck over veteran credentials?

2 Likes

While I appreciate everyone’s concerns on these changes, please keep in mind that if you want to submit 40 names, you can still do so. The point of these changes is to set the initial limit based upon contestant’s past ratings. If you submit 15 names initially, and find that the CH did not like any of them, you still have the ability to withdraw your entries and submit more names.

This change is only meant to act as a throttling mechanism so that there is an initial limit on how many entries you can submit, before getting ANY feedback from CH.

What we want to avoid is users submitting 30/ 40 names before getting any feedback from CH. If you want to build upon any names or ideas that the CH likes, you can continue to do so by withdrawing other entries that they did not like.

Regarding the ratings from CH, here are some stats to consider:

In July, 19904 of the 40451 submitted entries received ratings from CH (49% rated entries). In September, this number has already increased to 62%. We are implementing several capabilities and notifications to ensure there is an early feedback on your submission so that you can continue to refine your direction based on CH ratings.

We intend to stay with this change, as we believe this will reduce the number of abandoned contests, and allow the users to continue to submit more entries based upon feedback from the CH.

3 Likes

@Dan. Go the extra mile, you made some very good changes, but why not take this initial round of feedback and tweak it a bit, then set it in stone for a while? Jackie is a MVP and she has a great point. This new formula is biased too far toward defining best creative by “least number of entries producing highest percentage of better ratings”, and light on awarding value to the pure number of wins. I’m thinking wins trump just about everything, they are in fact why the CH is here, and they do represent ultimate CH satisfaction do they not? Simply adding something like an extra 10-20 initial entry allowance for every 5 wins might reduce the current bias and properly award our consistent winners. There are different “creative formulas” for producing the best creative solutions, why have a point system that says one is better than the other? Lets just reward best solution producers, period. Here’s the KEY Dan, your point about withdrawing and re-entering doesn’t take care of the possible downside to MVP’s who happen to have a creative style like Jackie. In the real world, many CHs don’t rate entries, delay rating entries, don’t give enough feedback (sometimes not a lick of it), use the point system entirely wrong, etc. In the meantime Jackie might be stuck on the sideline? The numbers do show what may have been an unintended bias. During the last 6 mo. I think Jackie has 15 wins versus my 4, but our creative style happens to be different, I get 30 initial entries and she gets 10? Ultimately Jackie is doing a better job of serving your clientele than I am yet the new points system clearly gives me more reward, and puts Jackie at a disadvantage. I hope you can fix this, creatives like Jackie are earning the right to an equitable point system each and every day.

5 Likes

Hey everyone,

I am fairly new to this site and thought I’d give my feedback on the current ratings/point system. I’ve only been part of the SquadHelp community for about a week and am already finding myself discouraged with how the points and ratings work.

I made and entry to a contest and the CH replied to my entry saying I was on the right track and asked me to make more submissions. After doing so, they then gave me 2-star ratings on that entry and all other entries I made based on their feedback. That took my overall rating score down significantly and decreased the number of entries I could submit from 20 to 5. I have been extremely thoughtful with my submissions and have had a similar experience with several other contests. My low-rated entries now account for about 80% of my total score.

I find this very discouraging, and it has made me consider stop participating all-together. I know it’s probably very difficult to find a solution that works well for everyone, but I wanted to share my experience so far. I consider myself a fairly creative person and I love taking some time to think-up ideas that might help others with their business. But when the contest holder is unfair with ratings and feedback or doesn’t understand how it works or how their ratings impact us, it creates more of a negative experience than positive one.

I can tell that the team at SquadHelp has been very appreciative and open to feedback from users. I hope this helps somehow!

4 Likes

My two cents is a hierarchy system:

Firstly creative has won over x amount of contests, 30 submissions plus additional for 4 or 5 star ratings within those thirty.
Obviously if the creative is winning contests they are finding what the CH is after whether that takes 5 entries or 30. At the end of the day they have found a winning name.

Secondly, if creative has won under x amount of contests take into account rating %. So 20% and over gets 30 entries, 15-20% gets 20 entries etc. plus additional entries for 4 and 5 star ratings.

Thoughts?

Veterans aren’t disadvantaged. New comers who are hitting a higher rating % aren’t disadvantaged. I guess the only group who are, are new comers with the trial and error approach but I think that is better then docking your vets

To all contestants: Check and make sure your wins are being posted for the right month. My October 1 win is being shown on my profile as a September win. I notified SH, but check yours if you’ve won recently.

2 Likes

i wasnt sure if the win was from when the contest started or when it finished

@emilybj1 and @Skevans, thanks for sharing your feedback. We will continue to evaluate the point system, as well as rules for limiting the entries. For example, we have now increased the initial limits for users who feature in our leaderboard, regardless of their points. We will continue to evaluate and make additional changes to ensure that the users who are genuinely putting in a good effort do not get discouraged due to point system.

In case of new users, since the total number of entries are very small, their rating percentage can vary quite rapidly. Therefore, we will likely make some changes to ensure that the newer contestants don’t see too many rapid fluctuations in their max entries for an initial period.

@auntshommy, on user details page, we show the total number of wins against the contests participated for the same month. Basically it shows the number of wins in the same month as the month when contest was launched.

Dan, if that’s the case, something still isn’t right. I am shown on the September Leaderboard as having one win, but my profile shows 2 wins for September and none for October. Normally, I wouldn’t have noticed the date of my wins, but one can make the Leaderboard based on the date they’re counted. So if one win was counted for September’s Leaderboard, shouldn’t the October 1 win be counted with October’s wins? This is potentially going to affect a lot of contestants. Thanks.

@auntshommy, the October leaderboard will count any wins that occurred during the month of October. So if you won a contest on October 1, it will get counted in October wins. The user profile chart is just a different way of showing a “side by side” view of contests participated vs contests won - so it uses a slightly different criteria.

Thanks, Dan: that definitely clarifies things. There were a number of contestants who won contests during the first week of October, and those wins hadn’t been counted on the September Leaderboard.

@Dan, just wondering if you have noticed the increase in 2 stars since they are now hidden for the CH and how it is fair we are penalized for their convenience?

@Tammy, there is certainly an increase in overall % of rated entries in contests which continues to be an important goal. This increase is primarily due to the implementation of system changes and communication to ensure that the CH rates most of the entries that are being submitted to the contest. In fact, our goal is to get to a point where every single entry submitted to a contest is rated by CH and we will continue to implement more tools to get closer to that goal.

Regarding your concern about whether this is causing more penalization for contestants, I would like to reiterate that the system evaluates the contestant’s performance on a relative basis. We look at the % of high rated entries vs total rated entries to determine a contestant’s performance, and we compare that to other contestants during the same time period. So if two different contestants received 100 total ratings each in the same time period, and one of them received 20 high ratings, while the other person received 10 high ratings, the first contestant has a greater percentage of high rated entries. The fact that more entries are being rated across all contests doesn’t change the relative performance of contestants.

We recognize that the point system is causing an unintended stress despite some of the changes we announced previously. We will be announcing some additional important changes to point system shortly, to ensure that it acts primarily as a vehicle to recognize positive performance, vs something that is seen as penalizing to the contestants. More to come on this soon.

5 Likes

Thanks @Dan, I’m happy about the increased ratings. It just seems their rating more for convenience than an actual fair rating. Some CH actually use the 3 rating but very few. You typically see a few 4s and the remaining names mostly or all 2s. Looking forward to seeing the changes.

6 Likes

@dan I remember way back there was going to be a bottom 25, is that still in place, I ask because I thought the system was being designed for quality over quantity!!!. Will you or do you stop someone from participating, at a point where submissions (over 2,000) to win ratio and high rated percentage(Low) play a role? Or even restrict the entries allowed including being able to delete and enter more, As I said at the top I thought we are focusing on quality not quantity.

1 Like

@seezall, we do not identify or have a separate policy for 25 contestants. The ability to submit entries is determined by their high rating score - so if their score is below a certain threshold, their number of entries drops to 5. However, you are correct that everyone still has an ability to delete their entries and submit more even when they hit this limit.
So if someone really wanted to, they can work around this limit, by simply deleting their entries and submitting more.

We are working on an important update which will change the role high rated % plays in being able to submit entries, including deleted entries. This will also include a change in the limits for new users to participate in contests (their limits will increase only after we see a consistent pattern of high ratings across multiple contests).

We analyze the rating and submission patterns every month so that we can make the necessary changes on an ongoing basis. However we don’t want to make too many changes too frequently.

You are correct in pointing out that our focus continues to be quality over quantity, and our changes to point system as well as high rating % will continue to be based on that primary focus. Our next update to this high rating score/ entry limits will happen in the next 1-2 weeks.

1 Like

Thank you Dan, excited to see the changes

Quality over quantity is of course the way it should be. That’s frustrating, however, when you come across the ubiquitous CHs that ask for something specific, give those entries (even available domains) :disappointed:s and then give the highest ratings and the win to registered domains, etc.