Recently, the game Legend of Solgard, which is developed by Snowprint and published by King, has done the following: (1) give one group much more gold than another for I think its been over a month for, their words, an "experiment," and indicated that whenever the experiment is over, no compensation will be given to the low gold group. They claim that companies do experiments like this all the time; (2) just today, pushed major new content onto only a small subset of android users for an involuntary beta test. I am aware that companies do limited releases in certain markets, but have never heard of a limited update. Am I wrong to think this feels different and unethical? I have never even heard of such conduct and find it reprehensible. Thoughts?
Do you have any links to them admitting they're doing this? A/B testing is sort of common but usually you don't ... talk about it, or let players figure out you're doing it.
Yeah, their community rep (who I assume works for King, although this has never been clarified to my knowledge) is on record admitting it in the reddit, defending the practice as routine. I've also seen a response from their customer service to a guildmate indicating that no compensation will be given once the test is over because A/B testing is routine and legitimate. A/B testing may be common for little stuff, but doing it for the amount of gold given, which is the major bottleneck currency, really floored me. Also, once you have admitted it publicly, seems like any value of the test is gone. Not to mention that people definitely quit over this issue. They are now openly A/B testing for a lot of different things: a 4x button in the arena, some people get fewer arena chests but get an arena currency, that sort of thing. A lot of those strike me as weird too, but the gold thing is the one that really struck me as unfair and possibly unethical treatment of their customers. The difference in gold they gave is enormous -- probably over $100 worth over the course of the test so far, if you were to buy it in their store. It also struck me as bizarre that only a tiny number of people are getting this major update to use them as an involuntary beta test. It led, for example, to the ridiculous result of a user posting on reddit that a big content update was available, you know trying to be helpful, even if he just confused everyone. Maybe its a stretch to call that unethical, given that limited releases are a well established thing, but an unannounced limited update struck me as the worst possible way to get help debugging.
Sure. Here is the community rep's posts: https://www.reddit.com/user/RatatoskTheSquirrel And here is what he said on this: *** "The only stuff everyone is guaranteed to have right now is the stuff in the notes that I posted. The differences in gold bounties are a test we're doing. Many people still have the old reward levels. There are some people who were randomly selected to try out some new rewards, and the results are eye popping enough that people are posting screenshots Bear in mind there's no such thing as a free lunch. Increases in treasure are always paid for with an increase in cost elsewhere. The goal is something that feels good. Anyway, back to Solgard: You aren't losing anything, and you're not being disadvantaged. You aren't seeing increases in costs or time or other requirements because those things don't show up in screenshots. I think I posted in another thread - last patch we were testing changes to the experience curve. This time we're doing some economy tests. When we're done running the numbers we'll push an update that puts everyone on the same formulas." *** It turns out that his "no such thing as a free lunch" point was simply untrue. One group got massively more gold. While they got less of the second currency, dust, you can buy dust for gold. It wasn't "some people" either -- a major % of the community was in the high gold group (including me, but not many of my guildies.) The test is actually still in progress...think its been over a month, but I'd have to double check. I can probably also find the response from customer service to my guildies request for compensation for being in the low gold group if it adds value. Also Eli, is it really the getting in front of it part of this that is odd -- if you are going to give a high % of your players much higher rewards, how can you say nothing? I think A/B testing is normally not for such a major issue? Anyway, thanks for taking the time to respond, was curious if the experts thought this was as odd/unfair as I do.
In this case, the reward systems were so dramatically different there’s no chance it would have stayed hidden for long. People were posting on reddit almost immediately about how much gold they got from different activities. There were other changes, like some players had a button to change game speed while others didn’t. There were two very different award sets for hero arena, with one group having a shop that didn’t exist in the other. There were shop offers unique to one group. None of this was subtle.
To be fair, it is pretty universal that developers in online games do stuff like this. Changes are usually rolled out to small parts of the population, based on something like level, playtime, etc., as a way to test if the updates are balanced or achieving the desired goal. It sounds like these devs were doing tests to see if increased rewards encouraged more playtime or IAPs on a small section of the player base before rolling it out. They sometimes try to be subtle about it, but many games (like Pokemon Go) straight up announce that only certain players will get new features during testing. I get that it seems unfair when you aren't part of the population that gets these experimental changes when they are beneficial, but keep in mind that almost every online game (on mobile and elsewhere) without a dedicated testing server does this, and the changes will probably be rolled out to everyone soon enough. Remember that this is done to avoid a bad change being rolled out to everyone and completely breaking the balance of the game and ruining people's experiences. Hopefully you are one of the lucky ones to get tested on next time.
This doesn't seem like a smooth way to handle such a test. I would think a better solution would be to invite users to help test new changes, but not have those changes impact public gameplay. I can't say that I've come across a situation where it was done this way, and I can definitely understand the community's frustration with it. Seems like inviting PR fires to put out!
