So, ever since the 7Sigma argument a while back, I've been paying a bit closer attention to this, and I'm starting to see more and more skills pop up where they are self-reviewing to get themselves on the front of the "Average review" list. We wrote a blog post about it, calling out several skills that have done it, but I think in general it's important that we have a conversation about how to [i]prevent[/i] it from becoming the de facto state. Otherwise everybody loses.
This is a very interesting article, Galactoise, thanks for taking the time to write it. We'll be passing this along where applicable, and we appreciate the community coming out and sharing their thoughts surrounding this.
@jjaquinta -- about this: "If you look at the normal Amazon store (and why aren't we using that?) in addition to ranked reviews, it also has best sellers. " I thought of that too, but I'm not sure that would help us here. I'm guessing that Amazon could easily track which skills are most in use. But does usage follow quality? Suppose they measured usage in terms of number of HTTP requests. I can imagine some of the low or medium level skills get a lot of HTTP requests. We could encourage Amazon to track the number of dedicated, repeat users a skill has, which might be a good measure. But mere usage, as in HTTP requests, might favor the stupid skills over the good ones.
>But does usage follow quality? On the Skill Metrics tab in the Dev Console they graph "Total Utterance Count" and "Unique Customers Count". I think there is a strong argument to be made that if a whole bunch of people are invoking your skill, or that any number of people are invoking your skill a lot, then there must be something to it. Certainly it is a better metric to rank by than any of the current ones. Personally, I have my own logs that I do number crunching on. I distinguish between "casual" users and "dedicated" users. Since the main goal of my last release for Starlanes was to increase retention, I needed the numbers. Currently we're at about 10% retention. Given the casual nature of the Echo device, I'm pretty happy with that number. But what is a good or bad retention value varies considerably with the type of application being measured. This is why I don't think Amazon can realistically adopt a more sophisticated method such as this to determine what a dedicated user is, and is not. What I'm not happy with is the number of new users that are finding my apps. Given the bloat in the skills tab, and no way to effectively advertise, it's hard to address without Amazon fixing some of the discoverability problems with skills. Which is why the ranking methods in the Skills Store are so important.
I would suggest sorting by category as a start, but more fundamentally I think once more people start enabling skills the issue of one or two self-reviews will become moot. Still, I agree that the current "highest rated" thing is distracting, and might be creating a bit of a chicken and egg problem: once there's a greater adoption of skills then one or two self reviews won't make a difference. But the current skills tab might be a barrier to greater adoption. I would love to know what percentage of Echo users have enabled at least one skill. As a pure stab in the dark, I'd guess less than one percent. Steve p.s. I won't call it a skills *store* until people can buy skills. Right now it's just a tab in the app.
>I would love to know what percentage of Echo users have enabled at least one skill. >As a pure stab in the dark, I'd guess less than one percent. They won't even release how many Echo units have been sold. So I'm not sure you would get that information out of them. I haven't meet a (non-programmer) echo user yet who has ever enabled a skill. So, yeah, I'd be inclined to agree with you. I'm developing a list of "asks" for Amazon that are necessary to turn this into the sort of e-commerce platform I want it to be. Having an easier on-ramp for people is one of them. If we had a web-link that would take people right to where they need to be to enable a skill, then that would be something we could use in promotion. I know someone (Galacticose?) worked out how to backwards engineer it. But it really wants to be something that has a simple URL that works from anywhere.
@jjaquinta -- those are great ideas. It would also be great if Alexa had the support of the whole Amazon organization. I was delighted to find out that Amazon now runs a co-working space (in New York City) where anyone can walk in and schedule time to talk to an Amazon engineer about Amazon AWS services. I was sad to find out that the engineers here do not support Alexa. It is too new. When I talk to the Amazon engineers, they take careful notes and then send the notes to the Alexa team. The Alexa team never responds. So standing in the Amazon AWS co-working space is no more useful than asking a question on this forum. I am hoping that during 2016, Amazon increases its support for Alexa.
AWS is pretty mature. So I'm not surprised they have a lot of support behind it. I don't expect Alexa to be as mature. But they've lumped it in here with the Mobile App stuff. That would be fine. There's lots of cool libraries for Mobile that I [b]really[/b] want to use for Alexa. [i]But none of them work[/i]. Here's a list: * In App Purchasing. Bzzzt. I don't expect the UI to work. But I'd rather like to leverage the actual monetary transaction side of things. I'm not looking forward to inventing my own. * Ads. Bzzzt. I've already resigned myself to creating our own ad serving platform. But it would be nice if Amazon was already ahead of the game and was in talks with radio-ad serving agencies., * Merch by Amazon. Bzzzt. They only give out so many accounts. And they are all gone. It would be nice if they had reserved a few for Alexa developers. I've got users who want Starlanes T-Shirts. * GameCircle. Bzzzt. I've had to invent my own leaderboards and achievements. I'd much rather throw it away and use Amazon's. * Login With Amazon. Bzzzt. You would think this would be low hanging fruit. But, no. * Advertising. Bzzzt. There is no way to increase awareness of our skills. There is no way to, say, target ads at Echo owners who have installed games on their phones. No way to give them a link that automatically brings them to the Skills tab ready to enable. All we can do is name our skills with alphabetically low names and make their icons flash. (Now who would do that?) It's like we're the poor bastard stepchild.
If that AWS pop-up scenario was frustrating for being "close, but not quite enough", think of what it's like for me - I live in Seattle, and still couldn't get any traction on direct communication with them until the rant and ensuing hackernews post. I know probably 30+ people who [i]work for Amazon[/i], and still had no way of initiating a conversation. That said, the Alexa Roadshow is coming to NY and Boston soon. You guys should go, and pester their DAs and developers with questions, like I did. It was well worth my time (although I wish I had known more beforehand, so I could've asked the harder questions), plus I totally got a free kindle from the raffle.
>I know probably 30+ people who work for Amazon, and still had no way of initiating a conversation. Well, it's a big company. And Alexa, mostly, isn't done in Seattle. The father of one of my daughter's classmates works for Amazon, Boston. But he's on the AWS team, not the Echo team. And, even if he was, they do pipeline work here. They don't do business development or the SDK. So it wouldn't be helpful. I think the SDK and developer outreach stuff is done somewhere in the midwest. I can't remember where. >That said, the Alexa Roadshow is coming to NY and Boston soon. Huh. I was at the Amazon Developer Day last year. There really weren't any technical people at it. David Bazuski was there, but mostly gave a non-stop commercial for the other, unreleated, development stuff. The Alexa stuff he talked about was pretty canned. I talked to a few of the "judges", but half of them were with the conference agency who ran the event, and the rest were not subject matter experts. I ended up answering more questions for my table mates than the people giving the conference. It was reflecting on that experience that convinced me to write that book on developing for Alexa. (Which is still selling rather better than I would have expected.)