question

This is a very interesting article, Galactoise, thanks for taking the time to write it. We'll be passing this along where applicable, and we appreciate the community coming out and sharing their thoughts surrounding this.

Hey Justin, it would be great if you moved beyond just "appreciating" the community complaining about this and actually did something about it and/or shared what you were planning. Transparency leads to customer satisfaction and engagement. Discoverability of skills is a critical business need. I'm working on a log analyzer now to graph things, but my anecdotal feeling is that the number of new users of my skills has been dropping as the number of skills in the market has increased. (This is statistically difficult to definitively prove, since there was a HUGE surge over Christmas week.) That would be expected if I felt my skills were around the average in terms of use. But given articles like the "write an Alexa skill in one hour" and tools like "Alexa Designer" make it easy for people to post lots of worthless skills the quality has gone down a lot. Basing it solely on reviews is flawed, as Galactoise lengthily point out. In a high volume system, you can do statistical tricks to remove outliers. That's hard with the low volume here. If you look at the normal Amazon store (and why aren't we using that?) in addition to ranked reviews, it also has "best sellers". That at least tells you whose advertising is best. But, with the Kindle, Amazon can actually track the number of pages read. This gives the actual true engagement in the product. There is no reason this can't be done for Alexa. You have the numbers. These are already displayed in our personal "Skill Metrics" tab. The number of unique customers maps easily to "number of units sold" and the "total utterances" maps to the number of pages read. These are much more objective skills and, until such time as Chinese sweat shops can fake hundreds of Echo user, are much harder to game. It only makes sense.

@jjaquinta -- about this: "If you look at the normal Amazon store (and why aren't we using that?) in addition to ranked reviews, it also has best sellers. " I thought of that too, but I'm not sure that would help us here. I'm guessing that Amazon could easily track which skills are most in use. But does usage follow quality? Suppose they measured usage in terms of number of HTTP requests. I can imagine some of the low or medium level skills get a lot of HTTP requests. We could encourage Amazon to track the number of dedicated, repeat users a skill has, which might be a good measure. But mere usage, as in HTTP requests, might favor the stupid skills over the good ones.

>But does usage follow quality? On the Skill Metrics tab in the Dev Console they graph "Total Utterance Count" and "Unique Customers Count". I think there is a strong argument to be made that if a whole bunch of people are invoking your skill, or that any number of people are invoking your skill a lot, then there must be something to it. Certainly it is a better metric to rank by than any of the current ones. Personally, I have my own logs that I do number crunching on. I distinguish between "casual" users and "dedicated" users. Since the main goal of my last release for Starlanes was to increase retention, I needed the numbers. Currently we're at about 10% retention. Given the casual nature of the Echo device, I'm pretty happy with that number. But what is a good or bad retention value varies considerably with the type of application being measured. This is why I don't think Amazon can realistically adopt a more sophisticated method such as this to determine what a dedicated user is, and is not. What I'm not happy with is the number of new users that are finding my apps. Given the bloat in the skills tab, and no way to effectively advertise, it's hard to address without Amazon fixing some of the discoverability problems with skills. Which is why the ranking methods in the Skills Store are so important.

I would suggest sorting by category as a start, but more fundamentally I think once more people start enabling skills the issue of one or two self-reviews will become moot. Still, I agree that the current "highest rated" thing is distracting, and might be creating a bit of a chicken and egg problem: once there's a greater adoption of skills then one or two self reviews won't make a difference. But the current skills tab might be a barrier to greater adoption. I would love to know what percentage of Echo users have enabled at least one skill. As a pure stab in the dark, I'd guess less than one percent. Steve p.s. I won't call it a skills *store* until people can buy skills. Right now it's just a tab in the app.

>I would love to know what percentage of Echo users have enabled at least one skill. >As a pure stab in the dark, I'd guess less than one percent. They won't even release how many Echo units have been sold. So I'm not sure you would get that information out of them. I haven't meet a (non-programmer) echo user yet who has ever enabled a skill. So, yeah, I'd be inclined to agree with you. I'm developing a list of "asks" for Amazon that are necessary to turn this into the sort of e-commerce platform I want it to be. Having an easier on-ramp for people is one of them. If we had a web-link that would take people right to where they need to be to enable a skill, then that would be something we could use in promotion. I know someone (Galacticose?) worked out how to backwards engineer it. But it really wants to be something that has a simple URL that works from anywhere.

@jjaquinta -- those are great ideas. It would also be great if Alexa had the support of the whole Amazon organization. I was delighted to find out that Amazon now runs a co-working space (in New York City) where anyone can walk in and schedule time to talk to an Amazon engineer about Amazon AWS services. I was sad to find out that the engineers here do not support Alexa. It is too new. When I talk to the Amazon engineers, they take careful notes and then send the notes to the Alexa team. The Alexa team never responds. So standing in the Amazon AWS co-working space is no more useful than asking a question on this forum. I am hoping that during 2016, Amazon increases its support for Alexa.

AWS is pretty mature. So I'm not surprised they have a lot of support behind it. I don't expect Alexa to be as mature. But they've lumped it in here with the Mobile App stuff. That would be fine. There's lots of cool libraries for Mobile that I [b]really[/b] want to use for Alexa. [i]But none of them work[/i]. Here's a list: * In App Purchasing. Bzzzt. I don't expect the UI to work. But I'd rather like to leverage the actual monetary transaction side of things. I'm not looking forward to inventing my own. * Ads. Bzzzt. I've already resigned myself to creating our own ad serving platform. But it would be nice if Amazon was already ahead of the game and was in talks with radio-ad serving agencies., * Merch by Amazon. Bzzzt. They only give out so many accounts. And they are all gone. It would be nice if they had reserved a few for Alexa developers. I've got users who want Starlanes T-Shirts. * GameCircle. Bzzzt. I've had to invent my own leaderboards and achievements. I'd much rather throw it away and use Amazon's. * Login With Amazon. Bzzzt. You would think this would be low hanging fruit. But, no. * Advertising. Bzzzt. There is no way to increase awareness of our skills. There is no way to, say, target ads at Echo owners who have installed games on their phones. No way to give them a link that automatically brings them to the Skills tab ready to enable. All we can do is name our skills with alphabetically low names and make their icons flash. (Now who would do that?) It's like we're the poor bastard stepchild.

If that AWS pop-up scenario was frustrating for being "close, but not quite enough", think of what it's like for me - I live in Seattle, and still couldn't get any traction on direct communication with them until the rant and ensuing hackernews post. I know probably 30+ people who [i]work for Amazon[/i], and still had no way of initiating a conversation. That said, the Alexa Roadshow is coming to NY and Boston soon. You guys should go, and pester their DAs and developers with questions, like I did. It was well worth my time (although I wish I had known more beforehand, so I could've asked the harder questions), plus I totally got a free kindle from the raffle.

>I know probably 30+ people who work for Amazon, and still had no way of initiating a conversation. Well, it's a big company. And Alexa, mostly, isn't done in Seattle. The father of one of my daughter's classmates works for Amazon, Boston. But he's on the AWS team, not the Echo team. And, even if he was, they do pipeline work here. They don't do business development or the SDK. So it wouldn't be helpful. I think the SDK and developer outreach stuff is done somewhere in the midwest. I can't remember where. >That said, the Alexa Roadshow is coming to NY and Boston soon. Huh. I was at the Amazon Developer Day last year. There really weren't any technical people at it. David Bazuski was there, but mostly gave a non-stop commercial for the other, unreleated, development stuff. The Alexa stuff he talked about was pretty canned. I talked to a few of the "judges", but half of them were with the conference agency who ran the event, and the rest were not subject matter experts. I ended up answering more questions for my table mates than the people giving the conference. It was reflecting on that experience that convinced me to write that book on developing for Alexa. (Which is still selling rather better than I would have expected.)