question

Yup. Our experience exactly. I just finished yesterday the first draft report on just this problem for my CEO. It's intended to be the input by which she will decide if we consider pursuing Alexa or divesting. (Drop me a line at jo@111george.com and I'll send you a draft. And you can see how closely our experiences compare.) For me, it boils down to these three problems: 1) Many "requirements" are subjective. 2) Each time you submit, you get a different person. 3) There is no way to engage with Amazon and discuss/get clarity on these subjective judgements. These three together break certification. A different person each time makes different subjective decisions which cannot be explained, or discussed. There is no way to iterate towards the correct decision because the goalposts move each time you submit. The only skills that can get through unimpeded are the simple ones, which stifles innovation. Overall it is absurd. The majority of things they object to you can change the minute after they are certified. I just did a bunch of changes last night to once skill with a switch in the code. For everything they insisted I change that I disagreed with, I can flip the switch to do it their way. Once we're past certification I'll flip the switch and go back to doing it my way. You can't do this for everything, but you can do it for many things. Amazon Web Services (and most other companies with comparable offerings) don't do things like this. All they, pretty much, care about certifying is that you aren't going to break the system, and aren't doing anything they can get sued for. No mandatory program design constraints like when and where I should explicitly prompt the user (rather that re-prompt). Other than ranting in the forum, I know of no other way to get their attention. We've considered reworking the report I've done on certification into a chapter in our book and, bang, getting it in front of 200+ Alexa developers. But it seems kind of unprofessional to use a threat like that to just get some engagement on the subject. What I would really like to understand is Amazon's ambitions for the Alexa. Where do they see 3rd party developers fitting in? Where do they expect it to be in a year? Three years? Five years? If it's just really there for hobbyists and home automation enthusiasts, they should just say that. There are people like us, here, essentially giving them free labor that adds to their IP and the value added of their device to their users. I would think we deserve at least a little engagement for our efforts.

New plan. I'm going to resubmit without making any changes, and hope that I can trigger some engagement with them that way. At the very least, it makes my case to them that the current status quo - where submission is the only way to trigger a response from the certification team - is just as inefficient for them as it is for us.

Save your breath. I submitted our blackjack skill three times in quick succession with no change. They didn't even notice. (The basis of my report is comparing the results from the three submissions we did and analyzing the differences between the returns to draw conclusions on how the certification team operates.) You see, because it is a different person each time, and because they don't pay attention to either your notes or the previous notes, it doesn't matter if you resubmit without changing. They just won't notice because each time is a fresh slate. You won't drive engagement that way. This can, however, work to your advantage. You can use the "flip the switch after I have certified" for some issues, for the more subjective issues, you can just keep resubmitting until you get a certifier whose had their coffee or, otherwise, subjective analysis goes in your favor. It is kind of ironic. Most of the time it's casual time-saving actions on Amazon's side that burden the 3rd party developers with a lot of tail-chasing time-wasting activities. Repeated resubmission is a trivially easy thing for 3rd party developers to do, that does the same thing to Amazon. Professionally (and morally) I think it is really isn't the right thing to respond to bad business behavior with more bad behavior. I only did it with one of my four skills in certification hell, and only long enough to establish a baseline. Really, what needs to happen, is for Amazon to engage, and change the process so there is less time wasted on both sides.

Yeah, I fully expect you to be right on this. My thought was that by doing it and then having this conversation with you here, on the forums, maybe one of the employees who troll the forums will see the thread and realize "wow, our certification process is incredibly broken and wasting a ton of employee time (which translates to money)", and then do something about it.

jjaquinta -- that is a fascinating story. Sounds like someone with a skill to push could potentially just keep resubmitting, over and over again, without making changes, until finally they get lucky and they get someone who just happens to certify the skill. It also sounds like we can all simply do whatever they say, and then we can change the skill however we like as soon as they certify it. Amazon is trying to run the Alexa app store as if there was actual binary code in the store, as there is with the iPhone and Android phones. They don't seem to have thought about how much that model breaks down when one instead has web services that are being hosted outside of the store.

You see, I think the people who "troll the forum" are quite aware of the problems. I think, by and large, they are good people. The thing is, if you peek into any of the other forums, they work [i]all[/i] the forums. Not just Alexa. So they aren't subject matter experts. If you also note, most of their replies are either pointing to existing documentation, or else promises to send it to "relevant teams". They make a point of not commenting on future plans. I don't think they aren't being helpful because they don't want to be helpful, but because the constraints of their job role do not allow them to be helpful in the way we would like them to be helpful. I used to do tooling for support in my day-job company. There were several levels to support. Level one dealt with the raw incoming requests. They triaged them, made sure the customer was paying for support, got all the information and categorized the problem. It then got assigned to a Level two support engineer. These usually had some experience in the categorized area they dealt with. They would then be the liaison between the customer and Level 3 support. Level 3 were subject matter experts, and often on the actual product team. However, there was never any direct contact between the customers and Level 3. Barriers were intentionally put in place so that the Level 3 engineers could control the time they spent on support issues. I'm pretty sure that is what is happening here. Memo and Nick and the rest address what can be addressed via existing knowledge base. Anything else gets filtered back to the team. The problem is that it stops there. There's no way for anything to come back from the teams. No one who actually works on the system, or has authority to speak to the system has any way to communicate back. The people who run the form are not the cert team, so they can't tell us what the hell the cert team is thinking when they do their stuff. This is particularly frustrating for me, because I used to work in Business Partner Technical Enablement. Kind of like Level-2, as described above, but with our high value, high technical customers one the one side and with the software architects and product managers on the other side. The main objectives were to get customers deriving the most value from our product possible, and capturing the high value changes needed for our product back to the development team. Amazon's traditional areas (everything on AWS) follows a different model. They are more like a utility. Minimal engagement. No direct support. They provide a service. You take it or leave it. While this I think is a workable approach for a wide audience in a mature area, I do not think it is a wise approach for Alexa. But, that's if they give a hoot about 3rd party developers and think they are going to add any significant value to their platform. We don't know if this is so, because they won't say. I summarized a lot of this in a blog post I made a while back: "Explaining Amazon's Indifference" ( http://ocean-of-storms.com/tsatsatzu/explaining-amazons-indifference ) Sadly we don't seem to have made any real progress since then. Ranting in the forum is one of the few ways that seems to get any attention. Sadly I think it mostly has the effect of "shooting the messenger". I think the Amazonians monitoring the forum dutifully pass the information on. And that it's being ignored at a deeper level. Where our slings and arrows cannot reach. And that they've designed the system that way. TsaTsaTzu put in a proposal to the Alexa fund for creating a infrastructure for serving up ads. Much in the same way that mobile has. They were polite, but skeptical, and denied our application. That's fair enough. So I've created a skill with ads in it (Starlanes) and if I can get to the point where we are getting 1-10,000 impressions a month, that will be the signal to go back to them. We will have proven that Alexa skills have the potential to be a platform for advertising. In the same sense, we can rant all we want in the forum. Right now I think 3rd party skills are a very minor part of what Amazon see as the value their platform delivers. Sure, they talk about it in their newsletter, and wave the name of any "big name" who develops a skill for them. But they also talk about that banana "Name Game" thing which looks like it was written as a demo during an Amazon Developer Day. So we can't read too much into that. However, if one of us can create a skill, that is actually popular, and garners significant press coverage, then we might be in a position where we might be listened to. For my part, I am hoping to grow Starlanes into that. It's multi-user. It's a game. There's a level of interactivity and complexity. I'd really like to get it to the point where I can ping various game review companies and blogs and say "shit just got real on Alexa". But it's chicken and egg. I've got great engagement for Starlanes, but low retention. I'm trying to iterate towards a better solution, but to do so I have to run the gauntlet of certification. I've got a bunch of new features that are implemented, but that the users can't use because the voice model needs updating. I've got a comprehensive tutorial system that walks you through the gameplay elements, which I hope will help with retention. But it took two months to get through certification for the first release. To get these into people's hands I'm going to have to go through the new certification process and deal with asinine comments about how the "stream is left open" and other nonsense. I'm very much not looking forward to that.

I posted a short note about this to Hacker News. If you would like to upvote it, it is here: https://news.ycombinator.com/item?id=10876409

>Sounds like someone with a skill to push could potentially just keep resubmitting, >over and over again, without making changes, until finally they get lucky >and they get someone who just happens to certify the skill. Technically, yes. If there are only one or two narrowly contentious points, then yes. If you have several points, odds are one one of them will come up. That was part of the reason we did the analysis; to try to work out how consistent certification was. I wish this forum did tables and I'd copy our chart in. But here is the summary: [i]A total of ten different objections were raised. Three of these were raised across all three submissions. Six were raised across two submissions. And one was raised in a single submission. [/i] >It also sounds like we can all simply do whatever they say, >and then we can change the skill however we like as soon as they certify it. Yes and no. The interaction model (plus description, activation word, icon, etc) is defined in the developer console. These cannot be changed without going through certification. The code beyond your endpoint you can change as much as you want. So, for example, one of their objections was "F. Shorten Rules Prompt". [2/3] When the user asks for the rules of the game, Amazon felt the response was too long and asked us to shorten this. We disagree because the specifics of which rules are used influence the strategy you use to play the game. But there's no way to have that qualitative discussion with Amazon. So in my "mitigation" section I note: [i]The choice of what text to present is made in our application and is not controlled by certification. We can shorten the rule description down, and then put the full description back in after the certification process. Or, since it was not consistently reported, we can just keep resubmitting until they choose not to report it.[/i] However, another objection was "H. Game Should Start After 'I want to play'" [2/3]. We didn't have this certifiers pet invocation phrase "I want to play" in our utterances. So Alexa was matching "how do I play" and reading the rules. Since this is controlled by the utterance file, we cannot make a change to pass certification, and then revert it later. However, our objection is not as strong here. We would just have to do some testing to make sure other ambiguities aren't introduced. So, sometimes you can revert later, sometimes you can't. However, both these examples do illustrate how subjective interpretation of unclear rules escalate an individual certifier's personal preferences into publishing blocking requirements.

Thanks for doing that, Lawrence. This may be just the sort of visibility we need to get them to reform their process.

I jumped in and wrote a blog post about some of the points that got less traction on the HN article. http://www.derpgroup.com/blog/on-the-topic-of-certification