Wednesday, May 7th, 2008

Accountability: The Evolution of the eBay Feedback System

Yesterday I sat down with Brian Burke, Director of Global Feedback Policy, to get a full run-down of all the changes being applied to eBay’s Feedback system. During our discussion, I was able to get the rationale and reasoning behind the changes as well as ask some of the questions that had been provided by Ink readers over the past few days.

Before I jump into the overview, interview and presentation, I wanted to give my two cents. The overarching theme coming out of these changes seems to be a trade of transparency for accountability (hence the title of my post). I’ve received a few emails stating that this is very “un-eBay”. However, I get the impression that eBay’s public feedback system was always intended to provide transparency and accountability but that over time, the transparency of the feedback system has taken us away from the accountability goal. So, with these changes, eBay should return to a more healthy balance of the two.

NOTE: I’ve just checked out the AB post about the changes and in retrospect, I am definitely toeing the company line on this one.

Here is a breakdown of what is being introduced this month:

1. In order to encourage repeat transactions and reward good service, eBay will provide credit for similar repeat transactions going back to when the system was introduced in 1996.

2. In May, sellers will no longer be able to leave negative or neutral Feedback for buyers.

3. eBay will remove negative and neutral Feedback left by members who are suspended or who fail to respond to the Unpaid Item (UPI) Process.

4. Positive Feedback percentage will be based on the past 12 months of Feedback, rather than lifetime on the site.

5. Restrictions on when Feedback can be left:
- Buyers must wait seven days before leaving negative or neutral Feedback for active PowerSellers who have been registered for at least 12 months.
- Members must leave Feedback within 60 days of the transaction closing (today members have up to 90 days).

6. eBay is removing Mutual Feedback Withdrawal.

This is a global eBay site-wide initiative which will start to roll-out in May, 2008.
5/12 Australia
5/15 UK, Ireland
5/19 US, Canada, Canada.FR
5/20 France, Spain, Italy, Netherlands, Poland, Belgium.FR, Belgium.NL
5/21 HongKong, Singapore, India, Malaysia, Phillipines
5/22 Germany, Austria, Switzerland

Additionally, he provided me with the presentation that summarizes the evolution of the feedback system:

Q&A with Brian

Can you please explain how the 2004 and 2007 ratio of retaliatory feedback numbers (shown on slide 5) was determined?

We used a really simple definition when determining exactly what constituted retaliatory negative feedback. It was strictly a user who received a negative feedback and subsequently left a negative feedback. That second negative feedback was counted as being retaliatory. There wasn’t anything that we did to go back and look to see if that negative was justified rather we observed how behavior changed in the marketplace over time. We saw that 4 years ago sellers would do that two times as more frequently than a buyer and today it is eight times more likely.

It’s basically how our members interpret retaliatory negative feedback so we wanted to keep it simple; getting a negative after receiving a negative constitutes retaliatory feedback.

A week or two ago the eBay developer blog announced “We will base the Positive Feedback Percentage on the past 12 months of activity (and will include neutral feedback in the calculation.” Is the above quote still accurate? If so, why are neutrals now going to count as negatives?

The above quote is accurate. As for neutrals, they’re not counted as negatives but they’re going to not count as positives either. When we first introduced the feedback percentage we had, within our feedback system, something that was called NARU neutrals. Basically, what had happened up through 1999, whenever a member got suspended we took any feedback that they had left (whether it was negative, positive or neutral) and we converted it to Neutral. The majority of feedback that was left for users was predominantly positive so we didn’t think it was fair back then to include NARU neutrals in the overall feedback percentage so we chose to count only positives and negatives in calculating the feedback percentage (although ideally we would have liked to track all). Now we’re going to a 12-month window, we no longer have to worry about NARU neutrals unfairly affecting an overall feedback score. So what we can do is make the Positive Feedback Percentage be exactly that. What’s the percentage of positives that a seller has received over all of the feedback they have received? Which is really what sellers should be measured against.

How is eBay going to protect the sellers? How is eBay protecting the sellers from bad buyers? How can I truly block an eBay bidder given that anyone can create a new disposable eBay ID at any time to circumvent my blocked bidder list?

First, what is good for the buyers is ultimately going to be good for the sellers in our Marketplace and we’ve already seen a lot of sellers changing behavior and focusing on what’s most important to the buyers just by the announced changes we’ve made.

When we made this change we understood clearly that we were taking away both a real and a perceived protection. For the seller who had never left a negative – it was just a perceived protection that they had. We needed to make sure there was a balance to the system. The goal behind the changes was not to enable buyers to just leave negatives. Rather it was to go back to the original intent of the feedback system - to make sure that both buyers AND sellers are held accountable for their actions in the marketplace. We’re going to hold sellers accountable through a public reputation system, leveraging our buyers. The way we’re going to hold buyers accountable is through private reporting from sellers; through enhancing the tools that we’ve given sellers to help protect them from buyers. We’re going to rely on sellers to let us know when a buyer has violated policy.

Why is my blocked bidder list no longer alphabetized and why are some seller’s blocked bidder lists disappearing all together? How can I truly block an eBay bidder given that anyone can create a new disposable eBay ID?

We haven’t made any changes to the blocked bidder list system so I’m not sure why someone would be experiencing that. I’ll have to look into that further. I haven’t heard about blocked bidder lists disappearing all together so if you could get me an example I will get that to the product team to find out why that is happening. In the case of a bidder circumventing a seller’s blocked bidder list with a new ID, that is against the rules and there has been no change in policy here at all. If that buyer is identified, he/she is suspended from the site.

I rechecked Feedback criteria today and it seems that the reason eBay couldn’t censor feedback was due to potential legal liability to eBay if they did. How do the shifts in feedback procedures pan out legally? Does the shift in policy constitute censorship?

In the United States only, eBay is given protections as a hosting service for the comments and content that another person puts on our site. As we don’t actually edit the comments or make changes to the comments – there is no difference in how we’re protected. All we’ve changed is how a user is rated in the marketplace. We’ve always had rules in place that allow us to remove negative feedback or positive feedback and the comments – but we won’t go in and edit the comments. Under the US law we can’t do that. To be clear though, we don’t get those same protections outside of the US but we still have a very similar set of rules so even if that law was to go away in the US we probably wouldn’t make changes to the system because we’re trying to design a system that instills trust within the marketplace, not one that is in place because of a US law.

What steps did eBay take to educate the users, both buyers and sellers, during the 3-year period that showed a trend in retaliatory feedback?

Well, the focus on retaliatory negatives didn’t just happen overnight. We examined a number of other alternatives before coming to the conclusion that the feedback system needed to change. The one thing we learned was that when a buyer gets a negative feedback in the eBay marketplace, they discontinue participating in the marketplace and it’s not healthy for anyone. All of the alternatives we examined still didn’t address one of the leading reasons why buyers were not comfortable continuing to participate in our marketplace.

There’s been a lot of focus on buyers but what are the changes we’re making to protect sellers?

One is basing the feedback system on a 12-month window rather than a lifetime window. For example, in the past, a seller’s feedback score has been negatively impacted by a negative left 7 years ago even though it doesn’t necessarily accurately reflect the seller’s participation in the marketplace today. So they’ll no longer be dragged down by old feedback and it’s more relative information for the buyer.

Second, if the buyer receives an Unpaid Item Strike and fails to respond (and that occurs more than 70% of the time today) we’re now going to remove the rating and the comment (in the past we just removed the rating). In the future, if a buyer responds in a manner that doesn’t fault the seller, we’re going to take a look at what buyers are saying in the UPI console and remove it if it is arbitrary.

We’re going to remove all negative and neutral feedback when a member gets suspended.

We’re going to prevent negatives and neutrals being left for PowerSellers that have been on the site for more than 12 months (within 7 days of the transaction end). This is a big deal for us because we’ve always felt that we shouldn’t be able to prevent a buyer from leaving a negative feedback but we feel that PowerSellers who have been on the site for 12-months are pretty safe. We’re going to start with that group of sellers so that if a buyer tries to leave a negative or neutral within 7 days of the transaction end, we’re going to provide a link that encourages communication between the buyer and the seller to see if the buyer has given the seller enough time to deliver on the transaction.

In order to cut down on potential extortion situations we’re reducing the number of days someone can leave feedback from 90 to 60 days.

For Cross Border Trade we’re going to include messaging in the leave feedback flow to remind the buyer that it was a CBT transaction.

When a buyer goes to leave a negative or a neutral – and very single buyer will see this, every single time they leave a negative or neutral – we’re going to ask them three key questions:

1. Have you communicated with the seller?
2. Have you allowed enough time?
3. Have you kept the feedback factual?

The big difference between then and now is that before, buyers saw the above questions but moving forward, they will have to check each question before proceeding. So it is creating a speed bump for buyers in leaving a negative or neutral and we hope that it will educate buyers, the first time they go to leave a negative or a neutral, that it is important to communicate, it is important to allow time for shipping, and that it is important to be factual.

When satisfied buyers attempt to leave DSR ratings of all 5’s they are prompted with the following message: “Are you sure? Remember, the ratings left by you are anonymous and can’t be seen by the seller.”
Why does eBay feel they need to doubt the satisfied buyer’s desire to leave 5-star DSR ratings?

The short answer is we don’t do it just for those with 5. We do it for anyone who is leaving detailed seller ratings. Whether someone intends to leave a 1 or intends to leave a 5, they’re going to get that message. This isn’t about us doubting someone whose desire is to leave a 5. Rather, it was an effort by us last Summer to address an issue we saw arise when we first introduced detailed seller ratings. Our conventional feedback system is a public system that is transparent to the marketplace and we wanted to communicate to the buyers that DSRs were designed to be more anonymous with the ultimate goal to obtain more accurate, honest feedback from buyers.

Since eBay has made such a point that DSRs are anonymous, what faith do we as sellers have that eBay is accurately reporting the ratings that our buyers are actually leaving for us? While still protecting individual’s anonymity, why not provide to me some DSR information of value? For example, what is my DSR rating from international customers versus domestic buyers for each DSR?

To answer the first part of the question… it really is in eBay’s best interest to accurately represent how buyers are reporting on sellers because that will favor those sellers that are performing best which is exactly what we want.

With regard to seller reports and information, I absolutely believe this is an area in which we can dramatically improve. We’re introducing the Seller Dashboard that provides basic information to the seller about their DSRs but we’ve heard from a lot of sellers that they would like to slice and dice that information so we’re working on a separate project right now that will enable them to either create their own reports or for us to provide a much more robust reporting structure that our sellers can take advantage of. All I can say on that is to stay tuned; it’s not something folks are going to see in the next couple of months but it is something we’re working on.

Brian, thanks very much for taking the time to provide answers to Ink readers. It is greatly appreciated and we look forward to having you back on the blog soon.

My pleasure. Anytime.

Tagged: brian+burke, buyers, detailed+seller+rationgs, dsr, ebay, ebay.com, ecommerce, feedback, Marketplace, online+marketplace, sellers

StumbleUpon