“How much more complexity can human developers and organizations deal with?” – Tom Fastner, eBay.

I just finished reading an interview with Tom Fastner, senior member of technical staff and architect at eBay on ODBMS Industry Watch with Prof. Roberto V. Zicari. From a technical perspective it’s a great read but more importantly it represents a growing trend I’m seeing more of from our company: the very welcome “under-the-hood” conversations with our thought leaders and key drivers of change about the technology infrastructure of our marketplace.

Q1. What are the main technical challenges for big data analytics at eBay?

Tom Fastner: The primary challenges are:

I/O bandwidth: limited due to configuration of the nodes.

Concurrency/workload management: Workload management tools usually manage the limited resource. For many years EDW systems bottle neck on the CPU; big systems are configured with ample CPU making I/O the bottleneck. Vendors are starting to put mechanisms in place to manage I/O, but it will take some time to get to the same level of sophistication.

Data movement (loads, initial loads, backup/restores): As new platforms are emerging you need to make data available on more systems challenging networks, movement tools and support to ensure scalable operations that maintain data consistency

To read the full interview, please visit odbms.org. You can also follow @odbmsorg on Twitter.

Cheers!
RBH