Join us in returning to NYC on June 5th to collaborate with executive leaders in exploring comprehensive methods for auditing AI models regarding bias, performance, and ethical compliance across diverse organizations. Find out how you can attend here.
Over the past 10 years, the data tooling and infrastructure world has exploded. As the founder of a cloud data infrastructure company in the early days of cloud computing in 2009, plus the founder of a meetup community for the nascent data engineering crowd in 2013, I found a place at the center of this community even before “data engineer” was a job title. It is from this seat that I can reflect on the lessons learned from our recent data tooling past and how it should guide development of a new AI era.
In tech anthropology, 2013 was a period between the “big data” era and the “modern data stack” era. In the big data era, as the name suggests, more data was better. Data was purported to contain the analytical secrets to unlock new value in a business.
As a strategic consultant for a large internet company, I was once tasked to build a plan to chew through the data exhaust from billions of DNS queries per day and find a magical insight buried in this that could become a new line of business for the company worth $100 million. Did we find this insight? Not in the relatively short time (months) we had to spend on the project. As it turns out, storing big data is relatively easy, but generating big insights takes significant work.
But not everyone realized this. All they knew was that you couldn’t play the insights game if your data house wasn’t in order. So, companies of all shapes and sizes rushed to beef up their data stacks, causing an explosion in the number of data tools offered by vendors who proposed that their solution was the missing piece of a truly holistic data st …
Article Attribution | Read More at Article Source
Join us in returning to NYC on June 5th to collaborate with executive leaders in exploring comprehensive methods for auditing AI models regarding bias, performance, and ethical compliance across diverse organizations. Find out how you can attend here.
Over the past 10 years, the data tooling and infrastructure world has exploded. As the founder of a cloud data infrastructure company in the early days of cloud computing in 2009, plus the founder of a meetup community for the nascent data engineering crowd in 2013, I found a place at the center of this community even before “data engineer” was a job title. It is from this seat that I can reflect on the lessons learned from our recent data tooling past and how it should guide development of a new AI era.
In tech anthropology, 2013 was a period between the “big data” era and the “modern data stack” era. In the big data era, as the name suggests, more data was better. Data was purported to contain the analytical secrets to unlock new value in a business.
As a strategic consultant for a large internet company, I was once tasked to build a plan to chew through the data exhaust from billions of DNS queries per day and find a magical insight buried in this that could become a new line of business for the company worth $100 million. Did we find this insight? Not in the relatively short time (months) we had to spend on the project. As it turns out, storing big data is relatively easy, but generating big insights takes significant work.
But not everyone realized this. All they knew was that you couldn’t play the insights game if your data house wasn’t in order. So, companies of all shapes and sizes rushed to beef up their data stacks, causing an explosion in the number of data tools offered by vendors who proposed that their solution was the missing piece of a truly holistic data st …nnDiscussion:nn” ai_name=”RocketNews AI: ” start_sentence=”Can I tell you more about this article?” text_input_placeholder=”Type ‘Yes'”]