The list of technologies that have been created because of the limited nature of enterprise-based relational databases are quite long, but these tried and true technologies remain in the midst of modern enterprises.
But they are increasingly sharing space in datacenter with various databases and data stores trying to get performance constraints or scales to solve very bad problems.
Doug Cutting, creator of open source Hadoop platforms that mimic Google’s eponymous file system and MapReduce chunking and chewing systems, runs against any limitation while working on Yahoo, and doing something about it, in the end, helping to lay off new industries and solve some of the big problems.
As the chief architect of Cloudera, which is the largest distributor of commercial Hadoop platforms and soon to become larger when it takes over Hortonworks rivals, Knowledge Cutting where the Hadoop platform fits today and where it can grow in the future.
The Next Platform has recently talked to Cutting about the possibilities.
Timothy Prickett Morgan: I’ll know what’s happening next to Hadoop. Platforms come and go. What do you do to encore after Hadoop? I mean, we are the Next Platform, so as in the job title to think of what follows.
I just spent a week at the conference of Supercomputing 2018, where the hot topic of conversation was a kind of collecting traditional simulation and modeling and learning machine, either as part of the workflow or embedded in it as part of an ensemble or generate set simulation exercises and distributing stuff to in the simulation to find the next step in the simulation or use machine learning to provide the dataset and choose the algorithm or part of the algorithm to run.
Good platforms are relatively easy to set up to absorb new technologies, and infrastructure, which starts as a group processing engine, is a good example of Spark in memory processing and processing flow coupled with learning machine on and as a traditional SQL query database supported and as different file systems have been launched. All kinds of things have come and gone to the Hadoop platform.
I do not know if you can contact Hadoop again. It’s interesting to consider how this platform is more focused, or maybe they deviate or stay different forever so something new comes together, like Hadoop, when you do it.
It would be interesting to see if anyone can ever just analyze the platform, but if you look at the use in the market database, evidence shows that every time you think you have a database that can do everything, researchers with spring database just like mushrooms.
It’s hard to get the same platform uniformity.
I think it as an old ecosystem that is included primarily on open source projects that benefit from interoperability as new things come together. They interact with older things and some longer things become no longer interesting when something better comes together. Some things keep their utilities.
Therefore, many are more tools and faster and faster than they have in the past as open source innovation is driven more by users. Many new technologies come out of people’s frustration with existing tools and they see how they can add new gadgets that build existing objects and give them what they need.
Doug Cutting: The way I think we see more capabilities.
So you do not necessarily have solutions that are generated and promoted by vendors, so we get this fast new tool that can be used to create it.
They put it there and others can try and see who else is useful. When there are many people who find it useful, open source openers like ourselves begin to support them. We’ve seen this happen with Spark, with Kafka, with some things now.
For the Hadoop arrangement, I think it is less likely to be disturbed. We have this new ecosystem style, and I think it’s a basic disruption. It’s simply no longer everything builtaround the relational database management system that is heavily guarded by several vendors.