Logo

Hortonworks and Hewlett Packard Enterprise accelerate Apache Spark

Hortonworks and Hewlett Packard Labs are working together to enhance Apache Spark, one of the most active Apache big data projects. The collaboration will center around an entirely new class of analytic workloads that benefit from large pools of shared memory.

  • Thursday, 3rd March 2016 Posted 10 years ago in by Phil Alsop
Early results of the collaboration include the following:
  • Enhanced shuffle engine technologies: Faster sorting and in-memory computations, which has the potential to dramatically improve Spark performance.   
  • Better memory utilization: Improved performance and usage for broader scalability, which will help enable new large-scale use cases.

 

“This collaboration indicates our mutual support of and commitment to the growing Spark community and its solutions,” said Scott Gnau, chief technology officer, Hortonworks.  “We will continue to focus on the integration of Spark into broad data architectures supported by Apache YARN as well as enhancements for performance and functionality and better access points for applications like Apache Zeppelin.” 

 

“We’re hoping to enable the Spark community to derive insight more rapidly from much larger data sets without having to change a single line of code,” said Martin Fink, EVP and CTO, Hewlett Packard Enterprise and Hortonworks Board Member. “We’re very pleased to be able to work with Hortonworks to broaden the range of challenges that Spark can address.”

 

Hortonworks and Hewlett Packard Enterprise plan to contribute the new technologies to the Apache Spark community.

Fiverr’s new AI Video Hub enables brands to work directly with AI video creators on a range of...
Foxit's recent report challenges prevailing assumptions about AI's productivity benefits, revealing...
Exploring Keysight's new solution for error performance validation in AI-focused data centres,...
Databricks launches Genie Code, an autonomous AI agent designed to assist data engineers with...
Zapier's research reveals the potential of AI-powered systems in business operations, highlighting...
As AI becomes central to business operations, UK CIOs focus on aligning strategies, governance, and...
Cohesity and ServiceNow introduce solutions to support resilient and recoverable AI agent...
AvePoint introduces AgentPulse Command Centre, advancing AI agent governance across multicloud...