Solutions Review’s listing of the best data engineering tools is an annual mashup of products that best represent current market conditions, according to the crowd. Our editors selected the best data engineering tools and software based on each solution’s Authority Score; a meta-analysis of real user sentiment through the web’s most trusted business software review sites and our own proprietary five-point inclusion criteria.

The editors at Solutions Review have developed this resource to assist buyers in search of the data engineering tools to fit the needs of their organization. Choosing the right vendor and solution can be a complicated process — one that requires in-depth research and often comes down to more than just the solution and its technical capabilities. To make your search a little easier, we’ve profiled the best data engineering tools and software providers all in one place. We’ve also included platform and product line names and introductory software tutorials straight from the source so you can see each solution in action.

Note: The best data engineering tools are listed in alphabetical order.

The Best Data Engineering Tools

Amazon Web Services

Platform: Amazon Redshift

Description: Amazon Redshift is a fully-managed cloud data warehouse that lets customers scale up from a few hundred gigabytes to a petabyte or more. The solution enables users to upload any data set and perform data analysis queries. Regardless of the size of the data set, Redshift offers fast query performance using familiar SQL-based tools and business intelligence applications. AWS also has multiple ways to do cluster management depending on user skill level.

Cloudera

Description: Cloudera provides a data storage and processing platform based on the Apache Hadoop ecosystem, as well as a proprietary system and data management tools for design, deployment, operations, and production management. Cloudera acquired Hortonworks in October 2018. It followed that up with a buy of San Mateo-based big data analytics provider Arcadia Data last September. Cloudera’s new integrated data management product (Cloudera Data Platform) enables analytics across hybrid and multi-cloud.

Fivetran

Platform: Fivetran

Description: Fivetran is an automated data integration platform that delivers ready-to-use connectors, transformations and analytics templates that adapt as schemas and APIs change. The product can sync data from cloud applications, databases, and event logs. Integrations are built for analysts who need data centralized but don’t want to spend time maintaining their own pipelines or ETL systems. Fivetran is easy to deploy, scalable, and offers some of the best security features of any provider in the space.

Google Cloud

Platform: Google BigQuery

Description: Google offers a fully-managed enterprise data warehouse for analytics via its BigQuery product. The solution is serverless and enables organizations to analyze any data by creating a logical data warehouse over managed, columnar storage, and data from object storage and spreadsheets. BigQuery captures data in real-time using a streaming ingestion feature, and it’s built atop the Google Cloud Platform. The product also provides users the ability to share insights via datasets, queries, spreadsheets and reports.

Looker

Platform: Looker

Related products: Powered by Looker

Description: Looker offers a BI and data analytics platform that is built on LookML, the company’s proprietary modeling language. The product’s application for web analytics touts filtering and drilling capabilities, enabling users to dig into row-level details at will. Embedded analytics in Powered by Looker utilizes modern databases and an agile modeling layer that allows users to define data and control access. Organizations can use Looker’s full RESTful API or the schedule feature to deliver reports by email or webhook.

Microsoft

Platform: Power BI

Related products: Power BI Desktop, Power BI Report Server

Description: Microsoft is a major player in enterprise BI and analytics. The company’s flagship platform, Power BI, is cloud-based and delivered on the Azure Cloud. On-prem capabilities also exist for individual users or when power users are authoring complex data mashups using in-house data sources. Power BI is unique because it enables users to do data preparation, data discovery, and dashboards with the same design tool. The platform integrates with Excel and Office 365, and has a very active user community that extends the tool’s capabilities.

Mongo DB

Platform: Mongo DB Atlas

Description: MongoDB is a cross-platform document-oriented database. It is classified as a NoSQL database program and uses JSON-like documents with schema. The software is developed by MongoDB and licensed under the Server Side Public License. Key features include ad hoc queries, indexing, and real-time aggregation, as well as a document model that maps to the objects in your application code. MongoDB provides drivers for more than 10 languages, and the community has built dozens more.

Segment

Platform: Segment

Description: Segment offers a customer data platform (CDP) that collects user events from we band mobile apps and provides a complete data toolkit to the organization. The product is available in three iterations, depending on the user persona (Segment for Marketing Teams, Product Teams or Engineering Teams). Segment works by letting you standardize data collection, unify user records, and route customer data into any system where it’s needed. The solution also touts more than 300 integrations.

Snowflake

Platform: Snowflake Cloud Data Platform

Description: Snowflake offers a cloud data warehouse built atop Amazon Web Services. The solution loads and optimizes data from virtually any source, both structured and unstructured, including JSON, Avro, and XML. Snowflake features broad support for standard SQL, and users can do updates, deletes, analytical functions, transactions, and complex joins as a result. The tool requires zero management and no infrastructure. The columnar database engine uses advanced optimizations to crunch data, process reports, and run analytics.

Tableau Software

Platform: Tableau Desktop

Related products: Tableau Prep, Tableau Server, Tableau Online, Tableau Data Management

Description: Tableau offers an expansive visual BI and analytics platform, and is widely regarded as the major player in the marketplace. The company’s analytic software portfolio is available through three main channels: Tableau Desktop, Tableau Server, and Tableau Online. Tableau connects to hundreds of data sources and is available on-prem or in the cloud. The vendor also offers embedded analytics capabilities, and users can visualize and share data with Tableau Public.

The post The 10 Best Data Engineering Tools (Commercial) for 2025 appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.

Analyst house Gartner, Inc. is back at it with another major report, having just released their new 2017 Magic Quadrant for Data Integration Tools. Integration tools fill an important role in the enterprise, acting as the data acquisition stalwart for Business Intelligence, analytics and data warehousing. Integration software also sources and delivers application and master data in support of Data Management. The ultimate benefit of Data Integration comes from the consistency that it breeds across operational applications and the enterprise environment.

Gartner highlights that the emerging concept of a Data Lake is one of the driving forces behind widespread adoption of integration tools. Report authors explain: “The need for integrating nonrelational structures and distributing computing workloads to parallelized processes elevates data integration challenges. At the same time, it also provides opportunities to assist in the application of schemas at data read time, if needed, and to deliver data to business users, processes or applications, or to use data iteratively.”

In this Magic Quadrant, Gartner evaluates the strengths and weaknesses of 15 providers that it considers most significant in the marketplace and provides readers with a graph (the Magic Quadrant) plotting the vendors based on their ability to execute and their completeness of vision. The graph is divided into four quadrants: niche players, challengers, visionaries, and leaders. At Solutions Review, we read the report, available here, and pulled out a few observations.

Mega-vendors Informatica and IBM remain the two top dogs for another year, though the former increased its distance from the pack with a broad presence in the Big Data market. The leaders bracket also includes big names SAP, Talend, Oracle, and SAS. No surprises here, as this space continues to be dominated by large-scale, enterprise-focused companies. The only change in pecking order among these providers is Talend leapfrogging Oracle based on better product execution.

Keeping on theme with the large integration tools providers, Microsoft headlines the market challengers. The world’s largest technology company had their standing improved considerably as a result of strong support for diverse data types, and synergy between data, applications, business roles and artificial intelligence. Attunity and Adeptia round out this quadrant, with both toeing the niche players line, though Attunity did show some improvement in its completeness of vision ranking.

Information Builders tumbled on the vertical scale a bit, but still remains the class of the visionaries column. They offer a diverse integration tools portfolio and scored within the top-25 percent in this report for positive customer experience. Cisco and Denodo remain tightly grouped in the middle-left portion of the bracket. Both providers have made their mark with strong data virtualization capabilities, but Denodo has achieved strong momentum, growth and “mind share” within this sub-market by expanding partnerships with technology and service providers.

Actian saw their standing regress vertically while Syncsort has supplanted them as the most notable provider in the niche players graph. Pentaho makes a triumphant return to this Magic Quadrant between the two. Their customer reference base includes examples of all three deployment models of integration, including very large customers across back-office, Internet of Things and machine/sensor data tools, as well as traditional integration software demands.

Though up to 80 percent of all organizations still make significant use of bulk/batch and traditional integration techniques, message, virtualization and synchronization tools are also being used. Gartner believes that somewhere between 35 and 45 percent of enterprise companies are using two or more approaches. Gartner believes that this will result in a need to blend traditional deployments with modern infrastructure practices in the near future.

Read Gartner’s Magic Quadrant.

The post What’s Changed: 2017 Gartner Magic Quadrant for Data Integration Tools appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.

Source: Forrester

Enterprise technology analyst house Forrester Research has recently released the latest version of its Data Preparation Tools Wave Report for Q1 2017. In the 21-criteria evaluation of data prep solutions, Forrester researcher Cinny Little identifies the seven providers whom are most significant in the category – Alteryx, Datawatch, Oracle, Paxata, SAS, Trifacta, and Unifi Software – then researched, analyzed, and scored them. The Wave report details the findings and examines how each vendor meets (or falls short of) Forrester’s evaluation criteria and where vendors stand in relation to each other.

According to Forrester, data prep tools are now must haves, as their proprietary survey data has showed that analytics professionals seek low-friction access to data on-demand. They add: “Customer-obsessed firms align on key customer-centric metrics and take the actions that matter most on the insights they derive from data. Forrester projects that insights-driven businesses — companies that embed analytics and software deeply into their customer-centric operating model — will grow revenue at least eight times faster than global GDP.” That’s an impressive projection.

In order to help Big Data professionals select the right tools, The Forrester Wave Report outlines the current state of the market and separates the top providers into Leaders, Strong performers and Contenders. At Solutions Review, we’ve read the report, available for download here, and pulled a few of  the most important takeaways:

Business users require ease of use and scalable execution architectures

Data preparation is a pre-processing step that allows for the transformation of data before analysis to ensure quality and consistency, providing enterprises with maximum potential for Business Intelligence. Given the growing volumes and velocity of Big Data, integration acts as a significant barrier to the overall data preparation scheme. From a tactical perspective, generating data quality too remains a challenge.

Traditional Data Management techniques get in the way of analytical agility. As a result, business users are choosing tools that provide not only speed, but transparency and oversight that will provide scalability. Machine learning and cross-enterprise collaboration are also key features on many organizational wish lists.

Trifacta and Paxata are on a planet of their own

Forrester argues that these two providers offer the most comprehensive and scalable platforms of any of the providers covered in this report, citing self-service and speed as major advantages. These solution providers are staples in the Big Data software market, and have staked their respective claims on the throne of the rapidly growing data prep sector.

Trifacta leverages machine learning algorithms to automate data interactions that allow self-service data wrangling for analysts and business users alike. Not only is their platform top-of-class, but they offer an expansive list of customer programs and resources including a curriculum, certification program, and more. Paxata’s platform is based on a set of technologies that unite Data Integration, quality, governance, collaboration and enrichment. They combine a well-received user interface with machine learning, text and semantic analytics for quick speedy data connection. Customers enjoy Paxata’s usability and time-to-value.

Datawatch and Unifi each lead their respective niche

Datawatch has made semi-structured and unstructured data sources the priority, and as Forrester points out, they’ve been in this business long before the buzzword ‘Big Data’ was ever mainstream. Customers gave Monarch, their flagship offering, high scores for ease of use and automation capabilities. Unifi Software combines self-service data discovery and prep into a unified platform, and does it all in a patented six-step process that includes: connect, discover, cleanse/enrich, transform, and format. Machine learning capabilities enable the tool to learn from organizational actions and make recommendations to the user at each step. According to the report, Unifi’s natural language search is the strongest among all the vendors in this market.

Read Forrester’s Wave for Data Preparation Tools, Q1 2017.

[youtube https://www.youtube.com/watch?v=beLcYabuj6c&w=560&h=315]

The post Key Takeaways: Forrester Data Preparation Tools, Q1 2017 appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.

Gartner recently released the 2016 version of their Magic Quadrant for Data Integration Tools. According to Gartner, the market for enterprise integration tools was worth $2.8 billion at the end of 2015, representing a 10.5 percent increase from the year prior. The technology research giant defines Data Integration capabilities as: “Comprising the practices, architectural techniques and tools that ingest, transform, combine and provision data across the spectrum of information types in the enterprise and beyond — to meet the data consumption requirements of all applications and business processes.”

According to Gartner, organizations are increasingly looking for solutions that provide them with Data Virtualization capabilities and the ability to combine Data Lakes with their existing platforms since the overbearing expectation is that Data Integration will become cloud and on-premise agnostic. As a result, vendors are increasingly offering tools that enable widespread data access and data delivery infrastructure for a wide variety of integration scenarios, including data acquisition for Business Intelligence, Data Analytics, and data warehousing, data migration, data sharing, and support for Master Data Management and Data Governance.

Demand for legacy tools has waned as modern capabilities evolve, with new tools offering ways for organizations to split the demands of Data Integration so that they may integrate both data and processes with partners and growing digital customer bases. While legacy bulk/batch integration still makes up the vast majority of the software market (Gartner estimates 80 percent), forward-thinking companies have begun to shift their focus to Data Virtualization and synchronization. In this way, mobile devices, consumer applications, multi-channel interactions and social media are driving enterprises to build sophisticated integration architectures.

Informatica and IBM remain the class of this market, furthering their standing as the unquestioned leaders in enterprise Data Integration. Informatica doesn’t have much room left to expand, and if they continue to dominate the market they way they have in recent years, Gartner may have to handicap them. Informatica continues to build upon their impressive offerings portfolio, and recently unveiled several new hourly-priced AWS Data Management tools. IBM remains Informatica’s one true challenger for dominance in the integration sector.

SAP and Oracle remain tightly grouped in the middle of the leaders bracket, though both mega-vendors lost standing in Gartner’s completeness of vision metric. SAP has supplanted itself as a leader in this software sector, and with no end in sight, the company continues to attract customers that seek a mix of granularity, latency and physical and virtualized data delivery. Oracle’s strong 2015 came as a result of releasing GoldenGate for Big Data, which includes push-down to Spark and the introduction of self-service integration capabilities for data preparation. Talend and SAS round out the leaders column in this year’s study.

Microsoft now finds itself well within striking distance of joining Talend as a market leader after improving in Gartner’s completeness of vision metric this year. Attunity, the only new vendor in this year’s report, and Chicago-based Adeptia, join Microsoft as market challengers. Attunity has strong traction in this space and has been delivering data replication and synchronization capabilities for more than two decades. Adeptia’s integration tool offers attractive pricing and flexibility, as well as iPaaS capabilities, enabling inenterprise data sharing use cases.

Visionaries in this market segment for 2016 are Information Builders, Cisco, and Denodo. Information Builders does it all, offering integration software, Big Data support, and Business Intelligence and Data Analytics. Denodo’s placement as a challenger is an upgrade over last year’s stay in the niche player’s bracket. Actian regressed in this year’s report, though did improve their ability to executive metric, and although Gartner views the vendor as a niche player for 2016, Actian is within striking distance of upgrading their standing considerably with a positive year ahead. Syncsort remains a niche player, and their standing in the column was downgraded noticeably. Better days are surely ahead, as Syncsort offers high performance ETL processing, a low cost of ownership compared to market leaders, and swift time-to-value.

Read Gartner’s Magic Quadrant.

[youtube https://www.youtube.com/watch?v=beLcYabuj6c&w=560&h=315]

The post What’s Changed: 2016 Gartner Magic Quadrant for Data Integration Tools appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.

Source: Gartner

Gartner recently released the 2016 version of their Magic Quadrant for Enterprise Integration Platforms as a Service (iPaaS). iPaaS tools are cloud-based and provide platform support to application and Data Integration projects that involve a combination of cloud and on-premise data sources. According to Gartner, iPaaS will be the integration platform of choice for new integration projects in the very near future, and from an annual revenue perspective, will leave traditional application integration suites in the dust.

The expansion in popularity of iPaaS tools is evident by the fact that the market exploded in 2015, exceeding $400 million, good for a growth rate of more than 50 percent as a whole. Some of the top solution providers even saw their revenues grow by triple digits. Though iPaas is considered the ‘Data Integration of the future’, it is possible that adoption will be hampered by a lack of standards and skills, incomplete offerings and the trouble that organizations may have federating it with legacy on-premise tools. In addition, there are sure to be security and privacy concerns with iPaaS just like there are with every other cloud-based solution offering.

This year’s report features 17 vendors, up from 16 in 2014. Gartner adjusts its inclusion criteria as markets evolve. As a result, solution providers are added and dropped from time to time. Four new providers were added in 2016, including Actian, DBSync, Oracle and Scribe Software. These four vendors’ offerings and commercial operations have evolved to a point where they have collected the minimum number of paid clients to meet the criteria. On the opposite side of the coin, Gartner has decided to drop Cloud Elements, Flowgear and Fujitsu, three vendors that were included in the 2015 report.

The leaders column is made up of five key players, with two in a class of their own. Dell Boomi is the definitive top dog in this iPaaS Magic Quadrant, touting 3,800 total clients (adding 1,500 in 2015 alone) and catering to medium and large organizations. Customers rate their experience using Dell iPaaS as above average. Not far behind on the graphic is Data Integration behemoth Informatica, a company that needs no explanation. While Dell and Informatica are on an island of their own, MuleSoft is not that far behind. The California-based vendor provides both iPaaS and traditional integration tools, and has a growing portfolio of capabilities. The leaders quadrant is rounded out by Jitterbit and SnapLogic.

The lone challenger in the 2016 iteration of Gartner’s Magic Quadrant is Adaptris, a UK-based company that offers a mature iPaaS tool with 3,000 global customers. Adaptris is accustomed to working with large enterprise companies and has a cloud solution that is very scalable.

Gartner has named five visionaries in this report, headlined by four of the biggest technology companies in the world: Oracle, SAP, IBM and Microsoft. These vendors have their tentacles in almost every sector of enterprise technology, with these providers being commonly included in many of Gartner’s other Magic Quadrant reports. The lone outcast of the group is Celigo, an integration vendor that entered the iPaaS market in 2008. Celigo offers ease of use and attention to non-specialists in integration, with customers rating the company as above average for commercial support, reliability and professional services.

Three new vendors make their debut in the niche players bracket, including Scribe Software, Actian and DBSync. Scribe Software is based in New Hampshire and offers a multitenant iPaaS solution that runs on all of the major IaaS providers as well as inside a client’s data center. Youredi is another niche player, which offers an iPaaS tool that focuses heavily on logistics and supply chain integration and has been built upon the Microsoft Azure platform. Attunity is the final vendor included in this report. Attunity is no stranger to the Magic Quadrant, as they are represented in a variety of industries and offer tools which satisfy companies in many verticles.

Read Gartner’s Magic Quadrant.

The post 2016 Gartner Magic Quadrant for Enterprise Integration Platform as a Service (iPaaS): Key Takeaways appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.

If you work with data on a daily basis, you’re probably familiar with the process. Data is captured through any number of sources and stored inside of a data warehouse until it is ready for extraction and transfer into an analytics tool. Traditionally speaking, the integration and analytics tools would be separate entities altogether. More and more BI providers are beginning to include integration capabilities inside their analytics platforms, and even more of the top enterprise integration vendors are moving away from the relational integration method of ETL. Vendors that started out offering Data Integration solutions are now branching out in other directions, with many of them moving in the direction of data management platforms (Apache Hadoop and Spark).

With data volumes exploding, unstructured data on the rise, and analytics vendors siphoning off features that used to be specific to integration providers alone, one has to ask, is Data Integration dead?

The truth is, Data Integration has never been more important. However, the way in which organizations traditionally integrated data, via the “relational” ETL method, is beginning to whither away. Legacy ETL integration tools cannot withstand the sheer volume, variety or velocity of data that companies now generate, rendering these tools unusable. Companies are collecting more data than ever, and from an infinite number of additional sources which include IoT device sensors, mobile applications, machine data, social media, and more. With the size, scope and diversity of data growing at an unprecedented rate, the traditional data warehouse/ETL system no longer holds up.

Business users no longer speak regularly to their data warehouse. Services now do that on our behalf, moving mixing, and matching those data sources with Big Data systems. This has created a need for Data Lakes, which will largely replace data warehouses in the future, according to some experts. Thus, existing systems have to evolve further to ensure the automation of data access for the end-user. The complexity of data sources cannot be discounted, and now that companies are moving scores and scores of data to the cloud, these pipelines need to be replaced. For Data Integration to work properly for enterprise organizations in the modern day, it has to be able to collect all of the data, not just stores that are held within a specific source.

In a world where the growing majority of collected data is unstructured in nature, it is impossible to expect a relational data warehouse to integrate in a way that will drive valuable business insight. These legacy systems simply cannot keep up with the amount of data that needs to be integrated into other environments. Data Lakes are dealing with this problem. By having the technical ability to store any type of data in any realistic capacity, newer data management tools such as Hadoop and Spark are able to act as the middleware that legacy integration tools once did. Within this evolving paradigm, every piece of data should be collected.

Traditional integration tools and techniques required stringent controls over data quality and governance in order to extract usable sets from the data warehouse. This is no longer the case, as the Data Lake can house any data type without much thought. This method also allows for a serviced-based approach to data management, which, as a result of important applications and enterprise analytics platforms, enterprises are now employing. For what it’s worth, Gartner rolled ETL into their pure-play Data Integration Tools Magic Quadrant way back in 2006. If your organization relies on only structured data and relational data warehouse techniques, then you’ve got nothing to worry about. But if your company collects or plans on collecting data from different sources in the future, the ETL method of doing business has certainly gone the way of the dinosaurs.

Widget not in any sidebars

The post Is Data Integration Dead? appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.

The concept of the Data Lake is shrouded in mystery, what is it? Is there some kind of Loch Ness data monster swimming around at the bottom of it?

As a term, Data Lake is being defined as it develops. But simply, a Data Lake holds raw data in its native format for later use. Pentaho‘s CTO James Dixon is credited with inventing the term. The practice has grown in popularity recently, and is used heavily in Big Data initiatives. As a tool, Data Lake is disrupting the Data Integration market and helping to redefine the way enterprises handle their data. Providing a more in-depth definition, a Data Lake stores disparate information while ignoring almost everything. Unlike a data warehouse or datamart, which is a small slice of a data warehouse that users extract their data from, the lake pays no attention to how or when its data will be used, governed, defined or secured.

Big Data initiatives have begun to use Data Lakes much more of late because a Data Lake holds all of its data in an unstructured, unorganized format. The data is not specialized, meaning that it can be manipulated in a variety of ways. In lots of cases, Big Data works better in this way. In the past, data warehouses were sufficient storage areas for data because they were organized better, and that’s still true. However, it becomes difficult for data scientists to uncover insights when data is pre-organized. Sure, it may take longer to get from point A to point B, but what the Data Lake has going for it is that all of the information stored within it is available at any given time, in its native format.

In a competitive world where every scrap of data matters, the Data Lake can be intriguing. Considering that the Internet of Things is the next big topic in Data Integration, its popularity should continue to grow. The Data Lake is not limited to specific, static structures like a data warehouse is. Further, in using a Data Lake, one can dictate the kinds of analysis that are possible using that data, not the other way around.

Not everything is perfect at the lake, however. While it does allow for more advanced searching of larger volumes of data, there are no unique identifiers. The extractor has to start from scratch in order to create a new data analysis since there is no metadata. It’s a lot more difficult to search through a pool of unfiltered data when nothing has a category or class designation. In short, gaining value from a Data Lake is difficult.

Since the data cannot be defined, there is no oversight as to what exactly is being dumped into a lake. Is the data useful? No one knows until it is analyzed. At least in the use of a data warehouse, data can be organized by quality. Here, it’s all meshed together. This also raises security concerns. If no one knows what kind of data resides in the lake, they might not find out that some of it is corrupt until it’s too late. Shortcomings in this space are important to note, as organizations have started using this technology with no real push for security measures. Compromises in security need to be addressed.

Business Intelligence tools have a tough time sifting through all the mud at the bottom of the lake. BI solutions, for the most part, are engineered to analyze organized data. They simply don’t function at a high level when asked to take on the task of completely unstructured information. Though data warehouses provide a lot less raw data, they are drastically more defined. One of the biggest problems in the Data Integration space to begin with was a skills gap. The use of the data lake requires more highly-skilled integrators, something that may not be available for quite some time.

In a recent post, Gartner warned against falling into the “Data Lake Fallacy.” Their viewpoint was clear: enterprises need to be careful of jumping right into Data Lakes and using them as their main integration source for analytics. They argue that while there are benefits, the industry has yet to adapt, and applications within the enterprise environment are uncertain at this point.

Andrew White, VP of Gartner writes: “The need for increased agility and accessibility for data analysis is the primary driver for Data Lakes. Nevertheless, while it is certainly true that Data Lakes can provide value to various parts of the organization, the proposition of enterprise-wide data management has yet to be realized.”

The industry has started to latch on to the Data Lake initiative. Informatica has just joined forces with Pivotal and Capgemini to put forth a Data Lake program they call Business Data Lake, a solution that aims to reign in the Data Lake and make it usable for a wider audience of businesses.

They describe the initiative: “Current Big Data solutions face limitations and are not comprehensive enough to support the data pipelines and real-time capabilities required for operational systems and often do not meet the required levels of data governance, quality and security. The Business Data Lake addresses these issues and helps businesses leverage their data in a way that makes sense, from both an individual and business perspective, rather than just a single enterprise view.”

One has to think that Data Lakes will continue to grow in popularity as the Internet of Things boom looms and all its connected devices begin to stream into the marketplace. For now though, it seems safer to store data outside of the lake, citing the concerns outlined above. It doesn’t look like Data Lakes will make warehouses obsolete any time soon, at least until someone finds a way to provide enough organization and security to them to make them worthwhile for medium-sized initiatives. But if someone swoops in to organize and secure the Data Lakes, won’t that just make them Big Data warehouses?

Read Forrester’s report for delivering governed Data Analytics at scale.

The post The Emergence of Data Lake: Pros and Cons appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.

Companies are relying on their data like never before, and this has created a need for organizations to hunker down and get serious about the quality of the information they use to gain business insights. Data Governance is by no means a new phenomenon, but one that is certainly becoming vital to the modern data-driven enterprise. In fact, Forrester polled 164 companies recently to see how they are balancing the need for analytics and Data Governance simultaneously. To successfully highlight the growing importance of governance planning, we first need to put it into perspective.

Think of the relationship between members of government and those they are elected to serve. First and foremost, the governors are supposed to keep their constituents safe. Governments organize themselves in such a way that facilitates learning, growth and the overall success of the state they represent. Governing bodies typically focus on maintaining order and ensuring the smooth function of the state.

Not to get political, but a good enterprise Data Governance plan will consist of a similar framework. Since a governance strategy consists of Data Management and data quality techniques through a system of rights to decision, the same factors will be relevant. Real Data Governance requires ongoing monitoring to promote continuous improvement, and the formation of a governance council is vital to this kind of program. The organization of the governance council should be solely based on the culture of the organization, but should include all of the company’s key stakeholders from departments near and far.

Data Governance has one goal: to standardize the efforts of people and processes to optimize data integrity and quality. However, in our modern world of increasing data volumes, it can be a great challenge for companies to balance the need for Data Governance and Data Analytics.

Like a functional government, a data-governing body should be able to compromise for the good of the organization. The governing council will exercise near-hegemonic authority, control and shared decision-making over the management of the organization’s data assets, similar to how a political governing body would a state’s natural resources. As the core component of Data Management, a proper Data Governance strategy will get stakeholders involved in deciding on data definitions and in supporting consistent data use across the business.

Governments come in many forms, shapes and sizes. Data Governance strategies can come in either highly or loosely-structured bodies, again depending on the culture that already exists within an organization. Much of what a governance plan aims to accomplish will depend on the personnel. However, when all is said and done, success depends solely on stakeholder involvement and representation, so it’s important to ensure that all of the prominent departments are accounted for and have their voices heard.

Deploying Data Governance, as you can probably imagine, is no picnic. Initially, companies must be prepared for more questions than answers, as there are sure to be challenges to data ownership and lots of inconsistencies across competing departments. However, with careful planning, the right tools, and a data governing council willing to come together for the common good of the organization, data quality can be achieved.

Many organizations struggle to find the balance between governance procedures and giving themselves the ability to perform data analytics. Thus, governance plans need to be developed in a way that still enables the deployment of analytics that can have a positive impact on the company’s bottom line.

Click here for a free download of Forrester’s study on delivering governed data for analytics at scale.

The post The Growing Importance of Data Governance appeared first on Best Data Integration Vendors, News & Reviews for Big Data, Applications, ETL and Hadoop.