DataForge expands its commitment to empowering enterprises to build, grow, and develop data products

CHICAGO, IL, UNITED STATES, June 11, 2024 /EINPresswire.com/ -- DataForge, an emerging and disruptive data integration provider, has released a new version that enhances ease of use and adds new connectivity options for Databricks customers. With these latest updates, DataForge continues to expand its commitment to empowering enterprises to build, grow, and expand data products.

Introducing Talos: DataForge’s LLM-powered assistant

With Talos, developers and analysts can now configure and interact with complex data flows using the power of generative AI. Talos democratizes pipeline infrastructure, transformation, and observability to all team members. In the first release of Talos now available in DataForge Cloud, users can perform:

Data Discovery and Automated Ingestion: Explore external database systems with plain English in order to better understand their data models prior to ingestion. Once identified, ingest and replicate the data tables to DataForge’s cloud-based storage with a simple command such as: “create a DataForge source from these tables and begin processing”.

Column-level Lineage Exploration: Search and explore the end-to-end processing chain in DataForge. From raw data to transformation logic to target mappings, quickly find and navigate from any point in the processing chain with a few quick questions to Talos. “Find me all the raw data and
transformations used in the calculation for Revenue in the Reporting output”

Enhanced Complex Data Type Support

DataForge Cloud now allows developers to avoid verbose code when working with complex data types. DataForge Cloud offers native support for array, struct, and other data types commonly encountered in semi-structured datasets such as JSON, API results, and streaming data. Users can:

Extend Schema Evolution: Automatically adapt to changing data structures by leveraging DataForge Cloud’s configuration options to add, remove, upcast, clone, or lock the schema when elements change. Users can rely on consistent behavior and treatment regardless of how the complex type changes.

Build Rule Logic: Avoid JSON parsing scripts and simplify code by using DataForge Cloud’s new simple dot notation to navigate through multiple levels—no need to build separate code for each element in the nested structure.

Simplify Table Mapping: Flatten hierarchical data with one click and publish complex data to third-party tools to support integrations with BI and Analytics technologies that do not work well with complex types. Easily mass-create columns mapped to each attribute within the nested structures.

Native Kafka integration

DataForge Cloud’s new Kafka connector enables users to ingest and publish data to and from Kafka topics without writing any custom code. Apache Kafka, popularized for its open-source distributed event streaming platform, is utilized by thousands of companies for high-performance data pipelines, streaming analytics, and mission-critical applications. With DataForge Cloud's Kafka Events integration, users can:

Integrate Kafka with your Lakehouse: Use DataForge Cloud to manage your Kafka Events, handle the ingestion, storage, and transformation of your topic datasets, and combine Kafka data with tables or files from other connectors.

Confluent Support: DataForge Cloud’s connector has deep integrations available for customers leveraging the most popular commercial offering for Kafka: Confluent. Leverage Confluent’s integrated schema registry to simplify setup and configuration of integrations.

"These new features match our mission to automate the tedious and simplify the complex" said Matt Kosovec, co-founder and CEO of DataForge. "By adding complex data types and Kafka support, we have set the foundation for adding true streaming support and automation for unstructured data in the future."

For more information about DataForge and its latest enhancements, visit www.dataforgelabs.com.

Join DataForge at Data + AI Summit 2024. Visit Booth 2 to learn more.

About DataForge

At DataForge, our mission is to make data management, integration, and analysis faster and easier than ever. DataForge, the Declarative Data Management platform, automates data transformation, orchestration, and observability. By bringing functional programming to data engineering, DataForge introduces a new paradigm for building data solutions. Avoid the pitfalls of procedural scripting and take advantage of modern software engineering principles to automate orchestration, promote code reuse, and maximize observability. Experience a new era of data engineering with DataForge, where functional programming and automation pave the way for scalable data platforms.

Dataforge Cloud is the fastest and most reliable way to deploy DataForge. Develop, orchestrate, operate, and audit functional code pipelines in an all-in-one web-based UI.

Dataforge Core is an open-source code framework and command-line tool for developing transformation functions and compiling them into executable Spark SQL. Visit GitHub to download the CLI.

Paula David
DataForge
+1 844-222-6974
paula@dataforgelabs.com
Visit us on social media:
Facebook
LinkedIn
YouTube

Legal Disclaimer:

EIN Presswire provides this news content "as is" without warranty of any kind. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author above.