High-Volume Data Ingest¶
For AI to provide the best business value, enterprise data inputs need to be extracted from various sources (locked in various silos, of different types and formats, etc.) and loaded into a single point of access to effortlessly explore and discover patterns of interest. The Lucd Data Ingest Module uses Apache Niagara Files (NiFi) to accomplish this ETL task, which targets Lucd Unified Data Source (UDS) as a central repository for collecting and indexing large amounts of data, while automatically providing data governance. To support high-volume data demands with NiFi, Lucd deploys a highly performant NiFi setup that tailors the layout and configuration of NiFi on top of high-end Lucd compute nodes, while also leveraging pre-built/customized processors for ingesting high-volume datasets into the Lucd platform.
Large-Scale Secure Data Management¶
The Lucd Unified Data Space (UDS) provides an indexed NoSQL fused data store with role-based access control down to a data element (as opposed to record) enabling rapid and secure data discovery, controlled analytics/model building, and application integration leveraging data in the Lucd platform. The Lucd UDS fuses multiple data sources and augments with background analytics to build temporal, geospatial, entity, and other analytic indices (e.g., N-Gram). Lucd UDS Analytics also provides a metadata index across sources providing detail about data attribute classes, data and object models, as well as additional source-based statistics. The Lucd UDS accomplishes this for all data modalities, as well as for structured, unstructured, and semi-structured data. This information is exposed back to data engineers and scientists to support highly informed AI model development.
Data Exploration and Preparation¶
High-Performance 3D Client¶
As opposed to conventional data dashboards, Lucd JedAI manages data exploration and governed AI model management using the Unity 3D engine. Hence, Lucd JedAI can more fully utilize client resources (compared to a common Internet browser) in order to provide a richer, more detailed experience. Overall, this enables more possibilities to illustrate the vastness of data involved in building AI solutions for the modern enterprise.
Easy Large-Scale Data Processing Capabilities¶
Enterprise data scientists need to quickly and confidently analyze and prepare large amounts data for stakeholder education and AI model training. Lucd JedAI provides intuitive graphical controls for data search and fusion of multiple sources to enable powerful visual analysis and data transformation. Additionally, easy no-code transformation of rich data types (e.g., text, images) are supported. Data transformation workflows (powered by Dask for transparent scalability) can be split and merged and saved for rapid iteration in data preparation. For advanced scenarios, python can be used for defining and integrating custom feature transformation operations into Lucd, enabling fully extensible data processing capabilities. With Lucd’s data processing capabilities, users can focus on data, not dataframes.
Intuitive AI Model Development Capabilities¶
Developers use a range of frameworks for AI model development (e.g., TensorFlow, PyTorch). Furthermore, developing the best solutions for the enterprise requires constant model prototyping. Lucd supports this with multiple tools. The Lucd Modeling Framework provides APIs enabling seamless use of developers’ large-scale datasets (crafted in Lucd JedAI) for their models. This enables developers to focus on “modeling” and not the common burdens of coding custom data interfaces for TensorFlow and the like. The Lucd Python Library provides access to data in a user’s python environment (e.g., Jupyter Notebook), enabling model prototyping and tuning before submitting it for longer training sessions on full datasets. Overall, this makes model development for Lucd a total breeze, in many cases enabling easy re-use of previously developed models.
Enterprise Governance and Explainability¶
AI development within the enterprise requires the highest levels of trust, transparency, and interpretability. The combination of Lucd’s secure underlying data storage, comprehensive logging, and graphical views support governance over the entire development pipeline. Associations among models, data transformation, and training attempts are intuitively visualized to understand the lineage of a proposed AI solution. Model decision explainability capabilities are fully integrated into Lucd JedAI, enabling fine-grained analysis of model behavior, and supporting insightful team decisions regarding improved data preparation and model tuning. Overall, Lucd JedAI offers a common ground for multiple stakeholders to inspect AI models’ performance.