Machine Learning Data Engineer

Lucd is relentlessly focused on revolutionizing how the AI development and production lifecycles are executed and governed in enterprise settings. This includes how technical experts traverse and collaborate on AI development workflows (e.g., perform data feature transformations, AI experimentation, AI governance and integration), as well as how business experts extract actionable insights, value, and interpretability from data analytics and AI model output. Highly performant and well-architected distributed ML data management is a core part of Lucd’s goals for success. Enterprise AI development is a rapidly evolving space, and hence our customers need a highly capable and dynamic ML platform for exploiting large multi-modal datasets for experimentation and production purposes. As a ML Data Engineer, you will be responsible for the Lucd platform’s ML data processing capabilities.

This primarily includes the following responsibilities.

  • Architecting and implementing large-scale distributed data transformation components and services for supporting applications and workflows such as AI experimentation and training, and visual analytics. There is an emphasis on using GPU-enabled capabilities.

  • Contributing to AI workflow governance solutions, especially regarding data quality and provenance.

  • Contributing to AI model production management and monitoring solutions for various types of platforms (e.g., enterprise APIs, embedded devices).

The ideal candidate is expected to embody the following characteristics.

  • Exemplary communication and (non-technical and technical) documentation skills.

  • Extreme passion for understanding and anticipating users’ requirements and preferences.

  • Boundless curiosity for identifying emerging trends in ML data engineering and processing.

  • A self-starter that has excellent time management skills, is comfortable dealing with ambiguity.

  • Highly collaborative, innovative and a creative non-linear thinker, with the execution skills to make it all count.

  • Ability to work within an exciting, distributed startup environment.

Basic Qualifications

  • Solid understanding of machine learning, data engineering, and the overall AI development workflow.

  • Strong command of computer science fundamentals (object-oriented design, data structures, algorithms, etc.).

  • Solid understanding of scalable (enterprise) software architectures, including Kubernetes and application monitoring.

  • Strong Python and C-based language development skills.

  • Working knowledge of Linux systems.

  • 2+ years of experience with distributed AI model training on large-scale datasets (e.g., using a distributed framework such as Horovod).

  • 2+ years of experience working with text and image data.

  • 2+ years of experience with using frameworks distributed computation and data processing (e.g., Dask, Spark, RAPIDS).

  • 2+ years of experience developing AI models and workflows with industry-leading frameworks such as TensorFlow, PyTorch, XGBoost, Scikit-Learn, and MXNet.

Preferred Qualifications

  • 1+ years of experience with opensource data search and storage systems such as Elasticsearch and/or Apache Accumulo.

  • 1+ years of experience with workflow management frameworks (e.g., Apache Airflow, Kubeflow).

  • 1+ years of experience with agile development, including knowledge of CI/CD toolsets (e.g., Gitlab).

  • 1+ years of experience with AI model and/or data bias detection and mitigation, and other related “AI ethics” topics.

  • 2+ years of experience with AI model deployment and monitoring for enterprise (or similar) applications.

  • M.S. in Computer Science or related field of study, with a concentration in AI or data science.