Skip to content

Tutorial: Pull a VDS from the Lucd UDS into a Local Dask Dataframe

Background on Lucd

The Lucd Enterprise AI Data Science Platform is a highly secure, scalable, open and flexible platform for persisting an fusing large and numerous datasets and training AI models for production against those datasets. The Lucd platform is an end to end platform that can be deployed in public cloud environments, on premise on bare metal hardware, or the Lucd multi-tenant PaaS can be directly accessed. The platform consists of:

  • A scalable open data ingest capability
  • A petabyte scale unified data space data repository
  • 3-D Visualization and Exploration
  • An Exploratory Data Analysis Rest Service
  • A Kubernetes environment to train PyTorch and TensorFlow models
  • NLP Word Embedding and Explainable AI Assets
  • Model results visualization and exporting to internal or external serving capability

Introduction, Prerequisites

This tutorial covers leveraging the Lucd Python Client to pull a Virtual Data Set (VDS) from data in the Lucd Unified Data Space (UDS) into a Local Dask Dataframe via a Jupyter Notebook. Creating the VDS leveraging the Lucd 3D Graphical UI; Creating custom EDA operations; or Locally creating AI models to upload to the Lucd platform are outside of scope of this Tutorial and (are/will be) covered in other Tutorials.

Prerequisites are:

  • Obtaining a Lucd account with appropriate security settings to access/retrieve data. https://community.lucd.ai/hc/en-us/articles/360037995531
  • Obtaining the Lucd Python Client package (in the future pip install will be available). For now, obtain by contacting marketing@lucd.ai
  • Downloading and installing a Jupyter notebook (this tutorial assumes that an Anaconda Jupyter notebook is used)

1. Run Setup on the Lucd Python Client

  • Extract the Lucd Python Package from the zip file.
  • From the Anaconda Cmd Prompt or from Anaconda Navigator, navigate to the Lucd Python Package folder
  • run: python setup.py

2. Import the following into your notebook

In [1]:
import lucd
from lucd import LucdClient, log
from eda.int import asset
from eda.int import vds
from eda.int import uds
from eda.lib import lucd_uds

3. Access Lucd with your account information

In [2]:
client = lucd.LucdClient(domain="<your domain>", #i.e. "https://p1.lucd.ai"
                         username="<your username>",
                         password="<your password>",
                         )

Look at data in the Lucd Unified Data Space

In [3]:
all_uds = uds.sources({"uid": "<your username"})

Your view will look different depending on your security group, the below is an example of the result

In [4]:
all_uds
Out[4]:
[{'bytes': 1840691474,
  'lastIngest': 1575571138618,
  'records': 1697533,
  'source': 'AMAZON'},
 {'bytes': 69111650,
  'lastIngest': 1575566305862,
  'records': 50000,
  'source': 'IMDB'},
 {'bytes': 10000,
  'lastIngest': 1575564236229,
  'records': 150,
  'source': 'IRIS'},
 {'bytes': 5033542244,
  'lastIngest': 1575657022337,
  'records': 2121379,
  'source': 'MIMICIII'},
 {'bytes': 3465039079,
  'lastIngest': 1575582766041,
  'records': 8807303,
  'source': 'NYC_GREEN_TAXI'},
 {'bytes': 47316594862,
  'lastIngest': 1575932176659,
  'records': 111330249,
  'source': 'NYC_YELLOW_TAXI'},
 {'bytes': 2165822,
  'lastIngest': 1575584734133,
  'records': 7032,
  'source': 'TELCO_CHURN'}]

creating a Virtual Data Set (VDS) from data in the Unified Data Space (UDS) via access to the Lucd 3D UI Client is outside the scope of this tutorial, but when a VDS is created from data in the UDS, you can view it as follows:

In [5]:
all_vds, http = vds.read({"uid": "<your username>"})

Your VDS view will be different, the below is an example

In [6]:
all_vds
Out[6]:
{'demouser_9223370452718499796': {'description': 'single day',
  'model': {'data': ['green-taxi.extra',
    'green-taxi.fare_amount',
    'green-taxi.mta_tax',
    'green-taxi.passenger_count',
    'green-taxi.total_amount',
    'green-taxi.trip_distance'],
   'labels': []},
  'name': 'Taxi Dataset',
  'operations': [],
  'query': {'aggs': {'agg_source': {'aggs': {'agg_model': {'aggs': {'topHits': {'top_hits': {'size': 10}}},
       'terms': {'field': 'model'}}},
     'terms': {'field': 'source'}}},
   'dataset': '637197317169885378',
   'query': {'function_score': {'functions': [{'random_score': {}}],
     'query': {'bool': {'filter': [],
       'must': [{'bool': {'should': [{'match_phrase': {'source': 'nyc_green_taxi'}}]}},
        {'range': {'content_date': {'gte': 1514782800000,
           'lt': 1514869200000}}}],
       'must_not': []}}}},
   'size': 100},
  'query_size': 284306,
  'username': 'demouser'},
 'demouser_9223370455919155658': {'description': '',
  'model': {'data': ['processed_text'], 'labels': ['movie.merged_rating']},
  'name': 'Fused Movie Reviews',
  'operations': [{'command': 'replace',
    'dataset': '637165315088193671',
    'orient': 'records',
    'parameters': {'to_replace': {'movie.sentiment': 'positive'}, 'value': 1},
    'return': '637165319546076371'},
   {'command': 'replace',
    'dataset': '637165319546076371',
    'orient': 'records',
    'parameters': {'to_replace': {'movie.sentiment': 'negative'}, 'value': 0},
    'return': '637165319694960920'},
   {'command': 'replace',
    'dataset': '637165319694960920',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 1}, 'value': 0},
    'return': '637165320266581891'},
   {'command': 'replace',
    'dataset': '637165320266581891',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 2}, 'value': 0},
    'return': '637165320430572434'},
   {'command': 'replace',
    'dataset': '637165320430572434',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 3}, 'value': 1},
    'return': '637165320537972770'},
   {'command': 'replace',
    'dataset': '637165320537972770',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 4}, 'value': 1},
    'return': '637165320676379191'},
   {'command': 'replace',
    'dataset': '637165320676379191',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 5}, 'value': 1},
    'return': '637165320806496350'},
   {'command': 'fuse_wherenull',
    'dataset': '637165320806496350',
    'orient': 'records',
    'parameters': {'column_a': 'movie-tv.review_text',
     'column_b': 'movie.review',
     'column_new': 'movie.merged_review'},
    'return': '637165321507283631'},
   {'command': 'fuse_wherenull',
    'dataset': '637165321507283631',
    'orient': 'records',
    'parameters': {'column_a': 'movie.sentiment',
     'column_b': 'movie-tv.rating_overall',
     'column_new': 'movie.merged_rating'},
    'return': '637165321967327511'},
   {'command_sequence': {'operations': {'lemmatize': True,
      'remove_digits': True,
      'remove_punctuation': True,
      'remove_stopwords': True,
      'remove_whitespace': True},
     'tokenizer_mode': 'document'},
    'dataset': '637165321967327511',
    'orient': 'records',
    'return': '637165322522343277',
    'text_attribute': 'movie.merged_review'}],
  'query': {'aggs': {'agg_source': {'aggs': {'agg_model': {'aggs': {'topHits': {'top_hits': {'size': 10}}},
       'terms': {'field': 'model'}}},
     'terms': {'field': 'source'}}},
   'query': {'function_score': {'functions': [{'random_score': {}}],
     'query': {'bool': {'filter': [],
       'must': [{'query_string': {'query': 'wesley blade'}},
        {'bool': {'should': [{'match_phrase': {'source': 'amazon'}},
           {'match_phrase': {'source': 'imdb'}}]}},
        {'range': {'content_date': {'gte': None, 'lt': None}}}],
       'must_not': []}}}},
   'size': 1000},
  'query_size': 7956,
  'statistics': {},
  'username': 'demouser'},
 'demouser_9223370456091712540': {'description': 'IMDB Reviews Only',
  'model': {'data': ['processed_text'], 'labels': ['movie.sentiment']},
  'name': 'IMDB Reviews',
  'operations': [{'command': 'replace',
    'dataset': '637163591949123503',
    'orient': 'records',
    'parameters': {'to_replace': {'movie.sentiment': 'positive'}, 'value': 1},
    'return': '637163593401661726'},
   {'command': 'replace',
    'dataset': '637163593401661726',
    'orient': 'records',
    'parameters': {'to_replace': {'movie.sentiment': 'negative'}, 'value': 0},
    'return': '637163593629770013'},
   {'command_sequence': {'operations': {'lemmatize': True,
      'remove_digits': True,
      'remove_punctuation': True,
      'remove_stopwords': True,
      'remove_whitespace': True},
     'tokenizer_mode': 'document'},
    'dataset': '637163593629770013',
    'orient': 'records',
    'return': '637163597024183142',
    'text_attribute': 'movie.review'}],
  'query': {'aggs': {'agg_source': {'aggs': {'agg_model': {'aggs': {'topHits': {'top_hits': {'size': 10}}},
       'terms': {'field': 'model'}}},
     'terms': {'field': 'source'}}},
   'query': {'function_score': {'functions': [{'random_score': {}}],
     'query': {'bool': {'filter': [],
       'must': [{'query_string': {'query': 'tom'}},
        {'bool': {'should': [{'match_phrase': {'source': 'imdb'}}]}},
        {'range': {'content_date': {'gte': None, 'lt': None}}}],
       'must_not': []}}}},
   'size': 1000},
  'query_size': 1004,
  'statistics': {'postcompute': {'counts': {'art_entities': {'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'date_entities': {'art_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'fac_entities': {'art_entities': {},
      'date_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'gpe_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'language_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'law_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'loc_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'money_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.file_hash': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.file_name': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.file_seq': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.review': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.sentiment': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'norp_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'ordinal_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'org_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'original_text_length': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'percent_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'person_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'processed_text': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'processed_text_length': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'product_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'quantity_entities': {},
      'time_entities': {}},
     'quantity_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'time_entities': {}},
     'time_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {}}}},
   'precompute': {'counts': {'movie.file_hash': {'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {}},
     'movie.file_name': {'movie.file_hash': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {}},
     'movie.file_seq': {'movie.file_hash': {},
      'movie.file_name': {},
      'movie.review': {},
      'movie.sentiment': {}},
     'movie.review': {'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.sentiment': {}},
     'movie.sentiment': {'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {}}}}},
  'username': 'demouser'},
 'demouser_9223370456095032890': {'description': 'Using only Amazon Reviews',
  'model': {'data': ['processed_text'], 'labels': ['movie-tv.rating_overall']},
  'name': 'Amazon Reviews',
  'operations': [{'command': 'replace',
    'dataset': '637163558301241882',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 1.0}, 'value': 0},
    'return': '637163560064858063'},
   {'command': 'replace',
    'dataset': '637163560064858063',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 2.0}, 'value': 0},
    'return': '637163560249436240'},
   {'command': 'replace',
    'dataset': '637163560249436240',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 3.0}, 'value': 1},
    'return': '637163560381207503'},
   {'command': 'replace',
    'dataset': '637163560381207503',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 4.0}, 'value': 1},
    'return': '637163560492518501'},
   {'command': 'replace',
    'dataset': '637163560492518501',
    'orient': 'records',
    'parameters': {'to_replace': {'movie-tv.rating_overall': 5.0}, 'value': 1},
    'return': '637163560605692551'},
   {'command_sequence': {'operations': {'lemmatize': True,
      'remove_digits': True,
      'remove_punctuation': True,
      'remove_stopwords': True,
      'remove_whitespace': True},
     'tokenizer_mode': 'document'},
    'dataset': '637163560605692551',
    'orient': 'records',
    'return': '637163561425954826',
    'text_attribute': 'movie-tv.review_text'}],
  'query': {'query': {'function_score': {'functions': [{'random_score': {}}],
     'query': {'bool': {'filter': [],
       'must': [{'query_string': {'query': 'wesley'}},
        {'bool': {'should': [{'match_phrase': {'source': 'amazon'}}]}},
        {'range': {'content_date': {'gte': None, 'lt': None}}}],
       'must_not': []}}}},
   'size': 1000},
  'query_size': 2437,
  'statistics': {},
  'username': 'demouser'},
 'demouser_9223370456702943543': {'description': 'regression dataset',
  'model': {'data': ['flower.petal_length',
    'flower.petal_width',
    'flower.sepal_length'],
   'labels': ['flower.sepal_width']},
  'name': 'IRIS Regression',
  'operations': [],
  'query': {'aggs': {'agg_source': {'aggs': {'agg_model': {'aggs': {'topHits': {'top_hits': {'size': 10}}},
       'terms': {'field': 'model'}}},
     'terms': {'field': 'source'}}},
   'query': {'function_score': {'functions': [{'random_score': {}}],
     'query': {'bool': {'filter': [],
       'must': [{'bool': {'should': [{'match_phrase': {'source': 'iris'}}]}},
        {'range': {'content_date': {'gte': None, 'lt': None}}}],
       'must_not': []}}}},
   'size': 1000},
  'query_size': 150,
  'statistics': {'postcompute': {'counts': {'flower.petal_length': {'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.petal_width': {'flower.petal_length': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_length': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_width': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.species': {}},
     'flower.species': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {}}}},
   'precompute': {'counts': {'flower.petal_length': {'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.petal_width': {'flower.petal_length': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_length': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_width': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.species': {}},
     'flower.species': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {}}}}},
  'username': 'demouser'},
 'demouser_9223370456709135637': {'description': 'NLP data',
  'model': {'data': ['processed_text'], 'labels': ['movie.sentiment']},
  'name': 'IMDB: wesley',
  'operations': [{'command_sequence': {'operations': {'lemmatize': True,
      'remove_digits': True,
      'remove_punctuation': True,
      'remove_stopwords': True,
      'remove_whitespace': True},
     'tokenizer_mode': 'document'},
    'dataset': '637157422643312643',
    'orient': 'records',
    'return': '637157423577005709',
    'text_attribute': 'movie.review'}],
  'query': {'aggs': {'agg_source': {'aggs': {'agg_model': {'aggs': {'topHits': {'top_hits': {'size': 10}}},
       'terms': {'field': 'model'}}},
     'terms': {'field': 'source'}}},
   'query': {'function_score': {'functions': [{'random_score': {}}],
     'query': {'bool': {'filter': [],
       'must': [{'query_string': {'query': 'wesley'}},
        {'bool': {'should': [{'match_phrase': {'source': 'imdb'}}]}},
        {'range': {'content_date': {'gte': None, 'lt': None}}}],
       'must_not': []}}}},
   'size': 1000},
  'query_size': 96,
  'statistics': {'postcompute': {'counts': {'art_entities': {'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'date_entities': {'art_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'fac_entities': {'art_entities': {},
      'date_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'gpe_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'language_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'law_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'loc_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'money_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.file_hash': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.file_name': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.file_seq': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.review': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'movie.sentiment': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'norp_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'ordinal_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'org_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'original_text_length': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'percent_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'person_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'processed_text': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'processed_text_length': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'product_entities': {},
      'quantity_entities': {},
      'time_entities': {}},
     'product_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'quantity_entities': {},
      'time_entities': {}},
     'quantity_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'time_entities': {}},
     'time_entities': {'art_entities': {},
      'date_entities': {},
      'fac_entities': {},
      'gpe_entities': {},
      'language_entities': {},
      'law_entities': {},
      'loc_entities': {},
      'money_entities': {},
      'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {},
      'norp_entities': {},
      'ordinal_entities': {},
      'org_entities': {},
      'original_text_length': {},
      'percent_entities': {},
      'person_entities': {},
      'processed_text': {},
      'processed_text_length': {},
      'product_entities': {},
      'quantity_entities': {}}}},
   'precompute': {'counts': {'movie.file_hash': {'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {}},
     'movie.file_name': {'movie.file_hash': {},
      'movie.file_seq': {},
      'movie.review': {},
      'movie.sentiment': {}},
     'movie.file_seq': {'movie.file_hash': {},
      'movie.file_name': {},
      'movie.review': {},
      'movie.sentiment': {}},
     'movie.review': {'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.sentiment': {}},
     'movie.sentiment': {'movie.file_hash': {},
      'movie.file_name': {},
      'movie.file_seq': {},
      'movie.review': {}}}}},
  'username': 'demouser'},
 'demouser_9223370459654976634': {'description': 'Entire IRIS Dataset',
  'model': {'data': ['flower.petal_length',
    'flower.petal_width',
    'flower.sepal_length',
    'flower.sepal_width'],
   'labels': ['flower.species']},
  'name': 'IRIS Dataset',
  'operations': [],
  'query': {'aggs': {'agg_source': {'aggs': {'agg_model': {'aggs': {'topHits': {'top_hits': {'size': 10}}},
       'terms': {'field': 'model'}}},
     'terms': {'field': 'source'}}},
   'query': {'bool': {'filter': [],
     'must': [{'bool': {'should': [{'match_phrase': {'source': 'iris'}}]}},
      {'range': {'content_date': {'gte': None, 'lt': None}}}],
     'must_not': []}},
   'size': 1000},
  'statistics': {'postcompute': {'counts': {'flower.petal_length': {'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.petal_width': {'flower.petal_length': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_length': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_width': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.species': {}},
     'flower.species': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {}}}},
   'precompute': {'counts': {'flower.petal_length': {'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.petal_width': {'flower.petal_length': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_length': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_width': {},
      'flower.species': {}},
     'flower.sepal_width': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.species': {}},
     'flower.species': {'flower.petal_length': {},
      'flower.petal_width': {},
      'flower.sepal_length': {},
      'flower.sepal_width': {}}}}},
  'username': 'demouser'}}

Pull a VDS into a local Dask Dataframe

Identify the VDS ID

In [7]:
for my_dict_list in all_vds:
    print(all_vds[my_dict_list]['name'] + " is from key: " + my_dict_list)
Taxi Dataset is from key: demouser_9223370452718499796
Fused Movie Reviews is from key: demouser_9223370455919155658
IMDB Reviews is from key: demouser_9223370456091712540
Amazon Reviews is from key: demouser_9223370456095032890
IRIS Regression is from key: demouser_9223370456702943543
IMDB: wesley is from key: demouser_9223370456709135637
IRIS Dataset is from key: demouser_9223370459654976634

I.e. if you want pull the Taxi Dataset VDS, its VDS ID is: demouser_9223370452718499796

Pull the VDS into a local Dask Dataframe:

In [ ]:
ddf = lucd_uds.get_dataframe("<VDS ID>")

Now you have a local copy of the VDS in a dask dataframe to work with per your requirements

It is assumed that appropriate packages are imported into your notebook (i.e. dask, pandas)

I.e. Working with a dask dataframe:

In [ ]:
ddf.head()

I.e. Converting the dask dataframe to a pandas dataframe

In [ ]:
pdf = ddf.compute()

I.e. Writing the dask or pandas dataframe to a csv

In [ ]:
ddf.to_csv('/path/to/myfiles.csv', single_file = True)
pdf.to_csv('/path/to/myfiles.csv')
In [ ]: