FreeDiscovery

Build the categorization ML model

The option use_hashing=True must be set for the feature extraction. Recommended options also include, weighting="ntc".

Parameters

• parent_id: dataset_id or lsi_id
• data: a list of dict which have a category field and one or several fields that can be used for indexing, such as document_id and optionally rendition_id.
• method: classification algorithm to use (default: LogisticRegression),
  • "LogisticRegression": LogisticRegression
  • "LinearSVC": Linear SVM,
  • "NearestNeighbor": nearest neighbor classifier (requires LSI)
• cv: binary, if true optimal parameters of the ML model are determined by cross-validation over 5 stratified K-folds (default False).
• training_scores: binary, compute the efficiency scores on the training dataset. This would make computations much slower for NearestNeighbors (default False).

Compute birch clustering

The option use_hashing=False must be set for the feature extraction. Recommended options for data ingestion also include, ntc.

Parameters

• parent_id: dataset_id or lsi_id
• n_clusters: the number of clusters or -1 to use hierarchical clustering (default: -1)
• min_similarity: The radius of the subcluster obtained by merging a new sample and the closest subcluster should be lesser than the threshold. Otherwise a new subcluster is started. See sklearn.cluster.Birch. Increasing this value would increase the hierarchical tree depth (and the number of clusters).
• branching_factor: Maximum number of CF subclusters in each node. If a new samples enters such that the number of subclusters exceed the branching_factor then the node has to be split. The corresponding parent also has to be split and if the number of subclusters in the parent is greater than the branching factor, then it has to be split recursively. Decreasing this value would increase the number of clusters.
• max_tree_depth : Maximum hierarchy depth (only applicable when n_clusters=-1)
• metric : The similarity returned by nearest neighbor classifier in ['cosine', 'jaccard', 'cosine-positive'].

Compute clustering (DBSCAN)

The option use_hashing=False must be set for the feature extraction. Recommended options for the data ingestion also include, weighting="ntc".

Parameters

• parent_id: dataset_id or lsi_id
• min_similarity: The radius of the subcluster obtained by merging a new sample and the closest subcluster should be lesser than the threshold. Otherwise a new subcluster is started. See sklearn.cluster.Birch
• metric : The similarity returned by nearest neighbor classifier in ['cosine', 'jaccard', 'cosine-positive'].
• min_samples: (optional) int The number of samples (or total weight) in a neighborhood for a point to be considered as a core point. This includes the point itself.

Compute K-mean clustering

The option use_hashing=False must be set for the feature extraction. Recommended options for feature extraction include, weighting="ntc".

Parameters

• parent_id: dataset_id or lsi_id
• n_clusters: the number of clusters
• metric : The similarity returned by nearest neighbor classifier in ['cosine', 'jaccard', 'cosine-positive'].

Compute near duplicates

Parameters

• parent_id: dataset_id or lsi_id
• method: str, default='simhash' Method used for duplicate detection. One of "simhash", "i-match"

Initialize the feature extraction on a document collection.

Parameters

• n_features: [optional] number of features (overlapping character/word n-grams that are hashed). n_features refers to the number of buckets in the hash. The larger the number, the fewer collisions. (default: 1100000)
• analyzer: 'word', 'char', 'char_wb' Whether the feature should be made of word or character n-grams. Option ‘char_wb’ creates character n-grams only from text inside word boundaries. ( default: 'word')

• ngram_range : tuple (min_n, max_n), default=(1, 1) The lower and upper boundary of the range of n-values for different n-grams to be extracted. All values of n such that min_n <= n <= max_n will be used.

• stop_words: "english" or "None" Remove stop words from the resulting tokens. Only applies for the "word" analyzer. If "english", a built-in stop word list for English is used. ( default: "english")

• n_jobs: The maximum number of concurrently running jobs (default: 1)
• chuck_size: The number of documents simultaneously processed by a running job (default: 5000)
• weighting: the SMART notation for document term weighting and normalization. In the form [nlabL][ntp][ncb] , see https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System

• norm_alpha: the alpha value used for pivoted normalization

• use_hashing: Enable hashing. This option must be set to True for classification and set to False for clustering. (default: True)

• min_df: When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. This value is ignored when hashing is used.
• max_df: When building the vocabulary ignore terms that have a document frequency strictly higher than the given threshold. This value is ignored when hashing is used.
• parse_email_headers: when documents are emails, attempt to parse the information contained in the header (default: False)
• preprocess: a list of pre-processing steps to apply before vectorization. A subset of ['emails_ignore_header'], default: [].
• id: (optional) custom dataset id. Can only contain letters, numbers, "_" or "-". It must also be between 2 and 50 characters long.
• overwrite: if a custom dataset id was provided, and it already exists, overwrite it. Default: false
• column_ids : list of ints. If provided the input dataset is expected to be CSV, and the columns with the provided ids are selected. Documents can only be provided using dataset_definition parameter that must contain a single file path
• column_separator: str, character used to delimit columns. Only used if column_ids is provided. Default: ','

Run feature extraction on a dataset,

Parameters

• data_dir: [optional] relative path to the directory with the input files. Either data_dir or dataset_definition must be provided.
• dataset_definition: [optional] a list of dictionaries [{'file_path': <str>, 'content': <str>, 'document_id': <int>, 'rendition_id': <int>}, ...] where document_id and rendition_id are optional, while either file_path or content field must be provided.
• vectorize: [optional] this option can be used to ingest the dataset_definition in batches (optionally with document content), then make one final call to vectorize all sent documents (bool, default: True)
• document_id_generator: [optional] if the document_id is not provided, this specifies how it is generated. If indexed_file_path the document_id is given by the index of the sorted file_path, otherwise if infer_file_path the document_id is inferred from the file_path strings, removing all non digit characters. In this second case, the file_path must contain a unique numeric ID (default: indexed_file_path)

Add new documents to an existing processed dataset. This will also automatically update the LSI model if any is present. Raw documents on disk are not affected.

This operation cannot be undone.

Warning: all categorization, clustering, duplicate detection and email threading models associated with this dataset will be removed and need to be re-trained.

Parameters

• data_dir: [optional] relative path to the directory with the input files. Either data_dir or dataset_definition must be provided.
• dataset_definition: [optional] a list of dictionaries [{'file_path': <str>, 'document_id': <int>, 'rendition_id': <int>}, ...] where rendition_id are optional, while either file_path or content field must be provided.

Remove documents from an existing processed dataset. This will also automatically update the LSI model if any is present. Raw documents on disk are not affected.

     This operation cannot be undone.

     Warning: all categorization, clustering, duplicate detection and
     email threading models associated with this dataset will be removed and
     need to be re-trained.

     **Parameters**
      - `dataset_definition`: [optional] a list of dictionaries `[{'file_path': <str>, 'document_id': <int>, 'rendition_id': <int>}, ...]` where  `rendition_id` are optional.
            

Compute correspondence between id fields for documents. At least one of the fields used for indexing must be provided, and all the rest will be computed (if available). If the data parameter is not provided, return all the correspondence table

Parameters

• data: the ids of documents used as the query
• return_file_path: whether the results should include the file path

Build a Latent Semantic Indexing (LSI) model

Recommended data ingestion options also include, use_idf=1, sublinear_tf=0, binary=0.

The recommended value for the n_components (dimensions of the SVD decompositions) is in the [100, 200] range.

Parameters

• n_components: Desired dimensionality of the output data. Must be strictly less than the number of features.
• parent_id: parent dataset identified by dataset_id
• alpha: floor on the number of components used with small datasets
• id: (optional) custom model id. Can only contain letters, numbers, "_" or "-". It must also be between 2 and 50 characters long.
• overwrite: if a custom model id was provided, and it already exists, overwrite it. Default: false

Compute categorization metrics to assess the quality of categorization.

In the case of binary categrorization, category labels are sorted alphabetically and the second one is expected to be the positive one.

Parameters

• y_true: [required] ground truth categorization data
• y_pred: [required] predicted categorization results
• metrics: [required] list of str. Metrics to compute, any combination of "precision", "recall", "f1", "roc_auc"

Compute clustering metrics to assess the quality of categorization, comparing the groud truth labels with the predicted ones.

Parameters

• labels_true: [required] list of int. Ground truth clustering labels
• labels_pred: [required] list of int. Predicted clustering labels
• metrics: [required] list of str. Metrics to compute, any combination of "adjusted_rand", "adjusted_mutual_info", "v_measure"

Compute duplicate detection metrics to assess the quality of categorization, comparing the groud truth labels with the predicted ones.

Parameters

• labels_true: [required] list of int. Ground truth clustering labels
• labels_pred: [required] list of int. Predicted clustering labels
• metrics: [required] list of str. Metrics to compute, any combination of "ratio_duplicates", "f1_same_duplicates", "mean_duplicates_count"

Perform document search (if parent_id is a dataset_id) or a semantic search (if parent_id is a lsi_id).

Parameters

• parent_id : the id of the previous processing step (either dataset_id or lsi_id)
• query : the seach query. Either query or query_document_id must be provided.
• query_document_id : the id of the document used as the search query. Either query or query_document_id must be provided.
• metric : The similarity returned by nearest neighbor classifier in ['cosine', 'jaccard', 'cosine-positive'].
• min_score : filter out results below a similarity threashold
• max_results : return only the first max_results documents. If max_results <= 0 all documents are returned.
• sort_by : if provided and not None, the field used for sorting results. Valid values are [None, 'score']
• sort_order: the sort order (if applicable), one of ['ascending', 'descending']
• batch_id: retrieve a given subset of scores (-1 to retrieve all). Default: 0
• batch_size: the number of document scores retrieved per batch. Default: 10000
  • subset_document_id: apply prediction to a subset of document_id.

Store a list of custom stop words