
Auto-ML systems have made significant progress in recent years, with various approaches and techniques being developed to automate the machine learning process. These systems aim to reduce the manual effort required for building and deploying machine learning models, making them more accessible to non-experts.
One key aspect of Auto-ML is the use of hyperparameter tuning, which is a crucial step in model selection. According to research, the optimal hyperparameters can significantly impact the performance of a model, with some studies showing improvements of up to 30% in accuracy.
Auto-ML systems often employ ensemble methods to combine the predictions of multiple models. This approach can lead to improved performance and robustness, as seen in a study where an ensemble of models achieved a 25% reduction in error rate compared to a single model.
The Auto-ML landscape is diverse, with various techniques being used to automate different aspects of the machine learning process. From model selection to hyperparameter tuning, each approach has its strengths and weaknesses, and researchers are continually working to improve and refine these methods.
Intriguing read: Data Cleaning and Preprocessing
Related Work
Auto-ML systems are often aimed at developing an end-to-end ML workflow or model. This is in contrast to human-in-the-loop ML tools that focus on specific parts of the ML workflow.
Existing Auto-ML systems are a key area of prior work that our research builds upon. These systems aim to automate the process of developing a machine learning model from start to finish.
Human-centered ML work practices are another area of prior work that informs our research. This involves understanding how humans work with machine learning models and tools to improve the overall process.
Recommended read: Generative Ai Research Papers
vs Literature
Auto-ML systems are often aimed at developing an end-to-end ML workflow or model, unlike other human-in-the-loop ML tools that are more focused on specific parts of the ML workflow.
Existing Auto-ML systems are a key area of prior work that our research builds upon.
These systems aim to automate the entire machine learning process, from data collection to model deployment.
Human-centered ML work practices are another area that informs our approach, highlighting the importance of collaboration and communication in the ML development process.
Human-in-the-loop ML tools, which involve human interaction at various stages of the ML workflow, are also an area of prior work that we draw upon.
Our research seeks to combine the strengths of Auto-ML systems and human-centered work practices to create a more effective and efficient ML development process.
A different take: Right to Explanation
Deficiencies of Auto-ML
Auto-ML, or automated machine learning, is a powerful tool that can save time and effort in the machine learning process. However, it's not without its limitations.
One major deficiency of Auto-ML is its lack of interpretability, as mentioned in the "Interpretability of Auto-ML" section, where it's stated that Auto-ML models can be difficult to understand and explain.
Auto-ML models often rely on pre-defined hyperparameter settings, which can lead to suboptimal performance, as shown in the "Hyperparameter Tuning" section, where it's noted that manually tuning hyperparameters can lead to better results.
Another issue with Auto-ML is its dependence on high-quality training data, as highlighted in the "Data Quality" section, which emphasizes the importance of clean and accurate data for Auto-ML models to perform well.
The "Evaluation Metrics" section also points out that Auto-ML models can be biased towards certain evaluation metrics, leading to overfitting and poor generalization.
Furthermore, Auto-ML models can be vulnerable to adversarial attacks, as mentioned in the "Robustness" section, where it's noted that Auto-ML models can be easily fooled by carefully crafted inputs.
Overall, while Auto-ML is a valuable tool, its limitations and deficiencies must be carefully considered to ensure its effective use.
Auto-ML Systems
Auto-ML Systems are a game-changer in the field of machine learning. Auto-Keras is an efficient neural architecture search system that uses Bayesian optimization to guide the network morphism for efficient neural architecture search.
Several Auto-ML frameworks have been developed to simplify the process of machine learning. These frameworks include Auto-sklearn, H2O, and TPOT, which have been used in various benchmarks and studies. Auto-sklearn is among the first academic tools that is still actively maintained, while H2O is of interest due to its ability to run on distributed machines.
Auto-ML frameworks can be categorized into subtasks such as Neural Architecture Search, Hyperparameter Optimization, and Automated Feature Engineering. For example, Auto-Keras is specifically designed for Neural Architecture Search, while TabPFN is a Transformer that solves small tabular classification problems in a second, which can be related to Automated Feature Engineering.
Here are some of the key Auto-ML frameworks mentioned in the article:
- Auto-sklearn
- H2O
- TPOT
- MLjar
- AutoGluon
- Polyaxon
- PyCaret
Auto-Keras: Neural Architecture Search System
Auto-Keras is a novel framework that enables Bayesian optimization to guide the network morphism for efficient neural architecture search. It's a powerful tool for finding the best neural network architecture for a given problem.
Auto-Keras is designed to be efficient, which is crucial for large-scale neural architecture search. By using Bayesian optimization, it can search through a vast space of possible architectures quickly and effectively.
One of the key benefits of Auto-Keras is its ability to outperform state-of-the-art neural architecture search methods. In some cases, it can even achieve a 4.0% improvement in ImageNet top1 accuracy over MobileNetV3, or the same accuracy but 1.5x faster.
Auto-Keras is a part of the Auto-ML systems, which are designed to automate the process of building and deploying machine learning models. It's an exciting area of research that has the potential to revolutionize the way we build and use machine learning models.
Here are some key features of Auto-Keras:
- Efficient neural architecture search
- Bayesian optimization
- Ability to outperform state-of-the-art methods
Overall, Auto-Keras is a powerful tool that can help researchers and practitioners build more effective and efficient neural networks. Its ability to automate the process of neural architecture search makes it an essential part of any Auto-ML system.
Model Selection
Auto-ML systems are designed to automate the process of selecting the best machine learning model for a given problem. They can identify different types of models as best performing, even for the same dataset and task.
Auto-sklearn and TPOT, both built upon the sklearn library, found gradient boosting regressors as the best performing model in most cases. This suggests that tree-based models are a good starting point for many problems.
Other Auto-ML frameworks like MLjar found almost exclusively tree-based models as the best performing models, with extremely randomized trees being the most frequent. This highlights the robustness of tree-based models, which show less hyperparameter sensitive spread.
In contrast, H2O found neural networks as best performing ones, while Auto-sklearn also considered other models such as stochastic gradient descent (SGD) linear regressor, support vector machine (SVM), Gaussian process regression, and Bayesian automatic relevance determination (ARD) regression. This diversity of model selection is a key benefit of Auto-ML systems.
The tree-based models were most frequently evaluated as the best performing models, followed by neural networks (H2O only) and linear regressors. This suggests that tree-based models are a safe choice for many problems, but it's worth considering other options as well.
Explainability
Auto-ML systems are designed to make machine learning more accessible, but it's essential to understand how they work. Auto-ML objects are fully supported through the H2O Model Explainability interface.
One of the most impressive features of H2O Model Explainability is its ability to generate a large number of multi-model comparison and single model plots with just a single call to h2o.explain(). This makes it incredibly easy to visualize and compare the performance of different models.
You can use this feature to gain a deeper understanding of your Auto-ML model's strengths and weaknesses, and make informed decisions about how to improve it.
GBM Hyperparameters
GBM Hyperparameters are crucial for the performance of GBM models. The GBM model uses a set of hyperparameters to control its behavior.
The GBM model uses the following hyperparameters: col_sample_rate, col_sample_rate_per_tree, learn_rate, max_depth, min_rows, min_split_improvement, ntrees, and sample_rate.
The GBM model's learn_rate is hard coded to 0.1, which means it cannot be changed during the AutoML process. This value is a fundamental aspect of the GBM model.
The GBM model's max_depth is searchable over a range of values from 3 to 17. This means that the AutoML system can try different values of max_depth to find the best one.
The GBM model's min_rows is searchable over a range of values from 1 to 100. This means that the AutoML system can try different values of min_rows to find the best one.
Here are the searchable values for the GBM model's hyperparameters:
Benchmarks
Auto-ML Systems are constantly being improved and compared to one another through various benchmarks. The Chalearn-AutoML-1 dataset is one of the benchmarks used to evaluate these systems.
The best model for this dataset is aad_freiburg, according to the Analysis of the AutoML Challenge Series 2015–2018 paper. This model has been used to achieve impressive results.
OrdinalDataset is another benchmark used to evaluate Auto-ML systems. Zero-shot-BERT-SORT is the best model for this dataset, as described in the BERT-Sort: A Zero-shot MLM Semantic Encoder on Ordinal Features for AutoML paper.
Here are some of the key benchmarks used to evaluate Auto-ML systems:
Human-Centered ML Work Practices
Human-centered ML work practices are crucial for balancing user control and automation in Auto-ML systems.
Auto-ML systems are often aimed at developing an end-to-end ML workflow or model, but human-centered ML work practices propose a human-guided approach to ML that balances user control and automation.
Existing work has explored the use of visualization to help users gain insights into the black-box process of Auto-ML, but it's essential to understand the perspective of data science practitioners and their attitudes towards Auto-ML systems.
Researchers have studied the perceptions of transparency and trust in Auto-ML, but most of these studies have focused on participants without Auto-ML experience, demoing new Auto-ML prototypes to study subjects.
Our study builds on this line of work by studying users who have worked with present-day Auto-ML tools in real-world applications to investigate how Auto-ML fits into their end-to-end ML workflow and its limitations.
Human-guided ML builds on the premise that ML development should be a collaborative activity between human users and the machine, wherein users specify their domain knowledge or desirable model behavior at a high level and the system performs some form of automated search to generate an appropriate ML pipeline or model.
Study Analysis
In this study, the researchers conducted an in-depth analysis of the data collected from interviews with participants.
They audio-recorded and transcribed all but one interview, where the participant did not consent to being recorded.
The team used an online tool called Dedoose to independently code the data and identify common themes that arose in the interviews.
Study Analysis
In our study, we audio-recorded and transcribed all but one interview, where the participant didn't consent to being recorded.
We used an online tool called Dedoose for open coding, which allowed us to map data onto categories.
The first three co-authors independently coded the data, then met weekly to discuss themes and concepts and establish consensus in a code book.
We conducted a categorization exercise to identify common themes, including advantages and disadvantages of Auto-ML, perceptions of Auto-ML, and workflow strategies with and without Auto-ML.
We used codes to facilitate the process of theory development, and intentionally avoided calculating inter-rater reliability to avoid marginalizing perspectives.
Weight of Correlations
Understanding the weight of correlations in machine learning models is crucial to grasping the impact of whistleblowing attributes on results.
In this study, researchers used the validation dataset to determine the weight of correlations for each attribute in different machine learning algorithms.
The weight of correlations can be interpreted from the results of various machine learning models, specifically from the validation dataset.
The attributes' weight of correlations in the different machine learning algorithms are listed in Table 3, providing a clear picture of their influence.
Each attribute's weight of correlations can be used to understand how it affects the outcomes of different machine learning models.
The researchers aimed to understand how demographic and TPB attributes impact the results of machine learning models, making the weight of correlations a vital aspect of their research.
The weight of correlations helps to identify which attributes have the most significant influence on the models, allowing for more informed decision-making.
Results
The datasets used in this paper are presented in detail, providing a solid foundation for the evaluation process.
The four AutoML frameworks compared in this study are Auto-sklearn, MLjar, and two others.
A benchmark comparison is presented, highlighting the performance of each framework on various tasks.
Four Results
We grouped users and use cases into high-level categories in our research.
The participants reported common tasks in ML workflows, such as enumerating these tasks in Section 4.2.

Auto-ML tools can benefit from system and UI improvements to address deficiencies like those described in Section 4.4.
These deficiencies can be solved using existing techniques, as we suggest in Section 5.
The roles of the human developer that Auto-ML tools must respect and preserve are outlined in Section 4.5.
The datasets used in our research are presented in detail in the results section.
The AutoML frameworks were selected through a specific process, which is explained in the results.
NCV and unified data preparation are key components of our evaluation process.
Dataset Specific Results
In the dataset specific results, we see a significant improvement in accuracy when using the pre-trained model on the test dataset. The accuracy rate jumps to 92%.
The model's performance is particularly strong on images with clear lighting conditions, where it achieves an accuracy rate of 95%. This is likely due to the model's ability to learn from the large dataset of well-lit images.
However, the model struggles with images taken in low light conditions, where the accuracy rate drops to 80%. This highlights the importance of collecting and incorporating a diverse range of images into the dataset.
The model's performance also varies depending on the specific object being classified, with the highest accuracy rates achieved on images of cars and bicycles.
Performance Comparison
In this study, the performance of four AutoML frameworks was compared, and Auto-sklearn emerged as the best performing framework overall.
Auto-sklearn's strong performance in the short training time of 15 minutes is particularly remarkable, but its performance degrades with longer training times, indicating an overfitting effect.
The warm start method used by Auto-sklearn contributes to its strong performance in short training times, but this advantage disappears at longer training times, where MLjar can surpass its results.
MLjar's results are the most robust in this study, with the smallest gap between quantiles.
Each AutoML framework investigates a different model and hyperparameter space, and follows a different approach to solve the CASH problem, resulting in varying performance depending on the task and data.
To achieve the best performance for a given data-mining problem, it is beneficial to run several AutoML frameworks compared to just using one framework.
Here are the performance metrics used in this study to evaluate prediction accuracy:
Discussion
The results of the AutoML paper have far-reaching implications that warrant further discussion. Special attention should be drawn to the spread of the results.
Training time is a crucial aspect that has been evaluated in the study. The results show that training time is a significant factor.
Usability is another important aspect that has been considered in the research. The algorithm choices made in the study have a direct impact on usability.
The spread of the results is a critical aspect that needs to be evaluated. The study draws special attention to this aspect.
Algorithm choices have a significant impact on the overall performance of the AutoML system. The researchers have carefully considered these choices in the study.
Materials Engineering Datasets
Materials Engineering Datasets are a great place to start when it comes to exploring the world of AutoML.
One of the most well-known datasets in this space is NAS-Bench-101, which is a popular choice for researchers and engineers alike.
Here are some other notable datasets in the field of Materials Engineering:
- NAS-Bench-101
- MedMNIST
- PMLB
- NAS-Bench-1Shot1
- CSS10
- OrdinalDataset
- Chalearn-AutoML-1
Datasets
Datasets play a crucial role in materials engineering, and several notable ones have been developed for research and practical applications.
The NAS-Bench-101 dataset is a widely used benchmark for neural architecture search, containing 422,610 unique architectures and their corresponding performance metrics.
The MedMNIST dataset is a medical imaging dataset consisting of 60,000 images of medical conditions, useful for training machine learning models to diagnose and classify diseases.
PMLB stands for Python Machine Learning Benchmark, a collection of 64 datasets for comparing the performance of machine learning algorithms.
The NAS-Bench-1Shot1 dataset is another benchmark for neural architecture search, containing 20,000 unique architectures and their performance metrics.
The following datasets are commonly used in materials engineering:
Implementation Setup
To set up an implementation for materials engineering datasets, you'll first need to define the scope of your project. This involves identifying the specific materials you want to focus on, such as metals, ceramics, or polymers.
The next step is to gather relevant dataset sources, which can include government databases, academic repositories, and commercial providers. For instance, the National Institute of Standards and Technology (NIST) provides a comprehensive database of materials properties.
A key consideration is data standardization, ensuring that all datasets are formatted consistently to facilitate integration and analysis. This can be achieved through techniques like data normalization and feature scaling, as seen in the example of the Materials Project dataset.
Data preprocessing is also crucial, involving tasks like data cleaning, handling missing values, and removing outliers. This step is essential for ensuring the quality and reliability of your analysis results.
The choice of machine learning algorithm depends on the specific problem you're trying to solve and the characteristics of your dataset. For example, the use of neural networks can be effective for predicting material properties, as demonstrated in the article's discussion of the Materials Genome Initiative.
Implementation Setup
To set up an AutoML implementation, you'll need to have a dataset ready. This dataset should be in a format that can be easily read by the AutoML algorithm, such as a CSV file.
The AutoML algorithm will then use this dataset to train a model, which can be done using a variety of machine learning algorithms, including decision trees, random forests, and support vector machines.
For example, the AutoML algorithm can use a decision tree algorithm to train a model on a dataset of customer transactions, which can then be used to predict future transactions.
Training Time
Training time is a crucial aspect of AutoML frameworks, and it's interesting to note that extended training time does not necessarily lead to significantly increased performance. In fact, for Auto-sklearn, the median relative R^2 and relative RMSE are even higher for the short training time of 15 minutes.
The performance decrease with longer training times is related to overfitting induced by Hyperparameter Optimization (HPO). This means that runtime needs to be treated as a new hyperparameter.
One way to mitigate this overfitting is to use nested cross-validation, which strengthens the necessity of this approach. However, H2O and MLjar do show some improvements for a longer training time, likely due to the finer search using Grid Search or Random Search as HPO.
Here are the stopping strategies for AutoML runs that can be specified:
- max_runtime_secs: specifies the maximum time that the AutoML process will run for, with a default of 0 (no limit), but dynamically sets to 1 hour if none of max_runtime_secs and max_models are specified.
- max_models: specifies the maximum number of models to build in an AutoML run, excluding the Stacked Ensemble models, with a default of NULL/None.
In contrast, the best AutoML aggregation showed an increased performance of the median relative RMSE and consistent performance of the median relative R^2, indicating that aggregation can provide benefits in terms of performance.
EfficientDet: Scalable Object Detection
EfficientDet is a scalable object detection model that has become increasingly important in computer vision because model efficiency has become increasingly important in computer vision. This model is designed to be efficient, making it a great choice for applications where resources are limited.
In order to implement EfficientDet, you'll need to have a good understanding of object detection and how it works. Object detection is a type of computer vision task that involves identifying and locating objects within an image or video.
EfficientDet is a scalable model, meaning it can be easily adapted to different sizes and complexities of images. This makes it a great choice for applications where the size and complexity of the images may vary.
One of the key benefits of EfficientDet is its ability to detect objects quickly and accurately, even in complex scenes. This is due in part to its use of a novel architecture that is designed to be efficient and scalable.
Optional Data Parameters
If you want to exclude certain columns from the set of predictors, you can specify them in the x argument. This argument only needs to be specified if you want to exclude columns from the set of predictors.
You can also specify a validation frame using the validation_frame argument. However, this argument is ignored unless nfolds is set to 0.
The leaderboard_frame argument allows you to specify a particular data frame to use for scoring and ranking models on the leaderboard. If this argument is not specified, the leaderboard will use cross-validation metrics instead.
You can use the blending_frame argument to specify a frame for computing predictions that serve as the training frame for the Stacked Ensemble models metalearner. If provided, all Stacked Ensembles produced by AutoML will be trained using Blending (a.k.a. Holdout Stacking) instead of the default Stacking method based on cross-validation.
The fold_column argument specifies a column with cross-validation fold index assignment per observation. This is used to override the default, randomized, 5-fold cross-validation scheme for individual models in the AutoML run.
The weights_column argument specifies a column with observation weights. Giving some observation a weight of zero is equivalent to excluding it from the dataset; giving an observation a relative weight of 2 is equivalent to repeating that row twice.
Here are the optional data parameters in a concise list:
- x: A list/vector of predictor column names or indexes
- validation_frame: A frame to be used for early stopping of individual models and early stopping of the grid searches
- leaderboard_frame: A particular data frame to use for scoring and ranking models on the leaderboard
- blending_frame: A frame to be used for computing the predictions that serve as the training frame for the Stacked Ensemble models metalearner
- fold_column: A column with cross-validation fold index assignment per observation
- weights_column: A column with observation weights
Optional Miscellaneous Parameters
Some API calls require additional parameters to function properly, such as the API key.
The API key is usually a required parameter, and it's used to authenticate and authorize API requests.
You can also specify a specific version of the API to use, as shown in the "Versioning" section.
Versioning allows you to use different versions of the API, each with its own set of features and changes.
To specify the response format, you can use the "format" parameter, which can be set to "json" or "xml".
The "format" parameter is optional, but it's useful for customizing the response to your needs.
You can also specify a specific endpoint to use, as shown in the "Endpoints" section.
Endpoints allow you to access specific resources or operations within the API.
Some API calls also support pagination, which allows you to retrieve a limited number of results at a time.
Pagination is useful for large datasets, where retrieving all results at once would be impractical.
You can specify the number of results to retrieve using the "limit" parameter, as shown in the "Pagination" section.
The "limit" parameter is optional, but it's useful for controlling the amount of data returned.
Xgboost Memory Requirements
XGBoost requires its own memory outside the H2O cluster, so make sure to allocate enough RAM for it.
To avoid memory issues, leave at least 2/3 of the total available RAM for XGBoost, as it's included in H2O by default.
For example, if you have 60G of RAM, use h2o.init(max_mem_size="40G") to allocate 40G for H2O, leaving 20G for XGBoost.
See what others are reading: H2o Automl
Scikit-Learn Compatibility
H2OAutoML can interact with the h2o.sklearn module, allowing you to combine it with pure scikit-learn components in pipelines.
The h2o.sklearn module exposes two wrappers for H2OAutoML: H2OAutoMLClassifier and H2OAutoMLRegressor.
These wrappers expose the standard API familiar to scikit-learn users, including fit, predict, fit_predict, score, get_params, and set_params.
H2OAutoML can accept various formats as input data, including H2OFrame, numpy arrays, and pandas DataFrames.
For an example of using H2OAutoML with the h2o.sklearn module, you can click on the provided link.
Performance Metrics
Performance Metrics are crucial in evaluating the accuracy of a machine learning model, especially when dealing with imbalanced data. The study used metrics like accuracy and classification error to assess the model's performance, but these metrics have limitations as they don't distinguish between classes.
Accuracy can be misleading when dealing with imbalanced data, as it gives equal weight to all classes. In this study, the dataset had an inherent imbalance in the target label distribution, making it essential to use supplementary metrics.
Sensitivity and specificity were employed to gauge the model's performance with respect to specific classes. Sensitivity denotes the total number of correct predictions for the whistleblowing cases, while specificity signifies the accurate prediction of non-whistleblowing cases.
The study utilized a Receiver Operating Characteristics (ROC) Curve graph to provide a more comprehensive visualization of the trade-off between sensitivity and specificity. The Area Under Curve (AUC) of the ROC curve was then used to quantify the model's overall classification ability.
A higher AUC indicates a more potent model in effectively classifying both cases of whistleblowing and non-whistleblowing students. The study's design gained robustness through a clearer description of the model's performance with respect to the specific challenges posed by the imbalanced distribution of the target label.
Experimental Features
AutomL has made significant advancements in experimental features, allowing for more efficient and effective model training. One such feature is the ability to handle complex datasets with multiple features and outputs.
This is achieved through the use of ensemble methods, which combine the predictions of multiple models to produce a more accurate result. The authors of the AutomL paper demonstrate this by using an ensemble of 10 models to achieve a 10% improvement in accuracy.
The experimental results show that AutomL can handle datasets with up to 100 features, which is a significant improvement over traditional machine learning methods. This is particularly useful for datasets with a large number of irrelevant features.
AutomL also includes a feature for handling imbalanced datasets, where one class has a significantly larger number of instances than the others. This is achieved through the use of class weights, which assign more importance to the minority class.
Benchmarks
Automl paper has made significant strides in recent years, with various benchmarks showcasing its capabilities. One such benchmark is the Chalearn-AutoML-1 dataset, which was analyzed in the paper "Analysis of the AutoML Challenge Series 2015–2018".
The best model for this dataset is aad_freiburg. This model was used to achieve top results in the AutoML challenge series.
The OrdinalDataset is another benchmark, where the Zero-shot-BERT-SORT model achieved impressive results, as seen in the paper "BERT-Sort: A Zero-shot MLM Semantic Encoder on Ordinal Features for AutoML".
Check this out: Claude 3 Paper
Methods
AutomL methods can be broadly categorized into two types: model-based and ensemble-based. Model-based methods involve selecting a pre-trained model and fine-tuning it for the specific task at hand.
One common model-based approach is to use a pre-trained neural network as a starting point and then modify it to fit the specific problem. This can be done by adding or removing layers, adjusting the learning rate, and fine-tuning the model on the target dataset.
Ensemble-based methods, on the other hand, involve combining the predictions of multiple models to produce a final output. This can be done through techniques such as stacking, bagging, and boosting.
Preprocessing
Preprocessing is a crucial step in the machine learning process. As of H2O 3.32.0.1, AutoML now has a preprocessing option with minimal support for automated Target Encoding of high cardinality categorical variables.
The only currently supported option is preprocessing=["target_encoding"], which automatically tunes a Target Encoder model and applies it to columns that meet certain cardinality requirements for the tree-based algorithms (XGBoost, H2O GBM, and Random Forest).
Work to improve the automated preprocessing support is documented in this ticket, aiming to enhance model performance and customization.
Data Split Dependency
Data Split Dependency is a crucial aspect to consider in machine learning, especially when working with small datasets. The performance of models can vary significantly due to different random train-test splits in the outer loop of Nested Cross Validation (NCV).
For instance, in the task Su_bond-1, the performance of H2O for a runtime of 60 min varies between R^2_rel = 1.01 and R^2_rel = -1.66. This variation can lead to under- or overestimation of the true performance of the model.
It's essential to provide information about the mean performance and deviation to gain insight into the stability of the model's performance. Without this information, it's challenging to interpret the results accurately.
Small datasets are more susceptible to overfitting and underfitting due to the limited sample size used for the train or test split. This can result in models that don't capture enough information from the training dataset.
For tasks with high variance in performance, careful selection of training and test data is crucial. Nested cross-validation can increase the trustworthiness of the results by providing a more robust estimate of the model's performance.
Required Stopping Parameters
To ensure your AutoML run stops at the right time, you need to specify a stopping strategy. This can be either a time limit or a limit on the number of models built.
You can set a time limit using the max_runtime_secs argument. This specifies the maximum time the AutoML process will run for, and the default is 0, which means no limit. However, if you don't specify a time limit and a number of models, the AutoML process will dynamically set the time limit to 1 hour.
Alternatively, you can set a limit on the number of models built using the max_models argument. This defaults to None, meaning there's no limit, but it's always a good idea to set this parameter to ensure reproducibility.
Here are the details of the stopping parameters:
- max_runtime_secs: specifies the maximum time the AutoML process will run for (default is 0 or 1 hour if not specified)
- max_models: specifies the maximum number of models to build (defaults to None)
Experimental Features
Experimental Features are a crucial part of the research process. They allow scientists to test new ideas and techniques without committing to a full-scale study.
One example of an experimental feature is the randomized controlled trial, which involves randomly assigning participants to different groups to see how they respond to a particular treatment or intervention. This type of study helps researchers understand cause-and-effect relationships.
The use of statistical analysis is a key component of experimental features. By analyzing data from these studies, researchers can identify patterns and trends that might not be apparent through other methods.
For instance, in a study on the effects of a new medication, researchers might use statistical analysis to compare the outcomes of patients who received the medication to those who received a placebo. This helps identify any potential benefits or risks.
Experimental features can also involve the use of control groups, which serve as a baseline for comparison. By comparing the outcomes of the experimental group to the control group, researchers can determine the effectiveness of a particular treatment or intervention.
In some cases, experimental features may involve the use of pilot studies to test the feasibility and effectiveness of a larger study. These smaller-scale studies can help researchers refine their methods and identify potential issues before embarking on a full-scale study.
Glm Hyperparameters
GLM uses its own internal grid search rather than the H2O Grid interface. This means that AutoML builds a single model with lambda_search enabled and passes a list of alpha values.
The alpha values searched over during AutoML grid search are {0.0,0.2,0.4,0.6,0.8,1.0}.
Validation and Testing
To ensure accurate results, it's essential to validate and test machine learning models. If cross-validation is turned off, a holdout frame must be provided to score models on and generate performance metrics for the leaderboard.
Without cross-validation, a validation frame is also required for early stopping on models. If neither frame is provided, they will be automatically partitioned from the training data.
Up to 20% of the training data may be used to create a missing frame, depending on the specific circumstances. For example, if both the holdout and validation frames are missing, a total of 20% of the training data will be used to create a 10% validation and 10% leaderboard frame.
In TPOT, validation accuracy is printed out at each iteration to be averaged, and testing accuracy is displayed using the score function. This allows for the evaluation of different machine learning algorithms and their optimal hyper-parameters.
Table 2

Let's take a closer look at Table 2, which shows the results of prediction from TPOT Python. The population size is a crucial factor in determining the suggested machine learning algorithm with optimal hyper-parameters.
For a population size of 10, the suggested algorithm is RandomForestClassifier with specific parameters, resulting in a validation accuracy of 0.67 and a testing accuracy of 0.69.
As the population size increases to 20, the suggested algorithm shifts to GaussianNB with certain parameters, achieving a validation accuracy of 0.69 and a testing accuracy of 0.60.
With a population size of 30, the suggested algorithm becomes BernoulliNB with specific parameters, resulting in a validation accuracy of 0.71 and a testing accuracy of 0.55.
Lastly, for a population size of 40, the suggested algorithm is GradientBoostingClassifier with certain parameters, achieving a validation accuracy of 0.72 and a testing accuracy of 0.66.
Here's a summary of the results:
These results demonstrate the impact of population size on the suggested machine learning algorithms and their performance.
Validation and Testing
If the user turns off cross-validation, a holdout frame is required to score models on and generate model performance metrics for the leaderboard.
To ensure this, the holdout frame will be automatically partitioned from the training data if not provided by the user. This frame will be used to populate the leaderboard.
If the holdout frame is missing, 10% of the training data will be used to create it. If both the holdout and validation frames are missing, a total of 20% of the training data will be used to create a 10% validation and 10% leaderboard frame.
The TPOT Python output displays different machine learning algorithms with optimal hyper-parameters from various runs of population sizes.
The validation accuracy at each iteration/generation from the validation process is printed out to be averagely calculated, giving users a clear idea of how their models are performing.
In general, all algorithms suggested by TPOT are able to achieve good validation and testing accuracies, with the GradientBoostingClassifier outperforming others when the population size is set to 40.
Featured Images: pexels.com




