My Personal Holy Trinity for Machine Learning Reproducibility

Short and direct:

ML Flow
Why I do use? (a.k.a What was my pain?)
One of the most painful situations that I faced was spent a huge time coding doing hyperparameter search and track the whole experimental setup. With ML Flow right now the only thing that I need to do it’s just investing time to pre-process the data and choose the algorithm to train; the model serialization, data serialization, packaging it’s all done by MLFlow. A great advantage it is that the best model can be deployed in a REST/API easily instead to use a customized Flask script.

Caveats: I really love Databricks but I think sometimes they’re so fast in their development (sic.) and this can cause some problems, especially if you’re relying on a very stable version and suddenly with some migration you can lose a lot of work (e.g. RDD to Dataframe) because rewrite things again.

Why I do use? (a.k.a What was my pain?)
Data pre-processing sometimes can be very annoying and there’s a lot of new tools that actually overpromise to solve it, but in reality, it’s only a over-engineer stuff with a good Marketing (see this classic provided by Daniel Molnar to understand what I’m talking about (minute 15:48))

My main wish in the last 5 years it’s package all dirty SQL scripts in a single place just to execute with decent version control using Kubernetes and Docker and throw all ETLs made in Jenkins to trash (a.k.a embrace the dirty, cold, and complex reality of ETL). Nothing less, nothing more.

So, with Pachyderm I can do that.

Caveats: It’s necessary to say that you’ll need to know Docker and embrace all the problems related, and the bug list can be a little frightening.

Why I do use? (a.k.a What was my pain?)
ML Flow can serialize data and models. But DVC put this reproducibility in another level. With less than 15 commands in bash git-like you can easily serialize one versioning your data, code, and models. You can put the entire ML Pipeline in a single place and rolling back any point in time. In terms of reproducibility I think this is the best all-round tool.

Caveats: In comparison with ML Flow the navigation over the experiments here it’s a little bit hard tricky and demands some time to get used.

My Personal Holy Trinity for Machine Learning Reproducibility

Benchmark-ML: Cutting the Big Data Hype

This is the most important benchmark project already done in Machine Learning. I’ll let for you the summary provided:

When I started this benchmark in March 2015, the “big data” hype was all the rage, and the fanboys wanted to do machine learning on “big data” with distributed computing (Hadoop, Spark etc.), while for the datasets most people had single-machine tools were not only good enough, but also faster, with more features and less bugs. I gave quite a few talks at conferences and meetups about these benchmarks starting 2015 and while at the beginning I had several people asking angrily about my results on Spark, by 2017 most people realized single machine tools are much better for solving most of their ML problems. While Spark is a decent tool for ETL on raw data (which often is indeed “big”), its ML libraries are totally garbage and outperformed (in training time, memory footpring and even accuracy) by much better tools by orders of magnitude. Furthermore, the increase in available RAM over the last years in servers and also in the cloud, and the fact that for machine learning one typically refines the raw data into a much smaller sized data matrix is making the mostly single-machine highly-performing tools (such as xgboost, lightgbm, VW but also h2o) the best choice for most practical applications now. The big data hype is finally over.

Github Repo
Benchmark-ML: Cutting the Big Data Hype

Servicing models with R using Plumber package

Shirin Glander made a great post about how to use Plumber to provide some servicing of R models an API. For whom are looking for methods to deploy ML learning algorithms in R in production, this post is mandatory.

The plumber package for R makes it easy to expose existing R code as a webservice via an API (, Trestle Technology, LLC 2017).

You take an existing R script and make it accessible with plumber by simply adding a few lines of comments. If you have worked with Roxygen before, e.g. when building a package, you will already be familiar with the core concepts. If not, here are the most important things to know:

  • you define the output or endpoint
  • you can add additional annotation to customize your input, output and other functionalities of your API
  • you can define every input parameter that will go into your function
  • every such annotation will begin with either #' or #*


p class=”js-evernote-checked” style=”padding-left:60px;”>With this setup, we can take a trained machine learning model and make it available as an API. With this API, other programs can access it and use it to make predictions.

Servicing models with R using Plumber package

Pagou por múltiplas GPUs na Azure e não consegue usar com Deep Learning? Este post é pra você.

Você foi lá no portal da Azure pegou uma NC24R que tem maravilhosos 224 Gb de memória, 24 núcleos, mais de 1Tb de disco, e o melhor: 4 placas M80 para o seu deleite completo no treinamento com Deep Learning.

Tudo perfeito, certo? Quase.

Logo de início tentei usar um script para um treinamento e com um simples htop para monitorar o treinamento, vi que o Tensor Flow estava despejando todo o treinamento nos processadores.

Mesmo com esses 24 processadores maravilhosos batendo 100% de processamento, isso não chega nem perto do que as nossas GPUs mastodônticas podem produzir. (Nota: Você não trocaria 4 Ferraris 2017 por 24 Fiat 147 modelo 1985, certo?)

Acessando a nossa maravilhosa máquina para ver o que tinha acontecido, verifiquei primeiro se as GPUs estavam na máquina, o que de fato aconteceu.

azure_teste@deep-learning:~$ nvidia-smi
Tue Jun 27 18:21:05 2017
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|   0  Tesla K80           Off  | B2A3:00:00.0     Off |                    0 |
| N/A   47C    P0    71W / 149W |      0MiB / 11439MiB |      0%      Default |
|   1  Tesla K80           Off  | C4D8:00:00.0     Off |                    0 |
| N/A   57C    P0    61W / 149W |      0MiB / 11439MiB |      0%      Default |
|   2  Tesla K80           Off  | D908:00:00.0     Off |                    0 |
| N/A   52C    P0    56W / 149W |      0MiB / 11439MiB |      0%      Default |
|   3  Tesla K80           Off  | EEAF:00:00.0     Off |                    0 |
| N/A   42C    P0    69W / 149W |      0MiB / 11439MiB |      0%      Default |

| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|  No running processes found                                                 |
azure_teste@deep-learning:~/deep-learning-moderator-msft$ lspci | grep -i NVIDIA
b2a3:00:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
c4d8:00:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
d908:00:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
eeaf:00:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)

OK, as Teslas K80 estão prontas para o combate, contudo a própria Azure reconhece que há problemas no processo como um todo, e pra fazer isso basta alguns procedimentos bem simples em dois passos segundo a documentação.


1) Clone o repositório do Git (Sim, eu dei fork pois rotineiramente isso costuma sumir por motivos que nem sempre sabemos).

$ git clone

2) Entre na pasta que foi clonada

$ cd azure-gpu-setup

3) Execute o script que irá instalar algumas libs da NVIDIA e posteriormente fará o servidor reiniciar.

$ bash


1) Vá na pasta do repositório do git

$ cd azure-gpu-setup

2) Execute o segundo script que fará a instalação do Tensorflow, do Toolkit do CUDA, e do CUDNN além de fazer o set de uma porção de variáveis de ambiente.

$ bash

3) Depois faça o teste da instalação

$ python

Depois disso é só aproveitar as suas GPUs em carga total e aproveitar para treinar as suas GPUs.

Pagou por múltiplas GPUs na Azure e não consegue usar com Deep Learning? Este post é pra você.

Como criar um Virtualenv no Python sem bullshit

Via Eiti Kimura.

Direto e reto:

1) Realize a instalação do virtualenv pelo pip

$ pip install virtualenv

2) Faça a definição do seu diretório

$ mkdir deep-learning-virtual-env

3) Após a definição, entre no diretório

$ cd deep-learning-virtual-env

4) Faça a inicialização do seu virtualenv

$ virtualenv .

5) Com isso realize a ativação do seu virtualenv

$ source bin/activate

6) Para facilitar o seu trabalho, criamos até mesmo um arquivo de requirements com o Theano, Keras, Jupyter Notebook, Scikit-Learn. Para fazer isso basta rodar o seguinte comando:

$ pip install -r requirements.key




Como criar um Virtualenv no Python sem bullshit

Análise de Múltipla Correspondência no R para o problema de Churn

Via Data Science Plus

Analytical challenges in multivariate data analysis and predictive modeling include identifying redundant and irrelevant variables. A recommended analytics approach is to first address the redundancy; which can be achieved by identifying groups of variables that are as correlated as possible among themselves and as uncorrelated as possible with other variable groups in the same data set. On the other hand, relevancy is about potential predictor variables and involves understanding the relationship between the target variable and input variables.
Multiple correspondence analysis (MCA) is a multivariate data analysis and data mining tool for finding and constructing a low-dimensional visual representation of variable associations among groups of categorical variables. Variable clustering as a tool for identifying redundancy is often applied to get a first impression of variable associations and multivariate data structure.
The motivations of this post are to illustrate the applications of: 1) preparing input variables for analysis and predictive modeling, 2) MCA as a multivariate exploratory data analysis and categorical data mining tool for business insights of customer churn data, and 3) variable clustering of categorical variables for the identification of redundant variables.

Análise de Múltipla Correspondência no R para o problema de Churn

Previsões para Deep Learning para 2017

Por James Kobielus da IBM.

(…)The first hugely successful consumer application of deep learning will come to market: I predict that deep learning’s first avid embrace by the general public will come in 2017. And I predict that it will be to process the glut of photos that people are capturing with their smartphones and sharing on social media. In this regard, the golden deep-learning opportunities will be in apps that facilitate image search, auto-tagging, auto-correction, embellishment, photorealistic rendering, resolution enhancement, style transformation, and fanciful figure inception. (…)

(…)A dominant open-source deep-learning tool and library will take the developer community by storm: As 2016 draws to a close, we’re seeing more solution providers open-source their deep learning tools, libraries, and other intellectual property. This past year, Google open-sourced its DeepMind and TensorFlow code, Apple published its deep-learning research, and the OpenAI non-profit group has started to build its deep-learning benchmarking technology. Already, developers have a choice of open-source tools for development of deep-learning applications in Spark, Scala, Python, and Java, with support for other languages sure to follow. In addition to DeepMind and TensorFlow, open tools for deep-learning development currently include DeepLearning4J, Keras, Caffe, Theano, Torch, OpenBLAS and Mxnet.(…)

(…)A new generation of low-cost commercial off-the-shelf deep-learning chipsets will come to market: Deep learning relies on the application of multilevel neural-network algorithms to high-dimensional data objects. As such, it requires the execution of fast-matrix manipulations in highly parallel architectures in order to identify complex, elusive patterns—such as objects, faces, voices, threats, etc. For high-dimensional deep learning to become more practical and pervasive, the underlying pattern-crunching hardware needs to become faster, cheaper, more scalable, and more versatile. Also, the hardware needs to become capable of processing data sets that will continue to grow in dimensionality as new sources are added, merged with other data, and analyzed by deep learning algorithms of greater sophistication. (…)

(…)The algorithmic repertoire of deep learning will grow more diverse and sophisticated: Deep learning remains a fairly arcane, specialized, and daunting technology to most data professionals. The growing adoption of deep learning in 2017 will compel data scientists and other developers to grow their expertise in such cutting-edge techniques as recurrent neural networks, deep convolutional networks, deep belief networks, restricted Boltzmann machines, and stacked auto-encoders. (…)

Previsões para Deep Learning para 2017

Novo pacote do R – forecastHybrid

Direto do Peter Stats Blog

The new forecastHybrid package for R by David Shaub and myself provides convenient access to ensemble time series forecasting methods, in any combination of up to five of the modelling approaches from Hyndman’s forecast package. These are auto-selected autoregressive integrated moving average; exponential smoothing state space (both ets and tbats); feed forward neural network with a single hidden layer and lagged inputs; and forecasts of loess-based seasonal decomposition.

Background and motivation
In an earlier post I explored ways that might improve on standard methods for prediction intervals from univariate time series forecasting. One of the tools I used was a convenience function to combine forecasts from Rob Hyndman’s ets and auto.arima functions. David Shaub (with a small contribution from myself) has now built and published an R package forecastHybrid that expands on this idea to create ensembles from other forecasting methods from Hyndman’s forecast package.

The motivation is to make it easy to improve forecasts, both their point estimates and their prediction intervals. It has been well known for many years that taking the average of rival forecast methods improves the performance of forecasts. This new R package aims to make it as easy for people to do this as to fit the individual models in the first place.

Novo pacote do R – forecastHybrid

Gradient Boosted Trees Model no Microsoft R Server

Direto do Revolutions Blog

R is an open source, statistical programming language with millions of users in its community. However, a well-known weakness of R is that it is both single threaded and memory bound, which limits its ability to process big data. With Microsoft R Server (MRS), the enterprise grade distribution of R for advanced analytics, users can continue to work in their preferred R environment with following benefits: the ability to scale to data of any size, potential speed increases of up to one hundred times faster than open source R.

In this article, we give a walk-through on how to build a gradient boosted tree using MRS. We use a simple fraud data data set having approximately 1 million records and 9 columns. The last column “fraudRisk” is the tag: 0 stands for non-fraud and 1 stands for fraud. The following is a snapshot of the data.

Gradient Boosted Trees Model no Microsoft R Server

DL-Learner – Framework para Aprendizado de Máquina

The DL-Learner software learns concepts in Description Logics (DLs) from examples. Equivalently, it can be used to learn classes in OWL ontologies from selected objects. It extends Inductive Logic Programming to Descriptions Logics and the Semantic Web. The goal of DL-Learner is to provide a DL/OWL based machine learning tool to solve supervised learning tasks and support knowledge engineers in constructing knowledge and learning about the data they created.

Purposes of Class Expression Learning

  1. Learn Definitions for Classes: Based on existing instances of an OWL class, DL-Learner can make suggestions for class definitions to be included as an owl:equivalentClass or rdfs:subClassOf Axiom. As the algorithm is biased towards short and human readable definitions, a knowledge engineer can be supported when editing the TBox of an ontology (see Protege Plugin).
  2. Find similar instances: DL-Learner’s suggested class expressions can be used to find similar instances via retrieval (Concept definitions as search). Scalable methods allow the generation of recommendations on the fly, e.g. in a web scenario (see DBpedia Navigator – in experimental stage).
  3. Classify instances: The learned class descriptions can be used in a typical classification scenario, i.e. to decide for unknown instances whether they belong to a certain class. Common ILP benchmarks have been tested with DL-Learner. On the Carcinogenesis page, DL-Learner competes with other state-of-the-art ILP algorithms.


  1. Instance Classification: A user maintains a list of favorites. Based on these favorites, OWL Concepts are learned with DL-Learner and presented to the user in Natural Language. Such a concept could be all articles about proteins that are written by researchers from Germany (e.g. in Manchester syntax: Proteins and hasAuthor some (Person and hasLocation some Germany)). New articles, which fall in this category and are added to the knowledge base, are presented to the user automatically, like a customized RSS feed.
  2. Protégé: In a family ontology, a Protégé user wants to create a definition for the Concept ‘Father’. He / She already asserted some instances to the class Father. Now, the DL-Learner Protege plugin presents the definition (in Manchester OWL syntax): Male and hasChild some Thing.


The application is written in Java. A user manual can be found PDF Documenthere. Also there is an overview, a page about its architecture, and a feature list Features. DL-Learner is available as Open-Source at Sourceforge.

It has different learning algorithms, which offer several parameters for fine-tuning. It can solve four closely related learning problems: learning based on positive and negative examples, positive only learning, and learning definitions and subclass relationships in ontologies.


As reasoning is heavily used by DL-Learners algorithms, special methods were introduced to increase performance:

  1. Fast Instance Checker is a reasoning component, that is custom tailored for the needs of DL-Learner. After an initial reasoning step on the basis of Pellet, results are pre-calculated and cached. Besides the significant performance boost, the component can optionally apply a form of closed world reasoning, which allows to learn expressions like forall and max/min cardinality. It is an approximate reasoning method, where as usual rare cases of incomplete reasoning results are justified by a huge increase in performance.
  2. DL-Learner can also provide class suggestions for very large knowledge bases, since it uses local fragment reasoning, i.e. only the relevant part (which is small) is used for learning new classes. This enables class learning in real time on knowledge bases like DBpedia. More information can be found PDF Documenthere.
DL-Learner – Framework para Aprendizado de Máquina

Análise Exploratoria de Dados utilizando gráficos

É fato que a inconsistência de dados acaba com qualquer tipo de modelagem em Data Mining.

Dessa forma, ANTES de qualquer experimento com data mining é sempre desejável que se faça uma análise exploratória de dados utilizando estatísticas descritivas, gráficos, formulação de hipóteses para uma definição clara de quais técnicas serão utilizadas.

Esse post do Eli Bressert apresenta um tutorial muito bom usando R para a utilização de gráficos na análise exploratória de dados.

Análise Exploratoria de Dados utilizando gráficos