Which Big Data, Data Mining, and Data Science Tools go together?

comments

(co-authored with Shashank Iyer). We took anonymized data from the results of the  2015 KDnuggets Data Mining Software Poll,  and performed association analysis the top 20 tools. The dataset consisted of 2759 votes, each for one or more tools.   At the bottom of this post there is a link to download the anonymized dataset.

We used a version of Apriori algorithm to analyze the results.

There are many ways to measure how significant is associations between two nominal or binary features, eg chi-square or T-test, but we use a simple measure we call "Lift", defined as

Lift (X & Y) = pct (X & Y) / ( pct (X) * pct (Y) )

where pct(X) is the percent of users who selected X.

Lift (X&Y) > 1 indicates that X&Y appear together more than expected if they were independent,
Lift=1 if X & Y appear with frequency expected if they are independent, and
Lift < 1 if X & Y appear together less than expected (negatively correlated)

Note that this measure is symmetric:   Lift (X & Y) = Lift (Y & X)

Fig. 1 shows the heat map for the top 10 Data Mining tools.  The lift values are displayed in their respective matrix positions and the color gradient represents the degree of association from high to low.
If lift is > 1.2 the square is reddish, if less than 0.8, bluish, else grey.

Spark and Hadoop have the highest association with a lift=3.31, followed by Spark and Python (lift=2.05). We also note strong association between Excel and SQL, and Tableau and SQL.

The lowest associations were found between SAS base and KNIME (0.51), SAS base and RapidMiner (0.52), and KNIME and RapidMiner (0.56).

Fig 1: Association Matrix Heat Map for top 10 most popular data mining tools

 A similar heat map (Fig. 2) was computed showing the various associations between Open Source and Commercial tools.

 Fig. 2: Confusion Matrix Heat Map between Open Source and Commercial Tools

To visualize the correlations between top 20 most popular tools, a network graph was computed as in Fig. 3.

Each node represents a top 20 tool, and the nodes are colored Red: Free/Open Source tools, Green: Commercial tools, Fuchsia: Hadoop/Big Data tools. The node sizes  vary based on the percentage of votes each tool received. The edges are broadly categorized into two segments – low association to high association, and this is shown in the steep color gradient from light pink to dark purple respectively. This segmentation is also shown in the weights of each edge, the thicker ones showing a high association and the latter a low association.

Fig. 3: Network graph of top 20 most popular tools (click on the graph to get the higher resolution image

 Below are the top, high and low associations highlighting the data in the network graph in fig 3.

Table 1.  20 highest associations (lifts) among top 20 tools

As can be expected,  Excel frequently goes along with SQL, Hadoop with Spark, Pig with Hadoop,  IBM SPSS Modeler with IBM SPSS Statistics. Among less obvious associations we see Pig and scikit-learn, and Weka with KNIME and MATLAB.

Table 2.  20 lowest associations (lifts) among top 20 tools

Here we see that Alteryx users don't use much else (except Tableau) , IBM SPSS users don't use Unix, and KNIME and RapidMiner users don't use SAS.

Finally, we look at tools that go along with R.  Since R was used by almost half the voters, no tool can have association of more than 2, but here are the tools with the highest lifts for R.

Table 3.  Tools/Software most associated with R

Here is the link to anonymized data set (CSV format), with 3 columns

• ord: order of the vote
• ntools: number of tools
• votes: tool names, separated by ";". Note: for ease of pattern matching, R is encoded as Rlang, SQL as SQLang.

 
Let us know what you find!