For my project I decided to investigate the baseball data. The question I asked was, "How are the all-stars different from the entire player base?" I looked at two characteristics to investigate this: birth country (or state for American born players) and handedness for batting and throwing.
The data for this project was taken from SeanLahman.com. The CSVs investigated were AllstarFull.csv, Master.csv and Appearances.csv. I loaded the CSVs into dataframes using Pandas and printed their heads to get an idea of their structure.
import pandas as pd
# read CSVs and print their heads
allstar_df = pd.read_csv('AllstarFull.csv')
master_df = pd.read_csv('Master.csv')
appearances_df = pd.read_csv('Appearances.csv')
allstar_df.head()
master_df.head()
appearances_df.head()
I started by looking for missing values starting with the all-stars dataframe.
# list number of missing values in allstar_df
allstar_df.isnull().sum()
The gameID, GP and startingPos columns had missing values. The documentation included with this data said that startingPos would only have a value if the player was a starter, so it makes sense for there to be so many missing values. There was nothing explaining the other two columns, so I investigated them further.
# find entries with missing gameID
allstar_df.loc[pd.isnull(allstar_df['gameID'])]
All these entries were for 1945 all-star game. This game was cancelled due to World War II, which explains the missing values. I decided to keep these entries as these players were still chosen to be all-stars even if they did not play the game.
# find entries with missing GP
allstar_df.loc[pd.isnull(allstar_df['GP'])]
These entries corresponded to the 2015 all-star game. After doing a little research it seemed like these entries corresponded to players who were selected as all-stars, but did not appear in the game for some reason (such as injuries). I was unable to find a reason for every player (nor was every player who was replaced due to injury on the above list), so I was not completely confident in this explanation.
It was also noteworthy that these entries were the last 18 entries in the dataframe although I could not find a reason for the missing values from that angle.
I decided to keep these entries as all the players are listed on the roster,.
I moved on to the master dataframe.
# list number of missing values in master_df
master_df.isnull().sum()
Only playerID did not have missing values. Since this data goes back to 1820 the missing data was likely due to the information simply being unavailable. Furthermore, there would be no death time or place information for players who were still alive, which explains why there are significantly more missing values in those columns than the others. There was not much I could do to remedy the missing data, so I decided to continue using it as is.
The variables I were investigating had a relatively small amount of missing data points, but they would be affected by the missing data nonetheless. The most significant was the batting handedness where about 6% of the data was missing. It could also be possible for the missing birth country data to significantly affect the results if those players came from countries with low player contributions.
Finally, I looked for missing data in the appearances dataframe.
# list number of missing values in appearances_df
appearances_df.isnull().sum()
The columns lgID (league), G_all (total games played), GS (games started), G_defense (games in which the player appeared on defense), G_dh (games as designated pitcher), G_ph (games as pinch hitter) and G_pr (games as pinch runner) had missing data. Since G_defense, G_dh, G_ph and G_pr referred to positions, it made sense for there to be missing values as not every player would play in those positions in every season. Furthermore, GS would have missing data if a player did not start that season. However, lgID and G_all did not have a clear reason for missing data and needed to be investigated further. I started with with the league information.
# find entries with missing lgID
appearances_df.loc[pd.isnull(appearances_df['lgID'])].head()
appearances_df.loc[pd.isnull(appearances_df['lgID'])].tail()
These entries were from the years 1871 to 1875. Since the American League (the older of the two leagues that comprises the MLB) was founded in 1876, any entries prior to 1876 would not have a league ID. I looked at the games played information next.
# find entries with missing G_all
appearances_df.loc[pd.isnull(appearances_df['G_all'])].head()
All of these entries were from the 2013 season, but I could not find a reason why this data was missing. I was using the appearances data to help collect information on how handedness changed over time, so none of this missing data affected by analysis. I decided to ignore it.
I then cleaned up the inconsistencies in the country names in the master dataframe.
# make all country names consistent
def country_corrector(country):
if country == 'CAN':
return 'Canada'
elif country == 'D.R.':
return 'Dominican Republic'
elif country == 'P.R.':
return 'Puerto Rico'
elif country == 'V.I.':
return 'US Virgin Islands'
return country
master_df['birthCountry'] = master_df['birthCountry'].apply(country_corrector)
The all-stars dataframe contained an entry for every appearance a player made in an all-star game. This means the same player will have multiple entries if he played in more than one all-star game in his career. I used the all-stars dataframe to create an array of all-star players and then used this list to extract the all-stars from the master dataframe.
# all-stars array
unique_allstars = allstar_df['playerID'].unique()
# all-star players
allstars = master_df.loc[master_df['playerID'].isin(unique_allstars)]
I noticed a discrepancy in this data. The unique_allstars array had 1741 elements. However, the allstars dataframe had 1740 rows. I found the data point in unique_allstars, but not in allstars.
# discrepancy
print len(unique_allstars) # 1741
print allstars.shape[0] # 1740
# find discrepancy
for player in unique_allstars:
if player not in allstars['playerID'].tolist():
print player # carperma01
# print discrepancy
allstar_df.loc[allstar_df['playerID'] == 'carperma01']
It was not immediately clear to me what was wrong with this data point. After a bit of research I found the answer. All the player IDs were 9 characters long and 'carperma01' contained 10 characters. This was due to a typo as 'carperma01' should have been 'carpema01'. This data point refered to Matt Carpenter's second all-star selection. I corrected this discrepancy and recreated unique_allstars and allstars.
# correct discrepancy
allstar_df.set_value(4965, 'playerID', 'carpema01')
#recreate array and dataframe
unique_allstars = allstar_df['playerID'].unique()
allstars = master_df.loc[master_df['playerID'].isin(unique_allstars)]
# double check that values match
print len(unique_allstars) # 1740
print allstars.shape[0] # 1740
With the discrepancy cleared I organized the player data. Since America had a much larger number of players than any other country they would have dominated most measurements. Therefore, I decided to look at them separately. I wanted to create a dataframe containing all-star and player populations for American and non-American players.
I separated the all-stars and players (from the master dataframe) into American and non-American born players. I grouped each of the resultant dataframes by birth state/country and counted to get the all-star and player populations for each state/country. After renaming some columns for clarity, I concatenated these columns to get my desired dataframes. I also computed the percentage of all-stars in each dataframe for future analysis.
# divide all-stars and players into American and non-American
american_allstars = allstars.loc[allstars['birthCountry'] == 'USA']
non_american_allstars = allstars.loc[allstars['birthCountry'] != 'USA']
american_players = master_df.loc[master_df['birthCountry'] == 'USA']
non_american_players = master_df.loc[master_df['birthCountry'] != 'USA']
# group by state/country
american_allstars_by_state = american_allstars.groupby('birthState').count()
non_american_allstars_by_country = non_american_allstars.groupby('birthCountry').count()
american_players_by_state = american_players.groupby('birthState').count()
non_american_players_by_country = non_american_players.groupby('birthCountry').count()
# clean up column names to prepare for concatenation
american_allstars_by_state['allStars'] = american_allstars_by_state['playerID']
non_american_allstars_by_country['allStars'] = non_american_allstars_by_country['playerID']
american_players_by_state['players'] = american_players_by_state['playerID']
non_american_players_by_country['players'] = non_american_players_by_country['playerID']
# concatenate
american_players_by_state = pd.concat(
[american_players_by_state['players'], american_allstars_by_state['allStars']], axis=1)
non_american_players_by_country = pd.concat(
[non_american_players_by_country['players'], non_american_allstars_by_country['allStars']], axis=1)
# replace NaN with 0
american_players_by_state.fillna(0, inplace=True)
non_american_players_by_country.fillna(0, inplace=True)
# add percentAllStars column
american_players_by_state['percentAllStar'] = (american_players_by_state['allStars']
/ american_players_by_state['players']) * 100.00
non_american_players_by_country['percentAllStar'] = (non_american_players_by_country['allStars']
/ non_american_players_by_country['players']) * 100.00
# print final dataframes
american_players_by_state.head()
non_american_players_by_country.head()
The next step was to organize the data for handedness. I started by adding the handedness data from the master dataframe to the all-stars and appearances dataframes. I limited the player database to the same years as the all-stars.
# handedness of all players
handedness = master_df[['playerID', 'bats', 'throws']]
# add handedness to all-stars
allstar_handedness = allstar_df.merge(handedness, on='playerID', how='left')[['playerID', 'yearID', 'bats', 'throws']]
# print result
allstar_handedness.head()
# add handedness to player appearances
player_handedness = appearances_df.merge(handedness, on='playerID', how='left')[['playerID', 'yearID', 'bats', 'throws']]
player_handedness = player_handedness.loc[player_handedness['yearID'] >= 1933]
# print result
player_handedness.head()
I grouped the all-stars and players by year and used the value_counts method (as a proportion) to create four series containing different types of handedness: all-star batting, all-star throwing, player batting and player throwing.
# all-star batting
allstar_bat_handedness_by_year = allstar_handedness.groupby('yearID')['bats'].value_counts(normalize=True)
allstar_bat_handedness_by_year = allstar_bat_handedness_by_year.fillna(0)
# all-star throwing
allstar_throw_handedness_by_year = allstar_handedness.groupby('yearID')['throws'].value_counts(normalize=True)
allstar_throw_handedness_by_year = allstar_throw_handedness_by_year.fillna(0)
# player batting
player_bat_handedness_by_year = player_handedness.groupby('yearID')['bats'].value_counts(normalize=True)
player_bat_handedness_by_year = player_bat_handedness_by_year.fillna(0)
# player throwing
player_throw_handedness_by_year = player_handedness.groupby('yearID')['throws'].value_counts(normalize=True)
player_throw_handedness_by_year = player_throw_handedness_by_year.fillna(0)
I started by looking at the birth states/countries. I began by plotting the raw data. I decided to use bar graphs, because I was comparing the different amounts of all-stars and the difference in height would easily provide a visual indicator for these differences.
%pylab inline
import matplotlib.pyplot as plt
import seaborn as sns
master_df['birthCountry'].value_counts().plot(kind='bar', figsize=[15,14])
plt.title('Raw Birth Country Statistics')
plt.xlabel('Country')
plt.ylabel('Number of Players')
USA completely dominated every other country, which justified my choice in looking at it separately. I plotted the raw birth state data next.
master_df['birthState'].value_counts().plot(kind='barh', figsize=[15,45])
plt.title('Raw Birth State Statistics')
plt.xlabel('Number of Players')
plt.ylabel('State')
Again, American states dominated the data. Furthermore, the sheer amount of states in the master dataframe also justified my choice to group them by the more general category of countries. In particular, most countries contributed players from only a few states within them, so it made sense not to look at them separately. Although the argument could be made to give the other countries with relatively large amounts of player contribution (such as Canada and the Dominican Republic) the same treatment as USA.
I moved on to the American all-stars and their state of birth.
american_players_by_state.describe()
From the description of the data there was an average of about 323 players per state. About 29 or 9.3% of those players were all-stars. There was a large difference between the minimum and maximum values and the 75th percentile was very small compared to the maximum.
american_players_by_state['allStars'].plot(kind='barh', figsize=[15,14])
plt.title('American All-Stars by Birth State', fontsize=18)
plt.xlabel('Number of All-Stars', fontsize=14)
plt.ylabel('Birth State', fontsize=14)
What stuck out (literally) was that California had many more all-stars than the other states (about three times more than the state with the second most all-stars), which explained the statistics noted above. This could be attributed to their large population, so I also plotted the percentage of all-stars as well to reduce the effect of a large population.
american_players_by_state['percentAllStar'].plot(kind='barh', figsize=[15,14])
plt.title('American All-Stars by Birth State - Percentage', fontsize=16)
plt.xlabel('Percent of Player Base that are All-Stars', fontsize=14)
plt.ylabel('Birth State', fontsize=14)
This plot looked signficantly different from the previous one. California no longer had the largest value and in general the states' values were a lot closer together. The top three states were Idaho, Wyoming and Alaska. Maine had a low value in both plots.
While looking at the percentage did prevent large populations from dominating the plot, it also had the effect of overvaluing all-stars in states with smaller populations. A state with a small amount of players (such as Idaho, Wyoming and Alaska) only needed a small amount of all-stars to achieve a large percentage. However, there was no way of knowing if these all-stars were outliers and not indicative of the general player base. Therefore, in order to mitigate this effect I removed all states with less than 25 all-stars and plotted the percentages again.
to_plot = american_players_by_state.loc[american_allstars_by_state['allStars'] >= 25]
to_plot['percentAllStar'].plot(kind='barh', figsize=[15,14])
plt.title('American All-Stars by Birth State - Percentage', fontsize=16)
plt.xlabel('Percent of Player Base that are All-Stars', fontsize=14)
plt.ylabel('Birth State', fontsize=14)
Only 21 states remained after imposing this condition. And the percentages of these remaing states were relatively close together compared to the previous plots. The state with the largest percentage here was Arkansas even though it had a relatively low amount of all-stars. This implied that Arkansas born players are disproportionately more likely to become all-stars. New York was the state with the third most all-stars, but it had a relatively low percentage. This implied that the New York born players are disproportionately less likely to become all-stars. The same could also be said for Pennsylvania. However, the relative closeness of the percentages means that birth states did not drastically affect a player's chance at being an all-star. That is to say, California contributes more all-stars due to their larger population and not due to Californians being inherently good at baseball.
Finally, I plotted the number of players per birth state to look at this from another angle.
american_players_by_state['players'].plot(kind='barh', figsize=[15,14])
plt.title('American Players by Birth State', fontsize=18)
plt.xlabel('Number of Players', fontsize=14)
plt.ylabel('Birth State', fontsize=14)
Once again, California dominates the plot. Overall, the shape of this plot mimicked the shape of the all-stars plot. States with large amounts of all-stars also had large amounts of players and vice versa. The differences between the proportions were more easily seen in the percentages plot, but in general the all-star plot was a microcosm of the players plot. This reinforced my inital conclusion that birth state does not dratstically affect all-star likelihood.
I moved on to the rest of the countries next starting with a description of the data.
non_american_players_by_country.describe()
The other countries contributed an average of about 44 players with an average of 5 all-stars. However, the average percentage of all-stars contributed was 7.5% (although 5/44 is about 11%). This implied a heavily skewed distribution. This is further supported by the 75th percentile for the players and all-stars being extremely low compared to the maximum value.
It would turn out that some countries contributed one player who would also go on to become an all-star leading to a 100% all-star rate.
I plotted the all-stars for each country (for countries with at least one all-star).
to_plot = non_american_players_by_country.loc[non_american_players_by_country['allStars'] > 0]
to_plot['allStars'].plot(kind='barh', figsize=[15,14])
plt.title('Non-American All-Stars by Birth Country', fontsize=18)
plt.xlabel('Number of All-Stars', fontsize=14)
plt.ylabel('Birth Country', fontsize=14)
Several countried clearly dominated this plot. Again, I decided to look at percentages. This time I limited the results to countries with at least 10 all-stars (many countries had percentages over 50 without this limitation).
to_plot = non_american_players_by_country.loc[non_american_players_by_country['allStars'] > 10]
to_plot['percentAllStar'].plot(kind='barh', figsize=[15,14])
plt.title('Non-American All-Stars by Birth Country - Percentage', fontsize=18)
plt.xlabel('Percentage of Player Base that are All-Stars', fontsize=14)
plt.ylabel('Birth Country', fontsize=14)
Many of the countries were eliminated using this criteria and the countries that remained had percentages close to each other despite contributing largey disparate amounts of all-stars. This could be attributed to the data coming from American baseball leagues and so it made sense that foreigners would only compete if they had exceptional skill. A number of countries that contributed a very small number of players (such as two to five) would often have most of those players become all-stars.
Finally, I plotted the players per country.
non_american_players_by_country['players'].plot(kind='barh', figsize=[15,14])
plt.title('Non-American Players by Birth Country', fontsize=18)
plt.xlabel('Number of Players', fontsize=14)
plt.ylabel('Birth Country', fontsize=14)
This plot was consistent with the all-stars plot moreso than the American data. The countries with high amounts of all-stars also had high amounts of players and the countries that contributed few or no all-stars also contributed few players.
I looked at the handedness data next. I started by plotting the raw handedness data from the master database.
master_df['bats'].value_counts().plot(kind='bar')
plt.title('Raw Batting Statistics')
plt.xlabel('Handedness')
plt.ylabel('Number of Players')
master_df['throws'].value_counts().plot(kind='bar')
plt.title('Raw Throwing Statistics')
plt.xlabel('Handedness')
plt.ylabel('Number of Players')
The main distinction between the plots was that players were ambidextrous in batting, but not in throwing. This suggested that players were more versatile at batting. There were more right-handed than left-handed players as well, which was expected.
I moved on to the wrangled data to see how the handedness changed over time.
allstar_bat_handedness_by_year.unstack(level=1).describe()
The all-star batting handedness could be broken down to about 63% right-handed, 30% left-handed and 7% ambidextrous. The values had a relatively large range. There was a little as 52% right-handed all-stars at one point, which was surprising given the raw data above.
I plotted the batting data for all-stars and players. Since I was dealing with a change over time I decided to use a line graph.
allstar_bat_handedness_by_year.unstack(level=1).plot(figsize=[10,8])
plt.title('All-Star Batting Handedness')
plt.xlabel('Year')
plt.ylabel('Percentage of Players')
player_bat_handedness_by_year.unstack(level=1).describe()
The averages for the player base were similar to the all-stars, but it seemed the data did not vary as much.
player_bat_handedness_by_year.unstack(level=1).plot(figsize=[10,8])
plt.title('Player Batting Handedness')
plt.xlabel('Year')
plt.ylabel('Percentage of Players')
What stood out to me was that the all-star plot was much more sporadic than the general player base, which supported the conjectures made above. I suspected that the general player base handedness would be steady due to the population not changing much from year to year. However, the sporadic nature of the all-stars implied that this was not the case for them. This could mean it was hard for someone to beomce an all-star multiple years in a row. There was also a slight increase in the ambidextrous players as time went on, which could mean the player base was becoming more versatile over time.
allstar_throw_handedness_by_year.unstack(level=1).describe()
About 20% of players were left-handed and 80% were right-handed when it came to throwing. These numbers varied by about 20% over the years, which further indicated the all-star handedness was more sporadic than the general player base.
allstar_throw_handedness_by_year.unstack(level=1).plot(figsize=[10,8])
plt.title('All-Star Throwing Handedness')
plt.xlabel('Year')
plt.ylabel('Percentage of Players')
player_throw_handedness_by_year.unstack(level=1).describe()
The averages for the player base were similar to the all-stars. However, the varied by half as much as the all-stars, which further supported the trend of more stable statistics for the players compared to the all-stars.
player_throw_handedness_by_year.unstack(level=1).plot(figsize=[10,8])
plt.title('Player Throwing Handedness')
plt.xlabel('Year')
plt.ylabel('Percentage of Players')
Again, the sporadic nature of the all-stars relative to the player base further supported that it was hard for all-stars to have repeat showings. Furthermore, the difference in proportions was much greater for throwing than for batting. This was more proof that the players were better at batting than throwing. Overall, the plots reflected the conjectures made on the data's descriptions.
It is important to note that any relations implied by my analysis are not necessarily causal. There are many factors that could affect the results presented below. This project was focused on comparing all-stars to the general player base. However, all-stars are chosen by people whether it be the players, managers or fans rather than a competition or a player's stats. This means any trends in the all-stars' performance could be coincedental. For example, a popular player with average skill might have a better chance of becoming an all-star than a skilled, but unpopular player. Furthermore, being chosen to be an all-star once could affect your future performance. For example, maybe a player picked to be an all-star is more motivated and as a result performs better. Nevertheless, these are the conclusions I arrived to.
I would like to thank Caroline Buckey for creating the course on the data analyst process as well as numpy and Pandas, Mat Leonard for creating the course on Anaconda and Jupyter Notebook and Sean Lahman for providing the data.