{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6cbb4ca0",
   "metadata": {},
   "source": [
    "## ENG EK381: Probability, Statistics, and Data Science for Engineers  <br> Lab 1: Getting Started with Python and Jupyter Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd0ae9d-be7e-4f26-a64b-40b22ae169ee",
   "metadata": {},
   "source": [
    "### This lab was completed by [YOUR NAME HERE].\n",
    "\n",
    "(double click on this \"cell\" (box) to edit the above, then delete this helper sentence)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "985d54ab",
   "metadata": {},
   "source": [
    "Welcome to EK381's new set of labs! The goal is to give you hands-on experience with **exploratory data analysis**, which is a useful skill set for many disciplines. Unlike in previous semesters, we are not asking you to write programs yourself and you will not be asked to turn in your code. Instead, the provided Jupyter notebook will demonstrate how to use a data analysis technique using a couple of lines of Python code. Your task will be to apply these basic functions to analyze and plot the data. There will be clearly marked places for you to perform data analysis, generate plots, and write comments. At the end, you will submit your completed notebook for grading. Exams (and possibly quizzes) will include questions meant to test your understanding of data analysis concepts, such as interpreting plots."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "417e5dc2",
   "metadata": {},
   "source": [
    "In this first lab, you have three objectives: \n",
    "1. Install/setup a version of Jupyter notebook that you will be comfortable using throughout the semester. (If you are reading this Jupyter notebook, this step should have already been completed.) You can always switch or upgrade to an another version later.\n",
    "2. Download and read in a comma-separated value (CSV) dataset.\n",
    "3. Run the fully-written data analysis below, answering the three questions (L1.1, L1.2, and L1.3), and upload the completed Jupyter notebook for grading."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b1b8be8",
   "metadata": {},
   "source": [
    "Each of the rectangles that you see below is called a *cell*. To run the code in a cell (and make the results available to other cells), you can press the \"play\" button (triangle) or press \"Shift + Enter.\" "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91fd137c-ed6c-4f35-8354-fff63c8ee289",
   "metadata": {},
   "source": [
    "If you are using JupyterLab Desktop, you will need to install seaborn. Uncomment and run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dd1a122-4e06-4c39-bf38-c52cfa3637e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# JupyterLab Desktop (not Anaconda) users, delete the # symbol from the last two lines of this cell\n",
    "# and then run the cell to install the seaborn and sklearn packages (which is pre-installed on Anaconda\n",
    "# and Google Colab).\n",
    "\n",
    "#!pip install seaborn\n",
    "#!pip install scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc462ef6",
   "metadata": {},
   "source": [
    "Run the cell below to import the libraries/packages you will need to complete this lab. (If you are not able to run it successfully, please get help from a GST during a discussion before the lab is due.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b24ea8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary packages.\n",
    "# This cell must be run in order for the packages to be available below.\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "import seaborn as sns\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebe0d9ba",
   "metadata": {},
   "source": [
    "Next, we are going to read in the CSV file that contains our dataset. If you are working in Google colab, uncomment the code in the cell below and run it. For any other environments, skip the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbbc8934",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Google colab users, delete the # symbols from the two lines below and then run this cell to mount your Google drive.\n",
    "\n",
    "#from google.colab import drive\n",
    "#drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a701dac1",
   "metadata": {},
   "source": [
    "Download the dataset `penguins.csv` from the Lab 1 folder on Blackboard Learn and move it to the directory you will be using for this lab. You might want to create a new subfolder `datasets` to contain all the datasets for this course. When you are asked to do something in a lab (that will be graded), it will be written as LX.Y where X is the lab number and Y is the problem number. Your first lab problem is below.\n",
    "\n",
    "---\n",
    "#### L1.1 Determine the path to `penguins.csv` and set the variable `penguins_path` below to be equal to the resulting string.\n",
    "---\n",
    "There are a few options for this, ask for help from a TA if you are stuck: \n",
    "- Same Folder: If you will keep your datasets in the same folder as your Jupyter notebook, then the path is simply the filename `penguins.csv`\n",
    "- Subfolder: If you will keep your datasets in a subfolder inside the folder containing your Jupyter notebook, then you use the local filepath. For example, if the subdirectory is called `datasets`, then the local path would be `datasets/penguins.csv`\n",
    "- Some other folder: In this case, the easiest thing to do is to use the full path by first navigating to the file and then\n",
    "    - Google colab: Click on the three vertical dots to the right of the file name and select `Copy Path`\n",
    "    - Mac: Right click on the file, then press and hold option. Select `Copy \"penguins.csv\" as Pathname`\n",
    "    - Windows: Hold shift, right click on the file, and select `Copy as Path`\n",
    "      \n",
    "Once you have filled in `penguins_path`, run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03f073c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_path = \"\" # Put your copied filepath to penguins.csv inside the quotes to the left.\n",
    "penguins = pd.read_csv(penguins_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e673097",
   "metadata": {},
   "source": [
    "Let's take a quick look at the dataset, which we have loaded as a pandas dataframe. If you simply write the name of the dataframe, you will see a snapshot of the data. Run the cell below to do this. You will see that this is a dataset of 344 penguins, including their species, bill length, bill depth, and mass. Note also that, in Python, indexing starts at 0, not 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3b17d6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b42047-f351-4494-851c-73bb44554528",
   "metadata": {},
   "source": [
    "What if we want to see all of the rows?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62667497-fea7-4016-ac39-0e4aa6bddbe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", None)\n",
    "penguins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a65814",
   "metadata": {},
   "source": [
    "In previous semesters, our focus was on writing algorithms to implement some of the concepts we learned in class. For example, an important quantity is the average or mean of a variable. We used to write a for loop to sum up all the values, and then divide it by the total number of values. This semester, we will just use built-in functions to quickly compute many statistics, including the mean, standard deviation, minimum, and maximum. Run the cell below to see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa625315",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e153501",
   "metadata": {},
   "source": [
    "---\n",
    "#### L1.2 What is the body mass (in grams) of the lightest penguin in the dataset?\n",
    "\n",
    "#### Answer:\n",
    "---\n",
    "You can write comments directly in the Jupyter notebook using \"Markdown\" cells, like the one below. (You change the cell type from Code to Markdown and back using the menu or keyboard shortcuts.) Running a Markdown cell will display the text."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dccd9b3c",
   "metadata": {},
   "source": [
    "An easy way to visualize all of the values of a single variable is to use a histogram. Run the cell below to see a histogram for the bill length, with the y-axis representing the number of times each range (or bin) of values appears in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8a450fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.histplot(data=penguins, x=\"bill_length_mm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c648f15",
   "metadata": {},
   "source": [
    "In EK381, we will be primarily interested in probability distributions, rather than raw counts. If we divide the counts by the total number of entries in the dataset, we get normalized counts. These normalized counts (or frequencies) can be directly used as an estimate of the \"true\" probability distribution. Alternatively, we can fit a distribution to our data. We will explore both ideas throughout the semester. Run the cell below to see an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60652009",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.histplot(data=penguins, x=\"bill_length_mm\", stat = \"probability\", kde = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c81ad498",
   "metadata": {},
   "source": [
    "Does the distribution of bill lengths change based on the penguin species? We can visualize this easily by plotting a different histogram for each species as done below. Starting in Lecture 2, we will understand this as a form of \"conditioning.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bb81c59",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.histplot(data=penguins, x=\"bill_length_mm\", stat = \"probability\", hue = \"species\", kde = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f8a6997",
   "metadata": {},
   "source": [
    "We can also look at two variables at once, which will be important starting in Chapter 4. The scatter plot below places a dot at coordinates (x,y) for each penguin where x corresponds to the bill length, y to the bill depth, and the color to the species. We have also included the fitted distributions for the individual variables, but you will more often just see a simple scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf3c4571",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.jointplot(data=penguins, x=\"bill_length_mm\", y=\"bill_depth_mm\", hue=\"species\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d46b9e",
   "metadata": {},
   "source": [
    "Above, we calculated some interesting statistics, such as the mean and standard deviation. We can visualize these using bar plots, with the height corresponding to the mean and the \"error bar\" (black line) corresponding to the standard deviation. Again, we will encounter these ideas later in the semester."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd9e9046",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Draw a nested barplot by species and sex\n",
    "g = sns.catplot(\n",
    "    data=penguins, kind=\"bar\",\n",
    "    x=\"species\", y=\"body_mass_g\", hue=\"sex\", errorbar=\"sd\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebc34509",
   "metadata": {},
   "source": [
    "Everything we have done so far is descriptive. What about using some values to predict others? Below, we fit a line to predict the bill depth from the bill length. We will explore estimation and prediction more starting in Chapter 7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad2f399e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot bill depth as a function of bill length for each species\n",
    "g = sns.lmplot(data=penguins,\n",
    "    x=\"bill_length_mm\", y=\"bill_depth_mm\", hue=\"species\")\n",
    "\n",
    "# Use more informative axis labels than are provided by default\n",
    "g.set_axis_labels(\"Bill length (mm)\", \"Bill depth (mm)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2fb6db1-7ab1-49c7-ba77-dfa1c79f15b3",
   "metadata": {},
   "source": [
    "Another interesting possibility is to use the feature data for classification. Specifically, how well can we classify penguin species using only the bill depth and bill length? The code below displays one possible set of classification regions that work correctly for most of the penguins. Penguins that are misclassified by these regions are circled in red. The code also computes and outputs the percentage of misclassified penguins. We learn more about these machine learning techniques in Chapter 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d442ba8d-6409-400f-b313-4d7a345712e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_data = penguins.dropna()  # Drop rows with missing values\n",
    "\n",
    "# Extract features and target\n",
    "X = penguins_data[['bill_length_mm', 'bill_depth_mm']].values\n",
    "species_names = penguins_data['species'].astype('category').cat.categories\n",
    "y = penguins_data['species'].astype('category').cat.codes\n",
    "\n",
    "# Fit LDA model\n",
    "lda = LinearDiscriminantAnalysis()\n",
    "lda.fit(X, y)\n",
    "\n",
    "# Calculate training error\n",
    "predictions = lda.predict(X)\n",
    "train_error = 1 - accuracy_score(y, predictions)\n",
    "\n",
    "# Visualization of classification regions\n",
    "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1), np.arange(y_min, y_max, 0.1))\n",
    "\n",
    "Z = lda.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "Z = Z.reshape(xx.shape)\n",
    "\n",
    "# Plot decision boundaries\n",
    "custom_palette = ['#1f77b4', '#2ca02c','#ff7f0e']  # Blue, Orange, Green\n",
    "colormap = ListedColormap(custom_palette)\n",
    "\n",
    "plt.figure(figsize=(8, 8))\n",
    "plt.contourf(xx, yy, Z, alpha=0.2, cmap=colormap)\n",
    "\n",
    "# Plot training points\n",
    "scatter = plt.scatter(X[:, 0], X[:, 1], c=y, cmap=colormap, s=50)\n",
    "\n",
    "# Highlight misclassified points\n",
    "misclassified = X[predictions != y]\n",
    "plt.scatter(misclassified[:, 0], misclassified[:, 1], facecolors='none', edgecolors='red', s=100, linewidths=1.5, label='Misclassified')\n",
    "\n",
    "plt.legend(handles=[plt.Line2D([0], [0], marker='o', color='w', label=name, \n",
    "                                markerfacecolor=color, markersize=10) \n",
    "                     for name, color in zip(species_names, custom_palette)], title=\"Species\")\n",
    "\n",
    "plt.title(\"LDA Classification of Penguin Species\")\n",
    "plt.xlabel(\"Bill Length (mm)\")\n",
    "plt.ylabel(\"Bill Depth (mm)\")\n",
    "plt.show()\n",
    "\n",
    "print(f\"Percentage of data misclassified by these decision boundaries: {train_error:.2%}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f412af1a",
   "metadata": {},
   "source": [
    "Finally, another important issue we will examine throughout the labs are the limitations of statistics. For example, while a linear fit seems like a good estimator above, it is not always. The dataset below is known as Anscombe's quartet. Each set of 11 points generates the same linear fit. This seems like a good fit in dataset I where as a curve would do better in dataset II. A line would be a good fit in datasets III and IV, but a single \"outlier\" has tricked our simple line fitting algorithm. \n",
    "\n",
    "**Visualizing data is an important way to avoid these issues, and one of our goals throughout the semester is to provide you with tools and intuition to avoid statistical fallacies.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a24149a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set_theme(style=\"ticks\")\n",
    "\n",
    "# Load the example dataset for Anscombe's quartet\n",
    "df = sns.load_dataset(\"anscombe\")\n",
    "\n",
    "# Show the results of a linear regression within each dataset\n",
    "sns.lmplot(\n",
    "    data=df, x=\"x\", y=\"y\", col=\"dataset\", hue=\"dataset\",\n",
    "    col_wrap=2, palette=\"muted\", ci=None,\n",
    "    height=4, scatter_kws={\"s\": 50, \"alpha\": 1}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "417564a1-a0ac-49d5-83ea-6a7c37194a87",
   "metadata": {},
   "source": [
    "---\n",
    "#### L1.3 Please describe, in your own words, an engineering application from your major where some of the techniques showcased above might be useful. Briefly explain why.\n",
    "\n",
    "#### Answer:\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bbacbcd",
   "metadata": {},
   "source": [
    "That's it for the first lab. After you have successfully filled in and run all of the cells above, please upload your completed Jupyter notebook (ek381lab1.ipynb) to Gradescope for grading.\n",
    "\n",
    "There are many, many resources online for learning Python, Jupyter notebook, and exploratory data analysis. You can find a few options on the course website, please let us know if you find other particularly good resources that we failed to mention. The sample plots generated above were taken in part from the seaborn tutorial here https://seaborn.pydata.org/index.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "129a8260-639e-4fdf-9509-5f6b4524340f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}