Five classifiers on the same three datasets make the trade-offs visible: trees are readable, Naive Bayes is fast and needs little data, k-NN needs normalised attributes, SVM handles many attributes.
Objectives
Do not copy. Read for understanding and the viva- Complete questions 15 to 16 of the manual: classification: logistic regression, decision tree, naive bayes, k-nn, svm
- Prepare the deliverable before the lab and finish it during the session
- Be ready to explain every step in the viva
Questions Covered
Do not copy. Read for understanding and the viva| Question | Requirement | Status |
|---|---|---|
| Q15 | Demonstrate the classification rule process on the student.arff, employee.arff and… | Complete |
| Q16 | Demonstrate the classification rule process on the student.arff, employee.arff and… | Complete |
Preparation
Do not copy. Read for understanding and the viva- WEKA names:
functions.Logistic,trees.J48,bayes.NaiveBayes,lazy.IBk(k-NN, set K),functions.SMO(SVM). - Use 10-fold cross-validation for every run so results are comparable; note accuracy, kappa, and the confusion matrix for each.
- Build student.arff and employee.arff with a nominal class attribute and at least 30 rows each, or the classifiers will not have enough to learn.
Question 15
Problem Statement
Write in lab recordDemonstrate the classification rule process on the student.arff, employee.arff and labor.arff datasets using the following algorithms:
- Logistic Regression
- Decision Tree
- Naive Bayes
Solution
Write in lab recordSteps
Every number on this page and in Sessions 8 to 10 comes from the two 30-row data sets defined here (public/code/mcsl-223/section-2/session-7/): three numeric attributes, one nominal attribute and a two-valued class each, with no missing values, so the Python check script can reproduce WEKA’s evaluation. The Session 1 files (session-1/student.arff, session-1/employee.arff) have a richer layout with an identifier column, more attributes and a few ? cells; the WEKA procedure is identical on them, but first remove sid or eid on the Preprocess tab (an identifier makes every classifier overfit) and expect different numbers. labor.arff ships with WEKA (57 instances, 16 attributes plus the class class with values bad and good, many ? missing values).
- Explorer, Preprocess, Open file…,
student.arff. Check the status line: 30 instances, 5 attributes, and clickresultin the attribute list to confirm the class distribution (19 pass, 11 fail). Repeat later foremployee.arff(16 yes, 14 no) andlabor.arff(37 good, 20 bad). - Classify tab. Under Test options select Cross-validation, Folds 10. Leave the class chooser at the last attribute.
- Choose,
functions.Logistic(defaults:ridge1.0E-8,maxIts-1). Start. Record from the output: the coefficient table under “Classifier model”, then under “Stratified cross-validation” the Correctly Classified Instances line, the Kappa statistic and the Confusion Matrix. - Choose,
trees.J48(defaults:confidenceFactor0.25,minNumObj2). Start. Right-click the entry in the Result list and pick Visualize tree to see the tree drawn; the same tree is printed as text with leaf counts such aspass (16.0)orpass (4.0/1.0)(instances reaching the leaf, and how many of them are misclassified). - Choose,
bayes.NaiveBayes(defaults). Start. The model block lists, per class, the mean and standard deviation of every numeric attribute and the counts of every nominal value. - Repeat steps 3 to 5 for
employee.arffandlabor.arff; nine runs in all. Keep the Result list entries: the next question adds two more classifiers to the same list for a side-by-side comparison. - To check the numbers without WEKA run
python3 classify_cv.py student.arff employee.arff. The script reimplements the five classifiers of this session and the next in about 250 lines of standard-library Python and prints the same summary and confusion matrix layout as WEKA.
Program
Lab record: every tab is one file of the answer. Write all of them.
% student.arff : 30 students, class = result
% pass needs internal >= 12 and attendance >= 60 (one noisy row on purpose)
@relation student
@attribute hours numeric
@attribute attendance numeric
@attribute internal numeric
@attribute stream {science,commerce,arts}
@attribute result {pass,fail}
@data
12,85,24,science,pass
15,92,27,science,pass
8,70,15,commerce,pass
10,78,19,arts,pass
18,95,29,science,pass
6,65,13,commerce,pass
14,88,22,arts,pass
9,72,16,science,pass
11,80,20,commerce,pass
16,90,26,arts,pass
7,68,14,science,pass
13,84,23,commerce,pass
10,75,18,arts,pass
17,93,28,science,pass
9,74,17,commerce,pass
12,82,21,arts,pass
8,66,12,science,pass
15,89,25,commerce,pass
11,77,19,arts,pass
3,45,6,science,fail
4,50,8,commerce,fail
2,40,5,arts,fail
5,55,9,science,fail
6,58,11,commerce,fail
3,48,7,arts,fail
4,52,10,science,fail
7,60,11,commerce,fail
10,70,14,arts,fail
12,58,20,science,fail
6,62,10,commerce,fail% employee.arff : 30 employees, class = promoted
% promotion follows experience and salary; two noisy rows and two outliers on purpose
@relation employee
@attribute age numeric
@attribute experience numeric
@attribute department {sales,hr,it,finance}
@attribute salary numeric
@attribute promoted {yes,no}
@data
24,1,sales,28,no
25,2,it,35,no
26,3,hr,30,no
27,2,finance,33,no
28,4,sales,38,no
29,5,it,45,no
30,6,hr,42,yes
31,7,finance,55,yes
32,8,sales,52,yes
33,9,it,68,yes
34,10,hr,58,yes
35,11,finance,72,yes
36,12,sales,65,yes
37,13,it,85,yes
38,14,hr,70,yes
39,15,finance,90,yes
40,16,sales,78,yes
42,18,it,105,yes
44,20,hr,88,yes
46,22,finance,120,yes
23,0,it,25,no
25,1,hr,27,no
27,3,sales,36,no
29,4,finance,40,no
30,5,it,48,no
31,6,sales,44,no
34,9,hr,60,no
36,12,it,80,yes
58,35,finance,250,yes
22,0,sales,15,no#!/usr/bin/env python3
"""10-fold stratified cross-validation of five classifiers, standard library only.
NaiveBayes, IBk (k-NN, k = 1), an unpruned information-gain decision tree (ID3 with
J48-style binary splits on numeric attributes), Logistic regression (gradient descent)
and a linear SVM (Pegasos). For each one it prints accuracy, kappa, ROC area and the
confusion matrix in the layout WEKA uses, so the numbers can be checked against WEKA.
Run: python3 classify_cv.py student.arff employee.arff
"""
import math
import random
import sys
from collections import Counter
def load_arff(path):
attrs, rows, data = [], [], False
for line in open(path):
line = line.strip()
if not line or line.startswith('%') or line.lower().startswith('@relation'):
continue
if line.lower().startswith('@attribute'):
_, name, typ = line.split(None, 2)
typ = typ.strip()
nominal = None if typ.lower() in ('numeric', 'real', 'integer') else [v.strip() for v in typ.strip('{}').split(',')]
attrs.append((name, nominal))
elif line.lower() == '@data':
data = True
elif data:
vals = [v.strip() for v in line.split(',')]
rows.append([float(v) if a[1] is None else v for v, a in zip(vals, attrs)])
return attrs, rows
def folds(rows, k=10, seed=1):
"""Stratified folds: shuffle, then a stable sort by class stripes each class across folds."""
idx = list(range(len(rows)))
random.Random(seed).shuffle(idx)
idx.sort(key=lambda i: rows[i][-1])
return [idx[f::k] for f in range(k)]
def ranges(attrs, rows):
lo = {j: min(r[j] for r in rows) for j, a in enumerate(attrs[:-1]) if a[1] is None}
hi = {j: max(r[j] for r in rows) for j, a in enumerate(attrs[:-1]) if a[1] is None}
return lo, hi
def encoder(attrs, train):
"""Numeric attributes scaled to [0,1], nominal ones one-hot, plus a bias input."""
lo, hi = ranges(attrs, train)
def vec(row):
v = [1.0]
for j, (name, vals) in enumerate(attrs[:-1]):
if vals is None:
v.append((row[j] - lo[j]) / ((hi[j] - lo[j]) or 1))
else:
v.extend(1.0 if row[j] == x else 0.0 for x in vals)
return v
return vec
def entropy(rows):
n = len(rows)
return -sum(c / n * math.log2(c / n) for c in Counter(r[-1] for r in rows).values())
# ---- classifiers: each returns prob(row) -> {class: probability} ----
def naive_bayes(attrs, train):
classes = attrs[-1][1]
prior = Counter(r[-1] for r in train)
stats = {}
for j, (name, vals) in enumerate(attrs[:-1]):
for c in classes:
col = [r[j] for r in train if r[-1] == c]
if vals is None:
m = sum(col) / len(col) if col else 0.0
sd = math.sqrt(sum((v - m) ** 2 for v in col) / max(len(col) - 1, 1)) if col else 1.0
stats[j, c] = (m, sd or 1e-3)
else:
stats[j, c] = Counter(col)
def prob(row):
lp = {}
for c in classes:
s = math.log((prior[c] + 1) / (len(train) + len(classes)))
for j, (name, vals) in enumerate(attrs[:-1]):
if vals is None:
m, sd = stats[j, c]
s += -0.5 * ((row[j] - m) / sd) ** 2 - math.log(sd * math.sqrt(2 * math.pi))
else:
s += math.log((stats[j, c][row[j]] + 1) / (prior[c] + len(vals))) # Laplace
lp[c] = s
mx = max(lp.values())
z = sum(math.exp(v - mx) for v in lp.values())
return {c: math.exp(v - mx) / z for c, v in lp.items()}
return prob
def knn(attrs, train, k=1):
classes = attrs[-1][1]
lo, hi = ranges(attrs, train)
def dist(a, b):
d = 0.0
for j, (name, vals) in enumerate(attrs[:-1]):
if vals is None:
d += ((a[j] - b[j]) / ((hi[j] - lo[j]) or 1)) ** 2
else:
d += a[j] != b[j]
return d
def prob(row):
near = sorted(train, key=lambda t: dist(row, t))[:k]
cnt = Counter(t[-1] for t in near)
return {c: cnt[c] / k for c in classes}
return prob
def tree(attrs, train, min_leaf=2):
"""Unpruned tree on information gain; numeric attributes get one binary split per node."""
classes = attrs[-1][1]
def build(rows, used):
counts = Counter(r[-1] for r in rows)
if len(counts) == 1 or len(rows) < 2 * min_leaf:
return counts
h, best = entropy(rows), None
for j, (name, vals) in enumerate(attrs[:-1]):
if vals is not None:
if j in used:
continue
cands = [(None, {v: [r for r in rows if r[j] == v] for v in vals})]
else:
xs = sorted(set(r[j] for r in rows))
cands = [((a + b) / 2, None) for a, b in zip(xs, xs[1:])]
cands = [(t, {'<=': [r for r in rows if r[j] <= t], '>': [r for r in rows if r[j] > t]}) for t, _ in cands]
for t, parts in cands:
if any(0 < len(p) < min_leaf for p in parts.values()):
continue
gain = h - sum(len(p) / len(rows) * entropy(p) for p in parts.values() if p)
if best is None or gain > best[0] + 1e-12:
best = (gain, j, t, parts)
if best is None or best[0] < 1e-9:
return counts
gain, j, t, parts = best
kids = {v: build(p, used | ({j} if t is None else set())) for v, p in parts.items() if p}
return (j, t, kids, counts)
root = build(train, set())
def prob(row):
node = root
while isinstance(node, tuple):
j, t, kids, counts = node
key = row[j] if t is None else ('<=' if row[j] <= t else '>')
node = kids.get(key, counts)
n = sum(node.values())
return {c: node[c] / n for c in classes}
prob.root = root
return prob
def show_tree(attrs, node, depth=0):
"""Print a tree the way WEKA's J48 does: branch per line, leaf as class (n/errors)."""
j, t, kids, counts = node
for key, kid in kids.items():
label = f"{attrs[j][0]} = {key}" if t is None else f"{attrs[j][0]} {key} {t:g}"
if isinstance(kid, tuple):
print('| ' * depth + label)
show_tree(attrs, kid, depth + 1)
else:
top, n = kid.most_common(1)[0], sum(kid.values())
wrong = n - top[1]
print('| ' * depth + f"{label}: {top[0]} ({n}.0" + (f"/{wrong}.0)" if wrong else ")"))
def sigmoid(s):
return 1 / (1 + math.exp(-max(-30.0, min(30.0, s))))
def logistic(attrs, train, epochs=3000, lr=0.5, ridge=1e-4):
classes = attrs[-1][1]
assert len(classes) == 2, "logistic here is two-class only"
vec = encoder(attrs, train)
X = [vec(r) for r in train]
Y = [1.0 if r[-1] == classes[0] else 0.0 for r in train]
w = [0.0] * len(X[0])
for _ in range(epochs):
g = [ridge * wi for wi in w]
for x, y in zip(X, Y):
e = sigmoid(sum(wi * xi for wi, xi in zip(w, x))) - y
for i, xi in enumerate(x):
g[i] += e * xi
w = [wi - lr * gi / len(X) for wi, gi in zip(w, g)]
def prob(row):
p = sigmoid(sum(wi * xi for wi, xi in zip(w, vec(row))))
return {classes[0]: p, classes[1]: 1 - p}
prob.w = w
return prob
def svm(attrs, train, epochs=300, C=1.0, seed=1):
"""Linear SVM trained with Pegasos (stochastic sub-gradient); WEKA's SMO uses a linear kernel by default too."""
classes = attrs[-1][1]
assert len(classes) == 2, "svm here is two-class only"
vec = encoder(attrs, train)
X = [vec(r) for r in train]
Y = [1.0 if r[-1] == classes[0] else -1.0 for r in train]
n, lam, t = len(X), 1 / (C * len(X)), 0
w = [0.0] * len(X[0])
rnd = random.Random(seed)
for _ in range(epochs):
order = list(range(n))
rnd.shuffle(order)
for i in order:
t += 1
eta = 1 / (lam * t)
hinge = Y[i] * sum(wi * xi for wi, xi in zip(w, X[i])) < 1
w = [(1 - eta * lam) * wi + (eta * Y[i] * xi if hinge else 0.0) for wi, xi in zip(w, X[i])]
def prob(row):
p = sigmoid(sum(wi * xi for wi, xi in zip(w, vec(row)))) # only for ranking in ROC
return {classes[0]: p, classes[1]: 1 - p}
return prob
# ---- evaluation ----
def kappa(cm, classes):
n = sum(sum(r.values()) for r in cm.values())
po = sum(cm[c][c] for c in classes) / n
pe = sum(sum(cm[c].values()) * sum(cm[r][c] for r in classes) for c in classes) / n ** 2
return (po - pe) / (1 - pe) if pe < 1 else 1.0
def auc(scores):
"""Mann-Whitney: share of (positive, negative) pairs ranked correctly; ties count half."""
pos = [s for s, y in scores if y]
neg = [s for s, y in scores if not y]
if not pos or not neg:
return float('nan')
return sum((p > q) + 0.5 * (p == q) for p in pos for q in neg) / (len(pos) * len(neg))
def evaluate(attrs, rows, make):
classes = attrs[-1][1]
cm = {c: Counter() for c in classes}
scores = []
for f in folds(rows):
test = set(f)
prob = make(attrs, [r for i, r in enumerate(rows) if i not in test])
for i in f:
p = prob(rows[i])
pred = max(classes, key=lambda c: p[c])
cm[rows[i][-1]][pred] += 1
scores.append((p[classes[0]], rows[i][-1] == classes[0]))
return cm, kappa(cm, classes), auc(scores)
def report(name, attrs, rows, cm, k, a):
classes = attrs[-1][1]
n, correct = len(rows), sum(cm[c][c] for c in classes)
print(f"=== {name} : 10-fold cross-validation ===")
print(f"Correctly Classified Instances {correct:4d} {100 * correct / n:7.4f} %")
print(f"Incorrectly Classified Instances {n - correct:4d} {100 * (n - correct) / n:7.4f} %")
print(f"Kappa statistic {k:.4f}")
print(f"ROC Area ({classes[0]}) {a:.4f}")
print("=== Confusion Matrix ===")
letters = 'abcdefghijklmnopqrstuvwxyz'
print(' ' + ' '.join(f"{letters[i]:>3}" for i in range(len(classes))) + ' <-- classified as')
for i, c in enumerate(classes):
print(' ' + ' '.join(f"{cm[c][d]:3d}" for d in classes) + f" | {letters[i]} = {c}")
print()
MODELS = [('NaiveBayes', naive_bayes), ('IBk (k=1)', knn), ('Tree (ID3/J48, unpruned)', tree),
('Logistic', logistic), ('SVM (linear, C=1)', svm)]
def main(paths):
for path in paths:
attrs, rows = load_arff(path)
print(f"##### {path}: {len(rows)} instances, {len(attrs)} attributes, class = {attrs[-1][0]} #####\n")
full = tree(attrs, rows)
print("=== Tree on the full training set ===")
show_tree(attrs, full.root)
print()
summary = []
for name, make in MODELS:
cm, k, a = evaluate(attrs, rows, make)
report(name, attrs, rows, cm, k, a)
correct = sum(cm[c][c] for c in attrs[-1][1])
summary.append((name, 100 * correct / len(rows), k, a))
print(f"{'classifier':26} {'accuracy %':>10} {'kappa':>7} {'ROC area':>9}")
for name, acc, k, a in summary:
print(f"{name:26} {acc:10.2f} {k:7.4f} {a:9.4f}")
print()
if __name__ == '__main__':
# self-check: kappa for WEKA's J48 result on weather.nominal (5 4 / 3 2) is -0.0426
check = {'yes': Counter({'yes': 5, 'no': 4}), 'no': Counter({'yes': 3, 'no': 2})}
assert abs(kappa(check, ['yes', 'no']) + 0.0426) < 5e-4
main(sys.argv[1:] or ['student.arff', 'employee.arff'])Output
Decision tree on student.arff (classify_cv.py, run for real; J48’s pruning step would merge the two small leaves under internal <= 14.5 into one leaf pass (4.0/1.0), everything else is the same tree):
=== Tree on the full training set ===
attendance <= 63.5: fail (10.0)
attendance > 63.5
| internal <= 14.5
| | hours <= 7.5: pass (2.0)
| | hours > 7.5: pass (2.0/1.0)
| internal > 14.5: pass (16.0)
=== Tree (ID3/J48, unpruned) : 10-fold cross-validation ===
Correctly Classified Instances 29 96.6667 %
Incorrectly Classified Instances 1 3.3333 %
Kappa statistic 0.9268
ROC Area (pass) 0.9450
=== Confusion Matrix ===
a b <-- classified as
19 0 | a = pass
1 10 | b = failDecision tree on employee.arff:
=== Tree on the full training set ===
experience <= 5.5: no (12.0)
experience > 5.5
| experience <= 9.5
| | age <= 32.5
| | | experience <= 6.5: yes (2.0/1.0)
| | | experience > 6.5: yes (2.0)
| | age > 32.5: yes (2.0/1.0)
| experience > 9.5: yes (12.0)Results for the three classifiers of this question, 10-fold cross-validation (student and employee computed by the script; labor values are the ones WEKA 3.8 prints for its bundled file with the default seed, so paste the block from your own run):
| Data set | Classifier | Accuracy | Kappa | Confusion matrix (rows = actual) |
|---|---|---|---|---|
| student | Logistic | 86.67 % (26/30) | 0.7129 | pass: 17 2; fail: 2 9 |
| student | J48 | 96.67 % (29/30) | 0.9268 | pass: 19 0; fail: 1 10 |
| student | NaiveBayes | 83.33 % (25/30) | 0.6479 | pass: 16 3; fail: 2 9 |
| employee | Logistic | 83.33 % (25/30) | 0.6696 | yes: 12 4; no: 1 13 |
| employee | J48 | 86.67 % (26/30) | 0.7297 | yes: 15 1; no: 3 11 |
| employee | NaiveBayes | 86.67 % (26/30) | 0.7345 | yes: 13 3; no: 1 13 |
| labor | Logistic | about 88 % | record | record |
| labor | J48 | 73.68 % (42/57) | 0.4415 | bad: 14 6; good: 9 28 |
| labor | NaiveBayes | 89.47 % (51/57) | record | record |
The J48 block WEKA prints for labor.arff:
=== Stratified cross-validation ===
=== Summary ===
Correctly Classified Instances 42 73.6842 %
Incorrectly Classified Instances 15 26.3158 %
Kappa statistic 0.4415
Total Number of Instances 57
=== Confusion Matrix ===
a b <-- classified as
14 6 | a = bad
9 28 | b = goodExplanation
- Logistic regression fits one weight per attribute (nominal attributes become 0/1 indicator columns) and passes the weighted sum through the logistic function to get a probability of the first class. Its decision boundary is a straight line in attribute space, which suits
employee(promotion rises with experience and salary together) andstudentreasonably well. Theridgevalue shrinks the weights slightly so the solution exists even when a class is perfectly separable. - J48 picks the attribute and threshold with the highest information gain at each node (formula sheet), so the student tree splits first on
attendance <= 63.5, which alone sends all 10 low-attendance students tofailwith no error, then oninternal. It scores highest onstudentbecause the data was generated by exactly such a rule. Onemployeethe two noisy rows (experience 6 and 9, not promoted) force the small impure leaves and cost accuracy. - Naive Bayes multiplies the class prior by one Gaussian likelihood per numeric attribute and one frequency ratio per nominal attribute. It has no thresholds, so it is slightly worse on the rule-shaped
studentdata and best onemployee, where the class is a smooth function of two correlated numbers. - Kappa removes the agreement expected by chance. On
student, chance agreement is for the J48 matrix, so , exactly the printed value. - labor has only 57 rows and many missing values, so the pruned tree is unstable across folds (73.7 %) while Naive Bayes, which simply skips a missing attribute in the product, is the most accurate of the three.
Question 16
Problem Statement
Write in lab recordDemonstrate the classification rule process on the student.arff, employee.arff and labor.arff datasets using the following algorithms:
- K-Nearest Neighbour
- SVM
Solution
Write in lab recordSteps
- With
student.arffstill loaded and 10-fold cross-validation selected, Choose,lazy.IBk. Click the name to open the dialog:KNN1 (default). Start. Then setKNNto 3 anddistanceWeightingtoWeight by 1/distanceand Start again to see the effect of a larger neighbourhood. - Choose,
functions.SMO. Defaults:c1.0,kernelPolyKernel with exponent 1.0 (a linear SVM),filterTypeNormalize training data. Start. The model block prints one weight per (normalised) attribute and the bias, in the form-1.2 * (normalized) hours + ... + 0.8. - Repeat both for
employee.arffandlabor.arff. - For a side-by-side view of all five classifiers, open the Experimenter (New, Add new… data set, Add new… algorithm for each of the five, Run, then Analyse with Percent_correct as the comparison field). The Explorer is enough for the record book.
Output
classify_cv.py for the two classifiers of this question (run for real; k = 1 for IBk, linear kernel and C = 1 for the SVM, which is trained with the Pegasos sub-gradient method instead of SMO, so WEKA’s numbers can differ by a row or two):
=== IBk (k=1) : 10-fold cross-validation === student.arff
Correctly Classified Instances 27 90.0000 %
Kappa statistic 0.7805
=== Confusion Matrix ===
a b <-- classified as
18 1 | a = pass
2 9 | b = fail
=== SVM (linear, C=1) : 10-fold cross-validation === student.arff
Correctly Classified Instances 22 73.3333 %
Kappa statistic 0.4030
=== Confusion Matrix ===
a b <-- classified as
16 3 | a = pass
5 6 | b = fail
=== IBk (k=1) : 10-fold cross-validation === employee.arff
Correctly Classified Instances 23 76.6667 %
Kappa statistic 0.5374
=== Confusion Matrix ===
a b <-- classified as
11 5 | a = yes
2 12 | b = no
=== SVM (linear, C=1) : 10-fold cross-validation === employee.arff
Correctly Classified Instances 22 73.3333 %
Kappa statistic 0.4690
=== Confusion Matrix ===
a b <-- classified as
11 5 | a = yes
3 11 | b = noAll five classifiers together (the summary table the script prints at the end of each data set):
student.arff
classifier accuracy % kappa ROC area
NaiveBayes 83.33 0.6479 0.9187
IBk (k=1) 90.00 0.7805 0.8828
Tree (ID3/J48, unpruned) 96.67 0.9268 0.9450
Logistic 86.67 0.7129 0.9139
SVM (linear, C=1) 73.33 0.4030 0.8804
employee.arff
classifier accuracy % kappa ROC area
NaiveBayes 86.67 0.7345 0.9688
IBk (k=1) 76.67 0.5374 0.7723
Tree (ID3/J48, unpruned) 86.67 0.7297 0.8683
Logistic 83.33 0.6696 0.8973
SVM (linear, C=1) 73.33 0.4690 0.8705On labor.arff WEKA prints about 82 % for IBk with k = 1 and 89.47 % (51/57) for SMO; paste the kappa and confusion matrix from your run.
Explanation
- k-NN stores the training rows and, for each test row, takes the majority class of the k closest rows under Euclidean distance on attributes scaled to 0 to 1 (nominal attributes count 1 when different). It needs the scaling because
attendanceranges over 58 units andhoursover 16; without itattendancewould decide everything. With k = 1 one noisy neighbour flips a prediction, which is why theemployeeresult (76.7 %) is the weakest of the five there: the noisy rows 26 and 27 sit inside the promoted region. k = 3 with distance weighting smooths this out. - SVM finds the line (hyperplane) with the largest margin between the classes;
csets how many margin violations are tolerated. Onstudentthe true boundary is an L-shaped rule (attendance and internal both matter, but as thresholds, not as a weighted sum), so a linear kernel cannot follow it and the SVM comes last at 73.3 %. SwitchingkerneltoRBFKernelor raising the PolyKernel exponent to 2 lets it bend and recovers most of the loss. Onlabor, whose attributes are mostly ordinal wage figures, a linear boundary is right and SMO is joint best. - The ROC area column comes from ranking the test rows by the predicted probability of the first class; k = 1 gives only 0 or 1 scores, so its ROC area is little more than its accuracy, whereas Naive Bayes and Logistic produce graded scores and rank rows well even when their hard predictions are wrong. Session 8 uses this column.
Formula Sheet
Do not copy. Read for understanding and the vivaAssociation rules
For a rule over transactions:
WEKA’s Apriori starts at the upper bound of minimum support and lowers it by the delta each pass until the requested number of rules is found or the lower bound is reached.
Entropy, information gain and Gini
For a set with class proportions :
ID3 splits on the attribute with the highest gain; J48 (C4.5) uses the gain ratio where .
Classifier evaluation
From the confusion matrix with true positives , false positives , false negatives , true negatives :
Kappa compares observed agreement (accuracy) with the agreement expected by chance :
The ROC curve plots true positive rate against false positive rate ; the area under it (AUC) is 0.5 for guessing and 1.0 for a perfect classifier.
Naive Bayes and k-nearest neighbour
k-NN assigns the majority class among the nearest training records under Euclidean distance
after normalising each attribute to with .
Linear regression
WEKA reports the correlation coefficient, mean absolute error and root mean squared error .
Clustering
k-means minimises the within-cluster sum of squared errors over clusters with centroids :
Hierarchical (agglomerative) clustering merges the two closest clusters each step; linkage defines closeness: single , complete , average .
DBSCAN calls a point a core point when at least points lie within radius ; clusters grow from core points, and points reachable from none are noise.
Viva Questions
Do not copy. Read for understanding and the vivaQ: Why 10-fold cross-validation and not the training set? A: Training-set accuracy rewards memorising; cross-validation tests each row with a model that never saw it, so it estimates performance on new data.
Q: What does pass (4.0/1.0) on a J48 leaf mean? A: Four training instances reach that leaf and one of them is not pass.
Q: Why does IBk normalise attributes? A: Euclidean distance adds squared differences; an attribute with a large range would swamp the others unless all are scaled to 0 to 1.
Q: Why is Naive Bayes called naive? A: It assumes the attributes are conditionally independent given the class so the joint likelihood is a plain product.
Q: What is the c parameter of SMO? A: The penalty for points on the wrong side of the margin; a large c fits the training data more tightly.
Q: What does a kappa of 0 mean? A: The classifier agrees with the true labels no more often than random assignment with the same class proportions would.
Q: Which of the five classifiers has no training phase? A: IBk; it stores the data and does all the work at prediction time.
Q: How does Logistic regression handle a nominal attribute such as stream? A: It expands it into one 0/1 indicator column per value and learns a weight for each.
Common Mistakes
Do not copy. Read for understanding and the viva- Leaving Test options at “Use training set” and reporting 100 percent for J48 and IBk.
- Building
student.arffwith a numeric class such as marks and then wondering why the classifiers are greyed out. - Using the class attribute as an input by picking the wrong attribute in the class chooser below the Test options.
- Comparing classifiers run with different fold counts or seeds; keep every run at 10 folds, seed 1.
- Reading the confusion matrix with rows and columns swapped; rows are the actual class, columns the predicted one.
- Reporting IBk with k = 1 as “the k-NN result” without trying a larger k; one neighbour is the noisiest setting.
Session Summary
Write in lab record- Question 15:
student.arffandemployee.arff(30 rows each, defined here) andlabor.arffclassified with Logistic, J48 and NaiveBayes under 10-fold cross-validation; accuracy, kappa and confusion matrix tabulated, J48 trees recorded. - Question 16: the same three data sets with IBk (k = 1 and 3) and SMO; five-classifier comparison table per data set from
classify_cv.py.