import fastai
print(fastai.__version__)
path = Path("../data/")
The path variable is a os.pathlib.Path object that points to the competition data.
To print all the files this directory use the print_competition_data function.
print_competition_data(path)
The competition files in detail:
- train_images/ - folder of training images (12.5k images)
- test_images/ - folder of test images to segment and classify (5506 images)
- train.csv - training annotations which provide segments for defects (
ClassId = [1, 2, 3, 4]) - sample_submission.csv - a sample submission file in the correct format; note, each
ImageId4 rows, one for each of the 4 defect classes
train_path = path/"train_images"
test_path = path/"test_images"
train_pfiles = get_image_files(train_path)
test_pfiles = get_image_files(test_path)
The training data includes:
faulty images: images that have at least one defect
hard negative images: images with no defects
train_pfiles
The get_train_df function returns the DataFrame from the train.csv file, only faulty image names if only_faulty, with the training images metadata:
ImageId: image name
ClassId: the class type
EncodedPixels: the encoded pixels follows a run-length encoding rule, a sequence of pair values that contains a start position and a run length with the space as the delimiter. E.g.
1 3 10 5means pixels(1,2,3)and(10,11,12,13,14).
Each Image may have no defects, a single defect, or multiple defects.
hard_neg_patterns = pd.read_csv(
path/"hard_negatives_patterns.txt", header=None, names=["ImageId"])
hard_neg_patterns.head()
In hard_neg_patterns there are the ImageIds of training images with some patterns (from this kernel).
train = get_train_df(path, only_faulty=False)
assert isinstance(train, pd.DataFrame)
train_faulty = train.loc[train["status"]=="faulty"]
assert not train.ImageId_ClassId.duplicated().any(), "Found ImageId_ClassId duplicates"
train.describe(include='all')[:4]
print(train.shape)
train.head(2)
missing_imgs = train["ImageId"].map(lambda x: not ((path/"train_images"/str(x)).is_file()))
print(missing_imgs.sum())
print(train_faulty.shape)
train_faulty.head(2)
The ClassId column values from train are:
class_count = count_pct(train)
class_count
class_count["num"].plot.bar(title="Defects by ClassId count");
Images have at least one defect and there's a small number of images with two or three defects.
counts = train_faulty["ImageId"].value_counts()
hist_counts, _ = np.histogram(counts.values, bins=3)
nums = ['1', '2', '3']
plt.bar(x=nums, height=hist_counts)
plt.title("Num of defects per images")
plt.show()
{i+1: c for i, c in enumerate(hist_counts)}
unique_imgs = train_faulty.describe(include='all')["ImageId"].T[:2]
unique_imgs
fig, ax = plt.subplots(1, 1, figsize=(7, 6.5))
ax.set_title("Count imgs", pad=30, fontdict={'fontsize': 14})
ax.xaxis.tick_top() # Display x-axis ticks on top
(
train[['ImageId', 'status']]
.drop_duplicates()
.status.value_counts().iloc[:-1]
.plot.barh(table=True, ax=ax)
)
plt.show()
Loading the images for models requires some transformations to the DataFrames.
The get_train_pivot is the pivoted version of df. All the images are in the index, for each image the ClassId encoding values are in the columns.
train_pivot = get_train_df(path, pivot=True)
train_pivot.head(2)
count_pct(train_pivot, column='n')
count_pct(train_pivot, column='ClassIds').sort_values("freq")
The get_classification_df allows to build a DataFrame to classification models. In ClassId_multi are listed the ClassIds separated by a space.
train_multi = get_classification_df(train)
train_multi.head()
count_pct(train_multi, column='ClassId_multi').sort_values("freq", ascending=False)
train_multi.describe(include='all')
test_pfiles
test_df = pd.read_csv(path / 'sample_submission.csv')
test_df.head()
def old_rle2mask(rle: str, value: int, shape):
"""
From a RLE encoded pixels returns a mask
with `value` for defected pixels
(e.g. `value`=1 so 1 -> defected, 0 -> groundtruth)
and `shape` as tuple (height, width).
"""
assert len(shape) == 2, "The shape must be (height, width)"
assert isinstance(shape[0], int)
assert isinstance(shape[1], int)
h, w = shape
mask = np.zeros(h * w, dtype=np.uint8)
rle = rle.split(" ")
positions = map(int, rle[0::2])
length = map(int, rle[1::2])
for pos, le in zip(positions, length):
mask[pos:(pos + le)] = value
return mask.reshape(h, w, order='F')
item = train_faulty.iloc[1]
item_class_id = item["ClassId"]
item_rle = item["EncodedPixels"]
item_rle[:50]
s = item_rle.split()
how_many_values = 10
pixels = s[0:][::2]
lengths = s[1:][::2]
(pixels[:how_many_values], lengths[:how_many_values])
starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]
starts -= 1
starts, lengths
ends = starts + lengths
ends
mask_img = np.zeros(256*1600, dtype=np.uint8)
for lo, hi in zip(starts, ends):
mask_img[lo:hi] = 1
mask_img[18658:18698]
item_mask = rle2mask(item_rle, 1)
show_image(item_mask, figsize=(15,5));
mask = rle2mask(item_rle)
rle = mask2rle(mask)
rle[:100]
test_eq(rle, item_rle)
show_defects(path, train_pivot)