Results - Autonomous Driving - ELSA Benchmarks Platform

Task 1 - Track 1 – Single-domain training (Models must be trained exclusively on the published Cityscapes dataset)
Task 2 - Track 2 – Multi-domain training (Models may be trained over a mix of datasets, whose choice is strictly limited to the list provided in the submission

method: InternImage-OOD2024-08-23

Authors: Long Qian, Bingke Zhu, Yingying Chen, Ming Tang, Jinqiao Wang

Affiliation: CASIA

Email: qianlong2024@ia.ac.cn

Description: InternImage-OOD is a cutting-edge framework designed to enhance OOD detection by combining powerful image features with clustering techniques. This approach leverages the strengths of InternImage for feature extraction, followed by KMeans clustering to detect and refine OOD regions in the input images.

Source code

method: PixOOD w/ DeepLab Decoder Cityscapes+BDD100K2024-08-22

Authors: Tomáš Vojíř , Jan Šochman , and Jiří Matas

Affiliation: Czech Technical University in Prague

Description: This method is based on PixOOD but replace the simple MLP for semantic segmentation with more complex decoder from DeepLab v3 architecture.

This version was trained on Cityscapes and BDD100K datasets.

The details and differences w.r.t the PixOOD are described in the technical paper.

PixOOD: Pixel-Level Out-of-Distribution Detection

Technical Paper

Source code

method: Ensemble - Multi dataset C2024-08-23

Authors: Michael Smith, Frank Ferrie

Affiliation: McGill University

Description: Ensemble of 4 models - 2 from Hierarchical Multi-Scale Attention for Semantic Segmentation trained each on Cityscapes and Mapillary Vistas, and 2 from Mask2Former trained on Cityscapes (SwinL and SwinB)

BRAVO Challenge Report - Ensembles

Source code

Ranking Table

Description Paper Source Code

		BRAVO Index			Subset H-Means									ACDCfog						ACDCnight						ACDCrain						ACDCsnow						SMIYC			outofcontext						synflare						synobjs									synrain
Date	Method	bravo	semantic	ood	ACDCfog	ACDCnight	ACDCrain	ACDCsnow	SMIYC	outofcontext	synflare	synobjs	synrain	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_ood	auroc_ood	fpr95_ood	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_ood	auprc_suc	auroc_ood	auroc	ece	fpr95_ood	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou
2024-08-23	InternImage-OOD	0.6263	0.6934	0.5710	0.7131	0.5722	0.6904	0.6703	0.5553	0.7204	0.7461	0.6800	0.7338	0.4010	0.9960	0.9285	0.0237	0.3470	0.7791	0.4121	0.9727	0.8393	0.0711	0.6728	0.5859	0.3994	0.9938	0.9113	0.0167	0.4187	0.7432	0.3963	0.9908	0.8965	0.0357	0.4745	0.7335	0.5169	0.8485	0.5631	0.4087	0.9976	0.9462	0.0125	0.3164	0.7423	0.4664	0.9965	0.9413	0.0169	0.3472	0.7797	0.4205	0.3532	0.9972	0.9451	0.9426	0.0164	0.1777	0.2959	0.7987	0.4299	0.9974	0.9468	0.0148	0.3254	0.7729
2024-08-22	PixOOD w/ DeepLab Decoder Cityscapes+BDD100K	0.6081	0.5226	0.7272	0.5676	0.6115	0.6105	0.5309	0.6225	0.5019	0.6807	0.6591	0.3101	0.3578	0.9636	0.7792	0.0717	0.6481	0.6511	0.5362	0.9458	0.8015	0.0216	0.6034	0.4766	0.4057	0.9740	0.8154	0.0749	0.5983	0.6492	0.3279	0.9558	0.7480	0.0786	0.6929	0.6543	0.7053	0.8978	0.5628	0.2692	0.9743	0.7794	0.0409	0.6938	0.6168	0.5444	0.9632	0.8463	0.0410	0.5035	0.5880	0.3917	0.7202	0.9674	0.9902	0.7858	0.0554	0.0326	0.6380	0.6479	0.1198	0.9687	0.6814	0.0597	0.8345	0.7151
2024-08-23	Ensemble - Multi dataset C	0.5879	0.6448	0.5402	0.6254	0.6040	0.6707	0.6552	0.5840	0.6341	0.6867	0.5845	0.6535	0.3689	0.9929	0.9136	0.4075	0.4511	0.7865	0.4892	0.9799	0.8823	0.3664	0.6022	0.6086	0.4568	0.9952	0.9365	0.4061	0.4167	0.7777	0.4680	0.9922	0.9202	0.4062	0.4810	0.7602	0.4004	0.8445	0.3131	0.3470	0.9977	0.9446	0.4174	0.3330	0.7671	0.4879	0.9957	0.9412	0.4060	0.3934	0.7759	0.3423	0.2626	0.9966	0.9470	0.9330	0.4158	0.0960	0.3436	0.8353	0.3770	0.9977	0.9463	0.4188	0.3438	0.8158
2024-08-23	Ensemble - Multi dataset B	0.4554	0.6456	0.3517	0.6379	0.6014	0.6446	0.6449	0.4027	0.6613	0.6626	0.4783	0.6655	0.3708	0.9898	0.8725	0.2433	0.4755	0.7548	0.4566	0.9789	0.8677	0.2075	0.6201	0.5818	0.3729	0.9943	0.9122	0.2191	0.4791	0.7550	0.4018	0.9882	0.8871	0.2426	0.4847	0.7069	0.3926	0.7709	0.7226	0.3844	0.9960	0.9282	0.2010	0.4004	0.6802	0.4226	0.9917	0.9007	0.2255	0.4568	0.6954	0.3606	0.1451	0.9947	0.8621	0.9081	0.2072	0.3586	0.4241	0.7514	0.3883	0.9956	0.9217	0.2147	0.4167	0.7338
2024-08-22	Ensemble - Multi dataset A	0.4063	0.6599	0.2935	0.6507	0.6154	0.6590	0.6588	0.3537	0.6800	0.6724	0.4293	0.6814	0.3679	0.9889	0.8670	0.0781	0.4781	0.7477	0.4573	0.9781	0.8646	0.0497	0.6151	0.5769	0.3717	0.9939	0.9093	0.0462	0.4814	0.7505	0.4024	0.9893	0.8869	0.0797	0.4939	0.7035	0.3733	0.7475	0.7761	0.3921	0.9959	0.9274	0.0219	0.3989	0.6649	0.4190	0.9914	0.8970	0.0572	0.4751	0.6912	0.3595	0.1310	0.9946	0.8198	0.9057	0.0307	0.6781	0.4266	0.7476	0.3869	0.9954	0.9199	0.0388	0.4174	0.7284
2024-08-01	FAMix-R50-GTA5	0.2679	0.4953	0.1836	0.5142	0.4254	0.5238	0.4754	0.2046	0.5582	0.5241	0.2968	0.4778	0.5068	0.9340	0.8064	0.2073	0.6964	0.3534	0.5330	0.8548	0.7348	0.0807	0.7718	0.2365	0.4725	0.9466	0.8097	0.2389	0.6837	0.3889	0.4771	0.9118	0.7659	0.2193	0.7514	0.3545	0.2516	0.6551	0.8908	0.4821	0.9666	0.8428	0.2669	0.6095	0.3955	0.5114	0.9327	0.8037	0.2389	0.6980	0.3920	0.3878	0.0829	0.9482	0.7450	0.7781	0.2929	0.7834	0.7291	0.4392	0.4480	0.9291	0.7810	0.2570	0.7604	0.4079
2024-08-23	CoPT	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000

Autonomous Driving

Inactive evaluations

method: InternImage-OOD2024-08-23

method: PixOOD w/ DeepLab Decoder Cityscapes+BDD100K2024-08-22

method: Ensemble - Multi dataset C2024-08-23

Ranking Table

Ranking Graphic - BRAVO Index

Ranking Graphic - Semantic H-Mean

Ranking Graphic - OOD H-Mean