Results - Autonomous Driving - ELSA Benchmarks Platform

Task 1 - Track 1 – Single-domain training (Models must be trained exclusively on the published Cityscapes dataset)
Task 2 - Track 2 – Multi-domain training (Models may be trained over a mix of datasets, whose choice is strictly limited to the list provided in the submission

method: DINOv2, ViT-L, 8x8 patch size, linear decoder2024-08-23

Authors: Tommie Kerssies, Daan de Geus, and Gijs Dubbelman

Affiliation: Eindhoven University of Technology

Description: Fine-tuning for ~40 epochs on Cityscapes, following the setup described in: "How to Benchmark Vision Foundation Models for Semantic Segmentation?" (https://www.tue-mps.org/benchmark-vfm-ss/)

Tommie Kerssies, Daan de Geus, and Gijs Dubbelman. “How to Benchmark Vision Foundation Models for Semantic Segmentation?” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024

Source code

method: DINOv2, ViT-G, 16x16 patch size, linear decoder2024-08-23

Authors: Tommie Kerssies, Daan de Geus, and Gijs Dubbelman

Affiliation: Eindhoven University of Technology

Email: t.kerssies@tue.nl

Source code

method: DINOv2, ViT-B, 16x16 patch size, linear decoder2024-08-23

Authors: Tommie Kerssies, Daan de Geus, and Gijs Dubbelman

Affiliation: Eindhoven University of Technology

Email: t.kerssies@tue.nl

Source code

Ranking Table

Description Paper Source Code

		BRAVO Index			Subset H-Means									ACDCfog						ACDCnight						ACDCrain						ACDCsnow						SMIYC			outofcontext						synflare						synobjs									synrain
Date	Method	bravo	semantic	ood	ACDCfog	ACDCnight	ACDCrain	ACDCsnow	SMIYC	outofcontext	synflare	synobjs	synrain	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_ood	auroc_ood	fpr95_ood	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou	auprc_err	auprc_ood	auprc_suc	auroc_ood	auroc	ece	fpr95_ood	fpr95	miou	auprc_err	auprc_suc	auroc	ece	fpr95	miou
2024-08-23	DINOv2, ViT-L, 8x8 patch size, linear decoder	0.7789	0.6981	0.8807	0.6603	0.6808	0.6748	0.6756	0.8991	0.7104	0.7268	0.7666	0.7393	0.3585	0.9886	0.8619	0.0247	0.4451	0.7702	0.4415	0.9911	0.9102	0.0406	0.4767	0.6724	0.3580	0.9939	0.9005	0.0190	0.4097	0.7880	0.3686	0.9910	0.8839	0.0234	0.4242	0.7891	0.8901	0.9727	0.1563	0.4112	0.9962	0.9330	0.0119	0.3565	0.7288	0.4227	0.9966	0.9370	0.0157	0.3507	0.7916	0.4127	0.7442	0.9966	0.9804	0.9348	0.0135	0.1010	0.3328	0.7965	0.4234	0.9976	0.9482	0.0113	0.3057	0.8030
2024-08-23	DINOv2, ViT-G, 16x16 patch size, linear decoder	0.7612	0.7001	0.8340	0.6660	0.6786	0.6766	0.6871	0.8818	0.7100	0.7316	0.7469	0.7317	0.3492	0.9921	0.8877	0.0194	0.4170	0.7845	0.4046	0.9929	0.9141	0.0326	0.4529	0.7102	0.3517	0.9949	0.9118	0.0174	0.3960	0.8002	0.3734	0.9946	0.9153	0.0212	0.4093	0.8052	0.8649	0.9685	0.1757	0.4157	0.9958	0.9319	0.0122	0.3622	0.7207	0.4206	0.9967	0.9404	0.0145	0.3285	0.7980	0.4185	0.6212	0.9964	0.9691	0.9371	0.0149	0.1310	0.3338	0.7803	0.4073	0.9976	0.9474	0.0108	0.3055	0.8087
2024-08-23	DINOv2, ViT-B, 16x16 patch size, linear decoder	0.7554	0.7046	0.8142	0.6903	0.6650	0.7069	0.6795	0.8790	0.7116	0.7277	0.7404	0.7302	0.4063	0.9931	0.9075	0.0194	0.4071	0.7066	0.4596	0.9881	0.9021	0.0455	0.5233	0.6325	0.4140	0.9959	0.9317	0.0173	0.3818	0.7355	0.4058	0.9907	0.8980	0.0267	0.4615	0.7365	0.8440	0.9648	0.1607	0.4451	0.9955	0.9337	0.0121	0.3875	0.6829	0.4573	0.9955	0.9351	0.0175	0.3767	0.7339	0.4395	0.5689	0.9961	0.9671	0.9365	0.0131	0.1414	0.3427	0.7585	0.4382	0.9968	0.9430	0.0125	0.3428	0.7480
2024-08-23	PixOOD YOLO (="Model Selection")	0.6779	0.5706	0.8349	0.5688	0.4895	0.6199	0.5514	0.7487	0.6487	0.6496	0.5638	0.6328	0.6905	0.8930	0.7987	0.2852	0.5393	0.3166	0.7060	0.9099	0.8410	0.0887	0.6267	0.2094	0.6800	0.9261	0.8299	0.2018	0.5079	0.3790	0.8014	0.8365	0.8263	0.1796	0.5598	0.2676	0.7238	0.9332	0.3563	0.4486	0.9848	0.8586	0.0900	0.5117	0.5908	0.5739	0.9719	0.8848	0.1249	0.5064	0.4618	0.1998	0.8557	0.9826	0.9969	0.7863	0.0991	0.0081	0.6673	0.7285	0.4857	0.9753	0.8511	0.1386	0.5415	0.5315
2024-08-23	DeiT III (IN21K->IN1K), ViT-B, 16x16 patch size, linear decoder	0.6665	0.6649	0.6682	0.6256	0.6120	0.6701	0.6383	0.6820	0.6925	0.6796	0.6954	0.6985	0.4690	0.9755	0.8597	0.0182	0.5856	0.5436	0.5262	0.9759	0.8807	0.0633	0.6113	0.4754	0.4845	0.9879	0.9021	0.0190	0.5110	0.5864	0.4789	0.9791	0.8719	0.0134	0.5778	0.5679	0.5577	0.9014	0.3317	0.4809	0.9931	0.9246	0.0133	0.4462	0.5960	0.5192	0.9885	0.9136	0.0346	0.5220	0.5993	0.4692	0.4242	0.9951	0.9602	0.9347	0.0109	0.1539	0.3990	0.6758	0.4622	0.9941	0.9283	0.0110	0.4323	0.6368
2024-08-23	DINOv2, ViT-G, 16x16 patch size, Mask2Former decoder	0.6454	0.4968	0.9208	0.5143	0.6134	0.4062	0.4303	0.9437	0.4090	0.5389	0.6432	0.6007	0.2075	0.9877	0.8230	0.0389	0.5707	0.7996	0.3491	0.9931	0.9093	0.0545	0.5730	0.7088	0.1710	0.9930	0.8648	0.0642	0.7759	0.8138	0.1862	0.9907	0.8546	0.0657	0.7561	0.8160	0.9146	0.9849	0.0656	0.1905	0.9910	0.8674	0.0643	0.7922	0.7257	0.2358	0.9944	0.9013	0.0440	0.5968	0.7989	0.2505	0.7677	0.9936	0.9909	0.8981	0.0408	0.0251	0.5508	0.7850	0.2681	0.9955	0.9145	0.0309	0.4723	0.8099
2024-08-23	Model selection	0.6349	0.6939	0.5852	0.6882	0.6103	0.7066	0.6598	0.7071	0.7335	0.7765	0.5887	0.7639	0.4127	0.9911	0.8972	0.0385	0.4413	0.7453	0.5033	0.9720	0.8676	0.1050	0.6307	0.5438	0.4814	0.9902	0.9115	0.0499	0.4458	0.6897	0.4375	0.9838	0.8792	0.0583	0.5309	0.6849	0.7981	0.9137	0.4724	0.3961	0.9983	0.9547	0.0182	0.2410	0.7874	0.5483	0.9956	0.9464	0.0397	0.3215	0.7539	0.3646	0.2564	0.9963	0.9611	0.9255	0.4121	0.0654	0.3682	0.8577	0.4395	0.9985	0.9604	0.0182	0.2331	0.8237
2024-08-22	PixOOD w/ ResNet-101 DeepLab	0.6119	0.5866	0.6395	0.5688	0.4895	0.6199	0.5514	0.5318	0.6487	0.6496	0.6320	0.6328	0.6905	0.8930	0.7987	0.2852	0.5393	0.3166	0.7060	0.9099	0.8410	0.0887	0.6267	0.2094	0.6800	0.9261	0.8299	0.2018	0.5079	0.3790	0.8014	0.8365	0.8263	0.1796	0.5598	0.2676	0.3551	0.8430	0.3899	0.4486	0.9848	0.8586	0.0900	0.5117	0.5908	0.5739	0.9719	0.8848	0.1249	0.5064	0.4618	0.2963	0.5869	0.9849	0.9901	0.8268	0.1027	0.0263	0.5748	0.6972	0.4857	0.9753	0.8511	0.1386	0.5415	0.5315
2024-08-23	Ensemble C	0.6109	0.6428	0.5820	0.6518	0.6170	0.6867	0.6366	0.6307	0.6165	0.6958	0.5810	0.6463	0.4673	0.9890	0.9080	0.5330	0.3238	0.7503	0.5789	0.9753	0.8995	0.4566	0.5454	0.5493	0.5494	0.9932	0.9422	0.5178	0.2519	0.6954	0.4825	0.9833	0.8958	0.5106	0.4215	0.6912	0.4311	0.8886	0.2375	0.3737	0.9975	0.9422	0.5573	0.3175	0.7882	0.5596	0.9951	0.9475	0.5299	0.2489	0.7587	0.3555	0.2895	0.9957	0.9666	0.9204	0.5562	0.0598	0.3804	0.8576	0.4248	0.9975	0.9470	0.5556	0.2902	0.8248
2024-08-20	Ensemble A	0.5986	0.6725	0.5393	0.6837	0.6214	0.7014	0.6484	0.5866	0.6596	0.7399	0.5887	0.6935	0.4574	0.9910	0.9146	0.3900	0.3381	0.7479	0.5665	0.9734	0.8980	0.3288	0.5889	0.5471	0.5335	0.9921	0.9367	0.3755	0.3586	0.6926	0.4691	0.9814	0.8898	0.3680	0.4854	0.6880	0.3887	0.8667	0.2795	0.3816	0.9977	0.9451	0.4134	0.3062	0.7878	0.5617	0.9951	0.9481	0.3871	0.2404	0.7567	0.3646	0.2564	0.9963	0.9611	0.9255	0.4121	0.0654	0.3682	0.8577	0.4295	0.9979	0.9513	0.4121	0.2629	0.8243
2024-08-22	PixOOD w/ DeepLab Decoder	0.5937	0.4606	0.8349	0.3829	0.5583	0.4894	0.5071	0.7487	0.5615	0.4362	0.5638	0.3640	0.1670	0.9616	0.6659	0.2212	0.7759	0.7167	0.3461	0.9684	0.7937	0.0964	0.6462	0.6126	0.2292	0.9777	0.7735	0.1550	0.6607	0.6898	0.2801	0.9655	0.7431	0.1907	0.6893	0.7016	0.7238	0.9332	0.3563	0.3037	0.9835	0.8200	0.0778	0.6100	0.6630	0.2015	0.9788	0.7738	0.1340	0.7453	0.7161	0.1998	0.8557	0.9826	0.9969	0.7863	0.0991	0.0081	0.6673	0.7285	0.1524	0.9725	0.7236	0.1262	0.7987	0.7145
2024-08-20	PixOOD	0.5347	0.4038	0.7909	0.2588	0.5213	0.4291	0.4235	0.6670	0.4997	0.4837	0.4943	0.3640	0.1245	0.9408	0.6113	0.2961	0.8944	0.6389	0.3734	0.9519	0.7820	0.1105	0.7193	0.5511	0.2145	0.9685	0.7580	0.2177	0.7589	0.6480	0.2323	0.9519	0.7157	0.2965	0.7730	0.6293	0.7332	0.9057	0.5074	0.2598	0.9777	0.7880	0.0929	0.6897	0.6483	0.2611	0.9709	0.7645	0.1478	0.7201	0.6711	0.1723	0.9261	0.9745	0.9976	0.7387	0.1276	0.0064	0.7710	0.7038	0.1574	0.9653	0.6903	0.1226	0.8026	0.6938
2024-07-19	SegFormer-B5	0.4714	0.4530	0.4915	0.4646	0.4575	0.4340	0.4647	0.6049	0.3905	0.5063	0.4171	0.5148	0.2411	0.9708	0.7583	0.3682	0.7057	0.6949	0.3171	0.9333	0.7123	0.2668	0.7605	0.5136	0.2119	0.9750	0.7632	0.3128	0.7402	0.6813	0.2775	0.9602	0.7564	0.2925	0.7468	0.6317	0.5876	0.8746	0.5271	0.1665	0.9801	0.7453	0.2890	0.7590	0.6930	0.2804	0.9796	0.8077	0.3351	0.6777	0.7043	0.1761	0.4316	0.9861	0.8542	0.8010	0.3094	0.7341	0.7221	0.7449	0.2560	0.9883	0.8357	0.2851	0.6392	0.7306
2024-07-29	ObsNet-R101-DLv3plus	0.4530	0.5145	0.4046	0.5241	0.4775	0.5201	0.5118	0.3868	0.5070	0.5093	0.5018	0.5196	0.3216	0.9839	0.8544	0.4637	0.6362	0.6501	0.4296	0.9647	0.8388	0.3784	0.7603	0.4489	0.3394	0.9861	0.8740	0.4541	0.6635	0.6138	0.3336	0.9791	0.8407	0.4490	0.6780	0.6274	0.3291	0.7498	0.7045	0.2593	0.9952	0.9100	0.4901	0.5999	0.7082	0.3636	0.9815	0.8631	0.4323	0.7074	0.5971	0.2694	0.2385	0.9970	0.8671	0.9293	0.4888	0.4212	0.4683	0.8090	0.2551	0.9955	0.9118	0.4918	0.5575	0.7706
2024-07-30	Mask2Former-SwinB	0.3772	0.2767	0.5923	0.2328	0.3480	0.2851	0.2791	0.6445	0.2348	0.4112	0.2926	0.2667	0.0958	0.9085	0.4167	0.5104	0.8800	0.6882	0.2012	0.8517	0.5119	0.5326	0.8024	0.5150	0.1205	0.9120	0.4577	0.5860	0.8209	0.6820	0.1420	0.8663	0.4490	0.5491	0.8641	0.6702	0.8171	0.9173	0.5728	0.0869	0.9244	0.4354	0.5540	0.8569	0.6948	0.2194	0.9158	0.5694	0.5423	0.7133	0.6601	0.0861	0.6931	0.9301	0.9206	0.4444	0.5553	0.6605	0.8523	0.7542	0.1056	0.9195	0.4755	0.5779	0.8352	0.7133
2024-08-24	Physically Feasible Semantic Segmentation	0.3364	0.6630	0.2253	0.6486	0.6092	0.6610	0.6254	0.2090	0.6851	0.6813	0.4311	0.7096	0.3930	0.9860	0.8769	0.0473	0.5154	0.6949	0.4952	0.9719	0.8664	0.1018	0.6132	0.5136	0.4072	0.9888	0.8896	0.0423	0.4933	0.6813	0.4195	0.9758	0.8503	0.0679	0.5750	0.6317	0.4226	0.8139	0.9070	0.3981	0.9936	0.9120	0.0235	0.4032	0.6930	0.4353	0.9913	0.9099	0.0444	0.4833	0.7043	0.4009	0.1148	0.9941	0.8564	0.9167	0.0278	0.5831	0.3907	0.7449	0.4278	0.9951	0.9290	0.0268	0.3903	0.7306

Autonomous Driving

Inactive evaluations

method: DINOv2, ViT-L, 8x8 patch size, linear decoder2024-08-23

method: DINOv2, ViT-G, 16x16 patch size, linear decoder2024-08-23

method: DINOv2, ViT-B, 16x16 patch size, linear decoder2024-08-23

Ranking Table

Ranking Graphic - BRAVO Index

Ranking Graphic - Semantic H-Mean

Ranking Graphic - OOD H-Mean