Authors: Tommie Kerssies, Daan de Geus, and Gijs Dubbelman

Affiliation: Eindhoven University of Technology

Email: t.kerssies@tue.nl

Description: Fine-tuning for ~40 epochs on Cityscapes, following the setup described in: "How to Benchmark Vision Foundation Models for Semantic Segmentation?" (https://www.tue-mps.org/benchmark-vfm-ss/)

Authors: Tommie Kerssies, Daan de Geus, and Gijs Dubbelman

Affiliation: Eindhoven University of Technology

Email: t.kerssies@tue.nl

Description: Fine-tuning for ~40 epochs on Cityscapes, following the setup described in: "How to Benchmark Vision Foundation Models for Semantic Segmentation?" (https://www.tue-mps.org/benchmark-vfm-ss/)

Authors: Tommie Kerssies, Daan de Geus, and Gijs Dubbelman

Affiliation: Eindhoven University of Technology

Email: t.kerssies@tue.nl

Description: Fine-tuning for ~40 epochs on Cityscapes, following the setup described in: "How to Benchmark Vision Foundation Models for Semantic Segmentation?" (https://www.tue-mps.org/benchmark-vfm-ss/)

Ranking Table

Description Paper Source Code
BRAVO IndexSubset H-MeansACDCfogACDCnightACDCrainACDCsnowSMIYCoutofcontextsynflaresynobjssynrain
DateMethodbravosemanticoodACDCfogACDCnightACDCrainACDCsnowSMIYCoutofcontextsynflaresynobjssynrainauprc_errauprc_sucaurocecefpr95miouauprc_errauprc_sucaurocecefpr95miouauprc_errauprc_sucaurocecefpr95miouauprc_errauprc_sucaurocecefpr95miouauprc_oodauroc_oodfpr95_oodauprc_errauprc_sucaurocecefpr95miouauprc_errauprc_sucaurocecefpr95miouauprc_errauprc_oodauprc_sucauroc_oodaurocecefpr95_oodfpr95miouauprc_errauprc_sucaurocecefpr95miou
2024-08-23DINOv2, ViT-L, 8x8 patch size, linear decoder0.77890.69810.88070.66030.68080.67480.67560.89910.71040.72680.76660.73930.35850.98860.86190.02470.44510.77020.44150.99110.91020.04060.47670.67240.35800.99390.90050.01900.40970.78800.36860.99100.88390.02340.42420.78910.89010.97270.15630.41120.99620.93300.01190.35650.72880.42270.99660.93700.01570.35070.79160.41270.74420.99660.98040.93480.01350.10100.33280.79650.42340.99760.94820.01130.30570.8030
2024-08-23DINOv2, ViT-G, 16x16 patch size, linear decoder0.76120.70010.83400.66600.67860.67660.68710.88180.71000.73160.74690.73170.34920.99210.88770.01940.41700.78450.40460.99290.91410.03260.45290.71020.35170.99490.91180.01740.39600.80020.37340.99460.91530.02120.40930.80520.86490.96850.17570.41570.99580.93190.01220.36220.72070.42060.99670.94040.01450.32850.79800.41850.62120.99640.96910.93710.01490.13100.33380.78030.40730.99760.94740.01080.30550.8087
2024-08-23DINOv2, ViT-B, 16x16 patch size, linear decoder0.75540.70460.81420.69030.66500.70690.67950.87900.71160.72770.74040.73020.40630.99310.90750.01940.40710.70660.45960.98810.90210.04550.52330.63250.41400.99590.93170.01730.38180.73550.40580.99070.89800.02670.46150.73650.84400.96480.16070.44510.99550.93370.01210.38750.68290.45730.99550.93510.01750.37670.73390.43950.56890.99610.96710.93650.01310.14140.34270.75850.43820.99680.94300.01250.34280.7480
2024-08-23PixOOD YOLO (="Model Selection")0.67790.57060.83490.56880.48950.61990.55140.74870.64870.64960.56380.63280.69050.89300.79870.28520.53930.31660.70600.90990.84100.08870.62670.20940.68000.92610.82990.20180.50790.37900.80140.83650.82630.17960.55980.26760.72380.93320.35630.44860.98480.85860.09000.51170.59080.57390.97190.88480.12490.50640.46180.19980.85570.98260.99690.78630.09910.00810.66730.72850.48570.97530.85110.13860.54150.5315
2024-08-23DeiT III (IN21K->IN1K), ViT-B, 16x16 patch size, linear decoder0.66650.66490.66820.62560.61200.67010.63830.68200.69250.67960.69540.69850.46900.97550.85970.01820.58560.54360.52620.97590.88070.06330.61130.47540.48450.98790.90210.01900.51100.58640.47890.97910.87190.01340.57780.56790.55770.90140.33170.48090.99310.92460.01330.44620.59600.51920.98850.91360.03460.52200.59930.46920.42420.99510.96020.93470.01090.15390.39900.67580.46220.99410.92830.01100.43230.6368
2024-08-23DINOv2, ViT-G, 16x16 patch size, Mask2Former decoder0.64540.49680.92080.51430.61340.40620.43030.94370.40900.53890.64320.60070.20750.98770.82300.03890.57070.79960.34910.99310.90930.05450.57300.70880.17100.99300.86480.06420.77590.81380.18620.99070.85460.06570.75610.81600.91460.98490.06560.19050.99100.86740.06430.79220.72570.23580.99440.90130.04400.59680.79890.25050.76770.99360.99090.89810.04080.02510.55080.78500.26810.99550.91450.03090.47230.8099
2024-08-23Model selection0.63490.69390.58520.68820.61030.70660.65980.70710.73350.77650.58870.76390.41270.99110.89720.03850.44130.74530.50330.97200.86760.10500.63070.54380.48140.99020.91150.04990.44580.68970.43750.98380.87920.05830.53090.68490.79810.91370.47240.39610.99830.95470.01820.24100.78740.54830.99560.94640.03970.32150.75390.36460.25640.99630.96110.92550.41210.06540.36820.85770.43950.99850.96040.01820.23310.8237
2024-08-22PixOOD w/ ResNet-101 DeepLab0.61190.58660.63950.56880.48950.61990.55140.53180.64870.64960.63200.63280.69050.89300.79870.28520.53930.31660.70600.90990.84100.08870.62670.20940.68000.92610.82990.20180.50790.37900.80140.83650.82630.17960.55980.26760.35510.84300.38990.44860.98480.85860.09000.51170.59080.57390.97190.88480.12490.50640.46180.29630.58690.98490.99010.82680.10270.02630.57480.69720.48570.97530.85110.13860.54150.5315
2024-08-22PixOOD w/ DeepLab Decoder0.59370.46060.83490.38290.55830.48940.50710.74870.56150.43620.56380.36400.16700.96160.66590.22120.77590.71670.34610.96840.79370.09640.64620.61260.22920.97770.77350.15500.66070.68980.28010.96550.74310.19070.68930.70160.72380.93320.35630.30370.98350.82000.07780.61000.66300.20150.97880.77380.13400.74530.71610.19980.85570.98260.99690.78630.09910.00810.66730.72850.15240.97250.72360.12620.79870.7145
2024-08-20PixOOD0.53470.40380.79090.25880.52130.42910.42350.66700.49970.48370.49430.36400.12450.94080.61130.29610.89440.63890.37340.95190.78200.11050.71930.55110.21450.96850.75800.21770.75890.64800.23230.95190.71570.29650.77300.62930.73320.90570.50740.25980.97770.78800.09290.68970.64830.26110.97090.76450.14780.72010.67110.17230.92610.97450.99760.73870.12760.00640.77100.70380.15740.96530.69030.12260.80260.6938
2024-08-24Physically Feasible Semantic Segmentation0.33640.66300.22530.64860.60920.66100.62540.20900.68510.68130.43110.70960.39300.98600.87690.04730.51540.69490.49520.97190.86640.10180.61320.51360.40720.98880.88960.04230.49330.68130.41950.97580.85030.06790.57500.63170.42260.81390.90700.39810.99360.91200.02350.40320.69300.43530.99130.90990.04440.48330.70430.40090.11480.99410.85640.91670.02780.58310.39070.74490.42780.99510.92900.02680.39030.7306

Ranking Graphic - BRAVO Index

Ranking Graphic - Semantic H-Mean

Ranking Graphic - OOD H-Mean