method: DP-CLGECL-momentum2024-03-27
Authors: Takumi Fukami, Yusuke Yamasaki, Iifan Tyou , Kenta Niwa
Affiliation: NTT
Description: We propose DP-CLGECL-momentum expecting higher evaluation scores (ANLS and ACC) while complying with a privacy budget of eps = 1, 4, and 8.
We studied CLGECL [1] for federated learning. The update rule of CLGECL is formulated based on primal-dual formalism to solve constrained loss minimization problems. Although the details are noted in [1], CLGECL is associated with well-known SCAFFOLD [2]; namely, local stochastic gradient is modified using additional variance reduction terms to avoid client drifts. We think this term is also beneficial for reducing bias due to Gaussian noise addition due to DP. Moreover, we adopted SGD with momentum as a local update rule, instead of vanilla SGD. SGD with momentum utilizes the moment term, which accumulates the gradients over time. The moving average of the computed gradient serves to mitigate the variance in stochastic gradients across mini-batches. As a result, gradient clipping occurred less frequently compared to the instances observed with vanilla SGD.
Although our DP analysis is shown in the supplementary material, we followed DP analysis technique, which is shown in this competition cite.
Hyperparameters are experimentally selected as the number of clients K=2, the sensitivity of updates c = 0.5, the number of communication rounds R=14, and learning rate lr = 2e-4 since averaged ANLS score was then the highest. In our evaluation using the given validation dataset, averaged ANLS was 0.6156 and ACC was 0.5378.
eps=1 ANLS: 0.5981 Acc: 0.5155
eps=4 ANLS: 0.6206 Acc: 0.5450
eps=8 ANLS: 0.6282 Acc: 0.5530
[1] I. Tyou, T. Murata, T. Fukami, Y. Takezawa, and K. Niwa,
"A Localized Primal-Dual Method for Centralized/Decentralized Federated Learning Robust to Data Heterogeneity,"IEEE Trans., 2023.
[2] S. P . Karimireddy, et al. "SCAFFOLD: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.
method: Differentially Private Federated Learning with LoRA2023-10-25
Authors: Ragul N, Rintu Kutum
Affiliation: Department of Computer Science, Ashoka University
Email: ragul.n_asp24@ashoka.edu.in; rintu.kutum@ashoka.edu.in
Description: We used LoRA to finetune the model, which reduced the communication cost per round and total noise added to the model. The reduced communication cost per round allowed us to increase server rounds.
method: DP-CLGECL2023-10-27
Authors: Takumi Fukami, Yusuke Yamasaki, Iifan Tyou , Kenta Niwa
Affiliation: NTT
Description: We propose DP-CLGECL expecting higher evaluation scores (ANLS and ACC) while complying with a privacy budget of eps = 1, 4, and 8.
We studied CLGECL [1] for federated learning. The update rule of CLGECL is formulated based on primal-dual formalism to solve constrained loss minimization problems. Although the details are noted in [1], CLGECL is associated with well-known SCAFFOLD [2]; namely, local stochastic gradient is modified using additional variance reduction terms to avoid client drifts. We think this term is also beneficial for reducing bias due to Gaussian noise addition due to DP.
For this competition, we newly propose DP-CLGECL, which installed standard DP into CLGECL. Although our DP analysis is shown in the supplementary material, we followed DP analysis technique, which is shown in this competition cite.
Hyperparameters are experimentally tuned by fixing the number of clients K=2, the sensitivity of updates c = 0.5, and learning rate lr = 2e-4. Under this hyperparameter setting, we investigated evaluation metrics (ANLS and ACC) by varying the number of communication rounds from R=12 to 15. Then, we determined the noise multiplier value for each combination of R and privacy budget of eps = 1, 4, and 8. We selected R=14 since averaged ANLS score was then the highest. In our evaluation using the given validation dataset, averaged ANLS was 0.5698 and ACC was 0.4898.
eps=1 ANLS: 0.5487 Acc: 0.4678
eps=4 ANLS: 0.5778 Acc: 0.4976
eps=8 ANLS: 0.5829 Acc: 0.5042
[1] I. Tyou, T. Murata, T. Fukami, Y. Takezawa, and K. Niwa,
"A Localized Primal-Dual Method for Centralized/Decentralized Federated Learning Robust to Data Heterogeneity,"
IEEE Transaction on Signal and Information Processing Over Networks, (in press).
[2] S. P . Karimireddy, et al. "SCAFFOLD: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.
| Avg. Question-Answering | ε1 | ε4 | ε8 | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Date | Method | ANLS | Accuracy | ANLS | Accuracy | Total GB | FL Rounds | ANLS | Accuracy | Total GB | FL Rounds | ANLS | Accuracy | Total GB | FL Rounds | Privacy Proof Verified | |||
| 2024-03-27 | DP-CLGECL-momentum | 0.6411 | 57.0347 | 0.6242 | 0.5495 | 62.5185 | 14 | 0.6453 | 0.5761 | 62.5185 | 14 | 0.6537 | 0.5854 | 62.5185 | 14 | Yes | |||
| 2023-10-25 | Differentially Private Federated Learning with LoRA | 0.6066 | 53.5952 | 0.5854 | 0.5144 | 21.9521 | 30 | 0.6121 | 0.5418 | 22.3242 | 30 | 0.6225 | 0.5517 | 22.3242 | 30 | Yes | |||
| 2023-10-27 | DP-CLGECL | 0.5925 | 51.9113 | 0.5724 | 0.4989 | 62.5185 | 14 | 0.6018 | 0.5274 | 62.5185 | 14 | 0.6033 | 0.5311 | 62.5185 | 14 | Yes | |||
| 2023-10-24 | (Baseline) FedAvg + DP Baseline | 0.4996 | 43.3637 | 0.4832 | 0.4155 | 22.3281 | 5 | 0.5024 | 0.4368 | 22.3281 | 5 | 0.5132 | 0.4486 | 22.3281 | 5 | Yes | |||