• Problem identification: The model engineer identifies a problem to be solved with FL.
• Simulation prototyping (optional): The model engineer may prototype model architectures and test

  hyperparameters in an FL simulation using a proxy dataset or in real setting.
• Federated model training: Multiple federated training tasks are started to train different variations of the model, or use different optimization hyperparameters.
• (Federated) model evaluation: After the tasks have trained sufficiently, the models need to be analyzed. This analysis may include metrics computed on standard datasets in the datacenter, or federated evaluation wherein the models are pushed to held-out clients for evaluation on local client data.
• Deployment: Finally, once a good model is selected, it goes through a standard model launch process.

We term our approach Federated Learning, since the learning task is solved by a loose federation of participating devices which are coordinated by a central server.

• Selection: The server(i.e. the coordinator in the server) picks a subset of available devices to work on a specific FL task[1].
• Configuration: The server sends the FL plan(execution plan) and the FL checkpoint(i.e. a serialized state of a Tensorflow session) with the global model to each of the chosen devices. When receiving a FL task, the FL runtime will be responsible for performing local training.
• Reporting: The server waits for the participating devices to report updates. The round is considered successful if enough devices report in time. (The update to the model is often sent to the server using encrypted communication.)

• Coordinators are Top-level actors(one per population) which enable global synchronization and advancing rounds in lockstep. As previously mentioned, The Coordinator receives information about how many devices are connected to each Selector and instructs them how many devices to accept for participation, based on which FL tasks are scheduled.
• Selectors are responsible for accepting and forwarding device connections. After the Master Aggregator and set of Aggregators are spawned, the Coordinator instructs the Selectors to forward a subset of its connected devices to the Aggregators, allowing the Coordinator to efficiently allocate devices to FL tasks regardless of how many devices are available
• Master Aggregators manage the rounds of each FL task. In order to scale with the number of devices and update size, they make dynamic decisions to spawn one or more Aggregators to which work is delegated.

1. 1.

   Workers pull the latest model from the server
2. 2.

   Workers compute an update l based on their local data
3. 3.

   Workers send their local update to the server
4. 4.

   The server aggregates these updates (by averaging) to construct the new global model

1. 1.

   In data center setting, shared storage is usually used, which means the worker machine do not keep persistent data storage on their own, and they fetch data from the shared storage at the beginning of each iteration.
2. 2.

   In FL, the data, and thus the loss function, on the different clients may be very heterogeneous, and far from being representative of the joint data.(e.g. the data stored on each client may be highly non-IID)
3. 3.

   In FL, the server never keeps track of any individual client information and only uses aggregates to ensure privacy.
4. 4.

   Because of the high churn in FL setting, only a small subset of the devices are selected by the server in each round.

• Risk modeling
• Smart Devices

  • on-device item ranking, content suggestions for on-device keyboards, and next word prediction​
  • Detecting burglaries within smart homes
• Healthcare

  • predicting health events like low blood sugar or heart attack risk from wearable devices
• Browsers

  • ​Federated Learning for Firefox​

• On-device data is more relevant than server-side proxy data
• On-device data is privacy sensitive or large
• Labels can be inferred naturally from user interaction

• Highly efficient use of network bandwidth

  • Less information is required to be transmitted to the cloud.
• Privacy

  • As described above, the raw data of users need not be sent to the cloud.
  • With guaranteed privacy, more users will be willing to take part in collaborative model training and so, better inference models are built.
• Low latency

  • The latency is much lower than that when decisions are made in the cloud before transmitting them to the end devices. This is vital for time critical applications such as self-driving car systems in which the slightest delays can potentially be life threatening

• Does it work? And if so, why?

  • We can prove FL works for linear models and a couple of other special cases, but we cannot prove it works for more complicated things like neural networks unless we train the model in a non-federated way and demonstrate that it gets almost the same performance.
• Security

  • Recent study shows that a malicious participant may exist in FL and can infer the information of other participants from shared parameters. As such, privacy and security issues in FL need to be considered.
• Statistical and System heterogeneity

  • In a large and complex mobile edge network, the heterogeneity of participating devices in terms of data quality, computation power, and willingness to participate have to be well managed from the resource allocation perspective.
  • Problem of Non-IID data: In datacenters, workers usually use a shared storage. The worker machine do not keep persistent data storage on their own, and they fetch data from the shared storage at the beginning of the learning process(or each iteration). As a result, it's possible to guarantee that the data samples obtained by different workers are IID. In Federated Learning, we cannot make iid assumptions. ; that is, a device’s local data cannot be regarded as samples drawn from the overall distribution.
  • See next post for more detail.
• Slow, unstable and limited communication:

  • Due to the high dimensionality of model updates and limited communication bandwidth of participating mobile devices, communication costs remain an issue.
  • If we use mobile phone for Federated Learning, uploads are typically going to be much slower than downloads and network latency may be very slow as well. Even worse, the FL system does not have control over users’ devices. For example, when a mobile phone is turned off or WiFi access is unavailable, the central server will lose connection to this device.
  • One potential mitigation is that we can select N devices in each round and proceed with K response(

    $K\le N$

    ).
• May not work for tree-based algorithms(e.g. decision trees)

• ​Tensorflow Federated Github Repo ​
• ​Google's blog on Federated Learning(A great introduction)
• ​Pysyft - a Python library for secure, private Deep Learning
• ​Federated Learning for Firefox​
• ​Damien Desfontaines's blog post on differential privacy​
• ​Frank McSherry's blog on deep learning and differential privacy​
• ​IID Statistics: Independent and Identically Distributed Definition and Examples​