Interprocess Communication and Parallelism in C

Building a Weather Application Using IPC and Parallelism

Building a weather application using interprocess communication (IPC) like named pipes and shared memory. In addition to parallelism through posix threads to get weather data from a web API.

Interprocess Communication is when two processes or more communicate with each other to share data or information to be able to wrok concurrently. While parallelism is when the same process gets more resources, like more CPUs, to better perform the task.

Our Application will consist of 3 processes, as shown in the blog image. The processes will communicate with each other using different ways to get and view weather data obtained from

  1. reader.c: reads the input file and puts the city names into the named pipe for the worker process
  2. worker.c: reads data from the named pipe, parallelize the api requests using pthreads and writes the results into the shared memory object for the viewer process
  3. viewer.c: reads the results from the shared memory object, writes it into a file and filter it then view it.

The aim is to demonstrate some of C functionalities in doing tasks in parallel and/or concurrently, not to give a proper tutorial on these topics. For a good resource on these topics, I recommend this great guide on IPC.

The GitHub link for the repo and files:

Let’s get into work!

N.B. Comments in the code are very important!

Header File

Let’s first look at how the header file, “weather.h”, which has the variables the processes will use.

#include <errno.h>
#include <stdlib.h>
// number of cities in data
#define NUM_CITIES 15
// length of strings
#define STRLEN 256
// how many bytes the shared memory object is
#define SIZE 2048
// name of shared object
#define SHM_OBJ "weather_data"
// named pipe (FIFO)
#define CITY_FIFO "cityfifo"
// construct first command with the city
#define CMD1 "curl -s '"
// append api key and extract only needed info and convert to csv
#define CMD2 "&appid=<your-api-key>' | jq -r '{name: .name, temperature: .main.temp, desc: .weather[].description} | [.name, .temperature, .desc] | @csv'"
// filter only cities which have clear sky and some clouds & the degree is below or equal to 25 celsius and add a header then pretty print them
#define CMD3 "< ./weather_data.txt grep -E 'clear sky|clouds' | awk -F, '{if($2-273.15 <= 25) {print $1\",\"$2-273.15\",\"$3}}' | sed '1i city,temperature,description' | csvlook"

// error handling
void error(char * msg) {
  printf("%s: %s\n", msg, strerror(errno));

The code is straightforward. But let’s look at the commands, CMD1-3. In the code, we will use some command line tools to process data, instead of writing pure C code.

For a full example on using the command line to process data, you can have a look at this blog

In CMD1 macro, we simply use the curl command to submit the request to the api. While CMD2, second part of the first command, it appends the API Key you get when you sign into the website, then uses jq command to extract only the needed info from the response and converts the result into csv format.

On the other hand, CMD3, which is a command on its own, it reads from the file, where results will be stored, filters only lines with “clear sky” or “clouds” using grep, then uses awk to calculate the celsius degree from second column and filter rows with values <= 25. Then it uses sed to add a header. Finally, csvlook comes to prettify the output.

Now, let’s go to the processes!

Reader Process

The “reader.c” file looks like that

#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include "weather.h"

int main(int argc, char * argv[]) {

  if (argc != 2) {
    printf("Error: You need at least 1 arguemnt which is cities file ...\n");
    return 1;

  // temporary variables
  char city[STRLEN / 4], cities[NUM_CITIES][STRLEN / 4];
  int fd, i = 0;
  // reading the cities from file and storing in cities array
  FILE * input = fopen(argv[1], "r");
  while (fgets(city, sizeof(city), input)) {
    // removing the new line character that may come from fgets
    city[strcspn(city, "\n")] = 0;
    sprintf(cities[i], "%s", city);
    // tracking how many cities we have
  fclose(input); // closing the file
  // opening the pipe to send the cities to worker process
  // mkfifo(fifo_name, permission) 
  mkfifo(CITY_FIFO, 0666);
  // FIFO with write only 
  fd = open(CITY_FIFO, O_WRONLY);
  // writing them in backwards
  while (i--> 0) {
    write(fd, cities[i], strlen(cities[i]) + 1);
  // closing FIFO
  return 0;

This process does pretty simply task. It needs an input file to start, it reads it, opens a named pipe (FIFO) called CITY_FIFO, lastly it writes the input city names in it. Comments are explaining everything so no need to repeat.

Here is how our input file looks like with some example cities

Rio De Janeiro
New York City
Cape Town

Worker Process

The process which performs tha main task is the “worker.c“. The code looks as follows:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <pthread.h>
#include <errno.h>
#include <sys/shm.h>
#include <sys/mman.h>
#include "weather.h"

void * request(void * inp) {
  // the final command to be called
  char cmd[STRLEN];
  // void * into char *
  char * city = inp;
  char * res = (char * ) malloc(STRLEN * sizeof(char));
  // construct the command
  sprintf(cmd, CMD1, city);
  strcat(cmd, CMD2);
  // save the command output into a variable
  FILE * fp;
  fp = popen(cmd, "r");
  // error handling
  if (fp == NULL) {
    printf("Failed to run command\n");
  // read command output including spaces and commas
  // until new line
  fscanf(fp, "%[^\n]s", res);
  // return the pointer to res as void *
  return (void * ) res;

int main() {
  // temporary variables
  char cities[NUM_CITIES][STRLEN / 4];
  char city[STRLEN / 4];
  int fd, i = 0, ln;
  // to collect results from threads
  void * result;
  // open the pipe with read only
  mkfifo(CITY_FIFO, 0666); 
  fd = open(CITY_FIFO, O_RDONLY);
  // read data from the pipe
  while (read(fd, city, sizeof(city))) {
    if (i == 0)
      ln = 0;
      ln = 1;
    /* remove some unknown character at the end of
    cities names except for the first city
    it is coming from using fgets and the write to or/and
    read from named pip.
    I need to debug it more; but it works with this solution */
    strncpy(cities[i], city, strlen(city) - ln);

  // close pipe
  // create threads (backwards) to go in parallel hitting the api
  pthread_t threads[NUM_CITIES];
  while (i--> 0) {
    // create the threads and apply the request function
    if (pthread_create( & threads[i], NULL, request, (void * ) cities[i]) == -1) // passing arg pointer as void*
      error("Can't create thread\n");

  // shared memory object stuff to share results
  // shared memory file descriptor
  int shm_fd;
  // pointer to shared memory object
  char * shm_ptr, * msg;
  // creating shared memory object
  shm_fd = shm_open(SHM_OBJ, O_CREAT | O_RDWR, 0666);
  // adjusting size of shared memory object
  ftruncate(shm_fd, SIZE);
  // memory mapping the shared memory object
  shm_ptr = mmap(0, SIZE, PROT_WRITE, MAP_SHARED, shm_fd, 0);
  // error handling
  if (shm_ptr == MAP_FAILED)
    error("Can't map\n");

  // collecting results from threads and write into shared mem obj
  while (++i < NUM_CITIES) {
    if (pthread_join(threads[i], & result) == -1)
      error("Can't join thread\n");
    // void * to char *
    msg = result;
    // print the message into the shared mem obj ptr
    sprintf(shm_ptr, "%s\n", msg);
    /* extend the ptr length with city result length and
    and the new line character */ 
    shm_ptr += strlen(msg) + 1;
  return 0;

It is a big one isn’t it? That’s because it does almost everything. Let’s have a look at the important parts.

Again, comments are so important because I tried to explain every line as much as possible!

First, the request function that takes the city name as input, as void * pointer because this is the argument type to threads, and appends it to CMD1 and CMD2 to construct the proper command string. It then runs the system commands using popen function to collect the output and return it as a void * pointer again as it is the accepted type from threads.

N.B. Threads functions can accept and return any pointers but they will give warnings while compiled.

Then, the main function, which consists of several chunks. After declaring some variables, comes the chunk of the mkfifo(). It opens the named pipe with read only permissions and fills it content in cities array. Afterwards, the parallelism part, pthreads. A thread is created for each city in the array and calls the request function. Then, the process creates the shared memory object. Finally, another loop to collect the results from the threads and writes them into the shared memory object for the viewer process.

N.B. There are other ways for interprocess communication, unamed pipes and message passing for instance. For an example on how to use the unamed pipes as well as forking new processes and work concurrently, have a look at this blog

Viewer Process

The “viewer.c” process performs the last task.

#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/shm.h>
#include <sys/mman.h>
#include <string.h>
#include "weather.h"

int main() {
  // shared memory file descriptor
  int shm_fd;
  // pointer to shared memory object
  char * shm_ptr;
  // opening shared memory object with read only
  shm_fd = shm_open(SHM_OBJ, O_RDONLY, 0666);
  // memory mapping shared memory object
  shm_ptr = mmap(0, SIZE, PROT_READ, MAP_SHARED, shm_fd, 0);
  // unlink the shared memory object

  // writing shared memory object into a file
  FILE * out = fopen("./weather_data.txt", "w");
  fprintf(out, "%s", shm_ptr);
  // performing the third command to show filtered data
  char cmd[STRLEN];
  strcpy(cmd, CMD3);
  puts("\nCities with clear sky or clouds and degree <= 25:\n");
  return 0;

It opens the shared memory object and reads the information in it. Then it saves this data into disk in a file, “weather_data.txt“. Finally, it executes the last command which filters the data and prettily print it onto the console.

Compile & Run

Now, time to test everything. In three terminals, we compile and run the processes.

  1. Terminal A
    gcc reader.c -o read
    ./read cities.txt
  2. Terminal B
    gcc worker.c -o work -lpthread
  3. Terminal C
    gcc viewer.c -o view
    Cities with clear sky or clouds and degree <= 25:
    | city      | temperature | description      |
    | --------- | ----------- | ---------------- |
    | Amsterdam |        4.85 | few clouds       |
    | Berlin    |       -1.45 | broken clouds    |
    | Havana    |       23.00 | scattered clouds |
    | Cape Town |       23.17 | clear sky        |
    | Sydney    |       18.25 | clear sky        |
    | New York  |        6.03 | broken clouds    |
    | Tokyo     |        3.10 | few clouds       |
    | Madrid    |        4.43 | clear sky        |
    | Paris     |        4.25 | few clouds       |
    | Rome      |        8.05 | overcast clouds  |
    | Dubai     |       21.23 | clear sky        |
    | Cairo     |       16.56 | broken clouds    |


Great! Everything is working fine. Now let’s look at the full weather data file, “weather_data.txt“.

"Amsterdam",278,"few clouds"
"Berlin",271.7,"broken clouds"
"Havana",296.15,"scattered clouds"
"Cape Town",296.32,"clear sky"
"Bali",299.15,"light rain"
"Sydney",291.4,"clear sky"
"New York",279.18,"broken clouds"
"Bangkok",300.51,"broken clouds"
"Tokyo",276.25,"few clouds"
"Rio de Janeiro",308.6,"few clouds"
"Madrid",277.58,"clear sky"
"Paris",277.4,"few clouds"
"Rome",281.2,"overcast clouds"
"Dubai",294.38,"clear sky"
"Cairo",289.71,"broken clouds"


Leave a Reply

Your email address will not be published. Required fields are marked *