Building a Weather Application Using IPC and Parallelism
Building a weather application using interprocess communication (IPC) like named pipes and shared memory. In addition to parallelism through posix threads to get weather data from a web API.
Interprocess Communication is when two processes or more communicate with each other to share data or information to be able to wrok concurrently. While parallelism is when the same process gets more resources, like more CPUs, to better perform the task.
Our Application will consist of 3 processes, as shown in the blog image. The processes will communicate with each other using different ways to get and view weather data obtained from https://openweathermap.org/api
- reader.c: reads the input file and puts the city names into the named pipe for the worker process
- worker.c: reads data from the named pipe, parallelize the api requests using pthreads and writes the results into the shared memory object for the viewer process
- viewer.c: reads the results from the shared memory object, writes it into a file and filter it then view it.
The aim is to demonstrate some of C functionalities in doing tasks in parallel and/or concurrently, not to give a proper tutorial on these topics. For a good resource on these topics, I recommend this great guide on IPC.
The GitHub link for the repo and files: https://github.com/MNoorFawi/ipc-and-pthreads-in-c
Let’s get into work!
N.B. Comments in the code are very important!
Header File
Let’s first look at how the header file, “weather.h”, which has the variables the processes will use.
#include <errno.h>
#include <stdlib.h>
// number of cities in data
#define NUM_CITIES 15
// length of strings
#define STRLEN 256
// how many bytes the shared memory object is
#define SIZE 2048
// name of shared object
#define SHM_OBJ "weather_data"
// named pipe (FIFO)
#define CITY_FIFO "cityfifo"
// construct first command with the city
#define CMD1 "curl -s 'http://api.openweathermap.org/data/2.5/weather?q=%s"
// append api key and extract only needed info and convert to csv
#define CMD2 "&appid=<your-api-key>' | jq -r '{name: .name, temperature: .main.temp, desc: .weather[].description} | [.name, .temperature, .desc] | @csv'"
// filter only cities which have clear sky and some clouds & the degree is below or equal to 25 celsius and add a header then pretty print them
#define CMD3 "< ./weather_data.txt grep -E 'clear sky|clouds' | awk -F, '{if($2-273.15 <= 25) {print $1\",\"$2-273.15\",\"$3}}' | sed '1i city,temperature,description' | csvlook"
// error handling
void error(char * msg) {
printf("%s: %s\n", msg, strerror(errno));
exit(1);
}
The code is straightforward. But let’s look at the commands, CMD1-3. In the code, we will use some command line tools to process data, instead of writing pure C code.
For a full example on using the command line to process data, you can have a look at this blog
In CMD1 macro, we simply use the curl command to submit the request to the api. While CMD2, second part of the first command, it appends the API Key you get when you sign into the website, then uses jq command to extract only the needed info from the response and converts the result into csv format.
On the other hand, CMD3, which is a command on its own, it reads from the file, where results will be stored, filters only lines with “clear sky” or “clouds” using grep, then uses awk to calculate the celsius degree from second column and filter rows with values <= 25. Then it uses sed to add a header. Finally, csvlook comes to prettify the output.
Now, let’s go to the processes!
Reader Process
The “reader.c” file looks like that
#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include "weather.h"
int main(int argc, char * argv[]) {
if (argc != 2) {
printf("Error: You need at least 1 arguemnt which is cities file ...\n");
return 1;
}
// temporary variables
char city[STRLEN / 4], cities[NUM_CITIES][STRLEN / 4];
int fd, i = 0;
// reading the cities from file and storing in cities array
FILE * input = fopen(argv[1], "r");
while (fgets(city, sizeof(city), input)) {
// removing the new line character that may come from fgets
city[strcspn(city, "\n")] = 0;
sprintf(cities[i], "%s", city);
// tracking how many cities we have
++i;
}
fclose(input); // closing the file
// opening the pipe to send the cities to worker process
// mkfifo(fifo_name, permission)
mkfifo(CITY_FIFO, 0666);
// FIFO with write only
fd = open(CITY_FIFO, O_WRONLY);
// writing them in backwards
while (i--> 0) {
write(fd, cities[i], strlen(cities[i]) + 1);
}
// closing FIFO
close(fd);
return 0;
}
This process does pretty simply task. It needs an input file to start, it reads it, opens a named pipe (FIFO) called CITY_FIFO, lastly it writes the input city names in it. Comments are explaining everything so no need to repeat.
Here is how our input file looks like with some example cities
Cairo Dubai Rome Paris Madrid Rio De Janeiro Tokyo Bangkok New York City Sydney Bali Cape Town Havana Berlin Amsterdam
Worker Process
The process which performs tha main task is the “worker.c“. The code looks as follows:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <pthread.h>
#include <errno.h>
#include <sys/shm.h>
#include <sys/mman.h>
#include "weather.h"
void * request(void * inp) {
// the final command to be called
char cmd[STRLEN];
// void * into char *
char * city = inp;
char * res = (char * ) malloc(STRLEN * sizeof(char));
// construct the command
sprintf(cmd, CMD1, city);
strcat(cmd, CMD2);
// save the command output into a variable
FILE * fp;
fp = popen(cmd, "r");
// error handling
if (fp == NULL) {
printf("Failed to run command\n");
exit(1);
}
// read command output including spaces and commas
// until new line
fscanf(fp, "%[^\n]s", res);
pclose(fp);
// return the pointer to res as void *
return (void * ) res;
}
int main() {
// temporary variables
char cities[NUM_CITIES][STRLEN / 4];
char city[STRLEN / 4];
int fd, i = 0, ln;
// to collect results from threads
void * result;
// open the pipe with read only
mkfifo(CITY_FIFO, 0666);
fd = open(CITY_FIFO, O_RDONLY);
// read data from the pipe
while (read(fd, city, sizeof(city))) {
if (i == 0)
ln = 0;
else
ln = 1;
/* remove some unknown character at the end of
cities names except for the first city
it is coming from using fgets and the write to or/and
read from named pip.
I need to debug it more; but it works with this solution */
strncpy(cities[i], city, strlen(city) - ln);
i++;
}
// close pipe
close(fd);
// create threads (backwards) to go in parallel hitting the api
pthread_t threads[NUM_CITIES];
while (i--> 0) {
// create the threads and apply the request function
if (pthread_create( & threads[i], NULL, request, (void * ) cities[i]) == -1) // passing arg pointer as void*
error("Can't create thread\n");
}
// shared memory object stuff to share results
// shared memory file descriptor
int shm_fd;
// pointer to shared memory object
char * shm_ptr, * msg;
// creating shared memory object
shm_fd = shm_open(SHM_OBJ, O_CREAT | O_RDWR, 0666);
// adjusting size of shared memory object
ftruncate(shm_fd, SIZE);
// memory mapping the shared memory object
shm_ptr = mmap(0, SIZE, PROT_WRITE, MAP_SHARED, shm_fd, 0);
// error handling
if (shm_ptr == MAP_FAILED)
error("Can't map\n");
// collecting results from threads and write into shared mem obj
while (++i < NUM_CITIES) {
if (pthread_join(threads[i], & result) == -1)
error("Can't join thread\n");
// void * to char *
msg = result;
// print the message into the shared mem obj ptr
sprintf(shm_ptr, "%s\n", msg);
/* extend the ptr length with city result length and
and the new line character */
shm_ptr += strlen(msg) + 1;
}
return 0;
}
It is a big one isn’t it? That’s because it does almost everything. Let’s have a look at the important parts.
Again, comments are so important because I tried to explain every line as much as possible!
First, the request function that takes the city name as input, as void * pointer because this is the argument type to threads, and appends it to CMD1 and CMD2 to construct the proper command string. It then runs the system commands using popen function to collect the output and return it as a void * pointer again as it is the accepted type from threads.
N.B. Threads functions can accept and return any pointers but they will give warnings while compiled.
Then, the main function, which consists of several chunks. After declaring some variables, comes the chunk of the mkfifo(). It opens the named pipe with read only permissions and fills it content in cities array. Afterwards, the parallelism part, pthreads. A thread is created for each city in the array and calls the request function. Then, the process creates the shared memory object. Finally, another loop to collect the results from the threads and writes them into the shared memory object for the viewer process.
N.B. There are other ways for interprocess communication, unamed pipes and message passing for instance. For an example on how to use the unamed pipes as well as forking new processes and work concurrently, have a look at this blog
Viewer Process
The “viewer.c” process performs the last task.
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/shm.h>
#include <sys/mman.h>
#include <string.h>
#include "weather.h"
int main() {
// shared memory file descriptor
int shm_fd;
// pointer to shared memory object
char * shm_ptr;
// opening shared memory object with read only
shm_fd = shm_open(SHM_OBJ, O_RDONLY, 0666);
// memory mapping shared memory object
shm_ptr = mmap(0, SIZE, PROT_READ, MAP_SHARED, shm_fd, 0);
// unlink the shared memory object
shm_unlink(SHM_OBJ);
// writing shared memory object into a file
FILE * out = fopen("./weather_data.txt", "w");
fprintf(out, "%s", shm_ptr);
fclose(out);
// performing the third command to show filtered data
char cmd[STRLEN];
strcpy(cmd, CMD3);
puts("\nCities with clear sky or clouds and degree <= 25:\n");
system(cmd);
return 0;
}
It opens the shared memory object and reads the information in it. Then it saves this data into disk in a file, “weather_data.txt“. Finally, it executes the last command which filters the data and prettily print it onto the console.
Compile & Run
Now, time to test everything. In three terminals, we compile and run the processes.
- Terminal A
gcc reader.c -o read ./read cities.txt
- Terminal B
gcc worker.c -o work -lpthread ./work
- Terminal C
gcc viewer.c -o view ./view Cities with clear sky or clouds and degree <= 25: | city | temperature | description | | --------- | ----------- | ---------------- | | Amsterdam | 4.85 | few clouds | | Berlin | -1.45 | broken clouds | | Havana | 23.00 | scattered clouds | | Cape Town | 23.17 | clear sky | | Sydney | 18.25 | clear sky | | New York | 6.03 | broken clouds | | Tokyo | 3.10 | few clouds | | Madrid | 4.43 | clear sky | | Paris | 4.25 | few clouds | | Rome | 8.05 | overcast clouds | | Dubai | 21.23 | clear sky | | Cairo | 16.56 | broken clouds |
Great! Everything is working fine. Now let’s look at the full weather data file, “weather_data.txt“.
"Amsterdam",278,"few clouds" "Berlin",271.7,"broken clouds" "Havana",296.15,"scattered clouds" "Cape Town",296.32,"clear sky" "Bali",299.15,"light rain" "Sydney",291.4,"clear sky" "New York",279.18,"broken clouds" "Bangkok",300.51,"broken clouds" "Tokyo",276.25,"few clouds" "Rio de Janeiro",308.6,"few clouds" "Madrid",277.58,"clear sky" "Paris",277.4,"few clouds" "Rome",281.2,"overcast clouds" "Dubai",294.38,"clear sky" "Cairo",289.71,"broken clouds"
Leave a Reply