A belief rule based flood risk assessment expert system using real time sensor data streaming

(1)

1

Lappeenranta University of Technology School of Engineering Science

Degree Program in Computer Science

Ahmed Afif Monrat

A BELIEF RULE BASED FLOOD RISK ASSESSMENT EXPERT SYSTEM USING REAL TIME SENSOR DATA STREAMING

Examiners: Professor Eric Rondeau Professor Jari Porras Professor Karl Andersson

Supervisors: Professor Mohammad Shahadat Hossain Professor Karl Andersson

(2)

2

Ahmed Afif Monrat

(3)

3

ABSTRACT

Lappeenranta University of Technology School of Engineering Science

Degree Program in Computer Science

Ahmed Afif Monrat

A BELIEF RULE BASED FLOOD RISK ASSESSMENT EXPERT SYSTEM USING REAL TIME SENSOR DATA STREAMING

Master’s Thesis - 2018

Examiners: Professor Eric Rondeau Professor Jari Porras Professor Karl Andersson

Keywords: Belief Rule Base, Flood risk assessment, Uncertainty, Expert systems, Sensor data streaming, Bigdata.

Among the various natural calamities, flood is considered one of the most catastrophic natural hazards, which has a significant impact on the socio-economic lifeline of a country. The Assessment of flood risks facilitates taking appropriate measures to reduce the consequences of flooding. The flood risk assessment requires Big data which are coming from different sources, such as sensors, social media, and organizations. However, these data sources contain various types of uncertainties because of the presence of incomplete and inaccurate information. This paper presents a Belief rule-based expert system (BRBES) which is developed in Big data platform to assess flood risk in real time. The system processes extremely large dataset by integrating BRBES with Apache Spark while a web-based interface has developed allowing the visualization of flood risk in real time. Since the integrated BRBES employs knowledge driven learning mechanism, it has been compared with other data-driven learning mechanisms to determine the reliability in assessing flood risk. Integrated BRBES produces reliable results comparing from the other data- driven approaches. Data for the expert system has been collected targeting different case study areas from Bangladesh to validate the integrated system.

(4)

4

ACKNOWLEDGEMENTS

This research work was undertaken at Lulea University of Technology (campus skelleftea) under the supervision of Professor Mohammad Shahadat Hossain and Professor Karl Andersson. It was financially supported by Erasmus Mundus Joint Master Degree in Pervasive Computing and Communications for Sustainable Development (PERCCOM) and European Commission.

I would like to express my gratitude towards my supervisors for their support, guidance and inspiration throughout this thesis work. I would also like to thank Raihan Ul Islam (Ph.D.

Licentiate at Lulea University of Technology), from whom I have learned a lot during the tenure of this work. He followed my technical progress and continuously helped me improve my research by providing constructive criticism and suggestions. They monitored the progress of my research by arranging a physical or virtual meeting (via Skype) within every two weeks throughout this fourth semester which has been of paramount importance and vital for this work. I have thoroughly enjoyed the whole process even though there were some moments of despair when things were not working for me due to lack of research experience and knowledge. However, I have triumphed those feelings by working hard and published two paper: ”Challenges and Opportunities of Using Big Data for Assessing Flood Risks” in Springer (Applications of Big Data Analytics) and ” A Belief Rule Based Flood Risk Assessment Expert System using Real Time Sensor Data Streaming”

in IEEE Conference on Local Computer Networks Workshops (LCN), which was a part of this research work.

Thanks to the PERCCOM Consortium for giving me the opportunity to participate in this stupefying program. I would also like to thank Professor Eric Rondeau, Jean-Philippe Georges, Jari Porras, Josef Hallberg, Oleg Sadov, Ah-Lian Kor and Colin Pattinson along with all the supporting staff members from UL, LUT and LTU. I don’t want to waste any word praising my friends from cohort 4, as without them, I strongly believe, I would never be able to pull it off.

Finally, I would like to thank my family for being supportive all the time. This two years in PERCCOM, I had to go through so many hurdles, but my family was always there. Whenever it felt like I won’t be able to survive here, they motivated me not to give up.

Skelleftea, September 14, 2018

Ahmed Afif Monrat

(5)

5

LIST OF SYMBOLS AND ABBREVIATIONS

API Application Programming Interface AUC Area Under Curve

ANN Artificial Neural Networks BRBES Belief Rule Base Expert System BWDB Bangladesh Water Development Board DAG Directed Acyclic Graph

DSR Design Science Research ER Evidential Reasoning FRA Flood Risk Assessment GUI Graphical User Interface

HDFS Hadoop Distributed File System HUD Housing and Urban Development IS Information System

MSL Mean Sea Level

NFIP National Flood Insurance Program RDD Resilient Distributed Datasets REST Representational State Transfer RFID Radio Frequency Identification RMSE Root Mean Square Error

ROC Receiver Operating Characteristics

SC Spark Context

SDG SQL data generator UAT User Acceptance Testing WSN Wireless Sensor Networks

(All symbols and abbreviations are listed on this page in alphabetical order)

(9)

9

1 INTRODUCTION

This chapter presents the context of this research work. It describes the background and facts of flooding, along with stating the motivation and sustainability aspects of this work.

It also highlights the research objectives, challenges and contributions made by the author.

The limitations and scopes of this thesis are also discussed. Finally the chapter concludes with an overall outline for the whole content of this research.

1.1 Background

There are hardly few places on earth where people are not concerned about the disastrous impact of flooding. Although rain is not the only catalyst for flood, any place could become vulnerable where rain falls. Flood is one of the pervasive hazards with catastrophic consequences. It poses a greater threat to all aspects of sustainability including social, economic and environmental in comparison with other natural calamities such as earthquakes, landslides, volcanic eruptions and cyclones [1]. Apart from causing the loss of human lives, it also inundates residential and commercial properties harming local economies, causes disruption in national transport infrastructure, agriculture, electrical grid. Furthermore, severe water contamination and health hazards can occur due to flooding. For example, a huge tsunami hit Japan in 2011 and a part of the coastline was flooded by the sea water which caused massive leakage in nuclear plants that ended up contaminating waters with chemicals and hazardous substances [2].

Flood is considered responsible for killing more than 100,000 people and affected 1.4 billion people around the globe [3]. In the most recent decade of the twentieth century, the yearly cost to the world economy because of flooding is around 50-60 billion US dollars [4]. There were 157 major floods in Europe from 1971 to 1995. In 2016, more than 20 people were killed and the damage of property is estimated 1 billion euro approximately [6]. Among south asian countries, flooding occured in Bangladesh several occasions which caused major sufferings, especially in 1954, 1955, 1974, 1984, 1987, 1988 and 2004 [7].

The damages suffered by the developing nations are five times higher per unit of GDP than those of developed nations [8]. More than 1,200 people have died across India, Bangladesh

(10)

10

and Nepal and shut 1.8 million children out of school because of flooding in 2017 according to the reports of the Guardian [9].

Figure 1. Flood affected area of Chittagong, Bangladesh in 2017 [10]

Assessing the risk of flood is obligatory as it has a devastating impact on the socio- economic development of a country. It can save millions of lives by providing valuable insights for taking necessary steps during the flood to reduce the loss significantly. It is important to evaluate and monitor the factors regarding floods to develop a system that is capable of assessing the risk of the flood with the highest accuracy. Several factors such as topographical, meteorological, geological, river characteristics and human activities are responsible for flooding [11]. These factors can be categorized as quantitative and qualitative in nature. For example, while the financial loss can be determined in a quantitative way, qualitative measures can be taken for expressing health condition. As a result, different types of uncertainties can be noticed while assessing flood risk. The uncertainties can be categorized as vagueness, imprecision, ambiguity, ignorance, and incompleteness. In order to address this issue, a belief rule-based expert system (BRBES) can be employed which is developed in the big data analytics platform. This framework has the capability to address various types of uncertainties by processing large heterogeneous dataset regarding flood. Big data plays a vital role while assessing flood

(11)

11

risks due to its capability to visualize, analyze and predict the risks effectively. This integrated framework works as a single unit to assess the risks of the flood in an area.

We have already discussed different types of uncertainties that can be associated with sample dataset. Lack of human knowledge, insufficient data or sensor anomalies are mainly responsible for it. For this reason, it is required to consider uncertainty while designing an expert system. The main components of an expert system are knowledge base, inference engine and user interface. For representing uncertain knowledge, BRB expert system use belief rule base while Evidential Reasoning (ER) acts like an inference engine which has the capability of addressing both heterogeneous and uncertain data [12].

In this research, A belief rule-based expert system (BRBES) has been developed which has the capability of processing heterogeneous data with various types of uncertainties. It works like a single integrated framework to assess the consequences or risks of the flood in an area. In addition, to get better prediction and accuracy, BRBES has been integrated with a big data analytics platform because the traditional approach is not capable of handling voluminous dataset as well as performing complex mathematical computations. It also facilitates the widespread use of the integrated BRB framework for different domains.

Therefore, academia and companies can come forward with different innovative ideas, which will be capable of processing real time data such as disaster recovery, disease diagnosis, traffic management system. For this research, disaster prediction system like the assessment of flood risk is used as a use case.

1.2 Research Objectives and Questions

The main objective of this research is to develop a BRB expert system with the capability of accessing and processing large scale sensor data to predict and assess the risk of flooding in real time. It aims to the develop a new framework for flood risk assessment to assist the disaster management authorities and decision makers to assess different aspects of flood risk. For achieving the aims of this research, the following objectives are being identified:

1. Investigation of the existing flood risk assessment methodology

(12)

12

2. Critical review on various sensors from data acquisition and visualization perspectives.

3. Design and Implement BRB based inference algorithms and Big data based analytical methods

4. Development of the concrete prototype system by incorporating both real time data streaming and BRBES.

5. Development of optimal machine learning model to predict the risk of flooding in both the case study areas.

The following research questions have been raised to achieve the desired aim:

1. Which factors should be considered while predicting floods?

2. Why should we use BRB expert system to address this phenomena?

3. How to design and implement IoT and real time data streaming mechanisms with big data analytics platform?

4. What kinds of measurement techniques/parameters are available to compare the efficiency among the existing models?

1.3 Thesis Contribution

The contribution of this thesis is to develop a belief rule based expert system (BRBES) with big data analytics platform, which will have the capability of acquiring various sensor data as well as their processing, enabling the visualisation of flooding and its risk scenarios in real time. The important aspects of the solutions include the use of real time sensor data streaming in flood prediction as well as in assessing its risk, facilitating the sustainable decision making process. The novel contribution associated with this research are following:

1. Implementing IoT for data acquisition 2. Real time sensor data streaming

3. Big data analytics for large scale processing

4. Development of optimal machine learning model using BRBES to predict the risk of flooding

(13)

13

In addition, a comprehensive study of the data driven approach in comparison with the knowledge driven approach regarding flood risk assessment has been conducted. Since the integrated BRBES employs knowledge driven learning mechanism, it has been compared with other data-driven learning mechanisms to determine the reliability in assessing flood risk. A comparison has been performed among various approaches such as linear regression, random forest, decision tree, artificial neural network and experts’ opinion while the BRBES produces reliable results than from the other approaches.

1.4 Deliverables

This thesis investigates and evaluates the existing platforms used for assessing flood risk and come up with an integrated BRBES with big data platform considering uncertainties to mitigate the risk of flood. Data collected from Bangladesh Water Development Board (BWDB) and the Twitter stream was used to validate and test the system. We have used a dynamic tree traversal algorithm developed by Rafiul et al. [13], which helped us avoiding to write BRB algorithm from scratch. A hybrid Spark-Hadoop framework has been used as big data platform to populate the data from storage and trigger the BRB system to assess the risk of flood in real time. Moreover, to increase the accuracy and efficiency regarding the assessment, optimal learning model for BRB has been introduced. It helps to get better results by optimizing the parameters. It has features such as knowledge base, inference mechanism and learning parameters which makes the expert system intelligent. The novel contribution of this research to propose a system that can combine big data platform with the expert system to address various types of uncertainties. A REST (Representational State Transfer) API (Application Programming Interface) along with a web interface is built to visualize the pattern and behavior of different factors associated with flood through schematic diagrams. It also displays the risks of the flood in different zone of a country through a map using Google API. The system provides real-time flood forecasting ranging from a few hours or a day ahead which eventually helps the authority to make a quick response and take necessary measures before the flood occurs. Hence, It reduces the impact of the flood in different ways to mitigate the suffering of people.

(14)

14

1.5 Scopes and Delimitations

Risk assessment expert systems can be utilized in numerous domain such as medical diagnosis, e-government system, energy efficiency in data centers, traffic management system, disaster management etc. However, This research is mainly focused on flood risk assessment. As green ICT and sustainable development is the main agenda for PERCCOM consortium, this research also defines the context of sustainability in flood risk assessment.

This novel approach to implement BRB algorithm on big data platform facilitates the opportunity to assess the real situation of the flood in any area. The proposed system is capable of handling both structured and unstructured data to produce more accuracy in results. The FloodMap tool, which is a web interface for visualizing the risk in real time provides valuable insights to take proper precautions before the flood occurs. However, using satellite imagery as a data source to assess flood risk and suggesting immediate precautions regarding flood risk in FloodMap tool can be considered as future scope.

Although the system is providing a good assessment, the dataset regarding the computation is not quite as large as the prototype has been tested with gigabytes of data. It will be interesting to see how it performs while processing petabytes or yottabytes of data. The study used different sources of information (social networks, BWDB). It could be interesting to analyze the results in using separately the different sources of info in order to assess their impact on the global quality of results.

1.6 Sustainability Aspects

Sustainability is a term can be defined as “The development that can address the need of present without trading off the need of future generations” [14]. According to the Brundtland report, sustainable development was first introduced in 1987 [15]. It is a framework that can maintain the change in a balanced fashion avoiding the exploitation of natural resources. Sustainability has three major pillars, namely social, economic and environmental [16]. These pillars can also be referred to as people, profits and planet. In this research, sustainability has been considered as the core ideology regarding flood risk assessment which is covering all the three aspects of it.

(15)

15

Economic Pillar: This pillar focus on the economic and financial benefits.

Environment Pillar: This domain emphasizes the natural resources and effects of development on the environment for instance carbon footprint.

Social Pillar: Social stability and evolution are the main features of this factor of the sustainable development.

As this thesis is a part of the Erasmus Mundus PERCCOM program, it emphasizes the sustainability aspects under the scope of ICT. Sustainability in software system describes how it will perform in different circumstances. It is important to determine the software’s long-time consequences prior to designing the model architecture of the system. However, this approach to increase the longevity of a system is not quite enough to ensure all the aspects of sustainability [17]. A sustainable software system needs to address its immediate features along with their effects which will determine the cumulative impact in long run [18].

Figure 2. Immediate, enabling, and structural effects of the Flood Risk Assessment system in the five sustainability dimensions (Becker’s Sustainability Model).

(16)

16

Becker’s model is used to analyze the sustainability impact of a software service [19] as shown in Fig. 2. This involves five interrelated dimensions including individual, environmental, economic, technical and social. The individual dimension includes the inhabitant’s ability to exercise their freedom of expression and social rights. The environmental dimension concerns about climate change and consumption of natural resources to ensure balance in a local ecosystem. The economic dimension ensures transparency in financial aspects such as capital growth, the liquidity of the currency. The technical dimension is all about maintenance, resilience, and evolution of software systems over time and finally, the relationship between individuals in a group or society falls under the social dimension.

These dimensions can be categorized into three segments, namely immediate, enabling and structural [20]. The complete life cycle of the system can be considered an immediate impact, whereas the opportunities it facilitates in long-duration refers to enabling effects.

The structural effects are the recognizable transformations caused by the system that occur on a large scale. The FloodMap tool forecasts the system generated risk through a web interface which provides all the adequate information to the people regarding flood.

Therefore, it creates the immediate impact on citizens for taking proper precaution. As people are getting more information about the environment, they are gaining more knowledge which eventually increases awareness in society and stimulates people to help each other. Besides, structural impact helps to form a resilient and sustainable community by reducing disaster impact.

1.7 Thesis Outline

We now present an outline of the content of the next chapters in this thesis.

Chapter 2 - Background and Literature review

This chapter discusses about the factors considered for flooding, flood risk assessment framework and various types of uncertainties regarding flood. It also covers the existing methods for flood prediction and risk assessment.

(17)

17

Chapter 3 - Roles of Bigdata in Flood Risk Assessment

This chapter includes the defination and scope of big data for flood risk assessment. It also describes the opportunities and challenges of big data. Some case studies are also presented in this chapter to show how researchers are using big data for assessing flood risk.

Chapter 4 - Methodologies and System Framework

This chapter describes Design Science Research (DSR) and BRBES methodology, which has been used for assessing the flood risk. It also discusses how optimal learning can train the parameters of an expert system. Moreover, It gives detailed information about the integrated BRBES model with Big data framework (Spark), system architecture, data collection, implementation and the web interface tool (FloodMap) for risk visualization.

Chapter 5 - Results and Discussion

This chapter presents the results and discussion regarding risk assessment framework. A comparison between knowledge driven approach BRBES with other data driven approach has been performed in order to check the reliability of the system.

Chapter 6 - Conclusion and Future Works

This chapter draws conclusion from the evaluated results and observations. It also points out some limitations and future scope for this research.

(18)

18

2 BACKGROUND AND LITERATURE REVIEW

This chapter explains the impact of flood considering the way people suffer in different parts of the globe. Then it presents the concept of flood risk along with a detailed discussion about risk assessment framework for flooding. This is followed by the investigation of various factors that can possibly determine the risk of flood. It also elaborates the uncertainty issues which needs to be addressed while assessing flood risk.

Finally, the chapter ends with a comprehensive review of the existing methods for predicting flood risk.

2.1 Flooding

Rapid climate change and the growing population are playing a vital role to increase the risk of floods all over the world. Flooding can take place for different reasons such as the overflow of water from the lake, river or ocean. Severe rainfall on a saturated ground can cause an aerial flood [21]. However, as the amount of rainfall and snowmelt may change in every season, the size of these water source can also vary [22]. Therefore, these changes can not be considered significant unless they are responsible for causing damage on a large scale. It is also possible to occur floods in rivers when the capacity of river channel exceeds by the flow rate of water [23]. The properties and industries that are situated in the natural floodplains of the river, can get harmed by the flood. It is possible to eliminate the damage due to flood by shifting the accommodation and other activities away from water sources, However, it is very unlikely as the river provides better access and travel routes for business, as well as the land near embankment, is usually fertile which makes people live and work by the shores.

Flooding can be categorized into three types: coastal flood, river flood and surface flood [24]. Coastal flood is caused by severe weather conditions such as tropical storm, which push water onshore that overwhelms low lying land and become a serious threat to life and property [25]. River flood or often known as flash flood occurs due to excessive rainfall over an extended period of time and can be destructive because of the high velocity torrent of water [26]. Finally, surface flood is caused by urban infrastructure, for example poor

(19)

19

drainage system leads towards water clogging during intense rainfall and water flows out into streets and nearby structures [27]. In order to mitigate the consequences during flood, it is essential to determine the risk of flood.

2.2 DEFINITION OF FLOOD RISK

Flooding originates from a variety of sources, however it should be pointed out that, in reality, these types of flooding will often occur in combination. While assessing flood risk, it is necessary to consider the actual damage caused by flood. It can be defined as ”a combination of the probability (likelihood or chance) of an event happening and the consequences (impact) if it were to occur” [28]. In terms of water source the risk of flooding can be categorized as:

a) Fluvial (main river flooding)

Fluvial flooding occurs when the main river (e.g. the River Great Ouse) exceeds its capacity to accommodate the volume of water that is coming from the shore.

b) Coastal flood

Coastal flooding caused by storm surges that caused sudden rises in sea level due to strong winds. It may become severe when high tides coincide with high waves and surges.

c) Reservoir flood

Reservoir stands for artificially created lakes such as a dam. When water escapes from the reservoir by erosion or accidental damage to the structure, results land or properties being flooded is called reservoir flooding.

d) Sewer flood

Sewer flood occurs when heavy rainfall creates an overflow in sewerage or drainage system gets clogged due to heavy pressure and waste materials.

(20)

20

e) Burst water main

Busrt water main is a kind of flooding which is not caused by rainfall. It takes place when there is a burst or major leak occurs in the water supply system. This type of flooding cause disruption in transport system.

Kron describes flood risk as follows:

RISK = HAZARD * EXPOSURE * VULNERABILITY

This definition consists of four elements. Hazard is defined as ”the probability of the event and the size of the flood” [29]. Exposure is the part that represents the elements that could be affected by the flood, because they are located in a flood vulnerable location [30].

Vulnerability represents the potential damage that the exposed assets can catch when there an actual flood. Finally it is possible to define risk as the probability that the usual state is disturbed by an event. In terms of flood risk assessment this event will be a flood event [31].

2.2.1 HAZARDS

Flood hazard is the probability and the magnitude of a certain flood event. This probability is most often described in terms of a return period. A flood is less likely to occur when the return period has a higher value [32]. The magnitude can be expressed in multiple unites, such as duration, inundation depth, extent and flow velocity. In a large scale assessment like carried out in this study the inundation depth is most often used to quantify the flood risk. This is because inundation depth can be recorded very precise during a flood and the other unites are mostly not measured at all.

2.2.2 EXPOSURE

Exposure contains the elements that could be influenced by the event of a flood. In a small scale flood risk assessment you could describe the entire city in land use maps. Every building and road can be appointed to a land use category and you would get a reasonable precise flood risk estimation. In a large scale assessment as carried out in this study it is often a very time consuming job to describe entire cities in this precise level. Therefore in a large assessment a limited amount of land use categories is used. There are only five of

(21)

21

them because using more land use categories will take a lot of time to derive the land use maps and also a lot of computer power to calculate the total flood risk. The negative side of using only five land use categories is that the estimation is less accurate.

2.2.3 VULNERABILITY

Vulnerability represents to which extend the exposed elements are damages or affected by the floods. The standard approach is using depth-damage functions, which relate the inundation depth to a certain damage value. These depth damage functions are derived by Huizinga (2015) [33] and those are land use category – specific. Damage values are usually expressed in monetary values, but there is a difficulty in this scenario. This is because every country uses different currency, those have to be generalized to a certain currency and those values needs to be reconciled, especially in the case of inflation.

2.3 FLOOD RISK ASSESSMENT

Flood risk assessment (FRA) is a framework, which determines or evaluates the intensity level regarding flood risk [34]. It is also defined as ”the degree of interaction between various dimensions (depth, areal dimension) of flood and exposed socio-economic elements to the flooding” [35] as stated in Eq. 1.

f(R) = PF * C

Here, PF = probability of flood occurrence C = Consequences of exposed elements

FRA follows a specific approach in order to identify and assess the main causes of risk.

Moreover, the main purpose of this framework is to study and determine the measures to mitigate flood effect and providing guidelines for taking proper actions before and during a flood. It consists of three steps as shown in Fig. 3.

(22)

22

Figure 3. Framework for Flood Risk Assessment (FRA)

2.4 UNCERTAINTIES ASSOCIATED WITH FLOOD RISK ASSESSMENT

Flood risk assessment is associated with considerable uncertainty as it needs to deal with extreme events and failure scenarios. Different types of uncertainties can be observed while studying the factors of flooding. Hence, this segment discusses the uncertainty issues which are associated with different factors of flooding.

The term uncertainty refers to an unpredictable outcome, whereas risk can be considered as a consequence of actions, which is taken despite uncertainty. Different types of uncertainties such as vagueness, imprecision, ambiguity, incompleteness and ignorance are associated with the risk factors associated with flooding [36]. Therefore, it is mandatory to identify the uncertainties to get meaningful robust results regarding flood risk assessment.

(23)

23

TABLE I. FLOOD RISK ASSESSMENT FACTORS AND THEIR RELATED UNCERTAINTIES

Factors Types of

Uncertainty

Discussion Water Level Incompleteness

, Vagueness

Rise in sea/river/surface water level.

Accommodation Vagueness, Imprecision

Problem with accommodation in flood affected area.

Transport Availability

Incompleteness , Imprecision

Unavailability of transportation in flood affected area.

Delay in Transportation

Ignorance, Imprecision

Transportation system becomes standstill due to flood.

Road Damage Imprecision Percentage of road damage due to flood.

Financial Condition

Vagueness Financial condition of the flood affected people.

Social Condition Vagueness, Ignorance

Refers to the social condition of flood affected area.

Damage of Crops

Incompleteness damage of crops in the flood affected area.

Loss of Cattle Imprecision, Incompleteness

Percentage of cattle were died during the flood.

Casualties Imprecision How many people died during flood.

Health Hazards Ignorance, vagueness

Refers to health issues due to water borne out- breaks Water

Contamination

Incompleteness , Imprecision

Refers to the scarcity of fresh water

Lack of human knowledge or anomalous data from sensors can lead towards these kinds of uncertainties [37]. For example, the erroneous and misleading nature of sensor data makes the prediction highly unreliable. Battery power, computational and memory capacities, as well as communication bandwidths are the resource constraints which are responsible for missing, redundant or inconsistent data. Besides, sensors have vulnerability issues regarding the malicious attack, for example, eavesdropping, indiscretion, integrity violation, denial of service and black hole attack. Similarly, human ignorance to define qualitative data can lead towards uncertainties due to vagueness and ignorance, for example, while expressing ”social condition” in terms of ”high”,”medium” and ”low” as referential value, then the answer can be varied in a wide range of possible ways.

(24)

24

Therefore, it is mandatory to develop a reliable system that can address all sorts of uncertainties while assessing flood risks.

2.5 METHODS FOR FLOOD PREDICTION AND RISK ASSESSMENT

This segment presents the existing methods developed by different research groups and organizations all over the world for predicting the risk of flood.

There are traditional approaches for predicting flood assessment and monitoring which are usually based on the data from heterogenous geospatial resourcess to enable the expert system for hazard mapping. The apporach given in Kussul et al. [38] is based on probability density formula which highly dependenet on data-analytics of the geospatial heterogenous data following a sensor web method to provide standard services or utilities for establishing alerts, planning and acquiring observations and make alerts, etc.

SensorWeb Pilot initiative in Namibia is one of the examples of implementation of this approach.

United State’s HUD (Dept. Of Housing and Urban Development) developed another approach for flood assesment based on the hydrological method in 1960 which is often used by National Flood Insurance Program (NFIP) for house insurance purpose [39]. It is based on the principle of water surface elevation in a geographical area measuring probability to calculate the flood risk. Later, annual losses used to be calculated using the method of the probability function. In theory, the measure of the probability function is translated into depth inundation function by the adoption of damage-model. NFIP provides analysis for specific sites based on comprehensive measure and evaluation of the specific probability function to asses the risk. It also takes into account the reliability and performance of protection along with the flood failure effects into consideration.

For Addressing Uncertainty, Rafiul et al. [12] introduced a web based BRB expert system to assess flood risk. This proposed system is capable of reading sensor data, identifying the risk and visualize the risk in a web server. Moreover, the author developed a generic RESTful API which makes the system more robust and highly accessible by providing s a

(25)

25

layer of abstraction for the users who want to use the expert system without knowing much about the underlying algorithm. However, this approach is not scalable because the system is not capable to work with sensor data streaming as it provides huge amount of data.

Nowadays, the crisis response teams of a country have turned their interest to explore the potentialities of big data in managing disaster risks such as flooding. The reason for this is that, crisis response authorities need to react promptly while flooding based on the inaccurate and incomplete information. A real time map was built by a group of scientists (De Groeve, Kugler and Brakenridge), that can show the location, timing and impact of floods by collecting flood data from twitter and satellite [40]. The research proposed a space based river monitoring framework to facilitate a systematic and impartial approach to predict flood risk. The results obtained with passive microwave remote sensing considering flood extent and flood start and duration. Optical high-resolution satellite imagery has been used for validating extent measurements. It allows to have a quantitative evaluation of the impact of the floods. The technique has been demonstrated in Southern Africa during the floods of 2009.

Similar approaches have been proposed, for example, the state of New South Wales in Australia developed an early alarm system for risk analysis of various kinds of natural disasters as discussed in [41]. It is an interactive tool for natural hazards risk analysis based on the research proposal which goes by the name of web-GIS. It utilizes the open-source technologies and geospatial tools to predict the natural phenomenon of the disaster, especially landslides and floods. For accuracy and least probabilistic error this approach proposes to collect data from various resources, for instance: historical data sets from various resources, data from floodplain, meteorological dataset. Only specific geospatial data can be downloaded into the platform to analyze and asses the risk.

Considering the advent and usage of social media, ’Floodtags’ methodology has been proposed to extract information from the analysis of data shared by people on social medai, for instance, twitter tags. The information can be extracted using the filteration of data based on keywords by visualization and mapping social medida content as proposed in [42]. Moreover, it enables the system to identify the inundated areas by getting service

(26)

26

from satellite observations. This methodology has shown reasonable results to monitor the floods in Pakistan and Philipines especially in the areas which are densely populated.

All these flood risk assessment methods described above address different issues of flood risks for specific scenarios. While some of these methods explicitly consider uncertainty features such as web-based BRB expert system, some frameworks are well equipped to provide scalable solutions while working with huge amount of complex data. Therefore, it can be argued that none of these solutions can be considered as the best approach while dealing with both uncertainties as well as voluminous sensor streaming data. For this reason, this research emphasizes on integrating big data platform with BRBES to mitigate the constraints mentioned above and introduce a new learning model which will provide scalability and availability while addressing all sorts of uncertainties.

2.6 SUMMARY

This chapter discussed about the factors considered for flooding, flood risk assessment framework and various types of uncertainties regarding flood. It also described the existing methods for flood prediction and risk assessment. The Next chapter will discuss about the role of big data in flood risk assessment.

(27)

27

3 ROLES OF BIG DATA IN FLOOD RISK ASSESSMENT

This chapter will discuss the importance of big data associated with flood risk assessment.

It describes the features and characteristics of big data and how it becomes a new phenomena for disaster management, especially for identifying the risk of flood. It also explains the scope of big data that can bring new opportunities to improve the way flood risk managements are planned and executed.

3.1 DEFINITIONS AND FEATURES OF BIG DATA

Data sets are called big data when data becomes so large and complex that traditional data processing and management methods are not capable of handling it [43]. In other words, it is sets of data whose volume, velocity in terms of time variation or variety is so big that makes it difficult to be stored, processed and analyzed by using conventional databases and data processing units [44]. More formally, Big data is massively large datasets that require to be computationally analyzed in order to find trends, and patterns to make a strategy for the near future [45]. For instance, data collected from sensors, mobile devices, twitter tweets, security footage, large organizations, and youtube videos etc. can be considered as big data since they all require a big amount of storage and processing capability to handle these data.

At present, because of cheap and dense information sensing IoT devices such as software logs, Radio Frequency Identification (RFID) readers, mobile devices and Wireless Sensor Networks (WSN), data sets grow exponentially in a tremendous rate that 2.5 exabytes (2.5×1018) of data are generated daily since 2012 according to IDC report [46]. According to predictions, the global data volume will continue to grow enormously from 4.4 zettabytes to 44 zettabytes between 2013 and 2020 By 2025 [47]. The big data size can reach up to (yottabytes (1024) which makes the data process needs hundreds or thousands of servers to run software parallelly. It can be said that big data is a relative term and depends on the user's capabilities and tools. For instance, Some organizations are required to look through their data management systems while processing hundreds of gigabytes of

(28)

28

data, tens or hundreds of terabytes might be needed before data volume becomes considerable [48].

There is a possibility that traditional methods of analysis may not work with these large amounts of data, which raises the question of how to develop a highly efficient platform for analyzing data and extract useful information. The difficulty of dealing with enormous data sets is still present when we enter the era of big data despite the advances in computer and Internet technologies have witnessed the development of computers physical components after Moore's Law for many years. Fisher et al. [49] indicate that big data implies the data can not be loaded and handled by most of the traditional information methods and systems due to the fact that data in the big data age will not only turn into being too large to be processed into one machine, it also means that the majority of convenient data analytics or data mining methods are not designed and developed to be used for big data analysis but for a centralized data analysis process. It is also worth mentioning that extracting the meaningful data from the large sets of data by using user behavior and predictive analysis could be related to the Big data as a term.

There are a number of characteristics associated with big data: volume, variety, velocity, veracity and value as shown is Fig. 4 [50].

Figure 4. Characteristics of big data

(29)

29

Volume: It refers to the large datasets which are being shared or generated from various sources, for instance, smartphones, sensors etc [51]. There is no fixed threshold which can determine whether the volume of data to be considered as big data. Computation of voluminous data is required for reducing the cost and increasing the scalability as well as performance regarding their storage, access, and processing.

Variety: It refers to the increased diversity of data as data is coming from different sources in various formats, such as text, audio, binary, video, image etc [51].

Velocity: Velocity refers to the increasing speed at which big data is being generated and how the data needs to cope up with this scenario while being stored and analyzed [51]. We are getting information at a much rapid rate and in real time, unlike past because of the introduction of cheaper sensors, smartphones and social media.

Veracity: It refers to the quality of data and valence that measures the ratio of connected data items to the possible number of connections that could occur within the collection [51].

Value: The main objective of big data analytics is to extract the value by running complex computation on big data sets and without getting any valuable insights or solutions, all these computations will become worthless [51].

3.2 BIG DATA FOR FLOOD RISK MANAGEMENT

The reaction should be quick and effective when an emergency occurs, nevertheless, it is often tackled with disorder and confusion. Crises, whether natural, man-driven, or caused by various factors such as floods, make all communities vulnerable and may suffer significant losses [52]. Flood risk management requires special assistance to protect and rescue flood victims, as well as those who assist them in evacuation operations, regardless of the risk caused by the disaster. Therefore, Big data has a lot to do with flood-related disaster management. Big data is capable of dealing with an huge volume of data, which

(30)

30

comes from various sources in different formats [53]. The computing resources need to meet the challenge of Big data in order to make decisions based on the processed data and deliver useful information by evaluating trends, patterns and data interconnection. All types of corporations such as Amazon and Alibaba use the large amount of data analytics to find trends in order to enhance their strategies and increase revenues.

These e-commerce service providers analyze the big data to study and observe the behavior pattern of their customers which result in customer satisfaction and bring more revenues for the corporation. And in the same manner, different crisis management teams have shifted their attention to the potential of Big data in order to arrive at an enhanced pattern to predict the disasters such as floods, wildfires, earthquakes or storms [54]. The reason behind this is that data are coming from different sources, such as human, organizations and machines, related to natural disasters. This is due to the fact that the data sources associated with natural disasters are different, such as human, organizations and machines.

Predicting disasters before they occur can give enough time to evacuate people and make the necessary precautions based on information obtained from sensors, satellite images, social media (Facebook or Twitter), and disaster management through an API, since big data is known for converting unorganized groups of data into something useful and comprehensive. During the disaster, the most challenging task of an disaster managemetnt team becomes to react rapidly and in an appropriate manner based to the inaccurate and incomplete information that come from a variety of sources as happened when tsunami stuck Japan in 2011, in which rescue and crisis management teams struggled to reach stranded people to help them [55]. Big Data analytics can help in managing the disaster under such kind of conditions.

3.3 HOW CAN BIG DATA HELP?

Big data can increase social resilience to natural disaster by providing functionalities such as monitoring hazards, predicting exposure and vulnerabilities, managing disaster response, assessing the pliability of natural systems as well as engaging communities

(31)

31

throughout the disaster cycle [56]. Satellites, seismographs and drone provide consistently enhancing remote sensing abilities. Data that are coming from the smartphones and Twitter feeds create significant opportunities for monitoring hazards like floods or earthquakes [57]. Experts can identify geographical and infrastructural risks by using satellite images.

Volunteers, as well as general people, can add ground level data by using crowdsourcing applications like OpenStreetMap or Ushahidi, for instance, people can inform their status during a flood or any kind of disaster to the authorities [58]. Social media can be monitored to study the behavior and movement of people after a natural calamity for guiding disaster response accordingly. To improve the agricultural interventions in developing nations, different sensors can be used in the field to reveal the quality of air and soil. By raising awareness among citizens, big data helps to build strong communities that can manage their natural system, strengthen infrastructure and take effective decisions for a better future.

3.4 OPPORTUNITIES OF BIG DATA IN FLOOD RISK ASSESSMENT

This segment will explain the scope of big data that can bring new opportunities to improve the way flood risk managements are planned and executed. With big data, records of previous flood incidents such as fatality, the amount of damaged properties, rainfall during that period, infrastructures of the areas including coastal areas as well as cities and drainage system can be analyzed properly [59]. It can also pick out the specific mobility support or resources that are needed by the inhabitants of a flood affected area. Hence, identifying population hotspot gets easier with big data to provide real time alarm and warnings to the residents when a disaster approaches.

(32)

32

Figure 5. Opportunities for big data for flood risk assessment

Big data helps to study future reactions of the people who are living in a specific zone and suffered tremendously by flood [60]. By using geographical image mapping technique, it is possible to map the risk zones in real time of a city or area and viewers can observe the assessment through web services [61]. For instance, data scientist can extract detail information from local mobile network companies about how people reacted and responded to an emergency like flood. Moreover, big data makes a sort of spatial information framework to build the foundation which will make policies, protocols and the trade of information as an ongoing priority. Such sharing of information makes new best- case situations to help both the responders and survivors.

3.5 CHALLENGES OF PREDICTING FLOOD RISKS

Despite of getting very promising results, big data needs to deal with some barriers, uncertainties and risks associated with the assessment because of human and organizational capacity gaps along with lack of access to internet and IT infrastructure especially in the developing countries. While implementing and scaling new approaches, big data is open to new risks due to specific technological, political and economic obstacles [62]. For instance, the privacy and security of cell phone’s data can be hampered

(33)

33

by factors ranging from the large chunks of data sets due to uncertainty issues. Another example can be the analyses of social media data that works fine in developed countries however, it may not be reliable in developing countries due to much thinner and more skewed base users.

While leveraging big data to build resilience in a complex and volatile environment, it is necessary to consider some factors such as constraints on data access and completeness, analytical challenges to action ability and reliability. For example, finding out the approaches to mitigate verification technique and sample bias correction methods, human and technology capacity gaps as well as ethical and political risks [63]. Moreover, big data needs to comply with its major four V’s, that is volume, velocity, variety and veracity as large amount of data needs to be processed that are coming from different sources in various formats with a high rate which can be unreliable and associated with lots of uncertainties [64]. During flood risk assessment, big data has to deal with various constraints in different phases like data acquisition, information extraction, data integration and analysis, data lifecycle management, crowdsourcing and disaster response recovery [65].

3.6 CASE STUDY: FLOOD PREDICTION USING BIG DATA

Use case 1: A group of scientists (De Groeve, Kugler and Brakenridge) built a real-time map of location, timing, and impact of floods by combining information related to flood from twitter and satellite observations. It is possible to update the map constantly and can be accessed online.

Use case 2: In Netherlands, the government has started experimenting with how machine learning may help strengthen preparedness to future floods, where the vast majority of the population lives in flood-prone areas.

(34)

34

Figure 6. Monitoring flood risks using Big Data Analytics platform

Use case 3: An early warning system has been developed by the New South Wales state emergency service in Australia. It takes meteorological dataset such as data from flood plain, historical data information from various databases to perform predictive analysis of floods in different region.

Use case 4: A social media analytics platform named Floodtags was deployed for extracting information from twitter. It has the functionality to perform filtering, visualization and mapping social media content based on location and keywords. Besides, it also provides a service through microwave satellite observations for identifying inundated areas rapidly. The approach has been used in Philippines and Pakistan as case studies which later proved to be a great success monitoring large floods in densely populated areas.

3.7 SUMMARY

This chapter explained the defination and scope of big data for flood risk assessment. It also describes the opportunities and challenges of big data. Some case studies are also presented in this chapter to show how researchers are using big data for assessing flood risk.

(35)

35

4 METHODOLOGIES AND SYSTEM IMPLEMENTATION

This chapter discusses Design Science Research (DSR) methodology along with the procedure of knowledgebase construction and inference mechanism of the Belief Rule Based Expert System (BRBES). In addition, this chapter will also discuss the system architecture and framework of the risk assessment model. It also explains different components and tools used for building the expert system on big data platform.

4.1 DESIGN SCIENCE RESEARCH (DSR) METHODOLOGY

Improving artifact design knowledge is one of the most import aspects of Information System (IS) design research [66]. In this paper, Design Science Research Methodology (DSR) has been explored and utilized for developing a framework based on BRBES, a decision support system regarding IS design aspects. DSR provides precise guidelines to evaluate and iterate an artifact within the research project [67].

Figure 7. Cognition In The Design Science Research Cycle

(36)

36

It can enhance the functional performance of an artifact in order to get the better analysis.

Moreover, DSR can be applied on different artifacts, such as algorithms, human/computer interfaces, design methodologies including process models and languages [68]. The sole purpose of this approach is to make the assessment of BRBES inference engine more efficient and reliable. Fig 1. Shows the iterative knowledge distribution process of DSR methodology along with various phases.

The first phase considers an awareness of problem which may come from different sources including new developments in industry or in a reference discipline and determine the innovative approach to address research goal. The output of this phase is a formal or informal proposal for a new research effort. Suggestion phase is a creative step for providing new functionality to the new or existing elements. It recommends a tentative design for the solution prototype model. This model is further developed and implemented in development phase. The procedure of the implementation varies depending on the artifact to be created. Evaluation phase makes assumptions about the behavior of the artifact. This phase exposes an epistemic fluidity of deviations (both qualitative and quantitative) from expectations. Finally, Conclusion phase is the end of research cycle which is eventually an integral part of knowledge contribution where the iterative process extracts new knowledge and process the whole framework from the beginning. DSR approach ensures that the artifact is closely linked with the theory and practice [69].

4.2 BRB EXPERT SYSTEM

A Belief Rule Base is an extension of traditional IF THEN rule base, which is capable of representing more complicated non-linear causal relationships under uncertainty. A Belief Rule Based Expert System (BRBES) consisting two main parts named is knowledge base and inference engine. The main functionality of Belief Rule Base is to build initial rule base whereas evidential reasoning is used as inference engine. A belief rule is also associated with different learning parameters such as rule weight, antecedent attribute weight and belief degrees [70]. A belief rule is represented by Equation (1).

(37)

37

IF

(X₁is X₁^k) ∧ (X₂is X₂^k) ∧ … ∧ (X_T_kis X_T^k_k) THEN

{

{(P₁, β̅_k1), (P₂, β̅_k2), … , (P_N, β̅_kN)}, ((∑ β̅_kn≤ 1

N

n=1

)) , with rule weight 0 ≤ θ_k≤ 1, (1) and attribute weightδ₁^k, δ₂^k, … , δ_T^k ≥ 0

satisfying ∑ δ_i^k = 1

T_k

i=1

Where, 𝑋₁, 𝑋₂, … , 𝑋_𝑇_𝑘 , 𝑇_𝑘𝜖{1,2, … , 𝑇} represents the antecedent attributes used in the kth rule and 𝑃₁, 𝑃₂, … . , 𝑃_𝑁arethe referential values of the consequent attribute 𝑋_𝑖 where 𝛽_𝑘𝑖 is the belief degree.

For example, if onset rainfall is “High” and prolonged rainfall is “Low” then Meteorological factor is:

{(High, 0.8), (Medium, 0.2), (Low, 0)}

FIG. 8. A Simple BRB Tree Diagram

From FIG. 8. We can observe that, X19, X20 and X21 are the antecedent attributes, while

“X9” is the consequent attribute. The referential values associated with the antecedent attributes comprise high, medium and low. The illustrated rule is complete since the total sum of the degree of beliefs stands at '1'. “X9” BRB consists of three antecedent attributes, each with three referential values, namely high, medium and low. As a result, the “X9”

BRB consists of 27 rules according to (2) [71] [72] [73].

(38)

38

(2) Where,

L = The number of rules in a BRB

𝐽_𝑖 = Referential values of the i^th antecedent attribute

INFERENCE MECHANISMS OF BRBES

Evidential Reasoning (ER) is a decision analysis procedure which can handle heterogeneous data [74]. ER can handle different types of uncertainties such as incompleteness, ignorance, imprecision, vagueness exists in data [75]. The inference procedures using Evidential Reasoning approach consists of various steps including input transformation, rule activation, belief update, and rule aggregation. The goal of input transformation is to distribute the input data over the referential values of the attribute of a rule, which is called matching degree. After calculating the matching degree, the rules are called packet antecedent and they become active and are kept in the short-term memory.

This matching degree is used to calculate the activation weight of each rule. It is interesting to note that each rule has different weight while calculating the referential values of the consequent attribute. The summation of the rule activation weight of a rule base should be one. The belief degree associated with each rule in the rule base should be updated when an input data for any of the antecedent is ignored or missing. This is done by the procedures mentioned in [76]. The output of the BRB system is fuzzy, meaning that qualitative values can be transformed into crisp values, i.e. quantitative data, by assigning utility scores to each referential value of the consequent attributes.

(39)

39

FIG. 9. Sequence Of BRBES Inference Mechanisms

4.3 OPTIMAL LEARNING

This subsection will discuss the methodology of optimal learning procedures for more reliable assessment of flood risk. Optimal learning helps to make decisions by addressing the challenge of efficiently collecting information [77]. Primarily it is used in such circumstances where collecting information are expensive and time consuming. For example, finding the best drug to treat any disease out of several potential combinations.

Moreover, it can be used in expensive simulations where a single observation might take few days, laboratory sciences for testing drugs, and field experiments such as testing new energy saving technology in a data center. In this research, the optimal learning has been used to train the BRB expert system by intense parameter tuning to get more precise result regarding to flood risk assessment.

Optimal learning can be used to train BRB expert system’s parameters which include rule weights(𝜃_𝑘), attribute weights (𝛿_𝑖) and belief degrees (𝛽_𝑗𝑘) [78].These parameters can be obtained from experts of this particular domain. It is also possible to generate these parameters randomly for a BRB expert system. In this research, a multilevel hierarchical tree for BRBES has been developed as shown in Fig. 11. The top node X5 represents the final outcome of the BRB expert system. Several child nodes are placed in different levels, for example, X7 and X6 are the child node which represents antecedent attribute while X5

(40)

40

is the parent as well as a consequent attribute. The child node of BRB’s one level gets the input from the previous level output and that goes on until determining the value of the top most node. From this architecture, it is possible to understand that each level needs to produce better results to obtain an optimal assessment for flood risk and hence, the learning parameters can heavily influence the final outcome of the system. The importance of antecedent factors and rules can be determined by rule weights. The belief degrees are related to consequent attribute to identify the uncertainty of the final outcome [79].

Although, these parameters are not capable of producing highest accuracy without training the BRBES using sample historical dataset [80]. The aim of the learning module is to reduce the discrepancy 𝜉(𝑃) between the system generated results (𝒛_𝒎) and the BRBES results (𝒛̅_𝒎) by obtaining an optimal set of parameters (𝜃_𝑘, 𝛿_𝑖, 𝛽_𝑗𝑘).

Figure. 10. Optimal Learning Model for Flood Risk Assessment

There are some training models that consists of quantitative, qualitative and mixed parameters (both) are available both online and offline for BRB expert system [81]. This research follows an optimization model with mixed parameters to train BRB because we considered both quantitative and qualitative data for testing the proposed model. Therefore, the system is capable of handling heterogeneous data. Fig. 10 shows the schematic diagram of optimal learning model used in this research to train the parameters of BRBES.

The proposed model includes these following steps:

Step 1: Constructing an objective function

Step 2: Setting the constraints to train the parameters