My Foray into CartoDB

 One of the significant milestones of self learning this year was developing web maps using CartoDB.

My profile for CartoDB (https://daneshchacko.cartodb.com/maps)

I first learned about CartoDB through the Spatial Vision training (during the IMIA AP 2014 Conference). In this blog post, I will explore the ups and downs of using CartoDB for web map visualization

Why CartoDB? Up to 2015, the only online GIS platform I am accustomed with is ArcGIS Online. However, I would like to brush up my skills on new areas such as CartoDB. If you look at my profile above, I have developed around 6 maps for various topics .

If you want to design simple maps without much tweaking, CartoDB is easy tool to deploy web maps. They continuously improve the options for you to have for symbology or labelling. One thing that sets apart CartoDB and ArcGIS Online (from my personal experience) is the customization of the tools available in CartoDB.

If you have good programming and web design knowledge, you can exponentially enhance the experience of CartoDB maps. Let me give you a good example. Refer to this link: Sarawak Electoral Visualization

Example 1: Electoral Visualization

I was exploring myself on how to visualize electoral issues of the delimitation exercise in Malaysia. What started out as my personal hobby project, it evolved into a fully embedded map for my NGO's Homepage (http://www.tindakmalaysia.org/)

Referring to the link (Sarawak Electoral Visualization), basic tools of CartoDB weren't of much help. As I proposed CartoDB map to my NGO, they were happy to take onboard on the idea. However, my NGO has placed some expectations on the contents to be shown. After sourcing the datasets from my NGO, I was tasked to show the interrelationship between electoral seat category, representation value and its correspondence to local boundaries.

While it sounds simple, the basic tools of CartoDB could not meet the criteria of the symbology needed (i.e. the representation value or showing the symbol of administrative boundaries). I spent a significant amount of time to research html codes and examples of CartoDB to bring the source codes needed for the  visualization.

Subsequently, with close collaboration with my NGO, I spent additional time in writing out the correct terms for the legend and other necessities for the map to be understandable. Finally, I embedded the map of my NGO's website and customize the right size for the embedding. The default embedding option of CartoDB would result of compression of items in the layout.

Example 2: Time Based Visualization

Similar to ArcGIS Online, CartoDB has ventured into time based visualization (now with new enhancement). For only point based data, Torque (and Torque with category) allows the CartoDB users to visualize time based changes (which is becoming a big trend in maps). I previously designed on ArcGIS Online (and Mango Map) a static map of all the dam locations in Malaysia. It was built on intense research every possible freely available information back in 2013.

Referring to the link (Malaysian Dams), one of the big challenges I faced initially was the original dataset did not have proper timestamp needed for CartoDB. Preferably, in your shapefile or data file, the timestamp should have proper date or date-time format. I have created this dams dataset two years ago for static map and pop-up purposes.

The second challenge was the cleaning-up of classification. It was found through CartoDB upload there were duplicate classification values (i.e. Dams/Irrigation, Dams / Irrigation). I spent some amount of time in removing the duplicates in QGIS.

Since CartoDB torque visualization works for point based data (for the moment), I chose Torque with category to visualize the construction (opening date) history of 70+ Malaysian Dams and categorized them with the purposes. This allows the user to see where the dams are being constructed and see what types of dams are being related. I customized Torque category to show cumulative changes (as dams are rarely taken down). Choosing a suitable base map was another of challenge of its own as you want your content to be the Figure.

I did spend additional time to produce a understandable symbology for the dam purposes. I will be making further changes to make it more distinguishable. The rest was history. CartoDB Torque tool worked very well and now charting history of dams.

Conclusion

CartoDB has been a great tool with steep learning curve. My advice is before you embark on CartoDB maps, do consider your level of html experience and willingness to spend time, energy and (if necessary) money to develop the map. CartoDB email support is Great! They are swiftly respond to support enquiries.

Tell me your CartoDB experience...

QGIS Sample Atlas

Recently, I have been pushing myself to test my capability in generating high quality cartographic works in QGIS. Since I took on a mini project on Malaysian state of Kedah, this atlas was truly a combination of open source data and open source software.

For data in Malaysia, I manage to obtain through third party sources for district, subdistrict and voting locality boundaries. In Malaysia, these sorts of data are super hard to find (generally all forms of datasets for large scale maps of Malaysia are not for free). The roads, railways, rivers and land use data was sourced from Open Street Map (I recommend http://extract.bbbike.org/). Avoid choosing QGIS tool for Open Street Map as you get good coverage of the dataset with minimal attributes (making them redundant). The terrain is not exactly open source as it came from ASTER GDEM and hence, it restricts anyone to produce commercial maps.

The software I used was QGIS. In order to produce 200+ page atlas (based on sub district levels), I used Print Composer. It is the equivalent of Data Driven Pages in ArcGIS. I have made full use multi map concept where I can pinpoint the reader where they are (the inset maps). The legend size remains difficult thing to adjust (now I realized there is legend customization through filter) as it dominates a significant chunk of the map. The scale is tricky part and I resorted to numeric figure as QGIS hasn't improved on the variable scale legend. Now, I also realized there are better options to choose from for the North Arrows.

How about the map itself? I used pretty much standard colour schemes for every dataset for the atlas. Hillshading worked out very well  for the Bandar Padang Mat Sirat and Bandar Yan (impressed with QGIS capability). Choosing the colour of roads required much judgment as the terrain colouring exerted its influence significantly. The text visibility has been controlled after much playing around and deliberation. The original dataset of roads showed many road segmentation, Using the blending option, I was able to show a full continuity for the highways.A lot of judgement was used to determine how many layers are relevant to the atlas and which ones need to be kicked out.

Below here are some examples of urban, semi-urban and rural settings of Kedah (viewed through maps). When the final product came out, I was impressed with QGIS capability. For more maps, please contact me here.

Creating an 'Atlas' using ArcGIS and QGIS

Usually when I produce maps, I produce a single map based on single theme. However, back in my uni days and work training, I have learnt an important cartographic component in ArcGIS and QGIS.

A sample Local Government Area map of New South
Wales built on ArcGIS

It was about building an 'Atlas' in GIS. Multiple maps stored in one PDF and ready for print. This knowledge was put to the test when one of my Account Manager said he wants a map of all his clients (October 2014). His clients are spread over one state and he had a list of them in Excel. Good thing was they were Local Governments only.

In April 2015, another Account Manager approached me and asked me to produce a map of local governments and water authorities in New South Wales. You may think why do we need an Atlas style product for these two instances?

In this discussion, I will be discussing the challenges behind making these two 'atlases' and some comparisons between QGIS and ArcGIS.

Challenges/Issues to Consider 

• Data - In Australia, they have free data for Local Government Areas. However, the free data they had for the road network and localities are nearly 10 years old (Sourced from Geoscience Australia).
• Intended Audience - Account Managers use these maps as rough guides where their clients and prospective ones are found. Sometimes they may use for rough planning when they traveling around to see multiple clients. Hence, just showing the local government areas (LGAs) is not enough. I cannot expect my audience to know every LGA by name and location. Due to that, I incorporated roads and localities to give good geographical insight
• Level of Information Displayed - This becomes an issue on how many towns and cities the map should contain. If you refer to the first figure above, I classified the localities strictly to towns and cities. My Account Managers doesn't need to know every town or populated area in LGA. They need to know LGA name, the main town (s) and some roads connecting between towns.
• Scale - The most difficult issue I faced. New South Wales has over hundred LGAs of varying size and it is impossible to have clean map with all names of LGA. That is why the concept of mapbook was used in these two instances. ArcGIS and QGIS uses one shapefile to break down the mapbook into regions. I used Tourist Regions to divide the various states in Australia to produce multiple maps. However, in the context of New South Wales, I felt Tourist Region is very general division. I chose Statistical regional divisions to show more zoomed-in views of Sydney area. Sydney has many smaller size (only area we talking about) LGAs. 
• Text Layout - Text is necessary evil. I have ZERO intention of using any other graphic software to beautify the map. I am producing the map on the go and would like to deliver fast to my Account Managers. I fully capitalized Maplex Label Engine (in ArcGIS) to position text at best. 
• Software - I have used mapbook approach twice (in real life situation) so far in my cartographic career. The first instance, I was motivated to use QGIS to produce mapbooks for many states. The second instance, I used ArcGIS as I have not used Data-Driven pages before.Both instances they were learning curves for me.

ArcGIS/QGIS

I was very much grounded to ArcGIS (the only GIS software I knew at university). I came to know QGIS through my current work and learned further through QGIS training. Some opinions:

• In both GIS software, the final layout and content to be displayed are nearly the same. You can change the paper size, adjust the legends and placing inset maps in ArcGIS and QGIS
• From my experience, time consumed from the collecting data to final output are nearly the same for both software.
• However, the KEY difference between ArcGIS and QGIS I have noticed is the cartographic presentation. In ArcGIS, they have more in-built colour schemes than QGIS. Hence, I spend less time in determining on one of the fundamentals of maps. In text labeling, I was pleased both in performance by ArcGIS and QGIS

If you ask my stance, I tilt to ArcGIS due to my longer exposure comparatively to QGIS. However, the essence of this blog post is not about which software is better than other. In these two instances which used two different software, it is principles of mapping that played a big factor:

• Who is my Audience?
• How much information my Audience needs to see?
• Is the datasets needed for the map readily available?
• What scale level should I utilize? Should it be fixed or dynamic
• What is the best position for the labels
• Is the overall map layout conveys the right information the Audience needs to know?

Resources I used for building the map books using QGIS and ArcGIS

Atlas Generation in QGIS  and map book in ArcGIS: 

Visualizing Data

In one of my recent projects (back in January), I was working on converting text data of Wifi locations in Penang (Malaysia) to an online map of internet hot spots. Basically, I was visualizing boring wordy and lengthy data to graphically pleasing locator tool.

Refer to: Penang Free Wifi Project

Presentation from Geoplex

Thursday ago, through a friend of mine, I came to know an open event in Melbourne about Visualizing Data. It was hosted by GIS consulting & solutions firm, Geoplex and around 30-40 people (I think) attended for this exciting event. We had four speakers: Geoplex, City of Melbourne, The Age (newspaper) and Flink Labs. The first presentation was visualizing traffic accident data using CartoDB. Geoplex built an interactive web map whereby user can estimate the risk of accident on a specific route (they are travelling on). However, I couldn't understand the workflow from sourcing the information to the final product.

Melbourne Urban Forest Visual

 The second presenter came from City of Melbourne and he was talking about CityLab projects. Basically, CityLab is a physical and virtual space for the council and the community to engage to resolve urban problems. At the same time, there is push in City of Melbourne on the concept of open data (free datasets for public to download and use). While open data enhances transparency and accountability, there is a lot of challenges. For example, datasets relating to utilities comes from energy & telcos companies while dataset of tree & parking location come from the council. However, both datasets cover the same area (i.e. Melbourne). The push of open data would be obstructed as different policies govern about datasets in different institutions (e.g. telcos, council). Second challenge of open data is what and how much to share the datasets. Should we make all the datasets for public view or how much of utilities information can be downloaded free? Anyway, City of Melbourne has collaborative project with its concerned citizens on air quality (Citizen Science in action). For example, citizen scientists armed with air sensor networks feeds on air quality information to city of Melbourne. Yes, the city of Melbourne has its own network of air sensors but with citizen scientists, a lot of gaps in data are filled up. Whether it is related to CityLab or not, City of Melbourne developed online map showing all the trees in the council. Called as Melbourne Urban Forest Visual, users can click any of the trees and get basic information. For the council, it is great planning tool as they can identify which trees need to be removed (if they are dying soon). For users, interestingly, you can email to the tree and state your feelings towards each tree.

Map of Car Thefts incidents per council in greater Melbourne (The Age)

Third presentation was about data journalism and the speaker came from The Age (leading papers in Melbourne). Data journalism essentially is getting the story from datasets and nowadays, data literacy is critical for journalists to back up their stories with evidence. Data Visualization of particular topic (i.e. feelings of people of the day) opens more stories and exploring micro trends in local community. For example, The Age (with collaboration with Trend Maps) created a map of Twitter feelings in greater Melbourne. Interestingly,visualization presented concentration of positive feelings in shopping centres and negative feelings along train lines & stations. Similarly, in 2013, with collaboration with volunteers, The Age created a map of locations of high-risk cycling accidents in Melbourne.

Flink Labs project of visualizing trolley movements in shopping centres

Final presentation came from FlinkLabs. This company focuses heavily on data visualizations. Some of their interesting projects they did were visualizing movement of shopping trolleys in shopping centres (trolleys equipped with RFIDs) and dynamic cartograms of shifting world trademark applications from Japan to China. In the case of trolley movement, this visualization helps store managers to know which aisle is most frequented of and deduce some reasoning of the frequency (likewise for less frequented sections too). Visualization is growing rapidly as it creates 'techie' feel in every project done. However, the problem today a lot of visualization is data poor and lack of substance. This becomes more true with 3D printing as this visualization is not so much of data analytics tool, but for more artistic purposes. The speaker stated (following on what The Age journalist speaking on) data visualization is not the end product and it is tool for pattern analysis and stimulating ideas.

In short, Data visualization is converting long,wordy and boring Excel (and other data formats) into visual pleasing graphics to stimulate our thinking process. Expect more and more of data visualization in the years to come as tools of data visualization is being integrated more and more with our simple tools (i.e. Excel)