Awesome list of datasets in 100+ categories

With an estimated 44 zettabytes of data in existence in our digital world today and approximately 2.5 quintillion bytes of new data generated daily, there is a lot of data out there you could tap into for your data science projects. It's pretty hard to curate through such a massive universe of data, but this collection is a great start. Here, you can find data from cancer genomes to UFO reports, as well as years of air quality data to 200,000 jokes. Dive into this ocean of data to explore as you learn how to apply data science techniques or leverage your expertise to discover something new.


By Etienne D. Noumen, Senior Software Engineer.

Data science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains.

In this blog, we provide links to popular open-source and public data sets, data visualizations, data analytics resources, and data lakes.

 

Table of Contents

 

  1. Latest complete Netflix movie dataset
  2. Common Crawl
  3. Dataset on protein prices
  4. CPOST dataset on suicide attacks over four decades
  5. Credit Card Dataset – Survey of Consumer Finances (SCF) Combined Extract Data 1989-2019
  6. Drone imagery with annotations for small object detection and tracking dataset
  7. NOAA High-Resolution Rapid Refresh (HRRR) Model
  8. Registry of Open Data on AWS
  9. Textbook Question Answering (TQA)
  10. Harmonized Cancer Datasets: Genomic Data Commons Data Portal
  11. The Cancer Genome Atlas
  12. Therapeutically Applicable Research to Generate Effective Treatments (TARGET)
  13. Genome Aggregation Database (gnomAD)
  14. SQuAD (Stanford Question Answering Dataset)
  15. PubMed Diabetes Dataset
  16. Drug-Target Interaction Dataset
  17. Pharmacogenomics Datasets
  18. Pancreatic Cancer Organoid Profiling
  19. Africa Soil Information Service (AfSIS) Soil Chemistry
  20. Dataset for Affective States in E-Environments
  21. NatureServe Explorer Dataset
  22. Flight Records in the US
  23. Worldwide flight data
  24. 2019 Crime statistics in the USA
  25. Yahoo Answers DataSets
  26. History of America 1400-2021
  27. Persian words phonetics dataset
  28. Historical Air Quality Dataset
  29. Stack Exchange Dataset
  30. Awesome Public Datasets
  31. Agriculture Datasets
  32. Biology Datasets
  33. Climate and Weather Datasets
  34. Complex Network Datasets
  35. Computer Network Datasets
  36. CyberSecurity Datasets
  37. Data Challenges Datasets
  38. Earth Science Datasets
  39. Economics Datasets
  40. Education Datasets
  41. Energy Datasets
  42. Entertainment Datasets
  43. Finance Datasets
  44. GIS Datasets
  45. Government Datasets
  46. Healthcare Datasets
  47. Image Processing Datasets
  48. Machine Learning Datasets
  49. Museums Datasets
  50. Natural Language Datasets
  51. Neuroscience Datasets
  52. Physics Datasets
  53. Prostate Cancer Datasets
  54. Psychology and Cognition Datasets
  55. Public Domains Datasets
  56. Search Engines Datasets
  57. Social Networks Datasets
  58. Social Sciences Datasets
  59. Software Datasets
  60. Sports Datasets
  61. Time Series Datasets
  62. Transportation Datasets
  63. eSports Datasets
  64. Complementary Collections
  65. Categorized list of public datasets: Sindre Sorhus /awesome List
  66. Platforms
  67. Programming Languages
  68. Front-End Development
  69. Back-End Development
  70. Computer Science
  71. Big Data
  72. Theory
  73. Books
  74. Editors
  75. Gaming
  76. Development Environment
  77. Entertainment
  78. Databases
  79. Media
  80. Learn
  81. Security
  82. Content Management Systems
  83. Hardware
  84. Business
  85. Work
  86. Networking
  87. Decentralized Systems
  88. Higher Education
  89. Events
  90. Testing
  91. Miscellaneous
  92. Related
  93. US Department of Education CRDC Dataset
  94. Nasa Dataset: sequencing data from bacteria before and after being taken to space
  95. All Trump’s twitter insults from 2015 to 2021 in CSV.
  96. Data is plural
  97. Global terrorism database
  98. The dolphin social network
  99. Dataset of 200,000 jokes
  100. The Million Song Dataset
  101. Cornell University’s eBird dataset
  102. UFO Report Dataset
  103. CDC’s Trend Drug Data
  104. Health and Retirement study: Public Survey data

This is a huge list, so here are here are 100+ more categories

 

Latest complete Netflix movie dataset

Created from 4 APIs. 11K+ rows and 30+ attributes of Netflix (Ratings, earnings, actors, language, availability, movie trailers, and many more)

Dataset on Kaggle.

Explore this dataset using FlixGem.com (this dataset is powering this webapp)

Dataset on Google Sheets.

 

Common Crawl

A corpus of web crawl data composed of over 50 billion web pages. The Common Crawl corpus contains petabytes of data collected since 2008. It contains raw web page data, extracted metadata and text extractions.

AWS CLI Access (No AWS account required)

aws s3 ls s3://commoncrawl/ --no-sign-request

s3://commoncrawl/crawl-data/CC-MAIN-2021-17 – April 2021

 

Dataset on protein prices

Data on Primary Commodity Prices are updated monthly based on the IMF’s Primary Commodity Price System.

Excel Database

 

CPOST dataset on suicide attacks over four decades

The University of Chicago Project on Security and Threats presents the updated and expanded Database on Suicide Attacks (DSAT), which now links to Uppsala Conflict Data Program data on armed conflicts and includes a new dataset measuring the alliance and rivalry relationships among militant groups with connections to suicide attack groups. Access it here.

 

Credit Card Dataset – Survey of Consumer Finances (SCF) Combined Extract Data 1989-2019

You can do a lot of aggregated analysis in a pretty straightforward way there.

 

Drone imagery with annotations for small object detection and tracking dataset

11 TB dataset of drone imagery with annotations for small object detection and tracking

Download and more information are available here

Dataset License: CDLA-Sharing-1.0

Helper scripts for accessing the dataset: DATASET.md

Dataset Exploration: Colab

 

NOAA High-Resolution Rapid Refresh (HRRR) Model

The HRRR is a NOAA real-time 3-km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-h period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh.

 

Registry of Open Data on AWS

This registry exists to help people discover and share datasets that are available via AWS resources. Learn more about sharing data on AWS.

See all usage examples for datasets listed in this registry.

See datasets from Digital Earth Africa, Facebook Data for Good, NASA Space Act Agreement, NIH STRIDES, NOAA Big Data Program, Space Telescope Science Institute, and Amazon Sustainability Data Initiative.

 

Textbook Question Answering (TQA)

1,076 textbook lessons, 26,260 questions, 6229 images

Documentation: https://allenai.org/data/tqa

Download

 

Harmonized Cancer Datasets: Genomic Data Commons Data Portal

The GDC Data Portal is a robust data-driven platform that allows cancer researchers and bioinformaticians to search and download cancer data for analysis.

Genomic Data Commons Data Portal

Genomic Data Commons Data Portal

 

The Cancer Genome Atlas

The Cancer Genome Atlas (TCGA), a collaboration between the National Cancer Institute (NCI) and National Human Genome Research Institute (NHGRI), aims to generate comprehensive, multi-dimensional maps of the key genomic changes in major types and subtypes of cancer.

AWS CLI Access (No AWS account required)

aws s3 ls s3://tcga-2-open/ --no-sign-request

 

Therapeutically Applicable Research to Generate Effective Treatments (TARGET)

The Therapeutically Applicable Research to Generate Effective Treatments (TARGET) program applies a comprehensive genomic approach to determine molecular changes that drive childhood cancers. The goal of the program is to use data to guide the development of effective, less toxic therapies. TARGET is organized into a collaborative network of disease-specific project teams. TARGET projects provide comprehensive molecular characterization to determine the genetic changes that drive the initiation and progression of childhood cancers. The dataset contains open Clinical Supplement, Biospecimen Supplement, RNA-Seq Gene Expression Quantification, miRNA-Seq Isoform Expression Quantification, miRNA-Seq miRNA Expression Quantification data from Genomic Data Commons (GDC), and open data from GDC Legacy Archive. Access it here.

 

Genome Aggregation Database (gnomAD)

The Genome Aggregation Database (gnomAD) is a resource developed by an international coalition of investigators that aggregates and harmonizes both exome and genome data from a wide range of large-scale human sequencing projects. The summary data provided here are released for the benefit of the wider scientific community without restriction on use. Downloads

 

SQuAD (Stanford Question Answering Dataset)

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. Access it here.

 

PubMed Diabetes Dataset

The Pubmed Diabetes dataset consists of 19717 scientific publications from PubMed database pertaining to diabetes classified into one of three classes. The citation network consists of 44338 links. Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words. The README file in the dataset provides more details.

Download Link

 

Drug-Target Interaction Dataset

This dataset contains interactions between drugs and targets collected from DrugBank, KEGG Drug, DCDB, and Matador. It was originally collected by Perlman et al. It contains 315 drugs, 250 targets, 1,306 drug-target interactions, 5 types of drug-drug similarities, and 3 types of target-target similarities. Drug-drug similarities include Chemical-based, Ligand-based, Expression-based, Side-effect-based, and Annotation-based similarities. Target-target similarities include Sequence-based, Protein-protein interaction network-based, and Gene Ontology-based similarities. The original task on the dataset is to predict new interactions between drugs and targets based on different types of similarities in the network. Download link

 

Pharmacogenomics Datasets

PharmGKB data and knowledge is available as downloads. It is often critical to check with their curators at feedback@pharmgkb.org before embarking on a large project using these data, to be sure that the files and data they make available are being interpreted correctly. PharmGKB generally does NOT need to be a co-author on such analyses; They just want to make sure that there is a correct understanding of our data before lots of resources are spent.

 

Pancreatic Cancer Organoid Profiling

The dataset contains open RNA-Seq Gene Expression Quantification data and controlled WGS/WXS/RNA-Seq Aligned Reads, WXS Annotated Somatic Mutation, WXS Raw Somatic Mutation, and RNA-Seq Splice Junction Quantification. Documentation

AWS CLI Access (No AWS account required)

aws s3 ls s3://gdc-organoid-pancreatic-phs001611-2-open/ --no-sign-request

 

Africa Soil Information Service (AfSIS) Soil Chemistry

This dataset contains soil infrared spectral data and paired soil property reference measurements for georeferenced soil samples that were collected through the Africa Soil Information Service (AfSIS) project, which lasted from 2009 through 2018. Documentation

AWS CLI Access (No AWS account required)

aws s3 ls s3://afsis/ --no-sign-request

 

Dataset for Affective States in E-Environments

DAiSEE is the first multi-label video classification dataset comprising of 9068 video snippets captured from 112 users for recognizing the user affective states of boredom, confusion, engagement, and frustration “in the wild”. The dataset has four levels of labels namely – very low, low, high, and very high for each of the affective states, which are crowd annotated and correlated with a gold standard annotation created using a team of expert psychologists. Download it here.

 

NatureServe Explorer Dataset

NatureServe Explorer provides conservation status, taxonomy, distribution, and life history information for more than 95,000 plants and animals in the United States and Canada, and more than 10,000 vegetation communities and ecological systems in the Western Hemisphere.

The data available through NatureServe Explorer represents data managed in the NatureServe Central Databases. These databases are dynamic, being continually enhanced and refined through the input of hundreds of natural heritage program scientists and other collaborators. NatureServe Explorer is updated from these central databases to reflect information from new field surveys, the latest taxonomic treatments and other scientific publications, and new conservation status assessments. Explore Data here

 

Flight Records in the US

Airline On-Time Performance and Causes of Flight Delays – On_Time Data.

This database contains scheduled and actual departure and arrival times, reason of delay. reported by certified U.S. air carriers that account for at least one percent of domestic scheduled passenger revenues. The data is collected by the Office of Airline Information, Bureau of Transportation Statistics (BTS).

FlightAware.com has data but you need to pay for a full dataset.

The anyflights package supplies a set of functions to generate air travel data (and data packages!) similar to nycflights13. With a user-defined year and airport, the anyflights function will grab data on:

  • flights: all flights that departed a given airport in a given year and month
  • weather: hourly meterological data for a given airport in a given year and month
  • airports: airport names, FAA codes, and locations
  • airlines: translation between two letter carrier (airline) codes and names
  • planes: construction information about each plane found in flights

Airline On-Time Statistics and Delay Causes

The U.S. Department of Transportation’s (DOT) Bureau of Transportation Statistics (BTS) tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled and diverted flights appears in DOT’s monthly Air Travel Consumer Report, published about 30 days after the month’s end, as well as in summary tables posted on this website. BTS began collecting details on the causes of flight delays in June 2003. Summary statistics and raw data are made available to the public at the time the Air Travel Consumer Report is released. Access it here

 

Worldwide flight data

Open flights: As of January 2017, the OpenFlights Airports Database contains over 10,000 airports, train stations and ferry terminals spanning the globe

Download: airports.dat (Airports only, high quality)

Download: airports-extended.dat (Airports, train stations and ferry terminals, including user contributions)

Bureau of Transportation:

Flightera.net seems to have a lot of good data for free. It has in-depth data on flights and doesn’t seem limited by date. I can’t speak on the validity of the data though.

flightradar24.com has lots of data, also historically, they might be willing to help you get it in a nice format.

 

2019 Crime statistics in the USA

Dataset with arrest in US by race and separate states. Download Excel here

 

Yahoo Answers DataSets

Yahoo is shutting down in 2021. This is Yahoo Answers datasets (300MB gzip) that is fairly extensive from 2015 with about 1.4m rows. This dataset has the best questions answers, I mean all the answers, including the most insane awful answers and the worst questions people put together. Download it here.

Another option here: According to the tracker, there are 77M done, 20M out(?), and 40M to go:

https://wiki.archiveteam.org/index.php/Yahoo!_Answers

 

History of America 1400-2021

Sources:

https://os-connect.com/pop/p2an.asp

https://ourworldindata.org/

http://www.ggdc.net/maddison/oriindex.htm

https://www.globalfirepower.com/countries-comparison.asp

 

Persian words phonetics dataset

This is a dataset of about 55K Persian words with their phonetics. Each word is in a line and separated from its phonetic by a tab.

 

Historical Air Quality Dataset

Air Quality Data Collected at Outdoor Monitors Across the US. This is a BigQuery Dataset. There are no files to download, but you can query it through Kernels using the BigQuery API. The AQS Data Mart is a database containing all of the information from AQS. It has every measured value the EPA has collected via the national ambient air monitoring program. It also includes the associated aggregate values calculated by EPA (8-hour, daily, annual, etc.). The AQS Data Mart is a copy of AQS made once per week and made accessible to the public through web-based applications. The intended users of the Data Mart are air quality data analysts in the regulatory, academic, and health research communities. It is intended for those who need to download large volumes of detailed technical data stored at EPA and does not provide any interactive analytical tools. It serves as the back-end database for several Agency interactive tools that could not fully function without it: AirData, AirCompare, The Remote Sensing Information Gateway, the Map Monitoring Sites KML page, etc.

 

Stack Exchange Dataset

https://data.stackexchange.com/

 

Awesome Public Datasets

This list of a topic-centric public data sources in high quality. They are collected and tidied from blogs, answers, and user responses. Most of the data sets listed below are free, however, some are not.

 

Agriculture

 

Biology

 

Climate and Climate and Weather

 

Complex Complex Network

 

Computer Network

 

CyberSecurity

 

Data Challenges

 

Earth Science Datasets

 

Economics Datasets

 

Education Datasets

 

Energy Datasets

 

Entertainment Datasets

 

Finance Datasets

 

GIS Datasets

 

Government Datasets

 

Healthcare Datasets

 

Image Processing Datasets

 

Machine Learning Datasets

 

Museums Datasets

 

Natural Language Datasets

 

Neuroscience Datasets

 

Physics Datasets

 

Prostate Cancer Datasets

 

Psychology and Cognition Datasets

 

Public Domains Datasets

 

Search Engines Datasets

 

Social Networks Datasets

 

Social Sciences Datasets

 

Software Datasets

 

Sports Datasets

 

Time Series Datasets

 

Transportation

 

eSports Datasets

 

Complementary Collections

 

Categorized list of public datasets: Sindre Sorhus /awesome List

 

Platforms

  • Node.js – Async non-blocking event-driven JavaScript runtime built on Chrome’s V8 JavaScript engine.
  • Frontend Development
  • iOS – Mobile operating system for Apple phones and tablets.
  • Android – Mobile operating system developed by Google.
  • IoT & Hybrid Apps
  • Electron – Cross-platform native desktop apps using JavaScript/HTML/CSS.
  • Cordova – JavaScript API for hybrid apps.
  • React Native – JavaScript framework for writing natively rendering mobile apps for iOS and Android.
  • Xamarin – Mobile app development IDE, testing, and distribution.
  • Linux
    • Containers
    • eBPF – Virtual machine that allows you to write more efficient and powerful tracing and monitoring for Linux systems.
    • Arch-based Projects – Linux distributions and projects based on Arch Linux.
  • macOS – Operating system for Apple’s Mac computers.
  • watchOS – Operating system for the Apple Watch.
  • JVM
  • Salesforce
  • Amazon Web Services
  • Windows
  • IPFS – P2P hypermedia protocol.
  • Fuse – Mobile development tools.
  • Heroku – Cloud platform as a service.
  • Raspberry Pi – Credit card-sized computer aimed at teaching kids programming, but capable of a lot more.
  • Qt – Cross-platform GUI app framework.
  • WebExtensions – Cross-browser extension system.
  • RubyMotion – Write cross-platform native apps for iOS, Android, macOS, tvOS, and watchOS in Ruby.
  • Smart TV – Create apps for different TV platforms.
  • GNOME – Simple and distraction-free desktop environment for Linux.
  • KDE – A free software community dedicated to creating an open and user-friendly computing experience.
  • .NET
    • Core
    • Roslyn – Open-source compilers and code analysis APIs for C# and VB.NET languages.
  • Amazon Alexa – Virtual home assistant.
  • DigitalOcean – Cloud computing platform designed for developers.
  • Flutter – Google’s mobile SDK for building native iOS and Android apps from a single codebase written in Dart.
  • Home Assistant – Open source home automation that puts local control and privacy first.
  • IBM Cloud – Cloud platform for developers and companies.
  • Firebase – App development platform built on Google Cloud Platform.
  • Robot Operating System 2.0 – Set of software libraries and tools that help you build robot apps.
  • Adafruit IO – Visualize and store data from any device.
  • Cloudflare – CDN, DNS, DDoS protection, and security for your site.
  • Actions on Google – Developer platform for Google Assistant.
  • ESP – Low-cost microcontrollers with WiFi and broad IoT applications.
  • Deno – A secure runtime for JavaScript and TypeScript that uses V8 and is built in Rust.
  • DOS – Operating system for x86-based personal computers that was popular during the 1980s and early 1990s.
  • Nix – Package manager for Linux and other Unix systems that makes package management reliable and reproducible.

 

Programming Languages

  • JavaScript
  • Swift – Apple’s compiled programming language that is secure, modern, programmer-friendly, and fast.
  • Python – General-purpose programming language designed for readability.
    • Asyncio – Asynchronous I/O in Python 3.
    • Scientific Audio – Scientific research in audio/music.
    • CircuitPython – A version of Python for microcontrollers.
    • Data Science – Data analysis and machine learning.
    • Typing – Optional static typing for Python.
    • MicroPython – A lean and efficient implementation of Python 3 for microcontrollers.
  • Rust
  • Haskell
  • PureScript
  • Go
  • Scala
    • Scala Native – Optimizing ahead-of-time compiler for Scala based on LLVM.
  • Ruby
  • Clojure
  • ClojureScript
  • Elixir
  • Elm
  • Erlang
  • Julia – High-level dynamic programming language designed to address the needs of high-performance numerical analysis and computational science.
  • Lua
  • C
  • C/C++ – General-purpose language with a bias toward system programming and embedded, resource-constrained software.
  • R – Functional programming language and environment for statistical computing and graphics.
  • D
  • Common Lisp – Powerful dynamic multiparadigm language that facilitates iterative and interactive development.
  • Perl
  • Groovy
  • Dart
  • Java – Popular secure object-oriented language designed for flexibility to “write once, run anywhere”.
  • Kotlin
  • OCaml
  • ColdFusion
  • Fortran
  • PHP – Server-side scripting language.
  • Pascal
  • AutoHotkey
  • AutoIt
  • Crystal
  • Frege – Haskell for the JVM.
  • CMake – Build, test, and package software.
  • ActionScript 3 – Object-oriented language targeting Adobe AIR.
  • Eta – Functional programming language for the JVM.
  • Idris – General purpose pure functional programming language with dependent types influenced by Haskell and ML.
  • Ada/SPARK – Modern programming language designed for large, long-lived apps where reliability and efficiency are essential.
  • Q# – Domain-specific programming language used for expressing quantum algorithms.
  • Imba – Programming language inspired by Ruby and Python and compiles to performant JavaScript.
  • Vala – Programming language designed to take full advantage of the GLib and GNOME ecosystems, while preserving the speed of C code.
  • Coq – Formal language and environment for programming and specification which facilitates interactive development of machine-checked proofs.
  • V – Simple, fast, safe, compiled language for developing maintainable software.

 

Front-End Development

 

Back-End Development

  • Flask – Python framework.
  • Docker
  • Vagrant – Automation virtual machine environment.
  • Pyramid – Python framework.
  • Play1 Framework
  • CakePHP – PHP framework.
  • Symfony – PHP framework.
  • Laravel – PHP framework.
    • Education
    • TALL Stack – Full-stack development solution featuring libraries built by the Laravel community.
  • Rails – Web app framework for Ruby.
    • Gems – Packages.
  • Phalcon – PHP framework.
  • Useful .htaccess Snippets
  • nginx – Web server.
  • Dropwizard – Java framework.
  • Kubernetes – Open-source platform that automates Linux container operations.
  • Lumen – PHP micro-framework.
  • Serverless Framework – Serverless computing and serverless architectures.
  • Apache Wicket – Java web app framework.
  • Vert.x – Toolkit for building reactive apps on the JVM.
  • Terraform – Tool for building, changing, and versioning infrastructure.
  • Vapor – Server-side development in Swift.
  • Dash – Python web app framework.
  • FastAPI – Python web app framework.
  • CDK – Open-source software development framework for defining cloud infrastructure in code.
  • IAM – User accounts, authentication and authorization.
  • Chalice – Python framework for serverless app development on AWS Lambda.

 

Computer Science

 

Big Data

  • Big Data
  • Public Datasets
  • Hadoop – Framework for distributed storage and processing of very large data sets.
  • Data Engineering
  • Streaming
  • Apache Spark – Unified engine for large-scale data processing.
  • Qlik – Business intelligence platform for data visualization, analytics, and reporting apps.
  • Splunk – Platform for searching, monitoring, and analyzing structured and unstructured machine-generated big data in real-time.

 

Theory

 

Books

 

Editors

 

Gaming

 

Development Environment

 

Entertainment

 

Databases

  • Database
  • MySQL
  • SQLAlchemy
  • InfluxDB
  • Neo4j
  • MongoDB – NoSQL database.
  • RethinkDB
  • TinkerPop – Graph computing framework.
  • PostgreSQL – Object-relational database.
  • CouchDB – Document-oriented NoSQL database.
  • HBase – Distributed, scalable, big data store.
  • NoSQL Guides – Help on using non-relational, distributed, open-source, and horizontally scalable databases.
  • Contexture – Abstracts queries/filters and results/aggregations from different backing data stores like ElasticSearch and MongoDB.
  • Database Tools – Everything that makes working with databases easier.
  • Grakn – Logical database to organize large and complex networks of data as one body of knowledge.

 

Media

 

Learn

 

Security

 

Content Management Systems

  • Umbraco
  • Refinery CMS – Ruby on Rails CMS.
  • Wagtail – Django CMS focused on flexibility and user experience.
  • Textpattern – Lightweight PHP-based CMS.
  • Drupal – Extensible PHP-based CMS.
  • Craft CMS – Content-first CMS.
  • Sitecore – .NET digital marketing platform that combines CMS with tools for managing multiple websites.
  • Silverstripe CMS – PHP MVC framework that serves as a classic or headless CMS.

 

Hardware

 

Business

 

Work

 

Networking

 

Decentralized Systems

  • Bitcoin – Bitcoin services and tools for software developers.
  • Ripple – Open source distributed settlement network.
  • Non-Financial Blockchain – Non-financial blockchain applications.
  • Mastodon – Open source decentralized microblogging network.
  • Ethereum – Distributed computing platform for smart contract development.
  • Blockchain AI – Blockchain projects for artificial intelligence and machine learning.
  • EOSIO – A decentralized operating system supporting industrial-scale apps.
  • Corda – Open source blockchain platform designed for business.
  • Waves – Open source blockchain platform and development toolset for Web 3.0 apps and decentralized solutions.
  • Substrate – Framework for writing scalable, upgradeable blockchains in Rust.

 

Higher Education

  • Computational Neuroscience – A multidisciplinary science which uses computational approaches to study the nervous system.
  • Digital History – Computer-aided scientific investigation of history.
  • Scientific Writing – Distraction-free scientific writing with Markdown, reStructuredText and Jupyter notebooks.

 

Events

 

Testing

  • Testing – Software testing.
  • Visual Regression Testing – Ensures changes did not break the functionality or style.
  • Selenium – Open-source browser automation framework and ecosystem.
  • Appium – Test automation tool for apps.
  • TAP – Test Anything Protocol.
  • JMeter – Load testing and performance measurement tool.
  • k6 – Open-source, developer-centric performance monitoring and load testing solution.
  • Playwright – Node.js library to automate Chromium, Firefox and WebKit with a single API.
  • Quality Assurance Roadmap – How to start & build a career in software testing.

 

Miscellaneous

 

Related

 

US Department of Education CRDC Dataset

The US Department of Ed has a dataset called the CRDC that collects data from all the public schools in the US and has demographic, academic, financial and all sorts of other fun data points. They also have corollary datasets that use the same identifier—an expansion pack if you may. It comes out every 2-3 years. Access it here.

 

Nasa Dataset: sequencing data from bacteria before and after being taken to space

NASA has some sequencing data from bacteria before and after being taken to space, to look at genetic differences caused by lack of gravity, radiation and others. Very fun if you want to try your hand at some bio data science. Access it here.

 

All Trump’s twitter insults from 2015 to 2021 in CSV.

Extracted from the NYT story: here

 

Data is plural

Data is Plural is a really good newsletter published by Jeremy Singer-Vine. The datasets are very random, but super interesting. Access it here.

 

Global terrorism database

Huge list of terrorism incidents from inside the US and abroad. Each entry has date and location of the incident, motivations, whether people or property were lost, the size of the attack, type of attack, etc. Access it here.

Terrorist Attacks Dataset: This dataset consists of 1293 terrorist attacks each assigned one of 6 labels indicating the type of the attack. Each attack is described by a 0/1-valued vector of attributes whose entries indicate the absence/presence of a feature. There are a total of 106 distinct features. The files in the dataset can be used to create two distinct graphs. The README file in the dataset provides more details. Download Link

Terrorists: This dataset contains information about terrorists and their relationships. This dataset was designed for classification experiments aimed at classifying the relationships among terrorists. The dataset contains 851 relationships, each described by a 0/1-valued vector of attributes where each entry indicates the absence/presence of a feature. There are a total of 1224 distinct features. Each relationship can be assigned one or more labels out of a maximum of four labels making this dataset suitable for multi-label classification tasks. The README file provides more details. Download Link

 

The dolphin social network

This network dataset is in the category of Social Networks. A social network of bottlenose dolphins. The dataset contains a list of all of links, where a link represents frequent associations between dolphins. Access it here

 

Dataset of 200,000 jokes

There are about 208 000 jokes in this database scraped from three sources.

Access it here

 

The Million Song Dataset

The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.

Its purposes are:

  • To encourage research on algorithms that scale to commercial sizes
  • To provide a reference dataset for evaluating research
  • As a shortcut alternative to creating a large dataset with APIs (e.g. The Echo Nest’s)
  • To help new researchers get started in the MIR field

 

Cornell University’s eBird dataset

Decades of observations of birds all around the world, truly an impressive way to leverage citizen science. Access it here.

 

UFO Report Dataset

NUFORC geolocated and time standardized ufo reports for close to a century of data. 80,000 plus reports. Access it here

 

CDC’s Trend Drug Data

The CDC has a public database called NAMCS/NHAMCS that allows you to trend drug data. It has a lot of other data points so it can be used for a variety of other reasons. Access it here.

 

Health and Retirement study: Public Survey data

A listing of publicly available biennial, off-year, and cross-year data products.

Example: COVID-19 Data

Year Product
2020 2020 HRS COVID-19 Project

 

Original. Reposted with permission.

 

Related:


More On This Topic

Get the FREE ebook 'The Great Big Natural Language Processing Primer' and 'The Complete Collection of Data Science Cheat Sheets' along with the leading newsletter on Data Science, Machine Learning, AI & Analytics straight to your inbox.



By subscribing you accept KDnuggets Privacy Policy





By subscribing you accept KDnuggets Privacy Policy

By subscribing you accept KDnuggets Privacy Policy