OpenTSDB: Scalable time-series database

What's OpenTSDB?

OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on top of HBase. OpenTSDB was written to address a common need: store, index and serve metrics collected from computer systems (network gear, operating systems, applications) at a large scale, and make this data easily accessible and graphable.

Thanks to HBase's scalability, OpenTSDB allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). OpenTSDB will never delete or downsample data and can easily store billions of data points. As a matter of fact,StumbleUpon uses it to keep track of hundred of thousands of time series and collects over 1 billion data points per day in their main production datacenter. Other sites such asBox orTumblr are pushing tens of billions of data points per day.

Imagine having the ability to quickly plot a graph showing the number ofDELETE statements going to your MySQL database along with the number of slow queries and temporary files created, and correlate this with the 99th percentile of your service's latency. OpenTSDB makes generating such graphs on the fly a trivial operation, while manipulating millions of data point for very fine grained, real-time monitoring.

- 다수의 사용자 요청을 빠른 시간 내에 모두 처리

- 서비스에서 읽기/쓰기 비율 중 데이터 쓰기 비율이 높음

- 저장된 데이터의 일관성이 중요하게 여겨짐

- 스키마가 없는 데이터 형태(schema-less)

- 네트워크 기반 분산 데이터베이스가 가진 확장성

* 저장과 처리 시간 주기에 맞춰 데이터 종류를 나누어 보자.

 - 빠른 주기로 빈번하게 저장 및 처리되는 캐시 영역의 데이터

     : 수 ms 단위로 데이터 처리

 - 일정 주기로 아카이빙돼 시점 복원이 가능하도록 관리되는 데이터

     : 수 분 이내로 이전 상태로 복구 가능한 데이터 처리

 - 일정 주기로 데이터를 집계해 게임 내에 다시 반영하는 데이터

    : 실시간 랭킹, 개인 요약 정보, 최근 아이템 교환 비율 등

 - 각종 사용자 이벤트 발생에 따라 그 이력을 관리하는 로그 데이터

* 저장형태에 따른 구분

 Key-value

 Ordered Key-value

 Wide Column Store

 Document

 Graph

* NoSQL - 어떻게 선택해야 할까? : http://platformadvisory.kr/archives/608

* 한 눈에 살펴보는 PostgreSQL : http://helloworld.naver.com/helloworld/227936

Pinba — A MySQL storage engine and monitoring server

Pinba is a MySQL plugin and storage engine that acts as a real-time monitoring and statistics server for PHP. Its main goal is to let developers monitor their PHP scripts, locate bottlenecks, and direct attention to the code that most needs it.

2. Installation requirements
--------------------------------

Pinba requires MySQL 5.1.x sources to be built and MySQL 5.1.x installation to run. MySQL 5.1 is the first version which supports pluggable storage engines, 
so older MySQL versions cannot and will never be supported.

Libraries required:
* Google Protocol Buffers - http://code.google.com/p/protobuf
* Judy - http://judy.sf.net
* libevent - http://monkey.org/~provos/libevent/
(this one may be already installed in your system, just make sure it's new enough (1.4+ is enough) and check that you have -devel package installed).

Optionally used:
* Hoard memory allocator - http://www.hoard.org

Using --with-hoard option you can add compiled-in Hoard support. It helps to reduce memory consumption (and Pinba is quite memory hungry when it comes to millions of timers). 2x smaller memory footprint with Hoard is what I regularly see.

 SQL Relay - Database connection pool with support for lots of languages and databases.

Download : http://sourceforge.net/projects/sqlrelay/ 

Looking for the latest version? Download sqlrelay-0.46.zip (3.5 MB)