Dbid feature

Summary: Create a new type of file on startup if it doesn't already exist called DBID. This will store a unique number generated from boost library's uuid header file. The use-case is to identify the case of a db losing all its data and coming back up either empty or from an image(backup/live replica's recovery) the key point to note is that DBID is not stored in a backup or db snapshot It's preferable to use Boost for uuid because: 1) A non-standard way of generating uuid is not good 2) /proc/sys/kernel/random/uuid generates a uuid but only on linux environments and the solution would not be clean 3) c++ doesn't have any direct way to get a uuid 4) Boost is a very good library that was already having linkage in rocksdb from third-party Note: I had to update the TOOLCHAIN_REV in build files to get latest verison of boost from third-party as the older version had a bug. I had to put Wno-uninitialized in Makefile because boost-1.51 has an unitialized variable and rocksdb would not comiple otherwise. Latet open-source for boost is 1.54 but is not there in third-party. I have notified the concerned people in fbcode about it. @kailiu : While releasing to third-party, an additional dependency will need to be created for boost in TARGETS file. I can help identify. Test Plan: Expand db_test to test 2 cases 1) Restarting db with Id file present - verify that no change to Id 2)Restarting db with Id file deleted - verify that a different Id is there after reopen Also run make all check Reviewers: dhruba, haobo, kailiu, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13587
Disallow transaction log iterator to skip sequences
2026-11-06 02:52:35 -05:00 · 2013-10-22 12:23:34 -07:00 · 2013-10-22 11:45:35 -07:00 · 2013-10-21 10:23:56 -07:00 · 2013-10-20 22:02:05 -07:00 · 2013-10-17 22:15:57 -07:00
518 changed files with 3380 additions and 106584 deletions
@@ -1,3 +1,11 @@
+BSD License
+
+For rocksdb software
+
+Copyright (c) 2013, Facebook, Inc.
+All rights reserved.
+---------------------------------------------------------------------
+
 Copyright (c) 2011 The LevelDB Authors. All rights reserved.

 Redistribution and use in source and binary forms, with or without
@@ -48,6 +48,7 @@ TESTS = \
 	db_test \
 	dbformat_test \
 	env_test \
+	blob_store_test \
 	filelock_test \
 	filename_test \
 	filter_block_test \
@@ -156,7 +157,7 @@ valgrind_check: all $(PROGRAMS) $(TESTS)
 	done

 clean:
-	-rm -f $(PROGRAMS) $(BENCHMARKS) $(LIBRARY) $(SHARED) $(MEMENVLIBRARY) $(THRIFTSERVER) build_config.mk
+	-rm -f $(PROGRAMS) $(BENCHMARKS) $(LIBRARY) $(SHARED) $(MEMENVLIBRARY) build_config.mk
 	-rm -rf ios-x86/* ios-arm/*
 	-find . -name "*.[od]" | xargs --no-run-if-empty rm
 	-find . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" | xargs --no-run-if-empty rm
@@ -204,6 +205,9 @@ cache_test: util/cache_test.o $(LIBOBJECTS) $(TESTHARNESS)
 coding_test: util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(CXX) util/coding_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)

+blob_store_test: util/blob_store_test.o $(LIBOBJECTS) $(TESTHARNESS) $(TESTUTIL)
+	$(CXX) util/blob_store_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o$@ $(LDFLAGS) $(COVERAGEFLAGS)
+
 stringappend_test: utilities/merge_operators/string_append/stringappend_test.o $(LIBOBJECTS) $(TESTHARNESS)
 	$(CXX) utilities/merge_operators/string_append/stringappend_test.o $(LIBOBJECTS) $(TESTHARNESS) $(EXEC_LDFLAGS) -o $@ $(LDFLAGS) $(COVERAGEFLAGS)

@@ -0,0 +1,23 @@
+Additional Grant of Patent Rights
+
+“Software” means the rocksdb software distributed by Facebook, Inc.
+
+Facebook hereby grants you a perpetual, worldwide, royalty-free,
+non-exclusive, irrevocable (subject to the termination provision below)
+license under any rights in any patent claims owned by Facebook, to make,
+have made, use, sell, offer to sell, import, and otherwise transfer the
+Software. For avoidance of doubt, no license is granted under Facebook’s
+rights in any patent claims that are infringed by (i) modifications to the
+Software made by you or a third party, or (ii) the Software in combination
+with any software or other technology provided by you or a third party.
+
+The license granted hereunder will terminate, automatically and without
+notice, for anyone that makes any claim (including by filing any lawsuit,
+assertion or other action) alleging (a) direct, indirect, or contributory
+infringement or inducement to infringe any patent: (i) by Facebook or any
+of its subsidiaries or affiliates, whether or not such claim is related
+to the Software, (ii) by any party if such claim arises in whole or in
+part from any software, product or service of Facebook or any of its
+subsidiaries or affiliates, whether or not such claim is related to the
+Software, or (iii) by any party relating to the Software; or (b) that
+any right in any patent claim of Facebook is invalid or unenforceable.
@@ -134,14 +134,6 @@ $PWD/build_tools/build_detect_version
 # of all files matching either rule, so we need to append -print to make the
 # prune take effect.
 DIRS="util db table utilities"
-if test "$USE_THRIFT"; then
-  DIRS="$DIRS thrift/server_utils.cpp thrift/gen-cpp "
-  THRIFTSERVER=leveldb_server
-fi
-
-if test "$USE_SCRIBE"; then
-  DIRS="$DIRS scribe "
-fi

 set -f # temporarily disable globbing so that our patterns arent expanded
 PRUNE_TEST="-name *test*.cc -prune"
@@ -227,14 +219,6 @@ if test "$USE_HDFS"; then
  PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS $HDFS_LDFLAGS"
 fi

-# shall we build thrift server or scribe logger
-if test "$USE_THRIFT" ||  test "$USE_SCRIBE" ; then
-    THRIFT_CCFLAGS=" -I./thrift -I./thrift/gen-cpp -I./thrift/lib/cpp -I/usr/include -std=gnu++0x"
-    THRIFT_LDFLAGS=" -lexample -lserver -lthrift_base -ltransport -lthrift_exception -lutil -L./thrift/libs "
-    COMMON_FLAGS="$COMMON_FLAGS $THRIFT_CCFLAGS"
-    PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS $THRIFT_LDFLAGS"
-fi
-
 # if Intel SSE instruction set is supported, set USE_SSE=" -msse -msse4.2 "
 COMMON_FLAGS="$COMMON_FLAGS $USE_SSE"

@@ -254,5 +238,4 @@ echo "PLATFORM_SHARED_CFLAGS=$PLATFORM_SHARED_CFLAGS" >> $OUTPUT
 echo "PLATFORM_SHARED_EXT=$PLATFORM_SHARED_EXT" >> $OUTPUT
 echo "PLATFORM_SHARED_LDFLAGS=$PLATFORM_SHARED_LDFLAGS" >> $OUTPUT
 echo "PLATFORM_SHARED_VERSIONED=$PLATFORM_SHARED_VERSIONED" >> $OUTPUT
-echo "THRIFTSERVER=$THRIFTSERVER" >> $OUTPUT
 echo "EXEC_LDFLAGS=$EXEC_LDFLAGS" >> $OUTPUT
@@ -20,7 +20,7 @@ if [ "$?" = 0 ]; then
  BEGIN {
    print "#include \"build_version.h\"\n"
  }
-  { print "const char* leveldb_build_git_sha = \"leveldb_build_git_sha:" $0"\";" }
+  { print "const char* rocksdb_build_git_sha = \"rocksdb_build_git_sha:" $0"\";" }
  ' >  ${VFILE}
 else
  echo "git not found" |
@@ -28,13 +28,13 @@ else
    BEGIN {
      print "#include \"build_version.h\""
    }
-    { print "const char* leveldb_build_git_sha = \"leveldb_build_git_sha:git not found\";" }
+    { print "const char* rocksdb_build_git_sha = \"rocksdb_build_git_sha:git not found\";" }
    ' > ${VFILE}
 fi

-echo "const char* leveldb_build_git_datetime = \"leveldb_build_git_datetime:$(date)\";" >> ${VFILE}
-echo "const char* leveldb_build_compile_date = __DATE__;" >> ${VFILE}
-echo "const char* leveldb_build_compile_time = __TIME__;" >> ${VFILE}
+echo "const char* rocksdb_build_git_datetime = \"rocksdb_build_git_datetime:$(date)\";" >> ${VFILE}
+echo "const char* rocksdb_build_compile_date = __DATE__;" >> ${VFILE}
+echo "const char* rocksdb_build_compile_time = __TIME__;" >> ${VFILE}

 OUTFILE=$ROCKSDB_ROOT/util/build_version.cc
 if [ ! -e $OUTFILE ] || ! cmp -s $VFILE $OUTFILE; then
@@ -1,86 +0,0 @@
-#!/bin/bash -e
-function print_usage {
-  echo "Usage: $0 [build | release | bump_version version ]"
-}
-
-function check_env {
-  if [ -z $LEVELDB_HOME ] ; then
-    echo "Expect LEVELDB_HOME to be SET"
-    exit 1
-  fi
-  if [ -z $JAVA_HOME ] ; then
-    export JAVA_HOME=/usr/local/jdk-6u14-64/
-    echo "JAVA_HOME not set. Assuming JAVA_HOME=$JAVA_HOME"
-  fi
-  if [ -z $LEVELDBJNI_HOME ] ; then
-    export LEVELDBJNI_HOME=$LEVELDB_HOME/java/leveldbjni/
-    echo "LEVELDBJNI_HOME not set. Assuming LEVELDBJNI_HOME=$LEVELDBJNI_HOME"
-  fi
-  if [ -z $SNAPPY_HOME ] ; then
-    export SNAPPY_HOME="/home/dhruba/snappy-1.0.5"
-    echo "SNAPPY_HOME not set. Assuming SNAPPY_HOME=$SNAPPY_HOME"
-  fi
-  if [ -z $LEVELDB_PATCH ] ; then 
-    LEVELDB_PATCH=$LEVELDB_HOME/java/leveldbjni/db.h.patch
-    echo "LEVELDB_PATCH not set. Assuming LEVELDB_PATCH=$LEVELDB_PATCH"
-  fi
-}
-
-function build {
-  cd $LEVELDB_HOME
-  git apply $LEVELDB_PATCH
-  make clean libleveldb.a
-  cd $LEVELDB_HOME/java/leveldb/leveldb-api
-  mvn clean package
-  cd $LEVELDBJNI_HOME
-  mvn clean install -P linux64
-  cd $LEVELDB_HOME
-  git checkout $LEVELDB_HOME/include/leveldb/db.h
-}
-
-function release {
-  cd $LEVELDB_HOME
-  git apply $LEVELDB_PATCH
-  make clean libleveldb.a
-  cd $LEVELDB_HOME/java/leveldb/leveldb-api
-  mvn clean package
-  cd $LEVELDBJNI_HOME
-  mvn deploy -P linux64 -DskipTests
-  cd $LEVELDB_HOME
-  git checkout $LEVELDB_HOME/include/leveldb/db.h
-}
-
-CMD=$1
-if [ -z $CMD ]; then
-  print_usage
-  exit 1
-fi
-
-case "$CMD" in 
-  build) 
-    check_env
-    build
-    ;;
-  bump_version)
-    if [ -z $2 ] ; then
-      echo "bump_version requires a version.no parameter at the end"
-      exit 1
-    fi 
-    check_env
-    VERSION=$2
-    pushd $LEVELDBJNI_HOME
-    mvn versions:set -DnewVersion="$VERSION"
-    popd
-    echo "VERSION SET TO $VERSION"
-    ;;
-  release)
-    check_env
-    pushd $LEVELDBJNI_HOME
-    release
-    popd
-    ;;
-  *)
-    print_usage
-    exit 1
-    ;;
-esac
@@ -4,21 +4,16 @@
 # fbcode settings.  It uses the latest g++ compiler and also
 # uses jemalloc

-TOOLCHAIN_REV=f365dbeae46a30414a2874a6f45e73e10f1caf7d
+TOOLCHAIN_REV=fbe3b095a4cc4a3713730050d182b7b4a80c342f
 TOOLCHAIN_EXECUTABLES="/mnt/gvfs/third-party/$TOOLCHAIN_REV/centos5.2-native"
 TOOLCHAIN_LIB_BASE="/mnt/gvfs/third-party/$TOOLCHAIN_REV/gcc-4.7.1-glibc-2.14.1"
-TOOL_JEMALLOC=jemalloc-3.0.0/2f45f3a
+TOOL_JEMALLOC=jemalloc-3.3.1/9202ce3
 GLIBC_RUNTIME_PATH=/usr/local/fbcode/gcc-4.7.1-glibc-2.14.1

 # location of snappy headers and libraries
 SNAPPY_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/include"
 SNAPPY_LIBS=" $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/lib/libsnappy.a"

-# location of boost headers and libraries
-THRIFT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/bef9365/include -std=gnu++0x"
-THRIFT_INCLUDE+=" -I./thrift -I./thrift/gen-cpp -I./thrift/lib/cpp"
-THRIFT_LIBS=" -L $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/bef9365/lib"
-
 # location of libevent
 LIBEVENT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/include"
 LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
@@ -27,7 +22,7 @@ LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
 export USE_SSE=" -msse -msse4.2 "

 CC="$TOOLCHAIN_EXECUTABLES/clang/clang-3.1/6ca8a59/bin/clang $CLANG_INCLUDES"
-CXX="$TOOLCHAIN_EXECUTABLES/clang/clang-3.1/6ca8a59/bin/clang++ $CLANG_INCLUDES $JINCLUDE $SNAPPY_INCLUDE $THRIFT_INCLUDE $LIBEVENT_INCLUDE"
+CXX="$TOOLCHAIN_EXECUTABLES/clang/clang-3.1/6ca8a59/bin/clang++ $CLANG_INCLUDES $JINCLUDE $SNAPPY_INCLUDE $LIBEVENT_INCLUDE"
 AR=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ar
 RANLIB=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ranlib

@@ -37,8 +32,8 @@ CXXFLAGS="$CFLAGS -nostdinc++ -std=gnu++0x"
 CFLAGS+=" -I $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/include -DHAVE_JEMALLOC"

 EXEC_LDFLAGS=" -Wl,--whole-archive $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/lib/libjemalloc.a"
-EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/91ddd43/lib/libunwind.a"
-EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $THRIFT_LIBS $LIBEVENT_LIBS"
+EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/350336c/lib/libunwind.a"
+EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $LIBEVENT_LIBS"
 EXEC_LDFLAGS+=" -Wl,--dynamic-linker,$GLIBC_RUNTIME_PATH/lib/ld-linux-x86-64.so.2"
 EXEC_LDFLAGS+=" -B$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin"

@@ -4,10 +4,10 @@
 # fbcode settings.  It uses the latest g++ compiler and also
 # uses jemalloc

-TOOLCHAIN_REV=83eb773e262fa705eaebbf3de40db71d53fabf2e
+TOOLCHAIN_REV=fbe3b095a4cc4a3713730050d182b7b4a80c342f
 TOOLCHAIN_EXECUTABLES="/mnt/gvfs/third-party/$TOOLCHAIN_REV/centos5.2-native"
 TOOLCHAIN_LIB_BASE="/mnt/gvfs/third-party/$TOOLCHAIN_REV/gcc-4.7.1-glibc-2.14.1"
-TOOL_JEMALLOC=jemalloc-3.3.1/2f45f3a
+TOOL_JEMALLOC=jemalloc-3.3.1/9202ce3

 # location of libhdfs libraries
 if test "$USE_HDFS"; then
@@ -27,11 +27,6 @@ SNAPPY_LIBS=" $TOOLCHAIN_LIB_BASE/snappy/snappy-1.0.3/7518bbe/lib/libsnappy.a"
 ZLIB_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/zlib/zlib-1.2.5/91ddd43/include"
 ZLIB_LIBS=" $TOOLCHAIN_LIB_BASE/zlib/zlib-1.2.5/91ddd43/lib/libz.a"

-# location of boost headers and libraries
-THRIFT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/2a0840d/include"
-THRIFT_INCLUDE+=" -I./thrift -I./thrift/gen-cpp -I./thrift/lib/cpp"
-THRIFT_LIBS=" -L $TOOLCHAIN_LIB_BASE/boost/boost-1.48.0/2a0840d/lib"
-
 # location of libevent
 LIBEVENT_INCLUDE=" -I $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/include"
 LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
@@ -40,7 +35,7 @@ LIBEVENT_LIBS=" -L $TOOLCHAIN_LIB_BASE/libevent/libevent-1.4.14b/91ddd43/lib"
 export USE_SSE=" -msse -msse4.2 "

 CC="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/gcc"
-CXX="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/g++ $JINCLUDE $SNAPPY_INCLUDE $ZLIB_INCLUDE $THRIFT_INCLUDE $LIBEVENT_INCLUDE"
+CXX="$TOOLCHAIN_EXECUTABLES/gcc/gcc-4.7.1-glibc-2.14.1/bin/g++ $JINCLUDE $SNAPPY_INCLUDE $ZLIB_INCLUDE $LIBEVENT_INCLUDE"
 AR=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ar
 RANLIB=$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/da39a3e/bin/ranlib

@@ -48,8 +43,8 @@ CFLAGS="-B$TOOLCHAIN_EXECUTABLES/binutils/binutils-2.21.1/bin/gold -m64 -mtune=g
 CFLAGS+=" -I $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/include -DHAVE_JEMALLOC"

 EXEC_LDFLAGS=" -Wl,--whole-archive $TOOLCHAIN_LIB_BASE/jemalloc/$TOOL_JEMALLOC/lib/libjemalloc.a"
-EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/91ddd43/lib/libunwind.a"
-EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $ZLIB_LIBS $THRIFT_LIBS $LIBEVENT_LIBS"
+EXEC_LDFLAGS+=" -Wl,--no-whole-archive $TOOLCHAIN_LIB_BASE/libunwind/libunwind-1.0.1/350336c/lib/libunwind.a"
+EXEC_LDFLAGS+=" $HDFSLIB $SNAPPY_LIBS $ZLIB_LIBS $LIBEVENT_LIBS"

 PLATFORM_LDFLAGS="-L$TOOLCHAIN_LIB_BASE/libgcc/libgcc-4.7.1/afc21dc/lib -L$TOOLCHAIN_LIB_BASE/glibc/glibc-2.14.1/99df8fc/lib"

@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -25,7 +30,8 @@ Status BuildTable(const std::string& dbname,
                  FileMetaData* meta,
                  const Comparator* user_comparator,
                  const SequenceNumber newest_snapshot,
-                  const SequenceNumber earliest_seqno_in_memtable) {
+                  const SequenceNumber earliest_seqno_in_memtable,
+                  const bool enable_compression) {
  Status s;
  meta->file_size = 0;
  meta->smallest_seqno = meta->largest_seqno = 0;
@@ -46,7 +52,8 @@ Status BuildTable(const std::string& dbname,
    if (!s.ok()) {
      return s;
    }
-    TableBuilder* builder = new TableBuilder(options, file.get(), 0);
+    TableBuilder* builder = new TableBuilder(options, file.get(), 0,
+                                             enable_compression);

    // the first key is the smallest key
    Slice key = iter->key();
@@ -1,10 +1,11 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
-
-#ifndef STORAGE_LEVELDB_DB_BUILDER_H_
-#define STORAGE_LEVELDB_DB_BUILDER_H_
-
+#pragma once
 #include "rocksdb/comparator.h"
 #include "rocksdb/status.h"
 #include "rocksdb/types.h"
@@ -34,8 +35,7 @@ extern Status BuildTable(const std::string& dbname,
                         FileMetaData* meta,
                         const Comparator* user_comparator,
                         const SequenceNumber newest_snapshot,
-                         const SequenceNumber earliest_seqno_in_memtable);
+                         const SequenceNumber earliest_seqno_in_memtable,
+                         const bool enable_compression);

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_BUILDER_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -296,7 +296,7 @@ int main(int argc, char** argv) {
  {
    char* prop = leveldb_property_value(db, "nosuchprop");
    CheckCondition(prop == NULL);
-    prop = leveldb_property_value(db, "leveldb.stats");
+    prop = leveldb_property_value(db, "rocksdb.stats");
    CheckCondition(prop != NULL);
    Free(&prop);
  }
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -230,7 +235,7 @@ TEST(CorruptionTest, NewFileErrorDuringWrite) {
 TEST(CorruptionTest, TableFile) {
  Build(100);
  DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
  dbi->TEST_CompactRange(0, nullptr, nullptr);
  dbi->TEST_CompactRange(1, nullptr, nullptr);

@@ -241,7 +246,7 @@ TEST(CorruptionTest, TableFile) {
 TEST(CorruptionTest, TableFileIndexData) {
  Build(10000);  // Enough to build multiple Tables
  DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();

  Corrupt(kTableFile, -2000, 500);
  Reopen();
@@ -279,7 +284,7 @@ TEST(CorruptionTest, SequenceNumberRecovery) {
 TEST(CorruptionTest, CorruptedDescriptor) {
  ASSERT_OK(db_->Put(WriteOptions(), "foo", "hello"));
  DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
  dbi->TEST_CompactRange(0, nullptr, nullptr);

  Corrupt(kDescriptorFile, 0, 1000);
@@ -296,9 +301,9 @@ TEST(CorruptionTest, CorruptedDescriptor) {
 TEST(CorruptionTest, CompactionInputError) {
  Build(10);
  DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
  const int last = dbi->MaxMemCompactionLevel();
-  ASSERT_EQ(1, Property("leveldb.num-files-at-level" + NumberToString(last)));
+  ASSERT_EQ(1, Property("rocksdb.num-files-at-level" + NumberToString(last)));

  Corrupt(kTableFile, 100, 1);
  Check(9, 9);
@@ -319,12 +324,13 @@ TEST(CorruptionTest, CompactionInputErrorParanoid) {
  for (int level = 1; level < dbi->NumberLevels(); level++) {
    dbi->Put(WriteOptions(), "", "begin");
    dbi->Put(WriteOptions(), "~", "end");
-    dbi->TEST_CompactMemTable();
+    dbi->TEST_FlushMemTable();
  }

  Build(10);
-  dbi->TEST_CompactMemTable();
-  ASSERT_EQ(1, Property("leveldb.num-files-at-level0"));
+  dbi->TEST_FlushMemTable();
+  dbi->TEST_WaitForCompact();
+  ASSERT_EQ(1, Property("rocksdb.num-files-at-level0"));

  Corrupt(kTableFile, 100, 1);
  Check(9, 9);
@@ -341,7 +347,7 @@ TEST(CorruptionTest, CompactionInputErrorParanoid) {
 TEST(CorruptionTest, UnrelatedKeys) {
  Build(10);
  DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
  Corrupt(kTableFile, 100, 1);

  std::string tmp1, tmp2;
@@ -349,7 +355,7 @@ TEST(CorruptionTest, UnrelatedKeys) {
  std::string v;
  ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
  ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
-  dbi->TEST_CompactMemTable();
+  dbi->TEST_FlushMemTable();
  ASSERT_OK(db_->Get(ReadOptions(), Key(1000, &tmp1), &v));
  ASSERT_EQ(Value(1000, &tmp2).ToString(), v);
 }
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -196,6 +201,8 @@ static const char* FLAGS_db = nullptr;
 // if FLAGS_cache_size is non-negative.
 static int FLAGS_cache_numshardbits = -1;

+static int FLAGS_cache_remove_scan_count_limit = 32;
+
 // Verify checksum for every block read from storage
 static bool FLAGS_verify_checksum = false;

@@ -546,7 +553,7 @@ class Stats {

        if (FLAGS_stats_per_interval) {
          std::string stats;
-          if (db && db->GetProperty("leveldb.stats", &stats))
+          if (db && db->GetProperty("rocksdb.stats", &stats))
            fprintf(stderr, "%s\n", stats.c_str());
        }

@@ -816,7 +823,8 @@ class Benchmark {
  Benchmark()
  : cache_(FLAGS_cache_size >= 0 ?
           (FLAGS_cache_numshardbits >= 1 ?
-            NewLRUCache(FLAGS_cache_size, FLAGS_cache_numshardbits) :
+            NewLRUCache(FLAGS_cache_size, FLAGS_cache_numshardbits,
+                        FLAGS_cache_remove_scan_count_limit) :
            NewLRUCache(FLAGS_cache_size)) : nullptr),
    filter_policy_(FLAGS_bloom_bits >= 0
                   ? NewBloomFilterPolicy(FLAGS_bloom_bits)
@@ -985,11 +993,11 @@ class Benchmark {
      } else if (name == Slice("heapprofile")) {
        HeapProfile();
      } else if (name == Slice("stats")) {
-        PrintStats("leveldb.stats");
+        PrintStats("rocksdb.stats");
      } else if (name == Slice("levelstats")) {
-        PrintStats("leveldb.levelstats");
+        PrintStats("rocksdb.levelstats");
      } else if (name == Slice("sstables")) {
-        PrintStats("leveldb.sstables");
+        PrintStats("rocksdb.sstables");
      } else {
        if (name != Slice()) {  // No error message for empty name
          fprintf(stderr, "unknown benchmark '%s'\n", name.ToString().c_str());
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
@@ -75,7 +80,7 @@ Status DBImpl::GetSortedWalFiles(VectorLogPtr& files) {

  // list wal files in archive dir.
  Status s;
-  std::string archivedir = ArchivalDirectory(dbname_);
+  std::string archivedir = ArchivalDirectory(options_.wal_dir);
  if (env_->FileExists(archivedir)) {
    s = AppendSortedWalsOfType(archivedir, files, kArchivedLogFile);
    if (!s.ok()) {
@@ -83,19 +88,19 @@ Status DBImpl::GetSortedWalFiles(VectorLogPtr& files) {
    }
  }
  // list wal files in main db dir.
-  return AppendSortedWalsOfType(dbname_, files, kAliveLogFile);
+  return AppendSortedWalsOfType(options_.wal_dir, files, kAliveLogFile);
 }

 Status DBImpl::DeleteWalFiles(const VectorLogPtr& files) {
  Status s;
-  std::string archivedir = ArchivalDirectory(dbname_);
+  std::string archivedir = ArchivalDirectory(options_.wal_dir);
  std::string files_not_deleted;
  for (const auto& wal : files) {
    /* Try deleting in the dir that pathname points to for the logfile.
       This may fail if we try to delete a log file which was live when captured
       but is archived now. Try deleting it from archive also
     */
-    Status st = env_->DeleteFile(dbname_ + "/" + wal->PathName());
+    Status st = env_->DeleteFile(options_.wal_dir + "/" + wal->PathName());
    if (!st.ok()) {
      if (wal->Type() == kAliveLogFile &&
          env_->DeleteFile(LogFileName(archivedir, wal->LogNumber())).ok()) {
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -179,6 +184,11 @@ Options SanitizeOptions(const std::string& dbname,
      Log(result.info_log, "Prefix hash memtable rep is in use.");
    }
  }
+
+  if (result.wal_dir.empty()) {
+    // Use dbname as default
+    result.wal_dir = dbname;
+  }
  return result;
 }

@@ -245,10 +255,6 @@ DBImpl::DBImpl(const Options& options, const std::string& dbname)
  dumpLeveldbBuildVersion(options_.info_log.get());
  options_.Dump(options_.info_log.get());

-#ifdef USE_SCRIBE
-  logger_.reset(new ScribeLogger("localhost", 1456));
-#endif
-
  char name[100];
  Status st = env_->GetHostName(name, 100L);
  if (st.ok()) {
@@ -357,7 +363,7 @@ void DBImpl::MaybeIgnoreError(Status* s) const {

 const Status DBImpl::CreateArchivalDirectory() {
  if (options_.WAL_ttl_seconds > 0) {
-    std::string archivalPath = ArchivalDirectory(dbname_);
+    std::string archivalPath = ArchivalDirectory(options_.wal_dir);
    return env_->CreateDirIfMissing(archivalPath);
  }
  return Status::OK();
@@ -386,7 +392,7 @@ void DBImpl::MaybeDumpStats() {
    // period in rare cases.
    last_stats_dump_time_microsec_ = now_micros;
    std::string stats;
-    GetProperty("leveldb.stats", &stats);
+    GetProperty("rocksdb.stats", &stats);
    Log(options_.info_log, "%s", stats.c_str());
    PrintStatistics();
  }
@@ -423,6 +429,17 @@ void DBImpl::FindObsoleteFiles(DeletionState& deletion_state) {
  // set of all files in the directory
  env_->GetChildren(dbname_, &deletion_state.allfiles); // Ignore errors

+  //Add log files in wal_dir
+  if (options_.wal_dir != dbname_) {
+    std::vector<std::string> log_files;
+    env_->GetChildren(options_.wal_dir, &log_files); // Ignore errors
+    deletion_state.allfiles.insert(
+      deletion_state.allfiles.end(),
+      log_files.begin(),
+      log_files.end()
+    );
+  }
+
  // store the current filenum, lognum, etc
  deletion_state.filenumber = versions_->ManifestFileNumber();
  deletion_state.lognumber = versions_->LogNumber();
@@ -431,10 +448,10 @@ void DBImpl::FindObsoleteFiles(DeletionState& deletion_state) {

 Status DBImpl::DeleteLogFile(uint64_t number) {
  Status s;
-  auto filename = LogFileName(dbname_, number);
+  auto filename = LogFileName(options_.wal_dir, number);
  if (options_.WAL_ttl_seconds > 0) {
    s = env_->RenameFile(filename,
-                         ArchivedLogFileName(dbname_, number));
+                         ArchivedLogFileName(options_.wal_dir, number));

    if (!s.ok()) {
      Log(options_.info_log, "RenameFile logfile #%lu FAILED", number);
@@ -452,7 +469,7 @@ Status DBImpl::DeleteLogFile(uint64_t number) {
 // Diffs the files listed in filenames and those that do not
 // belong to live files are posibly removed. If the removed file
 // is a sst file, then it returns the file number in files_to_evict.
-// It is not necesary to hold the mutex when invoking this method.
+// It is not necessary to hold the mutex when invoking this method.
 void DBImpl::PurgeObsoleteFiles(DeletionState& state) {
  uint64_t number;
  FileType type;
@@ -492,6 +509,7 @@ void DBImpl::PurgeObsoleteFiles(DeletionState& state) {
          break;
        case kCurrentFile:
        case kDBLockFile:
+        case kIdentityFile:
        case kMetaDatabase:
          keep = true;
          break;
@@ -561,7 +579,7 @@ void DBImpl::PurgeObsoleteWALFiles() {
      return;
    }
    std::vector<std::string> wal_files;
-    std::string archival_dir = ArchivalDirectory(dbname_);
+    std::string archival_dir = ArchivalDirectory(options_.wal_dir);
    env_->GetChildren(archival_dir, &wal_files);
    for (const auto& f : wal_files) {
      uint64_t file_m_time;
@@ -622,6 +640,13 @@ Status DBImpl::Recover(VersionEdit* edit, MemTable* external_table,
            dbname_, "exists (error_if_exists is true)");
      }
    }
+    // Check for the IDENTITY file and create it if not there
+    if (!env_->FileExists(IdentityFileName(dbname_))) {
+      s = SetIdentityFile(env_, dbname_);
+      if (!s.ok()) {
+        return s;
+      }
+    }
  }

  Status s = versions_->Recover();
@@ -634,11 +659,11 @@ Status DBImpl::Recover(VersionEdit* edit, MemTable* external_table,
    //
    // Note that PrevLogNumber() is no longer used, but we pay
    // attention to it in case we are recovering a database
-    // produced by an older version of leveldb.
+    // produced by an older version of rocksdb.
    const uint64_t min_log = versions_->LogNumber();
    const uint64_t prev_log = versions_->PrevLogNumber();
    std::vector<std::string> filenames;
-    s = env_->GetChildren(dbname_, &filenames);
+    s = env_->GetChildren(options_.wal_dir, &filenames);
    if (!s.ok()) {
      return s;
    }
@@ -705,7 +730,7 @@ Status DBImpl::RecoverLogFile(uint64_t log_number,
  mutex_.AssertHeld();

  // Open the log file
-  std::string fname = LogFileName(dbname_, log_number);
+  std::string fname = LogFileName(options_.wal_dir, log_number);
  unique_ptr<SequentialFile> file;
  Status status = env_->NewSequentialFile(fname, &file, storage_options_);
  if (!status.ok()) {
@@ -804,7 +829,7 @@ Status DBImpl::WriteLevel0TableForRecovery(MemTable* mem, VersionEdit* edit) {
    s = BuildTable(dbname_, env_, options_, storage_options_,
                   table_cache_.get(), iter, &meta,
                   user_comparator(), newest_snapshot,
-                   earliest_seqno_in_memtable);
+                   earliest_seqno_in_memtable, true);
    mutex_.Lock();
  }

@@ -860,10 +885,15 @@ Status DBImpl::WriteLevel0Table(std::vector<MemTable*> &mems, VersionEdit* edit,
  Status s;
  {
    mutex_.Unlock();
+    // We skip compression if universal compression is used and the size
+    // threshold is set for compression.
+    bool enable_compression = (options_.compaction_style
+        != kCompactionStyleUniversal ||
+        options_.compaction_options_universal.compression_size_percent < 0);
    s = BuildTable(dbname_, env_, options_, storage_options_,
                   table_cache_.get(), iter, &meta,
                   user_comparator(), newest_snapshot,
-                   earliest_seqno_in_memtable);
+                   earliest_seqno_in_memtable, enable_compression);
    mutex_.Lock();
  }
  base->Unref();
@@ -912,13 +942,13 @@ Status DBImpl::WriteLevel0Table(std::vector<MemTable*> &mems, VersionEdit* edit,
  return s;
 }

-Status DBImpl::CompactMemTable(bool* madeProgress) {
+Status DBImpl::FlushMemTableToOutputFile(bool* madeProgress) {
  mutex_.AssertHeld();
  assert(imm_.size() != 0);

  if (!imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
-    Log(options_.info_log, "Memcompaction already in progress");
-    Status s = Status::IOError("Memcompaction already in progress");
+    Log(options_.info_log, "FlushMemTableToOutputFile already in progress");
+    Status s = Status::IOError("FlushMemTableToOutputFile already in progress");
    return s;
  }

@@ -986,7 +1016,7 @@ void DBImpl::CompactRange(const Slice* begin, const Slice* end,
      }
    }
  }
-  TEST_CompactMemTable(); // TODO(sanjay): Skip if memtable does not overlap
+  TEST_FlushMemTable(); // TODO(sanjay): Skip if memtable does not overlap
  for (int level = 0; level < max_level_with_files; level++) {
    TEST_CompactRange(level, begin, end);
  }
@@ -1105,24 +1135,18 @@ Status DBImpl::GetUpdatesSince(SequenceNumber seq,
    return s;
  }

-  if (wal_files->empty()) {
-    return Status::IOError(" NO WAL Files present in the db");
-  }
-  //  std::shared_ptr would have been useful here.
-
  s = RetainProbableWalFiles(*wal_files, seq);
  if (!s.ok()) {
    return s;
  }
  iter->reset(
-    new TransactionLogIteratorImpl(dbname_,
+    new TransactionLogIteratorImpl(options_.wal_dir,
                                   &options_,
                                   storage_options_,
                                   seq,
                                   std::move(wal_files),
                                   &last_flushed_sequence_));
-  iter->get()->Next();
-  return iter->get()->status();
+  return (*iter)->status();
 }

 Status DBImpl::RetainProbableWalFiles(VectorLogPtr& all_logs,
@@ -1151,7 +1175,8 @@ Status DBImpl::RetainProbableWalFiles(VectorLogPtr& all_logs,
 bool DBImpl::CheckWalFileExistsAndEmpty(const WalFileType type,
                                        const uint64_t number) {
  const std::string fname = (type == kAliveLogFile) ?
-    LogFileName(dbname_, number) : ArchivedLogFileName(dbname_, number);
+    LogFileName(options_.wal_dir, number) :
+    ArchivedLogFileName(options_.wal_dir, number);
  uint64_t file_size;
  Status s = env_->GetFileSize(fname, &file_size);
  return (s.ok() && (file_size == 0));
@@ -1161,23 +1186,24 @@ Status DBImpl::ReadFirstRecord(const WalFileType type, const uint64_t number,
                               WriteBatch* const result) {

  if (type == kAliveLogFile) {
-    std::string fname = LogFileName(dbname_, number);
+    std::string fname = LogFileName(options_.wal_dir, number);
    Status status = ReadFirstLine(fname, result);
    if (!status.ok()) {
      //  check if the file got moved to archive.
-      std::string archived_file = ArchivedLogFileName(dbname_, number);
+      std::string archived_file =
+        ArchivedLogFileName(options_.wal_dir, number);
      Status s = ReadFirstLine(archived_file, result);
      if (!s.ok()) {
-        return Status::IOError("Log File has been deleted");
+        return Status::IOError("Log File has been deleted: " + archived_file);
      }
    }
    return Status::OK();
  } else if (type == kArchivedLogFile) {
-    std::string fname = ArchivedLogFileName(dbname_, number);
+    std::string fname = ArchivedLogFileName(options_.wal_dir, number);
    Status status = ReadFirstLine(fname, result);
    return status;
  }
-  return Status::NotSupported("File Type Not Known");
+  return Status::NotSupported("File Type Not Known: " + type);
 }

 Status DBImpl::ReadFirstLine(const std::string& fname,
@@ -1331,7 +1357,7 @@ void DBImpl::TEST_CompactRange(int level, const Slice* begin,const Slice* end) {
    if (bg_compaction_scheduled_ == LargeNumber) {
      bg_compaction_scheduled_ = newvalue;
    }
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
    while (manual_compaction_ == &manual) {
      bg_cv_.Wait();
    }
@@ -1352,12 +1378,12 @@ Status DBImpl::FlushMemTable(const FlushOptions& options) {
  Status s = Write(WriteOptions(), nullptr);
  if (s.ok() && options.wait) {
    // Wait until the compaction completes
-    s = WaitForCompactMemTable();
+    s = WaitForFlushMemTable();
  }
  return s;
 }

-Status DBImpl::WaitForCompactMemTable() {
+Status DBImpl::WaitForFlushMemTable() {
  Status s;
  // Wait until the compaction completes
  MutexLock l(&mutex_);
@@ -1370,16 +1396,21 @@ Status DBImpl::WaitForCompactMemTable() {
  return s;
 }

-Status DBImpl::TEST_CompactMemTable() {
+Status DBImpl::TEST_FlushMemTable() {
  return FlushMemTable(FlushOptions());
 }

-Status DBImpl::TEST_WaitForCompactMemTable() {
-  return WaitForCompactMemTable();
+Status DBImpl::TEST_WaitForFlushMemTable() {
+  return WaitForFlushMemTable();
 }

 Status DBImpl::TEST_WaitForCompact() {
  // Wait until the compaction completes
+
+  // TODO: a bug here. This function actually does not necessarily
+  // wait for compact. It actually waits for scheduled compaction
+  // OR flush to finish.
+
  MutexLock l(&mutex_);
  while ((bg_compaction_scheduled_ || bg_flush_scheduled_) &&
         bg_error_.ok()) {
@@ -1388,7 +1419,7 @@ Status DBImpl::TEST_WaitForCompact() {
  return bg_error_;
 }

-void DBImpl::MaybeScheduleCompaction() {
+void DBImpl::MaybeScheduleFlushOrCompaction() {
  mutex_.AssertHeld();
  if (bg_work_gate_closed_) {
    // gate closed for backgrond work
@@ -1423,24 +1454,25 @@ void DBImpl::BGWorkCompaction(void* db) {
  reinterpret_cast<DBImpl*>(db)->BackgroundCallCompaction();
 }

-Status DBImpl::BackgroundFlush() {
+Status DBImpl::BackgroundFlush(bool* madeProgress) {
  Status stat;
  while (stat.ok() &&
         imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
    Log(options_.info_log,
-        "BackgroundCallFlush doing CompactMemTable, flush slots available %d",
+        "BackgroundCallFlush doing FlushMemTableToOutputFile, flush slots available %d",
        options_.max_background_flushes - bg_flush_scheduled_);
-    stat = CompactMemTable();
+    stat = FlushMemTableToOutputFile(madeProgress);
  }
  return stat;
 }

 void DBImpl::BackgroundCallFlush() {
+  bool madeProgress = false;
  assert(bg_flush_scheduled_);
  MutexLock l(&mutex_);

  if (!shutting_down_.Acquire_Load()) {
-    Status s = BackgroundFlush();
+    Status s = BackgroundFlush(&madeProgress);
    if (!s.ok()) {
      // Wait a little bit before retrying background compaction in
      // case this is an environmental problem and we do not want to
@@ -1456,7 +1488,9 @@ void DBImpl::BackgroundCallFlush() {
  }

  bg_flush_scheduled_--;
-
+  if (madeProgress) {
+    MaybeScheduleFlushOrCompaction();
+  }
  bg_cv_.SignalAll();
 }

@@ -1507,7 +1541,7 @@ void DBImpl::BackgroundCallCompaction() {
  // So reschedule another compaction if we made progress in the
  // last compaction.
  if (madeProgress) {
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
  }
  bg_cv_.SignalAll();

@@ -1521,9 +1555,10 @@ Status DBImpl::BackgroundCompaction(bool* madeProgress,
  // TODO: remove memtable flush from formal compaction
  while (imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
    Log(options_.info_log,
-        "BackgroundCompaction doing CompactMemTable, compaction slots available %d",
+        "BackgroundCompaction doing FlushMemTableToOutputFile, compaction slots "
+        "available %d",
        options_.max_background_compactions - bg_compaction_scheduled_);
-    Status stat = CompactMemTable(madeProgress);
+    Status stat = FlushMemTableToOutputFile(madeProgress);
    if (!stat.ok()) {
      return stat;
    }
@@ -1576,7 +1611,7 @@ Status DBImpl::BackgroundCompaction(bool* madeProgress,
    versions_->ReleaseCompactionFiles(c.get(), status);
    *madeProgress = true;
  } else {
-    MaybeScheduleCompaction(); // do more compaction work in parallel.
+    MaybeScheduleFlushOrCompaction(); // do more compaction work in parallel.
    CompactionState* compact = new CompactionState(c.get());
    status = DoCompactionWork(compact);
    CleanupCompaction(compact);
@@ -1702,7 +1737,8 @@ Status DBImpl::OpenCompactionOutputFile(CompactionState* compact) {
      1.1 * versions_->MaxFileSizeForLevel(compact->compaction->output_level()));

    compact->builder.reset(new TableBuilder(options_, compact->outfile.get(),
-                                            compact->compaction->output_level()));
+                                            compact->compaction->output_level(),
+                                            compact->compaction->enable_compression()));
  }
  return s;
 }
@@ -1900,7 +1936,7 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
      const uint64_t imm_start = env_->NowMicros();
      mutex_.Lock();
      if (imm_.IsFlushPending(options_.min_write_buffer_number_to_merge)) {
-        CompactMemTable();
+        FlushMemTableToOutputFile();
        bg_cv_.SignalAll();  // Wakeup MakeRoomForWrite() if necessary
      }
      mutex_.Unlock();
@@ -1952,9 +1988,9 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
          compaction_filter_value.clear();
          bool to_delete =
            compaction_filter->Filter(compact->compaction->level(),
-                                               ikey.user_key, value,
-                                               &compaction_filter_value,
-                                               &value_changed);
+                                      ikey.user_key, value,
+                                      &compaction_filter_value,
+                                      &value_changed);
          if (to_delete) {
            // make a copy of the original key
            delete_key.assign(key.data(), key.data() + key.size());
@@ -2342,7 +2378,7 @@ Status DBImpl::GetImpl(const ReadOptions& options,

  if (!options_.disable_seek_compaction &&
      have_stat_update && current->UpdateStats(stats)) {
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
  }
  mem->Unref();
  imm.UnrefAll();
@@ -2420,7 +2456,7 @@ std::vector<Status> DBImpl::MultiGet(const ReadOptions& options,
  mutex_.Lock();
  if (!options_.disable_seek_compaction &&
      have_stat_update && current->UpdateStats(stats)) {
-    MaybeScheduleCompaction();
+    MaybeScheduleFlushOrCompaction();
  }
  mem->Unref();
  imm.UnrefAll();
@@ -2815,7 +2851,7 @@ Status DBImpl::MakeRoomForWrite(bool force) {
      EnvOptions soptions(storage_options_);
      soptions.use_mmap_writes = false;
      s = env_->NewWritableFile(
-            LogFileName(dbname_, new_log_number),
+            LogFileName(options_.wal_dir, new_log_number),
            &lfile,
            soptions
          );
@@ -2839,7 +2875,7 @@ Status DBImpl::MakeRoomForWrite(bool force) {
      mem_->Ref();
      mem_->SetLogNumber(logfile_number_);
      force = false;   // Do not force another compaction if have room
-      MaybeScheduleCompaction();
+      MaybeScheduleFlushOrCompaction();
    }
  }
  return s;
@@ -2850,7 +2886,7 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {

  MutexLock l(&mutex_);
  Slice in = property;
-  Slice prefix("leveldb.");
+  Slice prefix("rocksdb.");
  if (!in.starts_with(prefix)) return false;
  in.remove_prefix(prefix.size());

@@ -3043,6 +3079,9 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
  } else if (in == "sstables") {
    *value = versions_->current()->DebugString();
    return true;
+  } else if (in == "num-immutable-mem-table") {
+    *value = std::to_string(imm_.size());
+    return true;
  }

  return false;
@@ -3177,8 +3216,15 @@ Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
    return Status::InvalidArgument(
        "no_block_cache is true while block_cache is not nullptr");
  }
+
  DBImpl* impl = new DBImpl(options, dbname);
-  Status s = impl->CreateArchivalDirectory();
+  Status s = impl->env_->CreateDirIfMissing(impl->options_.wal_dir);
+  if (!s.ok()) {
+    delete impl;
+    return s;
+  }
+
+  s = impl->CreateArchivalDirectory();
  if (!s.ok()) {
    delete impl;
    return s;
@@ -3190,8 +3236,11 @@ Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
    uint64_t new_log_number = impl->versions_->NewFileNumber();
    unique_ptr<WritableFile> lfile;
    soptions.use_mmap_writes = false;
-    s = options.env->NewWritableFile(LogFileName(dbname, new_log_number),
-                                     &lfile, soptions);
+    s = options.env->NewWritableFile(
+      LogFileName(impl->options_.wal_dir, new_log_number),
+      &lfile,
+      soptions
+    );
    if (s.ok()) {
      lfile->SetPreallocationBlockSize(1.1 * options.write_buffer_size);
      edit.SetLogNumber(new_log_number);
@@ -3202,7 +3251,7 @@ Status DB::Open(const Options& options, const std::string& dbname, DB** dbptr) {
    if (s.ok()) {
      impl->mem_->SetLogNumber(impl->logfile_number_);
      impl->DeleteObsoleteFiles();
-      impl->MaybeScheduleCompaction();
+      impl->MaybeScheduleFlushOrCompaction();
      impl->MaybeScheduleLogDBDeployStats();
    }
  }
@@ -3232,13 +3281,24 @@ Snapshot::~Snapshot() {
 }

 Status DestroyDB(const std::string& dbname, const Options& options) {
-  Env* env = options.env;
+  const InternalKeyComparator comparator(options.comparator);
+  const InternalFilterPolicy filter_policy(options.filter_policy);
+  const Options& soptions(SanitizeOptions(
+    dbname, &comparator, &filter_policy, options));
+  Env* env = soptions.env;
  std::vector<std::string> filenames;
  std::vector<std::string> archiveFiles;

+  std::string archivedir = ArchivalDirectory(dbname);
  // Ignore error in case directory does not exist
  env->GetChildren(dbname, &filenames);
-  env->GetChildren(ArchivalDirectory(dbname), &archiveFiles);
+
+  if (dbname != soptions.wal_dir) {
+    std::vector<std::string> logfilenames;
+    env->GetChildren(soptions.wal_dir, &logfilenames);
+    filenames.insert(filenames.end(), logfilenames.begin(), logfilenames.end());
+    archivedir = ArchivalDirectory(soptions.wal_dir);
+  }

  if (filenames.empty()) {
    return Status::OK();
@@ -3256,6 +3316,8 @@ Status DestroyDB(const std::string& dbname, const Options& options) {
        Status del;
        if (type == kMetaDatabase) {
          del = DestroyDB(dbname + "/" + filenames[i], options);
+        } else if (type == kLogFile) {
+          del = env->DeleteFile(soptions.wal_dir + "/" + filenames[i]);
        } else {
          del = env->DeleteFile(dbname + "/" + filenames[i]);
        }
@@ -3265,23 +3327,24 @@ Status DestroyDB(const std::string& dbname, const Options& options) {
      }
    }

+    env->GetChildren(archivedir, &archiveFiles);
    // Delete archival files.
    for (size_t i = 0; i < archiveFiles.size(); ++i) {
-      ParseFileName(archiveFiles[i], &number, &type);
-      if (type == kLogFile) {
-        Status del = env->DeleteFile(ArchivalDirectory(dbname) + "/" +
-                                     archiveFiles[i]);
+      if (ParseFileName(archiveFiles[i], &number, &type) &&
+          type == kLogFile) {
+        Status del = env->DeleteFile(archivedir + "/" + archiveFiles[i]);
        if (result.ok() && !del.ok()) {
          result = del;
        }
      }
    }
    // ignore case where no archival directory is present.
-    env->DeleteDir(ArchivalDirectory(dbname));
+    env->DeleteDir(archivedir);

    env->UnlockFile(lock);  // Ignore error since state is already gone
    env->DeleteFile(lockname);
    env->DeleteDir(dbname);  // Ignore error in case dir contains other files
+    env->DeleteDir(soptions.wal_dir);
  }
  return result;
 }
@@ -3289,9 +3352,9 @@ Status DestroyDB(const std::string& dbname, const Options& options) {
 //
 // A global method that can dump out the build version
 void dumpLeveldbBuildVersion(Logger * log) {
-  Log(log, "Git sha %s", leveldb_build_git_sha);
+  Log(log, "Git sha %s", rocksdb_build_git_sha);
  Log(log, "Compile time %s %s",
-      leveldb_build_compile_time, leveldb_build_compile_date);
+      rocksdb_build_compile_time, rocksdb_build_compile_date);
 }

 }  // namespace rocksdb
@@ -1,10 +1,12 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
-
-#ifndef STORAGE_LEVELDB_DB_DB_IMPL_H_
-#define STORAGE_LEVELDB_DB_DB_IMPL_H_
-
+#pragma once
 #include <atomic>
 #include <deque>
 #include <set>
@@ -20,10 +22,6 @@
 #include "util/stats_logger.h"
 #include "memtablelist.h"

-#ifdef USE_SCRIBE
-#include "scribe/scribe_logger.h"
-#endif
-
 namespace rocksdb {

 class MemTable;
@@ -89,11 +87,11 @@ class DBImpl : public DB {
  // Compact any files in the named level that overlap [*begin, *end]
  void TEST_CompactRange(int level, const Slice* begin, const Slice* end);

-  // Force current memtable contents to be compacted.
-  Status TEST_CompactMemTable();
+  // Force current memtable contents to be flushed.
+  Status TEST_FlushMemTable();

  // Wait for memtable compaction
-  Status TEST_WaitForCompactMemTable();
+  Status TEST_WaitForFlushMemTable();

  // Wait for any compaction
  Status TEST_WaitForCompact();
@@ -116,6 +114,9 @@ class DBImpl : public DB {
  // Trigger's a background call for testing.
  void TEST_PurgeObsoleteteWAL();

+  // get total level0 file size. Only for testing.
+  uint64_t TEST_GetLevel0TotalSize() { return versions_->NumLevelBytes(0);}
+
 protected:
  Env* const env_;
  const std::string dbname_;
@@ -155,9 +156,9 @@ class DBImpl : public DB {
  // Delete any unneeded files and stale in-memory entries.
  void DeleteObsoleteFiles();

-  // Compact the in-memory write buffer to disk.  Switches to a new
+  // Flush the in-memory write buffer to storage.  Switches to a new
  // log-file/memtable and writes a new descriptor iff successful.
-  Status CompactMemTable(bool* madeProgress = nullptr);
+  Status FlushMemTableToOutputFile(bool* madeProgress = nullptr);

  Status RecoverLogFile(uint64_t log_number,
                        VersionEdit* edit,
@@ -180,20 +181,20 @@ class DBImpl : public DB {
  // Force current memtable contents to be flushed.
  Status FlushMemTable(const FlushOptions& options);

-  // Wait for memtable compaction
-  Status WaitForCompactMemTable();
+  // Wait for memtable flushed
+  Status WaitForFlushMemTable();

  void MaybeScheduleLogDBDeployStats();
  static void BGLogDBDeployStats(void* db);
  void LogDBDeployStats();

-  void MaybeScheduleCompaction();
+  void MaybeScheduleFlushOrCompaction();
  static void BGWorkCompaction(void* db);
  static void BGWorkFlush(void* db);
  void BackgroundCallCompaction();
  void BackgroundCallFlush();
  Status BackgroundCompaction(bool* madeProgress,DeletionState& deletion_state);
-  Status BackgroundFlush();
+  Status BackgroundFlush(bool* madeProgress);
  void CleanupCompaction(CompactionState* compact);
  Status DoCompactionWork(CompactionState* compact);

@@ -240,7 +241,7 @@ class DBImpl : public DB {

  void PrintStatistics();

-  // dump leveldb.stats to LOG
+  // dump rocksdb.stats to LOG
  void MaybeDumpStats();

  // Return the minimum empty level that could hold the total data in the
@@ -445,5 +446,3 @@ extern Options SanitizeOptions(const std::string& db,
                               const Options& src);

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_DB_IMPL_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.

-#ifndef STORAGE_LEVELDB_DB_DB_IMPL_READONLY_H_
-#define STORAGE_LEVELDB_DB_DB_IMPL_READONLY_H_
-
+#pragma once
 #include "db/db_impl.h"

 #include <deque>
@@ -17,10 +20,6 @@
 #include "port/port.h"
 #include "util/stats_logger.h"

-#ifdef USE_SCRIBE
-#include "scribe/scribe_logger.h"
-#endif
-
 namespace rocksdb {

 class DBImplReadOnly : public DBImpl {
@@ -77,5 +76,3 @@ private:
 };

 }
-
-#endif
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -368,6 +373,7 @@ void DBIter::FindPrevUserEntry() {
  if (iter_->Valid()) {
    do {
      ParsedInternalKey ikey;
+      bool saved_key_cleared = false;
      if (ParseKey(&ikey) && ikey.sequence <= sequence_) {
        if ((value_type != kTypeDeletion) &&
            user_comparator_->Compare(ikey.user_key, saved_key_) < 0) {
@@ -378,6 +384,7 @@ void DBIter::FindPrevUserEntry() {
        if (value_type == kTypeDeletion) {
          saved_key_.clear();
          ClearSavedValue();
+          saved_key_cleared = true;
        } else {
          Slice raw_value = iter_->value();
          if (saved_value_.capacity() > raw_value.size() + 1048576) {
@@ -393,7 +400,7 @@ void DBIter::FindPrevUserEntry() {
      // found the prev user-key, then it is better to seek so that we can
      // avoid too many key comparisons. We seek to the first occurence of
      // our current key by looking for max sequence number.
-      if (num_skipped > max_skip_) {
+      if (!saved_key_cleared && num_skipped > max_skip_) {
        num_skipped = 0;
        std::string last_key;
        AppendInternalKey(&last_key,
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_DB_ITER_H_
-#define STORAGE_LEVELDB_DB_DB_ITER_H_
-
+#pragma once
 #include <stdint.h>
 #include "rocksdb/db.h"
 #include "db/dbformat.h"
@@ -23,5 +26,3 @@ extern Iterator* NewDBIterator(
    const SequenceNumber& sequence);

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_DB_ITER_H_
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef LEVELDB_STORAGE_DB_DB_STATISTICS_H_
-#define LEVELDB_STORAGE_DB_DB_STATISTICS_H_
-
+#pragma once
 #include <cassert>
 #include <stdlib.h>
 #include <vector>
@@ -60,5 +63,3 @@ std::shared_ptr<Statistics> CreateDBStatistics() {
 }

 } // namespace rocksdb
-
-#endif // LEVELDB_STORAGE_DB_DB_STATISTICS_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -49,6 +54,12 @@ static std::string RandomString(Random* rnd, int len) {
  return r;
 }

+static std::string CompressibleString(Random* rnd, int len) {
+  std::string r;
+  test::CompressibleString(rnd, 0.8, len, &r);
+  return r;
+}
+
 namespace anon {
 class AtomicCounter {
 private:
@@ -220,6 +231,7 @@ class DBTest {
    kUncompressed,
    kNumLevel_3,
    kDBLogDir,
+    kWalDir,
    kManifestFileSize,
    kCompactOnFlush,
    kPerfOptions,
@@ -242,21 +254,22 @@ class DBTest {
  enum OptionSkip {
    kNoSkip = 0,
    kSkipDeletesFilterFirst = 1,
-    kSkipUniversalCompaction = 2
+    kSkipUniversalCompaction = 2,
+    kSkipMergePut = 4
  };

  DBTest() : option_config_(kDefault),
             env_(new SpecialEnv(Env::Default())) {
    filter_policy_ = NewBloomFilterPolicy(10);
    dbname_ = test::TmpDir() + "/db_test";
-    DestroyDB(dbname_, Options());
+    ASSERT_OK(DestroyDB(dbname_, Options()));
    db_ = nullptr;
    Reopen();
  }

  ~DBTest() {
    delete db_;
-    DestroyDB(dbname_, Options());
+    ASSERT_OK(DestroyDB(dbname_, Options()));
    delete env_;
    delete filter_policy_;
  }
@@ -275,8 +288,11 @@ class DBTest {
        option_config_ == kUniversalCompaction) {
      option_config_++;
    }
-
+    if (skip_mask & kSkipMergePut && option_config_ == kMergePut) {
+      option_config_++;
+    }
    if (option_config_ >= kEnd) {
+      Destroy(&last_options_);
      return false;
    } else {
      DestroyAndReopen();
@@ -285,10 +301,14 @@ class DBTest {
  }

  // Switch between different compaction styles (we have only 2 now).
-  bool ChangeCompactOptions() {
+  bool ChangeCompactOptions(Options* prev_options = nullptr) {
    if (option_config_ == kDefault) {
      option_config_ = kUniversalCompaction;
-      DestroyAndReopen();
+      if (prev_options == nullptr) {
+        prev_options = &last_options_;
+      }
+      Destroy(prev_options);
+      TryReopen();
      return true;
    } else {
      return false;
@@ -318,10 +338,14 @@ class DBTest {
      case kDBLogDir:
        options.db_log_dir = test::TmpDir();
        break;
+      case kWalDir:
+        options.wal_dir = "/tmp/wal";
+        break;
      case kManifestFileSize:
        options.max_manifest_file_size = 50; // 50 bytes
      case kCompactOnFlush:
-        options.purge_redundant_kvs_while_flush = !options.purge_redundant_kvs_while_flush;
+        options.purge_redundant_kvs_while_flush =
+          !options.purge_redundant_kvs_while_flush;
        break;
      case kPerfOptions:
        options.hard_rate_limit = 2.0;
@@ -360,17 +384,22 @@ class DBTest {
  }

  void DestroyAndReopen(Options* options = nullptr) {
+    //Destroy using last options
+    Destroy(&last_options_);
+    ASSERT_OK(TryReopen(options));
+  }
+
+  void Destroy(Options* options) {
    delete db_;
    db_ = nullptr;
-    DestroyDB(dbname_, Options());
-    ASSERT_OK(TryReopen(options));
+    ASSERT_OK(DestroyDB(dbname_, *options));
  }

  Status PureReopen(Options* options, DB** db) {
    return DB::Open(*options, dbname_, db);
  }

-  Status TryReopen(Options* options) {
+  Status TryReopen(Options* options = nullptr) {
    delete db_;
    db_ = nullptr;
    Options opts;
@@ -489,7 +518,7 @@ class DBTest {
  int NumTableFilesAtLevel(int level) {
    std::string property;
    ASSERT_TRUE(
-        db_->GetProperty("leveldb.num-files-at-level" + NumberToString(level),
+        db_->GetProperty("rocksdb.num-files-at-level" + NumberToString(level),
                         &property));
    return atoi(property.c_str());
  }
@@ -522,7 +551,13 @@ class DBTest {
  int CountFiles() {
    std::vector<std::string> files;
    env_->GetChildren(dbname_, &files);
-    return static_cast<int>(files.size());
+
+    std::vector<std::string> logfiles;
+    if (dbname_ != last_options_.wal_dir) {
+      env_->GetChildren(last_options_.wal_dir, &logfiles);
+    }
+
+    return static_cast<int>(files.size() + logfiles.size());
  }

  int CountLiveFiles() {
@@ -549,7 +584,7 @@ class DBTest {
    for (int i = 0; i < n; i++) {
      Put(small, "begin");
      Put(large, "end");
-      dbfull()->TEST_CompactMemTable();
+      dbfull()->TEST_FlushMemTable();
    }
  }

@@ -574,7 +609,7 @@ class DBTest {

  std::string DumpSSTableList() {
    std::string property;
-    db_->GetProperty("leveldb.sstables", &property);
+    db_->GetProperty("rocksdb.sstables", &property);
    return property;
  }

@@ -599,7 +634,7 @@ class DBTest {
    const SequenceNumber seq) {
    unique_ptr<TransactionLogIterator> iter;
    Status status = dbfull()->GetUpdatesSince(seq, &iter);
-    ASSERT_TRUE(status.ok());
+    ASSERT_OK(status);
    ASSERT_TRUE(iter->Valid());
    return std::move(iter);
  }
@@ -607,6 +642,13 @@ class DBTest {
  std::string DummyString(size_t len, char c = 'a') {
    return std::string(len, c);
  }
+
+  void VerifyIterLast(std::string expected_key) {
+    Iterator* iter = db_->NewIterator(ReadOptions());
+    iter->SeekToLast();
+    ASSERT_EQ(IterStatus(iter), expected_key);
+    delete iter;
+  }
 };

 TEST(DBTest, Empty) {
@@ -718,7 +760,7 @@ TEST(DBTest, GetFromImmutableLayer) {
 TEST(DBTest, GetFromVersions) {
  do {
    ASSERT_OK(Put("foo", "v1"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_EQ("v1", Get("foo"));
  } while (ChangeOptions());
 }
@@ -733,7 +775,7 @@ TEST(DBTest, GetSnapshot) {
      ASSERT_OK(Put(key, "v2"));
      ASSERT_EQ("v2", Get(key));
      ASSERT_EQ("v1", Get(key, s1));
-      dbfull()->TEST_CompactMemTable();
+      dbfull()->TEST_FlushMemTable();
      ASSERT_EQ("v2", Get(key));
      ASSERT_EQ("v1", Get(key, s1));
      db_->ReleaseSnapshot(s1);
@@ -749,9 +791,9 @@ TEST(DBTest, GetLevel0Ordering) {
    // one has a smaller "smallest" key.
    ASSERT_OK(Put("bar", "b"));
    ASSERT_OK(Put("foo", "v1"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_OK(Put("foo", "v2"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_EQ("v2", Get("foo"));
  } while (ChangeOptions());
 }
@@ -763,7 +805,7 @@ TEST(DBTest, GetOrderedByLevels) {
    ASSERT_EQ("v1", Get("foo"));
    ASSERT_OK(Put("foo", "v2"));
    ASSERT_EQ("v2", Get("foo"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_EQ("v2", Get("foo"));
  } while (ChangeOptions());
 }
@@ -801,7 +843,7 @@ TEST(DBTest, GetEncountersEmptyLevel) {
      compaction_count++;
      Put("a", "begin");
      Put("z", "end");
-      dbfull()->TEST_CompactMemTable();
+      dbfull()->TEST_FlushMemTable();
    }

    // Step 2: clear level 1 if necessary.
@@ -895,7 +937,6 @@ TEST(DBTest, NonBlockingIteration) {
    options.statistics = rocksdb::CreateDBStatistics();
    non_blocking_opts.read_tier = kBlockCacheTier;
    Reopen(&options);
-
    // write one kv to the database.
    ASSERT_OK(db_->Put(WriteOptions(), "a", "b"));

@@ -904,7 +945,7 @@ TEST(DBTest, NonBlockingIteration) {
    Iterator* iter = db_->NewIterator(non_blocking_opts);
    int count = 0;
    for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
-      ASSERT_TRUE(iter->status().ok());
+      ASSERT_OK(iter->status());
      count++;
    }
    ASSERT_EQ(count, 1);
@@ -940,7 +981,7 @@ TEST(DBTest, NonBlockingIteration) {
    iter = db_->NewIterator(non_blocking_opts);
    count = 0;
    for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
-      ASSERT_TRUE(iter->status().ok());
+      ASSERT_OK(iter->status());
      count++;
    }
    ASSERT_EQ(count, 1);
@@ -1283,6 +1324,35 @@ TEST(DBTest, IterMultiWithDelete) {
  } while (ChangeOptions());
 }

+TEST(DBTest, IterPrevMaxSkip) {
+  do {
+    for (int i = 0; i < 2; i++) {
+      db_->Put(WriteOptions(), "key1", "v1");
+      db_->Put(WriteOptions(), "key2", "v2");
+      db_->Put(WriteOptions(), "key3", "v3");
+      db_->Put(WriteOptions(), "key4", "v4");
+      db_->Put(WriteOptions(), "key5", "v5");
+    }
+
+    VerifyIterLast("key5->v5");
+
+    ASSERT_OK(db_->Delete(WriteOptions(), "key5"));
+    VerifyIterLast("key4->v4");
+
+    ASSERT_OK(db_->Delete(WriteOptions(), "key4"));
+    VerifyIterLast("key3->v3");
+
+    ASSERT_OK(db_->Delete(WriteOptions(), "key3"));
+    VerifyIterLast("key2->v2");
+
+    ASSERT_OK(db_->Delete(WriteOptions(), "key2"));
+    VerifyIterLast("key1->v1");
+
+    ASSERT_OK(db_->Delete(WriteOptions(), "key1"));
+    VerifyIterLast("(invalid)");
+  } while (ChangeOptions(kSkipMergePut));
+}
+
 TEST(DBTest, IterWithSnapshot) {
  do {
    ASSERT_OK(Put("key1", "val1"));
@@ -1397,10 +1467,10 @@ TEST(DBTest, CheckLock) {
  do {
    DB* localdb;
    Options options = CurrentOptions();
-    ASSERT_TRUE(TryReopen(&options).ok());
+    ASSERT_OK(TryReopen(&options));

    // second open should fail
-    ASSERT_TRUE(!(PureReopen(&options, &localdb).ok()));
+    ASSERT_TRUE(!(PureReopen(&options, &localdb)).ok());
  } while (ChangeCompactOptions());
 }

@@ -1422,6 +1492,37 @@ TEST(DBTest, FlushMultipleMemtable) {
  } while (ChangeCompactOptions());
 }

+TEST(DBTest, NumImmutableMemTable) {
+  do {
+    Options options = CurrentOptions();
+    WriteOptions writeOpt = WriteOptions();
+    writeOpt.disableWAL = true;
+    options.max_write_buffer_number = 4;
+    options.min_write_buffer_number_to_merge = 3;
+    options.write_buffer_size = 1000000;
+    Reopen(&options);
+
+    std::string big_value(1000000, 'x');
+    std::string num;
+
+    ASSERT_OK(dbfull()->Put(writeOpt, "k1", big_value));
+    ASSERT_TRUE(dbfull()->GetProperty("rocksdb.num-immutable-mem-table", &num));
+    ASSERT_EQ(num, "0");
+
+    ASSERT_OK(dbfull()->Put(writeOpt, "k2", big_value));
+    ASSERT_TRUE(dbfull()->GetProperty("rocksdb.num-immutable-mem-table", &num));
+    ASSERT_EQ(num, "1");
+
+    ASSERT_OK(dbfull()->Put(writeOpt, "k3", big_value));
+    ASSERT_TRUE(dbfull()->GetProperty("rocksdb.num-immutable-mem-table", &num));
+    ASSERT_EQ(num, "2");
+
+    dbfull()->Flush(FlushOptions());
+    ASSERT_TRUE(dbfull()->GetProperty("rocksdb.num-immutable-mem-table", &num));
+    ASSERT_EQ(num, "0");
+  } while (ChangeCompactOptions());
+}
+
 TEST(DBTest, FLUSH) {
  do {
    Options options = CurrentOptions();
@@ -1529,12 +1630,12 @@ TEST(DBTest, ManifestRollOver) {
      ASSERT_OK(Put("manifest_key1", std::string(1000, '1')));
      ASSERT_OK(Put("manifest_key2", std::string(1000, '2')));
      ASSERT_OK(Put("manifest_key3", std::string(1000, '3')));
-      uint64_t manifest_before_fulsh =
+      uint64_t manifest_before_flush =
        dbfull()->TEST_Current_Manifest_FileNo();
      dbfull()->Flush(FlushOptions()); // This should trigger LogAndApply.
      uint64_t manifest_after_flush =
        dbfull()->TEST_Current_Manifest_FileNo();
-      ASSERT_GT(manifest_after_flush, manifest_before_fulsh);
+      ASSERT_GT(manifest_after_flush, manifest_before_flush);
      Reopen(&options);
      ASSERT_GT(dbfull()->TEST_Current_Manifest_FileNo(),
                manifest_after_flush);
@@ -1546,6 +1647,36 @@ TEST(DBTest, ManifestRollOver) {
  } while (ChangeCompactOptions());
 }

+TEST(DBTest, IdentityAcrossRestarts) {
+  do {
+    std::string idfilename = IdentityFileName(dbname_);
+    unique_ptr<SequentialFile> idfile;
+    const EnvOptions soptions;
+    ASSERT_OK(env_->NewSequentialFile(idfilename, &idfile, soptions));
+    char buffer1[100];
+    Slice id1;
+    ASSERT_OK(idfile->Read(100, &id1, buffer1));
+
+    Options options = CurrentOptions();
+    Reopen(&options);
+    char buffer2[100];
+    Slice id2;
+    ASSERT_OK(env_->NewSequentialFile(idfilename, &idfile, soptions));
+    ASSERT_OK(idfile->Read(100, &id2, buffer2));
+    // id1 should match id2 because identity was not regenerated
+    ASSERT_EQ(id1.ToString(), id2.ToString());
+
+    ASSERT_OK(env_->DeleteFile(idfilename));
+    Reopen(&options);
+    char buffer3[100];
+    Slice id3;
+    ASSERT_OK(env_->NewSequentialFile(idfilename, &idfile, soptions));
+    ASSERT_OK(idfile->Read(100, &id3, buffer3));
+    // id1 should NOT match id2 because identity was regenerated
+    ASSERT_NE(id1.ToString(0), id3.ToString());
+  } while (ChangeCompactOptions());
+}
+
 TEST(DBTest, RecoverWithLargeLog) {
  do {
    {
@@ -1617,7 +1748,7 @@ TEST(DBTest, CompactionTrigger) {
      values.push_back(RandomString(&rnd, 10000));
      ASSERT_OK(Put(Key(i), values[i]));
    }
-    dbfull()->TEST_WaitForCompactMemTable();
+    dbfull()->TEST_WaitForFlushMemTable();
    ASSERT_EQ(NumTableFilesAtLevel(0), num + 1);
  }

@@ -1655,7 +1786,7 @@ TEST(DBTest, UniversalCompactionTrigger) {
      ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 10000)));
      key_idx++;
    }
-    dbfull()->TEST_WaitForCompactMemTable();
+    dbfull()->TEST_WaitForFlushMemTable();
    ASSERT_EQ(NumTableFilesAtLevel(0), num + 1);
  }

@@ -1690,7 +1821,7 @@ TEST(DBTest, UniversalCompactionTrigger) {
      ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 10000)));
      key_idx++;
    }
-    dbfull()->TEST_WaitForCompactMemTable();
+    dbfull()->TEST_WaitForFlushMemTable();
    ASSERT_EQ(NumTableFilesAtLevel(0), num + 3);
  }

@@ -1719,7 +1850,7 @@ TEST(DBTest, UniversalCompactionTrigger) {
      ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 10000)));
      key_idx++;
    }
-    dbfull()->TEST_WaitForCompactMemTable();
+    dbfull()->TEST_WaitForFlushMemTable();
    ASSERT_EQ(NumTableFilesAtLevel(0), num + 3);
  }

@@ -1789,7 +1920,7 @@ TEST(DBTest, UniversalCompactionSizeAmplification) {
      ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 10000)));
      key_idx++;
    }
-    dbfull()->TEST_WaitForCompactMemTable();
+    dbfull()->TEST_WaitForFlushMemTable();
    ASSERT_EQ(NumTableFilesAtLevel(0), num + 1);
  }
  ASSERT_EQ(NumTableFilesAtLevel(0), 2);
@@ -1799,6 +1930,8 @@ TEST(DBTest, UniversalCompactionSizeAmplification) {
  // but will instead trigger size amplification.
  dbfull()->Flush(FlushOptions());

+  dbfull()->TEST_WaitForCompact();
+
  // Verify that size amplification did occur
  ASSERT_EQ(NumTableFilesAtLevel(0), 1);
 }
@@ -1809,6 +1942,7 @@ TEST(DBTest, UniversalCompactionOptions) {
  options.write_buffer_size = 100<<10; //100KB
  options.level0_file_num_compaction_trigger = 4;
  options.num_levels = 1;
+  options.compaction_options_universal.compression_size_percent = -1;
  Reopen(&options);

  Random rnd(301);
@@ -1822,7 +1956,7 @@ TEST(DBTest, UniversalCompactionOptions) {
      ASSERT_OK(Put(Key(key_idx), RandomString(&rnd, 10000)));
      key_idx++;
    }
-    dbfull()->TEST_WaitForCompactMemTable();
+    dbfull()->TEST_WaitForFlushMemTable();

    if (num < options.level0_file_num_compaction_trigger - 1) {
      ASSERT_EQ(NumTableFilesAtLevel(0), num + 1);
@@ -1836,6 +1970,97 @@ TEST(DBTest, UniversalCompactionOptions) {
  }
 }

+TEST(DBTest, UniversalCompactionCompressRatio1) {
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleUniversal;
+  options.write_buffer_size = 100<<10; //100KB
+  options.level0_file_num_compaction_trigger = 2;
+  options.num_levels = 1;
+  options.compaction_options_universal.compression_size_percent = 70;
+  Reopen(&options);
+
+  Random rnd(301);
+  int key_idx = 0;
+
+  // The first compaction (2) is compressed.
+  for (int num = 0; num < 2; num++) {
+    // Write 120KB (12 values, each 10K)
+    for (int i = 0; i < 12; i++) {
+      ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
+      key_idx++;
+    }
+    dbfull()->TEST_WaitForFlushMemTable();
+    dbfull()->TEST_WaitForCompact();
+  }
+  ASSERT_LT((int ) dbfull()->TEST_GetLevel0TotalSize(), 120000 * 2 * 0.9);
+
+  // The second compaction (4) is compressed
+  for (int num = 0; num < 2; num++) {
+    // Write 120KB (12 values, each 10K)
+    for (int i = 0; i < 12; i++) {
+      ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
+      key_idx++;
+    }
+    dbfull()->TEST_WaitForFlushMemTable();
+    dbfull()->TEST_WaitForCompact();
+  }
+  ASSERT_LT((int ) dbfull()->TEST_GetLevel0TotalSize(), 120000 * 4 * 0.9);
+
+  // The third compaction (2 4) is compressed since this time it is
+  // (1 1 3.2) and 3.2/5.2 doesn't reach ratio.
+  for (int num = 0; num < 2; num++) {
+    // Write 120KB (12 values, each 10K)
+    for (int i = 0; i < 12; i++) {
+      ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
+      key_idx++;
+    }
+    dbfull()->TEST_WaitForFlushMemTable();
+    dbfull()->TEST_WaitForCompact();
+  }
+  ASSERT_LT((int ) dbfull()->TEST_GetLevel0TotalSize(), 120000 * 6 * 0.9);
+
+  // When we start for the compaction up to (2 4 8), the latest
+  // compressed is not compressed.
+  for (int num = 0; num < 8; num++) {
+    // Write 120KB (12 values, each 10K)
+    for (int i = 0; i < 12; i++) {
+      ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
+      key_idx++;
+    }
+    dbfull()->TEST_WaitForFlushMemTable();
+    dbfull()->TEST_WaitForCompact();
+  }
+  ASSERT_GT((int ) dbfull()->TEST_GetLevel0TotalSize(),
+            120000 * 12 * 0.8 + 110000 * 2);
+}
+
+TEST(DBTest, UniversalCompactionCompressRatio2) {
+  Options options = CurrentOptions();
+  options.compaction_style = kCompactionStyleUniversal;
+  options.write_buffer_size = 100<<10; //100KB
+  options.level0_file_num_compaction_trigger = 2;
+  options.num_levels = 1;
+  options.compaction_options_universal.compression_size_percent = 95;
+  Reopen(&options);
+
+  Random rnd(301);
+  int key_idx = 0;
+
+  // When we start for the compaction up to (2 4 8), the latest
+  // compressed is compressed given the size ratio to compress.
+  for (int num = 0; num < 14; num++) {
+    // Write 120KB (12 values, each 10K)
+    for (int i = 0; i < 12; i++) {
+      ASSERT_OK(Put(Key(key_idx), CompressibleString(&rnd, 10000)));
+      key_idx++;
+    }
+    dbfull()->TEST_WaitForFlushMemTable();
+    dbfull()->TEST_WaitForCompact();
+  }
+  ASSERT_LT((int ) dbfull()->TEST_GetLevel0TotalSize(),
+            120000 * 12 * 0.8 + 110000 * 2);
+}
+
 TEST(DBTest, ConvertCompactionStyle) {
  Random rnd(301);
  int max_key_level_insert = 200;
@@ -1943,7 +2168,7 @@ void MinLevelHelper(DBTest* self, Options& options) {
      values.push_back(RandomString(&rnd, 10000));
      ASSERT_OK(self->Put(Key(i), values[i]));
    }
-    self->dbfull()->TEST_WaitForCompactMemTable();
+    self->dbfull()->TEST_WaitForFlushMemTable();
    ASSERT_EQ(self->NumTableFilesAtLevel(0), num + 1);
  }

@@ -2168,7 +2393,7 @@ TEST(DBTest, CompactionFilter) {
    snprintf(key, sizeof(key), "B%010d", i);
    Put(key, value);
  }
-  dbfull()->TEST_CompactMemTable();
+  dbfull()->TEST_FlushMemTable();

  // Push all files to the highest level L2. Verify that
  // the compaction is each level invokes the filter for
@@ -2195,7 +2420,7 @@ TEST(DBTest, CompactionFilter) {
  int total = 0;
  Iterator* iter = dbfull()->TEST_NewInternalIterator();
  iter->SeekToFirst();
-  ASSERT_EQ(iter->status().ok(), true);
+  ASSERT_OK(iter->status());
  while (iter->Valid()) {
    ParsedInternalKey ikey;
    ikey.sequence = -1;
@@ -2216,7 +2441,7 @@ TEST(DBTest, CompactionFilter) {
    snprintf(key, sizeof(key), "B%010d", i);
    Put(key, value);
  }
-  dbfull()->TEST_CompactMemTable();
+  dbfull()->TEST_FlushMemTable();

  // push all files to the highest level L2. This
  // means that all keys should pass at least once
@@ -2243,7 +2468,7 @@ TEST(DBTest, CompactionFilter) {
    snprintf(key, sizeof(key), "B%010d", i);
    Put(key, value);
  }
-  dbfull()->TEST_CompactMemTable();
+  dbfull()->TEST_FlushMemTable();
  ASSERT_NE(NumTableFilesAtLevel(0), 0);
  ASSERT_EQ(NumTableFilesAtLevel(1), 0);
  ASSERT_EQ(NumTableFilesAtLevel(2), 0);
@@ -2280,7 +2505,7 @@ TEST(DBTest, CompactionFilter) {
  count = 0;
  iter = dbfull()->TEST_NewInternalIterator();
  iter->SeekToFirst();
-  ASSERT_EQ(iter->status().ok(), true);
+  ASSERT_OK(iter->status());
  while (iter->Valid()) {
    ParsedInternalKey ikey;
    ASSERT_EQ(ParseInternalKey(iter->key(), &ikey), true);
@@ -2314,7 +2539,7 @@ TEST(DBTest, CompactionFilterWithValueChange) {
    }

    // push all files to  lower levels
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    dbfull()->TEST_CompactRange(0, nullptr, nullptr);
    dbfull()->TEST_CompactRange(1, nullptr, nullptr);

@@ -2327,7 +2552,7 @@ TEST(DBTest, CompactionFilterWithValueChange) {

    // push all files to  lower levels. This should
    // invoke the compaction filter for all 100000 keys.
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    dbfull()->TEST_CompactRange(0, nullptr, nullptr);
    dbfull()->TEST_CompactRange(1, nullptr, nullptr);

@@ -2365,14 +2590,14 @@ TEST(DBTest, SparseMerge) {
      Put(key, value);
    }
    Put("C", "vc");
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    dbfull()->TEST_CompactRange(0, nullptr, nullptr);

    // Make sparse update
    Put("A",    "va2");
    Put("B100", "bvalue2");
    Put("C",    "vc2");
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();

    // Compactions should not cause us to create a situation where
    // a file overlaps too much data at the next level.
@@ -2548,7 +2773,7 @@ TEST(DBTest, HiddenValuesAreRemoved) {
    Put("foo", "tiny");
    Put("pastfoo2", "v2");        // Advance sequence number one more

-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_GT(NumTableFilesAtLevel(0), 0);

    ASSERT_EQ(big, Get("foo", snapshot));
@@ -2583,7 +2808,7 @@ TEST(DBTest, CompactBetweenSnapshots) {

    // All entries (including duplicates) exist
    // before any compaction is triggered.
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ("sixth", Get("foo"));
    ASSERT_EQ("fourth", Get("foo", snapshot2));
    ASSERT_EQ("first", Get("foo", snapshot1));
@@ -2622,21 +2847,21 @@ TEST(DBTest, CompactBetweenSnapshots) {

 TEST(DBTest, DeletionMarkers1) {
  Put("foo", "v1");
-  ASSERT_OK(dbfull()->TEST_CompactMemTable());
+  ASSERT_OK(dbfull()->TEST_FlushMemTable());
  const int last = dbfull()->MaxMemCompactionLevel();
  ASSERT_EQ(NumTableFilesAtLevel(last), 1);   // foo => v1 is now in last level

  // Place a table at level last-1 to prevent merging with preceding mutation
  Put("a", "begin");
  Put("z", "end");
-  dbfull()->TEST_CompactMemTable();
+  dbfull()->TEST_FlushMemTable();
  ASSERT_EQ(NumTableFilesAtLevel(last), 1);
  ASSERT_EQ(NumTableFilesAtLevel(last-1), 1);

  Delete("foo");
  Put("foo", "v2");
  ASSERT_EQ(AllEntriesFor("foo"), "[ v2, DEL, v1 ]");
-  ASSERT_OK(dbfull()->TEST_CompactMemTable());  // Moves to level last-2
+  ASSERT_OK(dbfull()->TEST_FlushMemTable());  // Moves to level last-2
  if (CurrentOptions().purge_redundant_kvs_while_flush) {
    ASSERT_EQ(AllEntriesFor("foo"), "[ v2, v1 ]");
  } else {
@@ -2655,20 +2880,20 @@ TEST(DBTest, DeletionMarkers1) {

 TEST(DBTest, DeletionMarkers2) {
  Put("foo", "v1");
-  ASSERT_OK(dbfull()->TEST_CompactMemTable());
+  ASSERT_OK(dbfull()->TEST_FlushMemTable());
  const int last = dbfull()->MaxMemCompactionLevel();
  ASSERT_EQ(NumTableFilesAtLevel(last), 1);   // foo => v1 is now in last level

  // Place a table at level last-1 to prevent merging with preceding mutation
  Put("a", "begin");
  Put("z", "end");
-  dbfull()->TEST_CompactMemTable();
+  dbfull()->TEST_FlushMemTable();
  ASSERT_EQ(NumTableFilesAtLevel(last), 1);
  ASSERT_EQ(NumTableFilesAtLevel(last-1), 1);

  Delete("foo");
  ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, v1 ]");
-  ASSERT_OK(dbfull()->TEST_CompactMemTable());  // Moves to level last-2
+  ASSERT_OK(dbfull()->TEST_FlushMemTable());  // Moves to level last-2
  ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, v1 ]");
  dbfull()->TEST_CompactRange(last-2, nullptr, nullptr);
  // DEL kept: "last" file overlaps
@@ -2684,13 +2909,13 @@ TEST(DBTest, OverlapInLevel0) {
    int tmp = dbfull()->MaxMemCompactionLevel();
    ASSERT_EQ(tmp, 2) << "Fix test to match config";

-    // Fill levels 1 and 2 to disable the pushing of new memtables to levels > 0.
+    //Fill levels 1 and 2 to disable the pushing of new memtables to levels > 0.
    ASSERT_OK(Put("100", "v100"));
    ASSERT_OK(Put("999", "v999"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_OK(Delete("100"));
    ASSERT_OK(Delete("999"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_EQ("0,1,1", FilesPerLevel());

    // Make files spanning the following ranges in level-0:
@@ -2699,11 +2924,11 @@ TEST(DBTest, OverlapInLevel0) {
    // Note that files are sorted by smallest key.
    ASSERT_OK(Put("300", "v300"));
    ASSERT_OK(Put("500", "v500"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_OK(Put("200", "v200"));
    ASSERT_OK(Put("600", "v600"));
    ASSERT_OK(Put("900", "v900"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_EQ("2,1,1", FilesPerLevel());

    // Compact away the placeholder files we created initially
@@ -2715,7 +2940,7 @@ TEST(DBTest, OverlapInLevel0) {
    // not detect the overlap with level-0 files and would incorrectly place
    // the deletion in a deeper level.
    ASSERT_OK(Delete("600"));
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_EQ("3", FilesPerLevel());
    ASSERT_EQ("NOT_FOUND", Get("600"));
  } while (ChangeOptions(kSkipUniversalCompaction));
@@ -2771,7 +2996,7 @@ TEST(DBTest, L0_CompactionBug_Issue44_b) {
 TEST(DBTest, ComparatorCheck) {
  class NewComparator : public Comparator {
   public:
-    virtual const char* Name() const { return "leveldb.NewComparator"; }
+    virtual const char* Name() const { return "rocksdb.NewComparator"; }
    virtual int Compare(const Slice& a, const Slice& b) const {
      return BytewiseComparator()->Compare(a, b);
    }
@@ -2782,16 +3007,16 @@ TEST(DBTest, ComparatorCheck) {
      BytewiseComparator()->FindShortSuccessor(key);
    }
  };
-
+  Options new_options;
+  NewComparator cmp;
  do {
-    NewComparator cmp;
-    Options new_options = CurrentOptions();
+    new_options = CurrentOptions();
    new_options.comparator = &cmp;
    Status s = TryReopen(&new_options);
    ASSERT_TRUE(!s.ok());
    ASSERT_TRUE(s.ToString().find("comparator") != std::string::npos)
        << s.ToString();
-  } while (ChangeCompactOptions());
+  } while (ChangeCompactOptions(&new_options));
 }

 TEST(DBTest, CustomComparator) {
@@ -2820,10 +3045,10 @@ TEST(DBTest, CustomComparator) {
      return val;
    }
  };
-
+  Options new_options;
+  NumberComparator cmp;
  do {
-    NumberComparator cmp;
-    Options new_options = CurrentOptions();
+    new_options = CurrentOptions();
    new_options.create_if_missing = true;
    new_options.comparator = &cmp;
    new_options.filter_policy = nullptr;     // Cannot use bloom filters
@@ -2849,7 +3074,7 @@ TEST(DBTest, CustomComparator) {
      }
      Compact("[0]", "[1000000]");
    }
-  } while (ChangeCompactOptions());
+  } while (ChangeCompactOptions(&new_options));
 }

 TEST(DBTest, ManualCompaction) {
@@ -2888,7 +3113,7 @@ TEST(DBTest, ManualCompaction) {

 TEST(DBTest, DBOpen_Options) {
  std::string dbname = test::TmpDir() + "/db_options_test";
-  DestroyDB(dbname, Options());
+  ASSERT_OK(DestroyDB(dbname, Options()));

  // Does not exist, and create_if_missing == false: error
  DB* db = nullptr;
@@ -2927,7 +3152,7 @@ TEST(DBTest, DBOpen_Options) {

 TEST(DBTest, DBOpen_Change_NumLevels) {
  std::string dbname = test::TmpDir() + "/db_change_num_levels";
-  DestroyDB(dbname, Options());
+  ASSERT_OK(DestroyDB(dbname, Options()));
  Options opts;
  Status s;
  DB* db = nullptr;
@@ -2954,9 +3179,9 @@ TEST(DBTest, DestroyDBMetaDatabase) {
  std::string metametadbname = MetaDatabaseName(metadbname, 0);

  // Destroy previous versions if they exist. Using the long way.
-  DestroyDB(metametadbname, Options());
-  DestroyDB(metadbname, Options());
-  DestroyDB(dbname, Options());
+  ASSERT_OK(DestroyDB(metametadbname, Options()));
+  ASSERT_OK(DestroyDB(metadbname, Options()));
+  ASSERT_OK(DestroyDB(dbname, Options()));

  // Setup databases
  Options opts;
@@ -2973,13 +3198,13 @@ TEST(DBTest, DestroyDBMetaDatabase) {
  db = nullptr;

  // Delete databases
-  DestroyDB(dbname, Options());
+  ASSERT_OK(DestroyDB(dbname, Options()));

  // Check if deletion worked.
  opts.create_if_missing = false;
-  ASSERT_TRUE(!DB::Open(opts, dbname, &db).ok());
-  ASSERT_TRUE(!DB::Open(opts, metadbname, &db).ok());
-  ASSERT_TRUE(!DB::Open(opts, metametadbname, &db).ok());
+  ASSERT_TRUE(!(DB::Open(opts, dbname, &db)).ok());
+  ASSERT_TRUE(!(DB::Open(opts, metadbname, &db)).ok());
+  ASSERT_TRUE(!(DB::Open(opts, metametadbname, &db)).ok());
 }

 // Check that number of files does not grow when we are out of space
@@ -3053,7 +3278,7 @@ TEST(DBTest, ManifestWriteError) {
    ASSERT_EQ("bar", Get("foo"));

    // Memtable compaction (will succeed)
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    ASSERT_EQ("bar", Get("foo"));
    const int last = dbfull()->MaxMemCompactionLevel();
    ASSERT_EQ(NumTableFilesAtLevel(last), 1);   // foo=>bar is now in last level
@@ -3101,7 +3326,7 @@ TEST(DBTest, BloomFilter) {
    for (int i = 0; i < N; i += 100) {
      ASSERT_OK(Put(Key(i), Key(i)));
    }
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();

    // Prevent auto compactions triggered by seeks
    env_->delay_sstable_sync_.Release_Store(env_);
@@ -3218,7 +3443,7 @@ TEST(DBTest, SnapshotFiles) {
    DB* snapdb;
    opts.create_if_missing = false;
    Status stat = DB::Open(opts, snapdir, &snapdb);
-    ASSERT_TRUE(stat.ok());
+    ASSERT_OK(stat);

    ReadOptions roptions;
    std::string val;
@@ -3271,13 +3496,13 @@ TEST(DBTest, CompactOnFlush) {
    Reopen(&options);

    Put("foo", "v1");
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ(AllEntriesFor("foo"), "[ v1 ]");

    // Write two new keys
    Put("a", "begin");
    Put("z", "end");
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();

    // Case1: Delete followed by a put
    Delete("foo");
@@ -3286,7 +3511,7 @@ TEST(DBTest, CompactOnFlush) {

    // After the current memtable is flushed, the DEL should
    // have been removed
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ(AllEntriesFor("foo"), "[ v2, v1 ]");

    dbfull()->CompactRange(nullptr, nullptr);
@@ -3296,7 +3521,7 @@ TEST(DBTest, CompactOnFlush) {
    Delete("foo");
    Delete("foo");
    ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, DEL, v2 ]");
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, v2 ]");
    dbfull()->CompactRange(nullptr, nullptr);
    ASSERT_EQ(AllEntriesFor("foo"), "[ ]");
@@ -3305,7 +3530,7 @@ TEST(DBTest, CompactOnFlush) {
    Put("foo", "v3");
    Delete("foo");
    ASSERT_EQ(AllEntriesFor("foo"), "[ DEL, v3 ]");
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ(AllEntriesFor("foo"), "[ DEL ]");
    dbfull()->CompactRange(nullptr, nullptr);
    ASSERT_EQ(AllEntriesFor("foo"), "[ ]");
@@ -3314,7 +3539,7 @@ TEST(DBTest, CompactOnFlush) {
    Put("foo", "v4");
    Put("foo", "v5");
    ASSERT_EQ(AllEntriesFor("foo"), "[ v5, v4 ]");
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ(AllEntriesFor("foo"), "[ v5 ]");
    dbfull()->CompactRange(nullptr, nullptr);
    ASSERT_EQ(AllEntriesFor("foo"), "[ v5 ]");
@@ -3329,7 +3554,7 @@ TEST(DBTest, CompactOnFlush) {
    Put("foo", "v6");
    const Snapshot* snapshot = db_->GetSnapshot();
    Put("foo", "v7");
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ(AllEntriesFor("foo"), "[ v7, v6 ]");
    db_->ReleaseSnapshot(snapshot);

@@ -3343,7 +3568,7 @@ TEST(DBTest, CompactOnFlush) {
    const Snapshot* snapshot1 = db_->GetSnapshot();
    Put("foo", "v8");
    Put("foo", "v9");
-    ASSERT_OK(dbfull()->TEST_CompactMemTable());
+    ASSERT_OK(dbfull()->TEST_FlushMemTable());
    ASSERT_EQ(AllEntriesFor("foo"), "[ v9 ]");
    db_->ReleaseSnapshot(snapshot1);
  } while (ChangeCompactOptions());
@@ -3440,7 +3665,7 @@ void ExpectRecords(
    ASSERT_TRUE(res.sequence > lastSequence);
    ++i;
    lastSequence = res.sequence;
-    ASSERT_TRUE(iter->status().ok());
+    ASSERT_OK(iter->status());
    iter->Next();
  }
  ASSERT_EQ(i, expected_no_records);
@@ -3512,7 +3737,8 @@ TEST(DBTest, TransactionLogIteratorJustEmptyFile) {
    DestroyAndReopen(&options);
    unique_ptr<TransactionLogIterator> iter;
    Status status = dbfull()->GetUpdatesSince(0, &iter);
-    ASSERT_TRUE(!status.ok());
+    // Check that an empty iterator is returned
+    ASSERT_TRUE(!iter->Valid());
  } while (ChangeCompactOptions());
 }

@@ -3543,7 +3769,7 @@ TEST(DBTest, TransactionLogIteratorBatchOperations) {
    Reopen(&options);
    Put("key4", DummyString(1024));
    auto iter = OpenTransactionLogIter(3);
-    ExpectRecords(1, iter);
+    ExpectRecords(2, iter);
  } while (ChangeCompactOptions());
 }

@@ -3611,7 +3837,7 @@ TEST(DBTest, ReadCompaction) {
    }

    // clear level 0 and 1 if necessary.
-    dbfull()->TEST_CompactMemTable();
+    dbfull()->TEST_FlushMemTable();
    dbfull()->TEST_CompactRange(0, nullptr, nullptr);
    dbfull()->TEST_CompactRange(1, nullptr, nullptr);
    ASSERT_EQ(NumTableFilesAtLevel(0), 0);
@@ -4135,7 +4361,7 @@ void PrefixScanInit(DBTest *dbtest) {
  snprintf(buf, sizeof(buf), "%02d______:end", 10);
  keystr = std::string(buf);
  ASSERT_OK(dbtest->Put(keystr, keystr));
-  dbtest->dbfull()->TEST_CompactMemTable();
+  dbtest->dbfull()->TEST_FlushMemTable();
  dbtest->dbfull()->CompactRange(nullptr, nullptr); // move to level 1

  // GROUP 1
@@ -4146,7 +4372,7 @@ void PrefixScanInit(DBTest *dbtest) {
    snprintf(buf, sizeof(buf), "%02d______:end", i+1);
    keystr = std::string(buf);
    ASSERT_OK(dbtest->Put(keystr, keystr));
-    dbtest->dbfull()->TEST_CompactMemTable();
+    dbtest->dbfull()->TEST_FlushMemTable();
  }

  // GROUP 2
@@ -4159,7 +4385,7 @@ void PrefixScanInit(DBTest *dbtest) {
             small_range_sstfiles+i+1);
    keystr = std::string(buf);
    ASSERT_OK(dbtest->Put(keystr, keystr));
-    dbtest->dbfull()->TEST_CompactMemTable();
+    dbtest->dbfull()->TEST_FlushMemTable();
  }
 }

@@ -4203,7 +4429,7 @@ TEST(DBTest, PrefixScan) {
    assert(iter->key().starts_with(prefix));
    count++;
  }
-  ASSERT_TRUE(iter->status().ok());
+  ASSERT_OK(iter->status());
  delete iter;
  ASSERT_EQ(count, 2);
  ASSERT_EQ(env_->random_read_counter_.Read(), 2);
@@ -4220,7 +4446,7 @@ TEST(DBTest, PrefixScan) {
    assert(iter->key().starts_with(prefix));
    count++;
  }
-  ASSERT_TRUE(iter->status().ok());
+  ASSERT_OK(iter->status());
  delete iter;
  ASSERT_EQ(count, 2);
  ASSERT_EQ(env_->random_read_counter_.Read(), 2);
@@ -4237,7 +4463,7 @@ TEST(DBTest, PrefixScan) {
    }
    count++;
  }
-  ASSERT_TRUE(iter->status().ok());
+  ASSERT_OK(iter->status());
  delete iter;
  ASSERT_EQ(count, 2);
  ASSERT_EQ(env_->random_read_counter_.Read(), 11);
@@ -4252,8 +4478,8 @@ std::string MakeKey(unsigned int num) {
 }

 void BM_LogAndApply(int iters, int num_base_files) {
-  std::string dbname = test::TmpDir() + "/leveldb_test_benchmark";
-  DestroyDB(dbname, Options());
+  std::string dbname = test::TmpDir() + "/rocksdb_test_benchmark";
+  ASSERT_OK(DestroyDB(dbname, Options()));

  DB* db = nullptr;
  Options opts;
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_FORMAT_H_
-#define STORAGE_LEVELDB_DB_FORMAT_H_
-
+#pragma once
 #include <stdio.h>
 #include "rocksdb/comparator.h"
 #include "rocksdb/db.h"
@@ -90,7 +93,7 @@ class InternalKeyComparator : public Comparator {
  std::string name_;
 public:
  explicit InternalKeyComparator(const Comparator* c) : user_comparator_(c),
-    name_("leveldb.InternalKeyComparator:" +
+    name_("rocksdb.InternalKeyComparator:" +
          std::string(user_comparator_->Name())) {
  }

@@ -224,5 +227,3 @@ inline LookupKey::~LookupKey() {
 }

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_FORMAT_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -9,12 +14,11 @@
 #include "db/write_batch_internal.h"
 #include "util/testharness.h"
 #include "util/testutil.h"
-#include "boost/lexical_cast.hpp"
 #include "rocksdb/env.h"
 #include <vector>
-#include <boost/algorithm/string.hpp>
 #include <stdlib.h>
 #include <map>
+#include <string>

 namespace rocksdb {

@@ -57,7 +61,7 @@ class DeleteFileTest {
    options.sync = false;
    ReadOptions roptions;
    for (int i = startkey; i < (numkeys + startkey) ; i++) {
-      std::string temp = boost::lexical_cast<std::string>(i);
+      std::string temp = std::to_string(i);
      Slice key(temp);
      Slice value(temp);
      ASSERT_OK(db_->Put(options, key, value));
@@ -92,12 +96,12 @@ class DeleteFileTest {
  void CreateTwoLevels() {
    AddKeys(50000, 10000);
    DBImpl* dbi = reinterpret_cast<DBImpl*>(db_);
-    ASSERT_OK(dbi->TEST_CompactMemTable());
-    ASSERT_OK(dbi->TEST_WaitForCompactMemTable());
+    ASSERT_OK(dbi->TEST_FlushMemTable());
+    ASSERT_OK(dbi->TEST_WaitForFlushMemTable());

    AddKeys(50000, 10000);
-    ASSERT_OK(dbi->TEST_CompactMemTable());
-    ASSERT_OK(dbi->TEST_WaitForCompactMemTable());
+    ASSERT_OK(dbi->TEST_FlushMemTable());
+    ASSERT_OK(dbi->TEST_WaitForFlushMemTable());
  }

 };
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -57,8 +62,8 @@ std::string LogFileName(const std::string& name, uint64_t number) {
  return MakeFileName(name, number, "log");
 }

-std::string ArchivalDirectory(const std::string& dbname) {
-  return dbname + "/" + ARCHIVAL_DIR;
+std::string ArchivalDirectory(const std::string& dir) {
+  return dir + "/" + ARCHIVAL_DIR;
 }
 std::string ArchivedLogFileName(const std::string& name, uint64_t number) {
  assert(number > 0);
@@ -122,7 +127,12 @@ std::string MetaDatabaseName(const std::string& dbname, uint64_t number) {
  return dbname + buf;
 }

+std::string IdentityFileName(const std::string& dbname) {
+  return dbname + "/IDENTITY";
+}
+
 // Owned filenames have the form:
+//    dbname/IDENTITY
 //    dbname/CURRENT
 //    dbname/LOCK
 //    dbname/LOG
@@ -138,7 +148,10 @@ bool ParseFileName(const std::string& fname,
  if (fname.length() > 1 && fname[0] == '/') {
    rest.remove_prefix(1);
  }
-  if (rest == "CURRENT") {
+  if (rest == "IDENTITY") {
+    *number = 0;
+    *type = kIdentityFile;
+  } else if (rest == "CURRENT") {
    *number = 0;
    *type = kCurrentFile;
  } else if (rest == "LOCK") {
@@ -218,4 +231,18 @@ Status SetCurrentFile(Env* env, const std::string& dbname,
  return s;
 }

+Status SetIdentityFile(Env* env, const std::string& dbname) {
+  std::string id = env->GenerateUniqueId();
+  assert(!id.empty());
+  std::string tmp = TempFileName(dbname, id.size());
+  Status s = WriteStringToFileSync(env, id, tmp);
+  if (s.ok()) {
+    s = env->RenameFile(tmp, IdentityFileName(dbname));
+  }
+  if (!s.ok()) {
+    env->DeleteFile(tmp);
+  }
+  return s;
+}
+
 }  // namespace rocksdb
@@ -1,12 +1,15 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 //
 // File names used by DB code

-#ifndef STORAGE_LEVELDB_DB_FILENAME_H_
-#define STORAGE_LEVELDB_DB_FILENAME_H_
-
+#pragma once
 #include <stdint.h>
 #include <string>
 #include "rocksdb/slice.h"
@@ -25,7 +28,8 @@ enum FileType {
  kCurrentFile,
  kTempFile,
  kInfoLogFile,  // Either the current one, or an old one
-  kMetaDatabase
+  kMetaDatabase,
+  kIdentityFile
 };

 // Return the name of the log file with the specified number
@@ -79,7 +83,12 @@ extern std::string OldInfoLogFileName(const std::string& dbname, uint64_t ts,
 extern std::string MetaDatabaseName(const std::string& dbname,
                                    uint64_t number);

-// If filename is a leveldb file, store the type of the file in *type.
+// Return the name of the Identity file which stores a unique number for the db
+// that will get regenerated if the db loses all its data and is recreated fresh
+// either from a backup-image or empty
+extern std::string IdentityFileName(const std::string& dbname);
+
+// If filename is a rocksdb file, store the type of the file in *type.
 // The number encoded in the filename is stored in *number.  If the
 // filename was successfully parsed, returns true.  Else return false.
 extern bool ParseFileName(const std::string& filename,
@@ -91,7 +100,7 @@ extern bool ParseFileName(const std::string& filename,
 extern Status SetCurrentFile(Env* env, const std::string& dbname,
                             uint64_t descriptor_number);

+// Make the IDENTITY file for the db
+extern Status SetIdentityFile(Env* env, const std::string& dbname);

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_FILENAME_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -5,9 +10,7 @@
 // Log format information shared by reader and writer.
 // See ../doc/log_format.txt for more detail.

-#ifndef STORAGE_LEVELDB_DB_LOG_FORMAT_H_
-#define STORAGE_LEVELDB_DB_LOG_FORMAT_H_
-
+#pragma once
 namespace rocksdb {
 namespace log {

@@ -31,5 +34,3 @@ static const int kHeaderSize = 4 + 1 + 2;

 }  // namespace log
 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_LOG_FORMAT_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_LOG_READER_H_
-#define STORAGE_LEVELDB_DB_LOG_READER_H_
-
+#pragma once
 #include <memory>
 #include <stdint.h>

@@ -119,5 +122,3 @@ class Reader {

 }  // namespace log
 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_LOG_READER_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_LOG_WRITER_H_
-#define STORAGE_LEVELDB_DB_LOG_WRITER_H_
-
+#pragma once
 #include <memory>
 #include <stdint.h>
 #include "db/log_format.h"
@@ -50,5 +53,3 @@ class Writer {

 }  // namespace log
 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_LOG_WRITER_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_MEMTABLE_H_
-#define STORAGE_LEVELDB_DB_MEMTABLE_H_
-
+#pragma once
 #include <string>
 #include <memory>
 #include <deque>
@@ -147,5 +150,3 @@ class MemTable {
 };

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_MEMTABLE_H_
@@ -1,6 +1,8 @@
-// Copyright (c) 2012 Facebook.
-
-
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include "db/memtablelist.h"

 #include <string>
@@ -1,8 +1,10 @@
-// Copyright (c) 2012 Facebook.
-
-#ifndef STORAGE_LEVELDB_DB_MEMTABLELIST_H_
-#define STORAGE_LEVELDB_DB_MEMTABLELIST_H_
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//

+#pragma once
 #include <string>
 #include <list>
 #include <deque>
@@ -100,5 +102,3 @@ class MemTableList {
 };

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_MEMTABLELIST_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include "merge_helper.h"
 #include "db/dbformat.h"
 #include "rocksdb/comparator.h"
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #ifndef MERGE_HELPER_H
 #define MERGE_HELPER_H

@@ -1,8 +1,10 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 /**
 * Back-end implementation details specific to the Merge Operator.
- *
- * @author Deon Nicholas (dnicholas@fb.com)
- * Copyright 2013 Facebook
 */

 #include "rocksdb/merge_operator.h"
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include <assert.h>
 #include <memory>
 #include <iostream>
@@ -85,7 +90,7 @@ class Counters {
    }
  }

-  // mapped to a leveldb Delete
+  // mapped to a rocksdb Delete
  bool remove(const string& key) {
    auto s = db_->Delete(delete_option_, key);

@@ -97,7 +102,7 @@ class Counters {
    }
  }

-  // mapped to a leveldb Get
+  // mapped to a rocksdb Get
  bool get(const string& key, uint64_t *value) {
    string str;
    auto s = db_->Get(get_option_, key, &str);
@@ -159,7 +164,7 @@ class MergeBasedCounters : public Counters {
        merge_option_() {
  }

-  // mapped to a leveldb Merge operation
+  // mapped to a rocksdb Merge operation
  virtual bool add(const string& key, uint64_t value) override {
    char encoded[sizeof(uint64_t)];
    EncodeFixed64(encoded, value);
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include <algorithm>
 #include <iostream>
 #include <vector>
@@ -224,7 +229,7 @@ TEST(PerfContextTest, KeyComparisonCount) {
 }

 // make perf_context_test
-// export LEVELDB_TESTS=PerfContextTest.SeekKeyComparison
+// export ROCKSDB_TESTS=PerfContextTest.SeekKeyComparison
 // For one memtable:
 // ./perf_context_test --write_buffer_size=500000 --total_keys=10000
 // For two memtables:
@@ -1,6 +1,7 @@
-// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
-// Use of this source code is governed by a BSD-style license that can be
-// found in the LICENSE file. See the AUTHORS file for names of contributors.
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
 //
 // Wrap an underlying iterator, but exclude any results not starting
 // with a given prefix.  Seeking to keys not beginning with the prefix
@@ -9,9 +10,7 @@
 // underlying iterator would if there happened to be no non-matching
 // keys in the dataset.

-#ifndef STORAGE_LEVELDB_DB_PREFIX_FILTER_ITERATOR_H_
-#define STORAGE_LEVELDB_DB_PREFIX_FILTER_ITERATOR_H_
-
+#pragma once
 #include "rocksdb/iterator.h"

 namespace rocksdb {
@@ -72,5 +71,3 @@ class PrefixFilterIterator : public Iterator {
 };

 }  // namespace rocksdb
-
-#endif
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -74,7 +79,7 @@ class Repairer {
        bytes += tables_[i].meta.file_size;
      }
      Log(options_.info_log,
-          "**** Repaired leveldb %s; "
+          "**** Repaired rocksdb %s; "
          "recovered %d files; %llu bytes. "
          "Some data may have been lost. "
          "****",
@@ -221,7 +226,7 @@ class Repairer {
    Iterator* iter = mem->NewIterator();
    status = BuildTable(dbname_, env_, options_, storage_options_,
                        table_cache_, iter, &meta,
-                        icmp_.user_comparator(), 0, 0);
+                        icmp_.user_comparator(), 0, 0, true);
    delete iter;
    mem->Unref();
    mem = nullptr;
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -25,9 +30,7 @@
 // ... prev vs. next pointer ordering ...
 //

-#ifndef STORAGE_LEVELDB_DB_SKIPLIST_H_
-#define STORAGE_LEVELDB_DB_SKIPLIST_H_
-
+#pragma once
 #include <assert.h>
 #include <stdlib.h>
 #include "port/port.h"
@@ -401,5 +404,3 @@ bool SkipList<Key,Comparator>::Contains(const Key& key) const {
 }

 }  // namespace rocksdb
-
-#endif // STORAGE_LEVELDB_DB_SKIPLIST_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_SNAPSHOT_H_
-#define STORAGE_LEVELDB_DB_SNAPSHOT_H_
-
+#pragma once
 #include "rocksdb/db.h"

 namespace rocksdb {
@@ -81,5 +84,3 @@ class SnapshotList {
 };

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_SNAPSHOT_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -32,7 +37,10 @@ TableCache::TableCache(const std::string& dbname,
      dbname_(dbname),
      options_(options),
      storage_options_(storage_options),
-      cache_(NewLRUCache(entries, options->table_cache_numshardbits)) {}
+      cache_(
+        NewLRUCache(entries, options->table_cache_numshardbits,
+                    options->table_cache_remove_scan_count_limit)) {
+}

 TableCache::~TableCache() {
 }
@@ -1,12 +1,15 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 //
 // Thread-safe (provides internal synchronization)

-#ifndef STORAGE_LEVELDB_DB_TABLE_CACHE_H_
-#define STORAGE_LEVELDB_DB_TABLE_CACHE_H_
-
+#pragma once
 #include <string>
 #include <stdint.h>
 #include "db/dbformat.h"
@@ -73,5 +76,3 @@ class TableCache {
 };

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_TABLE_CACHE_H_
@@ -1,16 +1,21 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #include "db/transaction_log_impl.h"
 #include "db/write_batch_internal.h"

 namespace rocksdb {

 TransactionLogIteratorImpl::TransactionLogIteratorImpl(
-                           const std::string& dbname,
+                           const std::string& dir,
                           const Options* options,
                           const EnvOptions& soptions,
                           const SequenceNumber seq,
                           std::unique_ptr<VectorLogPtr> files,
                           SequenceNumber const * const lastFlushedSequence) :
-    dbname_(dbname),
+    dir_(dir),
    options_(options),
    soptions_(soptions),
    startingSequenceNumber_(seq),
@@ -18,31 +23,34 @@ TransactionLogIteratorImpl::TransactionLogIteratorImpl(
    started_(false),
    isValid_(false),
    currentFileIndex_(0),
+    currentBatchSeq_(0),
+    currentBatchCount_(0),
    lastFlushedSequence_(lastFlushedSequence) {
  assert(startingSequenceNumber_ <= *lastFlushedSequence_);
-  assert(files_.get() != nullptr);
+  assert(files_ != nullptr);

  reporter_.env = options_->env;
  reporter_.info_log = options_->info_log.get();
+  SeekToStartSequence(); // Seek till starting sequence
 }

 Status TransactionLogIteratorImpl::OpenLogFile(
-  const LogFile* logFile,
-  unique_ptr<SequentialFile>* file) {
+    const LogFile* logFile,
+    unique_ptr<SequentialFile>* file) {
  Env* env = options_->env;
  if (logFile->Type() == kArchivedLogFile) {
-    std::string fname = ArchivedLogFileName(dbname_, logFile->LogNumber());
+    std::string fname = ArchivedLogFileName(dir_, logFile->LogNumber());
    return env->NewSequentialFile(fname, file, soptions_);
  } else {
-    std::string fname = LogFileName(dbname_, logFile->LogNumber());
+    std::string fname = LogFileName(dir_, logFile->LogNumber());
    Status status = env->NewSequentialFile(fname, file, soptions_);
    if (!status.ok()) {
      //  If cannot open file in DB directory.
      //  Try the archive dir, as it could have moved in the meanwhile.
-      fname = ArchivedLogFileName(dbname_, logFile->LogNumber());
+      fname = ArchivedLogFileName(dir_, logFile->LogNumber());
      status = env->NewSequentialFile(fname, file, soptions_);
      if (!status.ok()) {
-        return Status::IOError(" Requested file not present in the dir");
+        return Status::IOError("Requested file not present in the dir");
      }
    }
    return status;
@@ -52,7 +60,7 @@ Status TransactionLogIteratorImpl::OpenLogFile(
 BatchResult TransactionLogIteratorImpl::GetBatch()  {
  assert(isValid_);  //  cannot call in a non valid state.
  BatchResult result;
-  result.sequence = currentSequence_;
+  result.sequence = currentBatchSeq_;
  result.writeBatchPtr = std::move(currentBatch_);
  return result;
 }
@@ -65,92 +73,155 @@ bool TransactionLogIteratorImpl::Valid() {
  return started_ && isValid_;
 }

-void TransactionLogIteratorImpl::Next() {
-  LogFile* currentLogFile = files_.get()->at(currentFileIndex_).get();
+bool TransactionLogIteratorImpl::RestrictedRead(
+    Slice* record,
+    std::string* scratch) {
+  // Don't read if no more complete entries to read from logs
+  if (currentBatchSeq_ >= *lastFlushedSequence_) {
+    return false;
+  }
+  return currentLogReader_->ReadRecord(record, scratch);
+}

-//  First seek to the given seqNo. in the current file.
+void TransactionLogIteratorImpl::SeekToStartSequence(
+    uint64_t startFileIndex,
+    bool strict) {
  std::string scratch;
  Slice record;
-  if (!started_) {
-    started_ = true;  // this piece only runs onced.
-    isValid_ = false;
-    if (startingSequenceNumber_ > *lastFlushedSequence_) {
-      currentStatus_ = Status::IOError("Looking for a sequence, "
-                                        "which is not flushed yet.");
-      return;
+  started_ = false;
+  isValid_ = false;
+  if (startingSequenceNumber_ > *lastFlushedSequence_) {
+    currentStatus_ = Status::IOError("Looking for a sequence, "
+                                     "which is not flushed yet.");
+    return;
+  }
+  if (files_->size() <= startFileIndex) {
+    return;
+  }
+  Status s = OpenLogReader(files_->at(startFileIndex).get());
+  if (!s.ok()) {
+    currentStatus_ = s;
+    return;
+  }
+  while (RestrictedRead(&record, &scratch)) {
+    if (record.size() < 12) {
+      reporter_.Corruption(
+        record.size(), Status::Corruption("very small log record"));
+      continue;
    }
-    Status s = OpenLogReader(currentLogFile);
-    if (!s.ok()) {
-      currentStatus_ = s;
+    UpdateCurrentWriteBatch(record);
+    if (currentBatchSeq_ + currentBatchCount_ - 1 >=
+        startingSequenceNumber_) {
+      if (strict && currentBatchSeq_ != startingSequenceNumber_) {
+        currentStatus_ = Status::Corruption("Gap in sequence number. Could not "
+                                            "seek to required sequence number");
+        reporter_.Info(currentStatus_.ToString().c_str());
+        return;
+      } else if (strict) {
+        reporter_.Info("Could seek required sequence number. Iterator will "
+                       "continue.");
+      }
+      isValid_ = true;
+      started_ = true; // set started_ as we could seek till starting sequence
+      return;
+    } else {
      isValid_ = false;
-      return;
-    }
-    while (currentLogReader_->ReadRecord(&record, &scratch)) {
-      if (record.size() < 12) {
-        reporter_.Corruption(
-          record.size(), Status::Corruption("log record too small"));
-        continue;
-      }
-      UpdateCurrentWriteBatch(record);
-      if (currentSequence_ >= startingSequenceNumber_) {
-        assert(currentSequence_ <= *lastFlushedSequence_);
-        isValid_ = true;
-        break;
-      } else {
-        isValid_ = false;
-      }
-    }
-    if (isValid_) {
-      // Done for this iteration
-      return;
    }
  }
-  bool openNextFile = true;
-  while(openNextFile) {
+  // Could not find start sequence in first file. Normally this must be the
+  // only file. Otherwise log the error and let the iterator return next entry
+  if (files_->size() != 1) {
+    currentStatus_ = Status::Corruption("Start sequence was not found, "
+                                        "skipping to the next available");
+    reporter_.Corruption(0, currentStatus_);
+    started_ = true; // Let Next find next available entry
+    Next();
+  }
+}
+
+void TransactionLogIteratorImpl::Next() {
+  std::string scratch;
+  Slice record;
+  isValid_ = false;
+  if (!started_) {  // Runs every time until we can seek to the start sequence
+    return SeekToStartSequence();
+  }
+  while(true) {
    assert(currentLogReader_);
-    if (currentSequence_ < *lastFlushedSequence_) {
+    if (currentBatchSeq_ < *lastFlushedSequence_) {
      if (currentLogReader_->IsEOF()) {
        currentLogReader_->UnmarkEOF();
      }
      while (currentLogReader_->ReadRecord(&record, &scratch)) {
        if (record.size() < 12) {
          reporter_.Corruption(
-            record.size(), Status::Corruption("log record too small"));
+            record.size(), Status::Corruption("very small log record"));
          continue;
        } else {
-          UpdateCurrentWriteBatch(record);
-          openNextFile = false;
-          break;
+          return UpdateCurrentWriteBatch(record);
        }
      }
    }

-    if (openNextFile) {
-      if (currentFileIndex_ < files_.get()->size() - 1) {
-        ++currentFileIndex_;
-        Status status =OpenLogReader(files_.get()->at(currentFileIndex_).get());
-        if (!status.ok()) {
-          isValid_ = false;
-          currentStatus_ = status;
-          return;
-        }
-      } else {
+    // Open the next file
+    if (currentFileIndex_ < files_->size() - 1) {
+      ++currentFileIndex_;
+      Status status =OpenLogReader(files_->at(currentFileIndex_).get());
+      if (!status.ok()) {
        isValid_ = false;
-        openNextFile = false;
-        if (currentSequence_ == *lastFlushedSequence_) {
-          currentStatus_ = Status::OK();
-        } else {
-          currentStatus_ = Status::IOError(" NO MORE DATA LEFT");
-        }
+        currentStatus_ = status;
+        return;
      }
+    } else {
+      isValid_ = false;
+      if (currentBatchSeq_ == *lastFlushedSequence_) {
+        currentStatus_ = Status::OK();
+      } else {
+        currentStatus_ = Status::IOError(" NO MORE DATA LEFT");
+      }
+      return;
    }
  }
 }

+bool TransactionLogIteratorImpl::IsBatchContinuous(
+    const WriteBatch* batch,
+    const SequenceNumber expectedSeq) {
+  assert(batch);
+  SequenceNumber batchSeq = WriteBatchInternal::Sequence(batch);
+  if (started_ && batchSeq != expectedSeq) {
+    char buf[200];
+    snprintf(buf, sizeof(buf),
+             "Discontinuity in log records. Got seq=%lu, Expected seq=%lu, "
+             "Last flushed seq=%lu. Log iterator will seek the correct batch.",
+             batchSeq, expectedSeq, *lastFlushedSequence_);
+    reporter_.Info(buf);
+    return false;
+  }
+  return true;
+}
+
 void TransactionLogIteratorImpl::UpdateCurrentWriteBatch(const Slice& record) {
  WriteBatch* batch = new WriteBatch();
  WriteBatchInternal::SetContents(batch, record);
-  currentSequence_ = WriteBatchInternal::Sequence(batch);
+
+  SequenceNumber expectedSeq = currentBatchSeq_ + currentBatchCount_;
+  if (!IsBatchContinuous(batch, expectedSeq)) {
+    // Seek to the batch having expected sequence number
+    if (expectedSeq < files_->at(currentFileIndex_)->StartSequence()) {
+      // Expected batch must lie in the previous log file
+      currentFileIndex_--;
+      currentFileIndex_ = (currentFileIndex_ >= 0) ? currentFileIndex_ : 0;
+    }
+    startingSequenceNumber_ = expectedSeq;
+    return SeekToStartSequence(currentFileIndex_, true);
+  }
+
+  currentBatchSeq_ = WriteBatchInternal::Sequence(batch);
+  currentBatchCount_ = WriteBatchInternal::Count(batch);
+  // currentBatchSeq_ can only change here
+  assert(currentBatchSeq_ <= *lastFlushedSequence_);
+
  currentBatch_.reset(batch);
  isValid_ = true;
  currentStatus_ = Status::OK();
@@ -1,7 +1,9 @@
-// Copyright 2008-present Facebook. All Rights Reserved.
-#ifndef STORAGE_LEVELDB_INCLUDE_WRITES_ITERATOR_IMPL_H_
-#define STORAGE_LEVELDB_INCLUDE_WRITES_ITERATOR_IMPL_H_
-
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
+#pragma once
 #include <vector>

 #include "rocksdb/env.h"
@@ -19,6 +21,9 @@ struct LogReporter : public log::Reader::Reporter {
  virtual void Corruption(size_t bytes, const Status& s) {
    Log(info_log, "dropping %zu bytes; %s", bytes, s.ToString().c_str());
  }
+  virtual void Info(const char* s) {
+    Log(info_log, "%s", s);
+  }
 };

 class LogFileImpl : public LogFile {
@@ -60,7 +65,7 @@ class LogFileImpl : public LogFile {

 class TransactionLogIteratorImpl : public TransactionLogIterator {
 public:
-  TransactionLogIteratorImpl(const std::string& dbname,
+  TransactionLogIteratorImpl(const std::string& dir,
                             const Options* options,
                             const EnvOptions& soptions,
                             const SequenceNumber seqNum,
@@ -76,10 +81,10 @@ class TransactionLogIteratorImpl : public TransactionLogIterator {
  virtual BatchResult GetBatch();

 private:
-  const std::string& dbname_;
+  const std::string& dir_;
  const Options* options_;
  const EnvOptions& soptions_;
-  const SequenceNumber startingSequenceNumber_;
+  SequenceNumber startingSequenceNumber_;
  std::unique_ptr<VectorLogPtr> files_;
  bool started_;
  bool isValid_;  // not valid when it starts of.
@@ -89,15 +94,19 @@ class TransactionLogIteratorImpl : public TransactionLogIterator {
  unique_ptr<log::Reader> currentLogReader_;
  Status OpenLogFile(const LogFile* logFile, unique_ptr<SequentialFile>* file);
  LogReporter reporter_;
+  SequenceNumber currentBatchSeq_; // sequence number at start of current batch
+  uint64_t currentBatchCount_; // count in current batch
  SequenceNumber const * const lastFlushedSequence_;
-  // represents the sequence number being read currently.
-  SequenceNumber currentSequence_;

+  // Reads from transaction log only if the writebatch record has been written
+  bool RestrictedRead(Slice* record, std::string* scratch);
+  // Seeks to startingSequenceNumber reading from startFileIndex in files_.
+  // If strict is set,then must get a batch starting with startingSequenceNumber
+  void SeekToStartSequence(uint64_t startFileIndex = 0, bool strict = false);
+  // Check if batch is continuous starting from expectedSeq, else return false
+  bool IsBatchContinuous(const WriteBatch* batch, SequenceNumber expectedSeq);
+  // Update current batch if a continuous batch is found, else return false
  void UpdateCurrentWriteBatch(const Slice& record);
  Status OpenLogReader(const LogFile* file);
 };
-
-
-
 }  //  namespace rocksdb
-#endif  //  STORAGE_LEVELDB_INCLUDE_WRITES_ITERATOR_IMPL_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_VERSION_EDIT_H_
-#define STORAGE_LEVELDB_DB_VERSION_EDIT_H_
-
+#pragma once
 #include <set>
 #include <utility>
 #include <vector>
@@ -123,5 +126,3 @@ class VersionEdit {
 };

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_VERSION_EDIT_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1430,9 +1435,9 @@ Status VersionSet::Recover() {
      if (s.ok()) {
        if (edit.has_comparator_ &&
            edit.comparator_ != icmp_.user_comparator()->Name()) {
-          s = Status::InvalidArgument(
-              edit.comparator_ + "does not match existing comparator ",
-              icmp_.user_comparator()->Name());
+          s = Status::InvalidArgument(icmp_.user_comparator()->Name(),
+                                      "does not match existing comparator " +
+                                      edit.comparator_);
        }
      }

@@ -1548,9 +1553,9 @@ Status VersionSet::DumpManifest(Options& options, std::string& dscname,
      if (s.ok()) {
        if (edit.has_comparator_ &&
            edit.comparator_ != icmp_.user_comparator()->Name()) {
-          s = Status::InvalidArgument(
-              edit.comparator_ + "does not match existing comparator ",
-              icmp_.user_comparator()->Name());
+          s = Status::InvalidArgument(icmp_.user_comparator()->Name(),
+                                      "does not match existing comparator " +
+                                      edit.comparator_);
        }
      }

@@ -1645,7 +1650,12 @@ void VersionSet::Finalize(Version* v,
  double max_score = 0;
  int max_score_level = 0;

-  for (int level = 0; level < NumberLevels()-1; level++) {
+  int num_levels_to_check =
+      (options_->compaction_style != kCompactionStyleUniversal) ?
+          NumberLevels() - 1 : 1;
+
+  for (int level = 0; level < num_levels_to_check; level++) {
+
    double score;
    if (level == 0) {
      // We treat level-0 specially by bounding the number of files
@@ -2240,8 +2250,10 @@ Compaction* VersionSet::PickCompactionUniversalSizeAmp(
  assert(start_index >= 0 && start_index < file_by_time.size() - 1);

  // create a compaction request
+  // We always compact all the files, so always compress.
  Compaction* c = new Compaction(level, level, MaxFileSizeForLevel(level),
-                                 LLONG_MAX, NumberLevels());
+                                 LLONG_MAX, NumberLevels(), false,
+                                 true);
  c->score_ = score;
  for (unsigned int loop = start_index; loop < file_by_time.size(); loop++) {
    int index = file_by_time[loop];
@@ -2346,11 +2358,30 @@ Compaction* VersionSet::PickCompactionUniversalReadAmp(
  if (!done || candidate_count <= 1) {
    return nullptr;
  }
+  unsigned int first_index_after = start_index + candidate_count;
+  // Compression is enabled if files compacted earlier already reached
+  // size ratio of compression.
+  bool enable_compression = true;
+  int ratio_to_compress =
+      options_->compaction_options_universal.compression_size_percent;
+  if (ratio_to_compress >= 0) {
+    uint64_t total_size = TotalFileSize(current_->files_[level]);
+    uint64_t older_file_size = 0;
+    for (unsigned int i = file_by_time.size() - 1; i >= first_index_after;
+        i--) {
+      older_file_size += current_->files_[level][file_by_time[i]]->file_size;
+      if (older_file_size * 100L >= total_size * (long) ratio_to_compress) {
+        enable_compression = false;
+        break;
+      }
+    }
+  }
  Compaction* c = new Compaction(level, level, MaxFileSizeForLevel(level),
-                                 LLONG_MAX, NumberLevels());
+                                 LLONG_MAX, NumberLevels(), false,
+                                 enable_compression);
  c->score_ = score;

-  for (unsigned int i = start_index; i < start_index + candidate_count; i++) {
+  for (unsigned int i = start_index; i < first_index_after; i++) {
    int index = file_by_time[i];
    FileMetaData* f = current_->files_[level][index];
    c->inputs_[0].push_back(f);
@@ -2874,7 +2905,7 @@ Compaction* VersionSet::CompactRange(

 Compaction::Compaction(int level, int out_level, uint64_t target_file_size,
  uint64_t max_grandparent_overlap_bytes, int number_levels,
-  bool seek_compaction)
+  bool seek_compaction, bool enable_compression)
    : level_(level),
      out_level_(out_level),
      max_output_file_size_(target_file_size),
@@ -2882,6 +2913,7 @@ Compaction::Compaction(int level, int out_level, uint64_t target_file_size,
      input_version_(nullptr),
      number_levels_(number_levels),
      seek_compaction_(seek_compaction),
+      enable_compression_(enable_compression),
      grandparent_index_(0),
      seen_key_(false),
      overlapped_bytes_(0),
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -12,9 +17,7 @@
 // Version,VersionSet are thread-compatible, but require external
 // synchronization on all accesses.

-#ifndef STORAGE_LEVELDB_DB_VERSION_SET_H_
-#define STORAGE_LEVELDB_DB_VERSION_SET_H_
-
+#pragma once
 #include <map>
 #include <memory>
 #include <set>
@@ -32,11 +35,9 @@ namespace log { class Writer; }
 class Compaction;
 class Iterator;
 class MemTable;
-class TableBuilder;
 class TableCache;
 class Version;
 class VersionSet;
-class WritableFile;

 // Return the smallest index i such that files[i]->largest >= key.
 // Return files.size() if there is no such file.
@@ -316,7 +317,17 @@ class VersionSet {
  // Returns true iff some level needs a compaction because it has
  // exceeded its target size.
  bool NeedsSizeCompaction() const {
-    for (int i = 0; i < NumberLevels()-1; i++) {
+    // In universal compaction case, this check doesn't really
+    // check the compaction condition, but checks num of files threshold
+    // only. We are not going to miss any compaction opportunity
+    // but it's likely that more compactions are scheduled but
+    // ending up with nothing to do. We can improve it later.
+    // TODO: improve this function to be accurate for universal
+    //       compactions.
+    int num_levels_to_check =
+        (options_->compaction_style != kCompactionStyleUniversal) ?
+            NumberLevels() - 1 : 1;
+    for (int i = 0; i < num_levels_to_check; i++) {
      if (current_->compaction_score_[i] >= 1) {
        return true;
      }
@@ -547,6 +558,9 @@ class Compaction {
  // Maximum size of files to build during this compaction.
  uint64_t MaxOutputFileSize() const { return max_output_file_size_; }

+  // Whether compression will be enabled for compaction outputs
+  bool enable_compression() const { return enable_compression_; }
+
  // Is this a trivial compaction that can be implemented by just
  // moving a single input file to the next level (no merging or splitting)
  bool IsTrivialMove() const;
@@ -581,7 +595,7 @@ class Compaction {

  explicit Compaction(int level, int out_level, uint64_t target_file_size,
    uint64_t max_grandparent_overlap_bytes, int number_levels,
-    bool seek_compaction = false);
+    bool seek_compaction = false, bool enable_compression = true);

  int level_;
  int out_level_; // levels to which output files are stored
@@ -592,6 +606,7 @@ class Compaction {
  int number_levels_;

  bool seek_compaction_;
+  bool enable_compression_;

  // Each compaction reads inputs from "level_" and "level_+1"
  std::vector<FileMetaData*> inputs_[2];      // The two sets of inputs
@@ -628,5 +643,3 @@ class Compaction {
 };

 }  // namespace rocksdb
-
-#endif  // STORAGE_LEVELDB_DB_VERSION_SET_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2012 Facebook. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,10 +1,13 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
-#define STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
-
+#pragma once
 #include "rocksdb/types.h"
 #include "rocksdb/write_batch.h"
 #include "rocksdb/db.h"
@@ -52,6 +55,3 @@ class WriteBatchInternal {
 };

 }  // namespace rocksdb
-
-
-#endif  // STORAGE_LEVELDB_DB_WRITE_BATCH_INTERNAL_H_
@@ -1,3 +1,8 @@
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 // Copyright (c) 2011 The LevelDB Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
@@ -1,4 +1,4 @@
-This directory contains the hdfs extensions needed to make leveldb store
+This directory contains the hdfs extensions needed to make rocksdb store
 files in HDFS.

 The hdfs.h file is copied from the Apache Hadoop 1.0 source code. 
@@ -7,12 +7,12 @@ It defines the libhdfs library
 data in HDFS.  The libhdfs.a is copied from the Apache Hadoop 1.0 build. 
 It implements the API defined in hdfs.h. If your hadoop cluster is running
 a different hadoop release, then install these two files manually from your
-hadoop distribution and then recompile leveldb.
+hadoop distribution and then recompile rocksdb.

-The env_hdfs.h file defines the leveldb objects that are needed to talk to an
+The env_hdfs.h file defines the rocksdb objects that are needed to talk to an
 underlying filesystem. 

-If you want to compile leveldb with hdfs support, please set the following
+If you want to compile rocksdb with hdfs support, please set the following
 enviroment variables appropriately:
   USE_HDFS=1
   JAVA_HOME=/usr/local/jdk-6u22-64
@@ -1,25 +1,10 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-// Copyright (c) 2012 Facebook. All rights reserved.
-
-#ifndef LEVELDB_HDFS_FILE_H
-#define LEVELDB_HDFS_FILE_H
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//

+#pragma once
 #include <algorithm>
 #include <stdio.h>
 #include <sys/time.h>
@@ -55,7 +40,7 @@ private:
 };

 //
-// The HDFS environment for leveldb. This class overrides all the
+// The HDFS environment for rocksdb. This class overrides all the
 // file/dir access methods and delegates the thread-mgmt methods to the
 // default posix environment.
 //
@@ -81,6 +66,10 @@ class HdfsEnv : public Env {
  virtual Status NewWritableFile(const std::string& fname,
                                 WritableFile** result);

+  virtual Status NewRandomRWFile(const std::string& fname,
+                                 unique_ptr<RandomRWFile>* result,
+                                 const EnvOptions& options);
+
  virtual bool FileExists(const std::string& fname);

  virtual Status GetChildren(const std::string& path,
@@ -227,7 +216,7 @@ class HdfsEnv : public Env {

 public:
  HdfsEnv(const std::string& fsname) {
-    fprintf(stderr, "You have not build leveldb with HDFS support\n");
+    fprintf(stderr, "You have not build rocksdb with HDFS support\n");
    fprintf(stderr, "Please see hdfs/README for details\n");
    throw new std::exception();
  }
@@ -251,6 +240,12 @@ class HdfsEnv : public Env {
    return notsup;
  }

+  virtual Status NewRandomRWFile(const std::string& fname,
+                                 unique_ptr<RandomRWFile>* result,
+                                 const EnvOptions& options) {
+    return notsup;
+  }
+
  virtual bool FileExists(const std::string& fname){return false;}

  virtual Status GetChildren(const std::string& path,
@@ -305,5 +300,3 @@ class HdfsEnv : public Env {
 }

 #endif // USE_HDFS
-
-#endif // LEVELDB_HDFS_FILE_H
@@ -1,21 +1,8 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
+//  Copyright (c) 2013, Facebook, Inc.  All rights reserved.
+//  This source code is licensed under the BSD-style license found in the
+//  LICENSE file in the root directory of this source tree. An additional grant
+//  of patent rights can be found in the PATENTS file in the same directory.
+//
 #ifndef LIBHDFS_HDFS_H
 #define LIBHDFS_HDFS_H

@@ -215,7 +215,7 @@ TEST(MemEnvTest, DBTest) {
  delete iterator;

  DBImpl* dbi = reinterpret_cast<DBImpl*>(db);
-  ASSERT_OK(dbi->TEST_CompactMemTable());
+  ASSERT_OK(dbi->TEST_FlushMemTable());

  for (size_t i = 0; i < 3; ++i) {
    std::string res;
@@ -38,6 +38,4 @@ class Arena {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_ARENA_H_
@@ -28,10 +28,22 @@ using std::shared_ptr;

 class Cache;

-// Create a new cache with a fixed size capacity.  This implementation
-// of Cache uses a least-recently-used eviction policy.
+// Create a new cache with a fixed size capacity. The cache is sharded
+// to 2^numShardBits shards, by hash of the key. The total capacity
+// is divided and evenly assigned to each shard. Inside each shard,
+// the eviction is done in two passes: first try to free spaces by
+// evicting entries that are among the most least used removeScanCountLimit
+// entries and do not have reference other than by the cache itself, in
+// the least-used order. If not enough space is freed, further free the
+// entries in least used order.
+//
+// The functions without parameter numShardBits and/or removeScanCountLimit
+// use default values. removeScanCountLimit's default value is 0, which
+// means a strict LRU order inside each shard.
 extern shared_ptr<Cache> NewLRUCache(size_t capacity);
 extern shared_ptr<Cache> NewLRUCache(size_t capacity, int numShardBits);
+extern shared_ptr<Cache> NewLRUCache(size_t capacity, int numShardBits,
+                                     int removeScanCountLimit);

 class Cache {
 public:
@@ -103,6 +115,4 @@ class Cache {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_UTIL_CACHE_H_
@@ -68,6 +68,4 @@ class DefaultCompactionFilterFactory : public CompactionFilterFactory {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_COMPACTION_FILTER_H_
@@ -13,7 +13,7 @@ class Slice;

 // A Comparator object provides a total order across slices that are
 // used as keys in an sstable or a database.  A Comparator implementation
-// must be thread-safe since leveldb may invoke its methods concurrently
+// must be thread-safe since rocksdb may invoke its methods concurrently
 // from multiple threads.
 class Comparator {
 public:
@@ -33,7 +33,7 @@ class Comparator {
  // the comparator implementation changes in a way that will cause
  // the relative ordering of any two keys to change.
  //
-  // Names starting with "leveldb." are reserved and should not be used
+  // Names starting with "rocksdb." are reserved and should not be used
  // by any clients of this package.
  virtual const char* Name() const = 0;

@@ -60,6 +60,4 @@ extern const Comparator* BytewiseComparator();

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_COMPARATOR_H_
@@ -175,11 +175,11 @@ class DB {
  //
  // Valid property names include:
  //
-  //  "leveldb.num-files-at-level<N>" - return the number of files at level <N>,
+  //  "rocksdb.num-files-at-level<N>" - return the number of files at level <N>,
  //     where <N> is an ASCII representation of a level number (e.g. "0").
-  //  "leveldb.stats" - returns a multi-line string that describes statistics
+  //  "rocksdb.stats" - returns a multi-line string that describes statistics
  //     about the internal operation of the DB.
-  //  "leveldb.sstables" - returns a multi-line string that describes all
+  //  "rocksdb.sstables" - returns a multi-line string that describes all
  //     of the sstables that make up the db contents.
  virtual bool GetProperty(const Slice& property, std::string* value) = 0;

@@ -308,6 +308,4 @@ Status RepairDB(const std::string& dbname, const Options& options);

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_DB_H_
@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.
 //
-// An Env is an interface used by the leveldb implementation to access
+// An Env is an interface used by the rocksdb implementation to access
 // operating system functionality like the filesystem etc.  Callers
 // may wish to provide a custom Env object when opening a database to
 // get fine gain control; e.g., to rate limit file system operations.
@@ -28,6 +28,7 @@ class RandomAccessFile;
 class SequentialFile;
 class Slice;
 class WritableFile;
+class RandomRWFile;
 class Options;

 using std::unique_ptr;
@@ -72,7 +73,7 @@ class Env {
  // system.  Sophisticated users may wish to provide their own Env
  // implementation instead of relying on this default environment.
  //
-  // The result of Default() belongs to leveldb and must never be deleted.
+  // The result of Default() belongs to rocksdb and must never be deleted.
  static Env* Default();

  // Create a brand new sequentially-readable file with the specified name.
@@ -109,6 +110,15 @@ class Env {
                                 unique_ptr<WritableFile>* result,
                                 const EnvOptions& options) = 0;

+  // Create an object that both reads and writes to a file on
+  // specified offsets (random access). If file already exists,
+  // does not overwrite it. On success, stores a pointer to the
+  // new file in *result and returns OK. On failure stores nullptr
+  // in *result and returns non-OK.
+  virtual Status NewRandomRWFile(const std::string& fname,
+                                 unique_ptr<RandomRWFile>* result,
+                                 const EnvOptions& options) = 0;
+
  // Returns true iff the named file exists.
  virtual bool FileExists(const std::string& fname) = 0;

@@ -223,6 +233,9 @@ class Env {
  // Converts seconds-since-Jan-01-1970 to a printable string
  virtual std::string TimeToString(uint64_t time) = 0;

+  // Generates a unique id that can be used to identify a db
+  virtual std::string GenerateUniqueId();
+
 private:
  // No copying allowed
  Env(const Env&);
@@ -329,7 +342,7 @@ class WritableFile {

  /*
   * Sync data and/or metadata as well.
-   * By default, sync only metadata.
+   * By default, sync only data.
   * Override this method for environments where we need to sync
   * metadata as well.
   */
@@ -418,6 +431,55 @@ class WritableFile {
  void operator=(const WritableFile&);
 };

+// A file abstraction for random reading and writing.
+class RandomRWFile {
+ public:
+  RandomRWFile() {}
+  virtual ~RandomRWFile() {}
+
+  // Write data from Slice data to file starting from offset
+  // Returns IOError on failure, but does not guarantee
+  // atomicity of a write.  Returns OK status on success.
+  //
+  // Safe for concurrent use.
+  virtual Status Write(uint64_t offset, const Slice& data) = 0;
+  // Read up to "n" bytes from the file starting at "offset".
+  // "scratch[0..n-1]" may be written by this routine.  Sets "*result"
+  // to the data that was read (including if fewer than "n" bytes were
+  // successfully read).  May set "*result" to point at data in
+  // "scratch[0..n-1]", so "scratch[0..n-1]" must be live when
+  // "*result" is used.  If an error was encountered, returns a non-OK
+  // status.
+  //
+  // Safe for concurrent use by multiple threads.
+  virtual Status Read(uint64_t offset, size_t n, Slice* result,
+                      char* scratch) const = 0;
+  virtual Status Close() = 0; // closes the file
+  virtual Status Sync() = 0; // sync data
+
+  /*
+   * Sync data and/or metadata as well.
+   * By default, sync only data.
+   * Override this method for environments where we need to sync
+   * metadata as well.
+   */
+  virtual Status Fsync() {
+    return Sync();
+  }
+
+  /*
+   * Pre-allocate space for a file.
+   */
+  virtual Status Allocate(off_t offset, off_t len) {
+    return Status::OK();
+  }
+
+ private:
+  // No copying allowed
+  RandomRWFile(const RandomRWFile&);
+  void operator=(const RandomRWFile&);
+};
+
 // An interface for writing log messages.
 class Logger {
 public:
@@ -497,6 +559,10 @@ class EnvWrapper : public Env {
                         const EnvOptions& options) {
    return target_->NewWritableFile(f, r, options);
  }
+  Status NewRandomRWFile(const std::string& f, unique_ptr<RandomRWFile>* r,
+                         const EnvOptions& options) {
+    return target_->NewRandomRWFile(f, r, options);
+  }
  bool FileExists(const std::string& f) { return target_->FileExists(f); }
  Status GetChildren(const std::string& dir, std::vector<std::string>* r) {
    return target_->GetChildren(dir, r);
@@ -565,6 +631,4 @@ class EnvWrapper : public Env {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_ENV_H_
@@ -4,8 +4,8 @@
 //
 // A database can be configured with a custom FilterPolicy object.
 // This object is responsible for creating a small filter from a set
-// of keys.  These filters are stored in leveldb and are consulted
-// automatically by leveldb to decide whether or not to read some
+// of keys.  These filters are stored in rocksdb and are consulted
+// automatically by rocksdb to decide whether or not to read some
 // information from disk. In many cases, a filter can cut down the
 // number of disk seeks form a handful to a single disk seek per
 // DB::Get() call.
@@ -66,6 +66,5 @@ class FilterPolicy {
 extern const FilterPolicy* NewBloomFilterPolicy(int bits_per_key);

 }
-#include "rocksdb/rocksdb_to_leveldb.h"

 #endif  // STORAGE_ROCKSDB_INCLUDE_FILTER_POLICY_H_
@@ -99,6 +99,4 @@ extern Iterator* NewErrorIterator(const Status& status);

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_ITERATOR_H_
@@ -10,11 +10,6 @@ class LDBTool {
  void Run(int argc, char** argv, Options = Options());
 };

-namespace leveldb = rocksdb;
-
-
 } // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif // STORAGE_ROCKSDB_INCLUDE_LDB_TOOL_H
@@ -10,7 +10,7 @@
 // The liberal use of assertions is encouraged to enforce (1).
 //
 // The factory will be passed an Arena object when a new MemTableRep is
-// requested. The API for this object is in leveldb/arena.h.
+// requested. The API for this object is in rocksdb/arena.h.
 //
 // Users can implement their own memtable representations. We include four
 // types built in:
@@ -177,7 +177,7 @@ public:

 // TransformReps are backed by an unordered map of buffers to buckets. When
 // looking up a key, the user key is extracted and a user-supplied transform
-// function (see leveldb/slice_transform.h) is applied to get the key into the
+// function (see rocksdb/slice_transform.h) is applied to get the key into the
 // unordered map. This allows the user to bin user keys based on arbitrary
 // criteria. Two example implementations are UnsortedRepFactory and
 // PrefixHashRepFactory.
@@ -256,6 +256,4 @@ public:

 }

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif // STORAGE_ROCKSDB_DB_MEMTABLEREP_H_
@@ -7,7 +7,7 @@

 #include <string>
 #include <deque>
-#include "rocksdb/slice.h"  // TODO: Remove this when migration is done;
+#include "rocksdb/slice.h"

 namespace rocksdb {

@@ -144,6 +144,4 @@ class AssociativeMergeOperator : public MergeOperator {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_MERGE_OPERATOR_H_
@@ -165,7 +165,7 @@ struct Options {
  // a block is the unit of reading from disk).

  // If non-NULL use the specified cache for blocks.
-  // If NULL, leveldb will automatically create and use an 8MB internal cache.
+  // If NULL, rocksdb will automatically create and use an 8MB internal cache.
  // Default: nullptr
  shared_ptr<Cache> block_cache;

@@ -351,13 +351,21 @@ struct Options {
  // Default value is 1800 (half an hour).
  int db_stats_log_interval;

-  // This specifies the log dir.
+  // This specifies the info LOG dir.
  // If it is empty, the log files will be in the same dir as data.
  // If it is non empty, the log files will be in the specified dir,
  // and the db data dir's absolute path will be used as the log file
  // name's prefix.
  std::string db_log_dir;

+  // This specifies the absolute dir path for write-ahead logs (WAL).
+  // If it is empty, the log files will be in the same dir as data,
+  //   dbname is used as the data dir by default
+  // If it is non empty, the log files will be in kept the specified dir.
+  // When destroying the db,
+  //   all log files in wal_dir and the dir itself is deleted
+  std::string wal_dir;
+
  // Disable compaction triggered by seek.
  // With bloomfilter and fast storage, a miss on one level
  // is very cheap if the file handle is cached in table cache
@@ -434,6 +442,15 @@ struct Options {
  // Number of shards used for table cache.
  int table_cache_numshardbits;

+  // During data eviction of table's LRU cache, it would be inefficient
+  // to strictly follow LRU because this piece of memory will not really
+  // be released unless its refcount falls to zero. Instead, make two
+  // passes: the first pass will release items with refcount = 1,
+  // and if not enough space releases after scanning the number of
+  // elements specified by this parameter, we will remove items in LRU
+  // order.
+  int table_cache_remove_scan_count_limit;
+
  // size of one block in arena memory allocation.
  // If <= 0, a proper value is automatically calculated (usually 1/10 of
  // writer_buffer_size).
@@ -461,7 +478,7 @@ struct Options {

  // The number of seconds a WAL(write ahead log) should be kept after it has
  // been marked as Not Live. If the value is set. The WAL files are moved to
-  // the archive direcotory and deleted after the given TTL.
+  // the archive directory and deleted after the given TTL.
  // If set to 0, WAL files are deleted as soon as they are not required by
  // the database.
  // If set to std::numeric_limits<uint64_t>::max() the WAL files will never be
@@ -497,7 +514,7 @@ struct Options {
  // Default: false
  bool skip_log_error_on_recovery;

-  // if not zero, dump leveldb.stats to LOG every stats_dump_period_sec
+  // if not zero, dump rocksdb.stats to LOG every stats_dump_period_sec
  // Default: 3600 (1 hour)
  unsigned int stats_dump_period_sec;

@@ -675,6 +692,4 @@ struct FlushOptions {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_OPTIONS_H_
@@ -36,6 +36,4 @@ extern __thread PerfContext perf_context;

 }

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif
@@ -1,9 +0,0 @@
-// Copyright (c) 2013 Facebook.
-
-#pragma once
-
-//
-// This is for backward compatibility with applications that use the
-// 'leveldb' namespace. This file will be deleted in a future release.
-//
-namespace leveldb = rocksdb;
@@ -91,7 +91,7 @@ class Slice {
            (memcmp(data_, x.data_, x.size_) == 0));
  }

- // private: make these public for leveldbjni access
+ // private: make these public for rocksdbjni access
  const char* data_;
  size_t size_;

@@ -119,6 +119,4 @@ inline int Slice::compare(const Slice& b) const {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_SLICE_H_
@@ -40,6 +40,4 @@ extern const SliceTransform* NewNoopTransform();

 }

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_SLICE_TRANSFORM_H_
@@ -198,7 +198,7 @@ class Histogram {

 /**
 * A dumb ticker which keeps incrementing through its life time.
- * Not thread safe. Locking is currently managed by external leveldb lock
+ * Thread safe. Locking managed by implementation of this interface.
 */
 class Ticker {
 public:
@@ -260,6 +260,4 @@ inline void SetTickerCount(std::shared_ptr<Statistics> statistics,

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_STATISTICS_H_
@@ -123,6 +123,4 @@ inline void Status::operator=(const Status& s) {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_STATUS_H_
@@ -30,7 +30,12 @@ class TableBuilder {
  // caller to close the file after calling Finish(). The output file
  // will be part of level specified by 'level'.  A value of -1 means
  // that the caller does not know which level the output file will reside.
-  TableBuilder(const Options& options, WritableFile* file, int level=-1);
+  //
+  // If enable_compression=true, this table will follow the compression
+  // setting given in parameter options. If enable_compression=false, the
+  // table will not be compressed.
+  TableBuilder(const Options& options, WritableFile* file, int level=-1,
+               const bool enable_compression=true);

  // REQUIRES: Either Finish() or Abandon() has been called.
  ~TableBuilder();
@@ -92,6 +97,4 @@ class TableBuilder {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_TABLE_BUILDER_H_
@@ -0,0 +1,46 @@
+// Copyright (c) 2013 Facebook
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file. See the AUTHORS file for names of contributors.
+#pragma once
+
+#include <string>
+#include <unordered_map>
+
+namespace rocksdb {
+
+// TableStats contains a bunch of read-only stats of its associated
+// table.
+struct TableStats {
+ public:
+  // TODO(kailiu) we do not support user collected stats yet.
+  //
+  // Other than basic table stats, each table may also have the user
+  // collected stats.
+  // The value of the user-collected stats are encoded as raw bytes --
+  // users have to interprete these values by themselves.
+  typedef
+    std::unordered_map<std::string, std::string>
+    UserCollectedStats;
+
+  // the total size of all data blocks.
+  uint64_t data_size = 0;
+  // the total size of all index blocks.
+  uint64_t index_size = 0;
+  // total raw key size
+  uint64_t raw_key_size = 0;
+  // total raw value size
+  uint64_t raw_value_size = 0;
+  // the number of blocks in this table
+  uint64_t num_data_blocks = 0;
+  // the number of entries in this table
+  uint64_t num_entries = 0;
+
+  // The name of the filter policy used in this table.
+  // If no filter policy is used, `filter_policy_name` will be an empty string.
+  std::string filter_policy_name;
+
+  // user collected stats
+  UserCollectedStats user_collected_stats;
+};
+
+}  // namespace rocksdb
@@ -67,8 +67,8 @@ class TransactionLogIterator {
  // REQUIRES: Valid() to be true.
  virtual void Next() = 0;

-  // Return's ok if the iterator is valid.
-  // Return the Error when something has gone wrong.
+  // Returns ok if the iterator is valid.
+  // Returns the Error when something has gone wrong.
  virtual Status status() = 0;

  // If valid return's the current write_batch and the sequence number of the
@@ -78,6 +78,4 @@ class TransactionLogIterator {
 };
 } //  namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_TRANSACTION_LOG_ITERATOR_H_
@@ -12,6 +12,4 @@ typedef uint64_t SequenceNumber;

 }  //  namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif //  STORAGE_ROCKSDB_INCLUDE_TYPES_H_
@@ -51,6 +51,23 @@ class CompactionOptionsUniversal {
  // 300 bytes of storage.
  unsigned int max_size_amplification_percent;

+  // If this option is set to be -1 (the default value), all the output files
+  // will follow compression type specified.
+  //
+  // If this option is not negative, we will try to make sure compressed
+  // size is just above this value. In normal cases, at least this percentage
+  // of data will be compressed.
+  // When we are compacting to a new file, here is the criteria whether
+  // it needs to be compressed: assuming here are the list of files sorted
+  // by generation time:
+  //    A1...An B1...Bm C1...Ct
+  // where A1 is the newest and Ct is the oldest, and we are going to compact
+  // B1...Bm, we calculate the total size of all the files as total_size, as
+  // well as  the total size of C1...Ct as total_C, the compaction output file
+  // will be compressed iff
+  //   total_C / total_size < this percentage
+  int compression_size_percent;
+
  // The algorithm used to stop picking files into a single compaction run
  // Default: kCompactionStopStyleTotalSize
  CompactionStopStyle stop_style;
@@ -61,12 +78,11 @@ class CompactionOptionsUniversal {
    min_merge_width(2),
    max_merge_width(UINT_MAX),
    max_size_amplification_percent(200),
+    compression_size_percent(-1),
    stop_style(kCompactionStopStyleTotalSize) {
  }
 };

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_UNIVERSAL_COMPACTION_OPTIONS_H
@@ -96,6 +96,4 @@ class WriteBatch {

 }  // namespace rocksdb

-#include "rocksdb/rocksdb_to_leveldb.h"
-
 #endif  // STORAGE_ROCKSDB_INCLUDE_WRITE_BATCH_H_
@@ -2,9 +2,7 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef LEVELDB_INCLUDE_UTILITIES_STACKABLE_DB_H_
-#define LEVELDB_INCLUDE_UTILITIES_STACKABLE_DB_H_
-
+#pragma once
 #include "rocksdb/db.h"

 namespace rocksdb {
@@ -161,5 +159,3 @@ class StackableDB : public DB {
 };

 } //  namespace rocksdb
-
-#endif  //  LEVELDB_INCLUDE_UTILITIES_STACKABLE_DB_H_
@@ -2,14 +2,12 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file. See the AUTHORS file for names of contributors.

-#ifndef LEVELDB_INCLUDE_UTILITIES_UTILITY_DB_H_
-#define LEVELDB_INCLUDE_UTILITIES_UTILITY_DB_H_
-
+#pragma once
 #include "stackable_db.h"

 namespace rocksdb {

-// This class contains APIs to open leveldb with specific support eg. TTL
+// This class contains APIs to open rocksdb with specific support eg. TTL
 class UtilityDB {

  public:
@@ -50,5 +48,3 @@ class UtilityDB {
 };

 } //  namespace rocksdb
-
-#endif  //  LEVELDB_INCLUDE_UTILITIES_UTILITY_DB_H_
@@ -1,20 +0,0 @@
-###############################################################################
-leveldb - A Java port of LevelDB (https://github.com/dain/leveldb)
-
-This is a Java port of LevelDB. We only need the interface part, so the
-implementation part is not checked in.
-
-This is based on commit: c8d074b3d95f30612e573bba689b85749031d639 from
-https://github.com/dain/leveldb.git
-
-###############################################################################
-
-leveldbjni - JNI Wrapper for LevelDB (https://github.com/fusesource/leveldbjni)
-
-Provide LevelDB implementation by using JNI wrapper. It is written using HawtJNI
-which is JNI code generatori (http://hawtjni.fusesource.org/).
-
-This is based on commmit: 8bac93ec1bcc97a098a1eaac265ea04b766ef574 from
-https://github.com/fusesource/leveldbjni.git
-
-###############################################################################
@@ -1,22 +0,0 @@
-target/
-/var
-pom.xml.versionsBackup
-test-output/
-/atlassian-ide-plugin.x
-.idea
-.*.swp
-.*.swo
-leveldb-c
-*~
-*.swp
-.idea
-.idea/*
-*.iml
-*.ipr
-*.iws
-.DS_Store
-.scala_dependencies
-.project
-.classpath
-.settings
-eclipse-classes
Author	SHA1	Message	Date
Mayank Agarwal	9b50106f9a	Dbid feature Summary: Create a new type of file on startup if it doesn't already exist called DBID. This will store a unique number generated from boost library's uuid header file. The use-case is to identify the case of a db losing all its data and coming back up either empty or from an image(backup/live replica's recovery) the key point to note is that DBID is not stored in a backup or db snapshot It's preferable to use Boost for uuid because: 1) A non-standard way of generating uuid is not good 2) /proc/sys/kernel/random/uuid generates a uuid but only on linux environments and the solution would not be clean 3) c++ doesn't have any direct way to get a uuid 4) Boost is a very good library that was already having linkage in rocksdb from third-party Note: I had to update the TOOLCHAIN_REV in build files to get latest verison of boost from third-party as the older version had a bug. I had to put Wno-uninitialized in Makefile because boost-1.51 has an unitialized variable and rocksdb would not comiple otherwise. Latet open-source for boost is 1.54 but is not there in third-party. I have notified the concerned people in fbcode about it. @kailiu : While releasing to third-party, an additional dependency will need to be created for boost in TARGETS file. I can help identify. Test Plan: Expand db_test to test 2 cases 1) Restarting db with Id file present - verify that no change to Id 2)Restarting db with Id file deleted - verify that a different Id is there after reopen Also run make all check Reviewers: dhruba, haobo, kailiu, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13587	2013-10-22 12:23:34 -07:00
Mayank Agarwal	ae8e0770b4	Disallow transaction log iterator to skip sequences Summary: This is expected to solve the "gaps in transaction log iterator" problem. * After a lot of observations on the gaps on the sigmafio machines I found that it is due to a race between log reader and writer always. * So when we drop the wormhole subscription and refresh the iterator, the gaps are not there. * It is NOT due to some boundary or corner case left unattended in the iterator logic because I checked many instances of the gaps against their log files with ldb. The log files are NOT corrupted also. * The solution is to not allow the iterator to read incompletely written sequences and detect gaps inside itself and invalidate it which will cause the application to refresh the iterator normally and seek to the required sequence properly. * Thus, the iterator can at least guarantee that it will not give any gaps. Test Plan: * db_test based log iterator tests * db_repl_stress * testing on sigmafio setup to see gaps go away Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13593	2013-10-22 11:45:35 -07:00
Igor Canadi	c674b42d52	Rephrasing the comment Summary: Per @haobo's request, rephrasing the comment for allocate Test Plan: It's a comment! Reviewers: haobo, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13575	2013-10-21 10:23:56 -07:00
Kai Liu	43ee5e2b3a	Fix the valgrind error in newly added unittests for table stats Summary: Previous the newly added test called NewBloomFilter without releasing it at the end of the test, which resulted in memory leak and was detected by valgrind. Test Plan: Ran valgrind test.	2013-10-20 22:02:05 -07:00
Igor Canadi	bcc8557901	tmpfs does not support fallocate Summary: This caused Siying's unit test to fail. Test Plan: Unittest Reviewers: dhruba, kailiu, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13539	2013-10-17 22:15:57 -07:00
Siying Dong	65428b0c0a	Fix Bug: iterator.Prev() or iterator.SeekToLast() might return the first element instead of the correct one Summary: Recent patch https://reviews.facebook.net/D11865 introduced a regression bug: DBIter::FindPrevUserEntry(), which is called by DBIter::Prev() (and also implicitly if calling iterator.SeekToLast()) might do issue a seek when having skipped too many entries. If the skipped entry just before the seek() is a delete, the saved key is erased so that it seeks to the front, so Prev() would return the first element. This patch fixes the bug by not doing seek() in DBIter::FindNextUserEntry() if saved key has been erased. Test Plan: Add a test DBTest.IterPrevMaxSkip which would fail without the patch and would pass with the change. Reviewers: dhruba, xjin, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13557	2013-10-17 18:33:18 -07:00
Siying Dong	9edda37027	Universal Compaction to Have a Size Percentage Threshold To Decide Whether to Compress Summary: This patch adds a option for universal compaction to allow us to only compress output files if the files compacted previously did not yet reach a specified ratio, to save CPU costs in some cases. Compression is always skipped for flushing. This is because the size information is not easy to evaluate for flushing case. We can improve it later. Test Plan: add test DBTest.UniversalCompactionCompressRatio1 and DBTest.UniversalCompactionCompressRatio12 Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13467	2013-10-17 13:33:39 -07:00
Kai Liu	aac44226a0	Add bloom filter to predefined table stats Summary: As title. Test Plan: Updated the unit tests to make sure new statistic is correctly written/read. Reviewers: dhruba, haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13497	2013-10-17 11:43:06 -07:00
Vamsi Ponnekanti	6731997f64	[ldb compact is not allowing ttl flag] Summary: Allow ttl flag Test Plan: tested on my database that has merge operations and ttl Revert Plan: OK Task ID: #3038186 Reviewers: emayanke, dhruba, haobo Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13503	2013-10-16 18:06:54 -07:00
Dhruba Borthakur	9cd221094c	Add appropriate LICENSE and Copyright message. Summary: Add appropriate LICENSE and Copyright message. Test Plan: make check Reviewers: CC: Task ID: # Blame Rev:	2013-10-16 17:48:41 -07:00
Igor Canadi	fc4616d898	External Value Store Summary: Developing a capability for storing values on external backing file(s). This is just a highly unoptimized first pass - supports: 1) Allocating some portion of external file to be used to store value 2) Freeing the range, enabling it to be reused by other values As next steps, I plan to: 1) Create some kind of stress testing. Once I can measure stuff, I can focus on optimizing. 2) Optimize locking. 3) Optimize freelist data structure. Currently we have O(n) for both freeing and allocation. 4) Figure out how to do recovery. Test Plan: Created a unit test. Reviewers: dhruba, haobo, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13389	2013-10-16 17:33:49 -07:00
Kai Liu	0f31843c15	Fix the patent format Summary: Formatted the PATENT file so that it's easier to read. Test Plan: Reviewers: CC: Task ID: # Blame Rev:	2013-10-16 15:37:32 -07:00
Siying Dong	073cbfc8f0	Enable background flush thread by default and fix issues related to it Summary: Enable background flush thread in this patch and fix unit tests with: (1) After background flush, schedule a background compaction if condition satisfied; (2) Fix a bug that if universal compaction is enabled and number of levels are set to be 0, compaction will not be automatically triggered (3) Fix unit tests to wait for compaction to finish instead of flush, before checking the compaction results. Test Plan: pass all unit tests Reviewers: haobo, xjin, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13461	2013-10-16 13:32:53 -07:00
Dhruba Borthakur	cb5b2baf18	Added Patent information to the source code repository. Summary: Added Patent information to the source code repository. Test Plan: Reviewers: CC: Task ID: # Blame Rev:	2013-10-15 21:08:47 -07:00
Dhruba Borthakur	b825df81e2	Fix error in previous commit of 'ftruncate' to 'fallocate'. Summary: Fix error in previous commit of 'ftruncate' to 'fallocate'. Test Plan: Reviewers: CC: Task ID: # Blame Rev:	2013-10-15 13:57:29 -07:00
Dhruba Borthakur	8457b74c26	Fix Unit test when run on tmpfs Summary: tmpfs might not support fallocate(). Fix unit test so that this does not cause a unit test to fail. Test Plan: ./env_test Reviewers: emayanke, igor, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13455	2013-10-15 12:03:09 -07:00
Mayank Agarwal	da2fd001a6	Fix rocksdb->levledb BytewiseComparator and inverted order of error in db/version_set.cc Summary: This is needed to make existing dbs be able to open and also because BytewiseComparator was not changed since leveldb. The inverted order in the error message caused confusion prebiously Test Plan: make; open existing db Reviewers: leveldb, dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D13449	2013-10-14 18:16:54 -07:00
Mayank Agarwal	fe3713961e	Features in Transaction log iterator Summary: * Logstore requests a valid change of reutrning an empty iterator and not an error in case of no log files. * Changed the code to return the writebatch containing the sequence number requested from GetupdatesSince even if it lies in the middle. Earlier we used to return the next writebatch,. This also allows me oto guarantee that no files played upon by the iterator are redundant. I mean the starting log file has at least a sequence number >= the sequence number requested form GetupdatesSince. * Cleaned up redundant logic in Iterator::Next and made a new function SeekToStartSequence for greater readability and maintainibilty. * Modified a test in db_test accordingly Please check the logic carefully and suggest improvements. I have a separate patch out for more improvements like restricting reader to read till written sequences. Test Plan: * transaction log iterator tests in db_test, * db_repl_stress. * rocks_log_iterator_test in fbcode/wormhole/rocksdb/test - 2 tests thriving on hacks till now can get simplified * testing on the shadow setup for sigma with replication Reviewers: dhruba, haobo, kailiu, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13437	2013-10-14 18:16:21 -07:00
Kai Liu	86ef6c3f74	Add statistics to sst file Summary: So far we only have key/value pairs as well as bloom filter stored in the sst file. It will be great if we are able to store more metadata about this table itself, for example, the entry size, bloom filter name, etc. This diff is the first step of this effort. It allows table to keep the basic statistics mentioned in http://fburl.com/14995441, as well as allowing writing user-collected stats to stats block. After this diff, we will figure out the interface of how to allow user to collect their interested statistics. Test Plan: 1. Added several unit tests. 2. Ran `make check` to ensure it doesn't break other tests. Reviewers: dhruba, haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13419	2013-10-14 15:56:13 -07:00
Siying Dong	88f2f89068	Change Function names from Compaction->Flush When they really mean Flush Summary: When I debug the unit test failures when enabling background flush thread, I feel the function names can be made clearer for people to understand. Also, if the names are fixed, in many places, some tests' bugs are obvious (and some of those tests are failing). This patch is to clean it up for future maintenance. Test Plan: Run test suites. Reviewers: haobo, dhruba, xjin Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13431	2013-10-14 15:12:15 -07:00
sdong	f8509653ba	LRUCache to try to clean entries not referenced first. Summary: With this patch, when LRUCache.Insert() is called and the cache is full, it will first try to free up entries whose reference counter is 1 (would become 0 after remo\ ving from the cache). We do it in two passes, in the first pass, we only try to release those unreferenced entries. If we cannot free enough space after traversing t\ he first remove_scan_cnt_ entries, we start from the beginning again and remove those entries being used. Test Plan: add two unit tests to cover the codes Reviewers: dhruba, haobo, emayanke Reviewed By: emayanke CC: leveldb, emayanke, xjin Differential Revision: https://reviews.facebook.net/D13377	2013-10-11 09:26:21 -07:00
Dhruba Borthakur	c0ce562c32	Bad nfs file checked in a long time back. Summary: Bad nfs file checked in a long time back. Test Plan: Reviewers: CC: Task ID: # Blame Rev:	2013-10-10 22:00:49 -07:00
Mayank Agarwal	a8b4a69de0	Fixing error in ParseFileName causing DestroyDB to fail on archive directory Summary: This careless error was causing ASSERT_OK(DestroyDB) to fail in db_test. Basically .. was being returned as a child of db/archive and ParseFileName returned false on that, but 'type' was set to LogFile from earlier and not reset. The return of ParseFileName was not being checked to delete the log file or not. Test Plan: make all check Reviewers: dhruba, haobo, xjin, kailiu, nkg- Reviewed By: nkg- CC: leveldb Differential Revision: https://reviews.facebook.net/D13413	2013-10-10 18:18:31 -07:00
Siying Dong	40a1e31fa5	Minor: Fix a lint error in cache_test.cc Summary: As title. Fix an lint error: Lint: CppLint Error Single-argument constructor 'Value(int v)' may inadvertently be used as a type conversion constructor. Prefix the function with the 'explicit' keyword to avoid this, or add an /* implicit */ comment to suppress this warning. Test Plan: N/A Reviewers: emayanke, haobo, dhruba Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13401	2013-10-10 13:47:25 -07:00
Igor Canadi	d2ca2bd183	Fixing build failure Summary: virtual NewRandomRWFile is not implemented on EnvHdfs, causing build failure. Test Plan: make clean; make all check Reviewers: dhruba, haobo, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13383	2013-10-10 01:01:16 -07:00
Igor Canadi	d0beadd456	Env class that can randomly read and write Summary: I have implemented basic simple use case that I need for External Value Store I'm working on. There is a potential for making this prettier by refactoring/combining WritableFile and RandomAccessFile, avoiding some copypasta. However, I decided to implement just the basic functionality, so I can continue working on the other diff. Test Plan: Added a unittest Reviewers: dhruba, haobo, kailiu Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13365	2013-10-10 00:03:08 -07:00
Dhruba Borthakur	7ac3c796f6	Add draft logo. Summary: Add draft logo in jpg format. Test Plan: Reviewers: CC: Task ID: # Blame Rev:	2013-10-09 22:55:30 -07:00
Dhruba Borthakur	6d5f6a4b1a	A bare-bones rocksdb logo. Summary: A hand-crafted rocksdb logo. Test Plan: Reviewers: CC: Task ID: # Blame Rev:	2013-10-09 20:32:19 -07:00
Dhruba Borthakur	3c37955a2f	Remove obsolete namespace mappings. Summary: The previous release 2.4 had a mapping to alias the older namespace to rocksdb. This mapping is not needed in the new release. Test Plan: make check make release Reviewers: emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13359	2013-10-08 22:53:38 -07:00
Naman Gupta	cbf4a06427	Add option for storing transaction logs in a separate dir Summary: In some cases, you might not want to store the data log (write ahead log) files in the same dir as the sst files. An example use case is leaf, which stores sst files in tmpfs. And would like to save the log files in a separate dir (disk) to save memory. Test Plan: make all. Ran db_test test. A few test failing. P2785018. If you guys don't see an obvious problem with the code, maybe somebody from the rocksdb team could help me debug the issue here. Running this on leaf worked well. I could see logs stored on disk, and deleted appropriately after compactions. Obviously this is only one set of options. The unit tests cover different options. Seems like I'm missing some edge cases. Reviewers: dhruba, haobo, leveldb CC: xinyaohu, sumeet Differential Revision: https://reviews.facebook.net/D13239	2013-10-08 17:40:27 -07:00
Naman Gupta	116071411b	Make db_test more robust Summary: While working on D13239, I noticed that the same options are not used for opening and destroying at db. So adding that. Also added asserts for successful DestroyDB calls. Test Plan: Ran unit tests. Atleast 1 unit test is failing. They failures are a result of some past logic change. I'm not really planning to fix those. But I would like to check this in. And hopefully the respective unit test owners can fix the broken tests Reviewers: leveldb, haobo CC: xinyaohu, sumeet, dhruba Differential Revision: https://reviews.facebook.net/D13329	2013-10-08 13:19:31 -07:00
Kai Liu	1f8ade6bd6	Fix a bug in table builder Summary: In talbe.cc, when reading the metablock, it uses BytewiseComparator(); However in table_builder.cc, we use r->options.comparator. After tracing the creation of r->options.comparator, I found this comparator is an InternalKeyComparator, which wraps the user defined comparator(details can be found in DBImpl::SanitizeOptions(). I encountered this problem when adding metadata about "bloom filter" before. With different comparator, we may fail to do the binary sort. Current code works well since there is only one entry in meta block. Test Plan: make all check I've also tested this change in https://reviews.facebook.net/D8283 before. Reviewers: dhruba, haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13335	2013-10-07 18:53:19 -07:00
Igor Canadi	fa46ddb41f	Move delete and free outside of crtical section Summary: Split Unref into two parts -> cheap and expensive. Try to call expensive Unref outside of critical section to decrease lock contention. Test Plan: unittests Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb, kailiu Differential Revision: https://reviews.facebook.net/D13299	2013-10-07 15:37:40 -07:00
Dhruba Borthakur	1a8c1b0817	Unit test failure in DBTest.NumImmutableMemTable. Summary: Previous patch introduced a unit test failure in DBTest.NumImmutableMemTable because of change in property names. Test Plan: Reviewers: CC: Task ID: # Blame Rev:	2013-10-06 01:12:02 -07:00
Dhruba Borthakur	4463b11cad	Migrate names of properties from 'leveldb' prefix to 'rocksdb' prefix. Summary: Migrate names of properties from 'leveldb' prefix to 'rocksdb' prefix. Test Plan: make check Reviewers: emayanke, haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13311	2013-10-06 00:14:26 -07:00
Haobo Xu	bf89edf78b	[RocksDB] Added a property "leveldb.num-immutable-mem-table" so that Flush can be called without blocking, and application still has a way to check when it's done also without blocking. Summary: as title Test Plan: DBTest.NumImmutableMemTable Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13305	2013-10-05 11:54:08 -07:00
Dhruba Borthakur	0a9f873f4b	Removed scribe, thrift and java modules. Summary: Removed scribe, thrift and java modules. Test Plan: make release make check Reviewers: emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13293	2013-10-04 15:36:00 -07:00