Compare commits

..
15 Commits
Author SHA1 Message Date
Yi Wu 808333e6f0 StackableDB optionally take shared ownership of the underlying DB
Summary:
Allow StackableDB optionally takes a shared_ptr on construction and thus hold shared ownership of the underlying DB.
Closes https://github.com/facebook/rocksdb/pull/3423

Differential Revision: D6824163

Pulled By: yiwu-arbug

fbshipit-source-id: dbdc30c42e007533a987ef413785e192340f03eb
2018-02-01 14:03:36 -08:00
Yi Wu f66da2e1ab Blob DB: miscellaneous changes
Summary:
* Expose garbage collection related options
* Minor logging and counter name update
* Remove unused constants.
Closes https://github.com/facebook/rocksdb/pull/3451

Differential Revision: D6867077

Pulled By: yiwu-arbug

fbshipit-source-id: 6c3272a9c9d78b125a0bd6b2e56d00d087cdd6c8
2018-01-31 18:15:10 -08:00
Yi Wu 004a02ee10 Blob DB: dump blob_db_options.min_blob_size
Summary:
min_blob_size was missing from BlobDBOptions::Dump.
Closes https://github.com/facebook/rocksdb/pull/3400

Differential Revision: D6781525

Pulled By: yiwu-arbug

fbshipit-source-id: 40d9b391578d7f8c91bd89f4ce2eda5064864c25
2018-01-31 18:14:53 -08:00
Yi Wu 258549b739 Blob DB: fix crash when DB full but no candidate file to evict
Summary:
When blob_files is empty, std::min_element will return blobfiles.end(), which cannot be dereference. Fixing it.
Closes https://github.com/facebook/rocksdb/pull/3387

Differential Revision: D6764927

Pulled By: yiwu-arbug

fbshipit-source-id: 86f78700132be95760d35ac63480dfd3a8bbe17a
2018-01-19 16:45:40 -08:00
Yi Wu 6d4ef9e839 Blob DB: avoid having a separate read of checksum
Summary:
Previously on a blob db read, we are making a read of the blob value, and then make another read to get CRC checksum. I'm combining the two read into one.

readrandom db_bench with 1G database with base db size of 13M, value size 1k:
`./db_bench --db=/home/yiwu/tmp/db_bench --use_blob_db --value_size=1024 --num=1000000 --benchmarks=readrandom --use_existing_db --cache_size=32000000`
master: throughput 234MB/s, get micros p50 5.984 p95 9.998 p99 20.817 p100 787
this PR: throughput 261MB/s, get micros p50 5.157 p95 9.928 p99 20.724 p100 190
Closes https://github.com/facebook/rocksdb/pull/3301

Differential Revision: D6615950

Pulled By: yiwu-arbug

fbshipit-source-id: 052410c6d8539ec0cc305d53793bbc8f3616baa3
2018-01-09 14:36:15 -08:00
Yi Wu af9746fd22 BlobDB: dump blob db options on open
Summary:
We dump blob db options on blob db open, but it was removed by mistake in #3246. Adding it back.
Closes https://github.com/facebook/rocksdb/pull/3298

Differential Revision: D6607177

Pulled By: yiwu-arbug

fbshipit-source-id: 2a4aacbfa52fd8f1878dc9e1fbb95fe48faf80c0
2018-01-09 14:36:10 -08:00
Yi Wu 5efc4bd8ea BlobDB: update blob_db_options.bytes_per_sync behavior
Summary:
Previously, if blob_db_options.bytes_per_sync, there is a background job to call fsync() for every bytes_per_sync bytes written to a blob file. With the change we simply pass bytes_per_sync as env_options_ to blob files so that sync_file_range() will be used instead.
Closes https://github.com/facebook/rocksdb/pull/3297

Differential Revision: D6606994

Pulled By: yiwu-arbug

fbshipit-source-id: 452424be52e32ba92f5ea603b564e9b88929af47
2018-01-09 14:36:03 -08:00
Yi Wu 1e21dcc833 BlobDB: Remove the need to get sequence number per write
Summary:
Previously we store sequence number range of each blob files, and use the sequence number range to check if the file can be possibly visible by a snapshot. But it adds complexity to the code, since the sequence number is only available after a write. (The current implementation get sequence number by calling GetLatestSequenceNumber(), which is wrong.) With the patch, we are not storing sequence number range, and check if snapshot_sequence < obsolete_sequence to decide if the file is visible by a snapshot (previously we check if first_sequence <= snapshot_sequence < obsolete_sequence).
Closes https://github.com/facebook/rocksdb/pull/3274

Differential Revision: D6571497

Pulled By: yiwu-arbug

fbshipit-source-id: ca06479dc1fcd8782f6525b62b7762cd47d61909
2018-01-09 14:35:52 -08:00
sdong f7285ce5be Fix HISTORY.md 2017-12-27 13:19:53 -08:00
yingsu00 76698fe15e Port 3 way SSE4.2 crc32c implementation from Folly
Summary:
**# Summary**

RocksDB uses SSE crc32 intrinsics to calculate the crc32 values but it does it in single way fashion (not pipelined on single CPU core). Intel's whitepaper () published an algorithm that uses 3-way pipelining for the crc32 intrinsics, then use pclmulqdq intrinsic to combine the values. Because pclmulqdq has overhead on its own, this algorithm will show perf gains on buffers larger than 216 bytes, which makes RocksDB a perfect user, since most of the buffers RocksDB call crc32c on is over 4KB. Initial db_bench show tremendous CPU gain.

This change uses the 3-way SSE algorithm by default. The old SSE algorithm is now behind a compiler tag NO_THREEWAY_CRC32C. If user compiles the code with NO_THREEWAY_CRC32C=1 then the old SSE Crc32c algorithm would be used. If the server does not have SSE4.2 at the run time the slow way (Non SSE) will be used.

**# Performance Test Results**
We ran the FillRandom and ReadRandom benchmarks in db_bench. ReadRandom is the point of interest here since it calculates the CRC32 for the in-mem buffers. We did 3 runs for each algorithm.

Before this change the CRC32 value computation takes about 11.5% of total CPU cost, and with the new 3-way algorithm it reduced to around 4.5%. The overall throughput also improved from 25.53MB/s to 27.63MB/s.

1) ReadRandom in db_bench overall metrics

    PER RUN
    Algorithm | run | micros/op | ops/sec |Throughput (MB/s)
    3-way      |  1   | 4.143   | 241387 | 26.7
    3-way      |  2   | 3.775   | 264872 | 29.3
    3-way      | 3    | 4.116   | 242929 | 26.9
    FastCrc32c|1  | 4.037   | 247727 | 27.4
    FastCrc32c|2  | 4.648   | 215166 | 23.8
    FastCrc32c|3  | 4.352   | 229799 | 25.4

     AVG
    Algorithm     |    Average of micros/op |   Average of ops/sec |    Average of Throughput (MB/s)
    3-way           |     4.01                               |      249,729                 |      27.63
    FastCrc32c  |     4.35                              |     230,897                  |      25.53

 2)   Crc32c computation CPU cost (inclusive samples percentage)
    PER RUN
    Implementation | run |  TotalSamples   | Crc32c percentage
    3-way                 |  1    |  4,572,250,000 | 4.37%
    3-way                 |  2    |  3,779,250,000 | 4.62%
    3-way                 |  3    |  4,129,500,000 | 4.48%
    FastCrc32c       |  1    |  4,663,500,000 | 11.24%
    FastCrc32c       |  2    |  4,047,500,000 | 12.34%
    FastCrc32c       |  3    |  4,366,750,000 | 11.68%

 **# Test Plan**
     make -j64 corruption_test && ./corruption_test
      By default it uses 3-way SSE algorithm

     NO_THREEWAY_CRC32C=1 make -j64 corruption_test && ./corruption_test

    make clean && DEBUG_LEVEL=0 make -j64 db_bench
    make clean && DEBUG_LEVEL=0 NO_THREEWAY_CRC32C=1 make -j64 db_bench
Closes https://github.com/facebook/rocksdb/pull/3173

Differential Revision: D6330882

Pulled By: yingsu00

fbshipit-source-id: 8ec3d89719533b63b536a736663ca6f0dd4482e9
2017-12-27 13:16:32 -08:00
Zhongyi Xie 9196e80b15 Reduce heavy hitter for Get operation
Summary:
This PR addresses the following heavy hitters in `Get` operation by moving calls to `StatisticsImpl::recordTick` from `BlockBasedTable` to `Version::Get`

- rocksdb.block.cache.bytes.write
- rocksdb.block.cache.add
- rocksdb.block.cache.data.miss
- rocksdb.block.cache.data.bytes.insert
- rocksdb.block.cache.data.add
- rocksdb.block.cache.hit
- rocksdb.block.cache.data.hit
- rocksdb.block.cache.bytes.read

The db_bench statistics before and after the change are:

|1GB block read|Children      |Self  |Command          |Shared Object        |Symbol|
|---|---|---|---|---|---|
|master:     |4.22%     |1.31%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|updated:    |0.51%     |0.21%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|     	     |0.14%     |0.14%  |db_bench  |db_bench  |[.] rocksdb::GetContext::record_counters|

|1MB block read|Children      |Self  |Command          |Shared Object        |Symbol|
|---|---|---|---|---|---|
|master:    |3.48%     |1.08%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|updated:    |0.80%     |0.31%  |db_bench  |db_bench  |[.] rocksdb::StatisticsImpl::recordTick|
|    	     |0.35%     |0.35%  |db_bench  |db_bench  |[.] rocksdb::GetContext::record_counters|
Closes https://github.com/facebook/rocksdb/pull/3172

Differential Revision: D6330532

Pulled By: miasantreble

fbshipit-source-id: 2b492959e00a3db29e9437ecdcc5e48ca4ec5741
2017-12-14 16:52:06 -08:00
Siying Dong 243ca14116 Print out compression type of new SST files in logging
Summary: Closes https://github.com/facebook/rocksdb/pull/3264

Differential Revision: D6552768

Pulled By: siying

fbshipit-source-id: 6303110aff22f341d5cff41f8d2d4f138a53652d
2017-12-14 16:51:26 -08:00
Siying Dong 1c3c1c4340 NUMBER_BLOCK_COMPRESSED, etc, shouldn't be treated as timer counter
Summary:
NUMBER_BLOCK_DECOMPRESSED and NUMBER_BLOCK_COMPRESSED are not reported unless the stats level contain detailed timers, which is wrong. They are normal counters. Fix it.
Closes https://github.com/facebook/rocksdb/pull/3263

Differential Revision: D6552519

Pulled By: siying

fbshipit-source-id: 40899ccea7b2856bb39752616657c0bfd432f6f9
2017-12-14 16:51:13 -08:00
Orvid King 3e40def2c1 Fix the build with MSVC 2017
Summary:
There were a few places where MSVC's implicit truncation warnings were getting triggered, which was causing the MSVC build to fail due to warnings being treated as errors. This resolves the issues by making the truncations in some places explicit, and by making it so there are no truncations of literals.

Fixes #3239
Supersedes #3259
Closes https://github.com/facebook/rocksdb/pull/3273

Reviewed By: yiwu-arbug

Differential Revision: D6569204

Pulled By: Orvid

fbshipit-source-id: c188cf1cf98d9acb6d94b71875041cc81f8ff088
2017-12-14 16:49:25 -08:00
Siying Dong 766f062bbb Switch version to 5.10
Summary: Closes https://github.com/facebook/rocksdb/pull/3252

Differential Revision: D6539373

Pulled By: siying

fbshipit-source-id: ce7c3d3fe625852179055295da9cf7bc80755025
2017-12-11 15:44:51 -08:00
193 changed files with 1512 additions and 15062 deletions

No files matched your search

+11 -21
View File
@@ -14,7 +14,7 @@
# cd build
# 3. Run cmake to generate project files for Windows, add more options to enable required third-party libraries.
# See thirdparty.inc for more information.
# sample command: cmake -G "Visual Studio 15 Win64" -DWITH_GFLAGS=1 -DWITH_SNAPPY=1 -DWITH_JEMALLOC=1 -DWITH_JNI=1 ..
# sample command: cmake -G "Visual Studio 14 Win64" -DGFLAGS=1 -DSNAPPY=1 -DJEMALLOC=1 -DJNI=1 ..
# 4. Then build the project in debug mode (you may want to add /m[:<N>] flag to run msbuild in <N> parallel threads
# or simply /m ot use all avail cores)
# msbuild rocksdb.sln
@@ -42,14 +42,7 @@ endif()
list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/modules/")
option(WITH_JEMALLOC "build with JeMalloc" OFF)
option(WITH_SNAPPY "build with SNAPPY" OFF)
option(WITH_LZ4 "build with lz4" OFF)
option(WITH_ZLIB "build with zlib" OFF)
if(MSVC)
# Defaults currently different for GFLAGS.
# We will address find_package work a little later
option(WITH_GFLAGS "build with GFlags" OFF)
option(WITH_XPRESS "build with windows built in compression" OFF)
include(${CMAKE_CURRENT_SOURCE_DIR}/thirdparty.inc)
else()
if(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
@@ -63,18 +56,8 @@ else()
include_directories(${JEMALLOC_INCLUDE_DIR})
endif()
endif()
# No config file for this
option(WITH_GFLAGS "build with GFlags" ON)
if(WITH_GFLAGS)
find_package(gflags)
if(gflags_FOUND)
add_definitions(-DGFLAGS=1)
include_directories(${gflags_INCLUDE_DIR})
list(APPEND THIRDPARTY_LIBS ${gflags_LIBRARIES})
endif()
endif()
option(WITH_SNAPPY "build with SNAPPY" OFF)
if(WITH_SNAPPY)
find_package(snappy REQUIRED)
add_definitions(-DSNAPPY)
@@ -82,6 +65,7 @@ else()
list(APPEND THIRDPARTY_LIBS ${SNAPPY_LIBRARIES})
endif()
option(WITH_ZLIB "build with zlib" OFF)
if(WITH_ZLIB)
find_package(zlib REQUIRED)
add_definitions(-DZLIB)
@@ -97,6 +81,7 @@ else()
list(APPEND THIRDPARTY_LIBS ${BZIP2_LIBRARIES})
endif()
option(WITH_LZ4 "build with lz4" OFF)
if(WITH_LZ4)
find_package(lz4 REQUIRED)
add_definitions(-DLZ4)
@@ -269,6 +254,13 @@ if(WITH_UBSAN)
endif()
endif()
find_package(gflags)
if(gflags_FOUND)
add_definitions(-DGFLAGS=1)
include_directories(${gflags_INCLUDE_DIR})
list(APPEND THIRDPARTY_LIBS ${gflags_LIBRARIES})
endif()
find_package(NUMA)
if(NUMA_FOUND)
add_definitions(-DNUMA)
@@ -498,7 +490,6 @@ set(SOURCES
table/block_based_table_factory.cc
table/block_based_table_reader.cc
table/block_builder.cc
table/block_fetcher.cc
table/block_prefix_index.cc
table/bloom_block.cc
table/cuckoo_table_builder.cc
@@ -586,7 +577,6 @@ set(SOURCES
utilities/leveldb_options/leveldb_options.cc
utilities/lua/rocks_lua_compaction_filter.cc
utilities/memory/memory_util.cc
utilities/merge_operators/bytesxor.cc
utilities/merge_operators/max.cc
utilities/merge_operators/put.cc
utilities/merge_operators/string_append/stringappend.cc
+2 -31
View File
@@ -1,43 +1,15 @@
# Rocksdb Change Log
## 5.11.4 (03/16/2018)
### Bug Fixes
* Fix WAL corruption caused by race condition between user write thread and backup/checkpoint thread.
## 5.11.2 (02/24/2018)
### Bug Fixes
* Fix bug in iterator readahead causing blocks to incorrectly be considered truncated (corrupted).
## 5.11.1 (02/22/2018)
### New Features
* Follow rsync-style naming convention for BackupEngine tempfiles. This enables some optimizations when run on GlusterFS.
## 5.11.0 (01/08/2018)
### Public API Change
* Add `autoTune` and `getBytesPerSecond()` to RocksJava RateLimiter
### New Features
* Add a new histogram stat called rocksdb.db.flush.micros for memtable flush.
* Add "--use_txn" option to use transactional API in db_stress.
* Disable onboard cache for compaction output in Windows platform.
* Improve the performance of iterators doing long range scans by using readahead.
### Bug Fixes
* Fix a stack-use-after-scope bug in ForwardIterator.
* Fix builds on platforms including Linux, Windows, and PowerPC.
* Fix buffer overrun in backup engine for DBs with huge number of files.
* Fix a mislabel bug for bottom-pri compaction threads.
* Fix DB::Flush() keep waiting after flush finish under certain condition.
## Unreleased
## 5.10.0 (12/11/2017)
### Public API Change
* When running `make` with environment variable `USE_SSE` set and `PORTABLE` unset, will use all machine features available locally. Previously this combination only compiled SSE-related features.
### New Features
* CRC32C is now using the 3-way pipelined SSE algorithm `crc32c_3way` on supported platforms to improve performance. The system will choose to use this algorithm on supported platforms automatically whenever possible. If PCLMULQDQ is not supported it will fall back to the old Fast_CRC32 algorithm.
* Provide lifetime hints when writing files on Linux. This reduces hardware write-amp on storage devices supporting multiple streams.
* Add a DB stat, `NUMBER_ITER_SKIP`, which returns how many internal keys were skipped during iterations (e.g., due to being tombstones or duplicate versions of a key).
* Add PerfContext counters, `key_lock_wait_count` and `key_lock_wait_time`, which measure the number of times transactions wait on key locks and total amount of time waiting.
* Support dynamically changing `ColumnFamilyOptions::compaction_options_universal`.
* Batch update stats at the end of each `Get`, rather than for each block cache access.
### Bug Fixes
* Fix IOError on WAL write doesn't propagate to write group follower
@@ -56,7 +28,6 @@
* Return an error on write if write_options.sync = true and write_options.disableWAL = true to warn user of inconsistent options. Previously we will not write to WAL and not respecting the sync options in this case.
### New Features
* CRC32C is now using the 3-way pipelined SSE algorithm `crc32c_3way` on supported platforms to improve performance. The system will choose to use this algorithm on supported platforms automatically whenever possible. If PCLMULQDQ is not supported it will fall back to the old Fast_CRC32 algorithm.
* `DBOptions::writable_file_max_buffer_size` can now be changed dynamically.
* `DBOptions::bytes_per_sync`, `DBOptions::compaction_readahead_size`, and `DBOptions::wal_bytes_per_sync` can now be changed dynamically, `DBOptions::wal_bytes_per_sync` will flush all memtables and switch to a new WAL file.
* Support dynamic adjustment of rate limit according to demand for background I/O. It can be enabled by passing `true` to the `auto_tuned` parameter in `NewGenericRateLimiter()`. The value passed as `rate_bytes_per_sec` will still be respected as an upper-bound.
-35
View File
@@ -107,41 +107,6 @@ to build a portable binary, add `PORTABLE=1` before your make commands, like thi
* run `brew tap homebrew/versions; brew install gcc48 --use-llvm` to install gcc 4.8 (or higher).
* run `brew install rocksdb`
* **FreeBSD** (11.01):
* You can either install RocksDB from the Ports system using `cd /usr/ports/databases/rocksdb && make install`, or you can follow the details below to install dependencies and compile from source code:
* Install the dependencies for RocksDB:
export BATCH=YES
cd /usr/ports/devel/gmake && make install
cd /usr/ports/devel/gflags && make install
cd /usr/ports/archivers/snappy && make install
cd /usr/ports/archivers/bzip2 && make install
cd /usr/ports/archivers/liblz4 && make install
cd /usr/ports/archivesrs/zstd && make install
cd /usr/ports/devel/git && make install
* Install the dependencies for RocksJava (optional):
export BATCH=yes
cd /usr/ports/java/openjdk7 && make install
* Build RocksDB from source:
cd ~
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
gmake static_lib
* Build RocksJava from source (optional):
cd rocksdb
export JAVA_HOME=/usr/local/openjdk7
gmake rocksdbjava
* **iOS**:
* Run: `TARGET_OS=IOS make static_lib`. When building the project which uses rocksdb iOS library, make sure to define two important pre-processing macros: `ROCKSDB_LITE` and `IOS_CROSS_COMPILE`.
+16 -35
View File
@@ -344,8 +344,6 @@ ifeq ($(HAVE_POWER8),1)
LIB_CC_OBJECTS = $(LIB_SOURCES:.cc=.o)
LIBOBJECTS += $(LIB_SOURCES_C:.c=.o)
LIBOBJECTS += $(LIB_SOURCES_ASM:.S=.o)
else
LIB_CC_OBJECTS = $(LIB_SOURCES:.cc=.o)
endif
LIBOBJECTS += $(TOOL_LIB_SOURCES:.cc=.o)
@@ -509,7 +507,6 @@ PARALLEL_TEST = \
backupable_db_test \
db_compaction_filter_test \
db_compaction_test \
db_merge_operator_test \
db_sst_test \
db_test \
db_universal_compaction_test \
@@ -521,7 +518,7 @@ PARALLEL_TEST = \
persistent_cache_test \
table_test \
transaction_test \
write_prepared_transaction_test \
write_prepared_transaction_test
SUBSET := $(TESTS)
ifdef ROCKSDBTESTS_START
@@ -1556,11 +1553,11 @@ BZIP2_DOWNLOAD_BASE ?= http://www.bzip.org
SNAPPY_VER ?= 1.1.4
SNAPPY_SHA256 ?= 134bfe122fd25599bb807bb8130e7ba6d9bdb851e0b16efcb83ac4f5d0b70057
SNAPPY_DOWNLOAD_BASE ?= https://github.com/google/snappy/releases/download
LZ4_VER ?= 1.8.0
LZ4_SHA256 ?= 2ca482ea7a9bb103603108b5a7510b7592b90158c151ff50a28f1ca8389fccf6
LZ4_VER ?= 1.7.5
LZ4_SHA256 ?= 0190cacd63022ccb86f44fa5041dc6c3804407ad61550ca21c382827319e7e7e
LZ4_DOWNLOAD_BASE ?= https://github.com/lz4/lz4/archive
ZSTD_VER ?= 1.3.3
ZSTD_SHA256 ?= a77c47153ee7de02626c5b2a097005786b71688be61e9fb81806a011f90b297b
ZSTD_VER ?= 1.2.0
ZSTD_SHA256 ?= 4a7e4593a3638276ca7f2a09dc4f38e674d8317bbea51626393ca73fc047cbfb
ZSTD_DOWNLOAD_BASE ?= https://github.com/facebook/zstd/archive
ifeq ($(PLATFORM), OS_MACOSX)
@@ -1574,7 +1571,7 @@ else
endif
endif
ifeq ($(PLATFORM), OS_FREEBSD)
JAVA_INCLUDE = -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/freebsd
JAVA_INCLUDE += -I$(JAVA_HOME)/include/freebsd
ROCKSDBJNILIB = librocksdbjni-freebsd$(ARCH).so
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-freebsd$(ARCH).jar
endif
@@ -1600,7 +1597,7 @@ libz.a:
exit 1; \
fi
tar xvzf zlib-$(ZLIB_VER).tar.gz
cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC ${EXTRA_CFLAGS}' LDFLAGS='${EXTRA_LDFLAGS}' ./configure --static && $(MAKE)
cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC ${EXTRA_CFLAGS}' LDFLAGS='${EXTRA_LDFLAGS}' ./configure --static && make
cp zlib-$(ZLIB_VER)/libz.a .
libbz2.a:
@@ -1612,7 +1609,7 @@ libbz2.a:
exit 1; \
fi
tar xvzf bzip2-$(BZIP2_VER).tar.gz
cd bzip2-$(BZIP2_VER) && $(MAKE) CFLAGS='-fPIC -O2 -g -D_FILE_OFFSET_BITS=64 ${EXTRA_CFLAGS}' AR='ar ${EXTRA_ARFLAGS}'
cd bzip2-$(BZIP2_VER) && make CFLAGS='-fPIC -O2 -g -D_FILE_OFFSET_BITS=64 ${EXTRA_CFLAGS}' AR='ar ${EXTRA_ARFLAGS}'
cp bzip2-$(BZIP2_VER)/libbz2.a .
libsnappy.a:
@@ -1625,7 +1622,7 @@ libsnappy.a:
fi
tar xvzf snappy-$(SNAPPY_VER).tar.gz
cd snappy-$(SNAPPY_VER) && CFLAGS='${EXTRA_CFLAGS}' CXXFLAGS='${EXTRA_CXXFLAGS}' LDFLAGS='${EXTRA_LDFLAGS}' ./configure --with-pic --enable-static --disable-shared
cd snappy-$(SNAPPY_VER) && $(MAKE) ${SNAPPY_MAKE_TARGET}
cd snappy-$(SNAPPY_VER) && make ${SNAPPY_MAKE_TARGET}
cp snappy-$(SNAPPY_VER)/.libs/libsnappy.a .
liblz4.a:
@@ -1638,7 +1635,7 @@ liblz4.a:
exit 1; \
fi
tar xvzf lz4-$(LZ4_VER).tar.gz
cd lz4-$(LZ4_VER)/lib && $(MAKE) CFLAGS='-fPIC -O2 ${EXTRA_CFLAGS}' all
cd lz4-$(LZ4_VER)/lib && make CFLAGS='-fPIC -O2 ${EXTRA_CFLAGS}' all
cp lz4-$(LZ4_VER)/lib/liblz4.a .
libzstd.a:
@@ -1651,45 +1648,29 @@ libzstd.a:
exit 1; \
fi
tar xvzf zstd-$(ZSTD_VER).tar.gz
cd zstd-$(ZSTD_VER)/lib && DESTDIR=. PREFIX= $(MAKE) CFLAGS='-fPIC -O2 ${EXTRA_CFLAGS}' install
cd zstd-$(ZSTD_VER)/lib && make CFLAGS='-fPIC -O2 ${EXTRA_CFLAGS}' all
cp zstd-$(ZSTD_VER)/lib/libzstd.a .
# A version of each $(LIBOBJECTS) compiled with -fPIC and a fixed set of static compression libraries
java_static_libobjects = $(patsubst %,jls/%,$(LIB_CC_OBJECTS))
java_static_libobjects = $(patsubst %,jls/%,$(LIBOBJECTS))
CLEAN_FILES += jls
java_static_all_libobjects = $(java_static_libobjects)
ifneq ($(ROCKSDB_JAVA_NO_COMPRESSION), 1)
JAVA_COMPRESSIONS = libz.a libbz2.a libsnappy.a liblz4.a libzstd.a
endif
JAVA_STATIC_FLAGS = -DZLIB -DBZIP2 -DSNAPPY -DLZ4 -DZSTD
JAVA_STATIC_INCLUDES = -I./zlib-$(ZLIB_VER) -I./bzip2-$(BZIP2_VER) -I./snappy-$(SNAPPY_VER) -I./lz4-$(LZ4_VER)/lib -I./zstd-$(ZSTD_VER)/lib/include
ifeq ($(HAVE_POWER8),1)
JAVA_STATIC_C_LIBOBJECTS = $(patsubst %.c.o,jls/%.c.o,$(LIB_SOURCES_C:.c=.o))
JAVA_STATIC_ASM_LIBOBJECTS = $(patsubst %.S.o,jls/%.S.o,$(LIB_SOURCES_ASM:.S=.o))
java_static_ppc_libobjects = $(JAVA_STATIC_C_LIBOBJECTS) $(JAVA_STATIC_ASM_LIBOBJECTS)
jls/util/crc32c_ppc.o: util/crc32c_ppc.c
$(AM_V_CC)$(CC) $(CFLAGS) $(JAVA_STATIC_FLAGS) $(JAVA_STATIC_INCLUDES) -c $< -o $@
jls/util/crc32c_ppc_asm.o: util/crc32c_ppc_asm.S
$(AM_V_CC)$(CC) $(CFLAGS) $(JAVA_STATIC_FLAGS) $(JAVA_STATIC_INCLUDES) -c $< -o $@
java_static_all_libobjects += $(java_static_ppc_libobjects)
endif
JAVA_STATIC_INCLUDES = -I./zlib-$(ZLIB_VER) -I./bzip2-$(BZIP2_VER) -I./snappy-$(SNAPPY_VER) -I./lz4-$(LZ4_VER)/lib -I./zstd-$(ZSTD_VER)/lib
$(java_static_libobjects): jls/%.o: %.cc $(JAVA_COMPRESSIONS)
$(AM_V_CC)mkdir -p $(@D) && $(CXX) $(CXXFLAGS) $(JAVA_STATIC_FLAGS) $(JAVA_STATIC_INCLUDES) -fPIC -c $< -o $@ $(COVERAGEFLAGS)
rocksdbjavastatic: $(java_static_all_libobjects)
rocksdbjavastatic: $(java_static_libobjects)
cd java;$(MAKE) javalib;
rm -f ./java/target/$(ROCKSDBJNILIB)
$(CXX) $(CXXFLAGS) -I./java/. $(JAVA_INCLUDE) -shared -fPIC \
-o ./java/target/$(ROCKSDBJNILIB) $(JNI_NATIVE_SOURCES) \
$(java_static_all_libobjects) $(COVERAGEFLAGS) \
$(java_static_libobjects) $(COVERAGEFLAGS) \
$(JAVA_COMPRESSIONS) $(JAVA_STATIC_LDFLAGS)
cd java/target;strip $(STRIPFLAGS) $(ROCKSDBJNILIB)
cd java;jar -cf target/$(ROCKSDB_JAR) HISTORY*.md
@@ -1750,7 +1731,7 @@ JAVA_C_LIBOBJECTS = $(patsubst %.c.o,jl/%.c.o,$(JAVA_C_OBJECTS))
JAVA_ASM_LIBOBJECTS = $(patsubst %.S.o,jl/%.S.o,$(JAVA_ASM_OBJECTS))
endif
java_libobjects = $(patsubst %,jl/%,$(LIB_CC_OBJECTS))
java_libobjects = $(patsubst %,jl/%,$(LIBOBJECTS))
CLEAN_FILES += jl
java_all_libobjects = $(java_libobjects)
+2 -4
View File
@@ -157,7 +157,6 @@ cpp_library(
"table/block_based_table_factory.cc",
"table/block_based_table_reader.cc",
"table/block_builder.cc",
"table/block_fetcher.cc",
"table/block_prefix_index.cc",
"table/bloom_block.cc",
"table/cuckoo_table_builder.cc",
@@ -242,7 +241,6 @@ cpp_library(
"utilities/leveldb_options/leveldb_options.cc",
"utilities/lua/rocks_lua_compaction_filter.cc",
"utilities/memory/memory_util.cc",
"utilities/merge_operators/bytesxor.cc",
"utilities/merge_operators/max.cc",
"utilities/merge_operators/put.cc",
"utilities/merge_operators/string_append/stringappend.cc",
@@ -568,7 +566,7 @@ ROCKS_TESTS = [
[
"db_merge_operator_test",
"db/db_merge_operator_test.cc",
"parallel",
"serial",
],
[
"db_options_test",
@@ -1018,7 +1016,7 @@ ROCKS_TESTS = [
[
"write_prepared_transaction_test",
"utilities/transactions/write_prepared_transaction_test.cc",
"parallel",
"serial",
],
]
+1 -1
View File
@@ -3,7 +3,7 @@ image: Visual Studio 2015
before_build:
- md %APPVEYOR_BUILD_FOLDER%\build
- cd %APPVEYOR_BUILD_FOLDER%\build
- cmake -G "Visual Studio 14 Win64" -DOPTDBG=1 -DWITH_XPRESS=1 -DPORTABLE=1 ..
- cmake -G "Visual Studio 14 2015 Win64" -DOPTDBG=1 -DXPRESS=1 -DPORTABLE=1 ..
- cd ..
build:
project: build\rocksdb.sln
+1 -1
View File
@@ -36,7 +36,7 @@ def parse_src_mk(repo_path):
# get all .cc / .c files
def get_cc_files(repo_path):
cc_files = []
for root, dirnames, filenames in os.walk(repo_path): # noqa: B007 T25377293 Grandfathered in
for root, dirnames, filenames in os.walk(repo_path):
root = root[(len(repo_path) + 1):]
if "java" in root:
# Skip java
-1
View File
@@ -1,5 +1,4 @@
#!/usr/bin/env bash
# Create a tmp directory for the test to use
TEST_DIR=$(mktemp -d /dev/shm/fbcode_rocksdb_XXXXXXX)
# shellcheck disable=SC2068
TEST_TMPDIR="$TEST_DIR" $@ && rm -rf "$TEST_DIR"
+3 -9
View File
@@ -51,13 +51,11 @@ if [ -z "$ROCKSDB_NO_FBCODE" -a -d /mnt/gvfs/third-party ]; then
FBCODE_BUILD="true"
# If we're compiling with TSAN we need pic build
PIC_BUILD=$COMPILE_WITH_TSAN
if [ -n "$ROCKSDB_FBCODE_BUILD_WITH_481" ]; then
# we need this to build with MySQL. Don't use for other purposes.
source "$PWD/build_tools/fbcode_config4.8.1.sh"
elif [ -n "$ROCKSDB_FBCODE_BUILD_WITH_5xx" ]; then
if [ -z "$ROCKSDB_FBCODE_BUILD_WITH_481" ]; then
source "$PWD/build_tools/fbcode_config.sh"
else
source "$PWD/build_tools/fbcode_config_platform007.sh"
# we need this to build with MySQL. Don't use for other purposes.
source "$PWD/build_tools/fbcode_config4.8.1.sh"
fi
fi
@@ -143,7 +141,6 @@ case "$TARGET_OS" in
;;
FreeBSD)
PLATFORM=OS_FREEBSD
CXX=clang++
COMMON_FLAGS="$COMMON_FLAGS -fno-builtin-memcmp -D_REENTRANT -DOS_FREEBSD"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpthread"
# PORT_FILES=port/freebsd/freebsd_specific.cc
@@ -484,9 +481,6 @@ if test -z "$PORTABLE"; then
COMMON_FLAGS="$COMMON_FLAGS -mcpu=$POWER -mtune=$POWER "
elif test -n "`echo $TARGET_ARCHITECTURE | grep ^s390x`"; then
COMMON_FLAGS="$COMMON_FLAGS -march=z10 "
elif test -n "`echo $TARGET_ARCHITECTURE | grep ^arm`"; then
# TODO: Handle this with approprite options.
COMMON_FLAGS="$COMMON_FLAGS"
elif [ "$TARGET_OS" != AIX ] && [ "$TARGET_OS" != SunOS ]; then
COMMON_FLAGS="$COMMON_FLAGS -march=native "
elif test "$USE_SSE"; then
-2
View File
@@ -13,12 +13,10 @@ error=0
function log {
DATE=`date +%Y-%m-%d:%H:%M:%S`
# shellcheck disable=SC2068
echo $DATE $@
}
function log_err {
# shellcheck disable=SC2145
log "ERROR: $@ Error code: $error."
}
-1
View File
@@ -1,4 +1,3 @@
# shellcheck disable=SC2148
GCC_BASE=/mnt/gvfs/third-party2/gcc/8219ec1bcedf8ad9da05e121e193364de2cc4f61/5.x/centos6-native/c447969
CLANG_BASE=/mnt/gvfs/third-party2/llvm-fb/64d8d58e3d84f8bde7a029763d4f5baf39d0d5b9/stable/centos6-native/6aaf4de
LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/ba9be983c81de7299b59fe71950c664a84dcb5f8/5.x/gcc-5-glibc-2.23/339d858
-1
View File
@@ -1,4 +1,3 @@
# shellcheck disable=SC2148
GCC_BASE=/mnt/gvfs/third-party2/gcc/cf7d14c625ce30bae1a4661c2319c5a283e4dd22/4.8.1/centos6-native/cc6c9dc
CLANG_BASE=/mnt/gvfs/third-party2/llvm-fb/8598c375b0e94e1448182eb3df034704144a838d/stable/centos6-native/3f16ddd
LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/d6e0a7da6faba45f5e5b1638f9edd7afc2f34e7d/4.8.1/gcc-4.8.1-glibc-2.17/8aac7fc
-18
View File
@@ -1,18 +0,0 @@
GCC_BASE=/mnt/gvfs/third-party2/gcc/6e8e715624fd15256a7970073387793dfcf79b46/7.x/centos7-native/b2ef2b6
CLANG_BASE=/mnt/gvfs/third-party2/llvm-fb/ef37e1faa1c29782abfac1ae65a291b9b7966f6d/stable/centos7-native/c9f9104
LIBGCC_BASE=/mnt/gvfs/third-party2/libgcc/c67031f0f739ac61575a061518d6ef5038f99f90/7.x/platform007/5620abc
GLIBC_BASE=/mnt/gvfs/third-party2/glibc/60d6f124a78798b73944f5ba87c2306ae3460153/2.26/platform007/f259413
SNAPPY_BASE=/mnt/gvfs/third-party2/snappy/7f9bdaada18f59bc27ec2b0871eb8a6144343aef/1.1.3/platform007/ca4da3d
ZLIB_BASE=/mnt/gvfs/third-party2/zlib/22c2d65676fb7c23cfa797c4f6937f38b026f3cf/1.2.8/platform007/ca4da3d
BZIP2_BASE=/mnt/gvfs/third-party2/bzip2/dc49a21c5fceec6456a7a28a94dcd16690af1337/1.0.6/platform007/ca4da3d
LZ4_BASE=/mnt/gvfs/third-party2/lz4/907b498203d297947f3bb70b9466f47e100f1873/r131/platform007/ca4da3d
ZSTD_BASE=/mnt/gvfs/third-party2/zstd/3ee276cbacfad3074e3f07bf826ac47f06970f4e/1.3.5/platform007/15a3614
GFLAGS_BASE=/mnt/gvfs/third-party2/gflags/0b9929d2588991c65a57168bf88aff2db87c5d48/2.2.0/platform007/ca4da3d
JEMALLOC_BASE=/mnt/gvfs/third-party2/jemalloc/9c910d36d6235cc40e8ff559358f1833452300ca/master/platform007/5b0f53e
NUMA_BASE=/mnt/gvfs/third-party2/numa/9cbf2460284c669ed19c3ccb200a71f7dd7e53c7/2.0.11/platform007/ca4da3d
LIBUNWIND_BASE=/mnt/gvfs/third-party2/libunwind/bf3d7497fe4e6d007354f0adffa16ce3003f8338/1.3/platform007/6f3e0a9
TBB_BASE=/mnt/gvfs/third-party2/tbb/ff4e0b093534704d8abab678a4fd7f5ea7b094c7/2018_U5/platform007/ca4da3d
KERNEL_HEADERS_BASE=/mnt/gvfs/third-party2/kernel-headers/b5c4a61a5c483ba24722005ae07895971a2ac707/fb/platform007/da39a3e
BINUTILS_BASE=/mnt/gvfs/third-party2/binutils/92ff90349e2f43ea0a8246d8b1cf17b6869013e3/2.29.1/centos7-native/da39a3e
VALGRIND_BASE=/mnt/gvfs/third-party2/valgrind/f3f697a28122e6bcd513273dd9c1ff23852fc59f/3.13.0/platform007/ca4da3d
LUA_BASE=/mnt/gvfs/third-party2/lua/f0cd714433206d5139df61659eb7b28b1dea6683/5.3.4/platform007/5007832
-157
View File
@@ -1,157 +0,0 @@
#!/bin/sh
#
# Set environment variables so that we can compile rocksdb using
# fbcode settings. It uses the latest g++ and clang compilers and also
# uses jemalloc
# Environment variables that change the behavior of this script:
# PIC_BUILD -- if true, it will only take pic versions of libraries from fbcode. libraries that don't have pic variant will not be included
BASEDIR=`dirname $BASH_SOURCE`
source "$BASEDIR/dependencies_platform007.sh"
CFLAGS=""
# libgcc
LIBGCC_INCLUDE="$LIBGCC_BASE/include/c++/7.3.0"
LIBGCC_LIBS=" -L $LIBGCC_BASE/lib"
# glibc
GLIBC_INCLUDE="$GLIBC_BASE/include"
GLIBC_LIBS=" -L $GLIBC_BASE/lib"
# snappy
SNAPPY_INCLUDE=" -I $SNAPPY_BASE/include/"
if test -z $PIC_BUILD; then
SNAPPY_LIBS=" $SNAPPY_BASE/lib/libsnappy.a"
else
SNAPPY_LIBS=" $SNAPPY_BASE/lib/libsnappy_pic.a"
fi
CFLAGS+=" -DSNAPPY"
if test -z $PIC_BUILD; then
# location of zlib headers and libraries
ZLIB_INCLUDE=" -I $ZLIB_BASE/include/"
ZLIB_LIBS=" $ZLIB_BASE/lib/libz.a"
CFLAGS+=" -DZLIB"
# location of bzip headers and libraries
BZIP_INCLUDE=" -I $BZIP2_BASE/include/"
BZIP_LIBS=" $BZIP2_BASE/lib/libbz2.a"
CFLAGS+=" -DBZIP2"
LZ4_INCLUDE=" -I $LZ4_BASE/include/"
LZ4_LIBS=" $LZ4_BASE/lib/liblz4.a"
CFLAGS+=" -DLZ4"
fi
ZSTD_INCLUDE=" -I $ZSTD_BASE/include/"
if test -z $PIC_BUILD; then
ZSTD_LIBS=" $ZSTD_BASE/lib/libzstd.a"
else
ZSTD_LIBS=" $ZSTD_BASE/lib/libzstd_pic.a"
fi
CFLAGS+=" -DZSTD"
# location of gflags headers and libraries
GFLAGS_INCLUDE=" -I $GFLAGS_BASE/include/"
if test -z $PIC_BUILD; then
GFLAGS_LIBS=" $GFLAGS_BASE/lib/libgflags.a"
else
GFLAGS_LIBS=" $GFLAGS_BASE/lib/libgflags_pic.a"
fi
CFLAGS+=" -DGFLAGS=gflags"
# location of jemalloc
JEMALLOC_INCLUDE=" -I $JEMALLOC_BASE/include/"
JEMALLOC_LIB=" $JEMALLOC_BASE/lib/libjemalloc.a"
if test -z $PIC_BUILD; then
# location of numa
NUMA_INCLUDE=" -I $NUMA_BASE/include/"
NUMA_LIB=" $NUMA_BASE/lib/libnuma.a"
CFLAGS+=" -DNUMA"
# location of libunwind
LIBUNWIND="$LIBUNWIND_BASE/lib/libunwind.a"
fi
# location of TBB
TBB_INCLUDE=" -isystem $TBB_BASE/include/"
if test -z $PIC_BUILD; then
TBB_LIBS="$TBB_BASE/lib/libtbb.a"
else
TBB_LIBS="$TBB_BASE/lib/libtbb_pic.a"
fi
CFLAGS+=" -DTBB"
# use Intel SSE support for checksum calculations
export USE_SSE=1
export PORTABLE=1
BINUTILS="$BINUTILS_BASE/bin"
AR="$BINUTILS/ar"
DEPS_INCLUDE="$SNAPPY_INCLUDE $ZLIB_INCLUDE $BZIP_INCLUDE $LZ4_INCLUDE $ZSTD_INCLUDE $GFLAGS_INCLUDE $NUMA_INCLUDE $TBB_INCLUDE"
STDLIBS="-L $GCC_BASE/lib64"
CLANG_BIN="$CLANG_BASE/bin"
CLANG_LIB="$CLANG_BASE/lib"
CLANG_SRC="$CLANG_BASE/../../src"
CLANG_ANALYZER="$CLANG_BIN/clang++"
CLANG_SCAN_BUILD="$CLANG_SRC/llvm/tools/clang/tools/scan-build/bin/scan-build"
if [ -z "$USE_CLANG" ]; then
# gcc
CC="$GCC_BASE/bin/gcc"
CXX="$GCC_BASE/bin/g++"
CFLAGS+=" -B$BINUTILS/gold"
CFLAGS+=" -isystem $LIBGCC_INCLUDE"
CFLAGS+=" -isystem $GLIBC_INCLUDE"
JEMALLOC=1
else
# clang
CLANG_INCLUDE="$CLANG_LIB/clang/stable/include"
CC="$CLANG_BIN/clang"
CXX="$CLANG_BIN/clang++"
KERNEL_HEADERS_INCLUDE="$KERNEL_HEADERS_BASE/include"
CFLAGS+=" -B$BINUTILS/gold -nostdinc -nostdlib"
CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/7.x "
CFLAGS+=" -isystem $LIBGCC_BASE/include/c++/7.x/x86_64-facebook-linux "
CFLAGS+=" -isystem $GLIBC_INCLUDE"
CFLAGS+=" -isystem $LIBGCC_INCLUDE"
CFLAGS+=" -isystem $CLANG_INCLUDE"
CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE/linux "
CFLAGS+=" -isystem $KERNEL_HEADERS_INCLUDE "
CFLAGS+=" -Wno-expansion-to-defined "
CXXFLAGS="-nostdinc++"
fi
CFLAGS+=" $DEPS_INCLUDE"
CFLAGS+=" -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX -DROCKSDB_FALLOCATE_PRESENT -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -DROCKSDB_SUPPORT_THREAD_LOCAL -DHAVE_SSE42"
CXXFLAGS+=" $CFLAGS"
EXEC_LDFLAGS=" $SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $NUMA_LIB $TBB_LIBS"
EXEC_LDFLAGS+=" -B$BINUTILS/gold"
EXEC_LDFLAGS+=" -Wl,--dynamic-linker,/usr/local/fbcode/platform007/lib/ld.so"
EXEC_LDFLAGS+=" $LIBUNWIND"
EXEC_LDFLAGS+=" -Wl,-rpath=/usr/local/fbcode/platform007/lib"
# required by libtbb
EXEC_LDFLAGS+=" -ldl"
PLATFORM_LDFLAGS="$LIBGCC_LIBS $GLIBC_LIBS $STDLIBS -lgcc -lstdc++"
EXEC_LDFLAGS_SHARED="$SNAPPY_LIBS $ZLIB_LIBS $BZIP_LIBS $LZ4_LIBS $ZSTD_LIBS $GFLAGS_LIBS $TBB_LIBS"
VALGRIND_VER="$VALGRIND_BASE/bin/"
# lua not supported because it's on track for deprecation, I think
LUA_PATH=
LUA_LIB=
export CC CXX AR CFLAGS CXXFLAGS EXEC_LDFLAGS EXEC_LDFLAGS_SHARED VALGRIND_VER JEMALLOC_LIB JEMALLOC_INCLUDE CLANG_ANALYZER CLANG_SCAN_BUILD LUA_PATH LUA_LIB
-5
View File
@@ -1,4 +1,3 @@
# shellcheck disable=SC1113
#/usr/bin/env bash
set -e
@@ -29,14 +28,12 @@ function package() {
if dpkg --get-selections | grep --quiet $1; then
log "$1 is already installed. skipping."
else
# shellcheck disable=SC2068
apt-get install $@ -y
fi
elif [[ $OS = "centos" ]]; then
if rpm -qa | grep --quiet $1; then
log "$1 is already installed. skipping."
else
# shellcheck disable=SC2068
yum install $@ -y
fi
fi
@@ -55,7 +52,6 @@ function gem_install() {
if gem list | grep --quiet $1; then
log "$1 is already installed. skipping."
else
# shellcheck disable=SC2068
gem install $@
fi
}
@@ -129,5 +125,4 @@ function main() {
include $LIB_DIR
}
# shellcheck disable=SC2068
main $@
+2 -3
View File
@@ -85,9 +85,8 @@ NON_SHM="TMPD=/tmp/rocksdb_test_tmp"
GCC_481="ROCKSDB_FBCODE_BUILD_WITH_481=1"
ASAN="COMPILE_WITH_ASAN=1"
CLANG="USE_CLANG=1"
# in gcc-5 there are known problems with TSAN like https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71090.
# using platform007 gives us gcc-8 or higher which has that bug fixed.
TSAN="ROCKSDB_FBCODE_BUILD_WITH_PLATFORM007=1 COMPILE_WITH_TSAN=1"
LITE="OPT=\"-DROCKSDB_LITE -g\""
TSAN="COMPILE_WITH_TSAN=1"
UBSAN="COMPILE_WITH_UBSAN=1"
DISABLE_JEMALLOC="DISABLE_JEMALLOC=1"
HTTP_PROXY="https_proxy=http://fwdproxy.29.prn1:8080 http_proxy=http://fwdproxy.29.prn1:8080 ftp_proxy=http://fwdproxy.29.prn1:8080"
-39
View File
@@ -53,45 +53,6 @@ function get_lib_base()
log_variable $__res_var
}
###########################################################
# platform007 dependencies #
###########################################################
OUTPUT="$BASEDIR/dependencies_platform007.sh"
rm -f "$OUTPUT"
touch "$OUTPUT"
echo "Writing dependencies to $OUTPUT"
# Compilers locations
GCC_BASE=`readlink -f $TP2_LATEST/gcc/7.x/centos7-native/*/`
CLANG_BASE=`readlink -f $TP2_LATEST/llvm-fb/stable/centos7-native/*/`
log_variable GCC_BASE
log_variable CLANG_BASE
# Libraries locations
get_lib_base libgcc 7.x platform007
get_lib_base glibc 2.26 platform007
get_lib_base snappy LATEST platform007
get_lib_base zlib LATEST platform007
get_lib_base bzip2 LATEST platform007
get_lib_base lz4 LATEST platform007
get_lib_base zstd LATEST platform007
get_lib_base gflags LATEST platform007
get_lib_base jemalloc LATEST platform007
get_lib_base numa LATEST platform007
get_lib_base libunwind LATEST platform007
get_lib_base tbb LATEST platform007
get_lib_base kernel-headers fb platform007
get_lib_base binutils LATEST centos7-native
get_lib_base valgrind LATEST platform007
get_lib_base lua 5.3.4 platform007
git diff $OUTPUT
###########################################################
# 5.x dependencies #
###########################################################
+1 -1
View File
@@ -72,7 +72,7 @@ def display_file_coverage(per_file_coverage, total_coverage):
header_template = \
"%" + str(max_file_name_length) + "s\t%s\t%s"
separator = "-" * (max_file_name_length + 10 + 20)
print header_template % ("Filename", "Coverage", "Lines") # noqa: E999 T25377293 Grandfathered in
print header_template % ("Filename", "Coverage", "Lines")
print separator
# -- Print body
+50 -32
View File
@@ -1388,24 +1388,23 @@ void rocksdb_writebatch_put_log_data(
b->rep.PutLogData(Slice(blob, len));
}
class H : public WriteBatch::Handler {
public:
void* state_;
void (*put_)(void*, const char* k, size_t klen, const char* v, size_t vlen);
void (*deleted_)(void*, const char* k, size_t klen);
virtual void Put(const Slice& key, const Slice& value) override {
(*put_)(state_, key.data(), key.size(), value.data(), value.size());
}
virtual void Delete(const Slice& key) override {
(*deleted_)(state_, key.data(), key.size());
}
};
void rocksdb_writebatch_iterate(
rocksdb_writebatch_t* b,
void* state,
void (*put)(void*, const char* k, size_t klen, const char* v, size_t vlen),
void (*deleted)(void*, const char* k, size_t klen)) {
class H : public WriteBatch::Handler {
public:
void* state_;
void (*put_)(void*, const char* k, size_t klen, const char* v, size_t vlen);
void (*deleted_)(void*, const char* k, size_t klen);
virtual void Put(const Slice& key, const Slice& value) override {
(*put_)(state_, key.data(), key.size(), value.data(), value.size());
}
virtual void Delete(const Slice& key) override {
(*deleted_)(state_, key.data(), key.size());
}
};
H handler;
handler.state_ = state;
handler.put_ = put;
@@ -1650,6 +1649,18 @@ void rocksdb_writebatch_wi_iterate(
void* state,
void (*put)(void*, const char* k, size_t klen, const char* v, size_t vlen),
void (*deleted)(void*, const char* k, size_t klen)) {
class H : public WriteBatch::Handler {
public:
void* state_;
void (*put_)(void*, const char* k, size_t klen, const char* v, size_t vlen);
void (*deleted_)(void*, const char* k, size_t klen);
virtual void Put(const Slice& key, const Slice& value) override {
(*put_)(state_, key.data(), key.size(), value.data(), value.size());
}
virtual void Delete(const Slice& key) override {
(*deleted_)(state_, key.data(), key.size());
}
};
H handler;
handler.state_ = state;
handler.put_ = put;
@@ -1988,11 +1999,6 @@ void rocksdb_options_optimize_universal_style_compaction(
opt->rep.OptimizeUniversalStyleCompaction(memtable_memory_budget);
}
void rocksdb_options_set_allow_ingest_behind(
rocksdb_options_t* opt, unsigned char v) {
opt->rep.allow_ingest_behind = v;
}
void rocksdb_options_set_compaction_filter(
rocksdb_options_t* opt,
rocksdb_compactionfilter_t* filter) {
@@ -3098,21 +3104,20 @@ void rocksdb_slicetransform_destroy(rocksdb_slicetransform_t* st) {
delete st;
}
struct Wrapper : public rocksdb_slicetransform_t {
const SliceTransform* rep_;
~Wrapper() { delete rep_; }
const char* Name() const override { return rep_->Name(); }
Slice Transform(const Slice& src) const override {
return rep_->Transform(src);
}
bool InDomain(const Slice& src) const override {
return rep_->InDomain(src);
}
bool InRange(const Slice& src) const override { return rep_->InRange(src); }
static void DoNothing(void*) { }
};
rocksdb_slicetransform_t* rocksdb_slicetransform_create_fixed_prefix(size_t prefixLen) {
struct Wrapper : public rocksdb_slicetransform_t {
const SliceTransform* rep_;
~Wrapper() { delete rep_; }
const char* Name() const override { return rep_->Name(); }
Slice Transform(const Slice& src) const override {
return rep_->Transform(src);
}
bool InDomain(const Slice& src) const override {
return rep_->InDomain(src);
}
bool InRange(const Slice& src) const override { return rep_->InRange(src); }
static void DoNothing(void*) { }
};
Wrapper* wrapper = new Wrapper;
wrapper->rep_ = rocksdb::NewFixedPrefixTransform(prefixLen);
wrapper->state_ = nullptr;
@@ -3121,6 +3126,19 @@ rocksdb_slicetransform_t* rocksdb_slicetransform_create_fixed_prefix(size_t pref
}
rocksdb_slicetransform_t* rocksdb_slicetransform_create_noop() {
struct Wrapper : public rocksdb_slicetransform_t {
const SliceTransform* rep_;
~Wrapper() { delete rep_; }
const char* Name() const override { return rep_->Name(); }
Slice Transform(const Slice& src) const override {
return rep_->Transform(src);
}
bool InDomain(const Slice& src) const override {
return rep_->InDomain(src);
}
bool InRange(const Slice& src) const override { return rep_->InRange(src); }
static void DoNothing(void*) { }
};
Wrapper* wrapper = new Wrapper;
wrapper->rep_ = rocksdb::NewNoopTransform();
wrapper->state_ = nullptr;
+14 -21
View File
@@ -344,13 +344,12 @@ void SuperVersionUnrefHandle(void* ptr) {
// When latter happens, we are in ~ColumnFamilyData(), no get should happen as
// well.
SuperVersion* sv = static_cast<SuperVersion*>(ptr);
bool was_last_ref __attribute__((__unused__));
was_last_ref = sv->Unref();
// Thread-local SuperVersions can't outlive ColumnFamilyData::super_version_.
// This is important because we can't do SuperVersion cleanup here.
// That would require locking DB mutex, which would deadlock because
// SuperVersionUnrefHandle is called with locked ThreadLocalPtr mutex.
assert(!was_last_ref);
if (sv->Unref()) {
sv->db_mutex->Lock();
sv->Cleanup();
sv->db_mutex->Unlock();
delete sv;
}
}
} // anonymous namespace
@@ -386,8 +385,7 @@ ColumnFamilyData::ColumnFamilyData(
pending_flush_(false),
pending_compaction_(false),
prev_compaction_needed_bytes_(0),
allow_2pc_(db_options.allow_2pc),
last_memtable_id_(0) {
allow_2pc_(db_options.allow_2pc) {
Ref();
// Convert user defined table properties collector factories to internal ones.
@@ -967,12 +965,6 @@ void ColumnFamilyData::InstallSuperVersion(
RecalculateWriteStallConditions(mutable_cf_options);
if (old_superversion != nullptr) {
// Reset SuperVersions cached in thread local storage.
// This should be done before old_superversion->Unref(). That's to ensure
// that local_sv_ never holds the last reference to SuperVersion, since
// it has no means to safely do SuperVersion cleanup.
ResetThreadLocalSuperVersions();
if (old_superversion->mutable_cf_options.write_buffer_size !=
mutable_cf_options.write_buffer_size) {
mem_->UpdateWriteBufferSize(mutable_cf_options.write_buffer_size);
@@ -988,6 +980,9 @@ void ColumnFamilyData::InstallSuperVersion(
sv_context->superversions_to_free.push_back(old_superversion);
}
}
// Reset SuperVersions cached in thread local storage
ResetThreadLocalSuperVersions();
}
void ColumnFamilyData::ResetThreadLocalSuperVersions() {
@@ -999,12 +994,10 @@ void ColumnFamilyData::ResetThreadLocalSuperVersions() {
continue;
}
auto sv = static_cast<SuperVersion*>(ptr);
bool was_last_ref __attribute__((__unused__));
was_last_ref = sv->Unref();
// sv couldn't have been the last reference because
// ResetThreadLocalSuperVersions() is called before
// unref'ing super_version_.
assert(!was_last_ref);
if (sv->Unref()) {
sv->Cleanup();
delete sv;
}
}
}
+1 -8
View File
@@ -239,11 +239,7 @@ class ColumnFamilyData {
void SetCurrent(Version* _current);
uint64_t GetNumLiveVersions() const; // REQUIRE: DB mutex held
uint64_t GetTotalSstFilesSize() const; // REQUIRE: DB mutex held
void SetMemtable(MemTable* new_mem) {
uint64_t memtable_id = last_memtable_id_.fetch_add(1) + 1;
new_mem->SetID(memtable_id);
mem_ = new_mem;
}
void SetMemtable(MemTable* new_mem) { mem_ = new_mem; }
// calculate the oldest log needed for the durability of this column family
uint64_t OldestLogToKeep();
@@ -423,9 +419,6 @@ class ColumnFamilyData {
// if the database was opened with 2pc enabled
bool allow_2pc_;
// Memtable id to track flush.
std::atomic<uint64_t> last_memtable_id_;
};
// ColumnFamilySet has interesting thread-safety requirements
+5 -25
View File
@@ -184,11 +184,8 @@ void CompactionIterator::InvokeFilterIfNeeded(bool* need_skip,
Slice* skip_until) {
if (compaction_filter_ != nullptr &&
(ikey_.type == kTypeValue || ikey_.type == kTypeBlobIndex) &&
(visible_at_tip_ || ignore_snapshots_ ||
ikey_.sequence > latest_snapshot_ ||
(snapshot_checker_ != nullptr &&
UNLIKELY(!snapshot_checker_->IsInSnapshot(ikey_.sequence,
latest_snapshot_))))) {
(visible_at_tip_ || ikey_.sequence > latest_snapshot_ ||
ignore_snapshots_)) {
// If the user has specified a compaction filter and the sequence
// number is greater than any external snapshot, then invoke the
// filter. If the return value of the compaction filter is true,
@@ -414,10 +411,7 @@ void CompactionIterator::NextFromInput() {
cmp_->Equal(ikey_.user_key, next_ikey.user_key)) {
// Check whether the next key belongs to the same snapshot as the
// SingleDelete.
if (prev_snapshot == 0 || next_ikey.sequence > prev_snapshot ||
(snapshot_checker_ != nullptr &&
UNLIKELY(!snapshot_checker_->IsInSnapshot(next_ikey.sequence,
prev_snapshot)))) {
if (prev_snapshot == 0 || next_ikey.sequence > prev_snapshot) {
if (next_ikey.type == kTypeSingleDeletion) {
// We encountered two SingleDeletes in a row. This could be due to
// unexpected user input.
@@ -428,12 +422,8 @@ void CompactionIterator::NextFromInput() {
// input_->Next().
++iter_stats_.num_record_drop_obsolete;
++iter_stats_.num_single_del_mismatch;
} else if (has_outputted_key_ ||
(ikey_.sequence <= earliest_write_conflict_snapshot_ &&
(snapshot_checker_ == nullptr ||
LIKELY(snapshot_checker_->IsInSnapshot(
ikey_.sequence,
earliest_write_conflict_snapshot_))))) {
} else if ((ikey_.sequence <= earliest_write_conflict_snapshot_) ||
has_outputted_key_) {
// Found a matching value, we can drop the single delete and the
// value. It is safe to drop both records since we've already
// outputted a key in this snapshot, or there is no earlier
@@ -481,9 +471,6 @@ void CompactionIterator::NextFromInput() {
// comparison, so the value of has_current_user_key does not matter.
has_current_user_key_ = false;
if (compaction_ != nullptr && ikey_.sequence <= earliest_snapshot_ &&
(snapshot_checker_ == nullptr ||
LIKELY(snapshot_checker_->IsInSnapshot(ikey_.sequence,
earliest_snapshot_))) &&
compaction_->KeyNotExistsBeyondOutputLevel(ikey_.user_key,
&level_ptrs_)) {
// Key doesn't exist outside of this range.
@@ -517,9 +504,6 @@ void CompactionIterator::NextFromInput() {
input_->Next();
} else if (compaction_ != nullptr && ikey_.type == kTypeDeletion &&
ikey_.sequence <= earliest_snapshot_ &&
(snapshot_checker_ == nullptr ||
LIKELY(snapshot_checker_->IsInSnapshot(ikey_.sequence,
earliest_snapshot_))) &&
ikeyNotNeededForIncrementalSnapshot() &&
compaction_->KeyNotExistsBeyondOutputLevel(ikey_.user_key,
&level_ptrs_)) {
@@ -554,10 +538,6 @@ void CompactionIterator::NextFromInput() {
// have hit (A)
// We encapsulate the merge related state machine in a different
// object to minimize change to the existing flow.
// In case snapshot_checker is present, we can probably merge further
// beyond prev_snapshot, since there could be more keys with sequence
// smaller than prev_snapshot, but reported by snapshot_checker as not
// visible by prev_snapshot. But it will make the logic more complicated.
Status s = merge_helper_->MergeUntil(input_, range_del_agg_,
prev_snapshot, bottommost_level_);
merge_out_iter_.SeekToFirst();
+2 -2
View File
@@ -625,8 +625,8 @@ Status CompactionJob::Install(const MutableCFOptions& mutable_cf_options) {
"[%s] compacted to: %s, MB/sec: %.1f rd, %.1f wr, level %d, "
"files in(%d, %d) out(%d) "
"MB in(%.1f, %.1f) out(%.1f), read-write-amplify(%.1f) "
"write-amplify(%.1f) %s, records in: %" PRIu64
", records dropped: %" PRIu64 " output_compression: %s\n",
"write-amplify(%.1f) %s, records in: %d, records dropped: %d "
"output_compression: %s\n",
cfd->GetName().c_str(), vstorage->LevelSummary(&tmp), bytes_read_per_sec,
bytes_written_per_sec, compact_->compaction->output_level(),
stats.num_input_files_in_non_output_levels,
-35
View File
@@ -126,41 +126,6 @@ TEST_F(DBFlushTest, FlushInLowPriThreadPool) {
ASSERT_EQ(1, num_compactions);
}
TEST_F(DBFlushTest, ManualFlushWithMinWriteBufferNumberToMerge) {
Options options = CurrentOptions();
options.write_buffer_size = 100;
options.max_write_buffer_number = 4;
options.min_write_buffer_number_to_merge = 3;
Reopen(options);
SyncPoint::GetInstance()->LoadDependency(
{{"DBImpl::BGWorkFlush",
"DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:1"},
{"DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:2",
"FlushJob::WriteLevel0Table"}});
SyncPoint::GetInstance()->EnableProcessing();
ASSERT_OK(Put("key1", "value1"));
port::Thread t([&]() {
// The call wait for flush to finish, i.e. with flush_options.wait = true.
ASSERT_OK(Flush());
});
// Wait for flush start.
TEST_SYNC_POINT("DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:1");
// Insert a second memtable before the manual flush finish.
// At the end of the manual flush job, it will check if further flush
// is needed, but it will not trigger flush of the second memtable because
// min_write_buffer_number_to_merge is not reached.
ASSERT_OK(Put("key2", "value2"));
ASSERT_OK(dbfull()->TEST_SwitchMemtable());
TEST_SYNC_POINT("DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge:2");
// Manual flush should return, without waiting for flush indefinitely.
t.join();
}
TEST_P(DBFlushDirectIOTest, DirectIO) {
Options options;
options.create_if_missing = true;
+2 -4
View File
@@ -1510,8 +1510,7 @@ ArenaWrappedDBIter* DBImpl::NewIteratorImpl(const ReadOptions& read_options,
ColumnFamilyData* cfd,
SequenceNumber snapshot,
ReadCallback* read_callback,
bool allow_blob,
bool allow_refresh) {
bool allow_blob) {
SuperVersion* sv = cfd->GetReferencedSuperVersion(&mutex_);
// Try to generate a DB iterator tree in continuous memory area to be
@@ -1560,8 +1559,7 @@ ArenaWrappedDBIter* DBImpl::NewIteratorImpl(const ReadOptions& read_options,
env_, read_options, *cfd->ioptions(), snapshot,
sv->mutable_cf_options.max_sequential_skip_in_iterations,
sv->version_number, read_callback,
((read_options.snapshot != nullptr) ? nullptr : this), cfd, allow_blob,
allow_refresh);
((read_options.snapshot != nullptr) ? nullptr : this), cfd, allow_blob);
InternalIterator* internal_iter =
NewInternalIterator(read_options, cfd, sv, db_iter->GetArena(),
+4 -9
View File
@@ -148,8 +148,7 @@ class DBImpl : public DB {
ColumnFamilyData* cfd,
SequenceNumber snapshot,
ReadCallback* read_callback,
bool allow_blob = false,
bool allow_refresh = true);
bool allow_blob = false);
virtual const Snapshot* GetSnapshot() override;
virtual void ReleaseSnapshot(const Snapshot* snapshot) override;
@@ -807,12 +806,8 @@ class DBImpl : public DB {
Status FlushMemTable(ColumnFamilyData* cfd, const FlushOptions& options,
bool writes_stopped = false);
// Wait for memtable flushed.
// If flush_memtable_id is non-null, wait until the memtable with the ID
// gets flush. Otherwise, wait until the column family don't have any
// memtable pending flush.
Status WaitForFlushMemTable(ColumnFamilyData* cfd,
const uint64_t* flush_memtable_id = nullptr);
// Wait for memtable flushed
Status WaitForFlushMemTable(ColumnFamilyData* cfd);
// REQUIRES: mutex locked
Status SwitchWAL(WriteContext* write_context);
@@ -951,7 +946,7 @@ class DBImpl : public DB {
// from the same write_thread_ without any locks.
uint64_t logfile_number_;
std::deque<uint64_t>
log_recycle_files_; // a list of log files that we can recycle
log_recycle_files; // a list of log files that we can recycle
bool log_dir_synced_;
// Without two_write_queues, read and writes to log_empty_ are protected by
// mutex_. Since it is currently updated/read only in write_thread_, it can be
+4 -15
View File
@@ -809,13 +809,7 @@ int DBImpl::Level0StopWriteTrigger(ColumnFamilyHandle* column_family) {
Status DBImpl::Flush(const FlushOptions& flush_options,
ColumnFamilyHandle* column_family) {
auto cfh = reinterpret_cast<ColumnFamilyHandleImpl*>(column_family);
ROCKS_LOG_INFO(immutable_db_options_.info_log, "[%s] Manual flush start.",
cfh->GetName().c_str());
Status s = FlushMemTable(cfh->cfd(), flush_options);
ROCKS_LOG_INFO(immutable_db_options_.info_log,
"[%s] Manual flush finished, status: %s\n",
cfh->GetName().c_str(), s.ToString().c_str());
return s;
return FlushMemTable(cfh->cfd(), flush_options);
}
Status DBImpl::RunManualCompaction(ColumnFamilyData* cfd, int input_level,
@@ -950,7 +944,6 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
const FlushOptions& flush_options,
bool writes_stopped) {
Status s;
uint64_t flush_memtable_id = 0;
{
WriteContext context;
InstrumentedMutexLock guard_lock(&mutex_);
@@ -968,7 +961,6 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
// SwitchMemtable() will release and reacquire mutex during execution
s = SwitchMemtable(cfd, &context);
flush_memtable_id = cfd->imm()->GetLatestMemTableID();
if (!writes_stopped) {
write_thread_.ExitUnbatched(&w);
@@ -983,19 +975,16 @@ Status DBImpl::FlushMemTable(ColumnFamilyData* cfd,
if (s.ok() && flush_options.wait) {
// Wait until the compaction completes
s = WaitForFlushMemTable(cfd, &flush_memtable_id);
s = WaitForFlushMemTable(cfd);
}
return s;
}
Status DBImpl::WaitForFlushMemTable(ColumnFamilyData* cfd,
const uint64_t* flush_memtable_id) {
Status DBImpl::WaitForFlushMemTable(ColumnFamilyData* cfd) {
Status s;
// Wait until the compaction completes
InstrumentedMutexLock l(&mutex_);
while (cfd->imm()->NumNotFlushed() > 0 && bg_error_.ok() &&
(flush_memtable_id == nullptr ||
cfd->imm()->GetEarliestMemTableID() <= *flush_memtable_id)) {
while (cfd->imm()->NumNotFlushed() > 0 && bg_error_.ok()) {
if (shutting_down_.load(std::memory_order_acquire)) {
return Status::ShutdownInProgress();
}
+4 -4
View File
@@ -238,11 +238,11 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
while (alive_log_files_.begin()->number < min_log_number) {
auto& earliest = *alive_log_files_.begin();
if (immutable_db_options_.recycle_log_file_num >
log_recycle_files_.size()) {
log_recycle_files.size()) {
ROCKS_LOG_INFO(immutable_db_options_.info_log,
"adding log %" PRIu64 " to recycle list\n",
earliest.number);
log_recycle_files_.push_back(earliest.number);
log_recycle_files.push_back(earliest.number);
} else {
job_context->log_delete_files.push_back(earliest.number);
}
@@ -283,8 +283,8 @@ void DBImpl::FindObsoleteFiles(JobContext* job_context, bool force,
// We're just cleaning up for DB::Write().
assert(job_context->logs_to_free.empty());
job_context->logs_to_free = logs_to_free_;
job_context->log_recycle_files.assign(log_recycle_files_.begin(),
log_recycle_files_.end());
job_context->log_recycle_files.assign(log_recycle_files.begin(),
log_recycle_files.end());
logs_to_free_.clear();
}
+8 -12
View File
@@ -521,6 +521,9 @@ Status DBImpl::WriteImplWALOnly(const WriteOptions& write_options,
PERF_TIMER_GUARD(write_pre_and_post_process_time);
WriteThread::Writer w(write_options, my_batch, callback, log_ref,
true /* disable_memtable */, pre_release_callback);
if (write_options.disableWAL) {
return status;
}
RecordTick(stats_, WRITE_WITH_WAL);
StopWatch write_sw(env_, immutable_db_options_.statistics.get(), DB_WRITE);
@@ -577,13 +580,7 @@ Status DBImpl::WriteImplWALOnly(const WriteOptions& write_options,
// LastAllocatedSequence is increased inside WriteToWAL under
// wal_write_mutex_ to ensure ordered events in WAL
size_t seq_inc = seq_per_batch_ ? write_group.size : 0 /*total_count*/;
if (!write_options.disableWAL) {
status =
ConcurrentWriteToWAL(write_group, log_used, &last_sequence, seq_inc);
} else {
// Otherwise we inc seq number to do solely the seq allocation
last_sequence = versions_->FetchAddLastAllocatedSequence(seq_inc);
}
status = ConcurrentWriteToWAL(write_group, log_used, &last_sequence, seq_inc);
auto curr_seq = last_sequence + 1;
for (auto* writer : write_group) {
if (writer->CallbackFailed()) {
@@ -596,7 +593,6 @@ Status DBImpl::WriteImplWALOnly(const WriteOptions& write_options,
// else seq advances only by memtable writes
}
if (status.ok() && write_options.sync) {
assert(!write_options.disableWAL);
// Requesting sync with two_write_queues_ is expected to be very rare. We
// hance provide a simple implementation that is not necessarily efficient.
if (manual_wal_flush_) {
@@ -697,7 +693,7 @@ Status DBImpl::PreprocessWrite(const WriteOptions& write_options,
}
if (UNLIKELY(status.ok() && !bg_error_.ok())) {
status = bg_error_;
return bg_error_;
}
if (UNLIKELY(status.ok() && !flush_scheduler_.Empty())) {
@@ -1215,9 +1211,9 @@ Status DBImpl::SwitchMemtable(ColumnFamilyData* cfd, WriteContext* context) {
}
uint64_t recycle_log_number = 0;
if (creating_new_log && immutable_db_options_.recycle_log_file_num &&
!log_recycle_files_.empty()) {
recycle_log_number = log_recycle_files_.front();
log_recycle_files_.pop_front();
!log_recycle_files.empty()) {
recycle_log_number = log_recycle_files.front();
log_recycle_files.pop_front();
}
uint64_t new_log_number =
creating_new_log ? versions_->NewFileNumber() : logfile_number_;
+6 -11
View File
@@ -217,9 +217,6 @@ class DBIter final: public Iterator {
*prop = "Iterator is not valid.";
}
return Status::OK();
} else if (prop_name == "rocksdb.iterator.internal-key") {
*prop = saved_key_.GetUserKey().ToString();
return Status::OK();
}
return Status::InvalidArgument("Undentified property.");
}
@@ -1381,19 +1378,17 @@ void ArenaWrappedDBIter::Init(Env* env, const ReadOptions& read_options,
const SequenceNumber& sequence,
uint64_t max_sequential_skip_in_iteration,
uint64_t version_number,
ReadCallback* read_callback, bool allow_blob,
bool allow_refresh) {
ReadCallback* read_callback, bool allow_blob) {
auto mem = arena_.AllocateAligned(sizeof(DBIter));
db_iter_ = new (mem)
DBIter(env, read_options, cf_options, cf_options.user_comparator, nullptr,
sequence, true, max_sequential_skip_in_iteration, read_callback,
allow_blob);
sv_number_ = version_number;
allow_refresh_ = allow_refresh;
}
Status ArenaWrappedDBIter::Refresh() {
if (cfd_ == nullptr || db_impl_ == nullptr || !allow_refresh_) {
if (cfd_ == nullptr || db_impl_ == nullptr) {
return Status::NotSupported("Creating renew iterator is not allowed.");
}
assert(db_iter_ != nullptr);
@@ -1411,7 +1406,7 @@ Status ArenaWrappedDBIter::Refresh() {
SuperVersion* sv = cfd_->GetReferencedSuperVersion(db_impl_->mutex());
Init(env, read_options_, *(cfd_->ioptions()), latest_seq,
sv->mutable_cf_options.max_sequential_skip_in_iterations,
cur_sv_number, read_callback_, allow_blob_, allow_refresh_);
cur_sv_number, read_callback_, allow_blob_);
InternalIterator* internal_iter = db_impl_->NewInternalIterator(
read_options_, cfd_, sv, &arena_, db_iter_->GetRangeDelAggregator());
@@ -1428,12 +1423,12 @@ ArenaWrappedDBIter* NewArenaWrappedDbIterator(
const ImmutableCFOptions& cf_options, const SequenceNumber& sequence,
uint64_t max_sequential_skip_in_iterations, uint64_t version_number,
ReadCallback* read_callback, DBImpl* db_impl, ColumnFamilyData* cfd,
bool allow_blob, bool allow_refresh) {
bool allow_blob) {
ArenaWrappedDBIter* iter = new ArenaWrappedDBIter();
iter->Init(env, read_options, cf_options, sequence,
max_sequential_skip_in_iterations, version_number, read_callback,
allow_blob, allow_refresh);
if (db_impl != nullptr && cfd != nullptr && allow_refresh) {
allow_blob);
if (db_impl != nullptr && cfd != nullptr) {
iter->StoreRefreshInfo(read_options, db_impl, cfd, read_callback,
allow_blob);
}
+3 -4
View File
@@ -73,7 +73,7 @@ class ArenaWrappedDBIter : public Iterator {
const ImmutableCFOptions& cf_options,
const SequenceNumber& sequence,
uint64_t max_sequential_skip_in_iterations, uint64_t version_number,
ReadCallback* read_callback, bool allow_blob, bool allow_refresh);
ReadCallback* read_callback, bool allow_blob);
void StoreRefreshInfo(const ReadOptions& read_options, DBImpl* db_impl,
ColumnFamilyData* cfd, ReadCallback* read_callback,
@@ -94,7 +94,6 @@ class ArenaWrappedDBIter : public Iterator {
ReadOptions read_options_;
ReadCallback* read_callback_;
bool allow_blob_ = false;
bool allow_refresh_ = true;
};
// Generate the arena wrapped iterator class.
@@ -105,6 +104,6 @@ extern ArenaWrappedDBIter* NewArenaWrappedDbIterator(
const ImmutableCFOptions& cf_options, const SequenceNumber& sequence,
uint64_t max_sequential_skip_in_iterations, uint64_t version_number,
ReadCallback* read_callback, DBImpl* db_impl = nullptr,
ColumnFamilyData* cfd = nullptr, bool allow_blob = false,
bool allow_refresh = true);
ColumnFamilyData* cfd = nullptr, bool allow_blob = false);
} // namespace rocksdb
-49
View File
@@ -55,7 +55,6 @@ TEST_F(DBIteratorTest, IteratorProperty) {
Options options = CurrentOptions();
CreateAndReopenWithCF({"pikachu"}, options);
Put(1, "1", "2");
Delete(1, "2");
ReadOptions ropt;
ropt.pin_data = false;
{
@@ -65,15 +64,9 @@ TEST_F(DBIteratorTest, IteratorProperty) {
ASSERT_NOK(iter->GetProperty("non_existing.value", &prop_value));
ASSERT_OK(iter->GetProperty("rocksdb.iterator.is-key-pinned", &prop_value));
ASSERT_EQ("0", prop_value);
ASSERT_OK(iter->GetProperty("rocksdb.iterator.internal-key", &prop_value));
ASSERT_EQ("1", prop_value);
iter->Next();
ASSERT_OK(iter->GetProperty("rocksdb.iterator.is-key-pinned", &prop_value));
ASSERT_EQ("Iterator is not valid.", prop_value);
// Get internal key at which the iteration stopped (tombstone in this case).
ASSERT_OK(iter->GetProperty("rocksdb.iterator.internal-key", &prop_value));
ASSERT_EQ("2", prop_value);
}
Close();
}
@@ -2164,48 +2157,6 @@ TEST_F(DBIteratorTest, SkipStatistics) {
ASSERT_EQ(skip_count, TestGetTickerCount(options, NUMBER_ITER_SKIP));
}
TEST_F(DBIteratorTest, SeekAfterHittingManyInternalKeys) {
Options options = CurrentOptions();
DestroyAndReopen(options);
ReadOptions ropts;
ropts.max_skippable_internal_keys = 2;
Put("1", "val_1");
// Add more tombstones than max_skippable_internal_keys so that Next() fails.
Delete("2");
Delete("3");
Delete("4");
Delete("5");
Put("6", "val_6");
unique_ptr<Iterator> iter(db_->NewIterator(ropts));
iter->SeekToFirst();
ASSERT_TRUE(iter->Valid());
ASSERT_EQ(iter->key().ToString(), "1");
ASSERT_EQ(iter->value().ToString(), "val_1");
// This should fail as incomplete due to too many non-visible internal keys on
// the way to the next valid user key.
iter->Next();
ASSERT_TRUE(!iter->Valid());
ASSERT_TRUE(iter->status().IsIncomplete());
// Get the internal key at which Next() failed.
std::string prop_value;
ASSERT_OK(iter->GetProperty("rocksdb.iterator.internal-key", &prop_value));
ASSERT_EQ("4", prop_value);
// Create a new iterator to seek to the internal key.
unique_ptr<Iterator> iter2(db_->NewIterator(ropts));
iter2->Seek(prop_value);
ASSERT_TRUE(iter2->Valid());
ASSERT_OK(iter2->status());
ASSERT_EQ(iter2->key().ToString(), "6");
ASSERT_EQ(iter2->value().ToString(), "val_6");
}
} // namespace rocksdb
int main(int argc, char** argv) {
-24
View File
@@ -376,30 +376,6 @@ TEST_F(DBSSTTest, RateLimitedDelete) {
rocksdb::SyncPoint::GetInstance()->DisableProcessing();
}
TEST_F(DBSSTTest, OpenDBWithExistingTrash) {
Options options = CurrentOptions();
options.sst_file_manager.reset(
NewSstFileManager(env_, nullptr, "", 1024 * 1024 /* 1 MB/sec */));
auto sfm = static_cast<SstFileManagerImpl*>(options.sst_file_manager.get());
Destroy(last_options_);
// Add some trash files to the db directory so the DB can clean them up
env_->CreateDirIfMissing(dbname_);
ASSERT_OK(WriteStringToFile(env_, "abc", dbname_ + "/" + "001.sst.trash"));
ASSERT_OK(WriteStringToFile(env_, "abc", dbname_ + "/" + "002.sst.trash"));
ASSERT_OK(WriteStringToFile(env_, "abc", dbname_ + "/" + "003.sst.trash"));
// Reopen the DB and verify that it deletes existing trash files
ASSERT_OK(TryReopen(options));
sfm->WaitForEmptyTrash();
ASSERT_NOK(env_->FileExists(dbname_ + "/" + "001.sst.trash"));
ASSERT_NOK(env_->FileExists(dbname_ + "/" + "002.sst.trash"));
ASSERT_NOK(env_->FileExists(dbname_ + "/" + "003.sst.trash"));
}
// Create a DB with 2 db_paths, and generate multiple files in the 2
// db_paths using CompactRangeOptions, make sure that files that were
// deleted from first db_path were deleted using DeleteScheduler and
+6 -55
View File
@@ -3488,16 +3488,12 @@ TEST_F(DBTest, GetThreadStatus) {
const int kTestCount = 3;
const unsigned int kHighPriCounts[kTestCount] = {3, 2, 5};
const unsigned int kLowPriCounts[kTestCount] = {10, 15, 3};
const unsigned int kBottomPriCounts[kTestCount] = {2, 1, 4};
for (int test = 0; test < kTestCount; ++test) {
// Change the number of threads in high / low priority pool.
env_->SetBackgroundThreads(kHighPriCounts[test], Env::HIGH);
env_->SetBackgroundThreads(kLowPriCounts[test], Env::LOW);
env_->SetBackgroundThreads(kBottomPriCounts[test], Env::BOTTOM);
// Wait to ensure the all threads has been registered
unsigned int thread_type_counts[ThreadStatus::NUM_THREAD_TYPES];
// TODO(ajkr): it'd be better if SetBackgroundThreads returned only after
// all threads have been registered.
// Try up to 60 seconds.
for (int num_try = 0; num_try < 60000; num_try++) {
env_->SleepForMicroseconds(1000);
@@ -3512,21 +3508,20 @@ TEST_F(DBTest, GetThreadStatus) {
if (thread_type_counts[ThreadStatus::HIGH_PRIORITY] ==
kHighPriCounts[test] &&
thread_type_counts[ThreadStatus::LOW_PRIORITY] ==
kLowPriCounts[test] &&
thread_type_counts[ThreadStatus::BOTTOM_PRIORITY] ==
kBottomPriCounts[test]) {
kLowPriCounts[test]) {
break;
}
}
// Verify the total number of threades
ASSERT_EQ(thread_type_counts[ThreadStatus::HIGH_PRIORITY] +
thread_type_counts[ThreadStatus::LOW_PRIORITY],
kHighPriCounts[test] + kLowPriCounts[test]);
// Verify the number of high-priority threads
ASSERT_EQ(thread_type_counts[ThreadStatus::HIGH_PRIORITY],
kHighPriCounts[test]);
// Verify the number of low-priority threads
ASSERT_EQ(thread_type_counts[ThreadStatus::LOW_PRIORITY],
kLowPriCounts[test]);
// Verify the number of bottom-priority threads
ASSERT_EQ(thread_type_counts[ThreadStatus::BOTTOM_PRIORITY],
kBottomPriCounts[test]);
}
if (i == 0) {
// repeat the test with multiple column families
@@ -4380,6 +4375,7 @@ TEST_F(DBTest, DynamicFIFOCompactionOptions) {
ASSERT_EQ(dbfull()->GetOptions().compaction_options_fifo.allow_compaction,
true);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, DynamicUniversalCompactionOptions) {
Options options;
@@ -4448,7 +4444,6 @@ TEST_F(DBTest, DynamicUniversalCompactionOptions) {
dbfull()->GetOptions().compaction_options_universal.allow_trivial_move,
false);
}
#endif // ROCKSDB_LITE
TEST_F(DBTest, FileCreationRandomFailure) {
Options options;
@@ -5656,50 +5651,6 @@ TEST_F(DBTest, PauseBackgroundWorkTest) {
// now it's done
ASSERT_TRUE(done.load());
}
// Keep spawning short-living threads that create an iterator and quit.
// Meanwhile in another thread keep flushing memtables.
// This used to cause a deadlock.
TEST_F(DBTest, ThreadLocalPtrDeadlock) {
std::atomic<int> flushes_done{0};
std::atomic<int> threads_destroyed{0};
auto done = [&] {
return flushes_done.load() > 10;
};
std::thread flushing_thread([&] {
for (int i = 0; !done(); ++i) {
ASSERT_OK(db_->Put(WriteOptions(), Slice("hi"),
Slice(std::to_string(i).c_str())));
ASSERT_OK(db_->Flush(FlushOptions()));
int cnt = ++flushes_done;
fprintf(stderr, "Flushed %d times\n", cnt);
}
});
std::vector<std::thread> thread_spawning_threads(10);
for (auto& t: thread_spawning_threads) {
t = std::thread([&] {
while (!done()) {
{
std::thread tmp_thread([&] {
auto it = db_->NewIterator(ReadOptions());
delete it;
});
tmp_thread.join();
}
++threads_destroyed;
}
});
}
for (auto& t: thread_spawning_threads) {
t.join();
}
flushing_thread.join();
fprintf(stderr, "Done. Flushed %d times, destroyed %d threads\n",
flushes_done.load(), threads_destroyed.load());
}
} // namespace rocksdb
int main(int argc, char** argv) {
+1 -3
View File
@@ -1157,13 +1157,11 @@ UpdateStatus DBTestBase::updateInPlaceNoAction(char* prevValue,
// Utility method to test InplaceUpdate
void DBTestBase::validateNumberOfEntries(int numValues, int cf) {
ScopedArenaIterator iter;
Arena arena;
auto options = CurrentOptions();
InternalKeyComparator icmp(options.comparator);
RangeDelAggregator range_del_agg(icmp, {} /* snapshots */);
// This should be defined after range_del_agg so that it destructs the
// assigned iterator before it range_del_agg is already destructed.
ScopedArenaIterator iter;
if (cf != 0) {
iter.set(
dbfull()->NewInternalIterator(&arena, &range_del_agg, handles_[cf]));
-1
View File
@@ -364,7 +364,6 @@ Status FlushJob::WriteLevel0Table() {
InternalStats::CompactionStats stats(1);
stats.micros = db_options_.env->NowMicros() - start_micros;
stats.bytes_written = meta_.fd.GetFileSize();
MeasureTime(stats_, FLUSH_TIME, stats.micros);
cfd_->internal_stats()->AddCompactionStats(0 /* level */, stats);
cfd_->internal_stats()->AddCFStats(InternalStats::BYTES_FLUSHED,
meta_.fd.GetFileSize());
+6 -16
View File
@@ -40,7 +40,6 @@ class FlushJobTest : public testing::Test {
EXPECT_OK(env_->CreateDirIfMissing(dbname_));
db_options_.db_paths.emplace_back(dbname_,
std::numeric_limits<uint64_t>::max());
db_options_.statistics = rocksdb::CreateDBStatistics();
// TODO(icanadi) Remove this once we mock out VersionSet
NewDB();
std::vector<ColumnFamilyDescriptor> column_families;
@@ -139,22 +138,16 @@ TEST_F(FlushJobTest, NonEmpty) {
EventLogger event_logger(db_options_.info_log.get());
SnapshotChecker* snapshot_checker = nullptr; // not relavant
FlushJob flush_job(dbname_, versions_->GetColumnFamilySet()->GetDefault(),
db_options_, *cfd->GetLatestMutableCFOptions(),
env_options_, versions_.get(), &mutex_, &shutting_down_,
{}, kMaxSequenceNumber, snapshot_checker, &job_context,
nullptr, nullptr, nullptr, kNoCompression,
db_options_.statistics.get(), &event_logger, true);
HistogramData hist;
FlushJob flush_job(
dbname_, versions_->GetColumnFamilySet()->GetDefault(), db_options_,
*cfd->GetLatestMutableCFOptions(), env_options_, versions_.get(), &mutex_,
&shutting_down_, {}, kMaxSequenceNumber, snapshot_checker, &job_context,
nullptr, nullptr, nullptr, kNoCompression, nullptr, &event_logger, true);
FileMetaData fd;
mutex_.Lock();
flush_job.PickMemTable();
ASSERT_OK(flush_job.Run(&fd));
mutex_.Unlock();
db_options_.statistics->histogramData(FLUSH_TIME, &hist);
ASSERT_GT(hist.average, 0.0);
ASSERT_EQ(ToString(0), fd.smallest.user_key().ToString());
ASSERT_EQ("9999a",
fd.largest.user_key().ToString()); // range tombstone end key
@@ -217,15 +210,12 @@ TEST_F(FlushJobTest, Snapshots) {
env_options_, versions_.get(), &mutex_, &shutting_down_,
snapshots, kMaxSequenceNumber, snapshot_checker,
&job_context, nullptr, nullptr, nullptr, kNoCompression,
db_options_.statistics.get(), &event_logger, true);
nullptr, &event_logger, true);
mutex_.Lock();
flush_job.PickMemTable();
ASSERT_OK(flush_job.Run());
mutex_.Unlock();
mock_table_factory_->AssertSingleFile(inserted_keys);
HistogramData hist;
db_options_.statistics->histogramData(FLUSH_TIME, &hist);
ASSERT_GT(hist.average, 0.0);
job_context.Clean();
}
+2 -2
View File
@@ -566,7 +566,7 @@ void ForwardIterator::RebuildIterators(bool refresh_sv) {
sv_ = cfd_->GetReferencedSuperVersion(&(db_->mutex_));
}
RangeDelAggregator range_del_agg(
cfd_->internal_comparator(), {} /* snapshots */);
InternalKeyComparator(cfd_->internal_comparator()), {} /* snapshots */);
mutable_iter_ = sv_->mem->NewIterator(read_options_, &arena_);
sv_->imm->AddIterators(read_options_, &imm_iters_, &arena_);
if (!read_options_.ignore_range_deletions) {
@@ -621,7 +621,7 @@ void ForwardIterator::RenewIterators() {
mutable_iter_ = svnew->mem->NewIterator(read_options_, &arena_);
svnew->imm->AddIterators(read_options_, &imm_iters_, &arena_);
RangeDelAggregator range_del_agg(
cfd_->internal_comparator(), {} /* snapshots */);
InternalKeyComparator(cfd_->internal_comparator()), {} /* snapshots */);
if (!read_options_.ignore_range_deletions) {
std::unique_ptr<InternalIterator> range_del_iter(
svnew->mem->NewRangeTombstoneIterator(read_options_));
-1
View File
@@ -54,7 +54,6 @@ class Reader {
// The Reader will start reading at the first record located at physical
// position >= initial_offset within the file.
Reader(std::shared_ptr<Logger> info_log,
// @lint-ignore TXT2 T25377293 Grandfathered in
unique_ptr<SequentialFileReader>&& file,
Reporter* reporter, bool checksum, uint64_t initial_offset,
uint64_t log_num);
-8
View File
@@ -368,11 +368,6 @@ class MemTable {
return oldest_key_time_.load(std::memory_order_relaxed);
}
// REQUIRES: db_mutex held.
void SetID(uint64_t id) { id_ = id; }
uint64_t GetID() const { return id_; }
private:
enum FlushStateEnum { FLUSH_NOT_REQUESTED, FLUSH_REQUESTED, FLUSH_SCHEDULED };
@@ -442,9 +437,6 @@ class MemTable {
// Timestamp of oldest key
std::atomic<uint64_t> oldest_key_time_;
// Memtable id to track flush.
uint64_t id_ = 0;
// Returns a heuristic flush decision
bool ShouldFlushNow() const;
+3 -20
View File
@@ -5,12 +5,11 @@
//
#pragma once
#include <deque>
#include <limits>
#include <list>
#include <set>
#include <string>
#include <list>
#include <vector>
#include <set>
#include <deque>
#include "db/dbformat.h"
#include "db/memtable.h"
@@ -245,22 +244,6 @@ class MemTableList {
uint64_t GetMinLogContainingPrepSection();
uint64_t GetEarliestMemTableID() const {
auto& memlist = current_->memlist_;
if (memlist.empty()) {
return std::numeric_limits<uint64_t>::max();
}
return memlist.back()->GetID();
}
uint64_t GetLatestMemTableID() const {
auto& memlist = current_->memlist_;
if (memlist.empty()) {
return 0;
}
return memlist.front()->GetID();
}
private:
// DB mutex held
void InstallNewVersion();
@@ -1,28 +0,0 @@
---
title: Auto-tuned Rate Limiter
layout: post
author: ajkr
category: blog
---
### Introduction
Our rate limiter has been hard to configure since users need to pick a value that is low enough to prevent background I/O spikes, which can impact user-visible read/write latencies. Meanwhile, picking too low a value can cause memtables and L0 files to pile up, eventually leading to writes stalling. Tuning the rate limiter has been especially difficult for users whose DB instances have different workloads, or have workloads that vary over time, or commonly both.
To address this, in RocksDB 5.9 we released a dynamic rate limiter that adjusts itself over time according to demand for background I/O. It can be enabled simply by passing `auto_tuned=true` in the `NewGenericRateLimiter()` call. In this case `rate_bytes_per_sec` will indicate the upper-bound of the window within which a rate limit will be picked dynamically. The chosen rate limit will be much lower unless absolutely necessary, so setting this to the device's maximum throughput is a reasonable choice on dedicated hosts.
### Algorithm
We use a simple multiplicative-increase, multiplicative-decrease algorithm. We measure demand for background I/O as the ratio of intervals where the rate limiter is drained. There are low and high watermarks for this ratio, which will trigger a change in rate limit when breached. The rate limit can move within a window bounded by the user-specified upper-bound, and a lower-bound that we derive internally. Users can expect this lower bound to be 1-2 orders of magnitude less than the provided upper-bound (so don't provide INT64_MAX as your upper-bound), although it's subject to change.
### Benchmark Results
Data is ingested at 10MB/s and the rate limiter was created with 1000MB/s as its upper bound. The dynamically chosen rate limit hovers around 125MB/s. The other clustering of points at 50MB/s is due to number of compaction threads being reduced to one when there's no compaction pressure.
![](/static/images/rate-limiter/write-KBps-series.png)
![](/static/images/rate-limiter/auto-tuned-write-KBps-series.png)
The following graph summarizes the above two time series graphs in CDF form. In particular, notice the p90 - p100 for background write rate are significantly lower with auto-tuned rate limiter enabled.
![](/static/images/rate-limiter/write-KBps-cdf.png)
@@ -1,37 +0,0 @@
---
title: WritePrepared Transactions
layout: post
author: maysamyabandeh
category: blog
---
RocksDB supports both optimistic and pessimistic concurrency controls. The pessimistic transactions make use of locks to provide isolation between the transactions. The default write policy in pessimistic transactions is _WriteCommitted_, which means that the data is written to the DB, i.e., the memtable, only after the transaction is committed. This policy simplified the implementation but came with some limitations in throughput, transaction size, and variety in supported isolation levels. In the below, we explain these in detail and present the other write policies, _WritePrepared_ and _WriteUnprepared_. We then dive into the design of _WritePrepared_ transactions.
> _WritePrepared_ are to be announced as production-ready soon.
### WriteCommitted, Pros and Cons
With _WriteCommitted_ write policy, the data is written to the memtable only after the transaction commits. This greatly simplifies the read path as any data that is read by other transactions can be assumed to be committed. This write policy, however, implies that the writes are buffered in memory in the meanwhile. This makes memory a bottleneck for large transactions. The delay of the commit phase in 2PC (two-phase commit) also becomes noticeable since most of the work, i.e., writing to memtable, is done at the commit phase. When the commit of multiple transactions are done in a serial fashion, such as in 2PC implementation of MySQL, the lengthy commit latency becomes a major contributor to lower throughput. Moreover this write policy cannot provide weaker isolation levels, such as READ UNCOMMITTED, that could potentially provide higher throughput for some applications.
### Alternatives: _WritePrepared_ and _WriteUnprepared_
To tackle the lengthy commit issue, we should do memtable writes at earlier phases of 2PC so that the commit phase become lightweight and fast. 2PC is composed of Write stage, where the transaction `::Put` is invoked, the prepare phase, where `::Prepare` is invoked (upon which the DB promises to commit the transaction if later is requested), and commit phase, where `::Commit` is invoked and the transaction writes become visible to all readers. To make the commit phase lightweight, the memtable write could be done at either `::Prepare` or `::Put` stages, resulting into _WritePrepared_ and _WriteUnprepared_ write policies respectively. The downside is that when another transaction is reading data, it would need a way to tell apart which data is committed, and if they are, whether they are committed before the transaction's start, i.e., in the read snapshot of the transaction. _WritePrepared_ would still have the issue of buffering the data, which makes the memory the bottleneck for large transactions. It however provides a good milestone for transitioning from _WriteCommitted_ to _WriteUnprepared_ write policy. Here we explain the design of _WritePrepared_ policy. We will cover the changes that make the design to also supported _WriteUnprepared_ in an upcoming post.
### _WritePrepared_ in a nutshell
These are the primary design questions that needs to be addressed:
1) How do we identify the key/values in the DB with transactions that wrote them?
2) How do we figure if a key/value written by transaction Txn_w is in the read snapshot of the reading transaction Txn_r?
3) How do we rollback the data written by aborted transactions?
With _WritePrepared_, a transaction still buffers the writes in a write batch object in memory. When 2PC `::Prepare` is called, it writes the in-memory write batch to the WAL (write-ahead log) as well as to the memtable(s) (one memtable per column family); We reuse the existing notion of sequence numbers in RocksDB to tag all the key/values in the same write batch with the same sequence number, `prepare_seq`, which is also used as the identifier for the transaction. At commit time, it writes a commit marker to the WAL, whose sequence number, `commit_seq`, will be used as the commit timestamp of the transaction. Before releasing the commit sequence number to the readers, it stores a mapping from `prepare_seq` to `commit_seq` in an in-memory data structure that we call _CommitCache_. When a transaction reading values from the DB (tagged with `prepare_seq`) it makes use of the _CommitCache_ to figure if `commit_seq` of the value is in its read snapshot. To rollback an aborted transaction, we apply the status before the transaction by making another write that cancels out the writes of the aborted transaction.
The _CommitCache_ is a lock-free data structure that caches the recent commit entries. Looking up the entries in the cache must be enough for almost all th transactions that commit in a timely manner. When evicting the older entries from the cache, it still maintains some other data structures to cover the corner cases for transactions that takes abnormally too long to finish. We will cover them in the design details below.
### Preliminary Results
The full experimental results are to be reported soon. Here we present the improvement in tps observed in some preliminary experiments with MyRocks:
* sysbench update-noindex: 25%
* sysbench read-write: 7.6%
* linkbench: 3.7%
Learn more [here](https://github.com/facebook/rocksdb/wiki/WritePrepared-Transactions).
Binary file not shown.

Before

Width:  |  Height:  |  Size: 172 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 303 KiB

-1
View File
@@ -1,4 +1,3 @@
# shellcheck disable=SC2148
export USE_HDFS=1
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:$JAVA_HOME/jre/lib/amd64:/usr/lib/hadoop/lib/native
-3
View File
@@ -710,9 +710,6 @@ extern ROCKSDB_LIBRARY_API void rocksdb_options_optimize_level_style_compaction(
extern ROCKSDB_LIBRARY_API void
rocksdb_options_optimize_universal_style_compaction(
rocksdb_options_t* opt, uint64_t memtable_memory_budget);
extern ROCKSDB_LIBRARY_API void
rocksdb_options_set_allow_ingest_behind(rocksdb_options_t*,
unsigned char);
extern ROCKSDB_LIBRARY_API void rocksdb_options_set_compaction_filter(
rocksdb_options_t*, rocksdb_compactionfilter_t*);
extern ROCKSDB_LIBRARY_API void rocksdb_options_set_compaction_filter_factory(
+4
View File
@@ -94,6 +94,10 @@ class CompactionFilter {
// be used by a single thread that is doing the compaction run, and this
// call does not need to be thread-safe. However, multiple filters may be
// in existence and operating concurrently.
//
// The last paragraph is not true if you set max_subcompactions to more than
// 1. In that case, subcompaction from multiple threads may call a single
// CompactionFilter concurrently.
virtual bool Filter(int level, const Slice& key, const Slice& existing_value,
std::string* new_value, bool* value_changed) const {
return false;
-3
View File
@@ -97,9 +97,6 @@ class Iterator : public Cleanable {
// Property "rocksdb.iterator.super-version-number":
// LSM version used by the iterator. The same format as DB Property
// kCurrentSuperVersionNumber. See its comment for more information.
// Property "rocksdb.iterator.internal-key":
// Get the user-key portion of the internal key at which the iteration
// stopped.
virtual Status GetProperty(std::string prop_name, std::string* prop);
private:
+1 -4
View File
@@ -415,7 +415,7 @@ const std::vector<std::pair<Tickers, std::string>> TickersNameMap = {
{BLOB_DB_WRITE_BLOB, "rocksdb.blobdb.write.blob"},
{BLOB_DB_WRITE_BLOB_TTL, "rocksdb.blobdb.write.blob.ttl"},
{BLOB_DB_BLOB_FILE_BYTES_WRITTEN, "rocksdb.blobdb.blob.file.bytes.written"},
{BLOB_DB_BLOB_FILE_BYTES_READ, "rocksdb.blobdb.blob.file,bytes.read"},
{BLOB_DB_BLOB_FILE_BYTES_READ, "rocksdb.blobdb.blob.file.bytes.read"},
{BLOB_DB_BLOB_FILE_SYNCED, "rocksdb.blobdb.blob.file.synced"},
{BLOB_DB_BLOB_INDEX_EXPIRED, "rocksdb.blobdb.blob.index.expired"},
{BLOB_DB_GC_NUM_FILES, "rocksdb.blobdb.gc.num.files"},
@@ -510,8 +510,6 @@ enum Histograms : uint32_t {
BLOB_DB_COMPRESSION_MICROS,
// BlobDB decompression time.
BLOB_DB_DECOMPRESSION_MICROS,
// Time spent flushing memtable to disk
FLUSH_TIME,
HISTOGRAM_ENUM_MAX, // TODO(ldemailly): enforce HistogramsNameMap match
};
@@ -562,7 +560,6 @@ const std::vector<std::pair<Histograms, std::string>> HistogramsNameMap = {
{BLOB_DB_GC_MICROS, "rocksdb.blobdb.gc.micros"},
{BLOB_DB_COMPRESSION_MICROS, "rocksdb.blobdb.compression.micros"},
{BLOB_DB_DECOMPRESSION_MICROS, "rocksdb.blobdb.decompression.micros"},
{FLUSH_TIME, "rocksdb.db.flush.micros"},
};
struct HistogramData {
+1 -2
View File
@@ -45,7 +45,6 @@ struct ThreadStatus {
HIGH_PRIORITY = 0, // RocksDB BG thread in high-pri thread pool
LOW_PRIORITY, // RocksDB BG thread in low-pri thread pool
USER, // User thread (Non-RocksDB BG thread)
BOTTOM_PRIORITY, // RocksDB BG thread in bottom-pri thread pool
NUM_THREAD_TYPES
};
@@ -164,7 +163,7 @@ struct ThreadStatus {
// The followings are a set of utility functions for interpreting
// the information of ThreadStatus
static std::string GetThreadTypeName(ThreadType thread_type);
static const std::string& GetThreadTypeName(ThreadType thread_type);
// Obtain the name of an operation given its type.
static const std::string& GetOperationName(OperationType op_type);
+2 -2
View File
@@ -5,8 +5,8 @@
#pragma once
#define ROCKSDB_MAJOR 5
#define ROCKSDB_MINOR 11
#define ROCKSDB_PATCH 4
#define ROCKSDB_MINOR 10
#define ROCKSDB_PATCH 0
// Do not use these. We made the mistake of declaring macros starting with
// double underscore. Now we have to live with our choice. We'll deprecate these
-15
View File
@@ -7,8 +7,6 @@ set(JNI_NATIVE_SOURCES
rocksjni/clock_cache.cc
rocksjni/columnfamilyhandle.cc
rocksjni/compaction_filter.cc
rocksjni/compaction_filter_factory.cc
rocksjni/compaction_filter_factory_jnicallback.cc
rocksjni/compaction_options_fifo.cc
rocksjni/compaction_options_universal.cc
rocksjni/comparator.cc
@@ -19,17 +17,14 @@ set(JNI_NATIVE_SOURCES
rocksjni/filter.cc
rocksjni/ingest_external_file_options.cc
rocksjni/iterator.cc
rocksjni/jnicallback.cc
rocksjni/loggerjnicallback.cc
rocksjni/lru_cache.cc
rocksjni/memtablejni.cc
rocksjni/merge_operator.cc
rocksjni/native_comparator_wrapper_test.cc
rocksjni/options.cc
rocksjni/options_util.cc
rocksjni/ratelimiterjni.cc
rocksjni/remove_emptyvalue_compactionfilterjni.cc
rocksjni/rocks_callback_object.cc
rocksjni/cassandra_compactionfilterjni.cc
rocksjni/cassandra_value_operator.cc
rocksjni/restorejni.cc
@@ -88,8 +83,6 @@ set(NATIVE_JAVA_CLASSES
org.rocksdb.LRUCache
org.rocksdb.MemTableConfig
org.rocksdb.MergeOperator
org.rocksdb.NativeComparatorWrapper
org.rocksdb.NativeComparatorWrapperTest.NativeStringComparatorWrapper
org.rocksdb.NativeLibraryLoader
org.rocksdb.Options
org.rocksdb.OptionsUtil
@@ -125,8 +118,6 @@ set(NATIVE_JAVA_CLASSES
org.rocksdb.WriteBatchTestInternalHelper
org.rocksdb.WriteBatchWithIndex
org.rocksdb.WriteOptions
org.rocksdb.util.CapturingWriteBatchHandler
org.rocksdb.util.WriteBatchGetter
)
include(FindJava)
@@ -182,7 +173,6 @@ add_jar(
src/main/java/org/rocksdb/CompactionStyle.java
src/main/java/org/rocksdb/Comparator.java
src/main/java/org/rocksdb/ComparatorOptions.java
src/main/java/org/rocksdb/ComparatorType.java
src/main/java/org/rocksdb/CompressionOptions.java
src/main/java/org/rocksdb/CompressionType.java
src/main/java/org/rocksdb/DBOptions.java
@@ -209,13 +199,11 @@ add_jar(
src/main/java/org/rocksdb/MergeOperator.java
src/main/java/org/rocksdb/MutableColumnFamilyOptions.java
src/main/java/org/rocksdb/MutableColumnFamilyOptionsInterface.java
src/main/java/org/rocksdb/NativeComparatorWrapper.java
src/main/java/org/rocksdb/NativeLibraryLoader.java
src/main/java/org/rocksdb/Options.java
src/main/java/org/rocksdb/OptionsUtil.java
src/main/java/org/rocksdb/PlainTableConfig.java
src/main/java/org/rocksdb/RateLimiter.java
src/main/java/org/rocksdb/RateLimiterMode.java
src/main/java/org/rocksdb/ReadOptions.java
src/main/java/org/rocksdb/ReadTier.java
src/main/java/org/rocksdb/RemoveEmptyValueCompactionFilter.java
@@ -251,14 +239,11 @@ add_jar(
src/main/java/org/rocksdb/WriteOptions.java
src/test/java/org/rocksdb/BackupEngineTest.java
src/test/java/org/rocksdb/IngestExternalFileOptionsTest.java
src/test/java/org/rocksdb/NativeComparatorWrapperTest.java
src/test/java/org/rocksdb/PlatformRandomHelper.java
src/test/java/org/rocksdb/RocksDBExceptionTest.java
src/test/java/org/rocksdb/RocksMemoryResource.java
src/test/java/org/rocksdb/SnapshotTest.java
src/test/java/org/rocksdb/WriteBatchTest.java
src/test/java/org/rocksdb/util/CapturingWriteBatchHandler.java
src/test/java/org/rocksdb/util/WriteBatchGetter.java
INCLUDE_JARS ${JAVA_TESTCLASSPATH}
)
-31
View File
@@ -1,7 +1,6 @@
NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
org.rocksdb.AbstractCompactionFilterFactory\
org.rocksdb.AbstractSlice\
org.rocksdb.AbstractTransactionNotifier\
org.rocksdb.BackupEngine\
org.rocksdb.BackupableDBOptions\
org.rocksdb.BlockBasedTableConfig\
@@ -30,9 +29,6 @@ NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
org.rocksdb.Logger\
org.rocksdb.LRUCache\
org.rocksdb.MergeOperator\
org.rocksdb.NativeComparatorWrapper\
org.rocksdb.OptimisticTransactionDB\
org.rocksdb.OptimisticTransactionOptions\
org.rocksdb.Options\
org.rocksdb.OptionsUtil\
org.rocksdb.PlainTableConfig\
@@ -49,10 +45,6 @@ NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
org.rocksdb.Slice\
org.rocksdb.SstFileWriter\
org.rocksdb.Statistics\
org.rocksdb.Transaction\
org.rocksdb.TransactionDB\
org.rocksdb.TransactionDBOptions\
org.rocksdb.TransactionOptions\
org.rocksdb.TransactionLogIterator\
org.rocksdb.TtlDB\
org.rocksdb.VectorMemTableConfig\
@@ -65,7 +57,6 @@ NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
org.rocksdb.WBWIRocksIterator
NATIVE_JAVA_TEST_CLASSES = org.rocksdb.RocksDBExceptionTest\
org.rocksdb.NativeComparatorWrapperTest.NativeStringComparatorWrapper\
org.rocksdb.WriteBatchTest\
org.rocksdb.WriteBatchTestInternalHelper
@@ -113,11 +104,7 @@ JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.MergeTest\
org.rocksdb.MixedOptionsTest\
org.rocksdb.MutableColumnFamilyOptionsTest\
org.rocksdb.NativeComparatorWrapperTest\
org.rocksdb.NativeLibraryLoaderTest\
org.rocksdb.OptimisticTransactionTest\
org.rocksdb.OptimisticTransactionDBTest\
org.rocksdb.OptimisticTransactionOptionsTest\
org.rocksdb.OptionsUtilTest\
org.rocksdb.OptionsTest\
org.rocksdb.PlainTableConfigTest\
@@ -133,10 +120,6 @@ JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.SliceTest\
org.rocksdb.SnapshotTest\
org.rocksdb.SstFileWriterTest\
org.rocksdb.TransactionTest\
org.rocksdb.TransactionDBTest\
org.rocksdb.TransactionOptionsTest\
org.rocksdb.TransactionDBOptionsTest\
org.rocksdb.TransactionLogIteratorTest\
org.rocksdb.TtlDBTest\
org.rocksdb.StatisticsTest\
@@ -226,20 +209,6 @@ column_family_sample: java
java $(JAVA_ARGS) -Djava.library.path=target -cp $(MAIN_CLASSES):$(SAMPLES_MAIN_CLASSES) RocksDBColumnFamilySample /tmp/rocksdbjni
$(AM_V_at)@rm -rf /tmp/rocksdbjni
transaction_sample: java
$(AM_V_GEN)mkdir -p $(SAMPLES_MAIN_CLASSES)
$(AM_V_at)javac -cp $(MAIN_CLASSES) -d $(SAMPLES_MAIN_CLASSES) $(SAMPLES_MAIN_SRC)/TransactionSample.java
$(AM_V_at)@rm -rf /tmp/rocksdbjni
java -ea -Xcheck:jni -Djava.library.path=target -cp $(MAIN_CLASSES):$(SAMPLES_MAIN_CLASSES) TransactionSample /tmp/rocksdbjni
$(AM_V_at)@rm -rf /tmp/rocksdbjni
optimistic_transaction_sample: java
$(AM_V_GEN)mkdir -p $(SAMPLES_MAIN_CLASSES)
$(AM_V_at)javac -cp $(MAIN_CLASSES) -d $(SAMPLES_MAIN_CLASSES) $(SAMPLES_MAIN_SRC)/OptimisticTransactionSample.java
$(AM_V_at)@rm -rf /tmp/rocksdbjni
java -ea -Xcheck:jni -Djava.library.path=target -cp $(MAIN_CLASSES):$(SAMPLES_MAIN_CLASSES) OptimisticTransactionSample /tmp/rocksdbjni
$(AM_V_at)@rm -rf /tmp/rocksdbjni
resolve_test_deps:
test -d "$(JAVA_TEST_LIBDIR)" || mkdir -p "$(JAVA_TEST_LIBDIR)"
test -s "$(JAVA_JUNIT_JAR)" || cp $(MVN_LOCAL)/junit/junit/4.12/junit-4.12.jar $(JAVA_TEST_LIBDIR) || curl -k -L -o $(JAVA_JUNIT_JAR) $(SEARCH_REPO_URL)junit/junit/4.12/junit-4.12.jar
-2
View File
@@ -1,4 +1,3 @@
# shellcheck disable=SC2148
PLATFORM=64
if [ `getconf LONG_BIT` != "64" ]
then
@@ -8,5 +7,4 @@ fi
ROCKS_JAR=`find target -name rocksdbjni*.jar`
echo "Running benchmark in $PLATFORM-Bit mode."
# shellcheck disable=SC2068
java -server -d$PLATFORM -XX:NewSize=4m -XX:+AggressiveOpts -Djava.library.path=target -cp "${ROCKS_JAR}:benchmark/target/classes" org.rocksdb.benchmark.DbBenchmark $@
+4 -46
View File
@@ -3,8 +3,8 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the "bridge" between Java and C++ for
// rocksdb::ColumnFamilyHandle.
// This file implements the "bridge" between Java and C++ and enables
// calling c++ rocksdb::Iterator methods from Java side.
#include <stdio.h>
#include <stdlib.h>
@@ -13,56 +13,14 @@
#include "include/org_rocksdb_ColumnFamilyHandle.h"
#include "rocksjni/portal.h"
/*
* Class: org_rocksdb_ColumnFamilyHandle
* Method: getName
* Signature: (J)[B
*/
jbyteArray Java_org_rocksdb_ColumnFamilyHandle_getName(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* cfh = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jhandle);
std::string cf_name = cfh->GetName();
return rocksdb::JniUtil::copyBytes(env, cf_name);
}
/*
* Class: org_rocksdb_ColumnFamilyHandle
* Method: getID
* Signature: (J)I
*/
jint Java_org_rocksdb_ColumnFamilyHandle_getID(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* cfh = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jhandle);
const int32_t id = cfh->GetID();
return static_cast<jint>(id);
}
/*
* Class: org_rocksdb_ColumnFamilyHandle
* Method: getDescriptor
* Signature: (J)Lorg/rocksdb/ColumnFamilyDescriptor;
*/
jobject Java_org_rocksdb_ColumnFamilyHandle_getDescriptor(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* cfh = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jhandle);
rocksdb::ColumnFamilyDescriptor desc;
rocksdb::Status s = cfh->GetDescriptor(&desc);
if (s.ok()) {
return rocksdb::ColumnFamilyDescriptorJni::construct(env, &desc);
} else {
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
return nullptr;
}
}
/*
* Class: org_rocksdb_ColumnFamilyHandle
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_ColumnFamilyHandle_disposeInternal(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* cfh = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jhandle);
JNIEnv* env, jobject jobj, jlong handle) {
auto* cfh = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(handle);
assert(cfh != nullptr);
delete cfh;
}
@@ -34,5 +34,4 @@ void Java_org_rocksdb_AbstractCompactionFilterFactory_disposeInternal(
auto* ptr_sptr_cff =
reinterpret_cast<std::shared_ptr<rocksdb::CompactionFilterFactoryJniCallback> *>(jhandle);
delete ptr_sptr_cff;
// @lint-ignore TXT4 T25377293 Grandfathered in
}
-13
View File
@@ -14,7 +14,6 @@
#include "include/org_rocksdb_Comparator.h"
#include "include/org_rocksdb_DirectComparator.h"
#include "include/org_rocksdb_NativeComparatorWrapper.h"
#include "rocksjni/comparatorjnicallback.h"
#include "rocksjni/portal.h"
@@ -50,16 +49,4 @@ jlong Java_org_rocksdb_DirectComparator_createNewDirectComparator0(
new rocksdb::DirectComparatorJniCallback(env, jobj, copt);
return reinterpret_cast<jlong>(c);
}
/*
* Class: org_rocksdb_NativeComparatorWrapper
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_NativeComparatorWrapper_disposeInternal(
JNIEnv* env, jobject jobj, jlong jcomparator_handle) {
auto* comparator =
reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
delete comparator;
}
// </editor-fold>
@@ -146,5 +146,4 @@ void Java_org_rocksdb_IngestExternalFileOptions_disposeInternal(
auto* options =
reinterpret_cast<rocksdb::IngestExternalFileOptions*>(jhandle);
delete options;
// @lint-ignore TXT4 T25377293 Grandfathered in
}
-1
View File
@@ -49,5 +49,4 @@ JniCallback::~JniCallback() {
releaseJniEnv(attached_thread);
}
// @lint-ignore TXT4 T25377293 Grandfathered in
} // namespace rocksdb
-1
View File
@@ -25,5 +25,4 @@ namespace rocksdb {
};
}
// @lint-ignore TXT4 T25377293 Grandfathered in
#endif // JAVA_ROCKSJNI_JNICALLBACK_H_
@@ -1,50 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#include <jni.h>
#include <string>
#include "rocksdb/comparator.h"
#include "rocksdb/slice.h"
#include "include/org_rocksdb_NativeComparatorWrapperTest_NativeStringComparatorWrapper.h"
namespace rocksdb {
class NativeComparatorWrapperTestStringComparator
: public Comparator {
const char* Name() const {
return "NativeComparatorWrapperTestStringComparator";
}
int Compare(
const Slice& a, const Slice& b) const {
return a.ToString().compare(b.ToString());
}
void FindShortestSeparator(
std::string* start, const Slice& limit) const {
return;
}
void FindShortSuccessor(
std::string* key) const {
return;
}
};
} // end of rocksdb namespace
/*
* Class: org_rocksdb_NativeComparatorWrapperTest_NativeStringComparatorWrapper
* Method: newStringComparator
* Signature: ()J
*/
jlong Java_org_rocksdb_NativeComparatorWrapperTest_00024NativeStringComparatorWrapper_newStringComparator(
JNIEnv* env , jobject jobj) {
auto* comparator =
new rocksdb::NativeComparatorWrapperTestStringComparator();
return reinterpret_cast<jlong>(comparator);
}
-267
View File
@@ -1,267 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the "bridge" between Java and C++
// for rocksdb::TransactionDB.
#include <jni.h>
#include "include/org_rocksdb_OptimisticTransactionDB.h"
#include "rocksdb/options.h"
#include "rocksdb/utilities/optimistic_transaction_db.h"
#include "rocksdb/utilities/transaction.h"
#include "rocksjni/portal.h"
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: open
* Signature: (JLjava/lang/String;)J
*/
jlong Java_org_rocksdb_OptimisticTransactionDB_open__JLjava_lang_String_2(
JNIEnv* env, jclass jcls, jlong joptions_handle, jstring jdb_path) {
const char* db_path = env->GetStringUTFChars(jdb_path, nullptr);
if (db_path == nullptr) {
// exception thrown: OutOfMemoryError
return 0;
}
auto* options = reinterpret_cast<rocksdb::Options*>(joptions_handle);
rocksdb::OptimisticTransactionDB* otdb = nullptr;
rocksdb::Status s =
rocksdb::OptimisticTransactionDB::Open(*options, db_path, &otdb);
env->ReleaseStringUTFChars(jdb_path, db_path);
if (s.ok()) {
return reinterpret_cast<jlong>(otdb);
} else {
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
return 0;
}
}
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: open
* Signature: (JLjava/lang/String;[[B[J)[J
*/
jlongArray Java_org_rocksdb_OptimisticTransactionDB_open__JLjava_lang_String_2_3_3B_3J(
JNIEnv* env, jclass jcls, jlong jdb_options_handle, jstring jdb_path,
jobjectArray jcolumn_names, jlongArray jcolumn_options_handles) {
const char* db_path = env->GetStringUTFChars(jdb_path, nullptr);
if (db_path == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
std::vector<rocksdb::ColumnFamilyDescriptor> column_families;
const jsize len_cols = env->GetArrayLength(jcolumn_names);
if (len_cols > 0) {
if (env->EnsureLocalCapacity(len_cols) != 0) {
// out of memory
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
jlong* jco =
env->GetLongArrayElements(jcolumn_options_handles, nullptr);
if(jco == nullptr) {
// exception thrown: OutOfMemoryError
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
for (int i = 0; i < len_cols; i++) {
const jobject jcn = env->GetObjectArrayElement(jcolumn_names, i);
if (env->ExceptionCheck()) {
// exception thrown: ArrayIndexOutOfBoundsException
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
const jbyteArray jcn_ba = reinterpret_cast<jbyteArray>(jcn);
const jsize jcf_name_len = env->GetArrayLength(jcn_ba);
if (env->EnsureLocalCapacity(jcf_name_len) != 0) {
// out of memory
env->DeleteLocalRef(jcn);
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
jbyte* jcf_name = env->GetByteArrayElements(jcn_ba, nullptr);
if (jcf_name == nullptr) {
// exception thrown: OutOfMemoryError
env->DeleteLocalRef(jcn);
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
const std::string cf_name(reinterpret_cast<char *>(jcf_name), jcf_name_len);
const rocksdb::ColumnFamilyOptions* cf_options =
reinterpret_cast<rocksdb::ColumnFamilyOptions*>(jco[i]);
column_families.push_back(
rocksdb::ColumnFamilyDescriptor(cf_name, *cf_options));
env->ReleaseByteArrayElements(jcn_ba, jcf_name, JNI_ABORT);
env->DeleteLocalRef(jcn);
}
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
}
auto* db_options = reinterpret_cast<rocksdb::DBOptions*>(jdb_options_handle);
std::vector<rocksdb::ColumnFamilyHandle*> handles;
rocksdb::OptimisticTransactionDB* otdb = nullptr;
const rocksdb::Status s = rocksdb::OptimisticTransactionDB::Open(*db_options,
db_path, column_families, &handles, &otdb);
env->ReleaseStringUTFChars(jdb_path, db_path);
// check if open operation was successful
if (s.ok()) {
const jsize resultsLen = 1 + len_cols; // db handle + column family handles
std::unique_ptr<jlong[]> results =
std::unique_ptr<jlong[]>(new jlong[resultsLen]);
results[0] = reinterpret_cast<jlong>(otdb);
for (int i = 1; i <= len_cols; i++) {
results[i] = reinterpret_cast<jlong>(handles[i - 1]);
}
jlongArray jresults = env->NewLongArray(resultsLen);
if (jresults == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
env->SetLongArrayRegion(jresults, 0, resultsLen, results.get());
if (env->ExceptionCheck()) {
// exception thrown: ArrayIndexOutOfBoundsException
return nullptr;
}
return jresults;
}
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
return nullptr;
}
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: beginTransaction
* Signature: (JJ)J
*/
jlong Java_org_rocksdb_OptimisticTransactionDB_beginTransaction__JJ(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jwrite_options_handle) {
auto* optimistic_txn_db =
reinterpret_cast<rocksdb::OptimisticTransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
rocksdb::Transaction* txn =
optimistic_txn_db->BeginTransaction(*write_options);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: beginTransaction
* Signature: (JJJ)J
*/
jlong Java_org_rocksdb_OptimisticTransactionDB_beginTransaction__JJJ(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jwrite_options_handle,
jlong joptimistic_txn_options_handle) {
auto* optimistic_txn_db =
reinterpret_cast<rocksdb::OptimisticTransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
auto* optimistic_txn_options =
reinterpret_cast<rocksdb::OptimisticTransactionOptions*>(
joptimistic_txn_options_handle);
rocksdb::Transaction* txn =
optimistic_txn_db->BeginTransaction(*write_options,
*optimistic_txn_options);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: beginTransaction_withOld
* Signature: (JJJ)J
*/
jlong Java_org_rocksdb_OptimisticTransactionDB_beginTransaction_1withOld__JJJ(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jwrite_options_handle,
jlong jold_txn_handle) {
auto* optimistic_txn_db =
reinterpret_cast<rocksdb::OptimisticTransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
auto* old_txn =
reinterpret_cast<rocksdb::Transaction*>(
jold_txn_handle);
rocksdb::OptimisticTransactionOptions optimistic_txn_options;
rocksdb::Transaction* txn =
optimistic_txn_db->BeginTransaction(*write_options,
optimistic_txn_options, old_txn);
// RocksJava relies on the assumption that
// we do not allocate a new Transaction object
// when providing an old_optimistic_txn
assert(txn == old_txn);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: beginTransaction_withOld
* Signature: (JJJJ)J
*/
jlong Java_org_rocksdb_OptimisticTransactionDB_beginTransaction_1withOld__JJJJ(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jwrite_options_handle,
jlong joptimistic_txn_options_handle, jlong jold_txn_handle) {
auto* optimistic_txn_db =
reinterpret_cast<rocksdb::OptimisticTransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
auto* optimistic_txn_options =
reinterpret_cast<rocksdb::OptimisticTransactionOptions*>(
joptimistic_txn_options_handle);
auto* old_txn =
reinterpret_cast<rocksdb::Transaction*>(
jold_txn_handle);
rocksdb::Transaction* txn =
optimistic_txn_db->BeginTransaction(*write_options,
*optimistic_txn_options, old_txn);
// RocksJava relies on the assumption that
// we do not allocate a new Transaction object
// when providing an old_optimisic_txn
assert(txn == old_txn);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: getBaseDB
* Signature: (J)J
*/
jlong Java_org_rocksdb_OptimisticTransactionDB_getBaseDB(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* optimistic_txn_db =
reinterpret_cast<rocksdb::OptimisticTransactionDB*>(jhandle);
return reinterpret_cast<jlong>(optimistic_txn_db->GetBaseDB());
}
/*
* Class: org_rocksdb_OptimisticTransactionDB
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_OptimisticTransactionDB_disposeInternal(JNIEnv* env,
jobject jobj, jlong jhandle) {
delete reinterpret_cast<rocksdb::OptimisticTransactionDB*>(jhandle);
}
@@ -1,72 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the "bridge" between Java and C++
// for rocksdb::OptimisticTransactionOptions.
#include <jni.h>
#include "include/org_rocksdb_OptimisticTransactionOptions.h"
#include "rocksdb/comparator.h"
#include "rocksdb/utilities/optimistic_transaction_db.h"
/*
* Class: org_rocksdb_OptimisticTransactionOptions
* Method: newOptimisticTransactionOptions
* Signature: ()J
*/
jlong Java_org_rocksdb_OptimisticTransactionOptions_newOptimisticTransactionOptions(
JNIEnv* env, jclass jcls) {
rocksdb::OptimisticTransactionOptions* opts =
new rocksdb::OptimisticTransactionOptions();
return reinterpret_cast<jlong>(opts);
}
/*
* Class: org_rocksdb_OptimisticTransactionOptions
* Method: isSetSnapshot
* Signature: (J)Z
*/
jboolean Java_org_rocksdb_OptimisticTransactionOptions_isSetSnapshot(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* opts =
reinterpret_cast<rocksdb::OptimisticTransactionOptions*>(jhandle);
return opts->set_snapshot;
}
/*
* Class: org_rocksdb_OptimisticTransactionOptions
* Method: setSetSnapshot
* Signature: (JZ)V
*/
void Java_org_rocksdb_OptimisticTransactionOptions_setSetSnapshot(JNIEnv* env,
jobject jobj, jlong jhandle, jboolean jset_snapshot) {
auto* opts =
reinterpret_cast<rocksdb::OptimisticTransactionOptions*>(jhandle);
opts->set_snapshot = jset_snapshot;
}
/*
* Class: org_rocksdb_OptimisticTransactionOptions
* Method: setComparator
* Signature: (JJ)V
*/
void Java_org_rocksdb_OptimisticTransactionOptions_setComparator(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jcomparator_handle) {
auto* opts =
reinterpret_cast<rocksdb::OptimisticTransactionOptions*>(jhandle);
opts->cmp = reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
}
/*
* Class: org_rocksdb_OptimisticTransactionOptions
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_OptimisticTransactionOptions_disposeInternal(JNIEnv* env,
jobject jobj, jlong jhandle) {
delete reinterpret_cast<rocksdb::OptimisticTransactionOptions*>(jhandle);
}
+20 -48
View File
@@ -146,33 +146,19 @@ void Java_org_rocksdb_Options_setComparatorHandle__JI(
/*
* Class: org_rocksdb_Options
* Method: setComparatorHandle
* Signature: (JJB)V
* Signature: (JJZ)V
*/
void Java_org_rocksdb_Options_setComparatorHandle__JJB(
void Java_org_rocksdb_Options_setComparatorHandle__JJZ(
JNIEnv* env, jobject jobj, jlong jopt_handle, jlong jcomparator_handle,
jbyte jcomparator_type) {
rocksdb::Comparator *comparator = nullptr;
switch(jcomparator_type) {
// JAVA_COMPARATOR
case 0x0:
comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
comparator =
reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
break;
}
jboolean is_direct) {
auto* opt = reinterpret_cast<rocksdb::Options*>(jopt_handle);
opt->comparator = comparator;
if(is_direct) {
opt->comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jcomparator_handle);
} else {
opt->comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
}
}
/*
@@ -2974,33 +2960,19 @@ void Java_org_rocksdb_ColumnFamilyOptions_setComparatorHandle__JI(
/*
* Class: org_rocksdb_ColumnFamilyOptions
* Method: setComparatorHandle
* Signature: (JJB)V
* Signature: (JJZ)V
*/
void Java_org_rocksdb_ColumnFamilyOptions_setComparatorHandle__JJB(
void Java_org_rocksdb_ColumnFamilyOptions_setComparatorHandle__JJZ(
JNIEnv* env, jobject jobj, jlong jopt_handle, jlong jcomparator_handle,
jbyte jcomparator_type) {
rocksdb::Comparator *comparator = nullptr;
switch(jcomparator_type) {
// JAVA_COMPARATOR
case 0x0:
comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
comparator =
reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
break;
}
jboolean is_direct) {
auto* opt = reinterpret_cast<rocksdb::ColumnFamilyOptions*>(jopt_handle);
opt->comparator = comparator;
if(is_direct) {
opt->comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jcomparator_handle);
} else {
opt->comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
}
}
/*
+66 -1424
View File
File diff suppressed because it is too large. Load diff
+3 -19
View File
@@ -12,21 +12,16 @@
/*
* Class: org_rocksdb_RateLimiter
* Method: newRateLimiterHandle
* Signature: (JJIBZ)J
* Signature: (JJI)J
*/
jlong Java_org_rocksdb_RateLimiter_newRateLimiterHandle(
JNIEnv* env, jclass jclazz, jlong jrate_bytes_per_second,
jlong jrefill_period_micros, jint jfairness, jbyte jrate_limiter_mode,
jboolean jauto_tune) {
auto rate_limiter_mode = rocksdb::RateLimiterModeJni::toCppRateLimiterMode(
jrate_limiter_mode);
jlong jrefill_period_micros, jint jfairness) {
auto * sptr_rate_limiter =
new std::shared_ptr<rocksdb::RateLimiter>(rocksdb::NewGenericRateLimiter(
static_cast<int64_t>(jrate_bytes_per_second),
static_cast<int64_t>(jrefill_period_micros),
static_cast<int32_t>(jfairness),
rate_limiter_mode,
jauto_tune));
static_cast<int32_t>(jfairness)));
return reinterpret_cast<jlong>(sptr_rate_limiter);
}
@@ -55,17 +50,6 @@ void Java_org_rocksdb_RateLimiter_setBytesPerSecond(
SetBytesPerSecond(jbytes_per_second);
}
/*
* Class: org_rocksdb_RateLimiter
* Method: getBytesPerSecond
* Signature: (J)J
*/
jlong Java_org_rocksdb_RateLimiter_getBytesPerSecond(
JNIEnv* env, jobject jobj, jlong handle) {
return reinterpret_cast<std::shared_ptr<rocksdb::RateLimiter> *>(handle)->get()->
GetBytesPerSecond();
}
/*
* Class: org_rocksdb_RateLimiter
* Method: request
-1
View File
@@ -24,5 +24,4 @@ void Java_org_rocksdb_RocksCallbackObject_disposeInternal(
// 2) Comparator -> BaseComparatorJniCallback + JniCallback -> ComparatorJniCallback
// I think this is okay, as Comparator and JniCallback both have virtual destructors...
delete reinterpret_cast<rocksdb::JniCallback*>(handle);
// @lint-ignore TXT4 T25377293 Grandfathered in
}
+14 -24
View File
@@ -20,34 +20,24 @@
/*
* Class: org_rocksdb_SstFileWriter
* Method: newSstFileWriter
* Signature: (JJJB)J
* Signature: (JJJZ)J
*/
jlong Java_org_rocksdb_SstFileWriter_newSstFileWriter__JJJB(JNIEnv *env,
jclass jcls, jlong jenvoptions, jlong joptions, jlong jcomparator_handle,
jbyte jcomparator_type) {
rocksdb::Comparator *comparator = nullptr;
switch(jcomparator_type) {
// JAVA_COMPARATOR
case 0x0:
comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
comparator =
reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
break;
}
jlong Java_org_rocksdb_SstFileWriter_newSstFileWriter__JJJZ(JNIEnv *env,
jclass jcls, jlong jenvoptions, jlong joptions, jlong jcomparator,
jboolean is_direct) {
auto *env_options =
reinterpret_cast<const rocksdb::EnvOptions *>(jenvoptions);
auto *options = reinterpret_cast<const rocksdb::Options *>(joptions);
rocksdb::Comparator *comparator = nullptr;
if(is_direct) {
comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jcomparator);
} else {
comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator);
}
rocksdb::SstFileWriter *sst_file_writer =
new rocksdb::SstFileWriter(*env_options, *options, comparator);
return reinterpret_cast<jlong>(sst_file_writer);
-1
View File
@@ -30,5 +30,4 @@ namespace rocksdb {
return true;
}
// @lint-ignore TXT4 T25377293 Grandfathered in
};
-1
View File
@@ -30,5 +30,4 @@ namespace rocksdb {
} // namespace rocksdb
// @lint-ignore TXT4 T25377293 Grandfathered in
#endif // JAVA_ROCKSJNI_STATISTICSJNI_H_
File diff suppressed because it is too large. Load diff
-431
View File
@@ -1,431 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the "bridge" between Java and C++
// for rocksdb::TransactionDB.
#include <jni.h>
#include <functional>
#include <memory>
#include <utility>
#include "include/org_rocksdb_TransactionDB.h"
#include "rocksdb/options.h"
#include "rocksdb/utilities/transaction.h"
#include "rocksdb/utilities/transaction_db.h"
#include "rocksjni/portal.h"
/*
* Class: org_rocksdb_TransactionDB
* Method: open
* Signature: (JJLjava/lang/String;)J
*/
jlong Java_org_rocksdb_TransactionDB_open__JJLjava_lang_String_2(JNIEnv* env,
jclass jcls, jlong joptions_handle, jlong jtxn_db_options_handle,
jstring jdb_path) {
auto* options = reinterpret_cast<rocksdb::Options*>(joptions_handle);
auto* txn_db_options =
reinterpret_cast<rocksdb::TransactionDBOptions*>(jtxn_db_options_handle);
rocksdb::TransactionDB* tdb = nullptr;
const char* db_path = env->GetStringUTFChars(jdb_path, nullptr);
if (db_path == nullptr) {
// exception thrown: OutOfMemoryError
return 0;
}
rocksdb::Status s =
rocksdb::TransactionDB::Open(*options, *txn_db_options, db_path, &tdb);
env->ReleaseStringUTFChars(jdb_path, db_path);
if (s.ok()) {
return reinterpret_cast<jlong>(tdb);
} else {
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
return 0;
}
}
/*
* Class: org_rocksdb_TransactionDB
* Method: open
* Signature: (JJLjava/lang/String;[[B[J)[J
*/
jlongArray Java_org_rocksdb_TransactionDB_open__JJLjava_lang_String_2_3_3B_3J(
JNIEnv* env, jclass jcls, jlong jdb_options_handle,
jlong jtxn_db_options_handle, jstring jdb_path,
jobjectArray jcolumn_names,
jlongArray jcolumn_options_handles) {
const char* db_path = env->GetStringUTFChars(jdb_path, nullptr);
if (db_path == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
const jsize len_cols = env->GetArrayLength(jcolumn_names);
if (env->EnsureLocalCapacity(len_cols) != 0) {
// out of memory
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
jlong* jco = env->GetLongArrayElements(jcolumn_options_handles, nullptr);
if (jco == nullptr) {
// exception thrown: OutOfMemoryError
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
std::vector<rocksdb::ColumnFamilyDescriptor> column_families;
for (int i = 0; i < len_cols; i++) {
const jobject jcn = env->GetObjectArrayElement(jcolumn_names, i);
if (env->ExceptionCheck()) {
// exception thrown: ArrayIndexOutOfBoundsException
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
const jbyteArray jcn_ba = reinterpret_cast<jbyteArray>(jcn);
jbyte* jcf_name = env->GetByteArrayElements(jcn_ba, nullptr);
if (jcf_name == nullptr) {
// exception thrown: OutOfMemoryError
env->DeleteLocalRef(jcn);
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
const int jcf_name_len = env->GetArrayLength(jcn_ba);
if (env->EnsureLocalCapacity(jcf_name_len) != 0) {
// out of memory
env->ReleaseByteArrayElements(jcn_ba, jcf_name, JNI_ABORT);
env->DeleteLocalRef(jcn);
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
env->ReleaseStringUTFChars(jdb_path, db_path);
return nullptr;
}
const std::string cf_name(reinterpret_cast<char *>(jcf_name), jcf_name_len);
const rocksdb::ColumnFamilyOptions* cf_options =
reinterpret_cast<rocksdb::ColumnFamilyOptions*>(jco[i]);
column_families.push_back(
rocksdb::ColumnFamilyDescriptor(cf_name, *cf_options));
env->ReleaseByteArrayElements(jcn_ba, jcf_name, JNI_ABORT);
env->DeleteLocalRef(jcn);
}
env->ReleaseLongArrayElements(jcolumn_options_handles, jco, JNI_ABORT);
auto* db_options = reinterpret_cast<rocksdb::DBOptions*>(jdb_options_handle);
auto* txn_db_options =
reinterpret_cast<rocksdb::TransactionDBOptions*>(jtxn_db_options_handle);
std::vector<rocksdb::ColumnFamilyHandle*> handles;
rocksdb::TransactionDB* tdb = nullptr;
const rocksdb::Status s = rocksdb::TransactionDB::Open(*db_options, *txn_db_options,
db_path, column_families, &handles, &tdb);
// check if open operation was successful
if (s.ok()) {
const jsize resultsLen = 1 + len_cols; // db handle + column family handles
std::unique_ptr<jlong[]> results =
std::unique_ptr<jlong[]>(new jlong[resultsLen]);
results[0] = reinterpret_cast<jlong>(tdb);
for (int i = 1; i <= len_cols; i++) {
results[i] = reinterpret_cast<jlong>(handles[i - 1]);
}
jlongArray jresults = env->NewLongArray(resultsLen);
if (jresults == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
env->SetLongArrayRegion(jresults, 0, resultsLen, results.get());
if (env->ExceptionCheck()) {
// exception thrown: ArrayIndexOutOfBoundsException
env->DeleteLocalRef(jresults);
return nullptr;
}
return jresults;
} else {
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
return nullptr;
}
}
/*
* Class: org_rocksdb_TransactionDB
* Method: beginTransaction
* Signature: (JJ)J
*/
jlong Java_org_rocksdb_TransactionDB_beginTransaction__JJ(JNIEnv* env,
jobject jobj, jlong jhandle, jlong jwrite_options_handle) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
rocksdb::Transaction* txn = txn_db->BeginTransaction(*write_options);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_TransactionDB
* Method: beginTransaction
* Signature: (JJJ)J
*/
jlong Java_org_rocksdb_TransactionDB_beginTransaction__JJJ(JNIEnv* env,
jobject jobj, jlong jhandle, jlong jwrite_options_handle,
jlong jtxn_options_handle) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
auto* txn_options =
reinterpret_cast<rocksdb::TransactionOptions*>(jtxn_options_handle);
rocksdb::Transaction* txn =
txn_db->BeginTransaction(*write_options, *txn_options);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_TransactionDB
* Method: beginTransaction_withOld
* Signature: (JJJ)J
*/
jlong Java_org_rocksdb_TransactionDB_beginTransaction_1withOld__JJJ(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jwrite_options_handle,
jlong jold_txn_handle) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
auto* old_txn = reinterpret_cast<rocksdb::Transaction*>(jold_txn_handle);
rocksdb::TransactionOptions txn_options;
rocksdb::Transaction* txn =
txn_db->BeginTransaction(*write_options, txn_options, old_txn);
// RocksJava relies on the assumption that
// we do not allocate a new Transaction object
// when providing an old_txn
assert(txn == old_txn);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_TransactionDB
* Method: beginTransaction_withOld
* Signature: (JJJJ)J
*/
jlong Java_org_rocksdb_TransactionDB_beginTransaction_1withOld__JJJJ(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jwrite_options_handle,
jlong jtxn_options_handle, jlong jold_txn_handle) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
auto* write_options =
reinterpret_cast<rocksdb::WriteOptions*>(jwrite_options_handle);
auto* txn_options =
reinterpret_cast<rocksdb::TransactionOptions*>(jtxn_options_handle);
auto* old_txn = reinterpret_cast<rocksdb::Transaction*>(jold_txn_handle);
rocksdb::Transaction* txn = txn_db->BeginTransaction(*write_options,
*txn_options, old_txn);
// RocksJava relies on the assumption that
// we do not allocate a new Transaction object
// when providing an old_txn
assert(txn == old_txn);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_TransactionDB
* Method: getTransactionByName
* Signature: (JLjava/lang/String;)J
*/
jlong Java_org_rocksdb_TransactionDB_getTransactionByName(JNIEnv* env,
jobject jobj, jlong jhandle, jstring jname) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
const char* name = env->GetStringUTFChars(jname, nullptr);
if (name == nullptr) {
// exception thrown: OutOfMemoryError
return 0;
}
rocksdb::Transaction* txn = txn_db->GetTransactionByName(name);
env->ReleaseStringUTFChars(jname, name);
return reinterpret_cast<jlong>(txn);
}
/*
* Class: org_rocksdb_TransactionDB
* Method: getAllPreparedTransactions
* Signature: (J)[J
*/
jlongArray Java_org_rocksdb_TransactionDB_getAllPreparedTransactions(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
std::vector<rocksdb::Transaction*> txns;
txn_db->GetAllPreparedTransactions(&txns);
const size_t size = txns.size();
assert(size < UINT32_MAX); // does it fit in a jint?
const jsize len = static_cast<jsize>(size);
jlong tmp[len];
for (jsize i = 0; i < len; ++i) {
tmp[i] = reinterpret_cast<jlong>(txns[i]);
}
jlongArray jtxns = env->NewLongArray(len);
if (jtxns == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
env->SetLongArrayRegion(jtxns, 0, len, tmp);
if (env->ExceptionCheck()) {
// exception thrown: ArrayIndexOutOfBoundsException
env->DeleteLocalRef(jtxns);
return nullptr;
}
return jtxns;
}
/*
* Class: org_rocksdb_TransactionDB
* Method: getLockStatusData
* Signature: (J)Ljava/util/Map;
*/
jobject Java_org_rocksdb_TransactionDB_getLockStatusData(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
const std::unordered_multimap<uint32_t, rocksdb::KeyLockInfo> lock_status_data =
txn_db->GetLockStatusData();
const jobject jlock_status_data = rocksdb::HashMapJni::construct(env,
static_cast<uint32_t>(lock_status_data.size()));
if (jlock_status_data == nullptr) {
// exception occurred
return nullptr;
}
const rocksdb::HashMapJni::FnMapKV<const int32_t, const rocksdb::KeyLockInfo> fn_map_kv =
[env, txn_db, &lock_status_data](const std::pair<const int32_t, const rocksdb::KeyLockInfo>& pair) {
const jobject jlong_column_family_id =
rocksdb::LongJni::valueOf(env, pair.first);
if (jlong_column_family_id == nullptr) {
// an error occurred
return std::unique_ptr<std::pair<jobject, jobject>>(nullptr);
}
const jobject jkey_lock_info =
rocksdb::KeyLockInfoJni::construct(env, pair.second);
if (jkey_lock_info == nullptr) {
// an error occurred
return std::unique_ptr<std::pair<jobject, jobject>>(nullptr);
}
return std::unique_ptr<std::pair<jobject, jobject>>(new std::pair<jobject, jobject>(jlong_column_family_id,
jkey_lock_info));
};
if(!rocksdb::HashMapJni::putAll(env, jlock_status_data,
lock_status_data.begin(), lock_status_data.end(), fn_map_kv)) {
// exception occcurred
return nullptr;
}
return jlock_status_data;
}
/*
* Class: org_rocksdb_TransactionDB
* Method: getDeadlockInfoBuffer
* Signature: (J)[Lorg/rocksdb/TransactionDB/DeadlockPath;
*/
jobjectArray Java_org_rocksdb_TransactionDB_getDeadlockInfoBuffer(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
const std::vector<rocksdb::DeadlockPath> deadlock_info_buffer =
txn_db->GetDeadlockInfoBuffer();
const jsize deadlock_info_buffer_len =
static_cast<jsize>(deadlock_info_buffer.size());
jobjectArray jdeadlock_info_buffer =
env->NewObjectArray(deadlock_info_buffer_len,
rocksdb::DeadlockPathJni::getJClass(env), nullptr);
if (jdeadlock_info_buffer == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
jsize jdeadlock_info_buffer_offset = 0;
auto buf_end = deadlock_info_buffer.end();
for (auto buf_it = deadlock_info_buffer.begin(); buf_it != buf_end; ++buf_it) {
const rocksdb::DeadlockPath deadlock_path = *buf_it;
const std::vector<rocksdb::DeadlockInfo> deadlock_infos
= deadlock_path.path;
const jsize deadlock_infos_len =
static_cast<jsize>(deadlock_info_buffer.size());
jobjectArray jdeadlock_infos = env->NewObjectArray(deadlock_infos_len,
rocksdb::DeadlockInfoJni::getJClass(env), nullptr);
if (jdeadlock_infos == nullptr) {
// exception thrown: OutOfMemoryError
env->DeleteLocalRef(jdeadlock_info_buffer);
return nullptr;
}
jsize jdeadlock_infos_offset = 0;
auto infos_end = deadlock_infos.end();
for (auto infos_it = deadlock_infos.begin(); infos_it != infos_end; ++infos_it) {
const rocksdb::DeadlockInfo deadlock_info = *infos_it;
const jobject jdeadlock_info = rocksdb::TransactionDBJni::newDeadlockInfo(
env, jobj, deadlock_info.m_txn_id, deadlock_info.m_cf_id,
deadlock_info.m_waiting_key, deadlock_info.m_exclusive);
if (jdeadlock_info == nullptr) {
// exception occcurred
env->DeleteLocalRef(jdeadlock_info_buffer);
return nullptr;
}
env->SetObjectArrayElement(jdeadlock_infos, jdeadlock_infos_offset++, jdeadlock_info);
if (env->ExceptionCheck()) {
// exception thrown: ArrayIndexOutOfBoundsException or ArrayStoreException
env->DeleteLocalRef(jdeadlock_info);
env->DeleteLocalRef(jdeadlock_info_buffer);
return nullptr;
}
}
const jobject jdeadlock_path =
rocksdb::DeadlockPathJni::construct(env, jdeadlock_infos,
deadlock_path.limit_exceeded);
if(jdeadlock_path == nullptr) {
// exception occcurred
env->DeleteLocalRef(jdeadlock_info_buffer);
return nullptr;
}
env->SetObjectArrayElement(jdeadlock_info_buffer, jdeadlock_info_buffer_offset++, jdeadlock_path);
if (env->ExceptionCheck()) {
// exception thrown: ArrayIndexOutOfBoundsException or ArrayStoreException
env->DeleteLocalRef(jdeadlock_path);
env->DeleteLocalRef(jdeadlock_info_buffer);
return nullptr;
}
}
return jdeadlock_info_buffer;
}
/*
* Class: org_rocksdb_TransactionDB
* Method: setDeadlockInfoBufferSize
* Signature: (JI)V
*/
void Java_org_rocksdb_TransactionDB_setDeadlockInfoBufferSize(
JNIEnv* env, jobject jobj, jlong jhandle, jint jdeadlock_info_buffer_size) {
auto* txn_db = reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
txn_db->SetDeadlockInfoBufferSize(jdeadlock_info_buffer_size);
}
/*
* Class: org_rocksdb_TransactionDB
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_TransactionDB_disposeInternal(JNIEnv* env, jobject jobj,
jlong jhandle) {
delete reinterpret_cast<rocksdb::TransactionDB*>(jhandle);
}
-147
View File
@@ -1,147 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the "bridge" between Java and C++
// for rocksdb::TransactionDBOptions.
#include <jni.h>
#include "include/org_rocksdb_TransactionDBOptions.h"
#include "rocksdb/utilities/transaction_db.h"
#include "rocksjni/portal.h"
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: newTransactionDBOptions
* Signature: ()J
*/
jlong Java_org_rocksdb_TransactionDBOptions_newTransactionDBOptions(
JNIEnv* env, jclass jcls) {
rocksdb::TransactionDBOptions* opts = new rocksdb::TransactionDBOptions();
return reinterpret_cast<jlong>(opts);
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: getMaxNumLocks
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionDBOptions_getMaxNumLocks(JNIEnv* env,
jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
return opts->max_num_locks;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: setMaxNumLocks
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionDBOptions_setMaxNumLocks(JNIEnv* env,
jobject jobj, jlong jhandle, jlong jmax_num_locks) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
opts->max_num_locks = jmax_num_locks;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: getNumStripes
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionDBOptions_getNumStripes(JNIEnv* env,
jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
return opts->num_stripes;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: setNumStripes
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionDBOptions_setNumStripes(JNIEnv* env,
jobject jobj, jlong jhandle, jlong jnum_stripes) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
opts->num_stripes = jnum_stripes;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: getTransactionLockTimeout
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionDBOptions_getTransactionLockTimeout(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
return opts->transaction_lock_timeout;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: setTransactionLockTimeout
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionDBOptions_setTransactionLockTimeout(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jtransaction_lock_timeout) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
opts->transaction_lock_timeout = jtransaction_lock_timeout;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: getDefaultLockTimeout
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionDBOptions_getDefaultLockTimeout(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
return opts->default_lock_timeout;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: setDefaultLockTimeout
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionDBOptions_setDefaultLockTimeout(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jdefault_lock_timeout) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
opts->default_lock_timeout = jdefault_lock_timeout;
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: getWritePolicy
* Signature: (J)B
*/
jbyte Java_org_rocksdb_TransactionDBOptions_getWritePolicy(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
return rocksdb::TxnDBWritePolicyJni::toJavaTxnDBWritePolicy(opts->write_policy);
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: setWritePolicy
* Signature: (JB)V
*/
void Java_org_rocksdb_TransactionDBOptions_setWritePolicy(
JNIEnv* env, jobject jobj, jlong jhandle, jbyte jwrite_policy) {
auto* opts = reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
opts->write_policy =
rocksdb::TxnDBWritePolicyJni::toCppTxnDBWritePolicy(jwrite_policy);
}
/*
* Class: org_rocksdb_TransactionDBOptions
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_TransactionDBOptions_disposeInternal(JNIEnv* env,
jobject jobj, jlong jhandle) {
delete reinterpret_cast<rocksdb::TransactionDBOptions*>(jhandle);
}
-42
View File
@@ -1,42 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the "bridge" between Java and C++
// for rocksdb::TransactionNotifier.
#include <jni.h>
#include "include/org_rocksdb_AbstractTransactionNotifier.h"
#include "rocksjni/transaction_notifier_jnicallback.h"
/*
* Class: org_rocksdb_AbstractTransactionNotifier
* Method: createNewTransactionNotifier
* Signature: ()J
*/
jlong Java_org_rocksdb_AbstractTransactionNotifier_createNewTransactionNotifier(
JNIEnv* env, jobject jobj) {
auto* transaction_notifier =
new rocksdb::TransactionNotifierJniCallback(env, jobj);
auto* sptr_transaction_notifier =
new std::shared_ptr<rocksdb::TransactionNotifierJniCallback>(
transaction_notifier);
return reinterpret_cast<jlong>(sptr_transaction_notifier);
}
/*
* Class: org_rocksdb_AbstractTransactionNotifier
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_AbstractTransactionNotifier_disposeInternal(JNIEnv* env,
jobject jobj, jlong jhandle) {
// TODO(AR) refactor to use JniCallback::JniCallback
// when https://github.com/facebook/rocksdb/pull/1241/ is merged
std::shared_ptr<rocksdb::TransactionNotifierJniCallback>* handle =
reinterpret_cast<std::shared_ptr<
rocksdb::TransactionNotifierJniCallback>*>(jhandle);
delete handle;
}
@@ -1,39 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the callback "bridge" between Java and C++ for
// rocksdb::TransactionNotifier.
#include "rocksjni/transaction_notifier_jnicallback.h"
#include "rocksjni/portal.h"
namespace rocksdb {
TransactionNotifierJniCallback::TransactionNotifierJniCallback(JNIEnv* env,
jobject jtransaction_notifier) : JniCallback(env, jtransaction_notifier) {
// we cache the method id for the JNI callback
m_jsnapshot_created_methodID =
AbstractTransactionNotifierJni::getSnapshotCreatedMethodId(env);
}
void TransactionNotifierJniCallback::SnapshotCreated(
const Snapshot* newSnapshot) {
jboolean attached_thread = JNI_FALSE;
JNIEnv* env = getJniEnv(&attached_thread);
assert(env != nullptr);
env->CallVoidMethod(m_jcallback_obj,
m_jsnapshot_created_methodID, reinterpret_cast<jlong>(newSnapshot));
if(env->ExceptionCheck()) {
// exception thrown from CallVoidMethod
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
releaseJniEnv(attached_thread);
}
} // namespace rocksdb
@@ -1,42 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the callback "bridge" between Java and C++ for
// rocksdb::TransactionNotifier.
#ifndef JAVA_ROCKSJNI_TRANSACTION_NOTIFIER_JNICALLBACK_H_
#define JAVA_ROCKSJNI_TRANSACTION_NOTIFIER_JNICALLBACK_H_
#include <jni.h>
#include "rocksdb/utilities/transaction.h"
#include "rocksjni/jnicallback.h"
namespace rocksdb {
/**
* This class acts as a bridge between C++
* and Java. The methods in this class will be
* called back from the RocksDB TransactionDB or OptimisticTransactionDB (C++),
* we then callback to the appropriate Java method
* this enables TransactionNotifier to be implemented in Java.
*
* Unlike RocksJava's Comparator JNI Callback, we do not attempt
* to reduce Java object allocations by caching the Snapshot object
* presented to the callback. This could be revisited in future
* if performance is lacking.
*/
class TransactionNotifierJniCallback: public JniCallback,
public TransactionNotifier {
public:
TransactionNotifierJniCallback(JNIEnv* env, jobject jtransaction_notifier);
virtual void SnapshotCreated(const Snapshot* newSnapshot);
private:
jmethodID m_jsnapshot_created_methodID;
};
} // namespace rocksdb
#endif // JAVA_ROCKSJNI_TRANSACTION_NOTIFIER_JNICALLBACK_H_
-166
View File
@@ -1,166 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the "bridge" between Java and C++
// for rocksdb::TransactionOptions.
#include <jni.h>
#include "include/org_rocksdb_TransactionOptions.h"
#include "rocksdb/utilities/transaction_db.h"
/*
* Class: org_rocksdb_TransactionOptions
* Method: newTransactionOptions
* Signature: ()J
*/
jlong Java_org_rocksdb_TransactionOptions_newTransactionOptions(JNIEnv* env,
jclass jcls) {
auto* opts = new rocksdb::TransactionOptions();
return reinterpret_cast<jlong>(opts);
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: isSetSnapshot
* Signature: (J)Z
*/
jboolean Java_org_rocksdb_TransactionOptions_isSetSnapshot(JNIEnv* env,
jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
return opts->set_snapshot;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: setSetSnapshot
* Signature: (JZ)V
*/
void Java_org_rocksdb_TransactionOptions_setSetSnapshot(JNIEnv* env,
jobject jobj, jlong jhandle, jboolean jset_snapshot) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
opts->set_snapshot = jset_snapshot;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: isDeadlockDetect
* Signature: (J)Z
*/
jboolean Java_org_rocksdb_TransactionOptions_isDeadlockDetect(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
return opts->deadlock_detect;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: setDeadlockDetect
* Signature: (JZ)V
*/
void Java_org_rocksdb_TransactionOptions_setDeadlockDetect(
JNIEnv* env, jobject jobj, jlong jhandle, jboolean jdeadlock_detect) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
opts->deadlock_detect = jdeadlock_detect;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: getLockTimeout
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionOptions_getLockTimeout(JNIEnv* env,
jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
return opts->lock_timeout;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: setLockTimeout
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionOptions_setLockTimeout(JNIEnv* env,
jobject jobj, jlong jhandle, jlong jlock_timeout) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
opts->lock_timeout = jlock_timeout;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: getExpiration
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionOptions_getExpiration(JNIEnv* env,
jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
return opts->expiration;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: setExpiration
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionOptions_setExpiration(JNIEnv* env,
jobject jobj, jlong jhandle, jlong jexpiration) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
opts->expiration = jexpiration;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: getDeadlockDetectDepth
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionOptions_getDeadlockDetectDepth(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
return opts->deadlock_detect_depth;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: setDeadlockDetectDepth
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionOptions_setDeadlockDetectDepth(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jdeadlock_detect_depth) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
opts->deadlock_detect_depth = jdeadlock_detect_depth;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: getMaxWriteBatchSize
* Signature: (J)J
*/
jlong Java_org_rocksdb_TransactionOptions_getMaxWriteBatchSize(
JNIEnv* env, jobject jobj, jlong jhandle) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
return opts->max_write_batch_size;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: setMaxWriteBatchSize
* Signature: (JJ)V
*/
void Java_org_rocksdb_TransactionOptions_setMaxWriteBatchSize(
JNIEnv* env, jobject jobj, jlong jhandle, jlong jmax_write_batch_size) {
auto* opts = reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
opts->max_write_batch_size = jmax_write_batch_size;
}
/*
* Class: org_rocksdb_TransactionOptions
* Method: disposeInternal
* Signature: (J)V
*/
void Java_org_rocksdb_TransactionOptions_disposeInternal(JNIEnv* env,
jobject jobj, jlong jhandle) {
delete reinterpret_cast<rocksdb::TransactionOptions*>(jhandle);
}
+34 -330
View File
@@ -27,34 +27,12 @@
* Method: newWriteBatch
* Signature: (I)J
*/
jlong Java_org_rocksdb_WriteBatch_newWriteBatch__I(
jlong Java_org_rocksdb_WriteBatch_newWriteBatch(
JNIEnv* env, jclass jcls, jint jreserved_bytes) {
auto* wb = new rocksdb::WriteBatch(static_cast<size_t>(jreserved_bytes));
return reinterpret_cast<jlong>(wb);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: newWriteBatch
* Signature: ([BI)J
*/
jlong Java_org_rocksdb_WriteBatch_newWriteBatch___3BI(
JNIEnv* env, jclass jcls, jbyteArray jserialized,
jint jserialized_length) {
jboolean has_exception = JNI_FALSE;
std::string serialized = rocksdb::JniUtil::byteString<std::string>(env,
jserialized, jserialized_length,
[](const char* str, const size_t len) { return std::string(str, len); },
&has_exception);
if(has_exception == JNI_TRUE) {
// exception occurred
return 0;
}
auto* wb = new rocksdb::WriteBatch(serialized);
return reinterpret_cast<jlong>(wb);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: count0
@@ -112,37 +90,6 @@ void Java_org_rocksdb_WriteBatch_rollbackToSavePoint0(
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: popSavePoint
* Signature: (J)V
*/
void Java_org_rocksdb_WriteBatch_popSavePoint(
JNIEnv* env, jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto s = wb->PopSavePoint();
if (s.ok()) {
return;
}
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: setMaxBytes
* Signature: (JJ)V
*/
void Java_org_rocksdb_WriteBatch_setMaxBytes(
JNIEnv* env, jobject jobj, jlong jwb_handle, jlong jmax_bytes) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
wb->SetMaxBytes(static_cast<size_t>(jmax_bytes));
}
/*
* Class: org_rocksdb_WriteBatch
* Method: put
@@ -155,13 +102,10 @@ void Java_org_rocksdb_WriteBatch_put__J_3BI_3BI(
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto put = [&wb] (rocksdb::Slice key, rocksdb::Slice value) {
return wb->Put(key, value);
wb->Put(key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(put, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(put, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
@@ -178,13 +122,10 @@ void Java_org_rocksdb_WriteBatch_put__J_3BI_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto put = [&wb, &cf_handle] (rocksdb::Slice key, rocksdb::Slice value) {
return wb->Put(cf_handle, key, value);
wb->Put(cf_handle, key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(put, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(put, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
@@ -199,13 +140,10 @@ void Java_org_rocksdb_WriteBatch_merge__J_3BI_3BI(
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto merge = [&wb] (rocksdb::Slice key, rocksdb::Slice value) {
return wb->Merge(key, value);
wb->Merge(key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(merge, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(merge, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
@@ -222,41 +160,34 @@ void Java_org_rocksdb_WriteBatch_merge__J_3BI_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto merge = [&wb, &cf_handle] (rocksdb::Slice key, rocksdb::Slice value) {
return wb->Merge(cf_handle, key, value);
wb->Merge(cf_handle, key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(merge, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(merge, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: delete
* Method: remove
* Signature: (J[BI)V
*/
void Java_org_rocksdb_WriteBatch_delete__J_3BI(
void Java_org_rocksdb_WriteBatch_remove__J_3BI(
JNIEnv* env, jobject jobj, jlong jwb_handle,
jbyteArray jkey, jint jkey_len) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto remove = [&wb] (rocksdb::Slice key) {
return wb->Delete(key);
wb->Delete(key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(remove, env,
jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::k_op(remove, env, jobj, jkey, jkey_len);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: delete
* Method: remove
* Signature: (J[BIJ)V
*/
void Java_org_rocksdb_WriteBatch_delete__J_3BIJ(
void Java_org_rocksdb_WriteBatch_remove__J_3BIJ(
JNIEnv* env, jobject jobj, jlong jwb_handle,
jbyteArray jkey, jint jkey_len, jlong jcf_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
@@ -264,55 +195,9 @@ void Java_org_rocksdb_WriteBatch_delete__J_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto remove = [&wb, &cf_handle] (rocksdb::Slice key) {
return wb->Delete(cf_handle, key);
wb->Delete(cf_handle, key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(remove, env,
jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
}
/*
* Class: org_rocksdb_WriteBatch
* Method: singleDelete
* Signature: (J[BI)V
*/
void Java_org_rocksdb_WriteBatch_singleDelete__J_3BI(
JNIEnv* env, jobject jobj, jlong jwb_handle, jbyteArray jkey,
jint jkey_len) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto single_delete = [&wb] (rocksdb::Slice key) {
return wb->SingleDelete(key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(single_delete,
env, jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
}
/*
* Class: org_rocksdb_WriteBatch
* Method: singleDelete
* Signature: (J[BIJ)V
*/
void Java_org_rocksdb_WriteBatch_singleDelete__J_3BIJ(
JNIEnv* env, jobject jobj, jlong jwb_handle, jbyteArray jkey,
jint jkey_len, jlong jcf_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto single_delete = [&wb, &cf_handle] (rocksdb::Slice key) {
return wb->SingleDelete(cf_handle, key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(single_delete,
env, jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::k_op(remove, env, jobj, jkey, jkey_len);
}
/*
@@ -320,20 +205,19 @@ void Java_org_rocksdb_WriteBatch_singleDelete__J_3BIJ(
* Method: deleteRange
* Signature: (J[BI[BI)V
*/
JNIEXPORT void JNICALL Java_org_rocksdb_WriteBatch_deleteRange__J_3BI_3BI(
JNIEnv*, jobject, jlong, jbyteArray, jint, jbyteArray, jint);
void Java_org_rocksdb_WriteBatch_deleteRange__J_3BI_3BI(
JNIEnv* env, jobject jobj, jlong jwb_handle, jbyteArray jbegin_key,
jint jbegin_key_len, jbyteArray jend_key, jint jend_key_len) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto deleteRange = [&wb](rocksdb::Slice beginKey, rocksdb::Slice endKey) {
return wb->DeleteRange(beginKey, endKey);
wb->DeleteRange(beginKey, endKey);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(
deleteRange, env, jobj, jbegin_key, jbegin_key_len, jend_key,
jend_key_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(deleteRange, env, jobj, jbegin_key, jbegin_key_len,
jend_key, jend_key_len);
}
/*
@@ -350,15 +234,11 @@ void Java_org_rocksdb_WriteBatch_deleteRange__J_3BI_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto deleteRange = [&wb, &cf_handle](rocksdb::Slice beginKey,
rocksdb::Slice endKey) {
return wb->DeleteRange(cf_handle, beginKey, endKey);
rocksdb::Slice endKey) {
wb->DeleteRange(cf_handle, beginKey, endKey);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(
deleteRange, env, jobj, jbegin_key, jbegin_key_len, jend_key,
jend_key_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(deleteRange, env, jobj, jbegin_key, jbegin_key_len,
jend_key, jend_key_len);
}
/*
@@ -372,13 +252,9 @@ void Java_org_rocksdb_WriteBatch_putLogData(
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto putLogData = [&wb] (rocksdb::Slice blob) {
return wb->PutLogData(blob);
wb->PutLogData(blob);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(putLogData,
env, jobj, jblob, jblob_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::k_op(putLogData, env, jobj, jblob, jblob_len);
}
/*
@@ -387,7 +263,7 @@ void Java_org_rocksdb_WriteBatch_putLogData(
* Signature: (JJ)V
*/
void Java_org_rocksdb_WriteBatch_iterate(
JNIEnv* env, jobject jobj, jlong jwb_handle, jlong handlerHandle) {
JNIEnv* env , jobject jobj, jlong jwb_handle, jlong handlerHandle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
@@ -400,178 +276,6 @@ void Java_org_rocksdb_WriteBatch_iterate(
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: data
* Signature: (J)[B
*/
jbyteArray Java_org_rocksdb_WriteBatch_data(
JNIEnv* env, jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto data = wb->Data();
return rocksdb::JniUtil::copyBytes(env, data);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: getDataSize
* Signature: (J)J
*/
jlong Java_org_rocksdb_WriteBatch_getDataSize(
JNIEnv* env, jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto data_size = wb->GetDataSize();
return static_cast<jlong>(data_size);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasPut
* Signature: (J)Z
*/
jboolean Java_org_rocksdb_WriteBatch_hasPut(
JNIEnv* env, jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasPut();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasDelete
* Signature: (J)Z
*/
jboolean Java_org_rocksdb_WriteBatch_hasDelete(
JNIEnv* env, jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasDelete();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasSingleDelete
* Signature: (J)Z
*/
JNIEXPORT jboolean JNICALL Java_org_rocksdb_WriteBatch_hasSingleDelete(
JNIEnv* env , jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasSingleDelete();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasDeleteRange
* Signature: (J)Z
*/
JNIEXPORT jboolean JNICALL Java_org_rocksdb_WriteBatch_hasDeleteRange(
JNIEnv* env , jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasDeleteRange();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasMerge
* Signature: (J)Z
*/
JNIEXPORT jboolean JNICALL Java_org_rocksdb_WriteBatch_hasMerge(
JNIEnv* env , jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasMerge();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasBeginPrepare
* Signature: (J)Z
*/
JNIEXPORT jboolean JNICALL Java_org_rocksdb_WriteBatch_hasBeginPrepare(
JNIEnv* env , jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasBeginPrepare();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasEndPrepare
* Signature: (J)Z
*/
JNIEXPORT jboolean JNICALL Java_org_rocksdb_WriteBatch_hasEndPrepare(
JNIEnv* env , jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasEndPrepare();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasCommit
* Signature: (J)Z
*/
JNIEXPORT jboolean JNICALL Java_org_rocksdb_WriteBatch_hasCommit(
JNIEnv* env , jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasCommit();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: hasRollback
* Signature: (J)Z
*/
JNIEXPORT jboolean JNICALL Java_org_rocksdb_WriteBatch_hasRollback(
JNIEnv* env , jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
return wb->HasRollback();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: markWalTerminationPoint
* Signature: (J)V
*/
void Java_org_rocksdb_WriteBatch_markWalTerminationPoint(
JNIEnv* env, jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
wb->MarkWalTerminationPoint();
}
/*
* Class: org_rocksdb_WriteBatch
* Method: getWalTerminationPoint
* Signature: (J)Lorg/rocksdb/WriteBatch/SavePoint;
*/
jobject Java_org_rocksdb_WriteBatch_getWalTerminationPoint(
JNIEnv* env, jobject jobj, jlong jwb_handle) {
auto* wb = reinterpret_cast<rocksdb::WriteBatch*>(jwb_handle);
assert(wb != nullptr);
auto save_point = wb->GetWalTerminationPoint();
return rocksdb::WriteBatchSavePointJni::construct(env, save_point);
}
/*
* Class: org_rocksdb_WriteBatch
* Method: disposeInternal
+2 -30
View File
@@ -87,32 +87,8 @@ jbyteArray Java_org_rocksdb_WriteBatchTest_getContents(
state.append(")");
count++;
break;
case rocksdb::kTypeSingleDeletion:
state.append("SingleDelete(");
state.append(ikey.user_key.ToString());
state.append(")");
count++;
break;
case rocksdb::kTypeRangeDeletion:
state.append("DeleteRange(");
state.append(ikey.user_key.ToString());
state.append(", ");
state.append(iter->value().ToString());
state.append(")");
count++;
break;
case rocksdb::kTypeLogData:
state.append("LogData(");
state.append(ikey.user_key.ToString());
state.append(")");
count++;
break;
default:
assert(false);
state.append("Err:Expected(");
state.append(std::to_string(ikey.type));
state.append(")");
count++;
break;
}
state.append("@");
@@ -120,12 +96,8 @@ jbyteArray Java_org_rocksdb_WriteBatchTest_getContents(
}
if (!s.ok()) {
state.append(s.ToString());
} else if (rocksdb::WriteBatchInternal::Count(b) != count) {
state.append("Err:CountMismatch(expected=");
state.append(std::to_string(rocksdb::WriteBatchInternal::Count(b)));
state.append(", actual=");
state.append(std::to_string(count));
state.append(")");
} else if (count != rocksdb::WriteBatchInternal::Count(b)) {
state.append("CountMismatch()");
}
delete mem->Unref();
+62 -178
View File
@@ -39,31 +39,19 @@ jlong Java_org_rocksdb_WriteBatchWithIndex_newWriteBatchWithIndex__Z(
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: newWriteBatchWithIndex
* Signature: (JBIZ)J
* Signature: (JZIZ)J
*/
jlong Java_org_rocksdb_WriteBatchWithIndex_newWriteBatchWithIndex__JBIZ(
jlong Java_org_rocksdb_WriteBatchWithIndex_newWriteBatchWithIndex__JZIZ(
JNIEnv* env, jclass jcls, jlong jfallback_index_comparator_handle,
jbyte jcomparator_type, jint jreserved_bytes, jboolean joverwrite_key) {
rocksdb::Comparator *fallback_comparator = nullptr;
switch(jcomparator_type) {
// JAVA_COMPARATOR
case 0x0:
fallback_comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jfallback_index_comparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
fallback_comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jfallback_index_comparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
fallback_comparator =
reinterpret_cast<rocksdb::Comparator*>(jfallback_index_comparator_handle);
break;
}
jboolean is_direct, jint jreserved_bytes, jboolean joverwrite_key) {
rocksdb::Comparator *fallback_comparator = nullptr;
if(is_direct) {
fallback_comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(jfallback_index_comparator_handle);
} else {
fallback_comparator =
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jfallback_index_comparator_handle);
}
auto* wbwi =
new rocksdb::WriteBatchWithIndex(
fallback_comparator,
@@ -95,13 +83,10 @@ void Java_org_rocksdb_WriteBatchWithIndex_put__J_3BI_3BI(
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto put = [&wbwi] (rocksdb::Slice key, rocksdb::Slice value) {
return wbwi->Put(key, value);
wbwi->Put(key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(put, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(put, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
@@ -118,13 +103,10 @@ void Java_org_rocksdb_WriteBatchWithIndex_put__J_3BI_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto put = [&wbwi, &cf_handle] (rocksdb::Slice key, rocksdb::Slice value) {
return wbwi->Put(cf_handle, key, value);
wbwi->Put(cf_handle, key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(put, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(put, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
@@ -138,13 +120,10 @@ void Java_org_rocksdb_WriteBatchWithIndex_merge__J_3BI_3BI(
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto merge = [&wbwi] (rocksdb::Slice key, rocksdb::Slice value) {
return wbwi->Merge(key, value);
wbwi->Merge(key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(merge, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(merge, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
@@ -161,41 +140,34 @@ void Java_org_rocksdb_WriteBatchWithIndex_merge__J_3BI_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto merge = [&wbwi, &cf_handle] (rocksdb::Slice key, rocksdb::Slice value) {
return wbwi->Merge(cf_handle, key, value);
wbwi->Merge(cf_handle, key, value);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(merge, env,
jobj, jkey, jkey_len, jentry_value, jentry_value_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(merge, env, jobj, jkey, jkey_len, jentry_value,
jentry_value_len);
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: delete
* Method: remove
* Signature: (J[BI)V
*/
void Java_org_rocksdb_WriteBatchWithIndex_delete__J_3BI(
void Java_org_rocksdb_WriteBatchWithIndex_remove__J_3BI(
JNIEnv* env, jobject jobj, jlong jwbwi_handle, jbyteArray jkey,
jint jkey_len) {
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto remove = [&wbwi] (rocksdb::Slice key) {
return wbwi->Delete(key);
wbwi->Delete(key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(remove, env,
jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::k_op(remove, env, jobj, jkey, jkey_len);
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: delete
* Method: remove
* Signature: (J[BIJ)V
*/
void Java_org_rocksdb_WriteBatchWithIndex_delete__J_3BIJ(
void Java_org_rocksdb_WriteBatchWithIndex_remove__J_3BIJ(
JNIEnv* env, jobject jobj, jlong jwbwi_handle, jbyteArray jkey,
jint jkey_len, jlong jcf_handle) {
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
@@ -203,55 +175,9 @@ void Java_org_rocksdb_WriteBatchWithIndex_delete__J_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto remove = [&wbwi, &cf_handle] (rocksdb::Slice key) {
return wbwi->Delete(cf_handle, key);
wbwi->Delete(cf_handle, key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(remove, env,
jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: singleDelete
* Signature: (J[BI)V
*/
void Java_org_rocksdb_WriteBatchWithIndex_singleDelete__J_3BI(
JNIEnv* env, jobject jobj, jlong jwbwi_handle, jbyteArray jkey,
jint jkey_len) {
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto single_delete = [&wbwi] (rocksdb::Slice key) {
return wbwi->SingleDelete(key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(single_delete,
env, jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: singleDelete
* Signature: (J[BIJ)V
*/
void Java_org_rocksdb_WriteBatchWithIndex_singleDelete__J_3BIJ(
JNIEnv* env, jobject jobj, jlong jwbwi_handle, jbyteArray jkey,
jint jkey_len, jlong jcf_handle) {
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto single_delete = [&wbwi, &cf_handle] (rocksdb::Slice key) {
return wbwi->SingleDelete(cf_handle, key);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(single_delete,
env, jobj, jkey, jkey_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::k_op(remove, env, jobj, jkey, jkey_len);
}
/*
@@ -265,14 +191,10 @@ void Java_org_rocksdb_WriteBatchWithIndex_deleteRange__J_3BI_3BI(
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto deleteRange = [&wbwi](rocksdb::Slice beginKey, rocksdb::Slice endKey) {
return wbwi->DeleteRange(beginKey, endKey);
wbwi->DeleteRange(beginKey, endKey);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(
deleteRange, env, jobj, jbegin_key, jbegin_key_len, jend_key,
jend_key_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(deleteRange, env, jobj, jbegin_key, jbegin_key_len,
jend_key, jend_key_len);
}
/*
@@ -289,15 +211,11 @@ void Java_org_rocksdb_WriteBatchWithIndex_deleteRange__J_3BI_3BIJ(
auto* cf_handle = reinterpret_cast<rocksdb::ColumnFamilyHandle*>(jcf_handle);
assert(cf_handle != nullptr);
auto deleteRange = [&wbwi, &cf_handle](rocksdb::Slice beginKey,
rocksdb::Slice endKey) {
return wbwi->DeleteRange(cf_handle, beginKey, endKey);
rocksdb::Slice endKey) {
wbwi->DeleteRange(cf_handle, beginKey, endKey);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::kv_op(
deleteRange, env, jobj, jbegin_key, jbegin_key_len, jend_key,
jend_key_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::kv_op(deleteRange, env, jobj, jbegin_key, jbegin_key_len,
jend_key, jend_key_len);
}
/*
@@ -311,13 +229,9 @@ void Java_org_rocksdb_WriteBatchWithIndex_putLogData(
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto putLogData = [&wbwi] (rocksdb::Slice blob) {
return wbwi->PutLogData(blob);
wbwi->PutLogData(blob);
};
std::unique_ptr<rocksdb::Status> status = rocksdb::JniUtil::k_op(putLogData,
env, jobj, jblob, jblob_len);
if (status != nullptr && !status->ok()) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, status);
}
rocksdb::JniUtil::k_op(putLogData, env, jobj, jblob, jblob_len);
}
/*
@@ -365,54 +279,6 @@ void Java_org_rocksdb_WriteBatchWithIndex_rollbackToSavePoint0(
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: popSavePoint
* Signature: (J)V
*/
void Java_org_rocksdb_WriteBatchWithIndex_popSavePoint(
JNIEnv* env, jobject jobj, jlong jwbwi_handle) {
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto s = wbwi->PopSavePoint();
if (s.ok()) {
return;
}
rocksdb::RocksDBExceptionJni::ThrowNew(env, s);
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: setMaxBytes
* Signature: (JJ)V
*/
void Java_org_rocksdb_WriteBatchWithIndex_setMaxBytes(
JNIEnv* env, jobject jobj, jlong jwbwi_handle, jlong jmax_bytes) {
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
wbwi->SetMaxBytes(static_cast<size_t>(jmax_bytes));
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: getWriteBatch
* Signature: (J)Lorg/rocksdb/WriteBatch;
*/
jobject Java_org_rocksdb_WriteBatchWithIndex_getWriteBatch(
JNIEnv* env, jobject jobj, jlong jwbwi_handle) {
auto* wbwi = reinterpret_cast<rocksdb::WriteBatchWithIndex*>(jwbwi_handle);
assert(wbwi != nullptr);
auto* wb = wbwi->GetWriteBatch();
// TODO(AR) is the `wb` object owned by us?
return rocksdb::WriteBatchJni::construct(env, wb);
}
/*
* Class: org_rocksdb_WriteBatchWithIndex
* Method: iterator0
@@ -685,15 +551,33 @@ jlongArray Java_org_rocksdb_WBWIRocksIterator_entry1(
jlong results[3];
// set the type of the write entry
results[0] = rocksdb::WriteTypeJni::toJavaWriteType(we.type);
//set the type of the write entry
switch (we.type) {
case rocksdb::kPutRecord:
results[0] = 0x1;
break;
// NOTE: key_slice and value_slice will be freed by org.rocksdb.DirectSlice#close
case rocksdb::kMergeRecord:
results[0] = 0x2;
break;
case rocksdb::kDeleteRecord:
results[0] = 0x4;
break;
case rocksdb::kLogDataRecord:
results[0] = 0x8;
break;
default:
results[0] = 0x0;
}
// key_slice and value_slice will be freed by org.rocksdb.DirectSlice#close
auto* key_slice = new rocksdb::Slice(we.key.data(), we.key.size());
results[1] = reinterpret_cast<jlong>(key_slice);
if (we.type == rocksdb::kDeleteRecord
|| we.type == rocksdb::kSingleDeleteRecord
|| we.type == rocksdb::kLogDataRecord) {
// set native handle of value slice to null if no value available
results[2] = 0;
+179 -394
View File
@@ -14,62 +14,24 @@ WriteBatchHandlerJniCallback::WriteBatchHandlerJniCallback(
JNIEnv* env, jobject jWriteBatchHandler)
: JniCallback(env, jWriteBatchHandler), m_env(env) {
m_jPutCfMethodId = WriteBatchHandlerJni::getPutCfMethodId(env);
if(m_jPutCfMethodId == nullptr) {
// exception thrown
return;
}
m_jPutMethodId = WriteBatchHandlerJni::getPutMethodId(env);
if(m_jPutMethodId == nullptr) {
// exception thrown
return;
}
m_jMergeCfMethodId = WriteBatchHandlerJni::getMergeCfMethodId(env);
if(m_jMergeCfMethodId == nullptr) {
// exception thrown
return;
}
m_jMergeMethodId = WriteBatchHandlerJni::getMergeMethodId(env);
if(m_jMergeMethodId == nullptr) {
// exception thrown
return;
}
m_jDeleteCfMethodId = WriteBatchHandlerJni::getDeleteCfMethodId(env);
if(m_jDeleteCfMethodId == nullptr) {
// exception thrown
return;
}
m_jDeleteMethodId = WriteBatchHandlerJni::getDeleteMethodId(env);
if(m_jDeleteMethodId == nullptr) {
// exception thrown
return;
}
m_jSingleDeleteCfMethodId =
WriteBatchHandlerJni::getSingleDeleteCfMethodId(env);
if(m_jSingleDeleteCfMethodId == nullptr) {
// exception thrown
return;
}
m_jSingleDeleteMethodId = WriteBatchHandlerJni::getSingleDeleteMethodId(env);
if(m_jSingleDeleteMethodId == nullptr) {
// exception thrown
return;
}
m_jDeleteRangeCfMethodId =
WriteBatchHandlerJni::getDeleteRangeCfMethodId(env);
if (m_jDeleteRangeCfMethodId == nullptr) {
// exception thrown
return;
}
m_jDeleteRangeMethodId = WriteBatchHandlerJni::getDeleteRangeMethodId(env);
if (m_jDeleteRangeMethodId == nullptr) {
// exception thrown
@@ -82,45 +44,6 @@ WriteBatchHandlerJniCallback::WriteBatchHandlerJniCallback(
return;
}
m_jPutBlobIndexCfMethodId =
WriteBatchHandlerJni::getPutBlobIndexCfMethodId(env);
if(m_jPutBlobIndexCfMethodId == nullptr) {
// exception thrown
return;
}
m_jMarkBeginPrepareMethodId =
WriteBatchHandlerJni::getMarkBeginPrepareMethodId(env);
if(m_jMarkBeginPrepareMethodId == nullptr) {
// exception thrown
return;
}
m_jMarkEndPrepareMethodId =
WriteBatchHandlerJni::getMarkEndPrepareMethodId(env);
if(m_jMarkEndPrepareMethodId == nullptr) {
// exception thrown
return;
}
m_jMarkNoopMethodId = WriteBatchHandlerJni::getMarkNoopMethodId(env);
if(m_jMarkNoopMethodId == nullptr) {
// exception thrown
return;
}
m_jMarkRollbackMethodId = WriteBatchHandlerJni::getMarkRollbackMethodId(env);
if(m_jMarkRollbackMethodId == nullptr) {
// exception thrown
return;
}
m_jMarkCommitMethodId = WriteBatchHandlerJni::getMarkCommitMethodId(env);
if(m_jMarkCommitMethodId == nullptr) {
// exception thrown
return;
}
m_jContinueMethodId = WriteBatchHandlerJni::getContinueMethodId(env);
if(m_jContinueMethodId == nullptr) {
// exception thrown
@@ -128,272 +51,196 @@ WriteBatchHandlerJniCallback::WriteBatchHandlerJniCallback(
}
}
rocksdb::Status WriteBatchHandlerJniCallback::PutCF(uint32_t column_family_id,
const Slice& key, const Slice& value) {
auto put = [this, column_family_id] (
jbyteArray j_key, jbyteArray j_value) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jPutCfMethodId,
static_cast<jint>(column_family_id),
j_key,
j_value);
};
auto status = WriteBatchHandlerJniCallback::kv_op(key, value, put);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
}
}
void WriteBatchHandlerJniCallback::Put(const Slice& key, const Slice& value) {
auto put = [this] (
jbyteArray j_key, jbyteArray j_value) {
m_env->CallVoidMethod(
const jbyteArray j_key = sliceToJArray(key);
if(j_key == nullptr) {
// exception thrown
if(m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
return;
}
const jbyteArray j_value = sliceToJArray(value);
if(j_value == nullptr) {
// exception thrown
if(m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
return;
}
m_env->CallVoidMethod(
m_jcallback_obj,
m_jPutMethodId,
j_key,
j_value);
};
WriteBatchHandlerJniCallback::kv_op(key, value, put);
}
if(m_env->ExceptionCheck()) {
// exception thrown
m_env->ExceptionDescribe();
if(j_value != nullptr) {
m_env->DeleteLocalRef(j_value);
}
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
return;
}
rocksdb::Status WriteBatchHandlerJniCallback::MergeCF(uint32_t column_family_id,
const Slice& key, const Slice& value) {
auto merge = [this, column_family_id] (
jbyteArray j_key, jbyteArray j_value) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jMergeCfMethodId,
static_cast<jint>(column_family_id),
j_key,
j_value);
};
auto status = WriteBatchHandlerJniCallback::kv_op(key, value, merge);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
if(j_value != nullptr) {
m_env->DeleteLocalRef(j_value);
}
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
}
void WriteBatchHandlerJniCallback::Merge(const Slice& key, const Slice& value) {
auto merge = [this] (
jbyteArray j_key, jbyteArray j_value) {
m_env->CallVoidMethod(
const jbyteArray j_key = sliceToJArray(key);
if(j_key == nullptr) {
// exception thrown
if(m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
return;
}
const jbyteArray j_value = sliceToJArray(value);
if(j_value == nullptr) {
// exception thrown
if(m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
return;
}
m_env->CallVoidMethod(
m_jcallback_obj,
m_jMergeMethodId,
j_key,
j_value);
};
WriteBatchHandlerJniCallback::kv_op(key, value, merge);
}
if(m_env->ExceptionCheck()) {
// exception thrown
m_env->ExceptionDescribe();
if(j_value != nullptr) {
m_env->DeleteLocalRef(j_value);
}
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
return;
}
rocksdb::Status WriteBatchHandlerJniCallback::DeleteCF(uint32_t column_family_id,
const Slice& key) {
auto remove = [this, column_family_id] (jbyteArray j_key) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jDeleteCfMethodId,
static_cast<jint>(column_family_id),
j_key);
};
auto status = WriteBatchHandlerJniCallback::k_op(key, remove);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
if(j_value != nullptr) {
m_env->DeleteLocalRef(j_value);
}
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
}
void WriteBatchHandlerJniCallback::Delete(const Slice& key) {
auto remove = [this] (jbyteArray j_key) {
m_env->CallVoidMethod(
const jbyteArray j_key = sliceToJArray(key);
if(j_key == nullptr) {
// exception thrown
if(m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
return;
}
m_env->CallVoidMethod(
m_jcallback_obj,
m_jDeleteMethodId,
j_key);
};
WriteBatchHandlerJniCallback::k_op(key, remove);
}
rocksdb::Status WriteBatchHandlerJniCallback::SingleDeleteCF(uint32_t column_family_id,
const Slice& key) {
auto singleDelete = [this, column_family_id] (jbyteArray j_key) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jSingleDeleteCfMethodId,
static_cast<jint>(column_family_id),
j_key);
};
auto status = WriteBatchHandlerJniCallback::k_op(key, singleDelete);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
if(m_env->ExceptionCheck()) {
// exception thrown
m_env->ExceptionDescribe();
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
return;
}
}
void WriteBatchHandlerJniCallback::SingleDelete(const Slice& key) {
auto singleDelete = [this] (jbyteArray j_key) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jSingleDeleteMethodId,
j_key);
};
WriteBatchHandlerJniCallback::k_op(key, singleDelete);
}
rocksdb::Status WriteBatchHandlerJniCallback::DeleteRangeCF(uint32_t column_family_id,
const Slice& beginKey, const Slice& endKey) {
auto deleteRange = [this, column_family_id] (
jbyteArray j_beginKey, jbyteArray j_endKey) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jDeleteRangeCfMethodId,
static_cast<jint>(column_family_id),
j_beginKey,
j_endKey);
};
auto status = WriteBatchHandlerJniCallback::kv_op(beginKey, endKey, deleteRange);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
if(j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
}
void WriteBatchHandlerJniCallback::DeleteRange(const Slice& beginKey,
const Slice& endKey) {
auto deleteRange = [this] (
jbyteArray j_beginKey, jbyteArray j_endKey) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jDeleteRangeMethodId,
j_beginKey,
j_endKey);
};
WriteBatchHandlerJniCallback::kv_op(beginKey, endKey, deleteRange);
const Slice& endKey) {
const jbyteArray j_beginKey = sliceToJArray(beginKey);
if (j_beginKey == nullptr) {
// exception thrown
if (m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
return;
}
const jbyteArray j_endKey = sliceToJArray(beginKey);
if (j_endKey == nullptr) {
// exception thrown
if (m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
return;
}
m_env->CallVoidMethod(m_jcallback_obj, m_jDeleteRangeMethodId,
j_beginKey, j_endKey);
if (m_env->ExceptionCheck()) {
// exception thrown
m_env->ExceptionDescribe();
if (j_beginKey != nullptr) {
m_env->DeleteLocalRef(j_beginKey);
}
if (j_endKey != nullptr) {
m_env->DeleteLocalRef(j_endKey);
}
return;
}
if (j_beginKey != nullptr) {
m_env->DeleteLocalRef(j_beginKey);
}
if (j_endKey != nullptr) {
m_env->DeleteLocalRef(j_endKey);
}
}
void WriteBatchHandlerJniCallback::LogData(const Slice& blob) {
auto logData = [this] (jbyteArray j_blob) {
m_env->CallVoidMethod(
const jbyteArray j_blob = sliceToJArray(blob);
if(j_blob == nullptr) {
// exception thrown
if(m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
return;
}
m_env->CallVoidMethod(
m_jcallback_obj,
m_jLogDataMethodId,
j_blob);
};
WriteBatchHandlerJniCallback::k_op(blob, logData);
}
rocksdb::Status WriteBatchHandlerJniCallback::PutBlobIndexCF(uint32_t column_family_id,
const Slice& key, const Slice& value) {
auto putBlobIndex = [this, column_family_id] (
jbyteArray j_key, jbyteArray j_value) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jPutBlobIndexCfMethodId,
static_cast<jint>(column_family_id),
j_key,
j_value);
};
auto status = WriteBatchHandlerJniCallback::kv_op(key, value, putBlobIndex);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
}
}
rocksdb::Status WriteBatchHandlerJniCallback::MarkBeginPrepare() {
m_env->CallVoidMethod(m_jcallback_obj, m_jMarkBeginPrepareMethodId);
// check for Exception, in-particular RocksDBException
if (m_env->ExceptionCheck()) {
if(m_env->ExceptionCheck()) {
// exception thrown
jthrowable exception = m_env->ExceptionOccurred();
std::unique_ptr<rocksdb::Status> status = rocksdb::RocksDBExceptionJni::toCppStatus(m_env, exception);
if (status == nullptr) {
// unkown status or exception occurred extracting status
m_env->ExceptionDescribe();
return rocksdb::Status::OK(); // TODO(AR) probably need a better error code here
} else {
m_env->ExceptionClear(); // clear the exception, as we have extracted the status
return rocksdb::Status(*status);
m_env->ExceptionDescribe();
if(j_blob != nullptr) {
m_env->DeleteLocalRef(j_blob);
}
return;
}
return rocksdb::Status::OK();
}
rocksdb::Status WriteBatchHandlerJniCallback::MarkEndPrepare(const Slice& xid) {
auto markEndPrepare = [this] (
jbyteArray j_xid) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jMarkEndPrepareMethodId,
j_xid);
};
auto status = WriteBatchHandlerJniCallback::k_op(xid, markEndPrepare);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
}
}
rocksdb::Status WriteBatchHandlerJniCallback::MarkNoop(bool empty_batch) {
m_env->CallVoidMethod(m_jcallback_obj, m_jMarkNoopMethodId, static_cast<jboolean>(empty_batch));
// check for Exception, in-particular RocksDBException
if (m_env->ExceptionCheck()) {
// exception thrown
jthrowable exception = m_env->ExceptionOccurred();
std::unique_ptr<rocksdb::Status> status = rocksdb::RocksDBExceptionJni::toCppStatus(m_env, exception);
if (status == nullptr) {
// unkown status or exception occurred extracting status
m_env->ExceptionDescribe();
return rocksdb::Status::OK(); // TODO(AR) probably need a better error code here
} else {
m_env->ExceptionClear(); // clear the exception, as we have extracted the status
return rocksdb::Status(*status);
}
}
return rocksdb::Status::OK();
}
rocksdb::Status WriteBatchHandlerJniCallback::MarkRollback(const Slice& xid) {
auto markRollback = [this] (
jbyteArray j_xid) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jMarkRollbackMethodId,
j_xid);
};
auto status = WriteBatchHandlerJniCallback::k_op(xid, markRollback);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
}
}
rocksdb::Status WriteBatchHandlerJniCallback::MarkCommit(const Slice& xid) {
auto markCommit = [this] (
jbyteArray j_xid) {
m_env->CallVoidMethod(
m_jcallback_obj,
m_jMarkCommitMethodId,
j_xid);
};
auto status = WriteBatchHandlerJniCallback::k_op(xid, markCommit);
if(status == nullptr) {
return rocksdb::Status::OK(); // TODO(AR) what to do if there is an Exception but we don't know the rocksdb::Status?
} else {
return rocksdb::Status(*status);
if(j_blob != nullptr) {
m_env->DeleteLocalRef(j_blob);
}
}
@@ -409,101 +256,39 @@ bool WriteBatchHandlerJniCallback::Continue() {
return static_cast<bool>(jContinue == JNI_TRUE);
}
std::unique_ptr<rocksdb::Status> WriteBatchHandlerJniCallback::kv_op(const Slice& key, const Slice& value, std::function<void(jbyteArray, jbyteArray)> kvFn) {
const jbyteArray j_key = JniUtil::copyBytes(m_env, key);
if (j_key == nullptr) {
// exception thrown
if (m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
/*
* Creates a Java Byte Array from the data in a Slice
*
* When calling this function
* you must remember to call env->DeleteLocalRef
* on the result after you have finished with it
*
* @param s A Slice to convery to a Java byte array
*
* @return A reference to a Java byte array, or a nullptr if an
* exception occurs
*/
jbyteArray WriteBatchHandlerJniCallback::sliceToJArray(const Slice& s) {
// TODO(AR) move to JniUtil
jbyteArray ja = m_env->NewByteArray(static_cast<jsize>(s.size()));
if(ja == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
const jbyteArray j_value = JniUtil::copyBytes(m_env, value);
if (j_value == nullptr) {
// exception thrown
if (m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
if (j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
m_env->SetByteArrayRegion(
ja, 0, static_cast<jsize>(s.size()),
const_cast<jbyte*>(reinterpret_cast<const jbyte*>(s.data())));
if(m_env->ExceptionCheck()) {
if(ja != nullptr) {
m_env->DeleteLocalRef(ja);
}
// exception thrown: ArrayIndexOutOfBoundsException
return nullptr;
}
kvFn(j_key, j_value);
// check for Exception, in-particular RocksDBException
if (m_env->ExceptionCheck()) {
if (j_value != nullptr) {
m_env->DeleteLocalRef(j_value);
}
if (j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
// exception thrown
jthrowable exception = m_env->ExceptionOccurred();
std::unique_ptr<rocksdb::Status> status = rocksdb::RocksDBExceptionJni::toCppStatus(m_env, exception);
if (status == nullptr) {
// unkown status or exception occurred extracting status
m_env->ExceptionDescribe();
return nullptr;
} else {
m_env->ExceptionClear(); // clear the exception, as we have extracted the status
return status;
}
}
if (j_value != nullptr) {
m_env->DeleteLocalRef(j_value);
}
if (j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
// all OK
return std::unique_ptr<rocksdb::Status>(new rocksdb::Status(rocksdb::Status::OK()));
}
std::unique_ptr<rocksdb::Status> WriteBatchHandlerJniCallback::k_op(const Slice& key, std::function<void(jbyteArray)> kFn) {
const jbyteArray j_key = JniUtil::copyBytes(m_env, key);
if (j_key == nullptr) {
// exception thrown
if (m_env->ExceptionCheck()) {
m_env->ExceptionDescribe();
}
return nullptr;
}
kFn(j_key);
// check for Exception, in-particular RocksDBException
if (m_env->ExceptionCheck()) {
if (j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
// exception thrown
jthrowable exception = m_env->ExceptionOccurred();
std::unique_ptr<rocksdb::Status> status = rocksdb::RocksDBExceptionJni::toCppStatus(m_env, exception);
if (status == nullptr) {
// unkown status or exception occurred extracting status
m_env->ExceptionDescribe();
return nullptr;
} else {
m_env->ExceptionClear(); // clear the exception, as we have extracted the status
return status;
}
}
if (j_key != nullptr) {
m_env->DeleteLocalRef(j_key);
}
// all OK
return std::unique_ptr<rocksdb::Status>(new rocksdb::Status(rocksdb::Status::OK()));
return ja;
}
} // namespace rocksdb
+1 -38
View File
@@ -9,9 +9,7 @@
#ifndef JAVA_ROCKSJNI_WRITEBATCHHANDLERJNICALLBACK_H_
#define JAVA_ROCKSJNI_WRITEBATCHHANDLERJNICALLBACK_H_
#include <functional>
#include <jni.h>
#include <memory>
#include "rocksjni/jnicallback.h"
#include "rocksdb/write_batch.h"
@@ -27,57 +25,22 @@ class WriteBatchHandlerJniCallback : public JniCallback, public WriteBatch::Hand
public:
WriteBatchHandlerJniCallback(
JNIEnv* env, jobject jWriteBackHandler);
Status PutCF(uint32_t column_family_id, const Slice& key,
const Slice& value);
void Put(const Slice& key, const Slice& value);
Status MergeCF(uint32_t column_family_id, const Slice& key,
const Slice& value);
void Merge(const Slice& key, const Slice& value);
Status DeleteCF(uint32_t column_family_id, const Slice& key);
void Delete(const Slice& key);
Status SingleDeleteCF(uint32_t column_family_id, const Slice& key);
void SingleDelete(const Slice& key);
Status DeleteRangeCF(uint32_t column_family_id, const Slice& beginKey,
const Slice& endKey);
void DeleteRange(const Slice& beginKey, const Slice& endKey);
void LogData(const Slice& blob);
Status PutBlobIndexCF(uint32_t column_family_id, const Slice& key,
const Slice& value);
Status MarkBeginPrepare();
Status MarkEndPrepare(const Slice& xid);
Status MarkNoop(bool empty_batch);
Status MarkRollback(const Slice& xid);
Status MarkCommit(const Slice& xid);
bool Continue();
private:
JNIEnv* m_env;
jmethodID m_jPutCfMethodId;
jbyteArray sliceToJArray(const Slice& s);
jmethodID m_jPutMethodId;
jmethodID m_jMergeCfMethodId;
jmethodID m_jMergeMethodId;
jmethodID m_jDeleteCfMethodId;
jmethodID m_jDeleteMethodId;
jmethodID m_jSingleDeleteCfMethodId;
jmethodID m_jSingleDeleteMethodId;
jmethodID m_jDeleteRangeCfMethodId;
jmethodID m_jDeleteRangeMethodId;
jmethodID m_jLogDataMethodId;
jmethodID m_jPutBlobIndexCfMethodId;
jmethodID m_jMarkBeginPrepareMethodId;
jmethodID m_jMarkEndPrepareMethodId;
jmethodID m_jMarkNoopMethodId;
jmethodID m_jMarkRollbackMethodId;
jmethodID m_jMarkCommitMethodId;
jmethodID m_jContinueMethodId;
/**
* @return A pointer to a rocksdb::Status or nullptr if an unexpected exception occurred
*/
std::unique_ptr<rocksdb::Status> kv_op(const Slice& key, const Slice& value, std::function<void(jbyteArray, jbyteArray)> kvFn);
/**
* @return A pointer to a rocksdb::Status or nullptr if an unexpected exception occurred
*/
std::unique_ptr<rocksdb::Status> k_op(const Slice& key, std::function<void(jbyteArray)> kFn);
};
} // namespace rocksdb
@@ -1,184 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
import org.rocksdb.*;
import static java.nio.charset.StandardCharsets.UTF_8;
/**
* Demonstrates using Transactions on an OptimisticTransactionDB with
* varying isolation guarantees
*/
public class OptimisticTransactionSample {
private static final String dbPath = "/tmp/rocksdb_optimistic_transaction_example";
public static final void main(final String args[]) throws RocksDBException {
try(final Options options = new Options()
.setCreateIfMissing(true);
final OptimisticTransactionDB txnDb =
OptimisticTransactionDB.open(options, dbPath)) {
try (final WriteOptions writeOptions = new WriteOptions();
final ReadOptions readOptions = new ReadOptions()) {
////////////////////////////////////////////////////////
//
// Simple OptimisticTransaction Example ("Read Committed")
//
////////////////////////////////////////////////////////
readCommitted(txnDb, writeOptions, readOptions);
////////////////////////////////////////////////////////
//
// "Repeatable Read" (Snapshot Isolation) Example
// -- Using a single Snapshot
//
////////////////////////////////////////////////////////
repeatableRead(txnDb, writeOptions, readOptions);
////////////////////////////////////////////////////////
//
// "Read Committed" (Monotonic Atomic Views) Example
// --Using multiple Snapshots
//
////////////////////////////////////////////////////////
readCommitted_monotonicAtomicViews(txnDb, writeOptions, readOptions);
}
}
}
/**
* Demonstrates "Read Committed" isolation
*/
private static void readCommitted(final OptimisticTransactionDB txnDb,
final WriteOptions writeOptions, final ReadOptions readOptions)
throws RocksDBException {
final byte key1[] = "abc".getBytes(UTF_8);
final byte value1[] = "def".getBytes(UTF_8);
final byte key2[] = "xyz".getBytes(UTF_8);
final byte value2[] = "zzz".getBytes(UTF_8);
// Start a transaction
try(final Transaction txn = txnDb.beginTransaction(writeOptions)) {
// Read a key in this transaction
byte[] value = txn.get(readOptions, key1);
assert(value == null);
// Write a key in this transaction
txn.put(key1, value1);
// Read a key OUTSIDE this transaction. Does not affect txn.
value = txnDb.get(readOptions, key1);
assert(value == null);
// Write a key OUTSIDE of this transaction.
// Does not affect txn since this is an unrelated key.
// If we wrote key 'abc' here, the transaction would fail to commit.
txnDb.put(writeOptions, key2, value2);
// Commit transaction
txn.commit();
}
}
/**
* Demonstrates "Repeatable Read" (Snapshot Isolation) isolation
*/
private static void repeatableRead(final OptimisticTransactionDB txnDb,
final WriteOptions writeOptions, final ReadOptions readOptions)
throws RocksDBException {
final byte key1[] = "ghi".getBytes(UTF_8);
final byte value1[] = "jkl".getBytes(UTF_8);
// Set a snapshot at start of transaction by setting setSnapshot(true)
try(final OptimisticTransactionOptions txnOptions =
new OptimisticTransactionOptions().setSetSnapshot(true);
final Transaction txn =
txnDb.beginTransaction(writeOptions, txnOptions)) {
final Snapshot snapshot = txn.getSnapshot();
// Write a key OUTSIDE of transaction
txnDb.put(writeOptions, key1, value1);
// Read a key using the snapshot.
readOptions.setSnapshot(snapshot);
final byte[] value = txn.getForUpdate(readOptions, key1, true);
assert(value == value1);
try {
// Attempt to commit transaction
txn.commit();
throw new IllegalStateException();
} catch(final RocksDBException e) {
// Transaction could not commit since the write outside of the txn
// conflicted with the read!
assert(e.getStatus().getCode() == Status.Code.Busy);
}
txn.rollback();
} finally {
// Clear snapshot from read options since it is no longer valid
readOptions.setSnapshot(null);
}
}
/**
* Demonstrates "Read Committed" (Monotonic Atomic Views) isolation
*
* In this example, we set the snapshot multiple times. This is probably
* only necessary if you have very strict isolation requirements to
* implement.
*/
private static void readCommitted_monotonicAtomicViews(
final OptimisticTransactionDB txnDb, final WriteOptions writeOptions,
final ReadOptions readOptions) throws RocksDBException {
final byte keyX[] = "x".getBytes(UTF_8);
final byte valueX[] = "x".getBytes(UTF_8);
final byte keyY[] = "y".getBytes(UTF_8);
final byte valueY[] = "y".getBytes(UTF_8);
try (final OptimisticTransactionOptions txnOptions =
new OptimisticTransactionOptions().setSetSnapshot(true);
final Transaction txn =
txnDb.beginTransaction(writeOptions, txnOptions)) {
// Do some reads and writes to key "x"
Snapshot snapshot = txnDb.getSnapshot();
readOptions.setSnapshot(snapshot);
byte[] value = txn.get(readOptions, keyX);
txn.put(valueX, valueX);
// Do a write outside of the transaction to key "y"
txnDb.put(writeOptions, keyY, valueY);
// Set a new snapshot in the transaction
txn.setSnapshot();
snapshot = txnDb.getSnapshot();
readOptions.setSnapshot(snapshot);
// Do some reads and writes to key "y"
// Since the snapshot was advanced, the write done outside of the
// transaction does not conflict.
value = txn.getForUpdate(readOptions, keyY, true);
txn.put(keyY, valueY);
// Commit. Since the snapshot was advanced, the write done outside of the
// transaction does not prevent this transaction from Committing.
txn.commit();
} finally {
// Clear snapshot from read options since it is no longer valid
readOptions.setSnapshot(null);
}
}
}
@@ -1,183 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
import org.rocksdb.*;
import static java.nio.charset.StandardCharsets.UTF_8;
/**
* Demonstrates using Transactions on a TransactionDB with
* varying isolation guarantees
*/
public class TransactionSample {
private static final String dbPath = "/tmp/rocksdb_transaction_example";
public static final void main(final String args[]) throws RocksDBException {
try(final Options options = new Options()
.setCreateIfMissing(true);
final TransactionDBOptions txnDbOptions = new TransactionDBOptions();
final TransactionDB txnDb =
TransactionDB.open(options, txnDbOptions, dbPath)) {
try (final WriteOptions writeOptions = new WriteOptions();
final ReadOptions readOptions = new ReadOptions()) {
////////////////////////////////////////////////////////
//
// Simple Transaction Example ("Read Committed")
//
////////////////////////////////////////////////////////
readCommitted(txnDb, writeOptions, readOptions);
////////////////////////////////////////////////////////
//
// "Repeatable Read" (Snapshot Isolation) Example
// -- Using a single Snapshot
//
////////////////////////////////////////////////////////
repeatableRead(txnDb, writeOptions, readOptions);
////////////////////////////////////////////////////////
//
// "Read Committed" (Monotonic Atomic Views) Example
// --Using multiple Snapshots
//
////////////////////////////////////////////////////////
readCommitted_monotonicAtomicViews(txnDb, writeOptions, readOptions);
}
}
}
/**
* Demonstrates "Read Committed" isolation
*/
private static void readCommitted(final TransactionDB txnDb,
final WriteOptions writeOptions, final ReadOptions readOptions)
throws RocksDBException {
final byte key1[] = "abc".getBytes(UTF_8);
final byte value1[] = "def".getBytes(UTF_8);
final byte key2[] = "xyz".getBytes(UTF_8);
final byte value2[] = "zzz".getBytes(UTF_8);
// Start a transaction
try(final Transaction txn = txnDb.beginTransaction(writeOptions)) {
// Read a key in this transaction
byte[] value = txn.get(readOptions, key1);
assert(value == null);
// Write a key in this transaction
txn.put(key1, value1);
// Read a key OUTSIDE this transaction. Does not affect txn.
value = txnDb.get(readOptions, key1);
assert(value == null);
// Write a key OUTSIDE of this transaction.
// Does not affect txn since this is an unrelated key.
// If we wrote key 'abc' here, the transaction would fail to commit.
txnDb.put(writeOptions, key2, value2);
// Commit transaction
txn.commit();
}
}
/**
* Demonstrates "Repeatable Read" (Snapshot Isolation) isolation
*/
private static void repeatableRead(final TransactionDB txnDb,
final WriteOptions writeOptions, final ReadOptions readOptions)
throws RocksDBException {
final byte key1[] = "ghi".getBytes(UTF_8);
final byte value1[] = "jkl".getBytes(UTF_8);
// Set a snapshot at start of transaction by setting setSnapshot(true)
try(final TransactionOptions txnOptions = new TransactionOptions()
.setSetSnapshot(true);
final Transaction txn =
txnDb.beginTransaction(writeOptions, txnOptions)) {
final Snapshot snapshot = txn.getSnapshot();
// Write a key OUTSIDE of transaction
txnDb.put(writeOptions, key1, value1);
// Attempt to read a key using the snapshot. This will fail since
// the previous write outside this txn conflicts with this read.
readOptions.setSnapshot(snapshot);
try {
final byte[] value = txn.getForUpdate(readOptions, key1, true);
throw new IllegalStateException();
} catch(final RocksDBException e) {
assert(e.getStatus().getCode() == Status.Code.Busy);
}
txn.rollback();
} finally {
// Clear snapshot from read options since it is no longer valid
readOptions.setSnapshot(null);
}
}
/**
* Demonstrates "Read Committed" (Monotonic Atomic Views) isolation
*
* In this example, we set the snapshot multiple times. This is probably
* only necessary if you have very strict isolation requirements to
* implement.
*/
private static void readCommitted_monotonicAtomicViews(
final TransactionDB txnDb, final WriteOptions writeOptions,
final ReadOptions readOptions) throws RocksDBException {
final byte keyX[] = "x".getBytes(UTF_8);
final byte valueX[] = "x".getBytes(UTF_8);
final byte keyY[] = "y".getBytes(UTF_8);
final byte valueY[] = "y".getBytes(UTF_8);
try (final TransactionOptions txnOptions = new TransactionOptions()
.setSetSnapshot(true);
final Transaction txn =
txnDb.beginTransaction(writeOptions, txnOptions)) {
// Do some reads and writes to key "x"
Snapshot snapshot = txnDb.getSnapshot();
readOptions.setSnapshot(snapshot);
byte[] value = txn.get(readOptions, keyX);
txn.put(valueX, valueX);
// Do a write outside of the transaction to key "y"
txnDb.put(writeOptions, keyY, valueY);
// Set a new snapshot in the transaction
txn.setSnapshot();
txn.setSavePoint();
snapshot = txnDb.getSnapshot();
readOptions.setSnapshot(snapshot);
// Do some reads and writes to key "y"
// Since the snapshot was advanced, the write done outside of the
// transaction does not conflict.
value = txn.getForUpdate(readOptions, keyY, true);
txn.put(keyY, valueY);
// Decide we want to revert the last write from this transaction.
txn.rollbackToSavePoint();
// Commit.
txn.commit();
} finally {
// Clear snapshot from read options since it is no longer valid
readOptions.setSnapshot(null);
}
}
}
@@ -17,23 +17,10 @@ package org.rocksdb;
public abstract class AbstractComparator<T extends AbstractSlice<?>>
extends RocksCallbackObject {
protected AbstractComparator() {
super();
}
protected AbstractComparator(final ComparatorOptions copt) {
super(copt.nativeHandle_);
}
/**
* Get the type of this comparator.
*
* Used for determining the correct C++ cast in native code.
*
* @return The type of the comparator.
*/
abstract ComparatorType getComparatorType();
/**
* The name of the comparator. Used to check for comparator
* mismatches (i.e., a DB created with one comparator is
@@ -1,54 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* Provides notification to the caller of SetSnapshotOnNextOperation when
* the actual snapshot gets created
*/
public abstract class AbstractTransactionNotifier
extends RocksCallbackObject {
protected AbstractTransactionNotifier() {
super();
}
/**
* Implement this method to receive notification when a snapshot is
* requested via {@link Transaction#setSnapshotOnNextOperation()}.
*
* @param newSnapshot the snapshot that has been created.
*/
public abstract void snapshotCreated(final Snapshot newSnapshot);
/**
* This is intentionally private as it is the callback hook
* from JNI
*/
private void snapshotCreated(final long snapshotHandle) {
snapshotCreated(new Snapshot(snapshotHandle));
}
@Override
protected long initializeNative(final long... nativeParameterHandles) {
return createNewTransactionNotifier();
}
private native long createNewTransactionNotifier();
/**
* Deletes underlying C++ TransactionNotifier pointer.
*
* Note that this function should be called only after all
* Transactions referencing the comparator are closed.
* Otherwise an undefined behavior will occur.
*/
@Override
protected void disposeInternal() {
disposeInternal(nativeHandle_);
}
protected final native void disposeInternal(final long handle);
}
@@ -18,80 +18,52 @@ public abstract class AbstractWriteBatch extends RocksObject
}
@Override
public void put(byte[] key, byte[] value) throws RocksDBException {
public void put(byte[] key, byte[] value) {
put(nativeHandle_, key, key.length, value, value.length);
}
@Override
public void put(ColumnFamilyHandle columnFamilyHandle, byte[] key,
byte[] value) throws RocksDBException {
byte[] value) {
put(nativeHandle_, key, key.length, value, value.length,
columnFamilyHandle.nativeHandle_);
}
@Override
public void merge(byte[] key, byte[] value) throws RocksDBException {
public void merge(byte[] key, byte[] value) {
merge(nativeHandle_, key, key.length, value, value.length);
}
@Override
public void merge(ColumnFamilyHandle columnFamilyHandle, byte[] key,
byte[] value) throws RocksDBException {
byte[] value) {
merge(nativeHandle_, key, key.length, value, value.length,
columnFamilyHandle.nativeHandle_);
}
@Override
@Deprecated
public void remove(byte[] key) throws RocksDBException {
delete(nativeHandle_, key, key.length);
public void remove(byte[] key) {
remove(nativeHandle_, key, key.length);
}
@Override
@Deprecated
public void remove(ColumnFamilyHandle columnFamilyHandle, byte[] key)
throws RocksDBException {
delete(nativeHandle_, key, key.length, columnFamilyHandle.nativeHandle_);
public void remove(ColumnFamilyHandle columnFamilyHandle, byte[] key) {
remove(nativeHandle_, key, key.length, columnFamilyHandle.nativeHandle_);
}
@Override
public void delete(byte[] key) throws RocksDBException {
delete(nativeHandle_, key, key.length);
}
@Override
public void delete(ColumnFamilyHandle columnFamilyHandle, byte[] key)
throws RocksDBException {
delete(nativeHandle_, key, key.length, columnFamilyHandle.nativeHandle_);
}
@Override
public void singleDelete(byte[] key) throws RocksDBException {
singleDelete(nativeHandle_, key, key.length);
}
@Override
public void singleDelete(ColumnFamilyHandle columnFamilyHandle, byte[] key)
throws RocksDBException {
singleDelete(nativeHandle_, key, key.length, columnFamilyHandle.nativeHandle_);
}
@Override
public void deleteRange(byte[] beginKey, byte[] endKey)
throws RocksDBException {
public void deleteRange(byte[] beginKey, byte[] endKey) {
deleteRange(nativeHandle_, beginKey, beginKey.length, endKey, endKey.length);
}
@Override
public void deleteRange(ColumnFamilyHandle columnFamilyHandle,
byte[] beginKey, byte[] endKey) throws RocksDBException {
public void deleteRange(ColumnFamilyHandle columnFamilyHandle, byte[] beginKey, byte[] endKey) {
deleteRange(nativeHandle_, beginKey, beginKey.length, endKey, endKey.length,
columnFamilyHandle.nativeHandle_);
}
@Override
public void putLogData(byte[] blob) throws RocksDBException {
public void putLogData(byte[] blob) {
putLogData(nativeHandle_, blob, blob.length);
}
@@ -110,67 +82,38 @@ public abstract class AbstractWriteBatch extends RocksObject
rollbackToSavePoint0(nativeHandle_);
}
@Override
public void popSavePoint() throws RocksDBException {
popSavePoint(nativeHandle_);
}
@Override
public void setMaxBytes(final long maxBytes) {
setMaxBytes(nativeHandle_, maxBytes);
}
@Override
public WriteBatch getWriteBatch() {
return getWriteBatch(nativeHandle_);
}
abstract int count0(final long handle);
abstract void put(final long handle, final byte[] key, final int keyLen,
final byte[] value, final int valueLen) throws RocksDBException;
final byte[] value, final int valueLen);
abstract void put(final long handle, final byte[] key, final int keyLen,
final byte[] value, final int valueLen, final long cfHandle)
throws RocksDBException;
final byte[] value, final int valueLen, final long cfHandle);
abstract void merge(final long handle, final byte[] key, final int keyLen,
final byte[] value, final int valueLen) throws RocksDBException;
final byte[] value, final int valueLen);
abstract void merge(final long handle, final byte[] key, final int keyLen,
final byte[] value, final int valueLen, final long cfHandle)
throws RocksDBException;
final byte[] value, final int valueLen, final long cfHandle);
abstract void delete(final long handle, final byte[] key,
final int keyLen) throws RocksDBException;
abstract void remove(final long handle, final byte[] key,
final int keyLen);
abstract void delete(final long handle, final byte[] key,
final int keyLen, final long cfHandle) throws RocksDBException;
abstract void singleDelete(final long handle, final byte[] key,
final int keyLen) throws RocksDBException;
abstract void singleDelete(final long handle, final byte[] key,
final int keyLen, final long cfHandle) throws RocksDBException;
abstract void remove(final long handle, final byte[] key,
final int keyLen, final long cfHandle);
abstract void deleteRange(final long handle, final byte[] beginKey, final int beginKeyLen,
final byte[] endKey, final int endKeyLen) throws RocksDBException;
final byte[] endKey, final int endKeyLen);
abstract void deleteRange(final long handle, final byte[] beginKey, final int beginKeyLen,
final byte[] endKey, final int endKeyLen, final long cfHandle) throws RocksDBException;
final byte[] endKey, final int endKeyLen, final long cfHandle);
abstract void putLogData(final long handle, final byte[] blob,
final int blobLen) throws RocksDBException;
final int blobLen);
abstract void clear0(final long handle);
abstract void setSavePoint0(final long handle);
abstract void rollbackToSavePoint0(final long handle);
abstract void popSavePoint(final long handle) throws RocksDBException;
abstract void setMaxBytes(final long handle, long maxBytes);
abstract WriteBatch getWriteBatch(final long handle);
}
@@ -5,8 +5,6 @@
package org.rocksdb;
import java.util.Arrays;
/**
* <p>Describes a column family with a
* name and respective Options.</p>
@@ -34,7 +32,7 @@ public class ColumnFamilyDescriptor {
* @since 3.10.0
*/
public ColumnFamilyDescriptor(final byte[] columnFamilyName,
final ColumnFamilyOptions columnFamilyOptions) {
final ColumnFamilyOptions columnFamilyOptions) {
columnFamilyName_ = columnFamilyName;
columnFamilyOptions_ = columnFamilyOptions;
}
@@ -45,63 +43,17 @@ public class ColumnFamilyDescriptor {
* @return column family name.
* @since 3.10.0
*/
public byte[] getName() {
public byte[] columnFamilyName() {
return columnFamilyName_;
}
/**
* Retrieve name of column family.
*
* @return column family name.
* @since 3.10.0
*
* @deprecated Use {@link #getName()} instead.
*/
@Deprecated
public byte[] columnFamilyName() {
return getName();
}
/**
* Retrieve assigned options instance.
*
* @return Options instance assigned to this instance.
*/
public ColumnFamilyOptions getOptions() {
return columnFamilyOptions_;
}
/**
* Retrieve assigned options instance.
*
* @return Options instance assigned to this instance.
*
* @deprecated Use {@link #getOptions()} instead.
*/
@Deprecated
public ColumnFamilyOptions columnFamilyOptions() {
return getOptions();
}
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
final ColumnFamilyDescriptor that = (ColumnFamilyDescriptor) o;
return Arrays.equals(columnFamilyName_, that.columnFamilyName_)
&& columnFamilyOptions_.nativeHandle_ == that.columnFamilyOptions_.nativeHandle_;
}
@Override
public int hashCode() {
int result = (int) (columnFamilyOptions_.nativeHandle_ ^ (columnFamilyOptions_.nativeHandle_ >>> 32));
result = 31 * result + Arrays.hashCode(columnFamilyName_);
return result;
return columnFamilyOptions_;
}
private final byte[] columnFamilyName_;
@@ -5,9 +5,6 @@
package org.rocksdb;
import java.util.Arrays;
import java.util.Objects;
/**
* ColumnFamilyHandle class to hold handles to underlying rocksdb
* ColumnFamily Pointers.
@@ -24,63 +21,6 @@ public class ColumnFamilyHandle extends RocksObject {
this.rocksDB_ = rocksDB;
}
/**
* Gets the name of the Column Family.
*
* @return The name of the Column Family.
*/
public byte[] getName() {
return getName(nativeHandle_);
}
/**
* Gets the ID of the Column Family.
*
* @return the ID of the Column Family.
*/
public int getID() {
return getID(nativeHandle_);
}
/**
* Gets the up-to-date descriptor of the column family
* associated with this handle. Since it fills "*desc" with the up-to-date
* information, this call might internally lock and release DB mutex to
* access the up-to-date CF options. In addition, all the pointer-typed
* options cannot be referenced any longer than the original options exist.
*
* Note that this function is not supported in RocksDBLite.
*
* @return the up-to-date descriptor.
*
* @throws RocksDBException if an error occurs whilst retrieving the
* descriptor.
*/
public ColumnFamilyDescriptor getDescriptor() throws RocksDBException {
assert(isOwningHandle());
return getDescriptor(nativeHandle_);
}
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
final ColumnFamilyHandle that = (ColumnFamilyHandle) o;
return rocksDB_.nativeHandle_ == that.rocksDB_.nativeHandle_ &&
getID() == that.getID() &&
Arrays.equals(getName(), that.getName());
}
@Override
public int hashCode() {
return Objects.hash(getName(), getID(), rocksDB_.nativeHandle_);
}
/**
* <p>Deletes underlying C++ iterator pointer.</p>
*
@@ -96,9 +36,6 @@ public class ColumnFamilyHandle extends RocksObject {
}
}
private native byte[] getName(final long handle);
private native int getID(final long handle);
private native ColumnFamilyDescriptor getDescriptor(final long handle) throws RocksDBException;
@Override protected final native void disposeInternal(final long handle);
private final RocksDB rocksDB_;
@@ -33,18 +33,6 @@ public class ColumnFamilyOptions extends RocksObject
super(newColumnFamilyOptions());
}
/**
* <p>Constructor to be used by
* {@link #getColumnFamilyOptionsFromProps(java.util.Properties)},
* {@link ColumnFamilyDescriptor#columnFamilyOptions()}
* and also called via JNI.</p>
*
* @param handle native handle to ColumnFamilyOptions instance.
*/
ColumnFamilyOptions(final long handle) {
super(handle);
}
/**
* <p>Method to get a options instance by using pre-configured
* property values. If one or many values are undefined in
@@ -143,7 +131,7 @@ public class ColumnFamilyOptions extends RocksObject
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
assert (isOwningHandle());
setComparatorHandle(nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue());
comparator instanceof DirectComparator);
comparator_ = comparator;
return this;
}
@@ -780,6 +768,17 @@ public class ColumnFamilyOptions extends RocksObject
return forceConsistencyChecks(nativeHandle_);
}
/**
* <p>Constructor to be used by
* {@link #getColumnFamilyOptionsFromProps(java.util.Properties)}</p>
* and also called via JNI.
*
* @param handle native handle to ColumnFamilyOptions instance.
*/
public ColumnFamilyOptions(final long handle) {
super(handle);
}
private static native long getColumnFamilyOptionsFromProps(
String optString);
@@ -795,7 +794,7 @@ public class ColumnFamilyOptions extends RocksObject
long memtableMemoryBudget);
private native void setComparatorHandle(long handle, int builtinComparator);
private native void setComparatorHandle(long optHandle,
long comparatorHandle, byte comparatorType);
long comparatorHandle, boolean isDirect);
private native void setMergeOperatorName(long handle, String name);
private native void setMergeOperator(long handle, long mergeOperatorHandle);
private native void setCompactionFilterHandle(long handle,
@@ -25,10 +25,5 @@ public abstract class Comparator extends AbstractComparator<Slice> {
return createNewComparator0(nativeParameterHandles[0]);
}
@Override
final ComparatorType getComparatorType() {
return ComparatorType.JAVA_COMPARATOR;
}
private native long createNewComparator0(final long comparatorOptionsHandle);
}
@@ -1,49 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
enum ComparatorType {
JAVA_COMPARATOR((byte)0x0),
JAVA_DIRECT_COMPARATOR((byte)0x1),
JAVA_NATIVE_COMPARATOR_WRAPPER((byte)0x2);
private final byte value;
ComparatorType(final byte value) {
this.value = value;
}
/**
* <p>Returns the byte value of the enumerations value.</p>
*
* @return byte representation
*/
byte getValue() {
return value;
}
/**
* <p>Get the ComparatorType enumeration value by
* passing the byte identifier to this method.</p>
*
* @param byteIdentifier of ComparatorType.
*
* @return ComparatorType instance.
*
* @throws IllegalArgumentException if the comparator type for the byteIdentifier
* cannot be found
*/
static ComparatorType getComparatorType(final byte byteIdentifier) {
for (final ComparatorType comparatorType : ComparatorType.values()) {
if (comparatorType.getValue() == byteIdentifier) {
return comparatorType;
}
}
throw new IllegalArgumentException(
"Illegal value provided for ComparatorType.");
}
}
@@ -269,10 +269,7 @@ public interface DBOptionsInterface<T extends DBOptionsInterface> {
* Statistics objects should not be shared between DB instances as
* it does not use any locks to prevent concurrent updates.</p>
*
* @param statistics The statistics to set
*
* @return the instance of the current object.
*
* @see RocksDB#open(org.rocksdb.Options, String)
*/
T setStatistics(final Statistics statistics);
@@ -280,9 +277,7 @@ public interface DBOptionsInterface<T extends DBOptionsInterface> {
/**
* <p>Returns statistics object.</p>
*
* @return the instance of the statistics object or null if there is no
* statistics object.
*
* @return the instance of the statistics object or null if there is no statistics object.
* @see #setStatistics(Statistics)
*/
Statistics statistics();
@@ -25,11 +25,6 @@ public abstract class DirectComparator extends AbstractComparator<DirectSlice> {
return createNewDirectComparator0(nativeParameterHandles[0]);
}
@Override
final ComparatorType getComparatorType() {
return ComparatorType.JAVA_DIRECT_COMPARATOR;
}
private native long createNewDirectComparator0(
final long comparatorOptionsHandle);
}
@@ -1,57 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* A simple abstraction to allow a Java class to wrap a custom comparator
* implemented in C++.
*
* The native comparator must directly extend rocksdb::Comparator.
*/
public abstract class NativeComparatorWrapper
extends AbstractComparator<Slice> {
@Override
final ComparatorType getComparatorType() {
return ComparatorType.JAVA_NATIVE_COMPARATOR_WRAPPER;
}
@Override
public final String name() {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
@Override
public final int compare(final Slice s1, final Slice s2) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
@Override
public final String findShortestSeparator(final String start, final Slice limit) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
@Override
public final String findShortSuccessor(final String key) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
/**
* We override {@link RocksCallbackObject#disposeInternal()}
* as disposing of a native rocksd::Comparator extension requires
* a slightly different approach as it is not really a RocksCallbackObject
*/
@Override
protected void disposeInternal() {
disposeInternal(nativeHandle_);
}
private native void disposeInternal(final long handle);
}
@@ -1,175 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
import java.util.List;
/**
* Database with Transaction support.
*/
public class OptimisticTransactionDB extends RocksDB
implements TransactionalDB<OptimisticTransactionOptions> {
/**
* Private constructor.
*
* @param nativeHandle The native handle of the C++ OptimisticTransactionDB
* object
*/
private OptimisticTransactionDB(final long nativeHandle) {
super(nativeHandle);
}
/**
* Open an OptimisticTransactionDB similar to
* {@link RocksDB#open(Options, String)}.
*
* @param options {@link org.rocksdb.Options} instance.
* @param path the path to the rocksdb.
*
* @return a {@link OptimisticTransactionDB} instance on success, null if the
* specified {@link OptimisticTransactionDB} can not be opened.
*
* @throws RocksDBException if an error occurs whilst opening the database.
*/
public static OptimisticTransactionDB open(final Options options,
final String path) throws RocksDBException {
final OptimisticTransactionDB otdb = new OptimisticTransactionDB(open(
options.nativeHandle_, path));
// when non-default Options is used, keeping an Options reference
// in RocksDB can prevent Java to GC during the life-time of
// the currently-created RocksDB.
otdb.storeOptionsInstance(options);
return otdb;
}
/**
* Open an OptimisticTransactionDB similar to
* {@link RocksDB#open(DBOptions, String, List, List)}.
*
* @param dbOptions {@link org.rocksdb.DBOptions} instance.
* @param path the path to the rocksdb.
* @param columnFamilyDescriptors list of column family descriptors
* @param columnFamilyHandles will be filled with ColumnFamilyHandle instances
*
* @return a {@link OptimisticTransactionDB} instance on success, null if the
* specified {@link OptimisticTransactionDB} can not be opened.
*
* @throws RocksDBException if an error occurs whilst opening the database.
*/
public static OptimisticTransactionDB open(final DBOptions dbOptions,
final String path,
final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
final List<ColumnFamilyHandle> columnFamilyHandles)
throws RocksDBException {
final byte[][] cfNames = new byte[columnFamilyDescriptors.size()][];
final long[] cfOptionHandles = new long[columnFamilyDescriptors.size()];
for (int i = 0; i < columnFamilyDescriptors.size(); i++) {
final ColumnFamilyDescriptor cfDescriptor = columnFamilyDescriptors
.get(i);
cfNames[i] = cfDescriptor.columnFamilyName();
cfOptionHandles[i] = cfDescriptor.columnFamilyOptions().nativeHandle_;
}
final long[] handles = open(dbOptions.nativeHandle_, path, cfNames,
cfOptionHandles);
final OptimisticTransactionDB otdb =
new OptimisticTransactionDB(handles[0]);
// when non-default Options is used, keeping an Options reference
// in RocksDB can prevent Java to GC during the life-time of
// the currently-created RocksDB.
otdb.storeOptionsInstance(dbOptions);
for (int i = 1; i < handles.length; i++) {
columnFamilyHandles.add(new ColumnFamilyHandle(otdb, handles[i]));
}
return otdb;
}
@Override
public Transaction beginTransaction(final WriteOptions writeOptions) {
return new Transaction(this, beginTransaction(nativeHandle_,
writeOptions.nativeHandle_));
}
@Override
public Transaction beginTransaction(final WriteOptions writeOptions,
final OptimisticTransactionOptions optimisticTransactionOptions) {
return new Transaction(this, beginTransaction(nativeHandle_,
writeOptions.nativeHandle_,
optimisticTransactionOptions.nativeHandle_));
}
// TODO(AR) consider having beingTransaction(... oldTransaction) set a
// reference count inside Transaction, so that we can always call
// Transaction#close but the object is only disposed when there are as many
// closes as beginTransaction. Makes the try-with-resources paradigm easier for
// java developers
@Override
public Transaction beginTransaction(final WriteOptions writeOptions,
final Transaction oldTransaction) {
final long jtxn_handle = beginTransaction_withOld(nativeHandle_,
writeOptions.nativeHandle_, oldTransaction.nativeHandle_);
// RocksJava relies on the assumption that
// we do not allocate a new Transaction object
// when providing an old_txn
assert(jtxn_handle == oldTransaction.nativeHandle_);
return oldTransaction;
}
@Override
public Transaction beginTransaction(final WriteOptions writeOptions,
final OptimisticTransactionOptions optimisticTransactionOptions,
final Transaction oldTransaction) {
final long jtxn_handle = beginTransaction_withOld(nativeHandle_,
writeOptions.nativeHandle_, optimisticTransactionOptions.nativeHandle_,
oldTransaction.nativeHandle_);
// RocksJava relies on the assumption that
// we do not allocate a new Transaction object
// when providing an old_txn
assert(jtxn_handle == oldTransaction.nativeHandle_);
return oldTransaction;
}
/**
* Get the underlying database that was opened.
*
* @return The underlying database that was opened.
*/
public RocksDB getBaseDB() {
final RocksDB db = new RocksDB(getBaseDB(nativeHandle_));
db.disOwnNativeHandle();
return db;
}
protected static native long open(final long optionsHandle,
final String path) throws RocksDBException;
protected static native long[] open(final long handle, final String path,
final byte[][] columnFamilyNames, final long[] columnFamilyOptions);
private native long beginTransaction(final long handle,
final long writeOptionsHandle);
private native long beginTransaction(final long handle,
final long writeOptionsHandle,
final long optimisticTransactionOptionsHandle);
private native long beginTransaction_withOld(final long handle,
final long writeOptionsHandle, final long oldTransactionHandle);
private native long beginTransaction_withOld(final long handle,
final long writeOptionsHandle,
final long optimisticTransactionOptionsHandle,
final long oldTransactionHandle);
private native long getBaseDB(final long handle);
@Override protected final native void disposeInternal(final long handle);
}
@@ -1,53 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
public class OptimisticTransactionOptions extends RocksObject
implements TransactionalOptions {
public OptimisticTransactionOptions() {
super(newOptimisticTransactionOptions());
}
@Override
public boolean isSetSnapshot() {
assert(isOwningHandle());
return isSetSnapshot(nativeHandle_);
}
@Override
public OptimisticTransactionOptions setSetSnapshot(
final boolean setSnapshot) {
assert(isOwningHandle());
setSetSnapshot(nativeHandle_, setSnapshot);
return this;
}
/**
* Should be set if the DB has a non-default comparator.
* See comment in
* {@link WriteBatchWithIndex#WriteBatchWithIndex(AbstractComparator, int, boolean)}
* constructor.
*
* @param comparator The comparator to use for the transaction.
*
* @return this OptimisticTransactionOptions instance
*/
public OptimisticTransactionOptions setComparator(
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
assert(isOwningHandle());
setComparator(nativeHandle_, comparator.nativeHandle_);
return this;
}
private native static long newOptimisticTransactionOptions();
private native boolean isSetSnapshot(final long handle);
private native void setSetSnapshot(final long handle,
final boolean setSnapshot);
private native void setComparator(final long handle,
final long comparatorHandle);
@Override protected final native void disposeInternal(final long handle);
}
+2 -2
View File
@@ -170,7 +170,7 @@ public class Options extends RocksObject
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
assert(isOwningHandle());
setComparatorHandle(nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue());
comparator instanceof DirectComparator);
comparator_ = comparator;
return this;
}
@@ -1734,7 +1734,7 @@ public class Options extends RocksObject
long memtableMemoryBudget);
private native void setComparatorHandle(long handle, int builtinComparator);
private native void setComparatorHandle(long optHandle,
long comparatorHandle, byte comparatorType);
long comparatorHandle, boolean isDirect);
private native void setMergeOperatorName(
long handle, String name);
private native void setMergeOperator(
Loaded 100 of 193 files, more files were not shown because too many files have changed in this diff. Show more